Machine synaesthesia enabled multi-modal intelligent channel modeling method

Abstract

The application discloses a machine transduction enabled multi-modal intelligent channel modeling method, and belongs to the technical field of wireless communication. By designing a scatterer identification algorithm, the nonlinear mapping relationship between the perception data LiDAR point cloud and electromagnetic propagation scatterers is intelligently analyzed and modeled, and a multi-modal intelligent channel model is constructed, which contains the number, distance, angle, time delay and power parameters of clusters, effectively captures the non-stationary and consistency of the wireless communication environment-channel, and can provide a high-precision channel modeling scheme for the algorithm design and performance evaluation of a communication system.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology and relates to wireless communication system channel modeling technology. Specifically, it relates to a multimodal intelligent channel modeling method enabled by machine synesthesia. The method is a multimodal intelligent channel modeling method assisted by a light detection and ranging (LiDAR) point cloud and enabled by machine synesthesia (SoM). The method can provide a high-precision channel modeling solution for algorithm design and performance evaluation of communication systems. Background Art

[0002] Vehicle-to-vehicle (V2V) communication plays an increasingly important role in intelligent transportation systems (ITS). To improve the safety and efficiency of ITS, sixth-generation (6G) V2V communication requires a comprehensive understanding of the vehicle's surrounding environment and advanced V2V channel modeling research. Existing V2V channel models are mainly divided into two categories: statistical models and deterministic models. Statistical models randomly generate and determine channel parameters through statistical methods, which has low complexity, but their accuracy is limited due to the random generation and determination of parameters. Unlike statistical models, deterministic models aim to reproduce the physical propagation process of radio electromagnetic waves in specific scenarios, with high accuracy but huge computational overhead. Therefore, these two models struggle to achieve a good balance between accuracy and complexity and cannot effectively support the design requirements of 6G V2V systems in ITS.

[0003] With the development of artificial intelligence (AI) and advanced technologies, a series of AI-based 6G V2V channel models have been proposed. AI's powerful ability to process massive amounts of data and recognize complex patterns has significantly enhanced the balance between channel modeling accuracy and complexity. Existing work includes the proposed multipath component (MPC) clustering method, which tracks MPCs based on Kalman filtering and the Kuhn-Munkres algorithm, can identify the dynamic behavior of MPCs, and divide them into time-varying clusters. Other work has proposed a fast angle-of-arrival (AoA) recognition model based on machine learning for high-mobility V2V channels. In addition, artificial neural networks (ANNs) have also been used for channel modeling, demonstrating powerful information processing capabilities. Recent work has used ANNs to propose V2V channel models for path loss and shadowing effects, effectively leveraging the information processing potential of ANNs. Other work combines generative adversarial networks (GANs) and feedforward neural networks (FNNs) to model the probability density function (PDF) and power distribution of received signals. Although AI-based V2V channel models have achieved significant improvements in accuracy, these models rely only on single radio frequency (RF) communication information and ignore the mapping relationship between the physical environment and the electromagnetic space. This limitation exacerbates the model's over-reliance on data sets and data collection scenarios, resulting in low scalability of the model. To overcome this limitation, a deeper understanding of the mapping relationship between the electromagnetic space and the physical environment is needed to achieve more accurate, universal, and low-complexity V2V channel modeling.

[0004] Integrated Sensing and Communications (ISAC) technology provides a new opportunity for more accurate, versatile, and low-complexity V2V channel modeling. ISAC, which provides deeper insight into the electromagnetic space and physical environment, has attracted widespread attention in 6G V2V communications, and the emergence of ISAC has opened up new possibilities for channel modeling. Furthermore, 6G autonomous vehicles in ITS will be equipped with multimodal devices, including millimeter-wave radar, LiDAR, RGB-D cameras, and communication equipment. Compared to single RF radar perception, non-RF sensing devices (such as LiDAR) can capture more diverse environmental features, which can be used to improve communication performance and perception functions. To fully utilize the multimodal information of communication devices and various sensing devices, existing work has proposed the new concept of "machine synesthesia" (SoM) based on the phenomenon of "human synesthesia," clarifying the goal of achieving intelligent multimodal perception and communication integration through artificial neural networks. Inspired by SoM, current research has explored the mapping relationship between sensory data and large-scale channel fading (path loss). However, the mapping relationship between sensory data and small-scale channel fading (multipath fading) remains relatively preliminary. Therefore, there is an urgent need to use artificial neural networks to analyze and model the nonlinear mapping relationship between sensory data and electromagnetic propagation scatterers, achieving more accurate, universal, and low-complexity V2V channel modeling. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned existing technologies, the present invention proposes a multimodal intelligent channel modeling method enabled by machine synesthesia (SoM). By analyzing and modeling the nonlinear mapping relationship between the perception data LiDAR point cloud and the electromagnetic propagation scatterers, a multimodal intelligent channel model is constructed. It can effectively capture the wireless communication environment - channel non-stationarity and consistency. The designed model provides high-precision channel modeling support for communication system algorithm design and performance evaluation.

[0006] The core of this invention is a multimodal intelligent channel modeling method assisted by LiDAR point clouds, a machine synesthesia (SoM). Leveraging a convolutional neural network (SegNet) and electromagnetic propagation mechanisms, a scatterer recognition (ScaR) algorithm is proposed to intelligently analyze and model the nonlinear mapping relationship between LiDAR point clouds and electromagnetic propagation scatterers. This method constructs a multimodal intelligent channel model that captures both environmental and channel nonstationarity and consistency. In this proposed multimodal intelligent channel model, a communication and perception intelligent fusion dataset covering scenarios with high, medium, and low mobile agent densities (MADs) is constructed using two highly reliable simulation platforms—Wireless InSite and AirSim. The LiDAR point cloud is subjected to ground point removal, height threshold filtering, and visible area filtering, and physical environment grid features, including point cloud density, height, and distance information, are extracted. Furthermore, the ScaR algorithm, taking into account electromagnetic propagation mechanisms, innovatively analyzes and models the complex mapping relationship between the LiDAR point cloud grid features and the electromagnetic propagation scatterer grid map. In order to model the non-stationarity and consistency of the environment-channel, the physical environment features obtained from the LiDAR point cloud are used in combination with the mapping relationship to identify the scatterer grid map, thereby modeling the spatial properties of the scatterers, namely the quantity and position. With the help of the complex mapping relationship between the LiDAR point cloud and the scatterers parsed and modeled by the ScaR algorithm, multimodal intelligent channel modeling assisted by the LiDAR point cloud enabled by SoM is realized. The multimodal intelligent channel model proposed in this invention has the ability to model and capture the non-stationarity and consistency of the wireless channel, thereby providing a high-precision channel model for the algorithm design, channel simulation and performance evaluation of the communication system. That is, by modeling the statistical characteristics of the channel, it helps designers evaluate the performance of the communication system and then optimize the communication system, including modulation scheme, signal coding, power control, etc.

[0007] To achieve the above objectives, the present invention designs a multimodal intelligent channel modeling method assisted by LiDAR point cloud, which is enabled by SoM. The method comprises the following steps: 1. constructing a communication and perception intelligent fusion dataset based on the Wireless InSite and AirSim simulation platforms; 2. proposing a ScaR algorithm to parse and model the complex mapping relationship between the LiDAR point cloud of perception data and electromagnetic propagation scatterers; 3. using the complex mapping relationship between the LiDAR point cloud and scatterers parsed and modeled by the ScaR algorithm, modeling the environment-channel non-stationarity and consistency, and constructing a LiDAR point cloud-assisted multimodal intelligent channel model, which includes the number of clusters, distance, angle, delay, and power parameters; 4. obtaining the channel impulse response and channel transfer function, and introducing a cluster frequency correlation factor to characterize the frequency correlation of the path gain within the cluster.

[0008] The SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel modeling method of the present invention is as follows:

[0009] 1) Achieve deep integration and precise matching of the Wireless InSite and AirSim simulation platforms to build a high-fidelity communication and perception intelligent fusion dataset, including LiDAR point cloud and electromagnetic propagation scatterer information.

[0010] 2) The LiDAR point clouds at the transmitting and receiving ends are fused to obtain complete physical environment feature information, and the ground points are further removed and height threshold filtering and visible area filtering are performed to reduce the redundant information of the LiDAR point cloud while extracting the part of the point cloud related to electromagnetic propagation.

[0011] 3) Considering the main factors affecting scatterers, including object position, material, and angle, rasterized features are extracted from the effective LiDAR point cloud preprocessed in step 2) in terms of density, height, and distance from the transceiver.

[0012] 4) Design the ScaR algorithm to parse and model the complex mapping relationship between the perception data LiDAR point cloud and electromagnetic propagation scatterers, obtain the number of scatterers in each LiDAR point cloud grid, and achieve high-resolution and high-precision scatterer prediction.

[0013] 5) At the initial moment, based on the complex mapping relationship parsed and modeled in step 4), the LiDAR point cloud is used to identify the electromagnetic propagation scatterer grid map, and the grid map is converted into scatterer point coordinates. At the same time, dynamic scatterers are distinguished from static scatterers, and the K-Means clustering algorithm is further used to obtain dynamic clusters and static clusters.

[0014] 6) At the next moment t n+1 =t n +Δt, as the physical environment changes, based on the complex mapping relationship analyzed and modeled in step 4), combined with the LiDAR point cloud data at the current moment, the scatterers at the current moment are identified, and dynamic scatterers and static scatterers are distinguished to obtain dynamic clusters and static clusters.

[0015] 7) Construct a multimodal intelligent channel model assisted by SoM-enabled LiDAR point cloud to model the number, distance, angle, delay, and delay parameters of clusters with high precision.

[0016] Furthermore, virtual links are introduced to characterize the complex processes between twin clusters (twin dynamic clusters and twin static clusters), and step 6) is continuously repeated. The ScaR algorithm is used to identify scatterers based on the LiDAR point cloud of perception data, and the channel impulse response and channel transfer function at each moment are obtained. At the same time, frequency-related factors are introduced to model the frequency-related path gain, providing a reliable simulation platform for performance evaluation and verification of system-level algorithms.

[0017] Through the above steps, we can achieve SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel modeling and model the non-stationarity and consistency of the environment-channel, thereby providing a reliable simulation platform for the performance evaluation and verification of system-level algorithms.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] This invention provides a SoM-enabled multimodal intelligent channel modeling method and proposes a highly accurate SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel model. Using two highly reliable simulation platforms, WirelessInSite and AirSim, a communication and perception intelligent fusion dataset with high, medium, and low MADs is constructed. The LiDAR point cloud is de-redundant, and physical environment grid features are innovatively extracted. Furthermore, a ScaR algorithm is proposed, utilizing a convolutional neural network (SegNet) and incorporating electromagnetic propagation mechanisms to deeply analyze and model the nonlinear mapping relationship between the perception data LiDAR point cloud and electromagnetic propagation scatterers. Based on this analyzed and modeled mapping relationship, scatterers are identified using the LiDAR point cloud, their spatial properties are derived, and the nonstationary and consistent nature of the environment-channel is captured, leading to the construction of a SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel model. This channel model enables the successful design of SoM systems and provides a reliable simulation platform for performance evaluation and verification of system-level algorithms.

[0020] The SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel model proposed in this invention has the following technical advantages:

[0021] (1) Based on the intelligent fusion dataset of communication and perception data under high, medium, and low MADs, a SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel model is constructed. It models the channel statistical characteristics with high accuracy and is applicable to scenarios under different MADs.

[0022] (2) Using the electromagnetic propagation mechanism and the ScaR algorithm, we innovatively process LiDAR point clouds and analyze and model the nonlinear mapping relationship between LiDAR point clouds and electromagnetic propagation scatterers;

[0023] (3) Based on the parsed and modeled mapping relationship, the LiDAR point cloud is used to identify electromagnetic propagation scatterers, and the number and position of scatterers are modeled with high resolution and high precision to capture the non-stationarity and consistency of the environment-channel, and to construct a multimodal intelligent channel model assisted by the SoM-enabled LiDAR point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention analyzes and models the mapping relationship between the LiDAR point cloud and the electromagnetic propagation scatterer to construct the ScaR algorithm architecture.

[0025] Figure 2 It is the geometric structure of the multimodal channel model proposed in this invention;

[0026] Among them, the distance between the transmitting and receiving ends is D tcv (t), the distance at the initial moment is D tcv (t0), the communication center frequency between the transmitter and receiver is f c , the kth static cluster close to the transmitter / receiver and the lth dynamic cluster close to the transmitter / receiver are defined as and The velocity vector of the dynamic cluster is The speeds of the transmitter and receiver are ν T (t) / ν R (t). DETAILED DESCRIPTION

[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] The communication and perception intelligent fusion dataset constructed by the present invention is based on two high-fidelity simulation softwares, Wireless InSite and AirSim. The dataset constructed by the present invention method contains low, medium and high mobile agent density MADs, including high-precision LiDAR point cloud and scatterer information. Based on this dataset, the LiDAR point cloud is processed in combination with the electromagnetic propagation mechanism and the ScaR algorithm is used. The network architecture is as follows: Figure 1 As shown in the figure, the density, height and distance information of the LiDAR point cloud are input. The network structure includes the SegNet backbone network and the designed task-specific modules. The output is a scatterer grid map, which can parse and model the complex mapping relationship between the perception data LiDAR point cloud and the electromagnetic propagation scatterers. Figure 2 The geometric architecture of the constructed SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel model is given.

[0029] The specific steps of the SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel modeling method proposed in the present invention are as follows:

[0030] S10: Achieve deep integration and precise matching between the two high-precision simulation platforms, Wireless InSite and AirSim, to build a high-fidelity communication and perception intelligent fusion dataset, including high-precision LiDAR point cloud and scatterer information.

[0031] S20: The LiDAR point clouds from the transmitter and receiver are integrated to obtain the physical environment information around the transmitter and receiver. Ground points are further removed and height threshold filtering and visible area filtering are performed to reduce the redundant information of the LiDAR point cloud while extracting the part of the point cloud related to electromagnetic propagation.

[0032] S30: Considering the main factors affecting scatterers, including object position, material, and angle, raster feature extraction is performed on the pre-processed valid LiDAR point cloud in terms of density, height, and distance from the transceiver.

[0033] S40: Design the ScaR algorithm to parse and model the complex mapping relationship between the perception data LiDAR point cloud and electromagnetic propagation scatterers, obtain the number of scatterers in each LiDAR point cloud grid, and achieve high-resolution and high-precision scatterer prediction.

[0034] S50: At the initial moment, based on the complex mapping relationship parsed and modeled by the ScaR algorithm in step S40, the LiDAR point cloud is used to identify the electromagnetic propagation scatterer grid map, and the grid map is converted into scatterer point coordinates. At the same time, dynamic scatterers and static scatterers are distinguished, and the K-Means clustering algorithm is further used to obtain dynamic clusters and static clusters.

[0035] S60: At the next moment t n+1 =t n +Δt, as the physical environment changes, based on the complex mapping relationship parsed and modeled by the ScaR algorithm in step S40, the LiDAR point cloud at that moment is used to complete the scatterer identification at that moment, and distinguish dynamic scatterers from static scatterers to obtain dynamic clusters and static clusters in the electromagnetic space.

[0036] S70: Build a multimodal intelligent channel model powered by SoM and assisted by LiDAR point clouds, accurately modeling the number, distance, angle, delay, and delay parameters of clusters in electromagnetic space. Virtual links are used to depict the complex transmission process between twin clusters in electromagnetic space. Frequency-dependent factors are introduced to model frequency-dependent path gains. Step S60 is repeated and iterated to obtain the channel impulse response and channel transfer function at each moment.

[0037] In step S10, a high-fidelity communication and perception intelligent fusion dataset is constructed using two high-precision simulation platforms, Wireless InSite and AirSim. This step includes the following processes S11-S12:

[0038] S11: Achieve deep integration of Wireless InSite and AirSim simulation platforms.

[0039] To construct simulation scenarios, the Wireless InSite and AirSim simulation platforms must be deeply integrated. The scene is rendered with high precision in AirSim, while being appropriately simplified in Wireless InSite to suit modeling requirements. When setting the trajectory of mobile agents, the coordinates of the mobile agents in both Wireless InSite and AirSim must be modified simultaneously, generating a dynamic scenario that deeply integrates communication and perception. Specifically, WirelessInSite requires parsing numerous internal files to batch adjust the coordinates of the mobile agents, while AirSim requires setting the position of each mobile agent frame by frame.

[0040] S12: Achieve precise matching between Wireless InSite and AirSim simulation platforms.

[0041] When constructing the simulation scene, it's crucial to ensure precise alignment of every object between the Wireless InSite and AirSim simulation platforms. Specifically, each object on both platforms needs to be individually drawn and built, ensuring they match in size, position, and scale. For mobile agent trajectory settings, it's crucial to ensure that each mobile agent's trajectory is perfectly synchronized between the Wireless InSite and AirSim platforms. This means ensuring that each mobile agent's position on both platforms is identical at every moment, ensuring precise matching of their trajectories.

[0042] In step S20: the LiDAR point cloud of the transmitting and receiving ends is integrated to remove the redundant information of the LiDAR point cloud data. This step includes the following processes S21 to S23:

[0043] S21: Based on the three-dimensional coordinates of the LiDAR point cloud at the transmitter and the three-dimensional coordinates of the LiDAR point cloud at the receiver, as well as the coordinates and movement direction of the intelligent body, point cloud registration is performed to place the LiDAR point clouds of the transmitter and receiver in the same world coordinate system, thereby obtaining complete physical environment information around the transmitter and receiver.

[0044] S22: Based on the spliced ​​and fused LiDAR point clouds of the transmitter and receiver, determine and delete ground points with a vertical coordinate of zero in the LiDAR point cloud to reduce the data redundancy of the LiDAR point cloud.

[0045] S23: A height threshold filtering method is introduced, that is, the LiDAR point cloud with a height greater than a certain threshold is removed, and the visible area is further introduced to filter the point cloud. The present invention sets the visible area as an ellipsoid with the transceiver as the focus, and extracts the point cloud within the ellipsoid as the preprocessed valid LiDAR point cloud, thereby screening out the LiDAR point cloud part that is most relevant to electromagnetic propagation.

[0046] In step S30: physical environment grid feature extraction is performed on the pre-processed valid LiDAR point cloud. This step includes the following processes S31 to S33.

[0047] S31: Extract the density grid features of the LiDAR point cloud. The value in each grid is the ratio of the total number of LiDAR point clouds in the grid to the area of ​​the grid, calculated as

[0048]

[0049] Among them, S grid (t) represents the LiDAR point cloud set in the grid, |·| represents the number of elements in the set, and x grid (t) and y grid (t) represents the length and width of the grid, respectively, x grid (t)×y grid (t) represents the area of ​​the grid.

[0050] S32: Extract the height grid features of the LiDAR point cloud. The value in each grid is the maximum value of the LiDAR point cloud height in that grid, calculated as

[0051]

[0052] in, For S grid Coordinates of the LiDAR point cloud along the z-axis in (t).

[0053] S33: Extract the distance grid features of the LiDAR point cloud. Consider a position encoding layer. For the grid with LiDAR point cloud, the distance is calculated as the average distance between the center of the grid and the communication transceiver, that is,

[0054] d grid (t)=(d Tg (t)+d Rg (t)) / 2

[0055] Among them, dTg (t) is the distance between the center of the grid and the communication transmitter, d Rg (t) is the distance between the center of the grid and the communication receiving end.

[0056] In step S40, a ScaR algorithm (scatterer recognition algorithm, or scatterer prediction network model) is designed to analyze and model the complex mapping relationship between the LiDAR point cloud data and electromagnetic propagation scatterers. This allows the spatial information of electromagnetic propagation scatterers to be predicted from the LiDAR point cloud data. This step includes the following processes S41-S43.

[0057] S41: The input of the ScaR algorithm is the density, height, and distance grid features of the valid LiDAR point cloud extracted in S30. The dimension of one sample is 3×80×80, that is, three channels, and the dimension of each channel is 80×80.

[0058] S42: The output of the ScaR algorithm is a grid map of scatterers within the visible area. Its dimension represents the degree of precision of the scatterer position prediction and can be selected according to actual needs.

[0059] S43: The hidden layer structure of the ScaR algorithm consists of two parts, the convolutional neural network SegNet and the task-specific module. SegNet adopts an encoder-decoder structure. The encoder extracts features through a series of convolutional layers, and the decoder restores the feature map into a segmentation map of the same size as the input image through deconvolution (transposed convolution). The decoder part of SegNet uses the pooling index information from the encoder pooling layer, which helps to restore the spatial resolution more accurately, especially in detail recovery. The task-specific module includes an image interpolation layer, a convolution layer, a batch normalization layer and an activation function, wherein the image interpolation layer is implemented using the F.interpolate function in PyTorch, and the activation function uses the ReLu function. The task-specific module adjusts the output dimension of the task head of SegNet to the dimension required by the present invention, thereby realizing the output of the scatterer grid map.

[0060] In step S50 , at the initial moment, the LiDAR point cloud is used to identify electromagnetic propagation scatterers, and further obtain and distinguish dynamic and static clusters. This step includes the following processes S51 to S52 .

[0061] S51: Identify electromagnetic propagation scatterers using perception data LiDAR point cloud. Based on the complex mapping relationship between the perception data LiDAR point cloud and the electromagnetic propagation scatterers resolved and modeled in step S40, the identification of electromagnetic propagation scatterers is completed based on the LiDAR point cloud at the initial time. The identified scatterer grid map is converted into a scatterer coordinate, and information including the number, time delay, and angle parameters of the scatterer can be obtained. Therefore, the scatterers identified based on the perception data LiDAR point cloud facilitate the construction of a multi-modal intelligent channel model with accurate channel parameters.

[0062] S52: Obtain and distinguish dynamic and static clusters. First, according to the height grid features of the LiDAR point cloud, since the height of the vehicle is small, if the value in the height grid of the LiDAR point cloud is less than a set height threshold, it is detected as a mobile agent; otherwise, the LiDAR point cloud grid is detected as a static object, such as a tree or a building. When a scatterer is located in the grid of a detected mobile agent, it is a dynamic scatterer; when a scatterer is located in the grid of a detected static object, it is a static scatterer. Further, the K-Means clustering algorithm is used for dynamic scatterers and static scatterers respectively to obtain dynamic clusters and static clusters in the electromagnetic space.

[0063] In step S60: At the next time t n+1 = t n + Δt, based on the complex mapping relationship resolved and modeled in step S40 using the ScaR algorithm, the scatterer identification at this time is completed using the LiDAR point cloud at this time, and the dynamic scatterers, static scatterers, dynamic clusters, and static clusters are distinguished. This step includes the following processes S61-S62.

[0064] S61: Identify the scatterers at the next time t n+1 = t n + Δt. Since the mapping relationship captures the close interaction between the physical environment and the electromagnetic space, the predicted electromagnetic propagation scatterers will change continuously and consistently with the LiDAR point cloud, thereby modeling the environment-channel non-stationarity in the time domain. In addition, the perception data LiDAR point cloud at adjacent times is similar, and the identified scatterers at adjacent times are also similar, thereby capturing the environment-channel consistency in the time domain.

[0065] S62: Distinguish the dynamic and static clusters at the next time t n+1 = t n + Δt. Similar to step S52, for the time t n+1 = t nThe height grid features of the LiDAR point cloud at +Δt are used to identify mobile agents and static objects in the physical environment. A scatterer is considered dynamic if it falls within the grid of a mobile agent; it is considered static if it falls within the grid of a static object. Furthermore, a K-Means clustering algorithm is applied to dynamic and static scatterers, respectively, to obtain dynamic and static clusters in the electromagnetic space.

[0066] In step S70, a multimodal intelligent channel model is constructed, powered by SoM and assisted by LiDAR point clouds. The number, distance, angle, delay, and delay parameters of the clusters are modeled with high precision. Virtual links are used to characterize the complex transmission process between twin clusters in electromagnetic space, and frequency-dependent factors are introduced to model frequency-dependent path gain.

[0067] S71: Build an architecture for wireless channel model embedded in the environment based on twin clusters, such as Figure 2 Specifically, the distance between the transmitter and receiver is D tcv (t), the distance at the initial moment is D tcv (t0), the communication center frequency between the transmitter and receiver is f c , the communication bandwidth is B. The number of static scatterers and dynamic scatterers is I s (t0) and I d (t0), the position and distance parameter D of the i / jth static / dynamic scatterer s,i (t) / D d,j (t) can be directly obtained from the predicted scatterers. The angular parameters of the scatterers are the horizontal departure angle (Azimuth Angle of Departure, AAoD) φ s,i (t) / φ d,j (t), horizontal angle of arrival (Azimuth Angle of Arrival, AAoA) ψ s,i (t) / ψ d,j (t), Elevation Angle of Departure (EAoD)ζ s,i (t) / ζ d,j (t), Elevation Angle of Arrival (EAoA) Generated by Gaussian distribution. Then, the static scatterers and dynamic scatterers are clustered using K-Means algorithm to obtain static clusters and dynamic clusters, respectively. The number of static clusters and dynamic clusters is J s (t0) and J d(t0) is generated by Logistic distribution. A part of the clusters close to the transmitter and the clusters close to the receiver are randomly matched to form twin clusters. There will be an abstract virtual link between the twin clusters for communication transmission, which represents the possible existence of other clusters or reflection and scattering components above the second order. This model defines the kth static cluster close to the transmitter / receiver and the lth dynamic cluster close to the transmitter / receiver as and The velocity vector of the dynamic cluster is The speeds of the transmitter and receiver are ν T (t) / ν R Furthermore, a virtual link approach is used to characterize the complex transmission process between twin clusters in electromagnetic space.

[0068] S72: Determine the channel impulse response. First, the complex channel gain of the Line-of-Sight (LoS) component of the present invention is given:

[0069]

[0070] Among them, R(t) is the rectangular window function, that is,

[0071]

[0072] Doppler frequency, phase shift and time delay are calculated as

[0073]

[0074] Among them, <·> represents the inner product, represents the phase shift at the initial moment, λ is the carrier wavelength, and c is the speed of light. Secondly, for the ground reflection component, the complex channel gain can be expressed as

[0075]

[0076] Among them, P GR (t), f GR,T / R (t) and are the power of the ground reflection component, the Doppler frequency and phase offset of the transmitter / receiver respectively. kth i The complex channel gain of a scatterer is expressed as

[0077]

[0078] in, is the normalized static scatterer power, and the Doppler frequency can be expressed as

[0079]

[0080] in, Indicates the transmitter / receiver and static cluster kth i The distance between the scatterers. The time delay can be calculated as

[0081]

[0082] Among them, τ i ′(t) represents the static twin cluster and The virtual delay of the virtual link between them follows an exponential distribution. The phase offset can be calculated as

[0083]

[0084] in, represents the phase shift at the initial moment.

[0085] Finally, similar to static clusters Through dynamic cluster The first j The complex channel gain of a dynamic scatterer can be expressed as

[0086]

[0087] in, is the normalized dynamic scatterer power, and the Doppler frequency can be expressed as

[0088]

[0089] Among them, represents the transmitter / receiver and the dynamic cluster The first j The distance between the dynamic scatterers. The time delay can be calculated as

[0090]

[0091] Among them, τ′ j (t) represents the static twin cluster and The virtual delay of the virtual link between them follows an exponential distribution. The phase offset can be calculated as

[0092]

[0093] in, represents the phase shift at the initial moment.

[0094] Finally, based on the above conclusions, the channel impulse response of the SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel model can be expressed as:

[0095]

[0096] Where h(t,τ) is the channel impulse response, J s (t) and J d (t) are the number of static clusters and dynamic clusters, I s (t) and I d (t) are the number of static scatterers and dynamic scatterers, is the Rice factor, h LoS (t), h GR (t), and They are LoS component, ground reflection component, and the kth static cluster. kth i scatterers and passing dynamic clusters The first j The complex channel gain of a dynamic scatterer, τ LoS , τ GR 、 and are the delays corresponding to the above complex channel gains, Θ GR 、Θ s and Θ d They represent the power ratios of the ground reflection component, static cluster, and dynamic cluster respectively, and their sum is equal to one.

[0097] S73: Calculate the channel transfer function and introduce the frequency-dependent factor to model the frequency-dependent path gains of the automatic and static clusters. Based on the channel impulse response obtained in step S72, perform Fourier transform with respect to the time delay τ to obtain the channel transfer function, which is written as:

[0098]

[0099] In order to obtain the frequency-dependent path gain, a frequency-dependent factor is introduced, which is obtained

[0100]

[0101] in, is the frequency-related factor that depends on the specific communication scenario. This factor can be used to calculate the frequency-related path gain, which enables the capture of the frequency non-stationarity of the channel. H(t,f) is the channel transfer function, and J s (t) and J d (t) are the number of static clusters and dynamic clusters, I s (t) and Id (t) are the number of static scatterers and dynamic scatterers, is the Rice factor, h LoS (t), h GR (t), and They are LoS component, ground reflection component, and the kth static cluster. kth i scatterers and passing dynamic clusters The first j The complex channel gain of a dynamic scatterer, τ LoS , τ GR 、 and are the delays corresponding to the above complex channel gains, Θ GR 、Θ s and Θ d They represent the power ratios of the ground reflection component, static cluster, and dynamic cluster respectively, and their sum is equal to one.

[0102] In summary, the present invention proposes a multimodal intelligent channel model assisted by a SoM-enabled LiDAR point cloud. Based on the intelligent fusion dataset of communication and perception data under high, medium and low MADs, a SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel model is constructed. The model has high-precision modeling of channel statistical characteristics and is applicable to scenarios under different MADs conditions. The electromagnetic propagation mechanism and ScaR algorithm are used to innovatively process the LiDAR point cloud and parse and model the nonlinear mapping relationship between the LiDAR point cloud and the electromagnetic propagation scatterers. Based on the parsed and modeled mapping relationship, the LiDAR point cloud is used to identify electromagnetic propagation scatterers, and the number and position of the scatterers are modeled with high resolution and high precision. The SoM-enabled LiDAR point cloud-assisted multimodal intelligent channel model is constructed to capture the non-stationarity and consistency of the environment-channel.

[0103] It should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A multimodal intelligent channel modeling method enabled by machine synesthesia, characterized in that: Design a scatterer recognition algorithm to intelligently analyze and model the nonlinear mapping relationship between the LiDAR point cloud and electromagnetic propagation scatterers, and build a multimodal intelligent channel model to effectively capture the wireless communication environment - channel non-stationarity and consistency; including: 1) Construct a high-fidelity communication and perception intelligent fusion dataset, including LiDAR point cloud and electromagnetic propagation scatterer information; 2) Perform data preprocessing to obtain valid LiDAR point cloud; 3) For the pre-processed valid LiDAR point cloud, perform raster feature extraction on the average value of density, height, and the sum of distances from the transmitter and receiver, respectively, to obtain the density raster feature of the LiDAR point cloud, the height raster feature of the LiDAR point cloud, and the distance raster feature of the LiDAR point cloud; 4) By designing a scatterer recognition algorithm (ScaR), the complex mapping relationship between the perception data LiDAR point cloud and electromagnetic propagation scatterers is analyzed and modeled, scatterer recognition and prediction are performed to obtain the number of scatterers in each LiDAR point cloud grid; this includes: The input of the ScaR algorithm is the density, height and distance raster features of the valid LiDAR point cloud; the output is a raster map formed by scatterers in the visible area; The hidden layer structure includes a convolutional neural network (SegNet) and task-specific modules. SegNet uses an encoder-decoder structure. The encoder extracts features through a series of convolutional layers, and the decoder restores the feature map into a segmentation map of the same size as the input image through deconvolution. The decoder uses the pooling index information from the encoder pooling layer. The task-specific module includes an image interpolation layer, a convolutional layer, a batch normalization layer, and an activation function, which is used to adjust the output dimension of the SegNet task head to achieve the output of the scatterer grid map. 5) At the initial moment, the LiDAR point cloud is used to identify the electromagnetic propagation scatterer grid map, which is converted into scatterer point coordinates. The information obtained includes the number of scatterers, time delay, and angle parameters. Dynamic scatterers are distinguished from static scatterers, and dynamic clusters and static clusters are further obtained. 6) At the next moment, based on the LiDAR point cloud data at the current moment, identify the scatterers at the current moment, distinguish dynamic scatterers from static scatterers, and obtain dynamic clusters and static clusters; 7) Build a multimodal intelligent channel model assisted by SoM-enabled LiDAR point cloud, and accurately model the number, distance, angle, delay, and delay parameters of clusters; including: S71: Build a wireless channel model architecture based on the environment embedding of twin clusters, and use virtual links to represent the transmission process between twin clusters in electromagnetic space; S72: Determine a channel impulse response. The channel impulse response of the multimodal intelligent channel model enabled by machine synesthesia is expressed as: Where h(t,τ) is the channel impulse response, J s (t) and J d (t) are the number of static clusters and dynamic clusters, I s (t) and I d (t) are the number of static scatterers and dynamic scatterers, is the Rice factor, h LoS (t), h GR (t), and They are LoS component, ground reflection component, and the kth static cluster kth i static scatterers and passing dynamic clusters The first j The complex channel gain of a dynamic scatterer, τ LoS , τ GR 、 and are the delays corresponding to the four complex channel gains mentioned above; Θ GR 、Θ s and Θ d They represent the power ratios of the ground reflection component, static cluster, and dynamic cluster respectively, and their sum is 1; S73: Calculate the channel transfer function and introduce the frequency-dependent factor to model the frequency-dependent path gains of the automatic and static clusters respectively; Based on the channel impulse response, the Fourier transform of the time delay τ is performed to obtain the channel transfer function, which is expressed as: After introducing the frequency-related factor, it can be expressed as: in, is the frequency correlation factor; H(t,f) is the channel transfer function; In this way, the frequency-related path gain is obtained, realizing multimodal intelligent channel modeling enabled by machine synesthesia.

2. The multimodal intelligent channel modeling method enabled by machine synesthesia according to claim 1, characterized in that: Step 1) uses the Wireless InSite and AirSim simulation platforms to build a high-fidelity communication and perception intelligent fusion dataset for constructing simulation scenarios.

3. The multimodal intelligent channel modeling method enabled by machine synesthesia according to claim 1, characterized in that: Step 1) includes: S11: Achieve deep integration of Wireless InSite and AirSim simulation platforms; The scene is rendered with high precision in AirSim and simplified in Wireless InSite. When setting the trajectory of the mobile agent, the coordinates of the mobile agent in both the Wireless InSite and AirSim simulation platforms are modified synchronously, thus generating a dynamic scene that deeply integrates communication and perception. S12: Achieve precise matching between Wireless InSite and AirSim simulation platforms; Draw and build each object in both simulation platforms one by one, so that the objects are exactly the same in size, position, and scale; set the trajectory of each mobile agent so that the trajectory of each mobile agent in the Wireless InSite and AirSim simulation platforms is completely synchronized.

4. The multimodal intelligent channel modeling method enabled by machine synesthesia according to claim 1, characterized in that: Step 2) Preprocess the data and fuse the LiDAR point clouds from the transmitter and receiver to obtain complete physical environment feature information; Remove ground points and perform height threshold filtering and visible area filtering to reduce redundant information in the LiDAR point cloud and extract the part of the point cloud related to electromagnetic propagation. Specifically, it includes: S21: Based on the 3D coordinates of the LiDAR point cloud at the transmitter and the 3D coordinates of the LiDAR point cloud at the receiver, as well as the coordinates and movement direction of the intelligent body, point cloud registration is performed to place the LiDAR point clouds of the transmitter and receiver in the same world coordinate system, thereby obtaining complete physical environment information around the transmitter and receiver; S22: Based on the spliced ​​and fused LiDAR point clouds of the transmitter and receiver, determine and delete ground points with zero vertical coordinates in the LiDAR point cloud to reduce data redundancy of the LiDAR point cloud; S23: A height threshold filtering method is used to remove LiDAR point clouds with heights greater than a certain threshold. The point cloud is further filtered according to the visible area, and the visible area is set as an ellipsoid with the transceiver as the focus. The point cloud inside the ellipsoid is taken out as the pre-processed valid LiDAR point cloud, thereby screening out the LiDAR point cloud part that is most relevant to electromagnetic propagation.

5. The multimodal intelligent channel modeling method enabled by machine synesthesia according to claim 1, characterized in that: Step 3) extracting rasterized features, specifically including: S31: Extract density grid features of LiDAR point cloud: the value in each grid is the ratio of the total number of LiDAR point clouds in the grid to the area of ​​the grid; S32: Extract the height grid features of the LiDAR point cloud: the value in each grid is the maximum value of the LiDAR point cloud height in the grid; S33: Extracting distance grid features of LiDAR point cloud: Based on the position encoding layer, for the grid where the LiDAR point cloud exists, the distance is calculated as the average of the sum of the distances between the center of the grid and the transmitter and receiver.

6. The multimodal intelligent channel modeling method enabled by machine synesthesia according to claim 1, characterized in that: Step 4) In the hidden layer of the ScaR algorithm, the image interpolation layer of the task-specific module is implemented using the F.interpolate function in PyTorch; the activation function uses the ReLu function.

7. The multimodal intelligent channel modeling method enabled by machine synesthesia according to claim 1, characterized in that: In step 5), a clustering algorithm is specifically used to obtain dynamic clusters and static clusters in the electromagnetic space.

8. The multimodal intelligent channel modeling method enabled by machine synesthesia according to claim 1, characterized in that: Step 6) distinguishes between dynamic clusters and static clusters, specifically including: First, based on the height grid map of the LiDAR point cloud, if the value in the height grid of the LiDAR point cloud is less than the set height threshold, it is detected as a mobile intelligent body; otherwise, the LiDAR point cloud grid is detected as a static object; when the scatterer is located in the grid of the detected mobile intelligent body, the scatterer is a dynamic scatterer; when the scatterer is located in the grid of the detected static object, the scatterer is a static scatterer.

9. The multimodal intelligent channel modeling method enabled by machine synesthesia according to claim 1, characterized in that: In step S71, the position and distance parameters of the static scatterer and the dynamic scatterer are obtained from the predicted scatterer; the multiple angular parameters of the scatterer are generated by Gaussian distribution; the number of static clusters and dynamic clusters is generated by Logistic distribution; and the formed twin clusters communicate and transmit through virtual links.

Images