A semantic environment perception method and system based on multi-modal dynamic fusion and a storage medium

Abstract

The application discloses a semantic environment perception method based on multi-modal dynamic fusion, which comprises the following steps: analyzing the time delay and angle characteristics of a millimeter wave signal, extracting echo peak information representing different reflection points, and mapping the echo peak information into spatial coordinates to construct a set of sparse and accurate point cloud; for each millimeter wave point, evaluating its signal strength, distance from the base station, and view deflection angle, introducing a reliability model, and calculating the corresponding reliability index; using a camera to obtain scene images, inputting the scene images into a deep convolutional neural network to generate a bird's eye view (BEV) semantic distribution map, mapping the pixel color value to a preset label color according to the minimum color difference principle, and uniformly encoding the diversified network output into a fixed semantic category; combining the spatial coordinates of the millimeter wave point cloud and the semantic information of the corresponding BEV pixel, dynamically weighting and integrating according to the reliability and semantic confidence, giving each point cloud the most possible semantic label, and eliminating low-confidence or inconsistent measurement data, finally outputting a wireless semantic point cloud map to complete environment perception.

Description

Technical Field

[0001] This invention proposes a semantic environment perception method, system, and storage medium based on multimodal dynamic fusion, belonging to the interdisciplinary field of multimodal fusion, environment perception, and communication perception integration. Background Technology

[0002] Integrated Sensing and Communication (ISAC) integrates communication and sensing functions into a unified architecture and is one of the core supporting technologies of the Sixth Generation Mobile Communication System (6G). Environmental perception is a particularly promising research direction in ISAC, enabling not only high-speed wireless data transmission and massive connectivity but also endowing the network with the ability to "observe," "understand," and interact with its environment. Furthermore, multimodal fusion offers the possibility of enriching sensing results with accurate semantic understanding, providing significant potential for improving the semantic understanding capabilities of ISAC's environmental perception.

[0003] Achieving integrated communication and sensing faces numerous challenges. First, millimeter-wave radar point clouds are often affected by multipath reflection and sidelobe interference in complex environments, leading to many false detections and missed detections. Second, most current fusion methods fail to adapt to the quality variations of different modal data in different environments, and cannot handle low-quality input or adapt to constantly changing environments. Finally, current research on millimeter-wave environmental sensing mainly focuses on simulations or indoor environments, lacking systematic testing and verification for complex outdoor scenarios. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to explore a multimodal data fusion method for radio point clouds and visual semantic information, so as to improve the semantic environment perception capability for mobile communication systems.

[0005] Technical Solution: To achieve the above objectives, this invention proposes a semantic environment perception method and system based on multimodal dynamic fusion. The method specifically includes the following steps:

[0006] Step 1: The single-station ISAC system generates an accurate point cloud set by analyzing the time delay and angular characteristics of the millimeter-wave signal, thus realizing the single-station sensing capability of the millimeter-wave communication system.

[0007] Step 2: Based on indicators such as signal strength, base station distance, and viewing angle deflection, perform reliability modeling on the single-station sensing results of the millimeter-wave communication system, construct a reliability distribution map for the entire scenario, and distinguish between high and low quality measurement points.

[0008] Step 3: The camera captures scene images, which are then input into a deep convolutional network to generate a Bird's-EyeView (BEV) semantic distribution map and pixel-level confidence scores to obtain visual semantic perception results.

[0009] Step 4: Compare the BEV output with the predefined semantic color palette for color difference. Based on the principle of minimum color difference, map the pixel color values ​​to the preset label colors, so that the diverse network outputs are uniformly encoded into fixed semantic categories.

[0010] Step 5: Align the millimeter-wave point cloud and the BEV semantic map on a spatial grid, dynamically assign fusion weights based on their respective confidence levels, assign semantic labels to the point cloud in a weighted manner, remove low-confidence or inconsistent data, and finally achieve semantic environment awareness.

[0011] Furthermore, in step 1, the specific method for generating millimeter-wave point clouds is as follows:

[0012] (1) Array A and array B are aligned in the same direction during transmission and reception. They use configurable beam sets to perform beam scanning in a predetermined direction to cover the entire space of a fan-shaped target area.

[0013] (2) Perform frequency domain analysis on the OFDM symbols and extract the channel impulse response h of the i-th beam. i (t);

[0014] (3) via h i (t) Obtain the power delay profile (PDP) of the i-th beam;

[0015] (4) Determine the delay corresponding to the PDP peak value in each beam direction and convert it into distance. Combine this with the corresponding beam angle positioning calculation to map the point location onto the reference system to form a point cloud map. This method can calculate the position of each point more accurately. Compared with the existing rough and simple estimation methods, this method can provide more accurate spatial positioning, thereby generating a higher precision point cloud map, which helps to improve the accuracy of environmental perception.

[0016] Furthermore, in step 2, the specific method for point cloud reliability modeling is as follows:

[0017] (1) For the point cloud in step 1, each point Energy intensity is represented by a state vector containing several key measurement attributes. Beam pointing angle Transmission distance and the reliability factor R to be evaluated (s) ;

[0018] (2) Based on the measurement attributes of each point in the point cloud, a general reliability function is introduced to calculate its score. This function can be combined with factors such as normalized energy intensity, distance attenuation and beam angle for comprehensive evaluation, and the output results are mapped to a unified confidence interval (0 to 1) to provide comparable weight input for multimodal fusion.

[0019] (3) For R (s) A threshold is set; points above the threshold are considered valid and retained, while those below are considered spurious and removed. Points with scores above the threshold are considered valid measurements and retained, while those below the threshold are marked as spurious noise and pseudo-echoes and discarded. This multi-dimensional scoring model can effectively distinguish between high-quality and low-quality point cloud data, avoiding the common problem in existing technologies that treat all data points the same, thus improving the overall quality of point cloud data and providing high-quality input for subsequent multimodal fusion.

[0020] Furthermore, in step 3, the specific methods for visual BEV semantic map and confidence estimation are as follows:

[0021] (1) Use a camera or multi-view camera to obtain scene image I, and combine the algorithm to perform gravity alignment on the image;

[0022] (2) Let E be the image width (i.e. the number of vertical columns) and D be the number of depth planes sampled at fixed intervals. The processed I is fed into a convolutional neural network (CNN) for feature extraction to obtain a feature map.

[0023] (3) Use a lightweight CNN for each column of the feature map BEV The columns are treated as polar coordinate rays to generate corresponding polar ray semantic representations. This process can fuse inter-column context or cross-channel information and support semantic aggregation and feature fusion on different depth profiles.

[0024] (4) Based on the aforementioned polarimetric representation, generate neural BEV images and confidence maps. Each element C i,j This represents the estimated reliability of the BEV pixel in the i-th column and depth plane j. This approach provides pixel-by-pixel semantic parsing, accurately and reliably representing the semantic information of the environment, and better understanding the semantic labels of each element in the environment.

[0025] Furthermore, in step 4, the specific method for semantic color mapping is as follows:

[0026] (1) Define a set of distinguishable colors for each target category, input the color mapping table and convert it to a unified color space so that consistency comparison can be performed in subsequent processing. The color space conversion can adopt common color model conversion algorithms, or it can be extended to other color difference measurement spaces with better perception as needed.

[0027] (2) Read the neural BEV semantic map generated by the network, map the valid pixel values ​​in it to a color space representation consistent with the mapping table, and interpolate or fill invalid or missing data in the image to ensure the integrity of subsequent mapping steps.

[0028] (3) For each pixel to be processed, compare its representation in the color space with the reference value in each semantic label color table, calculate the color difference, select the optimal matching category and record the corresponding original RGB value.

[0029] (4) Write the optimal category RGB value of each pixel back to the valid position of the BEV image to form a color bird's-eye view based on semantic labels. This method ensures the optimal matching of the semantic label of each pixel with the predefined color, thereby reducing the problems of color inconsistency or misclassification that may occur in the prior art, making the semantic information in the image more intuitive and easy to understand, and enhancing the accuracy and readability of the semantic labels in the image.

[0030] Furthermore, in step 5, the method for dynamic multimodal fusion is as follows:

[0031] (1) Since the BEV, confidence map, and radio point cloud share the same spatial grid, each radar point Each will be projected onto a unique BEV pixel. Based on each point... millimeter wave reliability R (s) and BEV confidence level C (s) Weighted sum of their contributions and

[0032] (2) Aggregate the initial weights of the two modes to obtain the total fused weight for each point s.

[0033] (3) For the total fusion weight Set a threshold, select valid points whose reliability and confidence sum exceeds the threshold, and clean up stray points below the threshold;

[0034] (4) Using the projected pixel of each filtered valid point as the center, extract the local neighborhood confidence segment from the BEV confidence map.

[0035] (5) Using the reliability index of the point itself, perform weighted fusion on the extracted neighborhood confidence block to generate local feature representation;

[0036] (6) Normalize the features after local weighted fusion to ensure that the weights of different neighborhoods are comparable, and generate a standardized output that can be directly used for label decision.

[0037] (7) According to step (6) The semantic label corresponding to the largest component is assigned to the entire segment, thereby effectively transferring BEV semantics into the radio point cloud and ultimately obtaining a semantic point cloud map. This dynamic weighting method can adjust the fusion weights in real time according to the quality of sensor data and environmental changes, avoiding the problem that existing static weighting methods cannot adapt to data quality fluctuations in different environments, thus ensuring the efficiency and accuracy of multimodal data fusion.

[0038] Furthermore, this invention proposes a semantic environment perception system based on multimodal dynamic fusion, comprising a sensor, a memory, and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described.

[0039] Furthermore, the present invention proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described herein.

[0040] Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0041] The technical solution proposed in this invention first establishes a reliability model for radio point clouds, performs quality scoring and screening for each echo point, and eliminates low-quality detections caused by pseudo-echoes and sidelobe scattering, thereby improving the accuracy and effectiveness of millimeter-wave point clouds. Subsequently, based on the reliability assessment of each modal sensor, the fusion weights are adaptively adjusted to achieve the collaborative fusion of millimeter-wave radar point cloud data and visual semantic information, fully leveraging the complementary advantages of different sensors. Secondly, relying on the existing ISAC prototype system, a joint experimental platform for fusion perception of millimeter-wave radar and cameras is constructed and tested and verified in real outdoor scenarios, completing the perception of target semantics in the environment. Finally, this solution eliminates the need for multi-base station synchronization or centralized computing, simplifying the system's communication architecture, reducing the difficulty of engineering implementation, and making the system more flexible and more fault-tolerant. Attached Figure Description

[0042] Figure 1 This is a schematic diagram illustrating the geometric relationship between the single-station ISAC prototype system, the user, and the target reflector in an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of visual sensing in an embodiment of the present invention;

[0044] Figure 3 This is a point cloud map generated by single-station sensing in an embodiment of the present invention;

[0045] Figure 4 These are the neural BEV and confidence maps obtained through CNN in this embodiment of the invention;

[0046] Figure 5 The flowchart illustrates a semantic environment perception method and system based on multimodal dynamic fusion, as provided in this embodiment of the invention. Detailed Implementation

[0047] In the following description, specific details, such as particular system architectures, are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the invention. However, those skilled in the art will appreciate that the invention may be practiced in other embodiments without these specific details.

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, a detailed description will be provided below in conjunction with the accompanying drawings.

[0049] Example: This embodiment of the invention provides a semantic environment perception method based on multimodal dynamic fusion, such as... Figure 5 As shown, the method includes the following steps:

[0050] Step 101: The single-station ISAC system generates an accurate point cloud set by analyzing the time delay and angular characteristics of the millimeter-wave signal, thus realizing the single-station sensing capability of the millimeter-wave communication system.

[0051] It should be noted that:

[0052] (1) The single-site ISAC system used is equipped with two uniform linear arrays (ULAs) to support downlink communication and radar sensing functions. In downlink communication, antenna array A transmits data symbols to the user. In sensing, antenna array B collects the signal echoes transmitted by array A and estimates the spatial parameters of targets in the environment from them, requiring a millimeter-wave-based point cloud map. The echo model received along the i-th beam on the q-th subcarrier and k-th OFDM symbol is as follows:

[0053]

[0054] Where L is the total number of propagation paths.

[0055]

[0056] It is the beam alignment factor, ξ BV,iIt is the analog beamforming vector of the i-th encoding sample (the size of the encoding sample is N), θ AoA,l ,θ AoD,l ∈[0,π] represents the departure angle and the arrival angle, (·) H This represents the Hermitian transpose, b l It is the complex path gain, τ l It is the round-trip time of the signal (target distance = cτ) l / 2, where c is the speed of light), Δf is the subcarrier spacing, x h,k It is a transmitted signal, n h,k It is additive white Gaussian noise. The m-th receiving steering vector element of antenna array B is...

[0057]

[0058] Where m = 1, ..., M, M is the number of elements in each array (A and B), and λ is the wavelength. It is the element spacing (usually λ / 2);

[0059] (2) Arrays A and B are aligned in the same direction during transmission and reception, and their beam directions sweep across all N turning angles sequentially. Frequency domain analysis of the OFDM symbols yields the channel impulse response of the i-th beam as follows:

[0060]

[0061] Where δ(·) is the Dirac function;

[0062] (3) via h i (t) The power delay spectrum of the i-th beam is obtained as follows

[0063]

[0064] Where |·| represents the amplitude. It is the statistical expected value. Connecting the PDPs of all beam directions can form a two-dimensional heatmap;

[0065] (4) Determine the delay corresponding to the PDP peak value in each beam direction and convert it into distance to form a... Figure 3 (b) shows the N-point millimeter-wave wireless point cloud map.

[0066] Step 102: Based on indicators such as signal strength, base station distance and viewing angle deflection, perform reliability modeling on the single-station sensing results of the millimeter-wave communication system, construct a reliability distribution map of the entire scenario, and distinguish between high and low quality measurement points.

[0067] It should be noted that:

[0068] (1) For the point cloud in step 1, the antenna pattern and the surface material of the object can introduce ranging estimation errors, especially for beams that deviate from the inner diameter. Therefore, a reliability model is designed to assign higher confidence to points with strong echo energy and small inner diameter angle deviation, while filtering out false detections. For the remaining points in the N point clouds after cleaning, the s-th point is represented by the following state vector:

[0069]

[0070] Where S is the number of points. It is energy intensity. It's the beam angle. It's distance;

[0071] (2) The reliability assessment of point clouds is calculated using the following formula:

[0072]

[0073] in, Therefore, higher echo energy and smaller off-axis angle will both increase R (s) .

[0074] Step 103: Use a camera to capture scene images, input them into a deep convolutional network to generate a BEV semantic distribution map and pixel-level confidence scores, and obtain visual semantic perception results.

[0075] It should be noted that:

[0076] (1) Use a camera to obtain scene image I and perform gravity alignment processing;

[0077] (2) Let E be the image width (i.e. the number of vertical columns) and D be the number of depth planes sampled at fixed intervals. The processed I is fed into the CNN for feature extraction to obtain the feature map.

[0078] (3) Each column of the feature map is composed of a lightweight CNN BEV The process involves treating columns as polar coordinate rays and generating polar ray representations.

[0079] (4) Output as shown Figure 4 (a) The neural BEV image shown and as Figure 4 (b) shows the confidence plot. Each element C i,j This represents the estimated reliability of the BEV pixel in the i-th column and depth plane j.

[0080] Step 104: Compare the BEV output with the predefined semantic color palette for color difference. Based on the principle of minimum color difference, map the pixel color values ​​to the preset label colors, so that the diverse network outputs are uniformly encoded into fixed semantic categories.

[0081] It should be noted that:

[0082] (1) Predefine the RGB triples corresponding to various semantic categories and convert them from sRGB space to Lab space;

[0083] (2) Read the neural BEV and convert the effective pixel values ​​into Lab colors;

[0084] (3) For the Lab value of each valid pixel, calculate the CIE2000 color difference with each semantic color, select the semantic category with the smallest color difference, and record the RGB value corresponding to the category in the list.

[0085] (4) Replace the valid positions in the neural BEV with the values ​​in the list to obtain the semantic mapping BEV of RGB.

[0086] Step 105: Align the millimeter-wave point cloud and the BEV semantic map on a spatial grid, dynamically assign fusion weights based on their respective confidence levels, assign semantic labels to the point cloud in a weighted manner, remove low-confidence or inconsistent data, and finally achieve semantic environment awareness.

[0087] It should be noted that:

[0088] (1) Radio point clouds from monostation millimeter-wave ISAC systems only provide object locations and lack semantic context. To enrich the understanding of the environment, these point clouds are fused with BEV representations extracted from camera images. Since the quality of each mode varies at different spatial locations, the fusion is performed according to the specific context of each point cloud. millimeter wave reliability R (s) and BEV confidence level C (s) Dynamically weight their contributions as

[0089]

[0090] Where v RPC v BEV >0, v<0 are hyperparameters, R (s) Or C (s) The larger the value, the higher the weight of the corresponding mode;

[0091] (2) The total fusion weight of point s is

[0092] (3) For the total fusion weight Set a threshold, select valid points whose reliability and confidence sum exceeds the threshold, and clean up stray points below the threshold;

[0093] (4) For each valid point, extract a 3×3 BEV confidence segment centered on its corresponding pixel:

[0094]

[0095] (5) Weighting is performed in this block based on the reliability of the central valid point.

[0096] (6) Obtained by normalization using the softmax function.

[0097] (7) According to step (6) The semantic label corresponding to its largest entry is assigned to the entire fragment, thereby effectively transferring the BEV semantics into the radio point cloud and finally obtaining a semantic point cloud map.

[0098] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A semantic environment perception method based on multimodal dynamic fusion, characterized in that, The method includes the following steps: Step 1: The single-station communication and sensing integrated system generates a precise point cloud set by analyzing the time delay and angular characteristics of the millimeter-wave signal, thereby realizing the single-station sensing capability of the millimeter-wave communication system. Step 2: Based on the signal strength, base station distance and viewing angle deflection, perform reliability modeling on the single-station sensing results of the millimeter-wave communication system, construct a reliability distribution map for the entire scenario, and distinguish between high and low quality measurement points; Step 3: Use a camera to capture scene images, input them into a deep convolutional network to generate BEV and pixel-level confidence scores, and obtain visual semantic perception results, where BEV is an environmental bird's-eye view semantic distribution map; Step 4: Compare the BEV output with the predefined semantic color palette for color difference. Based on the principle of minimum color difference, map the pixel color values ​​to the preset label colors, so that the diverse network outputs are uniformly encoded into fixed semantic categories. Step 5: Align the millimeter-wave point cloud and the BEV semantic map on the spatial grid, dynamically allocate fusion weights based on their respective confidence levels, assign semantic labels to the point cloud in a weighted manner, remove low-confidence or inconsistent data, and finally achieve semantic environment awareness. The specific method in step 4 is as follows: (1) Define a set of distinguishable colors for each target category, input the color mapping table and convert it to a unified color space so that consistency comparison can be performed in subsequent processing. The color space conversion can adopt common color model conversion algorithms, or be extended to other color difference measurement spaces with better perception as needed. (2) Read the neural BEV semantic map generated by the network, map the valid pixel values ​​in it to the color space representation consistent with the mapping table, and interpolate or fill invalid or missing data in the image to ensure the integrity of subsequent mapping steps; (3) For each pixel to be processed, compare its representation in the color space with the reference value in each semantic tag color table, calculate the color difference, select the optimal matching category and record the corresponding original RGB value; (4) Write the optimal category RGB value of each pixel back to the valid position of the BEV image to form a color bird's-eye view based on semantic tags.

2. The semantic environment perception method based on multimodal dynamic fusion according to claim 1, characterized in that, Step 1 is as follows: (1) Array A and array B are aligned in the same direction during transmission and reception, and use configurable beam sets to perform beam scanning in a predetermined direction to cover the entire space of a fan-shaped target area; (2) Perform frequency domain analysis on OFDM symbols and extract the first... Channel impulse response of each beam ; (3) Through Obtain the Power delay spectrum (PDP) of each beam; (4) Determine the delay corresponding to the PDP peak value in each beam direction, convert it into distance, combine it with the corresponding beam angle positioning calculation, and map the point position to the coordinate system to form a point cloud map; Step 2 is as follows: (1) For the point cloud in step 1, each point Energy intensity is represented by a state vector containing several key measurement attributes. Beam pointing angle Transmission distance and the reliability factors to be evaluated ; (2) Based on the measurement attributes of each point in the point cloud, a general reliability function is introduced to calculate its score. This function combines normalized energy intensity, distance attenuation and beam angle factors for comprehensive evaluation, and maps the output results to a unified confidence interval, providing comparable weight inputs for multimodal fusion. (3) To A threshold is set. Points above the threshold are considered valid and retained, while those below the threshold are considered spurious and removed. Points with scores above the threshold are considered valid measurements and retained, while points below the threshold are marked as spurious noise and removed, providing high-quality input for subsequent multimodal fusion.

3. The semantic environment perception method based on multimodal dynamic fusion according to claim 2, characterized in that, The specific method in step 3 is as follows: (1) Obtain scene images using a camera or multi-view camera. And combine the algorithm to perform gravity alignment on the image; (2) Record The image width, i.e., the number of vertical columns. The number of depth planes sampled at fixed intervals, after processing The data is fed into a convolutional neural network for feature extraction to obtain a feature map. (3) Use lightweight for each column of the feature map The columns are treated as polar coordinate rays to generate corresponding polar ray semantic representations. This process integrates inter-column context or cross-channel information, supporting semantic aggregation and feature fusion on different depth profiles. (4) Based on the aforementioned polarimetric semantic representation, generate neural BEV images and confidence maps. Each element Indicates the first Columns and depth plane Estimation reliability of BEV pixels.

4. The semantic environment perception method based on multimodal dynamic fusion according to claim 3, characterized in that, The method for step 5 is as follows: (1) Since the BEV, confidence map, and radio point cloud share the same spatial grid, each radar point All will be projected onto a single BEV pixel, based on each point millimeter wave reliability and BEV confidence Weighted summation of their contributions yields and ; (2) Aggregate the initial weights of the two modes to obtain the weights of each point. Total weight of fusion ; (3) For the total fusion weight Set a threshold, select valid points whose reliability and confidence sum exceeds the threshold, and clean up stray points below the threshold; (4) Using the projected pixel of each filtered valid point as the center, extract the local neighborhood confidence segment from the BEV confidence map. ; (5) Using the reliability index of the point itself, perform weighted fusion on the extracted neighborhood confidence block to generate local feature representation; (6) Normalize the features after local weighted fusion to ensure that the weights of different neighborhoods are comparable and generate a standardized output that can be directly used for label decision. ; (7) According to step (6) The semantic label corresponding to its largest component is assigned to the entire segment, thereby effectively transferring the BEV semantics to the radio point cloud and finally obtaining the semantic point cloud map.

5. The semantic environment perception method based on multimodal dynamic fusion according to claim 1, characterized in that, Step 1 is as follows: (1) The single-station ISAC system used is equipped with two uniform linear arrays, supporting downlink communication and radar sensing functions. In downlink communication, antenna array A sends data symbols to the user. In sensing, antenna array B collects the signal echoes emitted by array A and estimates the spatial parameters of the target in the environment from them. A point cloud map based on millimeter waves needs to be obtained. The subcarrier and the first The top edge of the OFDM symbol The echo model received by each beam is as follows in It is the total number of transmission paths. It is the beam alignment factor, where It is the first The simulated beamforming vector of each encoded text These are the departure angle and the arrival angle. This represents the Hermitian transpose. It is the complex path gain. It is the round-trip time of the signal. It is the subcarrier spacing. It is transmitting a signal. It is additive white Gaussian noise, the first of antenna array B. Each accepting steering vector element is in , It is the number of elements in each array. It's the wavelength. It refers to the spacing between elements; (2) Array A and array B are aligned in the same direction during transmission and reception, and their beam directions sweep across all [various targets] sequentially. The first steering angle is used to perform frequency domain analysis on the OFDM symbol to obtain the first... The channel impulse response of each beam is in, It is the Dirac function; (3) Through Obtain the The power delay spectrum of each beam is in It's the amplitude. It is a statistical expectation value, which connects the PDPs of all beam directions to form a two-dimensional heat map; (4) Determine the delay corresponding to the PDP peak value in each beam direction and convert it into range to form Point cloud map for millimeter-wave wireless transmission.

6. The semantic environment perception method based on multimodal dynamic fusion according to claim 1, characterized in that, Step 2 is as follows: (1) The point cloud in step 1, after cleaning The remaining points in the point cloud are represented by the following state vector. indivual: in, It is the number of dots. It is energy intensity. It is the beam pointing angle. It is the transmission distance; (2) The reliability assessment of point clouds is calculated using the following formula: in, .

7. The semantic environment perception method based on multimodal dynamic fusion according to claim 1, characterized in that, Step 5 is as follows: (1) Radio point clouds from a single-station millimeter-wave ISAC system can only provide the location of objects, based on each point millimeter wave reliability and BEV confidence Dynamically weight their contributions as in , It's a hyperparameter. or The larger the value, the higher the weight of the corresponding mode; (2) point The total fusion weight is ; (3) For the total fusion weight Set a threshold, select valid points whose reliability and confidence sum exceeds the threshold, and clean up stray points below the threshold; (4) For each valid point, extract a pixel centered on its corresponding pixel. BEV confidence segment: (5) Weighted summaries are applied in this block based on the reliability of the central valid point. ; (6) Normalization is performed using the softmax function to obtain the result. ; (7) According to step (6) The semantic label corresponding to the largest entry is assigned to the entire fragment, thereby effectively transferring the BEV semantics to the radio point cloud and finally obtaining a semantic point cloud map.

8. A semantic environment perception system based on multimodal dynamic fusion, comprising sensors, a memory, and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images