A large language model driven semantic communication method and system for cooperative three-dimensional perception

Abstract

The application belongs to the technical field of intelligent networked vehicle cooperative perception and semantic communication, and discloses a large language model driven semantic communication method and system for cooperative three-dimensional perception. The method obtains three-dimensional point cloud perception data of the ego vehicle and the cooperative vehicle, converts the cooperative vehicle data to the unified space reference system of the ego vehicle according to the relative position of the two ends, generates ego space perception features and cooperative space perception features; generates a basic importance response graph from the cooperative space perception features, and combines global scene embedding, regional semantic abstract, statistical information and task prompt information to obtain a cognitive guidance response graph through a large language model and spatial decoding; fuses the two types of response graphs to obtain a corrected importance response graph, and selects a cooperative semantic unit containing semantic features, spatial position and importance score according to the corrected importance response graph; selects a semantic representation dimension according to the signal-to-noise ratio, and compresses and jointly encodes the cooperative semantic unit.

Description

Technical Field

[0001] This invention belongs to the field of collaborative perception and semantic communication technology for intelligent connected vehicles, specifically a large language model-driven semantic communication method and system for collaborative three-dimensional perception. Background Technology

[0002] With the development of intelligent connected vehicles and high-level autonomous driving technology, vehicle collaborative 3D perception has become an important direction for improving the environmental perception capabilities of individual vehicles. Existing vehicle collaborative perception usually shares LiDAR point clouds, detection boxes, bird's-eye view features or intermediate semantic features through vehicle-to-vehicle communication links, enabling the vehicle to use the observation information from the collaborating vehicle to supplement the perception blind spots in complex scenarios such as occluded areas, distant areas, and intersections.

[0003] In real-world road environments, the point cloud data collected by collaborative vehicles and the resulting bird's-eye view features typically have high dimensionality. Directly transmitting the original point cloud or dense bird's-eye view features would consume significant communication bandwidth and be affected by wireless communication link latency, packet loss, signal-to-noise ratio fluctuations, and channel fading. Existing solutions reduce the amount of data transmitted through key region filtering, feature compression, or semantic communication. However, the selection of regions to be transmitted often relies on detection confidence, visual saliency, or local feature responses, failing to adequately consider the semantic relationships between occluded boundaries, distant weak targets, safety-critical areas, and long-tail risk targets. This can easily lead to the discarding of collaborative information with weak local responses but valuable for 3D detection during the sparsification process.

[0004] Meanwhile, existing channel adaptive transmission schemes typically focus on adjusting the code rate, modulation scheme, or redundancy based on channel quality. There is a lack of continuous data inheritance between the task importance information formed on the sensing side and the semantic compression dimension, power allocation, and coding protection on the communication side. When the wireless link is in a low signal-to-noise ratio or rapid fluctuation state, it is difficult to keep the semantic representation dimension, transmission protection strength, and importance of the area to be transmitted consistent, which can easily lead to problems such as insufficient protection of key semantic units and excessive transmission resources occupied by ordinary areas.

[0005] Furthermore, when the receiver fuses cooperative information, existing solutions often use feature stitching, direct projection, or conventional attention fusion to process cooperative features after channel perturbation. Since cooperative features are usually sparsely distributed after transmission and may be accompanied by spatial position deviation, noise distortion, or local semantic loss, if there is a lack of spatial reference relationships and reliability control consistent with the semantic selection results of the front end, the fusion stage is prone to insufficient utilization of cooperative information, error enhancement, or artifact interference, which in turn affects the stability of 3D target perception results in occlusion, long distance, and low signal-to-noise ratio scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a large language model-driven semantic communication method and system for collaborative 3D perception, which solves the problems of large amount of collaborative feature transmission in vehicle collaborative 3D perception, mismatch between semantic compression method and channel conditions when wireless communication link state changes, easy weakening of occluded boundary areas and candidate areas of distant traffic participants in sparse transmission, and easy influence of spatial misalignment and channel noise during receiver fusion.

[0007] To achieve the above objectives, this invention provides a large language model-driven semantic communication method for collaborative 3D perception, applicable to a vehicle collaborative 3D perception system including the vehicle end, the cooperating vehicle end, and a wireless communication link. The method includes the following process.

[0008] First, 3D point cloud perception data from the autonomous vehicle and the collaborating vehicle are acquired. Based on the relative pose relationship between the autonomous vehicle and the collaborating vehicle, the 3D point cloud perception data from the collaborating vehicle is converted to a unified spatial reference system corresponding to the autonomous vehicle. Under this unified spatial reference system, autonomous vehicle spatial perception features and collaborating spatial perception features are generated respectively. By completing the first spatial reference system in the feature generation stage, the collaborating semantic units selected from the collaborating spatial perception features can be directly represented by the spatial grid position corresponding to the autonomous vehicle, thus providing a consistent spatial reference basis for subsequent scattering, cross-attention aggregation, and gated residual fusion.

[0009] After obtaining the collaborative spatial perception features, a basic importance response map corresponding to the spatial grid under the unified spatial reference system is generated based on the collaborative spatial perception features.

[0010] Furthermore, global scene embeddings and regional semantic summaries are extracted from the collaborative space perception features, statistical information is extracted from the basic importance response map, and the global scene embeddings, regional semantic summaries, statistical information, and task prompt information are input into the large language model.

[0011] The task prompt information includes at least one of the learnable prompt embeddings related to occlusion boundaries, distant traffic participants, detection responses, or safety-critical areas. The large language model outputs a sequence of latent state vectors. After being mapped by a spatial decoder, the sequence of latent state vectors yields a cognitive guidance response map corresponding to a spatial grid in a unified spatial reference system. Subsequently, the basic importance response map and the cognitive guidance response map are fused to obtain a modified importance response map.

[0012] Based on the modified importance response map and transmission budget, a set of cooperative semantic units is selected from the cooperative spatially perceived features. Each cooperative semantic unit includes semantic features, spatial location, and importance score. The importance score is determined by the response value of the spatial location of the cooperative semantic unit in the modified importance response map. Thus, the selection of the spatial location and the generation of the importance score of the cooperative semantic unit both originate from the same modified importance response map, enabling subsequent semantic compression, joint source-channel coding, and receiver fusion to use the same set of spatial locations and importance scores of the cooperative semantic units.

[0013] After the set of cooperative semantic units is determined, the signal-to-noise ratio (SNR) of the wireless communication link is obtained. Based on the SNR, a target semantic representation dimension is selected from multiple preset semantic representation dimensions. While keeping the spatial position and importance score of each cooperative semantic unit unchanged, the semantic features of each cooperative semantic unit are compressed to obtain a compressed set of cooperative semantic units. In this process, the SNR is used to determine the semantic representation dimension without changing the previously determined set of cooperative semantic units or recalculating the importance score.

[0014] Subsequently, joint source channel coding is performed on the compressed cooperative semantic unit set to generate continuous channel symbols corresponding to each cooperative semantic unit. Power weights are calculated based on the importance scores of each cooperative semantic unit within the same transmission period, and global power normalization is performed on all adjusted continuous channel symbols to ensure that the average transmit power of all continuous channel symbols does not exceed a preset power budget before transmission via the wireless communication link. The importance scores are inherited from the modified importance response map and are not recalculated before joint source channel coding. Through this process, the importance scores obtained on the sensing side can continue to participate in the transmission amplitude adjustment of continuous channel symbols on the communication side, avoiding the separation between semantic selection and communication protection.

[0015] After receiving noisy channel symbols from the vehicle end, joint source channel decoding is performed to obtain a set of reconstructed cooperative semantic units corresponding to the set of cooperative semantic units. The reconstructed cooperative semantic units adopt the spatial position of the corresponding cooperative semantic units. Based on the spatial position of the reconstructed cooperative semantic units, the semantic features of the reconstructed cooperative semantic units are scattered onto a spatial grid under a unified spatial reference frame to form sparse cooperative spatial features. The filling value of the spatial grid into which no reconstructed cooperative semantic units fall is set to zero. When multiple reconstructed cooperative semantic units fall into the same spatial grid, the corresponding semantic features are accumulated and aggregated. Subsequently, the sparse cooperative spatial features are subjected to local convolution diffusion processing to obtain geometrically enhanced features.

[0016] In the semantic aggregation process, the vehicle spatial perception features are expanded into a query sequence, the semantic features and positional encoding of the reconstructed collaborative semantic units are used as key sequences, and the semantic features of the reconstructed collaborative semantic units are used as value sequences. Cross-attention aggregation is then performed to obtain semantic enhancement features. Subsequently, the geometric enhancement features and semantic enhancement features are fused into collaborative enhancement features, and a spatial adaptive gating map is generated based on the vehicle spatial perception features and collaborative enhancement features. The residuals of the collaborative enhancement features are injected into the vehicle spatial perception features through the spatial adaptive gating map to obtain fused spatial perception features. Finally, the 3D target perception results are output based on the fused spatial perception features.

[0017] Furthermore, the 3D point cloud perception data is LiDAR point cloud data. When generating the autonomous vehicle spatial perception features and collaborative spatial perception features, the LiDAR point cloud data from the collaborative vehicle is projected onto the autonomous vehicle's coordinate system. Within the unified bird's-eye view grid, point cloud cropping, voxelization, and cylinder feature aggregation are performed sequentially. Then, the autonomous vehicle bird's-eye view features and collaborative bird's-eye view features are obtained through a shared bird's-eye view encoder. The spatial position of the collaborative semantic unit is a 2D grid coordinate or a normalized 2D grid coordinate within the unified bird's-eye view grid.

[0018] Furthermore, when generating the basic importance response map, the collaborative spatial awareness features are input into the semantic importance head, which includes a convolutional layer, a normalization layer, a nonlinear activation layer, and a normalized output layer, and outputs a visual importance map with the same spatial grid size as the collaborative spatial awareness features.

[0019] The visual importance map serves as the basic importance response map, and its grid response value is jointly determined by the local texture features, geometric intensity features, and detection-related response features of the corresponding grid position.

[0020] Furthermore, the input to the large language model does not include full collaborative spatial awareness features expanded grid-by-grid.

[0021] The global scene embedding is obtained by global pooling and multilayer perceptron mapping of collaborative spatial perception features; the regional semantic summary is obtained by coarse-grained regional pooling of collaborative spatial perception features; the statistical information includes the mean, maximum, regional pooling response, and high-response region distribution features of the basic importance response map; the task prompt information includes at least one of the learnable prompt embeddings related to occlusion boundaries, distant traffic participants, detection responses, or safety-critical areas. The global scene embedding, regional semantic summary, statistical information, and task prompt information are mapped to the input dimension of the large language model and then concatenated in sequence to form the input sequence of the large language model.

[0022] Furthermore, the large language model outputs a sequence of latent state vectors. During spatial decoding, the sequence of latent state vectors is aggregated or cross-attention read, and then subjected to linear mapping, spatial rearrangement, and upsampling decoding to obtain a cognitive guidance response map aligned with the bird's-eye view grid under a unified spatial reference frame.

[0023] When generating the revised importance response map, learnable weights are used to weight the corresponding grid response values ​​of the basic importance response map and the cognitive guidance response map.

[0024] Furthermore, when selecting the set of cooperative semantic units, the number of cooperative semantic units to be transmitted is first determined based on the number of available channel symbols in the current transmission cycle and the channel symbol budget of a single cooperative semantic unit. Then, spatial diversity constraints are applied to the modified importance response map, and the corresponding number of spatial locations to be transmitted are selected according to the response value sorting results of the modified importance response map.

[0025] The spatial diversity constraints include at least one of local maximum suppression, non-maximum suppression, minimum spatial interval constraint, grid partitioning sampling, and regional equalization sampling. Before sorting and selection, adjacent high-response spatial locations within the same local region are suppressed. The importance score is the response value of the spatial location to be transmitted in the modified importance response map.

[0026] Furthermore, the signal-to-noise ratio is calculated by the communication module at the cooperating vehicle end based on the pilot signal, received power, and noise power, or estimated by the vehicle end and sent to the cooperating vehicle end through the control channel.

[0027] When selecting a semantic representation dimension based on the signal-to-noise ratio (SNR), a first SNR threshold and a second SNR threshold are set, with the second SNR threshold being greater than the first SNR threshold. When the SNR is less than the first SNR threshold, the first semantic representation dimension is selected. When the SNR is not less than the first SNR threshold and is less than the second SNR threshold, the second semantic representation dimension is selected. When the SNR is not less than the second SNR threshold, the third semantic representation dimension is selected. The first semantic representation dimension is less than the second semantic representation dimension, and the second semantic representation dimension is less than the third semantic representation dimension.

[0028] Furthermore, during joint source channel coding, each compressed cooperative semantic unit sequentially undergoes input linear projection, residual coding, channel attention, signal-to-noise ratio conditional modulation, importance protection gating, and amplitude limiting to obtain the corresponding real-valued transmission features. These real-valued transmission features are then packaged into continuous channel symbols according to in-phase and quadrature components.

[0029] When calculating power weights, the relative power weights of each cooperative semantic unit are calculated based on the importance scores of each cooperative semantic unit within the same transmission cycle, the mean of all importance scores within the same transmission cycle, and a preset stability term.

[0030] The importance score used in the importance protection gating is inherited from the modified importance response graph and is not recalculated before joint source-channel coding.

[0031] The present invention also provides a large language model-driven semantic communication system for collaborative 3D perception, including a vehicle-side terminal, a collaborative vehicle-side terminal, and a wireless communication link. The collaborative vehicle-side terminal includes a collaborative spatial feature encoding unit, a large language model-driven semantic selection unit, a channel adaptive semantic compression unit, and a semantic perception joint encoding unit. The vehicle-side terminal includes a vehicle spatial feature encoding unit, a joint source-channel decoding unit, and a sparse perception fusion unit.

[0032] The collaborative spatial feature encoding unit is used to convert the 3D point cloud perception data of the collaborative vehicle to the unified spatial reference system corresponding to the self vehicle based on the relative pose relationship between the self vehicle and the collaborative vehicle, and to generate collaborative spatial perception features.

[0033] The large language model-driven semantic selection unit is used to generate a basic importance response map based on the collaborative space-aware features. The global scene embedding and regional semantic summary extracted from the collaborative space-aware features, the statistical information extracted from the basic importance response map, and the task prompt information are input into the large language model. After spatial decoding, a cognitive guidance response map is obtained. The basic importance response map and the cognitive guidance response map are fused to obtain a modified importance response map. Finally, a set of collaborative semantic units is formed based on the modified importance response map and the transmission budget.

[0034] The channel-adaptive semantic compression unit is used to select the semantic representation dimension based on the signal-to-noise ratio and obtain a compressed set of cooperative semantic units while keeping the spatial location and importance score of each cooperative semantic unit unchanged.

[0035] The semantically aware joint coding unit is used to perform joint source-channel coding on the compressed cooperative semantic unit set, and then transmits the data after calculating power weights and normalizing global power based on importance scores.

[0036] The vehicle spatial feature encoding unit is used to acquire the three-dimensional point cloud perception data of the vehicle and generate the vehicle spatial perception features. The joint source channel decoding unit is used to decode the received noisy channel symbols to obtain the set of reconstructed cooperative semantic units.

[0037] The sparse perception fusion unit is used to perform geometric scattering, cross-attention aggregation, and gated residual fusion based on the spatial location, semantic features, and vehicle spatial perception features of the reconstructed collaborative semantic unit, to obtain fused spatial perception features and output three-dimensional target perception results.

[0038] The beneficial effects of this invention are as follows: 1. This invention fuses a basic importance response map generated from collaborative spatial perception features with a cognitive guidance response map output from a large language model and obtained through spatial decoding to form a modified importance response map for selecting collaborative semantic units. This allows the content to be transmitted to no longer rely solely on local detection confidence or visual saliency. Since the modified importance response map considers local geometric responses, detection-related responses, and scene semantic information such as occlusion boundaries, distant traffic participants, and key task areas in the bird's-eye view features of the collaborative end, it can prioritize retaining collaborative semantic units that contribute to 3D target perception when transmission budget is limited. This reduces the compression or discarding of key weak response areas, thereby improving the targeting and task relevance of collaborative information selection under limited bandwidth.

[0039] 2. This invention, after determining the set of cooperative semantic units, selects the semantic representation dimension based on the signal-to-noise ratio (SNR) while maintaining the spatial location and importance score of the cooperative semantic units, thus ensuring that adaptive channel compression does not destroy the front-end semantic selection results. Simultaneously, by using the importance score obtained from the modified importance response graph in the joint source channel coding for importance protection gating, power weight calculation, and global power normalization, the importance evaluation from the sensing side can be transmitted to the continuous channel symbol transmission process on the communication side. Therefore, under low SNR conditions, the symbol redundancy per unit semantic dimension can be increased by reducing the semantic representation dimension, and stronger protection can be provided for highly important cooperative semantic units through power reallocation, thereby mitigating the damage to key cooperative semantic information caused by wireless link noise, fading, and SNR fluctuations.

[0040] 3. This invention uses the spatial location of the cooperative semantic unit at the vehicle end for scattering, and restores the reconstructed cooperative semantic unit to the bird's-eye view grid under a unified spatial reference frame. Combined with local convolutional diffusion, cross-attention aggregation, and spatial adaptive gated residual fusion, the fusion process at the receiving end is kept on the same data chain as the front-end semantic selection and communication-side importance protection. The filling value of the spatial grid where no reconstructed cooperative semantic unit falls is set to zero. For cases where multiple reconstructed cooperative semantic units fall into the same spatial grid, they are accumulated and aggregated. Then, cooperative enhancement features are generated by combining geometric enhancement features and semantic enhancement features. This can reduce the impact of sparse transmission, spatial local missingness, and channel distortion on the fusion result, and achieve the effect of improving the stability of 3D target perception in occlusion, long distance, and low signal-to-noise ratio scenarios. Attached Figure Description

[0041] Figure 1 This is a diagram of the overall system architecture of the present invention; Figure 2 This is a detailed architecture diagram of the LLM-guided semantic sparse selection module of the present invention; Figure 3 This is a schematic diagram of the SNR adaptive progressive semantic compression module structure of the present invention; Figure 4 This is a schematic diagram of the semantic-aware JSCC transmission module structure of the present invention; Figure 5 AWGN results of this invention Figure 1 ; Figure 6 AWGN results of this invention Figure 2 ; Figure 7 This is a schematic diagram of the overall architecture of the large language model-driven semantic communication system for collaborative 3D perception according to the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] like Figures 1 to 7 As shown, this embodiment of the invention provides a large language model-driven semantic communication method and system for collaborative 3D perception. It is applied to a vehicle collaborative 3D perception system including a driver vehicle, a cooperating vehicle, and a wireless communication link. The driver vehicle is the end that outputs the 3D target perception result, the cooperating vehicle is the end that provides collaborative perception information to the driver vehicle, and the wireless communication link is used to transmit continuous channel symbols generated by the cooperating vehicle. The 3D target perception result may include target category, 3D bounding box center position, 3D bounding box size, and orientation information.

[0044] Point cloud acquisition, coordinate unification, and spatially aware feature generation: The vehicle-mounted 3D point cloud perception data is collected from the vehicle-mounted terminal, and the cooperative vehicle-mounted 3D point cloud perception data is collected from the cooperative vehicle-mounted terminal. In this embodiment, the 3D point cloud perception data is LiDAR point cloud data. Each frame of LiDAR point cloud data includes multiple point cloud points, and each point cloud point includes at least 3D coordinate information and may also include reflection intensity information.

[0045] Before performing collaborative 3D perception, the system acquires the relative pose relationship between the self-vehicle end and the collaborating vehicle end. The relative pose relationship can be jointly determined by the vehicle positioning module, the inertial measurement module, the vehicle network synchronization information, and the external parameter calibration relationship between vehicles, and at least includes the rotation and translation relationship of the collaborating vehicle end coordinate system relative to the self-vehicle end coordinate system.

[0046] The endpoint cloud of the collaborative vehicle is transformed to the coordinate system of the autonomous vehicle based on the rotation and translation relationship from the coordinate system of the collaborative vehicle to the coordinate system of the autonomous vehicle. This transformation can be achieved using rigid body coordinate transformation commonly used in this field. After the coordinate transformation is completed, point cloud clipping, voxelization, and cylindrical feature aggregation are performed on the endpoint cloud of the autonomous vehicle and the endpoint cloud of the collaborative vehicle within the unified bird's-eye view grid. Point cloud clipping is used to retain point cloud points within a preset perception range around the vehicle; voxelization is used to divide the point cloud points into preset spatial grids; cylindrical feature aggregation is used to aggregate the point cloud point features within the same cylindrical grid into corresponding grid features.

[0047] The aggregated raster features are input into a shared bird's-eye view encoder to obtain autonomous vehicle spatial perception features and cooperative spatial perception features, respectively. For ease of description, in this embodiment, the autonomous vehicle spatial perception BEV feature is denoted as... The collaborative spatial perception features are denoted as .

[0048] The symbols mentioned above are only used to represent feature variables in specific implementations and do not limit the specific network structure.

[0049] Since the endpoint cloud of the collaborative vehicle has been transformed to the unified spatial reference frame corresponding to the vehicle during the feature generation stage, the spatial location of the collaborative semantic unit selected from the collaborative spatial perception features can be directly represented as two-dimensional grid coordinates or normalized two-dimensional grid coordinates in the unified bird's-eye view grid.

[0050] This process provides a consistent spatial reference basis for subsequent scattering, cross-attention aggregation, and gated residual fusion, avoiding ambiguous coordinate transformations at the receiver during the fusion stage.

[0051] Basic Importance Response Map Generation In obtaining collaborative space perception features Then, it is input into the semantic importance head. The semantic importance head includes convolutional layers, normalization layers, non-linear activation layers, and normalized output layers. Convolutional layers are used to extract features related to local texture, geometric boundaries, and detection responses from collaborative spatial perception features; normalization layers are used to stabilize feature distribution; non-linear activation layers are used to enhance feature representation; and normalized output layers are used to compress the response value of each spatial raster to a preset range.

[0052] The semantic importance head outputs a visual importance map with the same spatial grid size as the collaborative spatial perception features. This visual importance map serves as the basic importance response map, denoted as... It satisfies:

[0053] Where H represents the height of the bird's-eye view grid and W represents the width of the bird's-eye view grid, the response value of each grid in the basic importance response map is jointly determined by the local texture features, geometric intensity features and detection correlation response features of the corresponding grid location. The local texture features are used to reflect the changes in the point cloud distribution near the grid; the geometric intensity features are used to reflect the target boundary, contour or obstacle structure near the grid; and the detection correlation response features are used to reflect the degree of correlation between the grid and the 3D target detection task.

[0054] The basic importance response map provides low-level visual and geometric saliency in collaborative spatial perception features, but it mainly comes from local spatial features and may still be insufficient for judging potential targets behind occluded boundaries, distant traffic participants, and weak response areas related to safety. Therefore, this embodiment further introduces the cognitive guidance response map output by the large language model to calibrate the basic importance response map.

[0055] Construction of input sequence for large language model This embodiment does not input the original point cloud, original image, or complete cooperative spatial perception features into the large language model grid by grid. Instead, it constructs a compact large language model input sequence from the cooperative spatial perception features and the basic importance response map.

[0056] First, the characteristics of collaborative space perception. Global pooling is performed, and global scene embedding is obtained through multilayer perceptron mapping. Global scene embedding is used to characterize the overall traffic scene state observed by the current collaborative vehicle, such as target distribution density, road area structure, and scene complexity from the collaborative perspective.

[0057] Secondly, the characteristics of collaborative space perception Coarse-grained region pooling is performed to obtain a region semantic summary. Coarse-grained region pooling can be performed according to the spatial partitioning of the bird's-eye view grid. For example, the collaborative spatial perception features are divided into several region blocks, and average pooling or max pooling is performed on the features in each region block to obtain a low-resolution region semantic summary. The region semantic summary is used to preserve the approximate spatial layout of the collaborative spatial perception features.

[0058] Secondly, from the basic importance response diagram Statistical information is extracted from the data, including the mean, maximum value, region pooling response, and high-response region distribution characteristics of the basic importance response map. The mean is used to describe the overall response level; the maximum value is used to describe the strongest visual response in the current frame; the region pooling response is used to describe the response differences between different regions; and the high-response region distribution characteristics are used to describe the spatial distribution of potential key regions.

[0059] In addition, task prompts are set, including occlusion boundary prompt embedding, distant traffic participant prompt embedding, and detection response prompt embedding. Occlusion boundary prompt embedding is used to enable the large language model to focus on building boundaries, large vehicle boundaries, or intersection edge areas that may generate blind spots; distant traffic participant prompt embedding is used to enable the large language model to focus on candidate areas of distant vehicles, pedestrians, or cyclists; and detection response prompt embedding is used to enable the large language model to focus on spatial regions related to 3D object detection.

[0060] Global scene embedding, regional semantic summarization, statistical information, and task prompt information are mapped to the input dimension of the large language model through modal alignment projection layers and then concatenated in sequence to form the input sequence of the large language model. The modal alignment projection layers can be implemented using linear layers or multilayer perceptrons to ensure that visual features, statistical features, and prompt embeddings have the same vector dimension as the input of the large language model.

[0061] In other implementations, the context of the input large language model can also include target candidate box confidence, occlusion region estimation results, historical frame motion information, road topology priors or vehicle pose information. The above information can be combined with global scene embedding, region semantic summarization, statistical information and task prompt information through modality alignment projection layer mapping and then concatenated to form an extended large language model input sequence.

[0062] In this way, the large language model receives compressed and aligned scene state, visual statistical information, and task prompts, rather than the complete collaborative spatial perception features unfolded grid by grid. This design enables the large language model to participate in semantic region judgment in collaborative 3D perception, rather than directly replacing the 3D detection network or directly outputting the final detection result.

[0063] Cognitive guidance response map generation and revision of importance response map generation After receiving the input sequence, the large language model outputs a sequence of latent state vectors. The large language model can be a pre-trained large language model with its main parameters frozen. It adapts to the vehicle collaborative 3D perception task through low-rank adaptation parameters, task prompt embedding, modality alignment projection layer, and spatial decoder. The latent state vector sequence output by the large language model is not used as a text description, nor is it directly used as the spatial location to be transmitted. Instead, it serves as an intermediate semantic representation that includes occlusion boundaries, candidate regions for distant traffic participants, and judgment information for detection-related regions.

[0064] In other implementations, the large language model can also be replaced by a multimodal language model; the output of the large language model can also be a structured vector, a region-level risk score, or a semantic label. For structured vectors, region-level risk scores, or semantic labels, they can be transformed into a cognitive guidance response map in the bird's-eye view space through a spatial decoder, a cross-attention mapper, or a query-based decoder. The above optional methods do not change the processing logic of calibrating the basic importance response map through the cognitive guidance response map in this embodiment.

[0065] The spatial decoder performs aggregated or cross-attention readings on the latent state vector sequence to obtain an aggregated semantic vector. Subsequently, the spatial decoder uses linear mapping, spatial rearrangement, and upsampling decoding to restore the aggregated semantic vector to a two-dimensional response map aligned with a unified bird's-eye view grid, resulting in a cognitively guided response map, denoted as . It satisfies:

[0066] in, Each grid response value represents the strength of the large language model's cognitive guidance for that spatial location based on the current scene input sequence.

[0067] After obtaining the basic importance response map and cognitive guidance response map Then, learnable weights are used to weight the corresponding raster response values ​​of the two to generate a corrected importance response map I, which can be specifically represented as:

[0068] in, The learnable weights are used to adjust the contribution ratio of the cognitive guidance response map and the basic importance response map in the modified importance response map. The modified importance response map I retains the local visual and geometric information in the collaborative spatial perception features, and also introduces the scene semantic guidance information output by the large language model.

[0069] In other implementations, the fusion method of the basic importance response map and the cognitive guidance response map is not limited to linear weighting. It can also adopt gated multilayer perceptron fusion, convolutional fusion, attention fusion or confidence adaptive fusion. All of the above fusion methods take the basic importance response map and the cognitive guidance response map as input and output a modified importance response map aligned with the unified bird's-eye view grid.

[0070] Compared to schemes that select collaborative features based solely on confidence maps or visual saliency, the selection criteria for collaborative semantic units in this embodiment are derived from the combined results of the basic importance response map and the cognitive guidance response map, rather than a single detection confidence or local texture intensity. This processing enables traceable selection criteria for occlusion boundaries, candidate regions of distant traffic participants, and detection-related weak response regions under a limited transmission budget.

[0071] Selection of Collaborative Semantic Unit Set Within the current transmission cycle, the system determines the number K of cooperative semantic units to be transmitted based on the number of available channel symbols and the channel symbol budget of a single cooperative semantic unit. K is a positive integer. The number of available channel symbols can be determined by the bandwidth resources of the wireless communication link, the scheduling cycle, and the current transmission configuration. The channel symbol budget of a single cooperative semantic unit can be preset by the system or determined based on the current communication resource configuration.

[0072] Before selecting the spatial location to be transmitted, spatial diversity constraints are applied to the modified importance response map I. The spatial diversity constraints include at least one of local maximum suppression, non-maximum suppression, minimum spatial interval constraint, grid partition sampling, and regional equalization sampling.

[0073] Local maximum suppression is used to preserve spatial locations with high responses within a local window; non-maximum suppression is used to reduce the probability of repeated selection of adjacent high-response points; minimum spatial interval constraint is used to limit the distance between selected spatial locations; grid partition sampling is used to ensure that different spatial partitions have candidate locations; regional equalization sampling is used to avoid all cooperative semantic units from being concentrated near the same target or the same occlusion boundary.

[0074] After applying spatial diversity constraints, K spatial locations to be transmitted are selected based on the ranking of response values ​​in the modified importance response map I.

[0075] For the nth spatial location to be transmitted, denoted as:

[0076] in, This represents the spatial location of the nth collaborative semantic unit. This indicates the vertical index of the spatial location within the bird's-eye view grid. This indicates the horizontal index of the spatial location within the bird's-eye view grid.

[0077] when When normalizing two-dimensional raster coordinates, their values ​​can be in the range of [0,1] and are remapped to the bird's-eye view raster index during the scattering stage.

[0078] For each spatial location to be transmitted Perceiving features from collaborative space Extract the semantic features corresponding to the spatial location and the spatial location The response values ​​in the modified importance response diagram I are used as importance scores. Ultimately, this forms a set of collaborative semantic units. , is represented as:

[0079] in, Represents a set of collaborative semantic units. The semantic features representing the nth collaborative semantic unit. This represents the spatial location of the nth collaborative semantic unit. Let K represent the importance score of the nth cooperative semantic unit, and K represent the number of cooperative semantic units selected in the current transmission cycle.

[0080] In this embodiment, the spatial location, semantic features, and importance score of the cooperative semantic unit have a one-to-one correspondence. Subsequent semantic compression, joint source-channel coding, power weight calculation, and receiver fusion all follow this correspondence, without reselecting the spatial location or recalculating the importance score. This is different from the scheme that temporarily estimates the transmission priority during the communication coding stage.

[0081] Channel Adaptive Semantic Compression Collaborative semantic unit set Once determined, the collaborative vehicle acquires the signal-to-noise ratio γ of the wireless communication link.

[0082] The signal-to-noise ratio γ can be calculated by the communication module at the cooperating vehicle end based on the pilot signal, received power, and noise power. When the self-vehicle end feeds back the link quality index, the signal-to-noise ratio γ can also be estimated by the self-vehicle end and sent to the cooperating vehicle end through the control channel. The signal-to-noise ratio γ can be used as input for each frame or each transmission cycle to select the semantic representation dimension of the cooperative semantic unit in the current transmission cycle.

[0083] It should be noted that the signal-to-noise ratio γ does not directly change the number of cooperative semantic units K, nor does it change the spatial location of the selected cooperative semantic units. And importance score .

[0084] The signal-to-noise ratio γ is used only to determine the semantic representation dimension of each cooperative semantic unit, thereby dynamically adjusting the relationship between semantic capacity and noise-resistant redundancy under a fixed channel symbol budget.

[0085] The target semantic representation dimension is selected from multiple preset semantic representation dimensions based on the signal-to-noise ratio γ. , can be represented as:

[0086] in, , and Let these represent the first semantic representation dimension, the second semantic representation dimension, and the third semantic representation dimension, respectively. < < .

[0087] In a specific implementation, the first semantic representation dimension, the second semantic representation dimension, and the third semantic representation dimension are 32, 64, and 128 dimensions, respectively.

[0088] The corresponding routing rules are as follows: when -10dB≤γ<3dB, a 32-dimensional semantic representation dimension is selected; when 3dB≤γ<20dB, a 64-dimensional semantic representation dimension is selected; and when 20dB≤γ≤30dB, a 128-dimensional semantic representation dimension is selected.

[0089] The above thresholds can be adjusted based on the actual wireless communication link environment, target detection accuracy requirements, and communication resource budget.

[0090] The reason for choosing a lower semantic representation dimension under low signal-to-noise ratio (SNR) conditions is that the channel symbol budget for each cooperative semantic unit remains fixed. If a higher-dimensional semantic representation is still transmitted under low SNR, the symbol protection available for each semantic dimension is insufficient and easily destroyed by channel noise. Therefore, choosing a lower dimension under low SNR is equivalent to protecting fewer semantic dimensions with a fixed number of channel symbols, thereby improving the noise resistance redundancy of a unit semantic dimension. Under high SNR conditions, the channel quality is better, and the system can choose a higher dimension to transmit richer semantic content and improve the upper limit of the three-dimensional target perception results.

[0091] During compression, the common implicit representation of cooperative semantic units is first extracted through shared coding subunits. For the semantic features of the nth cooperative semantic unit... A common implicit representation is obtained by sharing coding subunits. .

[0092] Subsequently, based on the target semantic representation dimension The corresponding compression header is invoked to obtain the compressed semantic vector. , can be represented as:

[0093] in, This represents the compressed semantic vector of the nth cooperative semantic unit at the lth compression level. This represents the common implicit representation of the output of the shared coding subunit. This represents the linear mapping weight of the l-th compression head. This represents the bias of the l-th compression head. This indicates the target semantic representation dimension corresponding to the current signal-to-noise ratio.

[0094] To preserve spatial location information in compressed semantic features, a position encoder is used to determine the spatial location of the nth cooperative semantic unit. Mapped to a position embedding vector that matches the dimension of the current semantic representation. , can be represented as:

[0095] in, This represents the position embedding vector of the nth cooperative semantic unit at the lth compression level. This indicates the position encoder corresponding to the first compression level.

[0096] Compress semantic vectors With position embedding vector By fusion, compressed collaborative semantic features are obtained. , can be represented as:

[0097] in, This represents the compressed collaborative semantic features after integrating semantic content and spatial location.

[0098] Therefore, compressed collaborative semantic features simultaneously include semantic content and spatial location.

[0099] For each collaborative semantic unit, only its semantic features are compressed, without changing its spatial location. And importance score This ultimately yields a set of compressed collaborative semantic units.

[0100] In practical deployments, to avoid frequent switching of compression levels due to short-term fluctuations in the signal-to-noise ratio (SNR) near the threshold, the SNR of several consecutive frames or several consecutive transmission cycles can be averaged or exponentially smoothed before being input into the dimension routing subunit. Alternatively, a hysteresis interval can be set between adjacent semantic representation dimensions, so that the system can only switch semantic representation dimensions when the SNR is continuously higher than the upper switching threshold or continuously lower than the lower switching threshold. Soft routing can also be used to generate fusion weights for multiple compression heads based on the current SNR.

[0101] Since different compression levels can share the front-end encoder and only the end compression head is switched, the switching overhead is low. The above stabilization process is an optional implementation of signal-to-noise ratio routing and does not change the core processing logic of selecting the semantic representation dimension based on the signal-to-noise ratio.

[0102] In other implementations, continuous mapping rules can be used to make the compression dimension or the fusion weight of multiple compression heads change continuously with the signal-to-noise ratio; routing strategies can also be jointly learned based on task loss, detection confidence or link quality. The above methods are extended implementations, but the core is still to dynamically adjust the semantic representation dimension based on channel quality.

[0103] Semantic-aware joint source-channel coding and power weight calculation The collaborative vehicle performs joint source-channel coding on the compressed collaborative semantic unit set.

[0104] For the nth compressed cooperative semantic unit, its compressed cooperative semantic features The signal passes sequentially through an input linear projection layer, several residual coding blocks, a post-normalization layer, a signal-to-noise ratio conditional modulation module, an importance protection gating module, an optional rate control mask, an optional cooperative semantic unit mask, and amplitude limiting to obtain the corresponding real-valued transmission features. .

[0105] The input linear projection layer is used to map compressed cooperative semantic features to a fixed-dimensional channel symbol representation space. The residual coding block is used to extract semantic representations suitable for wireless transmission. Each residual coding block may include layer normalization, multilayer perceptron, channel attention module and residual connection.

[0106] The multilayer perceptron can be composed of a linear layer, an activation layer, and another linear layer connected in sequence, with a post-normalization layer used to stabilize the feature distribution of the residual coding block output.

[0107] The signal-to-noise ratio conditional modulation module adjusts the coding features according to the current signal-to-noise ratio, so that the same semantic feature has different transmission representations under different channel conditions.

[0108] The importance protection gating module is used to assign importance scores to collaborative semantic units. An encoding process is introduced to enable highly important cooperative semantic units to obtain higher equivalent protection strength during the encoding stage. Optional rate control masks are used to shield channel symbols that exceed the budget under different transmission budgets. Optional cooperative semantic unit masks are used to shield invalid cooperative semantic units generated by padding. Amplitude limiting can be implemented using a hyperbolic tangent activation function to limit the amplitude range of real-valued transmission features.

[0109] Subsequently, the real value will be sent as a feature. The in-phase and quadrature components are packaged into continuous channel symbols. That is, the nth compressed cooperative semantic unit is mapped into a continuous channel symbol for wireless transmission after joint source channel coding, instead of being compressed into a bit stream and then modulated in a conventional way.

[0110] When calculating power weights, the system uses the importance scores of each cooperative semantic unit within the same transmission cycle.

[0111] First, calculate the mean of all importance scores within the current transmission cycle, denoted as . .

[0112] For the nth collaborative semantic unit, based on its importance score mean and preset stability terms Calculate relative power weights , can be represented as:

[0113] in, This represents the relative power weight of the nth cooperative semantic unit. This represents the importance score of the nth collaborative semantic unit. This represents the mean of the importance scores of all cooperative semantic units within the same transmission period. This indicates a preset stability term, used to avoid instability in weight calculation caused by a denominator that is zero or close to zero.

[0114] Importance score The higher the value, the higher the relative power weight. The larger the value, the better. To prevent low-importance cooperative semantic units from completely losing their basic transmission power, a smoothing coefficient or minimum weight lower limit can also be set. This setting is an optional stabilization method for power weight calculation.

[0115] Based on relative power weight The transmission amplitude of the corresponding continuous channel symbols is adjusted. Then, global power normalization is performed on all the adjusted continuous channel symbols so that the average transmission power of all continuous channel symbols does not exceed the preset power budget. It should be noted that this embodiment does not increase the total transmission power, but rather redistributes the power of continuous channel symbols corresponding to different cooperative semantic units under the constraint of average transmission power.

[0116] Importance score for the use of importance protection gating Inherited from the modified importance response diagram I, it is not recalculated before joint source-channel coding; that is to say... It is used both to determine the importance of the spatial location to be transmitted when selecting cooperative semantic units and to redistribute power after joint source-channel coding, forming a continuous data chain from semantic selection on the sensing side to power processing on the communication side.

[0117] In this embodiment, unequal error protection is mainly achieved through importance-driven power redistribution. At the same time, signal-to-noise ratio adaptive semantic compression reduces the semantic representation dimension under low signal-to-noise ratio conditions, so that a fixed number of channel symbols carry less semantic dimension, thereby improving the symbol redundancy per unit semantic dimension.

[0118] In other implementations, the protection strength of highly important cooperative semantic units can be further enhanced by combining variable symbol allocation, stronger coding headers, repeated transmissions, or layered modulation methods. The above optional methods do not change the core mechanism in this embodiment where the importance score is inherited from the modified importance response graph and used for power redistribution on the communication side.

[0119] Wireless communication link transmission and joint source channel decoding The continuous channel symbols, after power shaping and global power normalization, are transmitted to the vehicle end via a wireless communication link. This wireless communication link can employ an additive white Gaussian noise channel, a Rayleigh flat fading channel, a frequency-selective Rayleigh fading channel, or be modeled using a custom channel model given by an external channel frequency response. The noisy channel symbols obtained at the receiver can be represented as follows:

[0120] in, This represents the noisy channel symbol corresponding to the nth cooperative semantic unit received from the vehicle. This represents the consecutive channel symbols corresponding to the nth cooperative semantic unit sent by the cooperative vehicle. This represents the channel coefficients corresponding to the wireless communication link. This represents complex Gaussian noise. In an additive white Gaussian noise channel, we can let... =1; In Rayleigh flat fading channels These are the random fading coefficients; in frequency-selective Rayleigh fading channels, It can vary with symbol subcarriers or time to characterize the differences in channel frequency response caused by multipath propagation.

[0121] The vehicle end will display noisy channel symbols The input joint source-channel decoding unit includes, in sequence, input linear projection, signal-to-noise ratio conditional modulation, several decoding blocks, channel attention, layer normalization, and output linear projection.

[0122] Input linear projection is used to map noisy channel symbols to the decoding feature space; signal-to-noise ratio conditional modulation is used to adjust the decoding features according to the current link conditions; channel attention and layer normalization are used to stabilize the feature recovery process; output linear projection is used to obtain the semantic features of the reconstructed cooperative semantic units.

[0123] Each decoding block may include a first layer of normalization, multi-head self-attention, a first residual connection, a second layer of normalization, a feedforward network, and a second residual connection. Multi-head self-attention is used to model the contextual relationships between different cooperative semantic units. The feedforward network is used to perform nonlinear mapping on the decoding features of each cooperative semantic unit. The residual connection is used to stabilize the deep decoding process. The feedforward network may be composed of a linear layer, an activation layer, and a linear layer connected in sequence.

[0124] If there are invalid cooperative semantic units (SJUs) that are filled during the transmission cycle, the invalid cooperative semantic units are masked during the attention calculation in the decoding process and set to zero in the output stage.

[0125] After decoding at the vehicle end, a reconstructed set of collaborative semantic units corresponding to the set of collaborative semantic units is obtained. Each reconstructed collaborative semantic unit retains the spatial location of its corresponding collaborative semantic unit. .

[0126] In the joint source-channel decoding unit, the residual gating unit is used to perform weighted fusion between the decoding prediction result and the received input features to stabilize the decoding process and suppress noise amplification. This residual gating unit is not an explicit per-cooperative semantic unit distortion detector, nor does it need to judge whether a cooperative semantic unit is severely distorted one by one. Severely distorted or invalid cooperative semantic units are mainly handled through two types of mechanisms: First, at the transmitting end, high-importance cooperative semantic units obtain higher equivalent protection strength through importance protection gating and power redistribution; second, at the receiving end, invalid cooperative semantic units generated by padding are masked in the attention calculation and set to zero in the output stage.

[0127] In other implementations, to further enhance distortion perception capabilities, a quality estimator can be introduced. The quality estimator can generate dynamic cooperative semantic unit-level gating weights based on received symbol amplitude, signal-to-noise ratio estimation, reconstruction consistency, or contextual consistency among cooperative semantic units. The dynamic cooperative semantic unit-level gating weights can be used to adjust the participation intensity of the corresponding reconstructed cooperative semantic unit in subsequent scattering, cross-attention aggregation, or gated residual fusion. This optional implementation does not change the core mechanism of transmitter importance protection based on the modified importance response map in this embodiment.

[0128] Sparse cooperative spatial feature construction and geometry enhancement After obtaining the set of reconstructed collaborative semantic units from the vehicle, the semantic features of each reconstructed collaborative semantic unit are first mapped to the same channel dimension as the vehicle's spatial perception features through linear mapping or multilayer perceptron. This channel dimension adaptation is used to ensure that after the reconstructed collaborative semantic units are scattered onto the bird's-eye view grid, they can be subsequently fused with the vehicle's spatial perception features in the channel dimension.

[0129] Subsequently, the vehicle-side reconstructs the spatial location of the collaborative semantic units. The semantic features of the reconstructed collaborative semantic units after channel dimension adaptation are scattered onto a spatial grid under a unified spatial reference system, forming sparse collaborative spatial features. .

[0130] Since the spatial position of the reconstructed collaborative semantic unit is derived from the spatial position of the collaborative semantic unit, and the spatial position of the collaborative semantic unit has already been represented in the front end as two-dimensional grid coordinates or normalized two-dimensional grid coordinates under a unified spatial reference system, the transformation of the vehicle pose matrix or extrinsic parameter matrix is ​​no longer performed during the scattering process.

[0131] During scattering, if To normalize the two-dimensional grid coordinates, first convert them into integer grid indices according to the height and width of the bird's-eye view grid corresponding to the vehicle's spatial perception features; if If it is already an absolute raster index, it will be directly cropped to the valid bird's-eye view raster area for use.

[0132] If no reconstructed collaborative semantic units fall into a spatial grid, the fill value of that spatial grid is set to zero; if multiple reconstructed collaborative semantic units fall into the same spatial grid, the corresponding semantic features are accumulated and aggregated.

[0133] Subsequently, the characteristics of sparse cooperative spaces were studied. Local convolution diffusion processing is performed to obtain geometrically enhanced features. .

[0134] In a specific implementation, a 3x3 convolution kernel is used for local feature diffusion, which can be represented as:

[0135] in, This represents the characteristics of sparse cooperative spaces. Indicates geometric enhancement features, This indicates a 3x3 convolution process.

[0136] The purpose of this convolutional processing is not to re-align the global data, but to propagate the local information of sparse cooperative semantic units to adjacent grids in the bird's-eye view grid corresponding to the vehicle end, thereby enhancing the local spatial continuity, alleviating the hole problem caused by sparse reception, and preserving the target boundary and geometric structure as much as possible.

[0137] Cross-attention aggregation and gated residual fusion In the semantic aggregation process, the vehicle's spatial perception features are first... Expand into a query sequence and add location coding to preserve the vehicle's spatial layout.

[0138] The semantic features of the reconstructed collaborative semantic unit are fused with its positional encoding to obtain the key sequence.

[0139] The semantic features of the reconstructed collaborative semantic units are used as a value sequence.

[0140] The query sequence, key sequence, and value sequence are input across the attention aggregation module to obtain semantically enhanced features. .

[0141] The cross-attention aggregation module aggregates sparse collaborative information into the spatial perception representation of the vehicle by matching the feature relationship between the vehicle's spatial perception features and the reconstructed collaborative semantic units.

[0142] Geometric Enhancement Features and semantic enhancement features After weighted fusion, collaborative enhancement features are obtained. , can be represented as:

[0143] in, This indicates a collaborative enhancement feature. This represents semantic enhancement features. Indicates geometric enhancement features, This indicates learnable fusion weights. This means restoring the semantically enhanced features to the spatial shape corresponding to the bird's-eye view grid.

[0144] Subsequently, the vehicle's spatial perception features and collaboration enhancement features The graph is stitched together along the channel dimension and then input into a gating network to obtain a spatially adaptive gating graph. , can be represented as:

[0145] in, Represents an adaptive gating graph in the representation space. Represents the normalized activation function. Indicates a gating network. , This indicates that the vehicle's spatial perception features and collaborative enhancement features are spliced ​​together along the channel dimension.

[0146] Through spatial adaptive gating graph Enhance collaborative features Injecting vehicle spatial perception features using residual methods To obtain fused spatial perception features , can be represented as:

[0147] in, This indicates the fusion of spatial perception features. This indicates element-wise multiplication.

[0148] The spatially adaptive gating graph is used to represent the reliability and usefulness of cooperative information at each grid location in the bird's-eye view. When the cooperative features of certain areas are greatly affected by channel noise, sparse reception, or local misalignment, the gating weight is reduced to decrease the injection of unreliable cooperative information. When the cooperative semantic unit effectively complements the vehicle's perception, the gating weight is increased to enhance the utilization of cooperative information in that area.

[0149] Fusion of spatial perception features The input is a 3D detection head, which can include a classification branch, a position regression branch, a size regression branch, and an orientation regression branch. The output is a 3D target perception result, which includes the target category, the center position of the 3D bounding box, the size of the 3D bounding box, and the orientation information.

[0150] Through the above processing, the receiving end does not simply splice collaborative features, nor does it just perform conventional fusion of dense bird's-eye view features. Instead, it scatters the reconstructed collaborative semantic units under a unified spatial reference frame, and then injects collaborative information into the vehicle's spatial perception features through geometric enhancement, cross-attention aggregation, and gated residual fusion. This processing can form a continuous data chain with the spatial location selection of the front-end collaborative semantic units and the importance protection of the communication side.

[0151] System Implementation Examples This embodiment also provides a large language model-driven semantic communication system for collaborative 3D perception. The system includes a self-driving end, a collaborative vehicle end, and a wireless communication link.

[0152] The collaborative vehicle terminal includes a collaborative spatial feature encoding unit, a large language model-driven semantic selection unit, a channel adaptive semantic compression unit, and a semantic-aware joint encoding unit.

[0153] The collaborative spatial feature encoding unit is used to convert the 3D point cloud perception data of the collaborative vehicle to the unified spatial reference system corresponding to the self vehicle based on the relative pose relationship between the self vehicle and the collaborative vehicle, and to generate collaborative spatial perception features.

[0154] The large language model-driven semantic selection unit includes a semantic importance head, a semantic context construction subunit, a large language model, a spatial decoder, and a sparse selection subunit. The semantic importance head is used to generate a visual importance map based on collaborative spatial perception features, and the visual importance map serves as the basic importance response map.

[0155] The semantic context construction subunit is used to extract global scene embeddings and regional semantic summaries from collaborative space-aware features, extract statistical information from the basic importance response map, and generate input sequences for the large language model by combining task prompts.

[0156] The spatial decoder is used to map the sequence of latent state vectors output by the large language model into a cognitive guidance response map. The sparse selection sub-unit is used to fuse the cognitive guidance response map and the visual importance map to form a modified importance response map, and select a set of cooperative semantic units under the constraint of spatial diversity.

[0157] The channel-adaptive semantic compression unit includes a signal-to-noise ratio (SNR) acquisition subunit, a dimension routing subunit, a shared coding subunit, and a multi-level compression subunit. The SNR acquisition subunit acquires the SNR of the wireless communication link; the dimension routing subunit selects the target semantic representation dimension based on the SNR; and the shared coding subunit extracts the common implicit representation of the cooperative semantic units. The multi-level compression subunit outputs a compressed set of cooperative semantic units according to the target semantic representation dimension.

[0158] The semantically aware joint coding unit is used to perform joint source-channel coding on the compressed cooperative semantic unit set, and to transmit the data after calculating power weights and normalizing global power based on the importance scores of the cooperative semantic units. The importance scores are inherited from the modified importance response map.

[0159] The vehicle-side component includes a vehicle spatial feature encoding unit, a joint source-channel decoding unit, and a sparse sensing fusion unit. The vehicle spatial feature encoding unit is used to acquire the vehicle's 3D point cloud sensing data and generate vehicle spatial sensing features. The joint source-channel decoding unit is used to decode the received noisy channel symbols to obtain a set of reconstructed cooperative semantic units. The sparse sensing fusion unit includes a geometric branch, a semantic branch, and a gated fusion subunit. The geometric branch is used to scatter the reconstructed cooperative semantic units onto a spatial grid under a unified spatial reference frame and perform local convolutional diffusion processing.

[0160] The semantic branch is used to perform cross-attention aggregation with the vehicle's spatial perception features as the query, the semantic features and position encoding of the reconstructed collaborative semantic unit as the key, and the semantic features of the reconstructed collaborative semantic unit as the value. The gating fusion subunit is used to generate a spatial adaptive gating graph based on the vehicle's spatial perception features and collaborative enhancement features and perform residual fusion.

[0161] Model training and deployment methods To enable the modules in this embodiment to work together, training can be performed using publicly available vehicle cooperative perception datasets, simulated vehicle cooperative perception datasets, or self-collected vehicle cooperative perception data. The training data includes the endpoint cloud of the autonomous vehicle, the endpoint cloud of the cooperative vehicle, the relative pose relationship between the autonomous vehicle and the cooperative vehicle, the signal-to-noise ratio parameters of the wireless communication link, and the 3D target box labels.

[0162] In a specific implementation, the publicly available inter-vehicle cooperative perception dataset OPV2V can be used for training. OPV2V is a publicly available inter-vehicle cooperative perception dataset that can be jointly constructed by a cooperative driving simulation framework, an autonomous driving simulation platform, and a traffic flow simulation platform. It can provide the point cloud of the autonomous vehicle, the point cloud of the cooperative vehicle, the vehicle pose transformation relationship, and the 3D target bounding box labels.

[0163] Other publicly available vehicle cooperative perception datasets, simulation datasets, or self-collected data can also be used. When using different training data, as long as the training data can provide the endpoint cloud of the autonomous vehicle, the endpoint cloud of the cooperative vehicle, the pose relationship between vehicles, and the 3D target box labels, it can be used to train the vehicle cooperative 3D perception system of this embodiment.

[0164] The training process can be carried out in stages: In the first stage, the autonomous vehicle spatial feature coding unit, the cooperative spatial feature coding unit, and the 3D detection head are trained under ideal communication conditions, enabling the system to have basic 3D target perception capabilities.

[0165] In the second stage, the detection backbone is fixed or partially fixed, and a large language model is trained to drive semantic selection units, channel-adaptive semantic compression units, and semantic-aware joint coding units, enabling the system to learn how to select, compress, and recover key collaborative semantic units under limited bandwidth and different channel conditions.

[0166] In the third stage, the overall system is jointly fine-tuned to ensure that the data transmission relationship remains consistent among the revised importance response map, the collaborative semantic unit set, the compressed collaborative semantic unit set, the reconstructed collaborative semantic unit set, and the fused spatial perception features.

[0167] When training the relevant parts of the large language model, the main parameters of the large language model can be frozen. The training mainly focuses on low-rank adaptation parameters, learnable cue vectors, projection layers of visual statistics into the latent space of the large language model, spatial decoders, and subsequent semantic selection modules. Low-rank adaptation parameters can be inserted into the attention projection layer of the large language model or into the linear projection layer of the feedforward network to enable the large language model to better adapt to occlusion risk identification, distant target attention, and key task region selection in bird's-eye view collaborative perception.

[0168] During training, different signal-to-noise ratios can be randomly sampled within a preset signal-to-noise ratio range. The wireless channel module can perturb the continuous channel symbols after joint source channel coding. The wireless channel module can simulate additive white Gaussian noise channels, Rayleigh flat fading channels, frequency-selective Rayleigh fading channels, or custom channel models given by external channel frequency responses, so that the model covers low, medium, and high signal-to-noise ratio conditions during training.

[0169] Training losses can include 3D object detection loss, semantic topology constraint loss, semantic reconstruction loss, cooperative semantic unit selection constraint loss, power constraint loss, and fusion consistency loss.

[0170] The 3D target detection loss is used to supervise the final 3D target perception result; the semantic topology constraint loss is used to constrain the modified importance response map after the large language model is guided, so that it pays more attention to real targets, occlusion boundaries and safety-critical areas; the semantic reconstruction loss is used to constrain the reconstructed cooperative semantic units after joint source channel decoding to be consistent with the compressed cooperative semantic units before transmission; the cooperative semantic unit selection constraint loss is used to constrain the spatial distribution and task relevance of cooperative semantic units; the power constraint loss is used to constrain the average transmit power after global power normalization to not exceed the preset power budget; and the fusion consistency loss is used to constrain the task consistency between the fused spatial perception features and the vehicle's spatial perception features and cooperative enhancement features.

[0171] This can be implemented by those skilled in the art based on a vehicle collaborative 3D perception network and wireless communication simulation environment.

[0172] like Figures 5 to 6 Effect Verification Example To verify the effectiveness of this embodiment in harsh wireless communication environments, simulation tests can be conducted on a publicly available vehicle cooperative perception dataset. Different signal-to-noise ratio conditions are set using an additive white Gaussian noise channel. During the test, the driver vehicle and the cooperating vehicle respectively input the corresponding point cloud data. The cooperating vehicle generates a set of cooperative semantic units according to the method of this embodiment, and transmits it after channel adaptive semantic compression, joint source channel coding, power reallocation, and global power normalization. The driver vehicle receives the noisy channel symbols, and outputs the three-dimensional target perception results after joint source channel decoding, sparse cooperative space feature construction, cross-attention aggregation, and gated residual fusion.

[0173] Under harsh channel conditions of -10 dB, this embodiment achieves an average accuracy of 0.725, which is an improvement of about 6.6% compared to the SComCP scheme's 0.680; under high signal-to-noise ratio conditions of 20 dB, this embodiment still maintains an advantage of about 3.0%.

[0174] Compared with traditional digital communication schemes that employ low-density parity-check coding and hexadecimal quadrature amplitude modulation, this embodiment still exhibits smoother performance variations at low signal-to-noise ratios.

[0175] Traditional digital communication schemes tend to drop in performance to the level of single-vehicle perception under low signal-to-noise ratio conditions. However, this embodiment still maintains a high average target detection accuracy under -10 dB conditions, indicating that by modifying the cooperative semantic unit selection driven by the importance response map, channel adaptive semantic compression, and importance-driven power reallocation, the semantic information of the mission's critical areas can be prioritized for protection under adverse channel conditions.

[0176] The above test results are used to illustrate the effect of this embodiment under specific dataset and channel simulation conditions, and are not intended to limit the present invention to using the same dataset, the same evaluation index or the same channel parameters. In actual vehicle platforms, the training data, signal-to-noise ratio range and evaluation index can be reset according to sensor configuration, communication standard, detection task and road scene.

[0177] Differences between this embodiment and conventional solutions This embodiment does not simply send the bird's-eye view features from the collaborative vehicle end directly to the self vehicle end, nor does it perform feature compression solely based on detection confidence or visual saliency. In this embodiment, a modified importance response map is first formed by combining the basic importance response map and the cognitive guidance response map output by the large language model. Then, the spatial location and importance score of the collaborative semantic unit are determined simultaneously by the modified importance response map.

[0178] This embodiment does not simply adjust the communication bit rate based on the signal-to-noise ratio (SNR). The SNR is only used to select the target semantic representation dimension among multiple semantic representation dimensions, without changing the spatial location and importance score of the already selected cooperative semantic units. Therefore, the selection of cooperative semantic units, semantic compression, and receiver fusion maintain the same spatial location reference basis.

[0179] This embodiment is not a typical joint source-channel coding transmission. The power weight calculation in joint source-channel coding uses importance scores inherited from the modified importance response map, and satisfies a preset power budget through global power normalization. These importance scores are not recalculated before communication coding, but are generated in the front-end semantic selection stage and continue to be used in the communication stage.

[0180] In this embodiment, the receiving end does not simply splice together collaborative features and vehicle spatial perception features. Instead, it uses the spatial position of the reconstructed collaborative semantic unit for scattering and fuses it through geometric branches, semantic branches, and gated fusion subunits. Thus, the front-end selection, communication transmission, and receiving end fusion revolve around the same set of collaborative semantic units, forming a complete collaborative processing chain.

[0181] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0182] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A large language model-driven semantic communication method for collaborative 3D perception, characterized in that, The method, applied to a vehicle collaborative 3D perception system including the driver's end, the cooperating vehicle end, and a wireless communication link, includes: Acquire 3D point cloud perception data from the vehicle and the collaborating vehicle. Based on the relative poses of the two ends, convert the 3D point cloud perception data from the collaborating vehicle to the unified spatial reference system corresponding to the vehicle, and generate the vehicle spatial perception features and the collaborating spatial perception features. A basic importance response map is generated from the collaborative spatial perception features. The global scene embedding and regional semantic summary extracted from the collaborative spatial perception features, the statistical information extracted from the basic importance response map, and the task prompt information are input into the large language model. The cognitive guidance response map is obtained through spatial decoding. A modified importance response map is obtained by fusing the basic importance response map and the cognitive guidance response map. Based on the modified importance response map and the transmission budget, a set of collaborative semantic units is selected from the collaborative spatial perception features. Each collaborative semantic unit includes semantic features, spatial location, and importance score determined by the modified importance response map. The semantic representation dimension is selected based on the signal-to-noise ratio, and the collaborative semantic units are compressed while keeping the spatial position and importance score of each collaborative semantic unit unchanged, resulting in a compressed set of collaborative semantic units. Joint source-channel coding is performed on the compressed cooperative semantic unit set, and power weights are calculated and global power is normalized before transmission; The vehicle decodes to obtain a set of reconstructed collaborative semantic units. Based on the spatial location, semantic features and spatial perception features of the reconstructed collaborative semantic units, geometric scattering, cross-attention aggregation and gating residual fusion are performed to obtain fused spatial perception features and output three-dimensional target perception results.

2. The method for semantic communication driven by a large language model for collaborative 3D perception as described in claim 1, characterized in that: The three-dimensional point cloud perception data is LiDAR point cloud data; when generating autonomous vehicle spatial perception features and collaborative spatial perception features, the LiDAR point cloud data of the collaborative vehicle is projected onto the autonomous vehicle coordinate system, and point cloud clipping, voxelization and cylinder feature aggregation are performed sequentially within the unified bird's-eye view grid, and then autonomous vehicle BEV features and collaborative BEV features are obtained through a shared BEV encoder; the spatial position of the collaborative semantic unit is the two-dimensional grid coordinates or normalized two-dimensional grid coordinates in the unified bird's-eye view grid.

3. The method for semantic communication driven by a large language model for collaborative 3D perception as described in claim 2, characterized in that: When generating the basic importance response map, the collaborative spatial awareness features are input into the semantic importance head, which includes a convolutional layer, a normalization layer, a nonlinear activation layer, and a sigmoid output layer, and outputs a visual importance map with the same spatial grid size as the collaborative spatial awareness features. The visual importance map serves as the basic importance response map, and its grid response value is jointly determined by the local texture features, geometric intensity features, and detection-related response features at the corresponding grid position.

4. The semantic communication method driven by a large language model for collaborative 3D perception as described in claim 3, characterized in that: The input to the large language model does not include full collaborative spatial awareness features expanded grid-by-grid; The global scene embedding is obtained by global pooling and multilayer perceptron mapping of collaborative spatial perception features; the regional semantic summary is obtained by coarse-grained regional pooling of collaborative spatial perception features; the statistical information includes the mean, maximum, regional pooling response, and high-response region distribution features of the basic importance response map; The task prompt information includes at least one of the learnable prompt embeddings related to occluded areas, distant targets, target detection responses, or safety-critical areas; the global scene embedding, regional semantic summarization, statistical information, and task prompt information are mapped to the input dimension of the large language model and then concatenated in sequence to form the input sequence of the large language model.

5. A large language model-driven semantic communication method for collaborative 3D perception according to claim 4, characterized in that: The large language model outputs a sequence of latent state vectors. During spatial decoding, the sequence of latent state vectors is aggregated or cross-attention read, and then subjected to linear mapping, spatial rearrangement, and upsampling decoding to obtain a cognitive guidance response map aligned with the bird's-eye view grid under a unified spatial reference system. When generating the revised importance response map, learnable weights are used to weight the corresponding grid response values ​​of the basic importance response map and the cognitive guidance response map.

6. A large language model-driven semantic communication method for collaborative 3D perception according to claim 5, characterized in that: When selecting a set of cooperative semantic units, the number K of cooperative semantic units to be transmitted is first determined based on the number of available channel symbols in the current transmission cycle and the channel symbol budget of a single cooperative semantic unit. Then, spatial diversity is applied to the modified importance response map, and K spatial locations to be transmitted are selected according to the response value sorting result of the modified importance response map. The spatial diversity includes at least one of local maximum suppression, non-maximum suppression, minimum spatial interval constraint, grid partition sampling, and regional equalization sampling. Before sorting and selection, adjacent high-response spatial locations in the same local region are suppressed. The importance score is the response value of the spatial location to be transmitted in the modified importance response map.

7. A large language model-driven semantic communication method for collaborative 3D perception according to claim 6, characterized in that: The signal-to-noise ratio (SNR) is calculated by the communication module of the cooperating vehicle based on the pilot signal, received power, and noise power, or estimated by the autonomous vehicle and transmitted to the cooperating vehicle via the control channel. When selecting a semantic representation dimension based on the SNR, a first SNR threshold and a second SNR threshold are set, with the second SNR threshold being greater than the first SNR threshold. When the SNR is less than the first SNR threshold, the first semantic representation dimension is selected. When the SNR is not less than the first SNR threshold and is less than the second SNR threshold, the second semantic representation dimension is selected. When the SNR is not less than the second SNR threshold, the third semantic representation dimension is selected. The first semantic representation dimension is less than the second semantic representation dimension, and the second semantic representation dimension is less than the third semantic representation dimension.

8. A large language model-driven semantic communication method for collaborative 3D perception according to claim 7, characterized in that: When performing joint source channel coding, each compressed cooperative semantic unit sequentially undergoes input linear projection, residual coding, channel attention, signal-to-noise ratio conditional modulation, importance protection gating, and amplitude limiting to obtain the corresponding real-valued transmission features. The real-valued transmission features are then packaged into continuous channel symbols according to in-phase and quadrature components. When calculating power weights, the relative power weights of each cooperative semantic unit are calculated based on the importance scores of each cooperative semantic unit within the same transmission period, the mean of all importance scores within the same transmission period, and a preset stability term. The importance score used in the importance protection gating is inherited from the modified importance response graph and is not recalculated before joint source-channel coding; After obtaining the set of reconstructed collaborative semantic units from the vehicle-side decoding, the filling value of the spatial grid where no reconstructed collaborative semantic units fall is set to zero, and the semantic features of multiple reconstructed collaborative semantic units falling into the same spatial grid are accumulated and aggregated. When performing cross-attention aggregation, the vehicle's spatial perception features are used as the query, the semantic features and positional encoding of the reconstructed collaborative semantic unit are used as the key, and the semantic features of the reconstructed collaborative semantic unit are used as the value.

9. A large language model-driven semantic communication system for collaborative 3D perception, applied to the large language model-driven semantic communication method for collaborative 3D perception as described in any one of claims 1 to 8, characterized in that, This includes the vehicle-side, the collaborative vehicle-side, and the wireless communication link; The collaborative vehicle terminal includes a collaborative spatial feature encoding unit, a large language model-driven semantic selection unit, a channel adaptive semantic compression unit, and a semantic perception joint encoding unit. The collaborative space feature encoding unit is used to convert the 3D point cloud perception data of the collaborative vehicle to the unified spatial reference system corresponding to the self vehicle based on the relative pose relationship between the self vehicle and the collaborative vehicle, and to generate collaborative space perception features. The large language model-driven semantic selection unit is used to generate a basic importance response map based on the collaborative space perception features. The global scene embedding and regional semantic summary extracted from the collaborative space perception features, the statistical information extracted from the basic importance response map, and the task prompt information are input into the large language model. After spatial decoding, a cognitive guidance response map is obtained. The basic importance response map and the cognitive guidance response map are fused to obtain a modified importance response map. Finally, a set of collaborative semantic units is formed based on the modified importance response map and the transmission budget. The channel adaptive semantic compression unit is used to select the semantic representation dimension according to the signal-to-noise ratio, and to obtain a compressed set of cooperative semantic units while keeping the spatial position and importance score of each cooperative semantic unit unchanged. The semantically aware joint coding unit is used to perform joint source-channel coding on the compressed cooperative semantic unit set, and then transmit it after calculating power weights and normalizing global power based on importance scores; The vehicle-mounted unit includes a vehicle spatial feature coding unit, a joint source-channel decoding unit, and a sparse sensing fusion unit. The vehicle spatial feature encoding unit is used to acquire the three-dimensional point cloud perception data of the vehicle and generate the vehicle spatial perception features. The joint source channel decoding unit is used to decode the received noisy channel symbols to obtain a set of reconstructed cooperative semantic units; The sparse perception fusion unit is used to perform geometric scattering, cross-attention aggregation, and gated residual fusion based on the spatial location, semantic features, and vehicle spatial perception features of the reconstructed collaborative semantic unit, to obtain fused spatial perception features and output three-dimensional target perception results.

10. A large language model-driven semantic communication system for collaborative 3D perception according to claim 9, characterized in that: The large language model-driven semantic selection unit includes a semantic importance head, a semantic context construction subunit, a large language model, a spatial decoder, and a sparse selection subunit. The semantic importance head is used to generate a visual importance map based on the collaborative space perception features, and the visual importance map serves as the basic importance response map; the semantic context construction subunit is used to generate the input sequence of the large language model; The spatial decoder is used to map the sequence of latent state vectors output by the large language model into a cognitive guidance response map. The sparse selection subunit is used to form a modified importance response map based on the cognitive guidance response map and the visual importance map, and to select a set of cooperative semantic units under spatial diversity. The channel adaptive semantic compression unit includes a signal-to-noise ratio acquisition subunit, a dimension routing subunit, a shared coding subunit, and a multi-level compression subunit. The dimension routing subunit is used to select the target semantic representation dimension according to the signal-to-noise ratio. The shared coding subunit is used to extract the common implicit representation of the cooperative semantic units. The multi-level compression subunit is used to output a compressed cooperative semantic unit set according to the target semantic representation dimension. The sparse perception fusion unit includes a geometric branch, a semantic branch, and a gated fusion subunit. The geometric branch is used to scatter the reconstructed collaborative semantic units onto a spatial grid under a unified spatial reference frame and perform local convolutional diffusion processing. The semantic branch is used to perform cross-attention aggregation using the vehicle's spatial perception features as queries, the semantic features and position codes of the reconstructed collaborative semantic units as keys, and the semantic features of the reconstructed collaborative semantic units as values. The gated fusion subunit is used to generate a spatially adaptive gated map based on the vehicle's spatial perception features and collaborative enhancement features and perform residual fusion.

Images