Semantic communication method and device for vehicle networking collaborative perception, equipment and medium

Abstract

The application discloses a semantic communication method and device for vehicle networking collaborative perception, equipment and medium, and belongs to the technical field of communication. The method comprises the following steps: extracting a BEV feature map from environment observation data through a local perception backbone network, wherein the BEV feature map comprises a plurality of candidate semantic tokens; according to a target load budget, a target number of target semantic tokens are screened from the plurality of candidate semantic tokens through a semantic selector; a target number of target semantic tokens are encoded into a code word tensor through a semantic encoder; the code word tensor is transmitted to a receiving end; the code word tensor is used for decoding through a semantic decoder, and target token features with a first reliability score greater than or equal to a reliability score threshold are screened for spatial fusion; and a collaborative perception model composed of the semantic selector, the semantic encoder and the semantic decoder is used for constraining the consistency of multi-source observation domain semantic prototype distribution based on cross-domain prototype alignment loss in a training process. The application can improve the collaborative perception accuracy.

Description

Technical Field

[0001] This application belongs to the field of communication technology, specifically relating to a semantic communication method, device, equipment, and medium for collaborative perception in vehicle networking. Background Technology

[0002] In autonomous driving V2X (Vehicle to Everything) collaborative perception scenarios, multiple vehicles interact to perceive feature information, compensating for the limitations of single-vehicle perception, such as occlusion and limited detection range. This is a core supporting technology for achieving high-level autonomous driving. Traditional collaborative perception schemes transmit raw sensor data or full BEV features, resulting in high bandwidth consumption and transmission latency, which cannot meet the bandwidth constraints of in-vehicle wireless communication. Existing task-oriented semantic communication technologies effectively optimize the balance between perception accuracy and bandwidth by extracting key semantic tokens for compressed transmission.

[0003] However, in actual vehicle deployment scenarios, semantic communication systems face a dual-coupling problem of distribution offset degradation: on the one hand, adverse weather conditions such as rain, snow, and fog, road scene switching, and differences in different data sets can cause a shift in the distribution of observed features before encoding, resulting in semantic prototype drift; on the other hand, V2X wireless channels suffer from dynamic signal-to-noise ratio fluctuations, Rayleigh fading, quantization noise, and packet loss, leading to distortion and damage of semantic tokens during transmission. This dual-coupling problem of distribution offset degradation ultimately leads to a decrease in perception accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a semantic communication method, apparatus, device, and medium for collaborative perception in vehicle networks, which can solve the problem of decreased perception accuracy caused by the distribution offset degradation problem of dual coupling.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a semantic communication method for collaborative perception in vehicle networks. This method is applied at a sending end and includes: Acquire environmental observation data, and extract BEV feature maps from the environmental observation data through a local sensing backbone network. The BEV feature maps include multiple candidate semantic tokens, and each candidate semantic token corresponds to a spatial location in the BEV feature maps. Based on the target payload budget, a target number of candidate semantic tokens are selected from the multiple candidate semantic tokens by a semantic selector according to the semantic importance score of each candidate semantic token, and these are selected as the target semantic tokens. The target number of target semantic tokens are encoded into codeword tensors by a semantic encoder, and the codeword tensors are transmitted to the receiving end via a wireless link. The codeword tensors transmitted to the receiving end are used to filter target token features with a first reliability score greater than or equal to a reliability score threshold after decoding by a semantic decoder for spatial fusion. The collaborative perception model composed of the semantic selector, the semantic encoder, and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process.

[0006] Secondly, embodiments of this application provide a semantic communication method for collaborative perception in vehicle networks. This method is applied at a receiving end and includes: Receive at least one codeword tensor sent by a transmitter as the received codeword tensor; the codeword tensor sent by the transmitter is obtained by semantically encoding a target number of target semantic tokens by a semantic encoder, and the target number of target semantic tokens are obtained by filtering multiple candidate semantic tokens included in the BEV feature map by a semantic selector, and each candidate semantic token in the BEV feature map corresponds to a spatial location in the BEV feature map. For each received codeword tensor of the sending end, the received codeword tensor is decoded by a semantic decoder to obtain a target number of received token features; wherein, the collaborative perception model composed of the semantic selector, the semantic encoder and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process; A first reliability score is determined for each received token feature, and based on the first reliability score, received token features with a first reliability score greater than or equal to a reliability score threshold are selected from a target number of received token features as target token features. Spatial fusion is performed on the target token features corresponding to each of the aforementioned sending ends to obtain the collaborative perception result.

[0007] Thirdly, embodiments of this application provide a semantic communication device for collaborative perception in vehicle networks, applied at a sending end, comprising: The BEV feature extraction module is used to acquire environmental observation data and extract BEV feature maps from the environmental observation data through the local perception backbone network. The BEV feature map includes multiple candidate semantic tokens, and each candidate semantic token corresponds to a spatial location in the BEV feature map. The token selection module is used to select a target number of candidate semantic tokens from the multiple candidate semantic tokens based on the semantic importance score of each candidate semantic token, according to the target payload budget; The token encoding transmission module is used to encode a target number of target semantic tokens into codeword tensors through a semantic encoder, and transmit the codeword tensors to the receiving end through a wireless link. The codeword tensors transmitted to the receiving end are used to filter target token features with a first reliability score greater than or equal to a reliability score threshold after decoding by a semantic decoder for spatial fusion. The collaborative perception model composed of the semantic selector, the semantic encoder, and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process.

[0008] Fourthly, embodiments of this application provide a semantic communication device for collaborative perception in vehicle networking, applied at a receiving end, including: The receiving module is used to receive at least one codeword tensor sent by a sending end as a received codeword tensor; the codeword tensor sent by the sending end is obtained by semantically encoding a target number of target semantic tokens by a semantic encoder, and the target number of target semantic tokens are obtained by filtering multiple candidate semantic tokens included in the BEV feature map by a semantic selector, and each candidate semantic token in the BEV feature map corresponds to a spatial location in the BEV feature map. The decoding module is used to decode the received codeword tensor of each of the sending ends using a semantic decoder to obtain a target number of received token features; wherein, the collaborative perception model composed of the semantic selector, the semantic encoder and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process. The token filtering module is used to determine the first reliability score of each received token feature, and based on the first reliability score, to filter the received token features whose first reliability score is greater than or equal to the reliability score threshold from the target number of received token features as target token features. The spatial fusion module is used to spatially fuse the target token features corresponding to each of the sending ends to obtain the collaborative perception result.

[0009] Fifthly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first or second aspect.

[0010] In a sixth aspect, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0011] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the methods described in the first or second aspect.

[0012] In this embodiment, a BEV feature map is extracted from environmental observation data through a local sensing backbone network. The BEV feature map includes multiple candidate semantic tokens. Based on the target payload budget, a semantic selector selects a target number of candidate semantic tokens from the multiple candidate semantic tokens according to the semantic importance score of each candidate semantic token. These target semantic tokens are then encoded into codeword tensors by a semantic encoder. The codeword tensors are transmitted to the receiving end via a wireless link. The codeword tensors transmitted to the receiving end can be decoded by a semantic decoder to select target token features with a first reliability score greater than or equal to a reliability score threshold for spatial fusion. The collaborative sensing model composed of the semantic selector, semantic encoder, and semantic decoder constrains the consistency of the semantic prototype distribution of multi-source observation domains based on the cross-domain prototype alignment loss function during training, suppressing semantic drift between different observation domains. Moreover, the semantic decoder at the receiving end can select token features with higher first reliability scores for spatial fusion after decoding, which can avoid using distorted or damaged token features in spatial fusion. This solves the distribution offset degradation problem caused by the dual coupling of different observation domains and channel transmission process, and can improve the accuracy of collaborative sensing. Attached Figure Description

[0013] Figure 1 This is a flowchart of a semantic communication method for collaborative perception in vehicle networking provided in an embodiment of this application; Figure 2 This is a flowchart of the training process of the collaborative perception model in the embodiments of this application; Figure 3 This is a flowchart of a semantic communication method for collaborative perception in vehicle networking provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the fine-tuning of the parameters of the lightweight adapter in an embodiment of this application; Figure 5 This is a schematic diagram of the receiver reliability gating lightweight adaptive process in the embodiments of this application; Figure 6 This is a schematic diagram of the semantic communication system structure for collaborative perception in the Internet of Vehicles in this application embodiment; Figure 7 This is a schematic diagram of the structure of a semantic communication device for collaborative perception in the Internet of Vehicles provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a semantic communication device for collaborative perception in the Internet of Vehicles provided in an embodiment of this application; Figure 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0016] In existing technologies, to address the observation threshold drift problem, domain generalization methods can be used to optimize the observation domain adaptability, ignoring the impact of channel distortion. During testing, adaptive methods are used to assume clean input features. However, this approach cannot distinguish between semantic tokens with domain offset distortion and those with physical channel damage. Unified adaptation updates can be misled by low-quality tokens, leading to decreased perception accuracy. Furthermore, existing adaptive schemes suffer from large parameter update volumes, high vehicle-to-vehicle signaling overhead, and difficulties in edge deployment, failing to meet the real-time, lightweight, and highly robust requirements of practical autonomous driving scenarios. This application provides the following technical solution to address these technical problems.

[0017] The semantic communication method, apparatus, device, and medium for collaborative perception in vehicle networking provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0018] Figure 1 This is a flowchart illustrating a semantic communication method for collaborative perception in vehicle-to-everything (V2X) networks, provided in an embodiment of this application. This semantic communication method is applicable to multi-vehicle collaborative perception in autonomous vehicles, vehicle-to-infrastructure (V2I) environmental perception, lightweight in-vehicle semantic transmission, and high-precision intelligent perception scenarios under complex dynamic wireless channels. The semantic communication method for collaborative perception in V2X networks is applied at the transmitting end, which is the cooperating node in the collaborative perception process, and can be a vehicle, roadside unit, etc. Figure 1 As shown, the semantic communication method for collaborative perception in the Internet of Vehicles may include steps 110 to 130.

[0019] Step 110: Obtain environmental observation data and extract BEV feature maps from the environmental observation data through the local sensing backbone network. The BEV feature maps include multiple candidate semantic tokens, and each candidate semantic token corresponds to a spatial location in the BEV feature maps.

[0020] Environmental observation data includes LiDAR (Light Detection and Ranging) point clouds and / or camera images collected locally by collaborating nodes.

[0021] BEV (Bird's Eye View) is an algorithmic technology that transforms multi-sensor data into a unified bird's-eye view for perception and decision-making.

[0022] After acquiring environmental observation data based on sensors, the collaborative nodes process the data through a local perception backbone network, transforming it into a bird's-eye view and extracting features to obtain a BEV feature map. It should be noted that for environmental observation data of different modalities, the corresponding local perception backbone network can be used for BEV feature map extraction. For example, for point cloud data, the local perception backbone network corresponding to the point cloud data can be used for BEV feature map extraction; for image data, the local perception backbone network corresponding to the image data can be used. For instance, the local perception backbone network corresponding to point cloud data could be the PointPillar backbone network.

[0023] The BEV feature map is a dense two-dimensional grid, where each grid position corresponds to a C-dimensional feature vector, which can be considered as a candidate semantic token. Assume the BEV feature map of the i-th cooperative node is as follows:

[0024] in, This represents the environmental observation data obtained by the i-th collaborating node. This indicates that the i-th cooperating node obtains environmental observation data through the local sensing backbone network. BEV feature map extraction is performed for the i-th cooperative node. For size A two-dimensional grid, where each grid position corresponds to a C-dimensional feature vector.

[0025] The candidate semantic token corresponding to spatial location j is represented as:

[0026] in, This represents the spatial location of the BEV feature map determined by the i-th collaborating node ( Candidate semantic tokens.

[0027] Step 120: Based on the target payload budget, a target number of candidate semantic tokens are selected from the multiple candidate semantic tokens by a semantic selector according to the semantic importance score of each candidate semantic token, and these selected tokens are used as the target semantic tokens.

[0028] The target payload budget can provide the amount of data that can be transmitted. Based on this target payload budget and the amount of data for each candidate semantic token, the number of candidate semantic tokens that can be transmitted can be determined as the target number.

[0029] The semantic importance score corresponding to the candidate semantic token represents the degree of importance of the candidate semantic token to the current detection task. The current detection task can specify the types of targets to be detected during collaborative perception, such as pedestrians, lane lines, obstacles, etc.

[0030] A semantic selector determines the semantic importance score of each candidate semantic token, and selects a target number of candidate semantic tokens with higher semantic importance scores from all candidate semantic tokens as the target semantic tokens to be transmitted. The semantic selector, as part of the collaborative perception model, is pre-trained.

[0031] In some embodiments of this application, the step of selecting a target number of candidate semantic tokens from the plurality of candidate semantic tokens based on the semantic importance score of each candidate semantic token according to the target payload budget, and selecting a target number of candidate semantic tokens as the target semantic token according to the target payload budget and in descending order of the semantic importance score.

[0032] Each candidate semantic token is processed by a semantic selector, and a semantic importance score for each candidate semantic token is output. Based on the target payload budget, the candidate semantic tokens at the U positions (representing the number of targets) with the highest semantic importance scores are selected:

[0033]

[0034] in, This represents the position index of the U candidate semantic tokens selected by the i-th collaborating node. Let HW represent the semantic importance score of the candidate semantic token at the j-th position determined in the i-th collaborating node, and let HW represent the total number of candidate semantic tokens included in the BEV feature map. This represents the U candidate semantic tokens determined by the semantic selector for the i-th collaborating node. This represents the set of U candidate semantic tokens selected.

[0035] By selecting a number of candidate semantic tokens with high semantic importance scores based on the semantic importance scores of the candidate semantic tokens, and using these as the target semantic tokens to be transmitted, the semantic tokens with the highest relevance to the current detection task can be transmitted, thereby improving the accuracy of collaborative perception.

[0036] Step 130: The target number of target semantic tokens are encoded into codeword tensors by a semantic encoder, and the codeword tensors are transmitted to the receiving end via a wireless link. The codeword tensors transmitted to the receiving end are used to filter target token features with a first reliability score greater than or equal to a reliability score threshold after decoding by a semantic decoder for spatial fusion. The collaborative perception model composed of the semantic selector, the semantic encoder, and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process.

[0037] The semantic encoder, as part of the collaborative sensing model, is pre-trained. The collaborative sensing model may also include a semantic decoder applied at the receiver. During training, the collaborative sensing model uses a first training dataset that includes multiple source observation domains and various channel conditions. During training, cross-domain prototype alignment loss is used to constrain the consistency of the semantic prototype distribution of the multiple source observation domains, suppressing semantic drift in the observation domains. The observation domains represent different observation conditions such as weather and scenes, and may also include differences in dataset distribution.

[0038] A semantic encoder semantically encodes a target number of target semantic tokens and represents them as tensors, resulting in codeword tensors. These codeword tensors, as compact codeword tensors, are transmitted to the receiving end via a V2X wireless link. The receiving end performs semantic decoding on the received codeword tensors and selects target token features from the decoded target number of token features that have a first reliability score greater than or equal to a reliability score threshold. The selected target token features are then used for spatial fusion. The receiving end acts as the target node in collaborative sensing, responsible for aggregating information from various cooperating nodes, performing fusion, and conducting detection.

[0039] The U target semantic tokens obtained after filtering constitute The feature tensor, the semantic encoder maps each target semantic token to a d-dimensional continuous channel codeword, and the codeword tensor corresponding to U target semantic tokens is represented as:

[0040] in, Let represent the codeword tensor determined by the i-th collaborating node. Represents the network parameters of the semantic encoder. This represents the environmental observation data obtained by the i-th collaborating node. This represents the parameters of the local awareness backbone network in the i-th collaborating node. The network parameters represent the semantic selector.

[0041] The semantic communication method for collaborative perception in vehicle-to-everything (V2X) provided in this application extracts BEV feature maps from environmental observation data through a local perception backbone network. The BEV feature maps include multiple candidate semantic tokens. Based on the target payload budget, a semantic selector selects a target number of candidate semantic tokens from the multiple candidate semantic tokens according to their semantic importance scores. These target semantic tokens are then encoded into codeword tensors by a semantic encoder. The codeword tensors are transmitted to the receiving end via a wireless link. The codeword tensors transmitted to the receiving end can be decoded by a semantic decoder, and target token features with a first reliability score greater than or equal to a reliability score threshold are selected for spatial fusion. The collaborative perception model composed of the semantic selector, semantic encoder, and semantic decoder constrains the consistency of semantic prototype distribution across multiple observation domains during training based on a cross-domain prototype alignment loss function, suppressing semantic drift across different observation domains. Furthermore, the semantic decoder at the receiving end can select token features with higher first reliability scores for spatial fusion after decoding, avoiding the use of distorted or damaged token features in spatial fusion. This solves the distribution offset degradation problem caused by dual coupling between different observation domains and channel transmission processes, thereby improving the accuracy of collaborative perception.

[0042] Figure 2 This is a flowchart of the training process of the collaborative perception model in the embodiments of this application, such as... Figure 2 As shown, the training process of the collaborative perception model includes the following steps: Step 210: Obtain the first training dataset, which includes multiple first training samples. Each first training sample includes sample observation data and 3D annotations from different cooperative nodes. Different first training samples have multi-source observation domains and multiple channel conditions.

[0043] The first training dataset is a collection of multiple first training samples. Each first training sample includes sample observation data from one or more collaborating nodes and 3D annotations. The 3D annotations are generated after spatial fusion of the data from one or more collaborating nodes. All first training samples in the first training dataset include multi-source observation domains and multiple channel conditions, meaning that different first training samples can have different observation domains and different channel conditions; the observation domains of sample observation data from different collaborating nodes within the same first training sample can be the same or different; and the channel conditions of sample observation data from different collaborating nodes within the same first training sample can be the same or different.

[0044] Step 220: Extract sample BEV feature maps from sample observation data through the local perception backbone network. The sample BEV feature maps include multiple sample candidate semantic tokens.

[0045] During the training of the collaborative sensing model, the parameters of the local sensing backbone network are frozen.

[0046] The sample observation data is input into the local perception backbone network, which converts the sample observation data into a bird's-eye view and extracts features to obtain the sample BEV feature map. Each spatial location in the sample BEV feature map is a sample candidate semantic token.

[0047] Step 230: Based on the target payload budget, select a target number of sample candidate semantic tokens from the multiple sample candidate semantic tokens using an initialized semantic selector, and use them as the sample target semantic tokens.

[0048] During the training of the collaborative perception model, the parameters of the semantic selector, semantic encoder, and semantic decoder are updated.

[0049] After extracting sample BEV feature maps through the local perception backbone network, the initial semantic selector filters all candidate semantic tokens included in the BEV feature maps according to the target payload budget, and selects a number of target sample candidate semantic tokens with high reliability scores as sample target semantic tokens.

[0050] Step 240: Encode the target number of sample target semantic tokens into sample codeword tensors using the initialized semantic encoder.

[0051] Each sample target semantic token is encoded into a continuous channel codeword of the target dimension by an initialized semantic encoder. The continuous channel codewords corresponding to the target number of sample target semantic tokens are used as the sample codeword tensor.

[0052] Step 250: Based on the channel conditions corresponding to the sample observation data, determine the sample received codeword tensor corresponding to the sample codeword tensor.

[0053] Channel conditions may include additive white Gaussian noise (AWGN) channels or Rayleigh fading channels.

[0054] For each cooperating node, based on the channel conditions corresponding to the sample observation data in that cooperating node, the distortion caused by the channel conditions is superimposed on the sample codeword tensor corresponding to that cooperating node to obtain the sample received codeword tensor corresponding to that cooperating node.

[0055] Step 260: Decode the sample received codeword tensor using the initialized semantic decoder to obtain the target number of sample received token features.

[0056] The semantic decoder can include a semantic decoder backbone network, a lightweight adapter, and a reliability scoring unit. During training, the parameters of the semantic decoder backbone network, the lightweight adapter, and the reliability scoring unit are all updated. In the subsequent online deployment phase, the parameters of the lightweight adapter can be fine-tuned.

[0057] For each collaborative node's transmitted sample received codeword tensor, the sample received codeword tensor is decoded using the initialized semantic decoder backbone network and the initialized lightweight adapter to obtain the target number of sample received token features.

[0058] Step 270: For each collaborative node, determine the reliability score of each sample received token feature, and based on the reliability score, select sample received token features with a reliability score greater than or equal to the reliability score threshold from the target number of sample received token features as sample target token features.

[0059] For each collaborative node, the reliability score of each sample receiving token feature corresponding to the collaborative node is determined by the initialized reliability scoring unit. Sample receiving token features with a reliability score greater than or equal to the reliability score threshold are selected from the target number of sample receiving token features corresponding to the collaborative node and used as sample target token features.

[0060] When determining the reliability score of each sample's received token feature using the reliability scoring unit, the semantic importance score, channel confidence score, and prediction uncertainty score of the sample's received token feature are determined by the reliability scoring unit. Based on trainable weights, the semantic importance score, channel confidence score, and prediction uncertainty score are weighted and fused to obtain the reliability score of the sample's received token feature. Among these, the semantic importance score... The semantic importance score is generated by the semantic selector at the sending end when filtering target semantic tokens from a certain number of sample targets. It characterizes the token's contribution to the detection task; the semantic importance score can be provided through the token index or metadata. Channel confidence score. At the receiving end, this is formed based on the amplitude of the received signal of the token, used to characterize the reliability of physical transmission; and to predict uncertainty. The reliability scoring unit outputs the results based on the target token features of the sample, which is used to characterize the stability of the prediction.

[0061] Step 280: Spatial fusion is performed on the sample target token features corresponding to each collaborative node through the spatial fusion module, and the fusion result is subjected to three-dimensional detection by the three-dimensional detector to obtain the sample collaborative perception result.

[0062] Step 290: Determine the joint loss function value based on the sample collaborative perception results, the 3D annotation, the sample received token features, and the reliability score; and update the parameters of the initialized collaborative perception model based on the joint loss function value to obtain the pre-trained collaborative perception model.

[0063] The initialized collaborative perception model includes the initialized semantic selector, the initialized semantic encoder, and the initialized semantic decoder.

[0064] The joint loss function includes a detection loss function, a cross-domain prototype alignment loss function, a cross-channel gradient consistency loss function, a codeword compactness loss function, and an uncertainty regression loss function. The detection loss function is used to quantify the difference between the sample collaborative perception result and the 3D annotation. The cross-domain prototype alignment loss function is used to constrain the consistency of the semantic prototype distribution in the multi-source observation domain and suppress semantic drift in the observation domain. The cross-channel gradient consistency loss function is used to regularize the gradient statistical features of the semantic decoder. The codeword compactness loss function is used to reduce encoding energy. The uncertainty regression loss function is used to train the token uncertainty evaluation capability.

[0065] Detection loss function The standard for the three-dimensional detector used is supervised detection loss.

[0066] The cross-domain prototype alignment loss function is calculated as follows: the decoded feature semantic prototypes of each source observation domain are calculated separately, the mean of the semantic prototypes of the multi-source observation domains is obtained, and cross-domain semantic alignment is achieved by minimizing the Euclidean distance between the semantic prototypes of each source observation domain and the mean prototype, thereby eliminating the differences in the distribution of observation domain features caused by different weather, scenes, and datasets.

[0067] For the source observation domain m and semantic category c, the decoded feature semantic prototype and cross-domain average semantic prototype are represented as follows:

[0068] in, This represents the semantic prototype of the decoded features for the source observation domain m and the semantic category c. This represents the cross-domain average semantic prototype. The sample received token feature represents the target semantic token u of the sample. M represents the soft assignment probability of the target semantic token u to the semantic category c, and M represents the number of source observation domains. Represents the numerical stability constant. Represents the set of target semantic tokens.

[0069] The semantic prototype of a decoded feature refers to the representative center vector of a specific semantic category in the decoded feature space under a specific observation domain. The cross-domain average semantic prototype refers to the mean of the semantic prototypes of a specific semantic category across all observation domains.

[0070] Based on the decoded feature semantic prototype and the cross-domain average semantic prototype, the cross-domain prototype alignment loss function can be expressed as:

[0071] in, Indicates the number of semantic categories.

[0072] For the source observation domain m and semantic category c, use the soft-assignment probability that the target semantic token of each sample belongs to semantic category c. Decode token features of the sample Weighted average is used to obtain the semantic prototype of the decoded features. The average semantic prototype of the decoded features from each source observation domain is taken to obtain the cross-domain average semantic prototype. Cross-domain prototype alignment loss function constrains the decoding of feature semantic prototypes. Approximate cross-domain average semantic prototype This allows similar semantic features to cluster toward a common semantic center in different observation domains, thereby suppressing semantic drift caused by changes in weather, scene, and dataset.

[0073] The cross-channel gradient consistency loss function is based on Fisher's principle. It extracts the diagonal features of the decoder gradient covariance under different training channel conditions. By constraining the consistency of the statistical features of multi-channel gradients, it avoids the model from overfitting to a single channel and improves the adaptability to unknown channels such as Rayleigh fading.

[0074] For the k-th channel condition, the diagonal statistic of the semantic decoder gradient covariance is expressed as:

[0075] in, This represents the gradient covariance of the semantic decoder, which is the covariance matrix formed by the gradients of the semantic decoder parameters.

[0076] During training, the sample codeword tensor x generates the corresponding sample received codeword tensor according to the k-th channel condition. The detection loss under that channel condition is then obtained through the semantic decoder and the 3D detector. Then adjust the semantic decoder parameters. Calculate the gradient for each sample A set of samples corresponding to the k-th channel condition The covariance matrix of these gradients is statistically analyzed. Then, by taking the diagonal elements of the covariance matrix, the gradient statistic is obtained. .

[0077] Different channel conditions k will cause different reception distortions and detection losses, thus resulting in different gradient statistics. Cross-channel gradient consistency loss constrains the gradient statistics. The gradient consistency loss function approximates the cross-channel average of the gradient statistics, thereby reducing overfitting of the decoder to specific training channels and improving its generalization ability to unseen channel conditions. The cross-channel gradient consistency loss function is expressed as:

[0078] Where K represents the number of channel conditions during training.

[0079] The codeword compactness loss function reduces the overall energy of semantic coding by constraining the sum of squares of the F-norm of the semantic codeword tensor, reduces the interference of wireless channel fluctuations on coding features, and simultaneously reduces transmission power consumption and channel resource occupation.

[0080] The codeword compactness loss function is expressed as:

[0081] Where U represents the number of tokens to be retained, i.e., the target number, and d represents the codeword dimension of the target semantic token for each sample. Let N represent the codeword tensor transmitted by the i-th cooperating node, and let N represent the number of cooperating nodes.

[0082] The uncertainty regression loss function is calculated using the symmetric KL divergence of the prediction results from two independent channel samples. The model is pre-trained using the uncertainty regression loss function, enabling accurate assessment of token distortion in unlabeled scenarios during the deployment phase.

[0083] The posterior difference between the same sample target semantic token after two independent training channel samplings is as follows:

[0084] in, and Let represent the semantic category posterior probability distributions of the target semantic token u under two independently sampled training channel conditions (or channel implementations), obtained after decoding and task prediction at the receiver. The two independently sampled training channel conditions can be different channel implementations of the same channel type, or different channel types. The symmetric KL divergence of the semantic category posterior probability distributions under the two independently sampled training channel conditions is used to characterize the degree of prediction instability of the target semantic token u to channel perturbations, and serves as a monitoring signal for prediction uncertainty.

[0085] The regression loss function for uncertainty is expressed as:

[0086] Where U represents the number of tokens to be reserved, i.e., the target number. This represents the uncertain prediction value of the target semantic token u.

[0087] The joint loss function is expressed as:

[0088] in, , , , These represent the weights of the cross-domain prototype alignment loss function, the cross-channel gradient consistency loss function, the codeword compactness loss function, and the uncertainty regression loss function, respectively.

[0089] During the training of the collaborative sensing model, the constraints of the cross-domain prototype alignment loss function and the cross-channel gradient consistency loss function are used to achieve dual offset coupling adaptation, which greatly improves the robustness of complex scenarios. That is, by integrating cross-domain generalization and channel anti-interference optimization, the coupling degradation problem of weather and dataset offset in the observation domain with wireless channel fading and noise distortion is solved at the same time. It has been verified that in scenarios such as Rayleigh fading and unknown channels, severe weather, and cross-dataset deployment, the AP@0.7 index of the weather offset task is improved by 7.2 compared with the traditional domain generalization method, and the cross-dataset task is improved by 5.5, with a significant improvement in sensing accuracy.

[0090] By employing an uncertainty regression loss function, a reliability-gated adaptive mechanism is provided to address the issue of invalid token interference. This application's embodiments abandon the traditional indiscriminate adaptive update mode and instead use a reliability score fused from three dimensions (semantic importance score, channel confidence score, and prediction uncertainty score) to select high-quality semantic tokens. Gradient updates are performed only using valid tokens, avoiding corrupted or mismatched tokens from misleading model optimization. This effectively solves the adaptive model drift problem during testing and significantly improves adaptation stability in complex dynamic scenarios.

[0091] Figure 3 This is a flowchart illustrating a semantic communication method for collaborative perception in vehicle-to-everything (V2X) networks, provided in an embodiment of this application. This semantic communication method is applicable to multi-vehicle collaborative perception in autonomous vehicles, vehicle-to-infrastructure (V2I) environmental perception, lightweight in-vehicle semantic transmission, and high-precision intelligent perception scenarios under complex dynamic wireless channels. The semantic communication method for collaborative perception in V2X networks is applied at the receiving end, which is the target node responsible for aggregating information from various cooperative nodes, performing fusion, and detection in collaborative perception. This node can be a vehicle, roadside unit, etc. Figure 3 As shown, the semantic communication method for collaborative perception in the Internet of Vehicles may include steps 310 to 340.

[0092] Step 310: Receive at least one codeword tensor sent by a sending end as the received codeword tensor.

[0093] The codeword tensor sent by the sending end is obtained by semantically encoding a target number of target semantic tokens through a semantic encoder. The target number of target semantic tokens are obtained by filtering multiple candidate semantic tokens included in the BEV feature map through a semantic selector. Each candidate semantic token in the BEV feature map corresponds to a spatial location in the BEV feature map.

[0094] The BEV feature map is extracted from environmental observation data by the transmitting end through the local sensing backbone network. The transmitting end is a collaborative node in the collaborative sensing system. The environmental observation data includes LiDAR (Light Detection and Ranging) point clouds and / or camera images collected locally by the collaborative nodes.

[0095] The codeword tensor sent directly by the sending end is received by the receiving end only after being transmitted through the wireless link. Therefore, the received codeword tensor is somewhat distorted compared to the codeword tensor sent directly by the sending end.

[0096] Step 320: For each received codeword tensor of the sending end, the received codeword tensor is decoded by a semantic decoder to obtain the target number of received token features.

[0097] The collaborative perception model, composed of the semantic selector, the semantic encoder, and the semantic decoder, constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process.

[0098] The data sent by each sender is processed separately. For the received codeword tensor of the currently processed sender, it is decoded by a semantic decoder to obtain the target number of received token features.

[0099] The collaborative sensing model, including a semantic selector, semantic encoder, and semantic decoder, is pre-trained. During training, the collaborative sensing model uses a first training dataset that includes multiple observation domains and various channel conditions. During training, cross-domain prototype alignment loss is used to constrain the consistency of the semantic prototype distribution across the multiple observation domains, suppressing semantic drift. The observation domain represents different observation conditions such as weather and scenes, and may also include differences in dataset distribution.

[0100] Step 330: Determine the first reliability score for each received token feature, and based on the first reliability score, select received token features from the target number of received token features whose first reliability score is greater than or equal to the reliability score threshold as target token features.

[0101] For each received token feature, a first reliability score is calculated. The reliability score characterizes the reliability level of the received token feature; the higher the reliability score, the lower the degree to which the received token feature has been compromised.

[0102] In some embodiments of this application, determining the first reliability score for each received token feature includes: For each received token feature, a first semantic importance score is obtained from the received codeword tensor through a reliability scoring unit, a first channel confidence is determined based on the received signal amplitude of the received codeword tensor, and a first prediction uncertainty score is determined based on the received token feature. The first prediction uncertainty score is used to characterize the stability of the received token feature. The first semantic importance score, the first channel confidence score, and the first prediction uncertainty score are weighted and fused to obtain the first reliability score.

[0103] Among them, the first semantic importance score The first semantic importance score is generated by the semantic selector at the sending end when filtering a target number of target semantic tokens. It is used to characterize the contribution of the token to the detection task. The first semantic importance score can be provided through the token index or metadata; the first channel confidence score. At the receiving end, a score is formed based on the amplitude of the received signal of the token, used to characterize the reliability of physical transmission; the first prediction uncertainty score. The reliability scoring unit outputs the value based on the characteristics of the received token, which is used to characterize the degree of predictive stability.

[0104] The reliability scoring unit includes a lightweight uncertainty prediction head and reliability fusion weights. The reliability scoring unit, as part of the collaborative sensing model, is jointly trained with the collaborative sensing model during the offline pre-training phase. The reliability scoring unit employs a three-scalar sigmoid weighted fusion. During the online deployment phase, the uncertainty prediction head and weights... All settings remain frozen; only lightweight adapter parameters are updated. .

[0105] For each received token feature, a first semantic importance score is obtained from the metadata or token index of the received codeword tensor based on the uncertainty prediction header by a reliability scoring unit. A first channel confidence score is determined based on the received signal amplitude of the received codeword tensor, and a first prediction uncertainty score is determined based on the received token feature. For example, the ratio of the received signal amplitude to the target signal amplitude can be used as the first channel confidence score.

[0106] The reliability scoring unit performs a sigmoid-weighted fusion of the first semantic importance score, the first channel confidence score, and the first prediction uncertainty score based on the reliability fusion weights obtained during the training phase, to obtain the first reliability score. The first reliability score for the u-th received token feature can be expressed as:

[0107] in, This indicates the first semantic importance given by the sender's semantic selector. This represents the first channel confidence level formed by the receiving end based on the received signal. Indicates uncertainty in forecasting. These are the fusion weights obtained through offline joint training. Higher first semantic importance and first channel confidence result in a higher first reliability score; higher first prediction uncertainty scores result in a lower first reliability score. Used during the online deployment phase. By applying soft weights to the entropy loss and target prototype calculations, tokens with higher reliability scores contribute more to the parameter updates of the lightweight adapter.

[0108] Step 340: Spatial fusion is performed on the target token features corresponding to each of the sending ends to obtain the collaborative perception result.

[0109] After determining the target token features for each sender, the spatial fusion module performs semantic feature spatial fusion of the target token features corresponding to each sender across multiple collaborative nodes, and the fusion result is then subjected to 3D target detection by a 3D detector to obtain the collaborative perception result.

[0110] The semantic communication method for cooperative perception in vehicle-to-everything (V2X) provided in this application involves decoding the received codeword tensor of each transmitter using a semantic decoder to obtain a target number of received token features. A first reliability score is determined for each received token feature, and based on this first reliability score, received token features with a first reliability score greater than or equal to a reliability score threshold are selected from the target number of received token features as target token features. Spatial fusion is then performed on the target token features corresponding to each transmitter to obtain the cooperative perception result. The cooperative perception model, composed of a semantic selector, semantic encoder, and semantic decoder, constrains the consistency of the semantic prototype distribution of multi-source observation domains during training based on a cross-domain prototype alignment loss function, suppressing semantic drift across different observation domains. Furthermore, after decoding, the receiver can select token features with higher first reliability scores for spatial fusion, avoiding the use of distorted or damaged token features in spatial fusion. This solves the distribution offset degradation problem caused by dual coupling due to different observation domains and channel transmission processes, thereby improving the accuracy of cooperative perception.

[0111] In some embodiments of this application, the semantic decoder includes a semantic decoder backbone network, a lightweight adapter, and a reliability scoring unit. During offline training, the semantic decoder backbone network, lightweight adapter, and reliability scoring unit are trained together with the semantic selector and semantic encoder in the collaborative perception model. The lightweight adapter trained during offline training serves as a pre-trained lightweight adapter. After deploying the pre-trained semantic decoder backbone network, pre-trained lightweight adapter, and reliability scoring unit at the receiving end, before using the semantic decoder to decode a target number of received codeword tensors, the parameters of the lightweight adapter need to be fine-tuned. That is, during the online deployment phase, the parameters of the semantic selector, semantic encoder, semantic decoder backbone network, and reliability scoring unit are frozen, while the parameters of the lightweight adapter are updated.

[0112] Figure 4 This is a flowchart illustrating the fine-tuning of the parameters of the lightweight adapter in this embodiment of the application, such as... Figure 4 As shown, the process of fine-tuning the parameters of the lightweight adapter includes: Step 410: Obtain the second training dataset, which includes multiple second training samples, each of which includes the target number of distorted semantic tokens transmitted by multiple collaborative nodes.

[0113] Distorted semantic tokens are semantic tokens transmitted over a wireless link.

[0114] Step 420: For each cooperative node, the target number of distorted semantic tokens are decoded through the semantic decoder backbone network to obtain the target number of initial token features for samples. The target number of initial token features for samples are then processed through a pre-trained lightweight adapter to obtain the target number of target token features for samples.

[0115] The lightweight adapter is a two-layer bottleneck MLP (Multilayer Perceptron) structure with residual connections, implementing the mapping of feature dimensions C→C / 4→C. It only serves as a post-adaptation module at the receiver and does not participate in the encoding process at the transmitter, thus eliminating the need for multi-vehicle parameter synchronization. C represents the feature dimension of the candidate semantic token.

[0116] Step 430: For each of the sample target token features, the reliability scoring unit determines the second semantic importance score, the second channel confidence score, and the second prediction uncertainty score corresponding to the sample target token feature, and performs a weighted fusion of the second semantic importance score, the second channel confidence score, and the second prediction uncertainty score to obtain the second reliability score of the sample target token feature.

[0117] The second semantic importance score is obtained from the distorted semantic token corresponding to the sample target token feature by the reliability scoring unit, and the second channel confidence score and the second prediction uncertainty score are determined. The second semantic importance score, the second channel confidence score and the second prediction uncertainty score are fused by Sigmoid weighting to obtain the second reliability score of the sample target token feature.

[0118] The determination methods for the second semantic importance score, the second channel confidence score, and the second prediction uncertainty score are the same as those for the first semantic importance score, the first channel confidence score, and the first prediction uncertainty score.

[0119] Step 440: Determine the reliability gating entropy loss function value based on the second reliability score, determine the soft prototype anchoring loss function value based on the second reliability score and the target token feature, and determine the parameter regularization loss function value based on the pre-training parameters of the pre-trained lightweight adapter and the current parameters of the lightweight adapter.

[0120] Obtain the posterior probabilities of the target token features for each semantic category, and determine the token posterior entropy based on the posterior probabilities of the target token features for each semantic category. The token posterior entropy is defined as:

[0121] in, Let represent the posterior probability of the u-th target token feature with respect to the c-th semantic category. Indicates the number of semantic categories. This represents the numerical stability constant.

[0122] The reliability-gated entropy loss function value is determined based on the token's posterior entropy. The reliability-gated entropy loss function is expressed as:

[0123] Where U represents the number of target token features in the sample. This represents the second reliability score for the characteristics of the target token in the u-th sample. This represents the posterior probability of the category.

[0124] By using a reliability-gated entropy loss function, the full entropy loss gradient is retained for high-reliability tokens, while the gradient contribution of low-reliability tokens is weakened or even masked. This can prevent invalid tokens caused by channel distortion and domain offset mismatch from misleading the model's adaptive updates.

[0125] The soft prototype anchoring loss function is used to calculate the real-time semantic prototype in online deployment scenarios, constrain the deviation between the real-time semantic prototype and the source domain benchmark prototype in offline pre-training (i.e., the collaborative perception model training process), avoid excessive model drift during the deployment stage, and ensure the global consistency of semantic features.

[0126] The target domain soft prototype and the soft prototype anchoring loss are expressed as follows:

[0127] in, This represents a soft prototype of the target domain. This represents the soft prototype anchoring loss function, where U represents the number of target token features in the sample. This represents the second reliability score for the characteristics of the target token in the u-th sample. This indicates the target token feature of the sample. Let represent the posterior probability that the target token feature of the u-th sample belongs to the c-th semantic category. Indicates the number of semantic categories. This represents the average semantic prototype of the source domain, which is stored offline.

[0128] Target domain soft prototypes refer to category center features obtained by soft weighting based on continuous semantic category posterior probabilities and reliability scores when the target observation domain lacks true category labels. For semantic category c, the target token feature of the u-th sample is... As a weight, it participates in the feature mean, where The second reliability score for the target token feature of the u-th sample. This represents the posterior probability that the target token feature of this sample belongs to the c-th semantic category; therefore, tokens with high reliability and high category confidence contribute more. The soft prototype anchoring loss function will target the soft prototype of the target domain. Constraints on the average semantic prototype of the source domain obtained through offline training To maintain semantic structure stability and suppress model drift during the online adaptation process, the model is located nearby.

[0129] Step 450: Determine the adaptive total loss function value based on the reliability gated entropy loss function value, the soft prototype anchoring loss function value, and the parameter regularization loss function value, and iteratively update the parameters of the pre-trained lightweight adapter based on the adaptive total loss function value.

[0130] The adaptive total loss function is expressed as:

[0131] in, This indicates the current parameters of the lightweight adapter. These are the initial parameters for the adapter in the offline model, i.e., the parameters of the pre-trained lightweight adapter. and Indicates weight, This represents the parameter regularization loss function.

[0132] The parameters of the pre-trained lightweight adapter are updated based on the adaptive total loss function value. Then, steps 420 to 450 are executed iteratively. Iteration stops when the target number of iterations is reached, resulting in the fine-tuned lightweight adapter. After a few iterations, the online adaptation of the lightweight adapter can be completed.

[0133] Figure 5 This is a schematic diagram of the receiver reliability gating lightweight adaptive process in the embodiments of this application, as shown below. Figure 5 As shown, in the online deployment phase, offline pre-trained parameters, adapter initial parameters, and source domain prototypes are loaded; the distorted codeword tensors of the target batch are received and decoded into received token features; the class posterior and reliability score for each received token feature are calculated; reliability-gated entropy loss and soft prototype anchoring loss are constructed from the reliability scores; parameter regularization terms are added to the online total loss; one to two steps of Adam or SGD (Stochastic Gradient Descent) updates are performed on the lightweight adapter; and the updated lightweight adapter is used to complete the fusion and detection of the current batch. This process does not use source domain data, target labels, or vehicle-to-vehicle model synchronization; the semantic encoder, semantic decoder backbone network, spatial fusion module, and 3D detector are all kept frozen.

[0134] During the online deployment and adaptation phase, a single data batch can perform 1-2 steps of gradient iterative updates, with an additional latency of no more than 9ms for single-frame perception, balancing adaptive optimization effects with the real-time deployment requirements of in-vehicle systems.

[0135] During the online deployment phase, the backbone network parameters of the semantic encoder and decoder are frozen throughout the process, with only the lightweight adapter built into the receiver being updated, accounting for 0.21% of the parameter updates. First, based on three dimensions—semantic importance, channel confidence, and prediction uncertainty—a reliability score is calculated for each semantic token to accurately distinguish between high-quality, usable tokens and damaged or mismatched tokens. Then, reliability-gated entropy loss is minimized, and model adaptive optimization is achieved using high-reliability tokens to suppress gradient interference from low-quality tokens. Simultaneously, soft prototype anchoring loss is introduced to constrain the deviation between the online-adapted semantic prototype and the pre-trained source domain prototype, preventing excessive model drift. Finally, parameter regularization constraints are added to ensure smooth adapter parameter updates. After adaptive optimization, spatial fusion is performed on the decoded semantic features of multiple collaborative nodes to achieve high-precision 3D target perception and detection, without requiring any inter-vehicle synchronization signaling and with zero interaction overhead.

[0136] This application also provides a semantic communication system for collaborative perception in the Internet of Vehicles, including: a multi-vehicle perception and acquisition module, a semantic selection and coding module, a wireless channel transmission module, a lightweight adapter module for the receiver, and a feature fusion and detection module; The multi-vehicle perception and acquisition module is used by each cooperating node to collect environmental observation data through sensors and extract BEV feature maps through the local perception backbone network. The semantic selection encoding module is connected to the multi-vehicle perception and acquisition module. It is used to select the target semantic tokens with the highest semantic importance scores in the BEV feature map according to the preset load budget, and encode the target semantic tokens into compact codeword tensors through a pre-trained shared semantic encoder. The wireless channel transmission module is connected to the semantic selection coding module and is used to transmit the codeword tensor to the vehicle receiver through the V2X wireless link, simulate channel attenuation and noise interference, and output the distorted semantic token. The receiver lightweight adaptation module is connected to the wireless channel transmission module and has a built-in pre-trained semantic decoder backbone network, lightweight adapter and reliability scoring unit. The parameters of the semantic decoder backbone network are frozen throughout, and only the lightweight adapter supports online parameter updates. The reliability scoring unit is used to calculate the reliability score of each semantic token by fusing semantic importance score, channel confidence and token prediction uncertainty score, to select high-reliability tokens and complete reliability gating adaptive optimization. The feature fusion and detection module is connected to the lightweight adaptation module of the receiving end. It is used to spatially fuse the decoded semantic features of multiple cooperative nodes after adaptive optimization, and output the final collaborative perception detection result through the three-dimensional detector. The system achieves domain generalization and channel stability of semantic coding through offline multi-constraint pre-training, and achieves dynamic adaptation without source and with zero vehicle-to-vehicle synchronization through online receiver lightweight adaptation, adapting to V2X cooperative sensing scenarios coupled with observation domain offset and wireless channel distortion.

[0137] The system is compatible with multiple wireless channels including AWGN and Rayleigh fading, and is compatible with two types of domain offset scenarios: cross-weather and cross-dataset. It can stably achieve high-precision V2X collaborative perception semantic communication even without target domain labels or vehicle-to-vehicle synchronization.

[0138] Figure 6 This is a schematic diagram of the semantic communication system structure for collaborative perception in vehicle networking, as described in this application embodiment. Figure 6 As shown, during the offline training phase, vehicle point clouds or images from multi-observation domain source data are converted into BEV feature maps by the corresponding modality's local perception backbone network; the semantic selector retains Top-U semantic tokens based on semantic importance; the semantic encoder maps them to compact codewords; after sampling through the training channel, the codewords are recovered into semantic features by the receiving end semantic decoder, and then spatial fusion and 3D detection are performed. During training, detection loss, cross-domain prototype alignment loss, cross-channel gradient consistency loss, codeword compactness loss, and uncertainty regression loss are calculated on the features after channel decoding, and the offline model parameters are jointly updated.

[0139] like Figure 6 As shown, during the online deployment phase, the collaborative vehicles use a frozen local perception backbone network, semantic selector, and semantic encoder to generate and send Top-U codewords. The target receiving node receives the codewords through the actual V2X wireless link, decodes them through the frozen semantic decoder backbone network and a lightweight adapter with updatable parameters, calculates the semantic importance, channel confidence, and prediction uncertainty of each token, and forms a reliability score. The parameters of the lightweight adapter are updated based on the online adaptive total loss, and then spatial fusion and 3D detection are completed.

[0140] Figure 6 The “multi-domain source data” in the text refers to a collection of multiple source domains formed by point cloud or image samples according to weather, scene, or dataset sources. The arrow points to the cross-domain prototype alignment module, indicating that the domain prototypes are calculated and aligned using the source domain identifiers and decoding features of each domain, i.e., constraints are imposed by the cross-domain prototype alignment loss function.

[0141] Compared with the prior art, the embodiments of this application have multiple technical advantages and beneficial effects, as detailed below: First, it achieves dual offset coupling adaptation, significantly improving robustness in complex scenarios. The embodiments of this application integrate cross-domain generalization and channel anti-interference optimization, while simultaneously solving the coupling degradation problem between observation domain weather, dataset offset and wireless channel fading and noise distortion. In scenarios with Rayleigh fading, unknown channels, severe weather, and cross-dataset deployment, the AP@0.7 index of the weather offset task is improved by 7.2 compared to the traditional domain generalization method, and the cross-dataset task is improved by 5.5, demonstrating a significant improvement in perception accuracy.

[0142] Second, the reliability-gated adaptive mechanism completely solves the problem of invalid token interference. This application's embodiment abandons the traditional indiscriminate adaptive update mode, selects high-quality semantic tokens through a reliability score fused from three dimensions, and completes gradient updates only based on valid tokens. This avoids corrupted or mismatched tokens misleading model optimization, effectively solves the adaptive model drift problem during testing, and significantly improves the adaptation stability in complex dynamic scenarios.

[0143] Third, it features extremely lightweight deployment, adapting to the constraints of automotive edge computing power. The embodiments of this application adopt a one-way adaptive update strategy at the receiving end, freezing the backbone network parameters throughout the process and updating only 0.21% of the adapter parameters. Compared with traditional adaptive methods, the parameter update volume is significantly reduced, and the latency of a single adaptation is increased by only 8.9ms. It balances adaptation accuracy and real-time performance, perfectly adapting to low computing power deployment scenarios of automotive embedded terminals and edge chips.

[0144] Fourth, zero vehicle-to-vehicle synchronization overhead reduces system deployment costs. This application's embodiment eliminates the need for sending-end parameter updates and multi-vehicle signaling synchronization, completely removing control channel transmission overhead. It solves the communication congestion and high latency problems of traditional multi-vehicle adaptive solutions, improving the stability and compatibility of the multi-vehicle cooperative system, and can be directly adapted to existing V2X communication protocol architectures.

[0145] Fifth, it boasts a complete theoretical framework, strong versatility, and high applicability. The embodiments of this application establish a one-to-one correspondence between each loss function and degradation scenarios through task robustness risk decomposition, providing rigorous theoretical support. Furthermore, it is adaptable to various channels such as AWGN and Rayleigh fading, compatible with mainstream V2X collaborative sensing datasets, exhibits extremely strong scenario versatility, and possesses significant value for engineering implementation and industrial application.

[0146] It should be noted that the semantic communication method for vehicle-to-everything (V2X) cooperative perception provided in this application can be executed by a semantic communication device for V2X cooperative perception, or by a control module within that semantic communication device for executing the semantic communication method for V2X cooperative perception. This application uses the execution of the semantic communication method for V2X cooperative perception by a semantic communication device as an example to illustrate the semantic communication method for V2X cooperative perception provided in this application.

[0147] Figure 7 This is a schematic diagram of a semantic communication device for collaborative perception in vehicle-to-everything (V2X) networks, provided in an embodiment of this application. This semantic communication device is applied at the transmitting end, such as... Figure 7 As shown, the semantic communication device for collaborative perception in the Internet of Vehicles includes: BEV feature extraction module 710 is used to acquire environmental observation data and extract BEV feature map from the environmental observation data through local perception backbone network. The BEV feature map includes multiple candidate semantic tokens, and each candidate semantic token corresponds to a spatial location in the BEV feature map. The token selection module 720 is used to select a target number of candidate semantic tokens from the multiple candidate semantic tokens based on the semantic importance score of each candidate semantic token according to the target payload budget, and use them as the target semantic tokens. The token encoding transmission module 730 is used to encode a target number of target semantic tokens into codeword tensors through a semantic encoder, and transmit the codeword tensors to a receiving end through a wireless link. The codeword tensors transmitted to the receiving end are used to filter target token features with a first reliability score greater than or equal to a reliability score threshold after decoding by a semantic decoder for spatial fusion. The collaborative perception model composed of the semantic selector, the semantic encoder, and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process.

[0148] Optionally, the token selection module is specifically used for: The semantic selector determines the semantic importance score of each candidate semantic token, and selects a target number of candidate semantic tokens from the multiple candidate semantic tokens according to the target payload budget, in descending order of the semantic importance scores, as the target semantic token.

[0149] Optionally, the training process of the collaborative perception model includes: Obtain a first training dataset, which includes multiple first training samples. Each first training sample includes sample observation data and 3D annotations from at least one cooperative node. Different first training samples have multi-source observation domains and multiple channel conditions. The sample BEV feature map is extracted from the sample observation data through the local perception backbone network, and the sample BEV feature map includes multiple sample candidate semantic tokens; Based on the target payload budget, a target number of sample candidate semantic tokens are selected from the multiple sample candidate semantic tokens by an initialized semantic selector, and these are used as the sample target semantic tokens. The target number of sample target semantic tokens are encoded into sample codeword tensors by the initialized semantic encoder; Based on the channel conditions corresponding to the sample observation data, determine the sample received codeword tensor corresponding to the sample codeword tensor; The sample received codeword tensor is decoded by the initialized semantic decoder to obtain the target number of sample received token features; For each collaborative node, the reliability score of each sample received token feature is determined, and based on the reliability score, sample received token features with a reliability score greater than or equal to the reliability score threshold are selected from the target number of sample received token features as sample target token features. The spatial fusion module performs spatial fusion on the sample target token features corresponding to each collaborative node, and the three-dimensional detector performs three-dimensional detection on the fusion result to obtain the sample collaborative perception result. Based on the sample collaborative perception results, the 3D annotation, the sample received token features, and the reliability score, a joint loss function value is determined, and the parameters of the initialized collaborative perception model are updated based on the joint loss function value to obtain a pre-trained collaborative perception model; the initialized collaborative perception model includes the initialized semantic selector, the initialized semantic encoder, and the initialized semantic decoder. The joint loss function includes a detection loss function, a cross-domain prototype alignment loss function, a cross-channel gradient consistency loss function, a codeword compactness loss function, and an uncertainty regression loss function. The detection loss function is used to quantify the difference between the sample collaborative perception result and the 3D annotation. The cross-domain prototype alignment loss function is used to constrain the consistency of the semantic prototype distribution in the multi-source observation domain and suppress semantic drift in the observation domain. The cross-channel gradient consistency loss function is used to regularize the gradient statistical features of the semantic decoder. The codeword compactness loss function is used to reduce encoding energy. The uncertainty regression loss function is used to train the token uncertainty evaluation capability.

[0150] The semantic communication device for collaborative perception in vehicle-to-everything (V2X) provided in this application extracts BEV feature maps from environmental observation data through a local perception backbone network. The BEV feature maps include multiple candidate semantic tokens. Based on the target payload budget, a semantic selector selects a target number of candidate semantic tokens from the multiple candidate semantic tokens according to the semantic importance score of each candidate semantic token. These target semantic tokens are then encoded into codeword tensors by a semantic encoder. The codeword tensors are transmitted to the receiving end via a wireless link. The codeword tensors transmitted to the receiving end can be decoded by a semantic decoder to select target token features with a first reliability score greater than or equal to a reliability score threshold for spatial fusion. The collaborative perception model composed of the semantic selector, semantic encoder, and semantic decoder constrains the consistency of the semantic prototype distribution of multi-source observation domains based on a cross-domain prototype alignment loss function during training, suppressing semantic drift between different observation domains. Moreover, the semantic decoder at the receiving end can select token features with higher first reliability scores for spatial fusion after decoding, avoiding the use of distorted or damaged token features in spatial fusion. This solves the distribution offset degradation problem caused by dual coupling due to different observation domains and channel transmission processes, thereby improving the accuracy of collaborative perception.

[0151] Figure 8 This is a schematic diagram of a semantic communication device for collaborative perception in vehicle-to-everything (V2X) networks, provided in an embodiment of this application. This semantic communication device is applied at the receiving end, such as... Figure 8 As shown, the semantic communication device for collaborative perception in the Internet of Vehicles includes: The receiving module 810 is used to receive a codeword tensor sent by at least one sending end as a received codeword tensor; the codeword tensor sent by the sending end is obtained by semantically encoding a target number of target semantic tokens by a semantic encoder, and the target number of target semantic tokens are obtained by filtering multiple candidate semantic tokens included in the BEV feature map by a semantic selector, and each candidate semantic token in the BEV feature map corresponds to a spatial location in the BEV feature map. The decoding module 820 is used to decode the received codeword tensor of each of the sending ends using a semantic decoder to obtain a target number of received token features; wherein, the collaborative perception model composed of the semantic selector, the semantic encoder and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process. The token filtering module 830 is used to determine the first reliability score of each received token feature, and based on the first reliability score, to filter the received token features whose first reliability score is greater than or equal to the reliability score threshold from the target number of received token features as target token features. The spatial fusion module 840 is used to perform spatial fusion on the target token features corresponding to each of the sending ends to obtain the collaborative perception result.

[0152] Optionally, the token filtering module includes a reliability determination unit, which is used for: For each received token feature, a first semantic importance score is obtained from the received codeword tensor through a reliability scoring unit, a first channel confidence is determined based on the received signal amplitude of the received codeword tensor, and a first prediction uncertainty score is determined based on the received token feature. The first prediction uncertainty score is used to characterize the stability of the received token feature. The first semantic importance score, the first channel confidence score, and the first prediction uncertainty score are weighted and fused to obtain the first reliability score.

[0153] Optionally, the semantic decoder includes a semantic decoder backbone network, a lightweight adapter, and the reliability scoring unit; The device further includes an adapter module, which is used for: Obtain a second training dataset, which includes multiple second training samples, each of which includes a target number of distorted semantic tokens transmitted by multiple collaborative nodes. For each collaborative node, the semantic decoder backbone network decodes the target number of distorted semantic tokens to obtain the target number of initial token features for samples. The pre-trained lightweight adapter processes the target number of initial token features for samples to obtain the target number of target token features for samples. For each of the sample target token features, the reliability scoring unit determines the second semantic importance score, the second channel confidence score, and the second prediction uncertainty score corresponding to the sample target token feature, and performs a weighted fusion of the second semantic importance score, the second channel confidence score, and the second prediction uncertainty score to obtain the second reliability score of the sample target token feature; Based on the second reliability score, determine the reliability gated entropy loss function value, and based on the second reliability score and the target token feature, determine the soft prototype anchoring loss function value. Based on the pre-training parameters of the pre-trained lightweight adapter and the current parameters of the lightweight adapter, determine the parameter regularization loss function value. Based on the reliability-gated entropy loss function value, the soft prototype anchoring loss function value, and the parameter regularization loss function value, an adaptive total loss function value is determined, and the parameters of the pre-trained lightweight adapter are iteratively updated based on the adaptive total loss function value.

[0154] The semantic communication device for collaborative perception in vehicle-to-everything (V2X) provided in this application provides a semantic communication device that decodes the received codeword tensor of each transmitter using a semantic decoder to obtain a target number of received token features. A first reliability score is determined for each received token feature, and based on this first reliability score, received token features with a first reliability score greater than or equal to a reliability score threshold are selected from the target number of received token features as target token features. Spatial fusion is then performed on the target token features corresponding to each transmitter to obtain the collaborative perception result. The collaborative perception model, composed of a semantic selector, semantic encoder, and semantic decoder, constrains the consistency of the semantic prototype distribution of multi-source observation domains during training based on a cross-domain prototype alignment loss function, suppressing semantic drift across different observation domains. Furthermore, after decoding, the receiver can select token features with higher first reliability scores for spatial fusion, avoiding the use of distorted or damaged token features in spatial fusion. This solves the distribution offset degradation problem caused by dual coupling due to different observation domains and channel transmission processes, thereby improving the accuracy of collaborative perception.

[0155] The semantic communication device for vehicle-to-everything (V2X) collaborative perception in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0156] The semantic communication device for vehicle-to-everything (V2X) collaborative perception in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0157] The semantic communication device for vehicle-to-everything (V2X) cooperative perception provided in this application embodiment can achieve... Figures 1 to 4 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0158] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described semantic communication method embodiments for collaborative perception in vehicle networking and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0159] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0160] Figure 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0161] The electronic device 900 includes, but is not limited to, components such as: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910. The input unit 904 may include an image processor 9041 and a microphone 9042; the display unit 906 may include a display panel 9061; and the user input unit 907 may include a touch panel 9071 and other input devices (such as a keyboard) 9072.

[0162] Those skilled in the art will understand that the electronic device 900 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0163] The memory 909 stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor 910, it implements the various processes of the above-described semantic communication method embodiment for collaborative perception in vehicle networking and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0164] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described semantic communication method embodiments for collaborative perception in vehicle networking and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0165] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0166] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described semantic communication method embodiment for vehicle network collaborative perception, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0167] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0168] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0170] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A semantic communication method for collaborative perception in vehicle-to-everything (V2X) networks, characterized in that, Applied to the sending end, including: Acquire environmental observation data, and extract BEV feature maps from the environmental observation data through a local sensing backbone network. The BEV feature maps include multiple candidate semantic tokens, and each candidate semantic token corresponds to a spatial location in the BEV feature maps. Based on the target payload budget, a target number of candidate semantic tokens are selected from the multiple candidate semantic tokens by a semantic selector according to the semantic importance score of each candidate semantic token, and these are selected as the target semantic tokens. The target number of target semantic tokens are encoded into codeword tensors by a semantic encoder, and the codeword tensors are transmitted to the receiving end via a wireless link. The codeword tensors transmitted to the receiving end are used to filter target token features with a first reliability score greater than or equal to a reliability score threshold after decoding by a semantic decoder for spatial fusion. The collaborative perception model composed of the semantic selector, the semantic encoder, and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process.

2. The method according to claim 1, characterized in that, The step of selecting a target number of candidate semantic tokens from the plurality of candidate semantic tokens based on the semantic importance score of each candidate semantic token, according to the target payload budget, includes: The semantic selector determines the semantic importance score of each candidate semantic token, and selects a target number of candidate semantic tokens from the multiple candidate semantic tokens according to the target payload budget, in descending order of the semantic importance scores, as the target semantic token.

3. The method according to claim 1, characterized in that, The training process of the collaborative perception model includes: Obtain a first training dataset, which includes multiple first training samples. Each first training sample includes sample observation data and 3D annotations from at least one cooperative node. Different first training samples have multi-source observation domains and multiple channel conditions. The sample BEV feature map is extracted from the sample observation data through the local perception backbone network, and the sample BEV feature map includes multiple sample candidate semantic tokens; Based on the target payload budget, a target number of sample candidate semantic tokens are selected from the multiple sample candidate semantic tokens by an initialized semantic selector, and these are used as the sample target semantic tokens. The target number of sample target semantic tokens are encoded into sample codeword tensors by the initialized semantic encoder; Based on the channel conditions corresponding to the sample observation data, determine the sample received codeword tensor corresponding to the sample codeword tensor; The sample received codeword tensor is decoded by the initialized semantic decoder to obtain the target number of sample received token features; For each collaborative node, the reliability score of each sample received token feature is determined, and based on the reliability score, sample received token features with a reliability score greater than or equal to the reliability score threshold are selected from the target number of sample received token features as sample target token features. The spatial fusion module performs spatial fusion on the sample target token features corresponding to each collaborative node, and the three-dimensional detector performs three-dimensional detection on the fusion result to obtain the sample collaborative perception result. Based on the sample collaborative perception results, the 3D annotation, the sample received token features, and the reliability score, a joint loss function value is determined, and the parameters of the initialized collaborative perception model are updated based on the joint loss function value to obtain a pre-trained collaborative perception model; the initialized collaborative perception model includes the initialized semantic selector, the initialized semantic encoder, and the initialized semantic decoder. The joint loss function includes a detection loss function, a cross-domain prototype alignment loss function, a cross-channel gradient consistency loss function, a codeword compactness loss function, and an uncertainty regression loss function. The detection loss function is used to quantify the difference between the sample collaborative perception result and the 3D annotation. The cross-domain prototype alignment loss function is used to constrain the consistency of the semantic prototype distribution in the multi-source observation domain and suppress semantic drift in the observation domain. The cross-channel gradient consistency loss function is used to regularize the gradient statistical features of the semantic decoder. The codeword compactness loss function is used to reduce encoding energy. The uncertainty regression loss function is used to train the token uncertainty evaluation capability.

4. A semantic communication method for collaborative perception in vehicle-to-everything (V2X) networks, characterized in that, Applied to the receiving end, including: Receive at least one codeword tensor sent by a transmitter as the received codeword tensor; the codeword tensor sent by the transmitter is obtained by semantically encoding a target number of target semantic tokens by a semantic encoder, and the target number of target semantic tokens are obtained by filtering multiple candidate semantic tokens included in the BEV feature map by a semantic selector, and each candidate semantic token in the BEV feature map corresponds to a spatial location in the BEV feature map. For each received codeword tensor of the sending end, the received codeword tensor is decoded by a semantic decoder to obtain a target number of received token features; wherein, the collaborative perception model composed of the semantic selector, the semantic encoder and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process; A first reliability score is determined for each received token feature, and based on the first reliability score, received token features with a first reliability score greater than or equal to a reliability score threshold are selected from a target number of received token features as target token features. Spatial fusion is performed on the target token features corresponding to each of the aforementioned sending ends to obtain the collaborative perception result.

5. The method according to claim 4, characterized in that, The determination of the first reliability score for each received token feature includes: For each received token feature, a first semantic importance score is obtained from the received codeword tensor through a reliability scoring unit, a first channel confidence is determined based on the received signal amplitude of the received codeword tensor, and a first prediction uncertainty score is determined based on the received token feature. The first prediction uncertainty score is used to characterize the stability of the received token feature. The first semantic importance score, the first channel confidence score, and the first prediction uncertainty score are weighted and fused to obtain the first reliability score.

6. The method according to claim 5, characterized in that, The semantic decoder includes a semantic decoder backbone network, a lightweight adapter, and the reliability scoring unit; Before decoding the target number of received codeword tensors using a semantic decoder, the method further includes: Obtain a second training dataset, which includes multiple second training samples, each of which includes a target number of distorted semantic tokens transmitted by multiple collaborative nodes. For each collaborative node, the semantic decoder backbone network decodes the target number of distorted semantic tokens to obtain the target number of initial token features for samples. The pre-trained lightweight adapter processes the target number of initial token features for samples to obtain the target number of target token features for samples. For each of the sample target token features, the reliability scoring unit determines the second semantic importance score, the second channel confidence score, and the second prediction uncertainty score corresponding to the sample target token feature, and performs a weighted fusion of the second semantic importance score, the second channel confidence score, and the second prediction uncertainty score to obtain the second reliability score of the sample target token feature; Based on the second reliability score, determine the reliability gated entropy loss function value, and based on the second reliability score and the target token feature, determine the soft prototype anchoring loss function value. Based on the pre-training parameters of the pre-trained lightweight adapter and the current parameters of the lightweight adapter, determine the parameter regularization loss function value. Based on the reliability-gated entropy loss function value, the soft prototype anchoring loss function value, and the parameter regularization loss function value, an adaptive total loss function value is determined, and the parameters of the pre-trained lightweight adapter are iteratively updated based on the adaptive total loss function value.

7. A semantic communication device for collaborative perception in vehicle networking, characterized in that, Applied to the sending end, including: The BEV feature extraction module is used to acquire environmental observation data and extract BEV feature maps from the environmental observation data through the local perception backbone network. The BEV feature map includes multiple candidate semantic tokens, and each candidate semantic token corresponds to a spatial location in the BEV feature map. The token selection module is used to select a target number of candidate semantic tokens from the multiple candidate semantic tokens based on the semantic importance score of each candidate semantic token, according to the target payload budget; The token encoding transmission module is used to encode a target number of target semantic tokens into codeword tensors through a semantic encoder, and transmit the codeword tensors to the receiving end through a wireless link. The codeword tensors transmitted to the receiving end are used to filter target token features with a first reliability score greater than or equal to a reliability score threshold after decoding by a semantic decoder for spatial fusion. The collaborative perception model composed of the semantic selector, the semantic encoder, and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process.

8. A semantic communication device for collaborative perception in vehicle networking, characterized in that, Applied to the receiving end, including: The receiving module is used to receive at least one codeword tensor sent by a sending end as a received codeword tensor; the codeword tensor sent by the sending end is obtained by semantically encoding a target number of target semantic tokens by a semantic encoder, and the target number of target semantic tokens are obtained by filtering multiple candidate semantic tokens included in the BEV feature map by a semantic selector, and each candidate semantic token in the BEV feature map corresponds to a spatial location in the BEV feature map. The decoding module is used to decode the received codeword tensor of each of the sending ends using a semantic decoder to obtain a target number of received token features; wherein, the collaborative perception model composed of the semantic selector, the semantic encoder and the semantic decoder constrains the consistency of the semantic prototype distribution of the multi-source observation domain based on the cross-domain prototype alignment loss function during the training process. The token filtering module is used to determine the first reliability score of each received token feature, and based on the first reliability score, to filter the received token features whose first reliability score is greater than or equal to the reliability score threshold from the target number of received token features as target token features. The spatial fusion module is used to spatially fuse the target token features corresponding to each of the sending ends to obtain the collaborative perception result.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the semantic communication method for collaborative perception in vehicle networking as described in any one of claims 1-3 or 4-6.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the semantic communication method for collaborative perception in vehicle networking as described in any one of claims 1-3 or 4-6.

Images