Task-oriented millimeter wave link adaptive multi-mode semantic communication method in automatic driving scene

Abstract

The invention provides a task-oriented millimeter wave link adaptive multi-mode semantic communication method and device in an automatic driving scene, electronic equipment and a storage medium, and belongs to the technical field of automatic driving, and the method comprises the steps that a transmitting end inputs a target image into a semantic encoder to obtain a first feature map, and then inputs the first feature map into a joint information source channel encoder, obtaining a second feature map; performing feature screening on the second feature map to obtain a third feature map, performing digital pre-coding and analog pre-coding to obtain a corresponding sending signal matrix, and sending the sending signal matrix to a receiving end; the receiving end performs analog merging and digital merging on the received signal matrix to obtain a received feature map, and inputs the received feature map to a joint source channel decoder to obtain decoding features; and performing fusion processing on each decoding feature to obtain a fusion feature, and inputting the fusion feature into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder. According to the invention, the robustness and accuracy of semantic communication are improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario. Background Art

[0002] In autonomous driving technology, environmental perception, precise positioning, information interaction and decision control are its core components. Among them, the vehicle's perception and understanding of the surrounding environment is the basis for realizing autonomous driving. However, the perception range and computing power of a single vehicle are limited and easily blocked, which is inconsistent with the demand for high-quality, all-round perception of autonomous driving, and it is difficult to ensure truly effective and safe autonomous driving. Therefore, it is particularly important to share environmental perception data between multiple vehicles.

[0003] Semantic communication technology can significantly reduce the amount of data transmission by extracting and transmitting key semantic information, providing a new solution for autonomous driving scenarios. Current research has covered vehicle-mounted semantic communication systems for single-modal image segmentation. Considering a single-link V2V scenario, the image is compressed by an encoder, transmitted through a wireless channel, and a semantic segmentation map is generated by a decoder at the receiving end. However, the vehicle-mounted semantic communication system for single-modal image segmentation only performs environmental perception and information interaction based on images of one modality, which depends on lighting conditions. In different time periods and weather conditions, the quality of the image will greatly affect the accuracy of semantic segmentation.

[0004] Therefore, how to ensure the accuracy of semantic segmentation becomes a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario, so as to solve the defect in the prior art that the accuracy of single-modal image segmentation is affected by lighting conditions.

[0006] The present invention provides a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario, comprising the following steps: The transmitting end inputs the target image into the semantic encoder to obtain a first feature map output by the semantic encoder, wherein the target image is any modality image among the multimodal images collected by the transmitting end through the sensing device; The transmitting end inputs the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, wherein the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; The transmitting end performs feature screening on the second feature map to obtain a third feature map; The transmitting end performs digital precoding and analog precoding on the third feature map to obtain a transmission signal matrix corresponding to the target image, and then sends the transmission signal matrix corresponding to each target image to the receiving end; The receiving end performs analog merging and digital merging on multiple matrices received from multiple transmitting ends to obtain multiple receiving characteristic graphs; The receiving end inputs each of the merging matrices into a joint source channel decoder to obtain decoding features corresponding to the output of the joint source channel decoder; The receiving end performs fusion processing on each of the decoding features to obtain a fused feature, and then inputs the fused feature into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder.

[0007] According to a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario provided by the present invention, the method further includes: The transmitting end expands the user's channel matrix into After the vector is obtained, the vector is input into the jointly trained deep neural network model to obtain the simulated precoding matrix output by the deep neural network model, where is the number of receiving antennas at the receiving end, is the number of transmitting antennas at the transmitting end; The jointly trained deep neural network model includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, wherein the first linear layer and the second linear layer are connected to the first Dropout layer through a Leaky-Relu activation function, the second linear layer and the third linear layer are connected to the second Dropout layer through a Leaky-Relu activation function, the third linear layer and the fourth linear layer are connected to the third Dropout layer through a Leaky-Relu activation function, and the fourth linear layer and the fifth linear layer are connected to the fourth Dropout layer through a Leaky-Relu activation function.

[0008] According to a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario provided by the present invention, the number of channels of the second feature map is , the number of channels of the third feature map is ,in, , the transmitting end performs feature screening on the second feature map to obtain a third feature map, including: Calculating a semantic importance score of each channel feature in the second feature map, where the semantic importance score represents the semantic importance of the channel feature; Based on the semantic importance scores of the channel features, the channel features are sorted and the top Channel features form the original semantic feature set; For any remaining feature, use the remaining feature to replace the channel feature in the original semantic feature set, and the remaining feature is a channel feature that does not belong to the original semantic feature set; If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is used as the original semantic feature set, and the semantic score represents the semantic importance and semantic repetition of the semantic feature set; If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, the original semantic feature set is kept unchanged; Return to the step of replacing the channel feature in the original semantic feature set with any remaining feature, until all the remaining features have been attempted to be replaced.

[0009] According to a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario provided by the present invention, the semantic importance score of each channel feature is calculated based on the following formula: ; in, Indicates The semantic importance score of the channel feature, Indicates the number of transmitters, Indicates The transmitter The semantic importance scores of channel features.

[0010] According to a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario provided by the present invention, the semantic score is calculated based on the following formula: ; in, represents the semantic score, represents the channel feature set, Represents channel characteristics and channel characteristics The similarity of Represents the semantic importance score of each channel feature, and is the weight coefficient.

[0011] According to a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario provided by the present invention, each of the transmitting ends continuously shares the semantic importance score of each feature channel of each other based on the following steps: Each of the transmitting ends transmits the semantic importance scores of all its feature channels to the receiving end; The receiving end continuously feeds back the semantic importance scores of all transmitting ends to each of the transmitting ends.

[0012] The present invention also provides a task-oriented millimeter wave link adaptive multi-modal semantic communication system in an autonomous driving scenario, comprising a plurality of transmitting ends and a receiving end; Each of the transmitting ends is used to input a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, wherein the target image is any modal image of the multimodal images collected by the transmitting end through a sensing device; input the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, wherein the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; perform feature screening on the second feature map to obtain a third feature map; perform digital precoding and analog precoding on the third feature map to obtain a transmission signal matrix corresponding to the target image, and then send the transmission signal matrix corresponding to each target image to a receiving end; The receiving end is used to perform analog merging and digital merging on multiple receiving signal matrices received from multiple transmitting ends to obtain multiple receiving feature maps; input each of the receiving feature maps into a joint source channel decoder to obtain a decoding feature corresponding to the output of the joint source channel decoder; fuse each of the decoding features to obtain a fused feature, and then input the fused feature into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder.

[0013] According to a task-oriented millimeter wave link adaptive multimodal semantic communication system in an autonomous driving scenario provided by the present invention, the transmitting end is also used for: Expand the user's channel matrix into After the vector is obtained, the vector is input into the jointly trained deep neural network model to obtain the simulated precoding matrix output by the deep neural network model, where is the number of receiving antennas at the receiving end, is the number of transmitting antennas at the transmitting end; The pre-trained deep neural network model includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, the first linear layer and the second linear layer are connected to the first Dropout layer through a Leaky-Relu activation function, the second linear layer and the third linear layer are connected to the second Dropout layer through a Leaky-Relu activation function, the third linear layer and the fourth linear layer are connected to the third Dropout layer through a Leaky-Relu activation function, and the fourth linear layer and the fifth linear layer are connected to the fourth Dropout layer through a Leaky-Relu activation function.

[0014] According to a task-oriented millimeter wave link adaptive multimodal semantic communication system in an autonomous driving scenario provided by the present invention, the number of channels of the second feature map is , the number of channels of the third feature map is ,in, , the transmitting end is specifically used for: Calculating a semantic importance score of each channel feature in the second feature map, where the semantic importance score represents the semantic importance of the channel feature; Based on the semantic importance scores of the channel features, the channel features are sorted and the top Channel features form the original semantic feature set; For any remaining feature, use the remaining feature to replace the channel feature in the original semantic feature set, and the remaining feature is a channel feature that does not belong to the original semantic feature set; If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is used as the original semantic feature set, and the semantic score represents the semantic importance and semantic repetition of the semantic feature set; If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, the original semantic feature set is kept unchanged; Return to the step of replacing the channel feature in the original semantic feature set with any remaining feature, until all the remaining features have been attempted to be replaced.

[0015] According to a task-oriented millimeter wave link adaptive multimodal semantic communication system in an autonomous driving scenario provided by the present invention, the transmitting end calculates the semantic importance score of each channel feature based on the following formula: ; in, Indicates The semantic importance score of the channel feature, Indicates the number of transmitters, Indicates The transmitter The semantic importance scores of channel features.

[0016] According to a task-oriented millimeter wave link adaptive multimodal semantic communication system in an autonomous driving scenario provided by the present invention, the transmitter calculates the semantic score based on the following formula: ; in, represents the semantic score, represents the channel feature set, Represents channel characteristics and channel characteristics The similarity of Represents the semantic importance score of each channel feature, and is the weight coefficient.

[0017] According to a task-oriented millimeter wave link adaptive multimodal semantic communication system in an autonomous driving scenario provided by the present invention, each of the transmitting ends continuously shares the semantic importance score of each feature channel of each other based on the following steps: Each of the transmitting ends transmits the semantic importance scores of all its feature channels to the receiving end; The receiving end continuously feeds back the semantic importance scores of all transmitting ends to each of the transmitting ends.

[0018] The present invention provides a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario. The transmitting end inputs a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, and the target image is any modal image of the multimodal images collected by the transmitting end through a sensing device; the transmitting end inputs the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, and the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; the transmitting end performs feature screening on the second feature map to obtain a third feature map Figure; the transmitting end performs digital precoding and analog precoding on the third feature map, obtains the sending signal matrix corresponding to the target image, and then sends the sending signal matrix corresponding to each target image to the receiving end; the receiving end performs analog merging and digital merging on multiple receiving signal matrices received from multiple transmitting ends to obtain multiple receiving feature maps; the receiving end inputs each of the receiving feature maps into a joint source channel decoder to obtain the decoding features outputted by the joint source channel decoder; the receiving end performs fusion processing on each of the decoding features to obtain the fused features, and then inputs the fused features into the semantic decoder to obtain the semantic segmentation results outputted by the semantic decoder. The present invention fuses multi-modal image features to cope with complex and changeable driving environments, thereby improving the robustness and accuracy of semantic communication; in addition, the joint source channel encoder and the semantic encoder are jointly trained to further improve the accuracy of the image segmentation task and enhance the overall performance of the semantic communication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flowchart of a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario provided by the present invention; Figure 2 It is a schematic diagram of a multi-user multi-modal segmentation task scenario for semantic communication provided by the present invention; Figure 3 It is a schematic diagram of the structure of a multimodal semantic communication system model based on hybrid beamforming provided by the present invention; Figure 4 It is a schematic diagram of the DNN network structure for training the simulated beamforming matrix; Figure 5 is a schematic diagram of feature importance feedback provided by the present invention; Figure 6 is a schematic diagram showing the variation of the average intersection-to-parallel ratio with the signal-to-noise ratio provided by the present invention; Figure 7 It is a schematic diagram of the change of the average accuracy rate provided by the present invention with the signal-to-noise ratio; Figure 8 is a schematic diagram of the variation of the average intersection-to-parallel ratio with the signal-to-noise ratio when the compression rate provided by the present invention is 0.5; Fig. 9 is a schematic diagram of the variation of the average intersection-to-parallel ratio with the signal-to-noise ratio when the compression rate provided by the present invention is 0.75; Fig.10 It is a structural schematic diagram of the task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be a connection between two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] In the Internet of Vehicles environment, understanding traffic scenes is one of the key technologies for realizing autonomous driving, and image segmentation is the core task of scene understanding. Autonomous vehicles can use semantic segmentation graphs to complete important tasks such as vehicle perception, behavior prediction, pedestrian recognition, and path planning. However, the amount of data generated by visual sensors is huge, and existing vehicle-mounted communication technologies are difficult to meet their needs for high-speed and high-reliability transmission.

[0025] Semantic communication technology can significantly reduce the amount of data transmission by extracting and transmitting key semantic information, providing a new solution for autonomous driving scenarios. Current research has covered vehicle-mounted semantic communication systems for single-modal image segmentation. Considering a single-link V2V scenario, the image is compressed by an encoder, transmitted through a wireless channel, and a semantic segmentation map is generated by a decoder at the receiving end. In addition, the semantic communication system for multimodal image segmentation considers combining visible light images and infrared images, encoding and compressing them separately, and fusing them through the channel. At the same time, the design of semantic precoding technology can retain semantic information as much as possible when fusing channel-level information.

[0026] However, in a multi-user, multi-modal semantic communication network, the available bandwidth of different users varies greatly, which requires an adaptive feature screening algorithm and ensures that the screened multi-modal features can be aligned. In addition, millimeter wave communication technology provides greater bandwidth resources and higher data transmission efficiency, but the existing multi-modal semantic communication system design has not yet fully integrated millimeter wave communication technology. Therefore, for multi-user semantic communication scenarios for image segmentation tasks, there are two main challenges: 1) How to achieve adaptive screening of multi-modal features and ensure feature alignment; 2) How to design a task-oriented hybrid beamforming scheme suitable for millimeter wave semantic communication.

[0027] In view of the above problems, the present invention proposes the following embodiments. Figure 1-Figure 10 The present invention describes a method and system for adaptive multimodal semantic communication over a millimeter wave link for a task-oriented autonomous driving scenario provided by an embodiment of the present invention.

[0028] Figure 1is a flow chart of a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario provided by the present invention, such as Figure 1 As shown, the method includes the following: S110, the transmitting end inputs a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, wherein the target image is any modality image among the multimodal images collected by the transmitting end through a sensing device; S120, the transmitting end inputs the first feature map to a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, wherein the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; S130, the transmitting end performs feature screening on the second feature map to obtain a third feature map; S140, the transmitting end performs digital precoding and analog precoding on the third feature map to obtain a transmission signal matrix corresponding to the target image, and then sends the transmission signal matrix corresponding to each target image to the receiving end; S150, the receiving end performs analog merging and digital merging on a plurality of receiving signal matrices received from a plurality of transmitting ends to obtain a plurality of receiving characteristic graphs; S160, the receiving end inputs each of the receiving feature graphs into a joint source channel decoder to obtain a decoding feature outputted corresponding to the joint source channel decoder; S170, the receiving end performs fusion processing on each of the decoding features, and after obtaining the fused features, inputs the fused features into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder.

[0029] Figure 2 is a schematic diagram of a multi-user multi-modal segmentation task scenario for semantic communication provided by the present invention, such as Figure 2 As shown in the figure, in an autonomous driving environment, V2V-assisted collaborative perception can be abstracted as a process in which multiple users (i.e., vehicles) transmit perception information to a single user (target vehicle) and extract semantics. Specifically, the vehicle collects image data through the perception device and compresses the image locally through an encoder. The compressed image data is transmitted to the target vehicle through a wireless channel. At the receiving end, the target vehicle uses a decoder to decode the received data and generate a corresponding semantic segmentation map, thereby achieving collaborative perception of the surrounding environment.

[0030] Figure 3 is a schematic diagram of the structure of a multimodal semantic communication system model based on hybrid beamforming provided by the present invention, such as Figure 3As shown, a multi-user hybrid beamforming MIMO system is considered, the transmitter includes a semantic encoder, a joint source channel encoder, feature screening, digital precoding and analog precoding, and the receiver includes analog combining, digital combining, a joint source channel decoder, a fusion module and a semantic decoder.

[0031] In S110, the multimodal images may include but are not limited to visible light images, infrared images, radar images, etc. Figure 3 As shown, the transmitter The image semantic encoder uses a convolutional neural network to train the input image For feature extraction, it can be expressed as ,in It has trainable parameters Semantic extraction network.

[0032] In S120, the joint source channel encoder further extracts semantic information, which is expressed as ,in is a trainable parameter A joint source-channel encoder network.

[0033] In S130, the second feature map is compressed into ,in represents a feature screening operation, Indicates the semantic compression ratio.

[0034] In S150, each user at the transmitting end is equipped with Transmitting antennas, RF chains, the receiving end is equipped with Root receiving antenna, The analog precoding matrix at the transmitter is recorded as , the digital precoding matrix is ​​recorded as ,in is the number of data streams, and the simulated merging matrix is ​​recorded as , the digital merge matrix is ​​recorded as ,in Received by the receiving end User (transmitter) data streams, the channel matrix is ​​recorded as , additive Gaussian white noise is denoted as , the signal received by the receiving end can be expressed as: .

[0035] In S160, the information is semantically restored by the joint source channel decoder, which can be expressed as ,in It has trainable parameters A joint source-channel decoder network.

[0036] In S170, the features of different modalities are fused element by element, and the semantic decoder outputs the semantic segmentation result, which is expressed as ,in It has trainable parameters The semantic decoder network of

[0037] In the specific implementation process, the feature fusion method may include but is not limited to addition, splicing, etc.

[0038] In an embodiment of the present invention, a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario is provided. The transmitting end inputs a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, and the target image is any modal image of the multimodal images collected by the transmitting end through a sensing device; the transmitting end inputs the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, and the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; the transmitting end performs feature screening on the second feature map to obtain a third feature map. Feature map; the transmitting end performs digital precoding and analog precoding on the third feature map, obtains the sending signal matrix corresponding to the target image, and then sends the sending signal matrix corresponding to each target image to the receiving end; the receiving end performs analog merging and digital merging on multiple receiving signal matrices received from multiple transmitting ends to obtain multiple receiving feature maps; the receiving end inputs each of the receiving feature maps into a joint source channel decoder to obtain the decoding features outputted by the joint source channel decoder; the receiving end performs fusion processing on each of the decoding features to obtain the fused features, and then inputs the fused features into the semantic decoder to obtain the semantic segmentation results outputted by the semantic decoder. The present invention fuses multi-modal image features to cope with complex and changeable driving environments, thereby improving the robustness and accuracy of semantic communication; in addition, the joint source channel encoder and the semantic encoder are jointly trained to further improve the accuracy of the image segmentation task and enhance the overall performance of the semantic communication system.

[0039] In an optional embodiment, the method further comprises: The transmitting end expands the user's channel matrix into After the vector is obtained, the vector is input into the jointly trained deep neural network model to obtain the simulated precoding matrix output by the deep neural network model, where is the number of receiving antennas at the receiving end, is the number of transmitting antennas at the transmitting end; The pre-trained deep neural network model includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, the first linear layer and the second linear layer are connected to the first Dropout layer through a Leaky-Relu activation function, the second linear layer and the third linear layer are connected to the second Dropout layer through a Leaky-Relu activation function, the third linear layer and the fourth linear layer are connected to the third Dropout layer through a Leaky-Relu activation function, and the fourth linear layer and the fifth linear layer are connected to the fourth Dropout layer through a Leaky-Relu activation function.

[0040] In traditional communication systems, the goal of calculating the analog beamforming matrix is ​​usually to maximize the sum rate; in the embodiment of the present invention, the same loss function as the semantic segmentation network is used to solve the matrix through joint training. Specifically: Figure 4 It is a schematic diagram of the DNN network structure for training the simulated beamforming matrix, such as Figure 4 As shown, the user's channel matrix Expand to A vector is input into a DNN network, such as Figure 3 As shown in Figure 1, the DNN network consists of five linear layers, of which the first four linear layers each contain a Leaky-Relu activation function and a Dropout layer. The number of neurons in the five linear layers are 1024, 512, 256, 128, and . You can get The real-valued phase , and then convert the output phase into complex form: ; Similarly, the product of the simulated precoding matrix and the channel matrix is ​​expanded and input into the DNN network to obtain The real-valued phase , and then convert the output phase into complex form to obtain the analog merging matrix: ; After analog beamforming, the signal received by the receiver can be expressed as: ; in, is the equivalent channel matrix, = .

[0041] In the embodiment of the present invention, the calculation of digital beamforming is similar to that of traditional communication and is divided into two steps. The first step is to eliminate interference between users, and the second step is to eliminate interference between antennas within a user. Finally, the signal received by the receiving end realizes multi-user detection and decomposes the MIMO channel into sub-channels that do not interfere with each other. The signal received by the receiving end can be expressed as: .

[0042] In an optional embodiment, the number of channels of the second feature map is , the number of channels of the third feature map is ,in, , the transmitting end performs feature screening on the second feature map to obtain a third feature map, including: Calculating a semantic importance score of each channel feature in the second feature map, where the semantic importance score represents the semantic importance of the channel feature; Based on the semantic importance scores of the channel features, the channel features are sorted and the top Channel features form the original semantic feature set; For any remaining feature, use the remaining feature to replace the channel feature in the original semantic feature set, and the remaining feature is a channel feature that does not belong to the original semantic feature set; If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is used as the original semantic feature set, and the semantic score represents the semantic importance and semantic repetition of the semantic feature set; If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, the original semantic feature set is kept unchanged; Return to the step of replacing the channel feature in the original semantic feature set with any remaining feature, until all the remaining features have been attempted to be replaced.

[0043] Specifically, different systems have different available bandwidths, which requires adaptive feature screening. The number of channels of the feature map output by joint source channel coding is , the number of channels of the feature map after feature screening is ,in The goal of feature selection is to select the feature set with the highest semantic importance and the least semantic duplication from a large number of features. All users use the same feature selection algorithm, first selecting the feature set with the highest semantic importance score. The features are put into the semantic feature set to be transmitted Next, for the remaining features, try to replace them one by one The features in are used to obtain a new feature set, and the optimal association and minimum repetition value of the set are calculated. If the value of the new feature set is improved, the iterative exchange is continued. This process will continue until all the remaining features have been exchanged.

[0044] It is understandable that in the feature screening process, the order of channels discarded in the feature maps of all users must be completely consistent. Otherwise, in the feature element-by-element fusion operation at the receiving end, due to inconsistent channel arrangement, errors may occur in the fusion results, affecting the performance of the model. In order to ensure feature alignment, users need to use the same sorting and screening criteria when screening feature channels, and uniformly determine the order of channels to be retained and discarded.

[0045] Figure 5 is a schematic diagram of feature importance feedback provided by the present invention, such as Figure 5 As shown in the figure, in order to ensure that the transmitters of all users can continuously share the semantic importance scores of each feature channel of each other, it is assumed that the transmitter of each user transmits the semantic importance scores of all its channel features to the receiver to ensure that all users can continuously share each other's semantic importance information. At the same time, the receiver continuously feeds back the feature importance scores of all users to the transmitter. Considering that the features change relatively slowly during the transmission process of continuous time, it can be reasonably inferred that the feedback results received in the previous time slot can be approximately applied to the current time slot.

[0046] In the embodiment of the present invention, the front The time complexity of calculating the optimal correlation and minimum repetition value of features is For the remaining features, try to replace them one by one. Second outer loop; processing of each feature requires traversing the selected features and calculate the optimal association and minimum repetition value in turn. The time complexity of the inner loop is Therefore, the total time complexity of the feature screening part is The feature selection algorithm implemented by the incremental search method (adding one feature at a time) will perform The first feature selection directly selects the feature with the highest importance score, and the complexity is ; in the subsequent In each selection, we need to traverse the remaining features and calculate the cosine similarity with the currently selected feature. The cosine similarity calculation involves traversing all currently selected features, so the time complexity of each feature selection is The whole feature screening process is carried out times of selection, so the total time complexity of screening is In comparison, the algorithm provided by the embodiment of the present invention has lower computational complexity than this algorithm.

[0047] The task-oriented millimeter wave link adaptive multimodal semantic communication method in the autonomous driving scenario provided by the embodiment of the present invention continuously optimizes the feature set through the feature screening method of exchange matching, ensuring that the feature set finally transmitted has both strong correlation and minimizes redundancy, thereby improving transmission efficiency and model redundancy.

[0048] In an optional embodiment, the semantic importance score of each of the channel features is calculated based on the following formula: ; in, Indicates The semantic importance score of the channel feature, Indicates the number of transmitters, Indicates The transmitter The semantic importance scores of channel features.

[0049] In the embodiment of the present invention, it is assumed that User's The semantic importance of a channel is , averaging the semantic importance scores of all users, then the first The semantic importance of a channel is expressed as .

[0050] In an optional embodiment, the semantic score is calculated based on the following formula: ; in, represents the semantic score, represents the channel feature set, Represents channel characteristics and channel characteristics The similarity of Represents the semantic importance score of each channel feature, and is the weight coefficient.

[0051] In the embodiment of the present invention, the optimal association and minimum repetition value (OAMR) of the semantic feature set is defined as the difference between the average importance score and the average similarity score of all features.

[0052] Optionally, by adjusting the loss function relative to the last layer feature vector output by the encoder The gradient of is globally averaged and pooled to represent the semantic importance of each channel feature. .

[0053] Optionally, the similarity score between two features is measured using cosine similarity.

[0054] The following simulation experiments are combined to verify the performance of the task-oriented millimeter wave link adaptive multimodal semantic communication method in the autonomous driving scenario provided by the present invention.

[0055] The simulation settings are that the base station is equipped with 64 antennas and 4 RF chains, the user end is equipped with 4 antennas and 2 RF chains, and the channel model uses the millimeter wave Saleh-Valenzuela channel. During the training and testing phases, the pixel value of each image is mapped to [0,1]. The Adam optimizer is used with an initial learning rate of 0.001. The model is trained for 200 epochs, and each image is trained only once in an epoch. The batch size is set to 9, and the loss function is the cross entropy loss. The training is performed at a signal-to-noise ratio of 0dB and tested in the range of -25dB to 20dB. Benchmark algorithm 1 is a multimodal semantic segmentation model without channel noise and has a compression rate of 0; benchmark algorithm 2 uses a traditional hybrid beamforming transmission module, in which both the analog precoding matrix and the analog merging matrix are solved by SVD decomposition, and the receiving end fuses the multimodal features with a compression rate of 0; benchmark algorithm 3 combines traditional analog beamforming with a jointly trained digital precoding transmission module, adopts a channel fusion algorithm, has no multi-user detection, and has a compression rate of 0; benchmark algorithm 4 uses a jointly trained hybrid beamforming transmission module, and uses a greedy algorithm to comprehensively consider semantic importance and semantic similarity for feature screening; benchmark algorithm 5 is a jointly trained hybrid beamforming transmission module, and uses a feature screening algorithm that does not consider semantic similarity.

[0056] Figure 6 is a schematic diagram showing the variation of the average intersection-to-parallel ratio with the signal-to-noise ratio provided by the present invention; Figure 7 It is a schematic diagram of the change of the average accuracy rate provided by the present invention with the signal-to-noise ratio; like Figure 6 and Figure 7 As shown, the performance of the algorithm with joint training hybrid beamforming provided by the present invention is close to that of the original model without a transmission module when the signal-to-noise ratio is greater than -5dB, and is much higher than that of the channel-level fusion algorithm.

[0057] Figure 8 is a schematic diagram of the variation of the average intersection-to-parallel ratio with the signal-to-noise ratio when the compression rate provided by the present invention is 0.5; Fig. 9 is a schematic diagram of the variation of the average intersection-to-parallel ratio with the signal-to-noise ratio when the compression rate provided by the present invention is 0.75; like Figure 8 and Fig. 9 As shown, the algorithm provided by the present invention has a performance similar to that of a greedy algorithm with higher complexity, and has a greatly improved performance compared to a compression method that only considers semantic importance.

[0058] Table 1 Visual prediction example table

[0059] As shown in Table 1 above, the two groups of pictures are street images taken during the day and at night, and the semantic segmentation results of various algorithms. The data set used in the experiment divides the pixels into nine categories, including unlabeled, cars, pedestrians, bicycles, curves, parking stations, guardrails, traffic cones and bumps. Through the annotation of the two categories of cars and pedestrians, it can be clearly seen that the segmentation effect of the algorithm provided by the present invention is similar to that of the multimodal semantic segmentation model MFNet without a transmission module, and the influence of random wireless channels is basically eliminated. Compared with benchmark algorithms 2 and 3, the algorithm provided by the present invention has a clearer segmentation boundary.

[0060] In summary, the present invention proposes a task-oriented hybrid beamforming transmission architecture suitable for millimeter wave communication, and optimizes the analog beamforming matrix through joint training, thereby improving the accuracy of image segmentation tasks. Based on the designed semantic communication system, a feature alignment and feature screening algorithm is also proposed. The algorithm averages the semantic importance scores of all users to ensure that the features screened by each modality can be aligned. The feature screening algorithm provides a feature screening method based on exchange matching based on comprehensive consideration of semantic importance and semantic similarity, ensuring that the system can adaptively screen the optimal features according to its own available bandwidth.

[0061] The following describes a task-oriented millimeter-wave link adaptive multimodal semantic communication system for an autonomous driving scenario provided by an embodiment of the present application. The task-oriented millimeter-wave link adaptive multimodal semantic communication system for an autonomous driving scenario described below and the task-oriented millimeter-wave link adaptive multimodal semantic communication method for an autonomous driving scenario described above can be referenced to each other.

[0062] Fig.10 is a schematic diagram of the structure of the task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario provided by the present invention, such as Fig.10 As shown, the task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario may include multiple transmitting ends 1010 and one receiving end 1020; Each of the transmitting ends 1010 is used to input a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, wherein the target image is any modal image of the multimodal images collected by the transmitting end 1010 through a sensing device; input the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, wherein the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; perform feature screening on the second feature map to obtain a third feature map; perform digital precoding and analog precoding on the third feature map to obtain a transmission signal matrix corresponding to the target image, and then send the transmission signal matrix corresponding to each target image to the receiving end 1020; The receiving end 1020 is used to perform analog merging and digital merging on multiple receiving signal matrices received from multiple transmitting ends 1010 to obtain multiple receiving feature maps; input each of the receiving feature maps into a joint source channel decoder to obtain a decoding feature corresponding to the output of the joint source channel decoder; fuse each of the decoding features to obtain a fused feature, and then input the fused feature into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder.

[0063] In an optional embodiment, the transmitting end 1010 is further used for: Expand the user's channel matrix into After the vector is obtained, the vector is input into the jointly trained deep neural network model to obtain the simulated precoding matrix output by the deep neural network model, where is the number of receiving antennas at the receiving end, is the number of transmitting antennas at the transmitting end; The pre-trained deep neural network model includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, the first linear layer and the second linear layer are connected to the first Dropout layer through a Leaky-Relu activation function, the second linear layer and the third linear layer are connected to the second Dropout layer through a Leaky-Relu activation function, the third linear layer and the fourth linear layer are connected to the third Dropout layer through a Leaky-Relu activation function, and the fourth linear layer and the fifth linear layer are connected to the fourth Dropout layer through a Leaky-Relu activation function.

[0064] In an optional embodiment, the number of channels of the second feature map is , the number of channels of the third feature map is ,in, , the transmitting end 1010 is specifically used for: Calculating a semantic importance score of each channel feature in the second feature map, where the semantic importance score represents the semantic importance of the channel feature; Based on the semantic importance scores of the channel features, the channel features are sorted and the top Channel features form the original semantic feature set; For any remaining feature, use the remaining feature to replace the channel feature in the original semantic feature set, and the remaining feature is a channel feature that does not belong to the original semantic feature set; If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is used as the original semantic feature set, and the semantic score represents the semantic importance and semantic repetition of the semantic feature set; If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, the original semantic feature set is kept unchanged; Return to the step of replacing the channel feature in the original semantic feature set with any remaining feature, until all the remaining features have been attempted to be replaced.

[0065] In an optional embodiment, the transmitting end 1010 calculates the semantic importance score of each channel feature based on the following formula: ; in, Indicates The semantic importance score of the channel feature, Indicates the number of transmitters, Indicates The transmitter The semantic importance scores of channel features.

[0066] In an optional embodiment, the transmitting end 1010 calculates the semantic score based on the following formula: ; in, represents the semantic score, represents the channel feature set, Represents channel characteristics and channel characteristics The similarity of Represents the semantic importance score of each channel feature, and is the weight coefficient.

[0067] It should be noted that the task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario provided by the embodiment of the present invention can, during specific operation, execute the task-oriented millimeter wave link adaptive multimodal semantic communication method in the autonomous driving scenario described in any of the above embodiments, which will not be elaborated in this embodiment.

[0068] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario, characterized in that: include: The transmitting end inputs the target image into the semantic encoder to obtain a first feature map output by the semantic encoder, wherein the target image is any modality image among the multimodal images collected by the transmitting end through the sensing device; The transmitting end inputs the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, wherein the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; The transmitting end performs feature screening on the second feature map to obtain a third feature map; The transmitting end performs digital precoding and analog precoding on the third feature map to obtain a transmission signal matrix corresponding to the target image, and then sends the transmission signal matrix corresponding to each target image to the receiving end; The receiving end performs analog merging and digital merging on a plurality of receiving signal matrices received from a plurality of transmitting ends to obtain a plurality of receiving characteristic graphs; The receiving end inputs each of the receiving feature graphs into a joint source channel decoder to obtain a decoding feature outputted by the joint source channel decoder; The receiving end performs fusion processing on each of the decoding features to obtain a fused feature, and then inputs the fused feature into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder.

2. The task-oriented millimeter wave link adaptive multimodal semantic communication method in the autonomous driving scenario according to claim 1, characterized in that: The method further comprises: The transmitting end expands the user's channel matrix into After the vector is obtained, the vector is input into the jointly trained deep neural network model to obtain the simulated precoding matrix output by the deep neural network model, where is the number of receiving antennas at the receiving end, is the number of transmitting antennas at the transmitting end; The jointly trained deep neural network model includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, wherein the first linear layer and the second linear layer are connected to the first Dropout layer through a Leaky-Relu activation function, the second linear layer and the third linear layer are connected to the second Dropout layer through a Leaky-Relu activation function, the third linear layer and the fourth linear layer are connected to the third Dropout layer through a Leaky-Relu activation function, and the fourth linear layer and the fifth linear layer are connected to the fourth Dropout layer through a Leaky-Relu activation function.

3. The task-oriented millimeter wave link adaptive multimodal semantic communication method in the autonomous driving scenario according to claim 1, characterized in that: The number of channels of the second feature map is , the number of channels of the third feature map is ,in, , the transmitting end performs feature screening on the second feature map to obtain a third feature map, including: Calculating a semantic importance score of each channel feature in the second feature map, where the semantic importance score represents the semantic importance of the channel feature; Based on the semantic importance scores of the channel features, the channel features are sorted and the top Channel features form the original semantic feature set; For any remaining feature, use the remaining feature to replace the channel feature in the original semantic feature set, and the remaining feature is a channel feature that does not belong to the original semantic feature set; If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is used as the original semantic feature set, and the semantic score represents the semantic importance and semantic repetition of the semantic feature set; If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, the original semantic feature set is kept unchanged; Return to the step of replacing the channel feature in the original semantic feature set with any remaining feature, until all the remaining features have been attempted to be replaced.

4. The task-oriented millimeter wave link adaptive multimodal semantic communication method in the autonomous driving scenario according to claim 3, characterized in that: The semantic importance score of each channel feature is calculated based on the following formula: ； in, Indicates The semantic importance score of the channel feature, Indicates the number of transmitters, Indicates The transmitter The semantic importance scores of channel features.

5. The task-oriented millimeter wave link adaptive multimodal semantic communication method in the autonomous driving scenario according to claim 3 or 4, characterized in that: The semantic score is calculated based on the following formula: ； in, represents the semantic score, represents the channel feature set, Represents channel characteristics and channel characteristics The similarity of Represents the semantic importance score of each channel feature, and is the weight coefficient.

6. A task-oriented millimeter wave link adaptive multimodal semantic communication system in an autonomous driving scenario, characterized in that: It includes a plurality of transmitting ends and a receiving end; Each of the transmitting ends is used to input a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, wherein the target image is any modal image of the multimodal images collected by the transmitting end through a sensing device; input the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, wherein the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; perform feature screening on the second feature map to obtain a third feature map; perform digital precoding and analog precoding on the third feature map to obtain a transmission signal matrix corresponding to the target image, and then send the transmission signal matrix corresponding to each target image to a receiving end; The receiving end is used to perform analog merging and digital merging on multiple receiving signal matrices received from multiple transmitting ends to obtain multiple receiving characteristic graphs; each of the receiving characteristic graphs is input into a joint source channel decoder to obtain a decoding feature outputted by the joint source channel decoder; The decoding features are fused to obtain fused features, and then the fused features are input into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder.

7. The task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario according to claim 6, characterized in that: The transmitting end is also used for: Expand the user's channel matrix into After the vector is obtained, the vector is input into the jointly trained deep neural network model to obtain the simulated precoding matrix output by the deep neural network model, where is the number of receiving antennas at the receiving end, is the number of transmitting antennas at the transmitting end; The pre-trained deep neural network model includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, the first linear layer and the second linear layer are connected to the first Dropout layer through a Leaky-Relu activation function, the second linear layer and the third linear layer are connected to the second Dropout layer through a Leaky-Relu activation function, the third linear layer and the fourth linear layer are connected to the third Dropout layer through a Leaky-Relu activation function, and the fourth linear layer and the fifth linear layer are connected to the fourth Dropout layer through a Leaky-Relu activation function.

8. The task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario according to claim 6, characterized in that: The number of channels of the second feature map is , the number of channels of the third feature map is ,in, , the transmitting end is specifically used for: Calculating a semantic importance score of each channel feature in the second feature map, where the semantic importance score represents the semantic importance of the channel feature; Based on the semantic importance scores of the channel features, the channel features are sorted and the top Channel features constitute the original semantic feature set; For any remaining feature, use the remaining feature to replace the channel feature in the original semantic feature set, and the remaining feature is a channel feature that does not belong to the original semantic feature set; If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is used as the original semantic feature set, and the semantic score represents the semantic importance and semantic repetition of the semantic feature set; If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, the original semantic feature set is kept unchanged; Return to the step of replacing the channel feature in the original semantic feature set with any remaining feature, until all the remaining features have been attempted to be replaced.

9. The task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario according to claim 8, characterized in that: The transmitter calculates the semantic importance score of each channel feature based on the following formula: ； in, Indicates The semantic importance score of the channel feature, Indicates the number of transmitters, Indicates The transmitter The semantic importance scores of channel features.

10. The task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario according to claim 8 or 9, characterized in that: The transmitter calculates the semantic score based on the following formula: ； in, represents the semantic score, represents the channel feature set, Represents channel characteristics and channel characteristics The similarity of Represents the semantic importance score of each channel feature, and is the weight coefficient.

Images