Task-oriented millimeter wave link adaptation multi-modal semantic communication method under automatic driving scenario

Abstract

The application provides a task-oriented millimeter wave link adaptive multi-modal semantic communication method and device in an automatic driving scene, electronic equipment and a storage medium, and belongs to the technical field of automatic driving. The method comprises the following steps: inputting a target image into a semantic encoder to obtain a first feature map, and then inputting the first feature map into a joint source channel encoder to obtain a second feature map; performing feature screening on the second feature map to obtain a third feature map, and then performing digital precoding and analog precoding on the third feature map to obtain a corresponding sending signal matrix; and then sending the sending signal matrix to a receiving end; performing analog combining and digital combining on the receiving signal matrix to obtain a receiving feature map, and then inputting the receiving feature map into a joint source channel decoder to obtain decoded features; performing fusion processing on the decoded features to obtain fused features, and then inputting the fused features into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder. The application improves the robustness and accuracy of semantic communication.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a task-oriented millimeter wave link adaptive multi-modal semantic communication method in an automatic driving scene. BACKGROUND

[0002] In the automatic driving technology, environment perception, accurate positioning, information interaction and decision control are its core components. Among them, the perception and understanding of the surrounding environment by the vehicle are the basis for realizing automatic driving. However, the perception range and computing capacity of a single vehicle are limited and are easily affected by occlusion, which is inconsistent with the demand for high-quality and all-around perception of automatic driving, and it is difficult to ensure truly effective and safe automatic driving. Therefore, it is particularly important for multiple vehicles to share environment perception data.

[0003] Semantic communication technology can significantly reduce data transmission by extracting and transmitting key semantic information, providing a new solution for the automatic driving scene. Current research has covered a vehicle-mounted semantic communication system for single-modal image segmentation, considering that in a single-link V2V scene, the image is compressed by an encoder, transmitted through a wireless channel, and a semantic segmentation map is generated at the receiving end through a decoder. However, the vehicle-mounted semantic communication system for single-modal image segmentation only perceives the environment and interacts information according to one kind of modal image, and depends on the lighting conditions. In different time periods and weather conditions, the quality of the image will greatly affect the accuracy of semantic segmentation.

[0004] Therefore, how to ensure the accuracy of semantic segmentation has become a technical problem to be solved. SUMMARY

[0005] The present application provides a task-oriented millimeter wave link adaptive multi-modal semantic communication method in an automatic driving scene, which solves the defect that the accuracy of single-modal image segmentation is affected by lighting conditions in the prior art.

[0006] The present application provides a task-oriented millimeter wave link adaptive multi-modal semantic communication method in an automatic driving scene, comprising the following steps:

[0007] The transmitting end inputs a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, the target image being any one of the multi-modal images collected by the transmitting end through a perception device;

[0008] The transmitting end inputs the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, the joint source channel encoder being used for further extracting semantic information, and the joint source channel encoder being obtained by joint training with the semantic encoder;

[0009] The transmitting end performs feature screening on the second feature map to obtain a third feature map;

[0010] The transmitting end performs digital precoding and analog precoding on the third feature map to obtain a sending signal matrix corresponding to the target image, and then sends the sending signal matrix corresponding to each target image to the receiving end;

[0011] The receiving end respectively performs analog merging and digital merging on the plurality of matrices received from the plurality of transmitting ends to obtain a plurality of receiving feature maps;

[0012] The receiving end respectively inputs each of the merged matrices into a joint source channel decoder to obtain a decoding feature output by the joint source channel decoder;

[0013] The receiving end performs fusion processing on each of the decoding features to obtain a fusion feature, and then inputs the fusion feature into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder.

[0014] According to the task-oriented millimeter wave link adaptive multi-modal semantic communication method in an automatic driving scene provided by the application, the method further comprises:

[0015] The transmitting end expands the channel matrix of the user into a vector, and then inputs the vector into a jointly trained deep neural network model to obtain an analog precoding matrix output by the deep neural network model, wherein, is the number of receiving antennas of the receiving end, is the number of transmitting antennas of the transmitting end; The jointly trained deep neural network model comprises a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, the first linear layer and the second linear layer are connected through a Leaky-Relu activation function and a first Dropout layer, the second linear layer and the third linear layer are connected through a Leaky-Relu activation function and a second Dropout layer, the third linear layer and the fourth linear layer are connected through a Leaky-Relu activation function and a third Dropout layer, and the fourth linear layer and the fifth linear layer are connected through a Leaky-Relu activation function and a fourth Dropout layer.

[0016] According to the task-oriented millimeter wave link adaptive multi-modal semantic communication method in an automatic driving scene provided by the application, the number of channels of the second feature map is

[0017] , and the number of channels of the third feature map is , wherein, ​The transmitting end performs feature screening on the second feature map to obtain a third feature map, including:

[0018] The semantic importance score of each channel feature in the second feature map is calculated, and the semantic importance score represents the semantic importance of the channel feature.

[0019] Based on the semantic importance score of each channel feature, each channel feature is sorted, and the first K channel features are selected to form an original semantic feature set.

[0020] For any remaining feature, the remaining feature is used to replace the channel feature in the original semantic feature set, and the remaining feature is a channel feature that does not belong to the original semantic feature set.

[0021] If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is used as the original semantic feature set, and the semantic score represents the semantic importance and semantic redundancy of the semantic feature set.

[0022] If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, the original semantic feature set is kept unchanged.

[0023] Return to the step of replacing the channel feature in the original semantic feature set with the remaining feature until all the remaining features are replaced.

[0024] According to the task-oriented millimeter wave link adaptive multi-modal semantic communication method in an automatic driving scene provided by the application, the semantic importance score of each channel feature is calculated based on the following formula:

[0025]

[0026] Wherein, represents the semantic importance score of the i-th channel feature, represents the number of transmitting ends, represents the semantic importance score of the i-th channel feature of the j-th transmitting end.

[0027] According to the task-oriented millimeter wave link adaptive multi-modal semantic communication method in an automatic driving scene provided by the application, the semantic score is calculated based on the following formula:

[0028]

[0029] Wherein,​​​​​​ representing a semantic score, representing a set of channel features, representing a channel feature and a channel feature a similarity of, representing a semantic importance score of each channel feature, and is a weight coefficient.

[0030] According to the automatic driving scene task-oriented millimeter wave link adaptive multi-modal semantic communication method provided by the application, each transmitting end continuously shares the semantic importance score of each feature channel of each other based on the following steps:

[0031] Each transmitting end transmits the semantic importance score of all its feature channels to the receiving end;

[0032] The receiving end continuously feeds back the semantic importance score of all transmitting ends to each transmitting end.

[0033] The application also provides an automatic driving scene task-oriented millimeter wave link adaptive multi-modal semantic communication system, comprising a plurality of transmitting ends and a receiving end;

[0034] Each transmitting end is used for inputting a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, the target image being any one of a plurality of modal images collected by the transmitting end through a perception device; the first feature map is input into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, the joint source channel encoder being used for further extracting semantic information, and the joint source channel encoder being obtained by joint training with the semantic encoder; the second feature map is subjected to feature screening to obtain a third feature map; the third feature map is subjected to digital precoding and analog precoding to obtain a transmitting signal matrix corresponding to the target image, and the transmitting signal matrix corresponding to each target image is transmitted to the receiving end;

[0035] The receiving end is used for respectively performing analog and digital merging on a plurality of received signal matrices received from a plurality of transmitting ends to obtain a plurality of received feature maps; each received feature map is input into a joint source channel decoder to obtain a decoding feature corresponding to the output of the joint source channel decoder; after the fusion feature is obtained by fusing each decoding feature, the fusion feature is input into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder.

[0036] According to the automatic driving scene task-oriented millimeter wave link adaptive multi-modal semantic communication system provided by the application, the transmitting end is also used for:

[0037] The channel matrix of the user is unfolded into a vector , and the vector is input into a jointly trained deep neural network model to obtain a simulated precoding matrix output by the deep neural network model, wherein is the number of receiving antennas of the receiving end, is the number of transmitting antennas of the transmitting end;

[0038] The pre-trained deep neural network model comprises a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, the first linear layer and the second linear layer are connected through a Leaky-Relu activation function and a first Dropout layer, the second linear layer and the third linear layer are connected through a Leaky-Relu activation function and a second Dropout layer, the third linear layer and the fourth linear layer are connected through a Leaky-Relu activation function and a third Dropout layer, and the fourth linear layer and the fifth linear layer are connected through a Leaky-Relu activation function and a fourth Dropout layer.

[0039] According to the task-oriented millimeter wave link adaptive multi-modal semantic communication system in the automatic driving scene provided by the application, the number of channels of the second feature map is , and the number of channels of the third feature map is , wherein The transmitting end is specifically configured to:

[0040] Calculate the semantic importance score of each channel feature in the second feature map, and the semantic importance score represents the semantic importance of the channel feature;

[0041] Sort each channel feature based on the semantic importance score of each channel feature, and select the first channel features to form an original semantic feature set;

[0042] For any remaining feature, replace the channel feature in the original semantic feature set with the remaining feature, and the remaining feature is a channel feature that does not belong to the original semantic feature set;

[0043] If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is taken as the original semantic feature set, and the semantic score represents the semantic importance and semantic redundancy of the semantic feature set;

[0044] If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, the original semantic feature set remains unchanged;

[0045] Returning to the step of replacing the channel feature in the original semantic feature set with any remaining feature, until all remaining features are attempted to be replaced.

[0046] According to the task-oriented millimeter wave link adaptive multi-modal semantic communication system in the autonomous driving scene provided by the application, the transmitting end calculates the semantic importance score of each channel feature based on the following formula:

[0047] ;

[0048] wherein, the semantic importance score of the i-th channel feature, denotes the number of transmitting ends, the semantic importance score of the i-th channel feature of the j-th transmitting end, denotes the number of channel features, denotes the semantic importance score of the i-th channel feature of the j-th transmitting end. According to the task-oriented millimeter wave link adaptive multi-modal semantic communication system in the autonomous driving scene provided by the application, the transmitting end calculates the semantic score based on the following formula:

[0049]

[0050] ; wherein,

[0051] the semantic score, denotes the channel feature set, denotes the similarity of the channel feature and the channel feature , the semantic importance score of each channel feature, and are weight coefficients. According to the task-oriented millimeter wave link adaptive multi-modal semantic communication system in the autonomous driving scene provided by the application, each transmitting end continuously shares the semantic importance score of each feature channel of each other based on the following steps:

[0052] Each transmitting end transmits the semantic importance score of all its feature channels to the receiving end;

[0053] The receiving end continuously feeds back the semantic importance scores of all transmitting ends to each transmitting end.

[0054] The receiving end continuously feeds back the semantic importance scores of all transmitting ends to each transmitting end.

[0055] ​The application provides a task-oriented millimeter wave link adaptive multi-modal semantic communication method in an automatic driving scene, a target image is input into a semantic encoder at a transmitting end, a first feature map output by the semantic encoder is obtained, the target image is any one of multi-modal images collected by a perception device at the transmitting end; the first feature map is input into a joint source channel encoder at the transmitting end, a second feature map output by the joint source channel encoder is obtained, the joint source channel encoder is used for further extracting semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; feature screening is performed on the second feature map at the transmitting end, and a third feature map is obtained; digital precoding and analog precoding are performed on the third feature map at the transmitting end, a sending signal matrix corresponding to the target image is obtained, and the sending signal matrix corresponding to each target image is sent to a receiving end; analog merging and digital merging are respectively performed on a plurality of receiving signal matrices received from a plurality of transmitting ends at the receiving end, and a plurality of receiving feature maps are obtained; the receiving end inputs each receiving feature map into a joint source channel decoder, and obtains a decoding feature output by the joint source channel decoder; the receiving end performs fusion processing on each decoding feature, obtains a fusion feature, inputs the fusion feature into a semantic decoder, and obtains a semantic segmentation result output by the semantic decoder. The application fuses multi-modal image features, can cope with complex and variable driving environments, and thus improves the robustness and accuracy of semantic communication. In addition, the joint source channel encoder and the semantic encoder are jointly trained, the accuracy of the image segmentation task is further improved, and the overall performance of the semantic communication system is improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0057] Figure 1 is a flowchart of the task-oriented millimeter wave link adaptive multi-modal semantic communication method in the automatic driving scene provided by the application;

[0058] Figure 2 is a multi-user multi-modal segmentation task scene schematic diagram for semantic communication provided by the application;

[0059] Figure 3 is a multi-modal semantic communication system model structure schematic diagram based on hybrid beamforming provided by the application;

[0060] Figure 4is a schematic diagram of a DNN network structure for training an analog beamforming matrix;

[0061] Figure 5 is a schematic diagram of feature importance feedback provided by the present application;

[0062] Figure 6 is a schematic diagram of the change of average intersection over union with signal-to-noise ratio provided by the present application;

[0063] Figure 7 is a schematic diagram of the change of average accuracy with signal-to-noise ratio provided by the present application;

[0064] Figure 8 is a schematic diagram of the change of average intersection over union with signal-to-noise ratio when the compression rate is 0.5 provided by the present application;

[0065] Figure 9 is a schematic diagram of the change of average intersection over union with signal-to-noise ratio when the compression rate is 0.75 provided by the present application;

[0066] Figure 10 is a structural schematic diagram of a task-oriented millimeter wave link adaptive multi-modal semantic communication system in an autonomous driving scenario provided by the present application. DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0068] It should be noted that in the description of the embodiments of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element. The terms "upper", "lower" and the like indicate the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication between two elements inside. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0069] The terms "first", "second", and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class, and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in a "or" relationship.

[0070] In the Internet of Vehicles environment, traffic scene understanding is one of the key technologies to realize automatic driving, and image segmentation is the core task of scene understanding. An automatic driving vehicle can use a semantic segmentation map to complete important tasks such as vehicle perception, behavior prediction, pedestrian identification, and path planning. However, the amount of data generated by visual sensors is huge, and existing vehicle communication technologies are difficult to meet the demand for high-speed and high-reliability transmission.

[0071] Semantic communication technology can significantly reduce data transmission by extracting and transmitting key semantic information, providing a new solution for autonomous driving scenarios. Current research has covered vehicle-mounted semantic communication systems for single-modal image segmentation, considering that in a single-link V2V scenario, images are compressed by an encoder, transmitted through a wireless channel, and a semantic segmentation map is generated at the receiving end by a decoder. In addition, the semantic communication system for multi-modal image segmentation considers combining two modalities of visible light images and infrared images, respectively encoding and compressing, and fusing through the channel. At the same time, the design of semantic precoding technology can preserve as much semantic information as possible when fusing information at the channel level.

[0072] However, in a multi-user, multi-modal semantic communication network, the available bandwidth of different users differs greatly, which requires an adaptive feature selection algorithm, and ensures that the selected multi-modal features are aligned. In addition, millimeter wave communication technology provides greater bandwidth resources and higher data transmission efficiency, but the existing multi-modal semantic communication system design has not fully combined millimeter wave communication technology. Therefore, for the multi-user semantic communication scenario of the image segmentation task, there are two main challenges: 1) how to realize adaptive selection of multi-modal features and ensure feature alignment; 2) how to design a task-oriented hybrid beamforming scheme suitable for millimeter wave semantic communication.

[0073] To solve the above problems, the present application provides the following embodiments. The following describes the task-oriented millimeter wave link adaptive multi-modal semantic communication method and system provided by the embodiments of the present application. Figures 1-10 The task-oriented millimeter wave link adaptive multi-modal semantic communication method and system provided by the embodiments of the present application are described.

[0074] Figure 1 is a flowchart of the task-oriented millimeter wave link adaptive multi-modal semantic communication method provided by the present application, as Figure 1 shown, the method comprises the following steps:

[0075] S110, the transmitting end inputs a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, the target image being any one of the multi-modal images collected by the transmitting end through a perception device;

[0076] S120, the transmitting end inputs the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, the joint source channel encoder being used for further extracting semantic information, and the joint source channel encoder being obtained by joint training with the semantic encoder;

[0077] S130, the transmitting end performs feature selection on the second feature map to obtain a third feature map;

[0078] S140, the transmitter performs digital precoding and analog precoding on the third feature map to obtain a transmission signal matrix corresponding to the target image, and then transmits the transmission signal matrix corresponding to each target image to the receiver;

[0079] S150, the receiver performs analog merging and digital merging on the multiple received signal matrices received from the multiple transmitters respectively to obtain multiple received feature maps;

[0080] S160, the receiver inputs each of the received features into a joint source channel decoder to obtain a decoded feature output by the joint source channel decoder;

[0081] S170, the receiver performs fusion processing on each of the decoded features to obtain a fusion feature, and then inputs the fusion feature into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder.

[0082] Figure 2 is a multi-user multi-modal segmentation task scenario schematic diagram for semantic communication provided by the present application, as Figure 2 shown, in an automatic driving environment, V2V assisted cooperative perception can be abstracted as a process of transmitting perception information to a single user (target vehicle) and extracting semantics by multiple users (vehicles). Specifically, vehicles collect image data through perception devices, and compress the images locally through encoders. The compressed image data is transmitted to the target vehicle through the wireless channel. At the receiving end, the target vehicle decodes the received data using a decoder and generates a corresponding semantic segmentation map, thereby realizing cooperative perception of the surrounding environment.

[0083] Figure 3 is a multi-modal semantic communication system model structure schematic diagram based on hybrid beamforming provided by the present application, as Figure 3 shown, considering a multi-user hybrid beamforming MIMO system, the transmitter includes a semantic encoder, a joint source channel encoder, a feature screening, a digital precoding and an analog precoding, and the receiver includes an analog merging, a digital merging, a joint source channel decoder, a fusion module and a semantic decoder.

[0084] In S110, the multi-modal image can include but is not limited to visible light image, infrared image, radar image, etc. As Figure 3 shown, the image semantic encoder of the transmitter uses a convolutional neural network to extract features from the input image , which can be represented as , where is a semantic extraction network with trainable parameters .

[0085] In S120, the joint source channel encoder further extracts semantic information, which is expressed as ,in It has trainable parameters Joint source-channel encoder network.

[0086] In S130, the second feature map is compressed to ,in Represents a feature screening operation, Indicates the semantic compression ratio.

[0087] In S150, each user at the transmitting end is equipped with Root transmitting antenna, RF chains, the receiving end is equipped with Root receiving antenna, RF chains. The analog precoding matrix at the transmitter is denoted as , the digital precoding matrix is ​​recorded as ,in is the number of data streams, and the simulation merging matrix is ​​recorded as , the digital merge matrix is ​​recorded as ,in Received by the receiving end User (transmitter) data streams, the channel matrix is ​​recorded as , additive Gaussian white noise is recorded as , the signal received by the receiving end can be expressed as:

[0088] .

[0089] In S160, the information is semantically restored by the joint source-channel decoder, which can be expressed as ,in It has trainable parameters Joint source-channel decoder network.

[0090] In S170, the features of different modalities are fused element by element, and the semantic decoder outputs the semantic segmentation result, which is expressed as ,in It has trainable parameters Semantic decoder network.

[0091] In a specific implementation process, the feature fusion method may include but is not limited to addition, splicing, etc.

[0092] An embodiment of the present invention provides a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario. The transmitting end inputs a target image into a semantic encoder to obtain a first feature map output by the semantic encoder. The target image is any modal image in the multimodal images collected by the transmitting end through a perception device; the transmitting end inputs the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder. The joint source channel encoder is used to further extract semantic information. The joint source channel encoder is obtained by joint training with the semantic encoder; the transmitting end performs feature screening on the second feature map to obtain a third feature map. Feature map; the transmitting end performs digital precoding and analog precoding on the third feature map, obtains the transmission signal matrix corresponding to the target image, and then sends the transmission signal matrix corresponding to each target image to the receiving end; the receiving end performs analog merging and digital merging on the multiple reception signal matrices received from multiple transmitting ends, and obtains multiple reception feature maps; the receiving end inputs each of the reception feature maps into the joint source channel decoder, and obtains the decoding features corresponding to the output of the joint source channel decoder; the receiving end performs fusion processing on each of the decoding features, obtains the fusion features, and then inputs the fusion features into the semantic decoder, and obtains the semantic segmentation results output by the semantic decoder. The present invention fuses multimodal image features to cope with complex and changeable driving environments, thereby improving the robustness and accuracy of semantic communication; in addition, the joint source channel encoder and the semantic encoder are jointly trained to further improve the accuracy of the image segmentation task and enhance the overall performance of the semantic communication system.

[0093] In an optional embodiment, the method further includes:

[0094] The transmitting end expands the user's channel matrix into After the vector is obtained, the vector is input into the jointly trained deep neural network model to obtain the simulated precoding matrix output by the deep neural network model, where is the number of receiving antennas at the receiving end, is the number of transmitting antennas at the transmitting end;

[0095] The pre-trained deep neural network model comprises a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, the first linear layer is connected with the second linear layer through a Leaky-Relu activation function and a first Dropout layer, the second linear layer and the third linear layer are connected through a Leaky-Relu activation function and a second Dropout layer, the third linear layer and the fourth linear layer are connected through a Leaky-Relu activation function and a third Dropout layer, and the fourth linear layer and the fifth linear layer are connected through a Leaky-Relu activation function and a fourth Dropout layer.

[0096] In a conventional communication system, a computational analog beamforming matrix is usually computed to maximize the sum rate; while in the embodiment of the present application, the same loss function as the semantic segmentation network is adopted to solve the matrix through joint training. Specifically:

[0097] Figure 4 is a schematic diagram of a DNN network structure for training an analog beamforming matrix, as shown in Figure 4 , the channel matrix of a user is expanded into a vector , and input into a DNN network, as shown in Figure 3 , the DNN network is composed of five linear layers, and each of the first four linear layers contains a Leaky-Relu activation function and a Dropout layer, and the number of neurons of the five linear layers is 1024, 512, 256, 128 and , respectively. The real-valued phase of can be obtained, and the output phase is converted into a complex number form:

[0098] ;

[0099] Similarly, the product of the obtained analog precoding matrix and the channel matrix is expanded and input into the DNN network to obtain the real-valued phase of , and the output phase is converted into a complex number form to obtain the analog combining matrix:

[0100] ;

[0101] After analog beamforming, the signal received at the receiving end can be represented as:

[0102] ;

[0103] wherein is an equivalent channel matrix, = .

[0104] In the embodiment of the present application, the calculation of digital beamforming is similar to conventional communication, which is divided into two steps, the first step is to eliminate the interference between users, and the second step is to eliminate the interference between antennas within users. Finally, the signal received by the receiving end realizes multi-user detection, and the MIMO channel is decomposed into sub-channels that do not interfere with each other, and the signal received by the receiving end can be expressed as:

[0105] .

[0106] In an optional embodiment, the number of channels of the second feature map is , and the number of channels of the third feature map is , wherein The transmitting end performs feature screening on the second feature map to obtain a third feature map, including:

[0107] Calculate the semantic importance score of each channel feature in the second feature map, and the semantic importance score represents the semantic importance of the channel feature;

[0108] Based on the semantic importance score of each channel feature, sort each channel feature, and select the first channel features to form an original semantic feature set;

[0109] For any remaining feature, replace the channel feature in the original semantic feature set with the remaining feature, and the remaining feature is a channel feature that does not belong to the original semantic feature set;

[0110] If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is taken as the original semantic feature set, and the semantic score represents the semantic importance and semantic redundancy of the semantic feature set;

[0111] If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, the original semantic feature set remains unchanged;

[0112] Return to the step of replacing the channel feature in the original semantic feature set with the remaining feature for any remaining feature, until all remaining features are replaced.

[0113] Specifically, the available bandwidth of different systems is different, and adaptive feature screening is required. The number of channels of the feature map output by joint source channel coding is , and the number of channels of the feature map after feature screening is , wherein The goal of feature screening is to select a feature set with the highest semantic importance and the least semantic redundancy from a large number of features. All users use the same feature screening algorithm. First, the features with the highest semantic importance scores are selected and put into the semantic feature set to be transmitted. Next, for the remaining features, the features in the semantic feature set are replaced one by one to obtain a new feature set, and the optimal correlation and minimum redundancy of the new feature set are calculated. If the value of the new feature set improves, the iteration and exchange are continued. This process continues until all remaining features have been exchanged.

[0114] It can be understood that, in the feature screening process, the order of the discarded channel of the feature map of all users must be completely consistent, otherwise, in the element-by-element fusion operation of the features at the receiving end, due to the inconsistent arrangement of the channels, errors may occur in the fusion result, affecting the performance of the model. In order to ensure the alignment of the features, the same sorting and screening standard is used by the users when screening the channel of the features, and the order of the retained and discarded channels is uniformly determined.

[0115] Figure 5 is a schematic diagram of the feature importance feedback provided by the application, as shown in Figure 5 In order to ensure that the transmitters of all users can continuously share the semantic importance scores of each feature channel of each other, it is assumed that the transmitter of each user transmits the semantic importance scores of all channel features to the receiver, so that all users can continuously share the semantic importance information of each other. At the same time, the receiver continuously feeds back the feature importance scores of all users to the transmitter. Considering that the features change relatively gently in the transmission process in continuous time, it can be reasonably speculated that the feedback result received in the last time slot can be approximately applied to the current time slot.

[0116] In the embodiment of the application, the first features are selected and the optimal correlation and minimum redundancy of the selected features are calculated, and the time complexity is For the remaining features, each is tried to be replaced, and there are outer loops; the processing of each feature needs to traverse the selected features and calculate the optimal correlation and minimum redundancy one by one, and the time complexity of the inner loop is Therefore, the total time complexity of the feature screening part is The feature selection algorithm implemented by the incremental search method (adding one feature each time) will perform times of feature selection. The first feature selection directly selects the feature with the highest importance score, and the complexity is ; in the subsequent ​​​​In each selection, the remaining features and calculate the cosine similarity with the currently selected features. The cosine similarity calculation involves traversing all currently selected features, so the time complexity of each feature selection is The whole feature screening process is carried out times of selection, so the total time complexity of screening is In comparison, the algorithm provided by the embodiment of the present invention has lower computational complexity than this algorithm.

[0117] The embodiment of the present invention provides a task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario. The method continuously optimizes the feature set through an exchange-matching feature screening method to ensure that the feature set finally transmitted has both strong correlation and minimizes redundancy, thereby improving transmission efficiency and model redundancy.

[0118] In an optional embodiment, the semantic importance score of each channel feature is calculated based on the following formula:

[0119] ;

[0120] in, Indicates the The semantic importance score of the channel feature, Indicates the number of transmitters, Indicates the The first transmitter The semantic importance score of each channel feature.

[0121] In the embodiment of the present invention, it is assumed that User's The semantic importance of a channel is , average the semantic importance scores of all users, then the first The semantic importance of a channel is expressed as .

[0122] In an optional embodiment, the semantic score is calculated based on the following formula:

[0123] ;

[0124] in, represents the semantic score, represents the channel feature set, Represents channel characteristics and channel characteristics The similarity of represents the semantic importance score of each channel feature, and is a weight coefficient.

[0125] In the embodiments of the present application, the optimal association and minimal repetition (OAMR) of the semantic feature set is defined as the difference between the average importance score and the average similarity score of all features.

[0126] Optionally, the semantic importance value of each channel feature is represented by performing global average pooling on the gradient of the loss function with respect to the last layer feature vector of the encoder output. .

[0127] Optionally, the similarity score between two features is measured by cosine similarity.

[0128] The performance of the task-oriented millimeter wave link adaptive multi-modal semantic communication method in the autonomous driving scene provided by the present application is verified by the following simulation experiment.

[0129] The simulation setting is that the base station is equipped with 64 antennas and 4 radio frequency chains, and the user terminal is equipped with 4 antennas and 2 radio frequency chains, and the channel model adopts the millimeter wave Saleh-Valenzuela channel. In the training and testing stage, the pixel value of each image is mapped to [0, 1]. The Adam optimizer is used, and the initial learning rate is 0.001. The model is trained for 200 epochs, and each image is trained only once in an epoch. The batch size is set to 9, and the loss function is cross-entropy loss. It is trained at a signal-to-noise ratio of 0 dB and tested in the range of -25 dB to 20 dB. The benchmark algorithm 1 is a multi-modal semantic segmentation model without channel noise and a compression rate of 0; the benchmark algorithm 2 adopts a traditional hybrid beamforming transmission module, in which the analog precoding matrix and the analog combining matrix are solved by SVD decomposition, the receiving end fuses the multi-modal features, and the compression rate is 0; the benchmark algorithm 3 combines the traditional analog beamforming and the joint training digital precoding transmission module, adopts the channel fusion algorithm, has no multi-user detection, and the compression rate is 0; the benchmark algorithm 4 adopts the joint training hybrid beamforming transmission module, and performs feature screening by considering the semantic importance and semantic similarity through the greedy algorithm; and the benchmark algorithm 5 is the joint training hybrid beamforming transmission module, and adopts a feature screening algorithm without considering semantic similarity.

[0130] Figure 6 is a schematic diagram of the average intersection over union provided by the present application with the change of the signal-to-noise ratio;

[0131] Figure 7 is a schematic diagram of the average accuracy provided by the present application with the change of the signal-to-noise ratio; ​

[0132] As shown in Figure 6 and Figure 7 The algorithm provided by the present application has the performance of adding joint training mixed beamforming close to the original model without a transmission module when the signal-to-noise ratio is greater than -5dB, and is much higher than the algorithm of channel-level fusion.

[0133] Figure 8 is a schematic view of the average intersection-over-union ratio changing with the signal-to-noise ratio when the compression rate is 0.5;

[0134] Figure 9 is a schematic view of the average intersection-over-union ratio changing with the signal-to-noise ratio when the compression rate is 0.75;

[0135] As shown in Figure 8 and Figure 9 The algorithm provided by the present application has the performance close to the greedy algorithm with higher complexity, and has a great performance improvement compared with the compression method considering only semantic importance.

[0136] Table 1 visual prediction example table

[0137]

[0138] As shown in the above table 1, two groups of pictures are street shooting images in daytime and nighttime respectively, and semantic segmentation results of various algorithms. The data set used in the experiment has nine categories of pixel points, including unlabeled, car, pedestrian, bicycle, curve, parking station, guardrail, traffic cone and convex. Through the annotation of the two categories of car and pedestrian, it can be clearly seen that the segmentation effect of the algorithm provided by the present application is close to that of the multi-modal semantic segmentation model MFNet without a transmission module, and the influence of the random wireless channel is basically eliminated. Compared with the benchmark algorithms 2 and 3, the algorithm provided by the present application has clearer segmentation boundaries.

[0139] In summary, the present application proposes a task-oriented hybrid beamforming transmission architecture suitable for millimeter wave communication, and optimizes the analog beamforming matrix through joint training, thereby improving the accuracy of image segmentation tasks. On the basis of the designed semantic communication system, a feature alignment and feature screening algorithm is also proposed. The algorithm averages the semantic importance scores of all users to ensure that the features selected by each modality can be aligned. The feature screening algorithm provides a feature screening method based on exchange matching by comprehensively considering semantic importance and semantic similarity, ensuring that the system can adaptively select the optimal features according to its available bandwidth.

[0140] The following describes the task-oriented millimeter wave link adaptive multimodal semantic communication system for an autonomous driving scenario provided by an embodiment of the present application. The task-oriented millimeter wave link adaptive multimodal semantic communication system for an autonomous driving scenario described below and the task-oriented millimeter wave link adaptive multimodal semantic communication method for an autonomous driving scenario described above can be referenced to each other.

[0141] Figure 10 : is a schematic diagram of the structure of the task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario provided by the present invention, such as Figure 10 As shown, the task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario may include multiple transmitting terminals 1010 and one receiving terminal 1020;

[0142] Each transmitting end 1010 is used to input a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, where the target image is any modal image among the multimodal images collected by the transmitting end 1010 through a sensing device; input the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, where the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; perform feature screening on the second feature map to obtain a third feature map; perform digital precoding and analog precoding on the third feature map to obtain a transmission signal matrix corresponding to the target image, and then send the transmission signal matrix corresponding to each target image to the receiving end 1020;

[0143] The receiving end 1020 is used to perform analog merging and digital merging on multiple receiving signal matrices received from multiple transmitting ends 1010 to obtain multiple receiving feature maps; input each of the receiving feature maps into a joint source channel decoder to obtain the decoding features corresponding to the output of the joint source channel decoder; fuse each of the decoding features to obtain the fused features, and then input the fused features into a semantic decoder to obtain the semantic segmentation results output by the semantic decoder.

[0144] In an optional embodiment, the transmitting end 1010 is further configured to:

[0145] Expand the user's channel matrix into After the vector is obtained, the vector is input into the jointly trained deep neural network model to obtain the simulated precoding matrix output by the deep neural network model, where is the number of receiving antennas at the receiving end, is the number of transmitting antennas at the transmitting end;

[0146] The pre-trained deep neural network model comprises a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, the first linear layer is connected with the second linear layer through a Leaky-Relu activation function and a first Dropout layer, the second linear layer and the third linear layer are connected through a Leaky-Relu activation function and a second Dropout layer, the third linear layer and the fourth linear layer are connected through a Leaky-Relu activation function and a third Dropout layer, and the fourth linear layer and the fifth linear layer are connected through a Leaky-Relu activation function and a fourth Dropout layer.

[0147] In an optional embodiment, the number of channels of the second feature map is , and the number of channels of the third feature map is , wherein The transmitter 1010 is specifically configured to:

[0148] calculate a semantic importance score of each channel feature in the second feature map, the semantic importance score representing semantic importance of the channel feature;

[0149] sort each channel feature based on the semantic importance score of the channel feature, and select the first K channel features to form an original semantic feature set;

[0150] for any remaining feature, replace a channel feature in the original semantic feature set with the remaining feature, the remaining feature being a channel feature not belonging to the original semantic feature set;

[0151] if a semantic score of a new semantic feature set is higher than a semantic score of the original semantic feature set, the new semantic feature set is taken as the original semantic feature set, the semantic score representing semantic importance and semantic redundancy of a semantic feature set;

[0152] if the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, the original semantic feature set remains unchanged;

[0153] return to the step of replacing the channel feature in the original semantic feature set with the remaining feature, until all the remaining features are replaced.

[0154] In an optional embodiment, the transmitter 1010 calculates the semantic importance score of each channel feature based on the following formula:

[0155]

[0156] ​​wherein, denotes the semantic importance score of the i-th channel feature of the j-th transmitting end, denotes the number of transmitting ends, denotes the i-th channel feature of the j-th transmitting end, denotes the semantic importance score of the i-th channel feature of the j-th transmitting end. In an optional embodiment, the transmitting end 1010 calculates the semantic score based on the following formula:

[0157]

[0158] ;

[0159] wherein, denotes the semantic score, denotes the set of channel features, denotes the similarity of the channel feature and the channel feature , denotes the semantic importance score of each channel feature, and are weight coefficients.

[0160] It should be noted that the task-oriented millimeter wave link adaptive multi-modal semantic communication system in the autonomous driving scene provided by the embodiments of the present application can execute the task-oriented millimeter wave link adaptive multi-modal semantic communication method in the autonomous driving scene described in any of the above embodiments when actually running, and the present embodiment will not be described here.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and the necessary general hardware platform, and of course, it can also be realized by hardware. Based on such understanding, the above technical solutions or the essential part of the prior art can be embodied in the form of a software product, which can be stored in a computer readable storage medium such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in each embodiment or some part of the embodiment.

[0162] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.​​

Claims

1. A task-oriented millimeter wave link adaptive multimodal semantic communication method in an autonomous driving scenario, characterized by: include: The transmitting end inputs a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, wherein the target image is any modal image among the multimodal images collected by the transmitting end through a sensing device; The transmitting end inputs the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, wherein the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is jointly trained with the semantic encoder; The transmitting end performs feature screening on the second feature map to obtain a third feature map; The transmitting end performs digital precoding and analog precoding on the third feature map to obtain a transmission signal matrix corresponding to the target image, and then sends the transmission signal matrix corresponding to each target image to the receiving end; The receiving end performs analog merging and digital merging on multiple receiving signal matrices received from multiple transmitting ends to obtain multiple receiving characteristic graphs; The receiving end inputs each of the receiving feature maps into a joint source-channel decoder to obtain decoding features corresponding to the output of the joint source-channel decoder; The receiving end performs a fusion process on the decoding features to obtain a fusion feature, and then inputs the fusion feature into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder; The number of channels of the second feature map is , the number of channels of the third feature map is ,in, , the transmitting end performs feature screening on the second feature map to obtain a third feature map, including: Calculating a semantic importance score of each channel feature in the second feature map, where the semantic importance score represents the semantic importance of the channel feature; Based on the semantic importance score of each channel feature, the channel features are sorted and the top Channel features constitute the original semantic feature set; For any remaining feature, use the remaining feature to replace the channel feature in the original semantic feature set, where the remaining feature is a channel feature that does not belong to the original semantic feature set; If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is used as the original semantic feature set, and the semantic score represents the semantic importance and semantic repetition of the semantic feature set; If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, then the original semantic feature set remains unchanged; Return to the step of replacing the channel feature in the original semantic feature set with any remaining feature, until all the remaining features have been attempted to be replaced.

2. The task-oriented millimeter wave link adaptive multimodal semantic communication method in the autonomous driving scenario according to claim 1 is characterized in that: The method further comprises: The transmitting end expands the user's channel matrix into After the vector is obtained, the vector is input into the jointly trained deep neural network model to obtain the simulated precoding matrix output by the deep neural network model, where is the number of receiving antennas at the receiving end, is the number of transmitting antennas at the transmitting end; The jointly trained deep neural network model includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, wherein the first linear layer and the second linear layer are connected to the first Dropout layer through a Leaky-Relu activation function, the second linear layer and the third linear layer are connected to the second Dropout layer through a Leaky-Relu activation function, the third linear layer and the fourth linear layer are connected to the third Dropout layer through a Leaky-Relu activation function, and the fourth linear layer and the fifth linear layer are connected to the fourth Dropout layer through a Leaky-Relu activation function.

3. The task-oriented millimeter wave link adaptive multimodal semantic communication method in the autonomous driving scenario according to claim 1 is characterized in that: The semantic importance score of each channel feature is calculated based on the following formula: ； in, Indicates the The semantic importance score of the channel feature, Indicates the number of transmitters, Indicates the The first transmitter The semantic importance score of each channel feature.

4. The task-oriented millimeter wave link adaptive multimodal semantic communication method in the autonomous driving scenario according to claim 1 or 3, characterized in that: The semantic score is calculated based on the following formula: ； in, represents the semantic score, represents the channel feature set, Represents channel characteristics and channel characteristics The similarity of represents the semantic importance score of each channel feature, and is the weight coefficient.

5. A task-oriented millimeter wave link adaptive multimodal semantic communication system in an autonomous driving scenario, characterized by: It includes multiple transmitting ends and a receiving end; Each transmitting end is used to input a target image into a semantic encoder to obtain a first feature map output by the semantic encoder, where the target image is any modal image among the multimodal images collected by the transmitting end through a sensing device; input the first feature map into a joint source channel encoder to obtain a second feature map output by the joint source channel encoder, where the joint source channel encoder is used to further extract semantic information, and the joint source channel encoder is obtained by joint training with the semantic encoder; perform feature screening on the second feature map to obtain a third feature map; perform digital precoding and analog precoding on the third feature map to obtain a transmission signal matrix corresponding to the target image, and then send the transmission signal matrix corresponding to each target image to the receiving end; The receiving end is used to perform analog merging and digital merging on multiple receiving signal matrices received from multiple transmitting ends to obtain multiple receiving feature maps; each of the receiving feature maps is input into a joint source channel decoder to obtain a decoding feature corresponding to the output of the joint source channel decoder; Performing a fusion process on the decoding features to obtain a fused feature, and then inputting the fused feature into a semantic decoder to obtain a semantic segmentation result output by the semantic decoder; The number of channels of the second feature map is , the number of channels of the third feature map is ,in, , the transmitting end is specifically used for: Calculating a semantic importance score of each channel feature in the second feature map, where the semantic importance score represents the semantic importance of the channel feature; Based on the semantic importance score of each channel feature, the channel features are sorted and the top Channel features constitute the original semantic feature set; For any remaining feature, use the remaining feature to replace the channel feature in the original semantic feature set, where the remaining feature is a channel feature that does not belong to the original semantic feature set; If the semantic score of the new semantic feature set is higher than the semantic score of the original semantic feature set, the new semantic feature set is used as the original semantic feature set, and the semantic score represents the semantic importance and semantic repetition of the semantic feature set; If the semantic score of the new semantic feature set is less than or equal to the semantic score of the original semantic feature set, then the original semantic feature set remains unchanged; Return to the step of replacing the channel feature in the original semantic feature set with any remaining feature, until all the remaining features have been attempted to be replaced.

6. The task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario according to claim 5, characterized in that: The transmitting end is further used for: Expand the user's channel matrix into After the vector is obtained, the vector is input into the jointly trained deep neural network model to obtain the simulated precoding matrix output by the deep neural network model, where is the number of receiving antennas at the receiving end, is the number of transmitting antennas at the transmitting end; The jointly trained deep neural network model includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer and a fifth linear layer, wherein the first linear layer and the second linear layer are connected to the first Dropout layer through a Leaky-Relu activation function, the second linear layer and the third linear layer are connected to the second Dropout layer through a Leaky-Relu activation function, the third linear layer and the fourth linear layer are connected to the third Dropout layer through a Leaky-Relu activation function, and the fourth linear layer and the fifth linear layer are connected to the fourth Dropout layer through a Leaky-Relu activation function.

7. The task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario according to claim 5, characterized in that: The transmitter calculates the semantic importance score of each channel feature based on the following formula: ； in, Indicates the The semantic importance score of the channel feature, Indicates the number of transmitters, Indicates the The first transmitter The semantic importance score of each channel feature.

8. The task-oriented millimeter wave link adaptive multimodal semantic communication system in the autonomous driving scenario according to claim 5 or 7, characterized in that: The transmitter calculates the semantic score based on the following formula: ； in, represents the semantic score, represents the channel feature set, Represents channel characteristics and channel characteristics The similarity of represents the semantic importance score of each channel feature, and is the weight coefficient.

Images