Depth joint source channel coding method and related device

Abstract

The deep joint source channel coding method comprises the following steps: a sending end obtains a first semantic feature of a first image, and performs constellation mapping modulation on the first semantic feature to obtain a first initial constellation point; the sending end extracts semantic information of the first initial constellation point, generates a rotation angle corresponding to the semantic information, and rotates the first initial constellation point according to the rotation angle to obtain a first rotation constellation point; the sending end generates a real part interleaving matrix and an imaginary part interleaving matrix based on the first rotating constellation point and the real part and the imaginary part of the channel state information, resorts the real part and the imaginary part of the symbol sequence respectively and then recombines the real part and the imaginary part into a complex signal to obtain a first interleaving signal; the sending end transmits the first interleaved signal to a receiving end through a fading channel, and the receiving end receives a second interleaved signal; the receiving end performs semantic reconstruction decoding on the second interleaved signal to obtain a second image; or, the receiving end performs semantic classification decoding on the second interlaced signal to obtain a second image category.

Description

Technical Field

[0001] This disclosure relates to the field of communication technology, and in particular to a deep joint source-channel coding and decoding method based on semantic correlation interleaving and signal space diversity, applicable to semantic communication systems in wireless image transmission. Background Technology

[0002] With the rapid development of 6G wireless communication systems, semantic communication has attracted much attention due to its potential in data transmission efficiency and robustness. Compared to traditional communication systems that focus on bit-level lossless transmission, semantic communication significantly reduces redundant information by extracting and transmitting the core semantic content of data, making it suitable for high-dimensional data transmission (such as images and videos) in scenarios such as the Internet of Things, smart healthcare, and smart cities. In recent years, semantic communication research has made progress in optimizing the performance of anti-fading channels. For example, some studies have improved anti-fading capabilities by utilizing the importance of semantic symbols for channel estimation; others have proposed a Deep Joint Source-Channel Coding (DeepJSCC) scheme that combines Orthogonal Frequency Division Multiplexing (OFDM) and Channel State Information (CSI) feedback to optimize image transmission performance under multipath fading channels. These methods have improved the performance of semantic communication in complex channel environments to some extent.

[0003] Despite the progress made in resisting fading in existing semantic communication technologies, the following key issues still exist in practical applications:

[0004] Semantic symbol correlation is not fully utilized: Existing methods do not adequately consider the strong correlation between encoded semantic symbols. In fading channels, strongly correlated semantic symbols (such as features of adjacent pixels in an image) may be affected by deep fading simultaneously, leading to the loss of continuous semantic information and significantly reducing the performance of image reconstruction or classification at the receiver.

[0005] Insufficient adaptability of non-uniform constellation points: Constellation points generated by semantic encoders usually exhibit a non-uniform probability distribution. Traditional fixed-angle rotation signal space diversity (SSD) methods are difficult to adapt to this characteristic, resulting in limited transmission efficiency and anti-fading capability.

[0006] In addition, some existing methods are usually based on static coding strategies, which fail to effectively adapt to the real-time channel conditions. This results in large performance fluctuations of the system under different signal-to-noise ratio conditions, significant deterioration of image reconstruction quality, and difficulty in maintaining stable transmission quality in variable fading channels. Summary of the Invention

[0007] This disclosure provides an encoding method and related apparatus for intelligent simplified networks.

[0008] According to one aspect of this disclosure, a deep joint source-channel coding method is provided, comprising:

[0009] The sending end obtains the first semantic feature of the first image, and performs constellation mapping modulation on the first semantic feature to obtain the first initial constellation point;

[0010] The sending end extracts the semantic information of the first initial constellation point, generates a rotation angle corresponding to the semantic information, and rotates the first initial constellation point according to the rotation angle to obtain the first rotated constellation point;

[0011] The transmitting end generates a real part interleaving matrix and an imaginary part interleaving matrix based on the real part and imaginary part of the first rotating constellation point and the channel state information. The real part interleaving matrix is ​​used to reorder the real part of the symbol sequence of the first rotating constellation point, and the imaginary part interleaving matrix is ​​used to reorder the imaginary part of the symbol sequence of the first rotating constellation point. The real part and imaginary part of the symbol sequence are then recombined into a complex signal to obtain the first interleaved signal.

[0012] The transmitting end transmits the first interleaved signal to the receiving end through the fading channel, and the receiving end receives the second interleaved signal;

[0013] The receiving end performs semantic reconstruction decoding on the second interleaved signal to obtain the second image; or, the receiving end performs semantic classification decoding on the second interleaved signal to obtain the second image category.

[0014] Optionally, the sending end acquires the first semantic feature of the first image, including:

[0015] The first image is input into a deep neural network based on a convolutional neural network and a residual network to obtain the first semantic features of the first image.

[0016] Optionally, the step of performing constellation mapping modulation on the first semantic feature to obtain the first initial constellation point includes:

[0017] The first semantic feature is input into the constellation mapper, and the first semantic feature is mapped into a first structured tensor through a fully connected layer. The first structured tensor represents the unnormalized log probability distribution of each symbol.

[0018] Gumbell noise is added to the first structured tensor to achieve differentiable discrete sampling, and then a one-hot vector is generated; based on the weighted sum of the one-hot vector and the predefined constellation point values, a modulation output is generated; the modulation output is grouped by alternating indices and mapped to two-dimensional complex constellation points, i.e., the first initial constellation points, where even indices correspond to the real part of the signal and odd indices correspond to the imaginary part of the signal.

[0019] Optionally, the sending end extracts the semantic information of the first initial constellation point by: extracting the local spatial features and global structural features of the first initial constellation point sequence by alternately stacking three sets of one-dimensional convolutional layers and rectified linear unit pooling layers, and generating a feature sequence as the semantic information of the first initial constellation point;

[0020] The process of generating the rotation angle corresponding to the semantic information includes: setting the length of the convolutional output sequence to a first fixed length through an adaptive average pooling layer to generate a high-dimensional feature tensor, which is then processed by a fully connected layer, and the output is mapped to the [0,1] interval by combining a sigmoid activation function, and then adjusted to the rotation angle range of 0° to 90° through a linear transformation to obtain the rotation angle corresponding to the semantic information.

[0021] The step of rotating the first initial constellation point according to the rotation angle to obtain the first rotated constellation point includes: dynamically adjusting the generated rotation angle according to the importance of the semantic information carried by the first initial constellation point; and increasing the projection distance of the first initial constellation point carrying important semantic information in the I-axis and Q-axis directions through adaptive rotation to prioritize the protection of important semantic information.

[0022] Optionally, the transmitting end generates a real-part interleaving matrix and an imaginary-part interleaving matrix based on the real and imaginary parts of the first rotating constellation point and channel state information. The real-part interleaving matrix is ​​used to reorder the real parts of the symbol sequence of the first rotating constellation point, and the imaginary-part interleaving matrix is ​​used to reorder the imaginary parts of the symbol sequence of the first rotating constellation point. The real and imaginary parts of the symbol sequence are then recombine into a complex signal to obtain the first interleaved signal, including:

[0023] The correlation weight matrix between the real parts of the first rotating constellation points is calculated using a multi-head attention mechanism, and the correlation weight matrix between the imaginary parts of the first rotating constellation points is also calculated.

[0024] The real part correlation weight matrix is ​​concatenated with the real part of the amplitude feature of the channel state information to obtain the real part concatenated feature, and the imaginary part correlation weight matrix is ​​concatenated with the imaginary part of the amplitude feature of the channel state information to obtain the imaginary part concatenated feature.

[0025] The real part concatenation feature and the imaginary part concatenation feature are respectively input into a multilayer perceptron, which outputs a real part permutation probability matrix and an imaginary part permutation probability matrix respectively; the multilayer perceptron contains two fully connected layers;

[0026] Using the Gombell sigmoid function sampling technique, real part interleaving matrix and imaginary part interleaving matrix are generated from the real part permutation probability matrix and the imaginary part permutation probability matrix respectively through hard sampling; both the real part interleaving matrix and the imaginary part interleaving matrix are sparse matrices with only one 1 in each row and column and the rest being 0 values;

[0027] The real part of the first rotating constellation point is dynamically rearranged using the real part interleaving matrix, and the imaginary part of the first rotating constellation point is dynamically rearranged using the imaginary part interleaving matrix. The real and imaginary parts of the first rotating constellation point are then recombined into a complex signal to obtain the first interleaved signal.

[0028] Optionally, the step of calculating the real part correlation weight matrix between the real parts of the first rotating constellation points and the imaginary part correlation weight matrix between the imaginary parts of the first rotating constellation points using a multi-head attention mechanism includes: projecting the real and imaginary parts of the first rotating constellation points into query vectors, key vectors, and value vectors, respectively; calculating the attention scores of the real and imaginary parts using a scaled dot product attention mechanism based on the query vectors, key vectors, and value vectors; and normalizing the attention scores using a normalized exponential function to generate the real part correlation weight matrix and the imaginary part correlation weight matrix.

[0029] Optionally, the fading channel includes any of the following: Rayleigh fading channel and Rice fading channel.

[0030] Optionally, the receiving end performs semantic reconstruction decoding on the second interleaved signal to obtain a second image, including:

[0031] The second interleaved signal is input into a multi-layer convolutional neural network structure, which integrates residual connection mechanism and spatial resolution restoration operation. The spatial structure of the original image is gradually reconstructed through stepwise feature enhancement and nonlinear transformation. Gradient flow is optimized to improve reconstruction accuracy. Finally, a second image matching the size of the original image is generated through multi-stage resolution upsampling operation.

[0032] Optionally, the receiving end performs semantic classification decoding on the second interleaved signal to obtain a second image category, including:

[0033] The second interleaved signal is input into a multi-layer fully connected network based on a spine network architecture. Hierarchical feature fusion is implemented through progressive input. Multi-stage linear processing combined with dropout regularization is used to prevent overfitting, progressively optimizing feature representation capabilities and outputting a multi-class probability distribution. The class with the highest probability is taken as the second image class.

[0034] Optionally, the receiving end performs semantic reconstruction decoding on the second interleaved signal to obtain a second image, including:

[0035] The second interleaved signal is deinterleaved for semantic correlation to obtain the second constellation point; the second constellation point is then semantically reconstructed and decoded to obtain the second image.

[0036] Optionally, the receiving end performs semantic classification decoding on the second interleaved signal to obtain a second image category, including:

[0037] The second interleaved signal is deinterleaved based on semantic correlation to obtain the second constellation point; the second constellation point is then semantically classified and decoded to obtain the second image category.

[0038] According to another aspect of this disclosure, a deep joint source-channel coding apparatus is provided, characterized in that it comprises:

[0039] The constellation point generation module obtains the first semantic features of the first image from the sending end, and performs constellation mapping modulation on the first semantic features to obtain the first initial constellation points;

[0040] The adaptive angle rotation module extracts the semantic information of the first initial constellation point from the transmitting end, generates a rotation angle corresponding to the semantic information, and rotates the first initial constellation point according to the rotation angle to obtain the first rotated constellation point.

[0041] The semantic correlation interleaving module generates a real part interleaving matrix and an imaginary part interleaving matrix based on the real and imaginary parts of the first rotating constellation point and the channel state information. The real part interleaving matrix is ​​used to reorder the real part of the symbol sequence of the first rotating constellation point, and the imaginary part interleaving matrix is ​​used to reorder the imaginary part of the symbol sequence of the first rotating constellation point. The real and imaginary parts of the symbol sequence are then recombined into a complex signal to obtain the first interleaved signal.

[0042] The transmission module transmits the first interleaved signal to the receiving end through a fading channel, and the receiving end receives the second interleaved signal.

[0043] The semantic decoding module allows the receiving end to perform semantic reconstruction decoding on the second interleaved signal to obtain a second image; or, the receiving end to perform semantic classification decoding on the second interleaved signal to obtain a second image category.

[0044] This disclosure also provides an electronic device, including:

[0045] At least one processor; and

[0046] A memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the encoding method described in any of the above technical solutions.

[0048] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the encoding method according to any one of the above embodiments.

[0049] This disclosure provides a deep joint source-channel coding method and related apparatus, which optimizes semantic transmission performance through refined design.

[0050] First, the solution addresses the problem of insufficient utilization of semantic symbol correlation: the scheme generates corresponding rotation angles by extracting semantic information from the initial constellation points, constructs an interleaved matrix of real and imaginary parts based on the rotated constellation points, and reorders and recombines the real and imaginary parts of the constellation points accordingly, reducing the probability of strongly correlated symbols in the fading channel being simultaneously subjected to deep fading, avoiding the loss of continuous semantic information, and improving the image reconstruction or classification performance at the receiver.

[0051] Secondly, it addresses the issue of poor adaptability of non-uniform constellation points: instead of relying on traditional signal space diversity methods with fixed-angle rotation, it dynamically generates rotation angles based on semantic information, flexibly adapting to the probability distribution of non-uniform constellation points generated by the semantic encoder. By increasing the diversity order between constellation points through adaptive angle rotation and signal space diversity, it effectively improves transmission efficiency and anti-fading capability.

[0052] Furthermore, the problem of insufficient adaptability of static coding strategies is addressed: throughout the process, the generation of rotation angles depends on real-time semantic information, and the construction of the interleaving matrix combines the real and imaginary parts of channel state information, thereby realizing the dynamic adjustment of the coding strategy according to semantic features and channel state, reducing the performance fluctuation of the system under different signal-to-noise ratio conditions, and ensuring the stability of transmission quality in variable fading channels.

[0053] Finally, resource efficiency and system complexity are optimized: relying on neural networks to dynamically optimize constellation points and interleaving strategies, the entire optimization process does not require additional transmission resources. While improving transmission performance, it effectively reduces system complexity and bandwidth requirements, achieving dual optimization of efficient resource utilization and cost control.

[0054] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0055] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0056] Figure 1 This is a flowchart illustrating the steps of the encoding method in an embodiment of this disclosure;

[0057] Figure 2 This is a schematic diagram of the encoding method in the embodiments of this disclosure;

[0058] Figure 3 This is a schematic diagram of the encoding method in an embodiment of this disclosure;

[0059] Figure 4 This is another schematic diagram of the encoding method in the embodiments of this disclosure;

[0060] Figure 5 A schematic block diagram of the encoding device in the embodiments of this disclosure;

[0061] Figure 6 This is a schematic block diagram of the electronic device in the embodiments of this disclosure. Detailed Implementation

[0062] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0063] This disclosure provides a deep joint source-channel coding method, such as Figure 1 As shown, it includes:

[0064] Step S101: The transmitting end obtains the first semantic feature of the first image, and performs constellation mapping modulation on the first semantic feature to obtain the first initial constellation point;

[0065] Step S102: The sending end extracts the semantic information of the first initial constellation point, generates a rotation angle corresponding to the semantic information, and rotates the first initial constellation point according to the rotation angle to obtain the first rotated constellation point;

[0066] Step S103: The transmitting end generates a real part interleaving matrix and an imaginary part interleaving matrix based on the real and imaginary parts of the first rotating constellation point and the channel state information. The real part interleaving matrix is ​​used to reorder the real part of the symbol sequence of the first rotating constellation point, and the imaginary part interleaving matrix is ​​used to reorder the imaginary part of the symbol sequence of the first rotating constellation point. The real and imaginary parts of the symbol sequence are then recombined into a complex signal to obtain the first interleaved signal.

[0067] Step S104: The transmitting end transmits the first interleaved signal to the receiving end through the fading channel, and the receiving end receives the second interleaved signal.

[0068] In step S105, the receiving end performs semantic reconstruction decoding on the second interleaved signal to obtain the second image; or, the receiving end performs semantic classification decoding on the second interleaved signal to obtain the second image category.

[0069] like Figure 2 The diagram shown illustrates the principle of the encoding method disclosed herein.

[0070] Figure 2 The upper part illustrates the transmission scenario of existing technologies and the problems they encounter when faced with the loss of strongly correlated semantic symbols caused by fading channels. Figure 2 The lower part illustrates the coding method of this disclosure, which solves the problem of loss of strongly correlated semantic symbols in fading channels through an "interleaving-deinterleaving" process. The specific principle is as follows.

[0071] like Figure 2 The upper part shows the non-interleaved transmission scenario in the prior art. The transmitting end directly transmits semantically related signals (representing a sequence of strongly related semantic symbols with different gray-scale dots). When passing through the fading channel, the strongly related semantic symbols are concentrated and easily affected by the channel depth fading (manifested as continuous "hollow" dots replacing the original "gray-scale" dots), resulting in the loss of continuous semantic information at the receiving end and reducing the image reconstruction / classification performance.

[0072] like Figure 2 The lower half of the diagram illustrates the "interleaving-deinterleaving" transmission scenario of the coding method disclosed herein. The transmitting end performs an interleaving operation based on semantic correlation on strongly correlated semantic signals, disrupting the original continuous distribution of strongly correlated semantic symbols and dispersing the semantic symbols in the transmission sequence, thus avoiding the risk of concentrated exposure of strongly correlated symbols to channel fading.

[0073] When passing through a fading channel, although some signals are affected by fading (manifested as hollow dots replacing the original gray dots), the strong semantic symbols are dispersed, preventing the loss of continuous semantic information. The receiver performs deinterleaving to restore the original distribution of semantically related signals according to the interleaving rules of the transmitter, recovering effective semantic information, ensuring the performance of semantic processing tasks such as image reconstruction / classification, and improving transmission stability under fading channels.

[0074] It is evident that this disclosure optimizes semantic transmission performance through meticulous design.

[0075] First, the solution addresses the problem of insufficient utilization of semantic symbol correlation: the scheme generates corresponding rotation angles by extracting semantic information from the initial constellation points, constructs an interleaved matrix of real and imaginary parts based on the rotated constellation points, and reorders and recombines the real and imaginary parts of the constellation points accordingly, reducing the probability of strongly correlated symbols in the fading channel being simultaneously subjected to deep fading, avoiding the loss of continuous semantic information, and improving the image reconstruction or classification performance at the receiver.

[0076] Secondly, it addresses the issue of poor adaptability of non-uniform constellation points: instead of relying on traditional signal space diversity methods with fixed-angle rotation, it dynamically generates rotation angles based on semantic information, flexibly adapting to the probability distribution of non-uniform constellation points generated by the semantic encoder. By increasing the diversity order between constellation points through adaptive angle rotation and signal space diversity, it effectively improves transmission efficiency and anti-fading capability.

[0077] Furthermore, the problem of insufficient adaptability of static coding strategies is addressed: throughout the process, the generation of rotation angles depends on real-time semantic information, and the construction of the interleaving matrix combines the real and imaginary parts of channel state information, thereby realizing the dynamic adjustment of the coding strategy according to semantic features and channel state, reducing the performance fluctuation of the system under different signal-to-noise ratio conditions, and ensuring the stability of transmission quality in variable fading channels.

[0078] Finally, resource efficiency and system complexity are optimized: relying on neural networks to dynamically optimize constellation points and interleaving strategies, the entire optimization process does not require additional transmission resources. While improving transmission performance, it effectively reduces system complexity and bandwidth requirements, achieving dual optimization of efficient resource utilization and cost control.

[0079] like Figure 3 The diagram shown illustrates a deep joint source-channel coding method.

[0080] Figure 3 It comprises two main stages: semantic encoding and semantic decoding, involving two devices: a transmitter and a receiver. It realizes the process from inputting the original image to outputting the reconstructed image, encompassing key components such as a semantic encoding network, a constellation mapper, an adaptive angle generator, a rotation module, a semantic correlation interleaver, and a semantic decoder. These components are interconnected to complete the processing and transformation of semantic information.

[0081] 1. Extract semantic features at the sending end.

[0082] Assume the first input image X is an RGB image. A semantic encoder based on a convolutional neural network and a residual network is used. Hierarchical feature extraction is performed on the first image X to capture the global semantic information of the image, and a high-dimensional semantic feature matrix is ​​output, i.e., the first semantic feature S∈R. n×KWhere n represents the number of sub-features (e.g., n = 1024), and K represents the sub-feature dimension (e.g., K = 16). The semantic encoder is optimized through pre-training to ensure that the extracted features can effectively represent the core semantic content of the image (such as the shape, texture, or color of objects).

[0083] 2. The transmitting end generates discrete constellation points.

[0084] The first semantic feature S is input to the constellation mapper. The high-dimensional semantic features S are mapped to a structured tensor z through a fully connected layer, representing the unnormalized log probability distribution of each symbol in the corresponding modulation scheme.

[0085] When using M-QAM modulation, the dimension of the first structured tensor z is... (N is a predefined value, 1024 in this example; M is the number of constellation points).

[0086] Then, the Gumbel-Softmax technique is used to generate the first initial constellation point R. The specific process is as follows: Gumbel noise is added to the tensor z to achieve differentiable discrete sampling.

[0087] This is then converted into a one-hot vector, representing the discrete choices of the non-uniform probability distribution along the sign dimension. The dimension of this one-hot vector is the same as the dimension of the first structured tensor z, i.e.

[0088] Finally, based on the specified modulation scheme (such as BPSK, M-QAM, etc.), the one-hot vector is weighted and combined with its real and imaginary components to map it into a two-dimensional complex constellation point. That is, based on the one-hot vector, the weighted combination of its real and imaginary components is mapped into a two-dimensional complex constellation point, namely the first initial constellation point R.

[0089] Based on the weighted sum of the one-hot vector and the predefined constellation point values, a modulation output is generated. The modulation output is then grouped by alternating indices and mapped to two-dimensional complex constellation points, i.e., the first initial constellation points, where even indices correspond to the real part of the signal and odd indices correspond to the imaginary part. The predefined constellation point values ​​are equivalent to the preset x-coordinates or y-coordinates of the constellation points. Because a one-hot vector is a vector with only one 1 and the rest 0s, the weighted sum of the one-hot vector and the preset coordinates yields the x-coordinates or y-coordinates of the constellation points. Since there are N one-hot vectors, N / 2 constellation points can be generated (the x-coordinates and y-coordinates combine to form a constellation point).

[0090] Constellation points are specifically characterized by a higher probability in the central area and a lower probability in the outer area. (Parameters of the constellation mapper) Through end-to-end training and optimization, we ensure that the generated constellation points can efficiently represent semantic information and adapt to subsequent signal space diversity processing.

[0091] In communication systems, modulation is the crucial process of converting digital signals into analog signals suitable for transmission in a channel. The modulation method directly affects the transmission rate, noise immunity, spectral efficiency, and complexity of the communication system. Common digital modulation methods include BPSK and M-QAM, both of which belong to constellation modulation (transmitting information by defining discrete "constellation points" in the signal space).

[0092] Taking M-QAM modulation as an example, "M" represents the number of constellation points (i.e., the size of the symbol set), which is the general representation of QAM modulation. Common M-QAMs include 4-QAM, 16-QAM, and 64-QAM.

[0093] 3. The transmitting end performs adaptive angle rotation.

[0094] Adaptive Angle Generator Based on Convolutional Neural Network (CNN) The intrinsic features of the first initial constellation point set R are analyzed. The adaptive angle generator comprises a multi-layer convolutional network. First, it extracts the spatial distribution features of the first initial constellation points R, and then outputs a rotation angle θ (range 0°–90°) through fully connected layers. The rotation angle θ is dynamically adjusted based on task characteristics (such as image reconstruction or classification) and the non-uniform distribution of the constellation points. Rotation module. Rotate the first initial constellation point R to generate the rotation symbol R. r ∈R K×2 This serves as the first rotating constellation point to increase the diversity among constellation points. The rotated symbols do not require the receiver to recover the rotation angle; the decoder directly processes the semantic symbols embedded with adaptive rotation angle information.

[0095] The specific process is as follows:

[0096] (1) Multi-level feature extraction: By alternately stacking three sets of one-dimensional convolutional layers and ReLU pooling layers, the local spatial features (5×1 convolutional kernel) and global structural features (3×1 convolutional kernel) of the constellation point sequence are extracted respectively to generate feature sequences.

[0097] (2) Feature fusion and mapping: The length of the convolutional output sequence is fixed to N by an adaptive average pooling layer to generate a high-dimensional feature tensor. Then, it is processed by a fully connected layer and combined with the Sigmoid activation function (i.e., normalized exponential function) to map the output to the [0,1] interval. Then, it is precisely adjusted to the rotation angle range of 0° to 90° by a linear transformation.

[0098] (3) Dynamic adaptive optimization: The generated rotation angle is dynamically adjusted according to the importance of the semantic information carried by the constellation points. The constellation points in the outer region carry relatively important semantic information, while the constellation points in the central region carry relatively unimportant semantic information. Through adaptive rotation, the projection distance of the constellation points carrying important semantic information in the I-axis (i.e., real part axis) and Q-axis (i.e. imaginary part axis) directions is increased, and important semantic information is protected first. Even if one component suffers deep fading, the other component can still provide distinguishing information, reducing semantic loss.

[0099] 4. The sending end performs semantic correlation interleaving.

[0100] First rotating constellation point R r Input semantic relevance interleaver The correlation matrix between symbols is calculated using a multi-head attention (MHA) mechanism. Specifically, MHA uses multiple attention heads to capture the dependencies between symbols in the I-path and Q-path respectively, and concatenates the outputs of each attention head to generate the correlation weight z. w ∈R K At the same time, it combines the imperfect channel state information obtained from channel estimation. An end-to-end trainable interleaved matrix P is generated using a multilayer perceptron. i ∈R K×K , i∈{1,2}.

[0101] Interleaving matrix P i Acting on the first rotating constellation point R r The interleaved symbol Z is obtained as the first interleaved signal. This involves distributing strongly correlated symbols (e.g., semantic features of adjacent pixels) to locations with weaker channel fading to generate the interleaved symbol Z. The interleaver is optimized through end-to-end training to ensure that strongly correlated symbols are not simultaneously affected by deep fading, thereby guaranteeing maximum data recovery at the receiver and improving transmission reliability.

[0102] The specific process is as follows:

[0103] (1) Symbol Relevance Modeling: The relevance weights of the symbol sequence are calculated using a multi-head attention (MHA) mechanism. The input symbol sequence is projected as a query Q, a key K, and a value vector V, and a scaled dot product attention mechanism is used to calculate the score (d). k (Representing the dimension of each attention head), the Softmax function normalizes the scores and generates semantic relevance weights A. s The specific formula is as follows:

[0104]

[0105] (2) Channel Information Fusion: The correlation matrix output by MHA is concatenated with the amplitude features of imperfect channel state information (CSI) and input into a multi-layer perceptron (MLP). The MLP contains two fully connected layers (with 2048 and 4096 hidden neurons respectively, and the activation function is ReLU), and outputs a permutation probability matrix.

[0106] (3) Correlation Interleaving: Using the Gumbel-Softmax sampling technique, a discrete interleaving matrix is ​​generated from the permutation probability matrix through hard sampling (setting τ→0). Specifically, it is a sparse matrix where each row and column has only one 1 and the rest are 0 values. The interleaving matrix is ​​used to dynamically rearrange the real part (I part) and imaginary part (Q part) of the symbol sequence and then reassemble them into a complex signal to prevent the real part and imaginary part of the symbol sequence from encountering deep fading at the same time, thereby reducing the impact of deep fading.

[0107] 5. The transmitting end transmits the first interleaved signal to the receiving end through the channel.

[0108] The first interleaved signal Z is transmitted to the receiver through a fading channel. The channel model is as follows: Where h represents the channel gain, Z represents the first interleaved signal, and n represents additive Gaussian noise that follows a normal distribution.

[0109] 6. The receiving end performs semantic decoding.

[0110] The receiving end receives the second interleaved signal. Because the first interleaved signal may suffer losses during transmission through the channel, the signal received by the receiving end is called the second interleaved signal. The second interleaved signal is the signal that arrives at the receiving end after the first interleaved signal has been transmitted through the channel.

[0111] The decoder design fully utilizes the adaptive angle information embedded at the transmitter, eliminating the need for an additional rotation recovery process, thus simplifying receiver processing and improving robustness. The entire decoding process is optimized through end-to-end training, ensuring high-quality image reconstruction or accurate classification results even in Rayleigh fading channels.

[0112] Depending on the transmission task (image reconstruction or image classification), it is processed by the corresponding semantic decoder. Figure 3 The upper part of the semantic decoding stage illustrates the image reconstruction task, and the lower part illustrates the image classification task.

[0113] For image reconstruction tasks: use a semantic reconstruction decoder. The second interleaved signal is processed using a deep neural network (e.g., a decoder based on a convolutional neural network or Transformer). The reconstruction decoder, optimized through pre-training, is able to directly parse the rotated symbols, recover detailed features of the image using embedded semantic information, and generate a reconstructed image.

[0114] Image classification task: using a semantic classification decoder Based on a convolutional neural network and fully connected layers, class-related features are extracted from the symbols, and the classification result is output. The classification decoder is optimized through pre-training and can directly process the second interleaved signal to generate a classification probability vector, where the class with the highest probability represents the classification result.

[0115] like Figure 4 As shown, another schematic diagram of a deep joint source-channel coding method is provided. Figure 4 The illustration shows the implementation of the semantic relevance deinterleaver in the decoding stage.

[0116] Figure 4 and Figure 4 All of these are “interleaving-deinterleaving” processes, specifically an end-to-end semantic communication flow of “semantic encoding-fading channel-semantic decoding”. The first image X is input, processed by the semantic encoding module, transmitted through the fading channel, and then the semantic decoding module completes image reconstruction and / or semantic classification.

[0117] Figure 4 semantic encoding stage and Figure 3 Similarly, it also includes constellation mappers, rotation modules, adaptive angle generators, and semantic correlation interleavers, which perform angle generation, mapping, and correlation processing on semantic features to prepare for semantic transmission and ensure that semantic information is adapted and transmitted in the channel.

[0118] Figure 4 The semantic decoding stage is different Figure 3 . Figure 4 After the semantic encoding output passes through the fading channel, the semantic information transmitted through the channel is first de-interleaved by the semantic correlation deinterleaver before being sent to the semantic reconstruction and classification decoder. The addition of the semantic deinterleaving step optimizes the semantic information restoration process.

[0119] The receiver inputs the second interleaved signal into the semantic relevance deinterleaving unit. Combined parameters Obtain the second constellation point This prepares for further semantic understanding and reconstruction. The second constellation point... It is the recovery information corresponding to the first initial constellation point R.

[0120] Depending on the transmission task (image reconstruction or image classification), it is processed by the corresponding semantic decoder.

[0121] For image reconstruction tasks: use a semantic reconstruction decoder. Processing the second constellation points using deep neural networks (e.g., decoders based on convolutional neural networks or Transformers). The reconstruction decoder, optimized through pre-training, can directly parse rotated symbols, recover detailed features of the image using embedded semantic information, and generate a reconstructed image.

[0122] Image classification task: using a semantic classification decoder Based on convolutional neural networks and fully connected layers, class-related features are extracted from the symbols, and the classification result is output. The classification decoder is optimized through pre-training and is able to handle second constellation points. Generate a classification probability vector, where the category with the highest probability represents the classification result.

[0123] In short, Figure 4 exist Figure 3 Based on this, a semantic relevance deinterleaver is added between the fading channel and semantic decoding to optimize the processing flow of semantic information from the channel to decoding. The core semantic processing logic of the two is consistent, but... Figure 4 By refining the information restoration steps before semantic decoding through deinterleaving, the accuracy of semantic reconstruction and classification can be improved.

[0124] The modules and processing steps work together to fully realize the functions of semantic encoding and decoding through the above process. From semantic extraction and processing of the input image to final semantic understanding and related output, a closed-loop semantic information processing system is formed. Each parameter plays a role in controlling the processing logic and feature transformation in the corresponding module, ensuring the accuracy and effectiveness of the entire semantic processing process.

[0125] This disclosure also provides an encoding device 200, such as Figure 5 As shown, it includes:

[0126] The constellation point generation module 201 obtains the first semantic features of the first image from the sending end, and performs constellation mapping modulation on the first semantic features to obtain the first initial constellation points;

[0127] The adaptive angle rotation module 202 extracts the semantic information of the first initial constellation point from the transmitting end, generates a rotation angle corresponding to the semantic information, and rotates the first initial constellation point according to the rotation angle to obtain the first rotated constellation point.

[0128] The semantic correlation interleaving module 203 generates a real part interleaving matrix and an imaginary part interleaving matrix based on the real and imaginary parts of the first rotating constellation point and the channel state information. The real part interleaving matrix is ​​used to reorder the real part of the symbol sequence of the first rotating constellation point, and the imaginary part interleaving matrix is ​​used to reorder the imaginary part of the symbol sequence of the first rotating constellation point. The real and imaginary parts of the symbol sequence are then recombined into a complex signal to obtain the first interleaved signal.

[0129] Transmission module 204: The transmitting end transmits the first interleaved signal to the receiving end through the fading channel, and the receiving end receives the second interleaved signal;

[0130] The semantic decoding module 205 allows the receiving end to perform semantic reconstruction decoding on the second interleaved signal to obtain a second image; or, the receiving end can perform semantic classification decoding on the second interleaved signal to obtain a second image category.

[0131] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0132] Figure 6 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0133] like Figure 6 As shown, device 300 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 302 or a computer program loaded from storage unit 308 into random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0134] Multiple components in device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of monitors, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] The computing unit 301 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as the metaverse scene dynamic networking method. For example, in some embodiments, the metaverse scene dynamic networking method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the metaverse scene dynamic networking method described above can be performed. Alternatively, in other embodiments, computing unit 301 may be configured to perform a metaverse scene dynamic networking method by any other suitable means (e.g., by means of firmware).

[0136] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0138] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0140] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0141] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0142] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause the computer to perform the method according to any one of the above methods.

[0143] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0144] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0145] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A deep joint source-channel coding method, characterized in that, include: The sending end obtains the first semantic feature of the first image, and performs constellation mapping modulation on the first semantic feature to obtain the first initial constellation point; The sending end extracts the semantic information of the first initial constellation point, generates a rotation angle corresponding to the semantic information, and rotates the first initial constellation point according to the rotation angle to obtain the first rotated constellation point; The transmitting end generates a real part interleaving matrix and an imaginary part interleaving matrix based on the real part and imaginary part of the first rotating constellation point and the channel state information. The real part interleaving matrix is ​​used to reorder the real part of the symbol sequence of the first rotating constellation point, and the imaginary part interleaving matrix is ​​used to reorder the imaginary part of the symbol sequence of the first rotating constellation point. The real part and imaginary part of the symbol sequence are then recombined into a complex signal to obtain the first interleaved signal. The transmitting end transmits the first interleaved signal to the receiving end through the fading channel, and the receiving end receives the second interleaved signal; The receiving end performs semantic reconstruction decoding on the second interleaved signal to obtain the second image; or, the receiving end performs semantic classification decoding on the second interleaved signal to obtain the second image category.

2. The encoding method according to claim 1, characterized in that, The sending end acquires the first semantic feature of the first image, including: The first image is input into a deep neural network based on a convolutional neural network and a residual network to obtain the first semantic features of the first image.

3. The encoding method according to claim 1, characterized in that, The step of performing constellation mapping modulation on the first semantic feature to obtain the first initial constellation point includes: The first semantic feature is input into the constellation mapper, and the first semantic feature is mapped into a first structured tensor through a fully connected layer. The first structured tensor represents the unnormalized log probability distribution of each symbol. Gumbell noise is added to the first structured tensor to achieve differentiable discrete sampling, and then a one-hot vector is generated; based on the weighted sum of the one-hot vector and the predefined constellation point values, a modulation output is generated; the modulation output is grouped by alternating indices and mapped to two-dimensional complex constellation points, i.e., the first initial constellation points, where even indices correspond to the real part of the signal and odd indices correspond to the imaginary part of the signal.

4. The encoding method according to claim 1, characterized in that, The transmitting end extracts the semantic information of the first initial constellation point by: extracting the local spatial features and global structural features of the first initial constellation point sequence by alternately stacking three sets of one-dimensional convolutional layers and rectified linear unit pooling layers, and generating a feature sequence as the semantic information of the first initial constellation point; The process of generating the rotation angle corresponding to the semantic information includes: setting the length of the convolutional output sequence to a first fixed length through an adaptive average pooling layer to generate a high-dimensional feature tensor, which is then processed by a fully connected layer, and the output is mapped to the [0,1] interval by combining a sigmoid activation function, and then adjusted to the rotation angle range of 0° to 90° through a linear transformation to obtain the rotation angle corresponding to the semantic information. The step of rotating the first initial constellation point according to the rotation angle to obtain the first rotated constellation point includes: dynamically adjusting the generated rotation angle according to the importance of the semantic information carried by the first initial constellation point; and increasing the projection distance of the first initial constellation point carrying important semantic information in the I-axis and Q-axis directions through adaptive rotation to prioritize the protection of important semantic information.

5. The encoding method according to claim 1, characterized in that, The transmitting end generates a real-part interleaving matrix and an imaginary-part interleaving matrix based on the real and imaginary parts of the first rotating constellation point and channel state information. It then reorders the real parts of the symbol sequence of the first rotating constellation point using the real-part interleaving matrix and reorders the imaginary parts of the symbol sequence using the imaginary-part interleaving matrix, recombines the real and imaginary parts of the symbol sequence into a complex signal, and obtains the first interleaved signal, including: The correlation weight matrix between the real parts of the first rotating constellation points is calculated using a multi-head attention mechanism, and the correlation weight matrix between the imaginary parts of the first rotating constellation points is also calculated. The real part correlation weight matrix is ​​concatenated with the real part of the amplitude feature of the channel state information to obtain the real part concatenated feature, and the imaginary part correlation weight matrix is ​​concatenated with the imaginary part of the amplitude feature of the channel state information to obtain the imaginary part concatenated feature. The real part concatenation feature and the imaginary part concatenation feature are respectively input into a multilayer perceptron, which outputs a real part permutation probability matrix and an imaginary part permutation probability matrix respectively; the multilayer perceptron contains two fully connected layers; Using the Gombell sigmoid function sampling technique, real part interleaving matrix and imaginary part interleaving matrix are generated from the real part permutation probability matrix and the imaginary part permutation probability matrix respectively through hard sampling; both the real part interleaving matrix and the imaginary part interleaving matrix are sparse matrices with only one 1 in each row and column and the rest being 0 values; The real part of the first rotating constellation point is dynamically rearranged using the real part interleaving matrix, and the imaginary part of the first rotating constellation point is dynamically rearranged using the imaginary part interleaving matrix. The real and imaginary parts of the first rotating constellation point are then recombined into a complex signal to obtain the first interleaved signal.

6. The encoding method according to claim 5, characterized in that, The step of calculating the real part correlation weight matrix between the real parts of the first rotating constellation points and the imaginary part correlation weight matrix between the imaginary parts of the first rotating constellation points using a multi-head attention mechanism includes: The real and imaginary parts of the first rotating constellation point are projected as query vector, key vector, and value vector, respectively. Based on the query vector, key vector, and value vector, the attention scores of the real and imaginary parts are calculated using a scaled dot product attention mechanism. The attention scores are normalized using a normalized exponential function to generate a real part relevance weight matrix and an imaginary part relevance weight matrix.

7. The encoding method according to claim 1, characterized in that, The fading channel includes any one of the following: Rayleigh fading channel and Rice fading channel.

8. The decoding method according to claim 1, characterized in that, The receiving end performs semantic reconstruction decoding on the second interleaved signal to obtain a second image, including: The second interleaved signal is input into a multi-layer convolutional neural network structure, which integrates residual connection mechanism and spatial resolution restoration operation. The spatial structure of the original image is gradually reconstructed through stepwise feature enhancement and nonlinear transformation. Gradient flow is optimized to improve reconstruction accuracy. Finally, a second image matching the size of the original image is generated through multi-stage resolution upsampling operation.

9. The decoding method according to claim 1, characterized in that, The receiving end performs semantic classification decoding on the second interleaved signal to obtain a second image category, including: The second interleaved signal is input into a multi-layer fully connected network based on a spine network architecture. Hierarchical feature fusion is implemented through progressive input. Multi-stage linear processing combined with dropout regularization is used to prevent overfitting, progressively optimizing feature representation capabilities and outputting a multi-class probability distribution. The class with the highest probability is taken as the second image class.

10. The decoding method according to claim 1, characterized in that, The receiving end performs semantic reconstruction decoding on the second interleaved signal to obtain a second image, including: The second interleaved signal is deinterleaved for semantic correlation to obtain the second constellation points; the second constellation points are then semantically reconstructed and decoded to obtain the second image. The receiving end performs semantic classification decoding on the second interleaved signal to obtain a second image category, including: The second interleaved signal is deinterleaved based on semantic correlation to obtain the second constellation point; the second constellation point is then semantically classified and decoded to obtain the second image category.

11. A deep joint source-channel coding apparatus, characterized in that, include: The constellation point generation module obtains the first semantic features of the first image from the sending end, and performs constellation mapping modulation on the first semantic features to obtain the first initial constellation points; The adaptive angle rotation module extracts the semantic information of the first initial constellation point from the transmitting end, generates a rotation angle corresponding to the semantic information, and rotates the first initial constellation point according to the rotation angle to obtain the first rotated constellation point. The semantic correlation interleaving module generates a real part interleaving matrix and an imaginary part interleaving matrix based on the real and imaginary parts of the first rotating constellation point and the channel state information. The real part interleaving matrix is ​​used to reorder the real part of the symbol sequence of the first rotating constellation point, and the imaginary part interleaving matrix is ​​used to reorder the imaginary part of the symbol sequence of the first rotating constellation point. The real and imaginary parts of the symbol sequence are then recombined into a complex signal to obtain the first interleaved signal. The transmission module transmits the first interleaved signal to the receiving end through a fading channel, and the receiving end receives the second interleaved signal. The semantic decoding module allows the receiving end to perform semantic reconstruction decoding on the second interleaved signal to obtain a second image; or, the receiving end to perform semantic classification decoding on the second interleaved signal to obtain a second image category.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the encoding method according to any one of claims 1-10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the encoding method according to any one of claims 1-10.

Images