Collaborative non-orthogonal image semantic communication method and system based on information disentanglement

By adding Gaussian random matrix and de-entanglement module to the image, and combining course learning to train the joint source channel coding model, the channel environment problem in multi-user wireless image transmission is solved, and efficient and reliable non-orthogonal image transmission and multi-task are achieved. Concurrent execution.

CN118803277BActive Publication Date: 2024-11-15JINAN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411258882.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-11-15
Estimated Expiration
2044-09-10

AI Technical Summary

Technical Problem

The prior art cannot achieve efficient and reliable wireless image transmission in harsh channel environments, especially in multi-user transmission, and the computing complexity of deep neural networks for signal separation is high, which cannot meet unexpected intelligent tasks.

Method used

Adopting a collaborative non-orthogonal image semantic communication method based on information de-entanglement, by adding a trainable Gaussian random matrix to the image, de-entanglement module is used to de-entangle the user semantic information, and training the joint source channel coding model based on course learning , realize non-orthogonal transmission with high spectrum efficiency, and combine the combined source channel encoder and decoder for information transmission.

Benefits of technology

The image separation and recovery problems are effectively solved during non-orthogonal multi-transmission, and the coordinated transmission of multiple users is realized, spectrum utilization efficiency is improved, information redundancy is reduced, and reliable image recovery and classification is achieved under different signal-to-noise ratios. and clustering tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118803277B_ABST
    Figure CN118803277B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for cooperative non-orthogonal image semantic communication based on information disentanglement, the method comprising the following steps: obtaining two-view images of a target, connecting a random Gaussian matrix with an image pixel matrix to obtain an image splicing matrix; constructing and training a two-view distributed cooperative joint source channel coding model; completing image restoration, classification and clustering tasks based on the trained two-view distributed cooperative joint source channel coding model; mapping the image splicing matrix into semantic information through a joint source channel encoder, splicing private information and public information to obtain coded information, obtaining air aggregated information through air aggregation of the coded information, and restoring the two-view image through a joint source channel decoder; using public information as input, performing image classification based on a KNN algorithm, and obtaining image clustering results based on a k-means clustering algorithm. The present invention can realize cooperative transmission, improve spectrum utilization efficiency, and reduce the redundancy of transmission information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of semantic communication technology, and in particular to a collaborative non-orthogonal image semantic communication method and system based on information disentanglement. Background Art

[0002] With the increase of intelligent applications in edge networks, the requirements for efficient and reliable wireless image transmission are becoming higher and higher. Source coding and channel coding are key technologies for achieving efficient and reliable wireless image transmission. Although this method is optimal for infinitely long source and channel blocks, it cannot perform channel decoding normally in harsh channel environments, resulting in a "cliff effect". Most of the current work focuses on end-to-end single-user transmission. For multi-user transmission, it is transmitted in an orthogonal manner, which requires a large amount of spectrum resources.

[0003] In order to improve spectrum efficiency, semantic communication can adopt non-orthogonal multiple access (NOMA) scheme. Although signal interference between users is a challenge in NOMA, advanced multi-user identification (MUD) algorithms (such as serial interference cancellation technology (SIC)) can reduce mutual interference between users and improve signal quality. However, this technology has high computational complexity. NOMA in semantic communication uses deep neural networks to separate signals from superimposed signals. However, the existing deep joint source-channel coding framework focuses on solving specific tasks such as data recovery, classification or other artificial intelligence tasks, and cannot meet the requirements of performing other non-intended intelligent tasks. Summary of the invention

[0004] In order to overcome the defects and shortcomings of the prior art, the present invention provides a collaborative non-orthogonal image semantic communication method and system based on information disentanglement. The present invention adds a trainable Gaussian random matrix to the input image, disentangles the semantic information of each user through a disentanglement module, and finally trains based on curriculum learning to achieve non-orthogonal transmission with high spectral efficiency, thereby solving the problem of image separation and recovery when multi-user recognition is not used in non-orthogonal multi-transmission. In addition, for the problem of non-orthogonal collaborative transmission of multiple users under different signal-to-noise ratios, the joint source channel encoder and the joint source channel decoder are used to focus on transmitting information rather than features, thereby achieving efficient and reliable transmission.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a collaborative non-orthogonal image semantic communication method based on information disentanglement, comprising the following steps:

[0007] Get two-view images of the target, connect the random Gaussian matrix with the image pixel matrix to get the image stitching matrix;

[0008] Constructing a two-view distributed collaborative joint source channel coding model, including a joint source channel encoder, a disentanglement module and a joint source channel decoder, wherein the disentanglement module includes a private encoder and a public encoder;

[0009] A training method is constructed based on course learning to train the two-view distributed collaborative joint source channel coding model. Image restoration, classification and clustering tasks are completed based on the trained two-view distributed collaborative joint source channel coding model.

[0010] The image stitching matrix is ​​mapped into semantic information by a joint source channel encoder, the semantic information is passed through a private encoder and a public encoder to obtain private information and public information, the private information and the public information are spliced ​​to obtain coded information, the coded information is subjected to aerial aggregation to obtain aerial aggregate information, and the aerial aggregate information is restored by a joint source channel decoder to obtain a two-view image;

[0011] Taking public information as input, image data is classified based on KNN algorithm;

[0012] Taking public information as input, the clustering result of image data is obtained based on the k-means clustering algorithm.

[0013] As a preferred technical solution, the joint source channel encoder includes: a first residual block upsampling, a first attention feature module, a second residual block upsampling, a second attention feature module, a first residual block, a third attention feature module, a second residual block, a fourth attention feature module and a third residual block upsampling;

[0014] The image stitching matrix is ​​sequentially sampled by the first residual block, the first attention feature module, the second residual block, the second attention feature module, the first residual block, the third attention feature module, the second residual block, the fourth attention feature module and the third residual block to obtain semantic information.

[0015] As a preferred technical solution, private information and public information are concatenated to obtain coded information, which is specifically expressed as:

[0016] ;

[0017] in, Indicates public information. , represents a complex set, Indicates private information. , and They respectively represent the length of the corresponding information.

[0018] As a preferred technical solution, the coded information is aggregated in the air to obtain the air aggregated information, which is specifically expressed as:

[0019] ;

[0020] in, The variance is Additive Gaussian white noise, represents the M-dimensional identity matrix, and are public aggregate information and noisy private aggregate information, represents the channel matrix between edge device i and the server, is the preprocessing function, i={1,2}.

[0021] As a preferred technical solution, the joint source channel decoder includes: a first residual transposition block upsampling, a fifth attention feature module, a second residual transposition block upsampling, a sixth attention feature module, a first residual transposition block, a seventh attention feature module, a second residual transposition block, an eighth attention feature module and a third residual transposition block upsampling;

[0022] The aerial aggregated information is sequentially upsampled by the first residual transposition block, the fifth attention feature module, the second residual transposition block, the sixth attention feature module, the first residual transposition block, the seventh attention feature module, the second residual transposition block, the eighth attention feature module and the third residual transposition block to restore the two-view image.

[0023] As a preferred technical solution, a training method is constructed based on curriculum learning to train the two-view distributed collaborative joint source channel coding model, including: an end-to-end training phase, a training phase with image reconstruction loss, and a training phase with total loss;

[0024] In the end-to-end training phase, each pair of joint source channel encoders and joint source channel decoders is updated based on the joint source channel coding model of a single user;

[0025] The mean square error (MSE) is used to measure the difference between the reconstructed image and the original image of the two views to obtain the image reconstruction loss. Based on the trained single-user joint source channel coding model, the distributed joint source channel coding model is trained based on the image reconstruction loss.

[0026] The common encoder captures the common information across input views through consistency constraints and obtains the common information CMD constraints between devices;

[0027] The private encoder uses the Hilbert-Schmidt independence criterion to measure the difference between public information and private information in the device, and obtains the HSIC constraints of public information and private information;

[0028] Based on the weighted sum of image reconstruction loss, common information CMD constraint and HSIC constraint, the total loss is obtained. The optimal model parameters of the two-view distributed collaborative joint source channel coding model are obtained by minimizing the total loss.

[0029] As a preferred technical solution, in the end-to-end training phase, each pair of joint source channel encoders and joint source channel decoders is updated based on the joint source channel coding model of a single user, and the loss function is expressed as:

[0030] ;

[0031] in, represents the view image obtained by edge device i, i={1, 2}, Represents the restored view image.

[0032] As a preferred technical solution, the mean square error (MSE) is used to measure the difference between the reconstructed image and the original image to obtain the image reconstruction loss, which is specifically expressed as:

[0033] ;

[0034] in, represents the view image obtained by edge device i, i={1, 2}, Represents the restored view image.

[0035] As a preferred technical solution, the total loss is obtained based on the weighted sum of image reconstruction loss, common information CMD constraint, and HSIC constraint, which is specifically expressed as:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] in, , and Respectively represent the corresponding weight values, Indicates the public information corresponding to edge device i The expected vector of Represents the public information corresponding to edge device j The expected vector of Indicates the public information corresponding to edge device i The k-th central moment of Represents the public information corresponding to edge device j The k-th central moment of Indicates public information. Indicates private information. and is through and The size of the inner product is n×n The Gram matrix, represents the n-dimensional identity matrix, represents a full column vector, , and Respectively represent the information length corresponding to private information and public information.

[0047] The present invention also provides a collaborative non-orthogonal image semantic communication system based on information disentanglement, which is used to implement the above-mentioned collaborative non-orthogonal image semantic communication method based on information disentanglement, and the system includes: a target image acquisition module, a splicing module, a model construction module, a training method construction module, a model training module, an image restoration module, a classification module and a clustering module;

[0048] The target image acquisition module is used to acquire two-view images of the target;

[0049] The stitching module is used to connect the random Gaussian matrix with the image pixel matrix to obtain an image stitching matrix;

[0050] The model building module is used to build a two-view distributed collaborative joint source channel coding model, including a joint source channel encoder, a disentanglement module and a joint source channel decoder, and the disentanglement module includes a private encoder and a public encoder;

[0051] The training method construction module is used to construct a training method based on course learning;

[0052] The model training module is used to train the two-view distributed collaborative joint source channel coding model, and complete image restoration, classification and clustering tasks based on the trained two-view distributed collaborative joint source channel coding model;

[0053] The image restoration module is used to map the image stitching matrix into semantic information through a joint source channel encoder, the semantic information is passed through a private encoder and a public encoder to obtain private information and public information, the private information and the public information are spliced ​​to obtain coded information, the coded information is subjected to air aggregation to obtain air aggregate information, and the air aggregate information is restored through a joint source channel decoder to obtain a two-view image;

[0054] The classification module is used to take the public information as input and classify the image data based on the KNN algorithm;

[0055] The clustering module is used to take the public information as input and obtain the clustering result of the image data based on the k-means clustering algorithm.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] (1) The present invention constructs a two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D, designs consistency and inconsistency losses according to the objectives in the disentanglement module, disentangles semantic information by minimizing consistency and inconsistency losses, and effectively learns the public and private parts between different semantic information; it can directly use the public information in the feature space with sufficient discriminability to perform intelligent tasks, such as clustering and classification tasks;

[0058] It is beneficial for users in the same group to simultaneously transmit semantic information on the uplink channel, realize collaborative transmission, reliably restore image information, use non-orthogonal multiple access technology to share spectrum resources, improve spectrum utilization efficiency, and reduce the redundancy of transmitted information.

[0059] (2) The present invention proposes a two-view image semantic collaborative non-orthogonal transmission scheme in a distributed edge network, based on a constructed two-view distributed collaborative joint source channel coding model for semantic communication to support concurrent execution of multiple tasks.

[0060] (3) The present invention embeds trainable data into the original image and constructs a training method based on curriculum learning to train the two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D. Therefore, the two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D of the present invention can recover the image from the superimposed signal without using SIC or MUD.

[0061] (4) Combining the experimental results of image restoration, clustering and classification, the two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D of the present invention can effectively perform image restoration, classification and clustering tasks simultaneously, and has achieved significant improvements in classification and clustering accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flow chart of the collaborative non-orthogonal image semantic communication method based on information disentanglement of the present invention;

[0063] FIG2 (a) is a transmission schematic diagram of the transmitting end in the collaborative non-orthogonal image semantic communication method based on information disentanglement of the present invention;

[0064] FIG2 (b) is a transmission schematic diagram of a receiving end in the collaborative non-orthogonal image semantic communication method based on information disentanglement of the present invention;

[0065] FIG2 (c) is a schematic diagram of the network structure of each network module at the transmitting end and the receiving end in the collaborative non-orthogonal image semantic communication method based on information disentanglement of the present invention;

[0066] FIG. 3 (a) shows the present invention in In the case of , the training and test images are schematic diagrams of PSNR performance comparison under different signal-to-noise ratios on the Multi-COIL100 dataset;

[0067] FIG3( b ) is a schematic diagram of the present invention. In the case of , the training and test images are schematic diagrams of PSNR performance comparison under different signal-to-noise ratios on the Multi-COIL100 dataset;

[0068] Figure 4 For the present invention In the case of , the training and test images are schematic diagrams of the performance comparison of classification tasks under different signal-to-noise ratios on the Multi-COIL100 dataset. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0070] Example 1

[0071] The present embodiment provides a collaborative non-orthogonal image semantic communication method based on information disentanglement, which is used for uplink wireless image non-orthogonal transmission of an edge network, wherein multiple edge devices observe a target from different perspectives, and a transmitter uses a joint source channel encoder to extract semantic information from an input image, and the semantic information obtained by the joint source channel encoder is input into a disentanglement module to obtain private information and public information, and the private information and the public information are concatenated to obtain encoded information, and the encoded information is aggregated over the air to obtain air aggregated information, and after receiving the air aggregated information, the server recovers the two-view image through a joint source channel decoder.

[0072] like Figure 1 As shown, the collaborative non-orthogonal image semantic communication method of this embodiment specifically includes the following steps:

[0073] S1: Get two-view images of the target, connect the random Gaussian matrix with the image pixel matrix, and get the image stitching matrix ;

[0074] In this embodiment, the sending end records the image captured by device i as: ,in H and W Respectively represent the height and width of the image, 3 represents the RGB channel, represents the set of real numbers;

[0075] Random Gaussian Matrix Connect with the image pixel matrix to get the image stitching matrix , expressed as:

[0076] ;

[0077] S2: Construct a two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D that can simultaneously complete image restoration, clustering and classification tasks, including: a joint source channel encoder, a disentanglement module and a joint source channel decoder;

[0078] In the sending stage, the image stitching matrix By Parameterized Joint Source-Channel Coder Mapping to semantic information (M ≪ HW);

[0079] Among them, the joint source channel encoder Including: a first residual block upsampling RB upsample, a first attention feature module AF, a second residual block upsampling RB upsample, a second attention feature module AF, a first residual block RB, a third attention feature module AF, a second residual block RB, a fourth attention feature module AF and a third residual block upsampling RBupsample;

[0080] As shown in Figure 2(a)-Figure 2(c), the image stitching matrix Sequentially pass through the first residual block upsampling RBupsample, the first attention feature module AF, the second residual block upsampling RB upsample, the second attention feature module AF, the first residual block RB, the third attention feature module AF, the second residual block RB, the fourth attention feature module AF and the third residual block upsampling RB upsample to obtain semantic information;

[0081] In this embodiment, the residual block RB and the residual block upsample RB upsample include a convolution layer, a generalized split normalization layer GDN, and a nonlinear activation function PReLU layer. The convolution layer extracts task-related semantic visual information from the image, including color, texture, etc. The generalized split normalization layer GDN normalizes the network, broadens the convolution layer channel, and enhances the stability and generalization of the network.

[0082] In this embodiment, the attention feature module AF allows the network to use the signal-to-noise ratio as an input feature, randomly sample the signal-to-noise ratio for training, learn parameters of different signal-to-noise ratios, and test on channels with different signal-to-noise ratios, thereby enhancing the model's adaptability to different noise levels;

[0083] In this embodiment, the de-entanglement module includes Parameterized private encoder and Sutra Parameterized Common Encoders , the semantic information obtained by the joint source channel encoder Enter the disentanglement module to obtain private information and public information ,in represents a complex set, and Respectively represent the information length, so the encoded information of edge device i is expressed as ,and ; The bandwidth compression ratio is recorded as ;

[0084] Through air aggregation technology, the edge server receives the encoded information transmitted by the edge device. The signal received by the server is expressed as:

[0085] ;

[0086] in, The variance is Additive Gaussian white noise, represents the M-dimensional identity matrix;

[0087] In this embodiment, is the preprocessing function, expressed as:

[0088] ;

[0089] in, represents the channel matrix between edge device i and the server, Indicates the transmit power;

[0090] Assuming that both parties in communication know the channel information, the signal-to-noise ratio SNR is expressed as ,have , SNR is randomly sampled in [0, 20]dB;

[0091] Specifically, combining the private information and public information output by the disentanglement module, the air aggregation information y received by the server is expressed as:

[0092] ;

[0093] in, and are the public aggregate information and noisy private aggregate information received by the receiver respectively;

[0094] In the receiving stage, the Parameterized Joint Source-Channel Decoder Recover the image of each edge device from the received over-the-air aggregate information y , N is preferably 2, where is an image An estimated value of

[0095] In this embodiment, the joint source channel decoder It has a similar modular design to the joint source channel encoder, including: a first residual transposed block upsampling, a fifth attention feature module AF, a second residual transposed block upsampling, a sixth attention feature module AF, a first residual transposed block RTB, a seventh attention feature module AF, a second residual transposed block RTB, an eighth attention feature module AF and a third residual transposed block upsampling;

[0096] The aerial aggregate information y is sequentially upsampled by the first residual transposition block, the fifth attention feature module AF, the second residual transposition block, the sixth attention feature module AF, the first residual transposition block RTB, the seventh attention feature module AF, the second residual transposition block RTB, the eighth attention feature module AF and the third residual transposition block to restore the two-view image;

[0097] Among them, RTB and RTB upsampling consist of a transposed convolution layer, an inverse GDN layer (IGDN) and PReLU, among which the inverse GDN layer (IGDN) is the inverse process of GDN.

[0098] In this embodiment, the PReLU layer of the last module is replaced by a Sigmoid layer, the output is limited between 0 and 1, and the output channel of its transposed convolution layer is adjusted to 6. Through iterative training, the last layer has image-specific parameters, which can effectively distinguish signals and reconstruct images.

[0099] S3: Construct the loss function of the two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D, which can simultaneously complete image restoration, clustering and classification tasks;

[0100] The mean square error (MSE) is used to measure the gap between the reconstructed image and the original image. Therefore, the loss function of image reconstruction is expressed as:

[0101] ;

[0102] In this embodiment, in order to support the concurrent implementation of image restoration, clustering and classification tasks, it is necessary to mine the complementary information between images of different viewpoints to learn a comprehensive data representation, and the extracted semantic feature information is divided into the common information of the two viewpoints and the private information of each viewpoint. A neural network is used to define two nonlinear mappings from the extracted semantic feature information to the common subspace and the special subspace, namely, the public encoder f c and private encoder f p , the public encoder f c The common information across input views is captured by introducing consistency constraints. The goal of this embodiment is to minimize the difference in mapping within the common information subspace, promote the alignment of cross-viewpoint features in the shared subspace, and explicitly compare the moments of each order using the central moment difference, that is:

[0103] ;

[0104] in, yes The expected vector of yes If the central distance of each order is the same, then the distribution of the two random variables is consistent. The smaller the value, the closer the two distributions are. Therefore, the CMD constraint for the common information between two devices is expressed as:

[0105] ;

[0106] As shown in Figure 2(a), public information and private information are derived from the same semantic information. I want to learn in Capture different information in the private encoder f p Inconsistency constraints are introduced to maximize the independence between public information and private information. The Hilbert-Schmidt Independence Criterion (HSIC) is used to measure the difference between public information and private information in the device, namely:

[0107] ;

[0108] in, and is through and The size of the inner product is n×n The Gram matrix of and ,also, ,in, represents the n-dimensional identity matrix, is a full column vector, and , therefore, the HSIC constraints for public and private information are expressed as:

[0109] ;

[0110] At the same time, the mean square error (MSE) is used to measure the gap between the reconstructed image and the original image. Therefore, the loss function of the two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D is expressed as:

[0111] ;

[0112] in, , and is a weighting factor that balances the reconstruction and disentanglement performance;

[0113] S4: Build a training method based on curriculum learning to train the two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D;

[0114] In this embodiment, the focus of course learning is to first learn simple or basic concepts and then gradually learn more complex concepts. In the two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D, the interference effect of signal superposition is considered. Using the concept of course learning, first learn from the non-superposition channel, that is, the single-user joint source channel coding scheme, and then fine-tune according to future tasks. Therefore, the training of Two-view-DC-DeepJSCC-D includes three training stages, namely, the end-to-end training stage, the Two-view-DC-DeepJSCC training stage with reconstruction loss, and the Two-view-DC-DeepJSCC-D training stage with total loss.

[0115] In this embodiment, we first consider the single-user end-to-end model training to update each pair of joint source channel encoder and joint source channel decoder, where the loss function Defined as:

[0116] ;

[0117] Based on the loss function The gradient of updates the parameters of the single-user network, expressed as:

[0118] ;

[0119] ;

[0120] ;

[0121] in, , which gives:

[0122] ;

[0123] in is the activation function The derivative of and Represent element-wise multiplication and convolution of two tensors respectively. In addition, , Indicates rotating the matrix 180 degrees clockwise and then updating the network parameters and :

[0124] ;

[0125] ;

[0126] ;

[0127] in, is the learning rate;

[0128] For two-view cooperative transmission, the server aims to reconstruct the image of each view from the overlapping signals while improving the clustering and classification accuracy. Based on the trained single-user joint source channel coding model, the image reconstruction loss For training, the mean square error (MSE) is used to measure the difference between the reconstructed image and the original image;

[0129] Minimize total loss Get the optimal model parameters of Two-view-DC-DeepJSCC-D;

[0130] S5: After obtaining the trained two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D model, the server completes the image restoration, classification and clustering tasks;

[0131] The aerial aggregate information y is used to restore the two-view image through the joint source channel decoder, and the KNN algorithm is used to utilize the public information Classify and use k-means algorithm to classify public information Perform clustering.

[0132] As shown in Figure 3 (a) and Figure 3 (b), the present invention provides different transmission schemes in terms of compression ratio. , When , the training test images are the peak signal-to-noise ratios (PSNRs) at different signal-to-noise ratios on the Multi-COIL100 dataset. When , the reconstruction performance of Two-view-DC-DeepJSCC-D with information disentanglement is only 0.76 dB different from that of Two-view-DC-DeepJSCC. The degradation in reconstruction performance is due to the fact that constraints are imposed on the encoded information during model training to accommodate other tasks, thus sacrificing some accuracy in data recovery.

[0133] For clustering tasks, as shown in Table 1 below, the present invention provides ,The clustering performance of different schemes is on the Multi-COIL100 dataset with SNR=14 dB.,It can be seen from Table 1 that the two-view distributed collaborative joint source channel coding model Two-view-DC-DeepJSCC-D of the present invention outperforms other benchmark ,schemes in clustering tasks.

[0134] Table 1 Clustering performance comparison of different schemes at SNR=14 dB

[0135]

[0136] For classification tasks, DeepJSCC-SingleUser uses all encoding information, Two-view-DC-DeepJSCC-concat uses all aggregate information for classification, and the Two-view-DC-DeepJSCC-D scheme proposed in this embodiment only uses the public information in the aggregate information for classification tasks. Figure 4 As shown, despite using less information to distinguish categories, the classification performance of this embodiment is better than other schemes. Compared with the Two-view-DC-DeepJSCC scheme, the classification accuracy is improved by 5.6% on average, and it has better robustness to channel noise.

[0137] The experimental results show that the Two-view-DC-DeepJSCC-D scheme proposed in this embodiment can perform image restoration, classification and clustering tasks simultaneously. In addition, the proposed Two-view-DC-DeepJSCC-D not only maintains a competitive advantage in image restoration, but also has significant improvements in classification and clustering accuracy, and shows stronger robustness under various signal-to-noise ratios.

[0138] Example 2

[0139] This embodiment provides a collaborative non-orthogonal image semantic communication system based on information disentanglement, which is used to implement the collaborative non-orthogonal image semantic communication method based on information disentanglement of the above-mentioned embodiment 1, and the system includes: a target image acquisition module, a splicing module, a model construction module, a training method construction module, a model training module, an image restoration module, a classification module and a clustering module;

[0140] In this embodiment, the target image acquisition module is used to acquire two-view images of the target;

[0141] In this embodiment, the stitching module is used to connect the random Gaussian matrix with the image pixel matrix to obtain an image stitching matrix;

[0142] In this embodiment, the model construction module is used to construct a two-view distributed collaborative joint source channel coding model, including a joint source channel encoder, a de-entanglement module and a joint source channel decoder, wherein the de-entanglement module includes a private encoder and a public encoder;

[0143] In this embodiment, the training method construction module is used to construct a training method based on course learning;

[0144] In this embodiment, the model training module is used to train the two-view distributed collaborative joint source channel coding model, and complete the image restoration, classification and clustering tasks based on the trained two-view distributed collaborative joint source channel coding model;

[0145] In this embodiment, the image restoration module is used to map the image stitching matrix into semantic information through a joint source channel encoder, the semantic information is passed through a private encoder and a public encoder to obtain private information and public information, the private information and the public information are spliced ​​to obtain coded information, the coded information is subjected to air aggregation to obtain air aggregate information, and the air aggregate information is restored through a joint source channel decoder to obtain a two-view image;

[0146] In this embodiment, the classification module is used to take the public information as input and classify the image data based on the KNN algorithm;

[0147] In this embodiment, the clustering module is used to take the public information as input and obtain the clustering result of the image data based on the k-means clustering algorithm.

[0148] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A collaborative non-orthogonal image semantic communication method based on information disentanglement, characterized in that: The steps include: Get two-view images of the target, connect the random Gaussian matrix with the image pixel matrix to get the image stitching matrix; Constructing a two-view distributed collaborative joint source channel coding model, including a joint source channel encoder, a de-entanglement module and a joint source channel decoder, wherein the de-entanglement module includes a private encoder and a public encoder; A training method is constructed based on course learning to train a two-view distributed collaborative joint source channel coding model. Image restoration, classification, and clustering tasks are completed based on the trained two-view distributed collaborative joint source channel coding model. The image stitching matrix is ​​mapped into semantic information by a joint source-channel encoder. The semantic information is passed through a private encoder and a public encoder to obtain private information and public information. The private information and public information are concatenated to obtain coded information. The coded information is subjected to aerial aggregation to obtain aerial aggregated information. The aerial aggregated information is restored by a joint source-channel decoder to obtain a two-view image. Take public information as input and classify image data based on KNN algorithm; The public information is used as input and the clustering results of image data are obtained based on the k-means clustering algorithm.

2. The collaborative non-orthogonal image semantic communication method based on information disentanglement according to claim 1 is characterized in that: The joint source-channel encoder comprises: a first residual block upsampling, a first attention feature module, a second residual block upsampling, a second attention feature module, a first residual block, a third attention feature module, a second residual block, a fourth attention feature module and a third residual block upsampling; The image stitching matrix is ​​sequentially upsampled by the first residual block, the first attention feature module, the second residual block, the second attention feature module, the first residual block, the third attention feature module, the second residual block, the fourth attention feature module, and the third residual block to obtain semantic information.

3. The collaborative non-orthogonal image semantic communication method based on information disentanglement according to claim 1, characterized in that: The private information and public information are concatenated to obtain the encoded information, which is specifically expressed as: ; in, Indicates public information, , represents a complex set, Indicates private information. , and Respectively represent the length of the corresponding information.

4. The collaborative non-orthogonal image semantic communication method based on information disentanglement according to claim 3 is characterized in that: The coded information is aggregated over the air to obtain the air aggregate information, which is specifically expressed as: ; in, The variance is Additive Gaussian white noise, , represents the M-dimensional identity matrix, and are public aggregate information and noisy private aggregate information, represents the channel matrix between edge device i and the server, is the preprocessing function, i={1, 2}.

5. The collaborative non-orthogonal image semantic communication method based on information disentanglement according to claim 1, characterized in that: The joint source-channel decoder includes: a first residual transposed block upsampling, a fifth attention feature module, a second residual transposed block upsampling, a sixth attention feature module, a first residual transposed block, a seventh attention feature module, a second residual transposed block, an eighth attention feature module, and a third residual transposed block upsampling; The aerial aggregate information is sequentially upsampled by the first residual transposition block, the fifth attention feature module, the second residual transposition block, the sixth attention feature module, the first residual transposition block, the seventh attention feature module, the second residual transposition block, the eighth attention feature module and the third residual transposition block to restore the two-view image.

6. The collaborative non-orthogonal image semantic communication method based on information disentanglement according to claim 1, characterized in that: A training method based on curriculum learning is constructed to train a two-view distributed collaborative joint source channel coding model, including: end-to-end training phase, training phase with image reconstruction loss phase, and training phase with total loss phase; In the end-to-end training phase, each pair of joint source channel encoders and joint source channel decoders is updated based on the single-user joint source channel coding model; The mean square error (MSE) is used to measure the difference between the reconstructed image and the original image of the two views to obtain the image reconstruction loss. Based on the trained single-user joint source channel coding model, a distributed joint source channel coding model is trained based on the image reconstruction loss. The common encoder captures the common information across input views through consistency constraints, and obtains the common information CMD constraints between devices; The private encoder uses the Hilbert-Schmidt independence criterion to measure the difference between public information and private information within the device, and obtains the HSIC constraints of public information and private information; Based on the weighted sum of image reconstruction loss, common information CMD constraint and HSIC constraint, the total loss is obtained. The optimal model parameters of the two-view distributed collaborative joint source channel coding model are obtained by minimizing the total loss.

7. The collaborative non-orthogonal image semantic communication method based on information disentanglement according to claim 6, characterized in that: In the end-to-end training phase, each pair of joint source channel encoders and joint source channel decoders is updated based on the single-user joint source channel coding model, and its loss function is expressed as: ; in, represents the view image obtained by edge device i, i={1, 2}, Represents the restored view image.

8. The collaborative non-orthogonal image semantic communication method based on information disentanglement according to claim 6, characterized in that: The mean square error (MSE) is used to measure the gap between the reconstructed image and the original image, and the image reconstruction loss is obtained, which is specifically expressed as: ; in, represents the view image obtained by edge device i, i={1, 2}, Represents the restored view image.

9. The collaborative non-orthogonal image semantic communication method based on information disentanglement according to claim 8, characterized in that: Based on the weighted sum of image reconstruction loss, common information CMD constraint, and HSIC constraint, the total loss is obtained, which is specifically expressed as: ; ; ; ; ; ; ; ; ; ; in, 、 and Represent the corresponding weight values, Indicates the public information corresponding to edge device i The expected vector of Represents the public information corresponding to edge device j The expected vector of Indicates the public information corresponding to edge device i The k-th central moment of Represents the public information corresponding to edge device j The k-th central moment of Indicates public information, Indicates private information. and is through and The size of the inner product is n×n Gram matrix, represents the n-dimensional identity matrix, represents a full column vector, , and Respectively represent the information length corresponding to private information and public information.

10. A collaborative non-orthogonal image semantic communication system based on information disentanglement, characterized in that: For implementing the collaborative non-orthogonal image semantic communication method based on information disentanglement as described in any one of claims 1 to 9 above, the system comprises: a target image acquisition module, a splicing module, a model construction module, a training method construction module, a model training module, an image restoration module, a classification module and a clustering module; The target image acquisition module is used to acquire two-view images of the target; The stitching module is used to connect the random Gaussian matrix with the image pixel matrix to obtain an image stitching matrix; The model construction module is used to construct a two-view distributed collaborative joint source channel coding model, including a joint source channel encoder, a de-entanglement module and a joint source channel decoder, wherein the de-entanglement module includes a private encoder and a public encoder; The training method construction module is used to construct a training method based on course learning; The model training module is used to train the two-view distributed collaborative joint source channel coding model, and complete image restoration, classification and clustering tasks based on the trained two-view distributed collaborative joint source channel coding model; The image restoration module is configured to map the image stitching matrix into semantic information via a joint source-channel encoder, pass the semantic information through a private encoder and a public encoder to obtain private information and public information, concatenate the private information and the public information to obtain encoded information, perform air aggregation on the encoded information to obtain air aggregated information, and restore the air aggregated information via a joint source-channel decoder to obtain a two-view image; The classification module is used to take public information as input and classify image data based on the KNN algorithm; The clustering module is used to take public information as input and obtain clustering results of image data based on the k-means clustering algorithm.