Generative image coding knowledge base cross-domain synchronization method, system and medium

Abstract

The application provides a generative image coding knowledge base cross-domain synchronization method, system and medium. The generative image coding knowledge base cross-domain synchronization method guides the receiver to perform knowledge synchronization by sending a synchronization image. Since the noise resistance of the image itself is much higher than that of the neural network weight which is required to be precise, the noise resistance of the knowledge base cross-domain synchronization is improved. Secondly, in the application, the synchronization is performed only when the preset encoding number is reached, and only part of the image is used as an example to perform knowledge synchronization. Compared with the transmission of large model parameters, the bandwidth resource consumption is lower. Finally, the optimization strategy of meta learning is used to train the distribution transformation network, the knowledge bases of the two synchronization parties are better preserved, and the compatibility and scalability of the coding knowledge base cross-domain synchronization are improved.

Description

Technical Field

[0001] This invention belongs to the field of semantic communication technology, specifically relating to a method, system, and medium for cross-domain synchronization of generative image encoding and decoding knowledge bases. Background Technology

[0002] In recent years, with the improvement of graphics computing capabilities and the development of deep learning algorithms, various image communication methods based on deep learning have emerged. Deep learning-based communication systems rely on both the sender and receiver sharing the same encoding / decoding knowledge base. However, due to the limitations of the expressive power of neural networks, the training set cannot encompass all communication scenarios. Therefore, synchronizing the encoding / decoding knowledge base of neural networks across different data domains has become a major bottleneck affecting the performance of communication systems.

[0003] Synchronizing the encoding / decoding knowledge base between different data domains refers to the situation where the model weights of the receiver and sender are trained separately in non-independent and identically distributed data domains. For example, the receiver can encode and decode vehicle images, while the sender can encode and decode pedestrian images. In this case, the sender's encoded content cannot be recognized and correctly decoded by the receiver, thus requiring synchronization of the encoding / decoding knowledge base.

[0004] Existing synchronization methods typically involve directly transmitting neural network weights. First, the network weights are compressed and encoded, then transmitted through a channel. The receiver, upon receiving the parameters, decompresses and overwrites its own parameters. However, due to the large number of parameters in modern neural networks and the high requirements for floating-point precision, lossless compression and transmission require significant computational resources and channel bandwidth, and are easily affected by channel noise. Furthermore, overwriting the receiver's own parameters also overwrites its own encoding / decoding knowledge base, resulting in the loss of all network weights from the receiver and failing to achieve true knowledge synchronization. Meanwhile, deep learning-based synchronization methods require the aggregation and merging of model parameters. However, existing merging methods are limited by the black-box nature of neural networks, making it impossible to clearly define the parameter weights of the sender and receiver during parameter merging. This leads to one side's parameters dominating the synchronization process, preventing the aggregation of data knowledge within the parameters and ultimately resulting in synchronization failure.

[0005] Therefore, existing methods for cross-domain knowledge base synchronization suffer from high requirements for synchronization bandwidth, poor noise resistance, and the inability to integrate the knowledge bases of both parties. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a method, system, and medium for cross-domain synchronization of generative image encoding and decoding knowledge bases.

[0007] The technical problem to be solved by this invention is achieved through the following technical solution:

[0008] In a first aspect, the present invention provides a method for cross-domain synchronization of a generative image encoding / decoding knowledge base, comprising:

[0009] The preset synchronization condition is determined by the number of times the sender encodes the code.

[0010] When the preset synchronization condition is reached, the first synchronized image data is obtained through the first pre-trained distribution transformation network and the first pre-trained encoder-decoder knowledge base;

[0011] The first synchronized image data is sent to the receiver, and the second synchronized image data is obtained through the second pre-trained distribution transformation network and the second pre-trained encoding and decoding knowledge base;

[0012] The first synchronized image data and the second synchronized image data are merged to obtain synchronized merged data;

[0013] A portion of samples is randomly selected from the synchronized merged data and used as synchronized sampling data;

[0014] The synchronous sampling data is input into the second pre-trained encoder-decoder knowledge base to obtain synchronous prediction data;

[0015] By simultaneously sampling data and simultaneously predicting data, the parameters of the second pre-trained encoder-decoder knowledge base are optimized to obtain an optimized encoder-decoder knowledge base;

[0016] The optimized codec knowledge base will cover the second pre-trained codec knowledge base to complete the current cross-domain synchronization of the knowledge base;

[0017] The first pre-trained encoding / decoding knowledge base and the first pre-trained distribution transformation network are set at the sender, and the second pre-trained encoding / decoding knowledge base and the second pre-trained distribution transformation network are set at the receiver. The first pre-trained distribution transformation network and the second pre-trained distribution transformation network are obtained through meta-learning optimization training.

[0018] Optionally, the first pre-trained encoder-decoder knowledge base includes: a first pre-trained encoder knowledge base and a first pre-trained decoder knowledge base;

[0019] The second pre-trained encoder-decoder knowledge base includes: a second pre-trained encoder knowledge base and a second pre-trained decoder knowledge base.

[0020] Optionally, when a preset synchronization condition is reached, first synchronized image data is obtained through a first pre-trained distribution transformation network and a first pre-trained encoder-decoder knowledge base, including:

[0021] When the preset synchronization condition is reached, the first initial normal distribution data is sampled through the preset normal distribution sampling function in the first pre-trained distribution transformation network, and the first initial normal distribution data is input into the first pre-trained distribution transformation network to obtain the first normal distribution data;

[0022] The first normal distribution data is input into the first pre-trained decoding knowledge base to obtain the first synchronized image data.

[0023] Optionally, by synchronizing sampled data and synchronized prediction data, the parameters of the second pre-trained encoder-decoder knowledge base are optimized to obtain an optimized encoder-decoder knowledge base, including:

[0024] The mean squared error loss function (MSE) is used to calculate the loss between synchronously sampled data and synchronously predicted data to obtain the predicted loss value.

[0025] The optimized codec knowledge base is obtained by predicting the loss value and optimizing the parameters of the second pre-trained codec knowledge base using the backpropagation algorithm.

[0026] Optionally, both the first pre-trained distribution transformation network and the second pre-trained distribution transformation network include: a linear transformation layer, a sigmoid nonlinear transformation layer, and a self-attention layer;

[0027] The linear transformation layer, the sigmoid nonlinear transformation layer, and the self-attention layer are connected in series; the self-attention layer includes multiple attention heads.

[0028] Optionally, the training process of the first pre-trained distribution transformation network includes:

[0029] S201. Obtain the first training dataset A and the second training dataset B, and merge A and B to form the third training dataset AB;

[0030] S202. Use A to train the first initial codec knowledge base to obtain the first pre-trained codec knowledge base; the first initial codec knowledge base includes: the first initial encoding knowledge base and the first initial decoding knowledge base;

[0031] S203. Use B to train the second initial codec knowledge base to obtain the second intermediate codec knowledge base; the second intermediate codec knowledge base includes: the second intermediate encoding knowledge base and the second intermediate decoding knowledge base;

[0032] S204. Obtain the normal distribution training samples (h) through the first initial distribution transformation network and the preset normal distribution sampling function inside the first initial distribution transformation network. 1 ,h 2 ,h 3 ); where h 1 h 2 and h 3 These represent training samples 1, 2, and 3, respectively, which are normally distributed training samples.

[0033] S205, h 1Input the first pre-trained decoding knowledge base to obtain the first decoding training image a;

[0034] S206, h 2 Input the second intermediate decoding knowledge base to obtain the second decoding training image b;

[0035] S207. Add Gaussian noise to a and perform union processing with b to obtain the third training image ab.

[0036] S208. Randomly select a portion of samples from ab as the first sampled training image x;

[0037] S209. Input x into the second intermediate encoder-decoder knowledge base to obtain the prediction training image.

[0038] S210, Calculate x and using Mean Squared Error Loss (MSE) The training loss value between;

[0039] S211. Optimize the parameters of the second intermediate codec knowledge base by training loss value and backpropagation algorithm to obtain the second pre-trained codec knowledge base;

[0040] S212, h 3 Input the second pre-trained encoder-decoder knowledge base to obtain the test image TestAB;

[0041] S213. Using TestAB and AB, optimize the parameters of the first initial distribution transformation network to obtain the third loss value;

[0042] S214. Repeat the process S204-S213 until the third loss value is less than the preset loss threshold, and use the first initial distribution transformation network corresponding to the third loss value being less than the preset loss threshold as the first pre-trained distribution transformation network.

[0043] Optionally, A is used to train the first initial codec knowledge base to obtain a first pre-trained codec knowledge base, including:

[0044] S301. Randomly sample a portion of data from A to use as the second sampled training image;

[0045] S302. Input the second sampled training image into the first initial coding knowledge base to obtain the second training coded image;

[0046] S303. Input the second training encoded image into the first initial decoding knowledge base to obtain the second training decoded image;

[0047] S304. The mean square error loss function (MSE) is used to calculate the fourth loss value between the second sampled training image and the second trained decoded image, and the parameters of the first initial encoding and decoding knowledge base are optimized by the backpropagation algorithm and the fourth loss value.

[0048] S305. Repeat steps S301-S304 until the first initial codec knowledge base reaches the preset convergence condition, and use the first initial codec knowledge base that meets the preset convergence condition as the first pre-trained codec knowledge base.

[0049] Optionally, using TestAB and AB, the parameters of the first initial distribution transformation network are optimized to obtain a third loss value, including:

[0050] Image data of equal quantity to TestAB is sampled from AB and used as the third sampling training data;

[0051] The third loss value is obtained by calculating the loss between the third sampled training data and TestAB using LPIPS loss.

[0052] In a second aspect, the present invention provides a cross-domain synchronization system for a generative image encoding and decoding knowledge base, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the generative image encoding and decoding knowledge base cross-domain synchronization system is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method described in the first aspect above.

[0053] Thirdly, the present invention provides a storage medium on which a computer program is stored, and the computer program is executed by a processor to perform the steps of the method described in the first aspect above.

[0054] This invention provides a method, system, and medium for cross-domain synchronization of a generative image encoding / decoding knowledge base. The method includes: determining whether a preset synchronization condition has been reached based on the number of encoding attempts by the sender; when the preset synchronization condition is reached, acquiring first synchronized image data using a first pre-trained distribution transformation network and a first pre-trained encoding / decoding knowledge base; sending the first synchronized image data to the receiver and obtaining second synchronized image data using a second pre-trained distribution transformation network and a second pre-trained encoding / decoding knowledge base; merging the first and second synchronized image data to obtain synchronized merged data; and randomly selecting a portion of samples from the synchronized merged data as synchronized sampling data. Synchronous sampling data is input into the second pre-trained codec knowledge base to obtain synchronous prediction data. The parameters of the second pre-trained codec knowledge base are optimized using the synchronous sampling data and synchronous prediction data to obtain an optimized codec knowledge base. The optimized codec knowledge base is then overlaid on the second pre-trained codec knowledge base to complete the current cross-domain synchronization of the knowledge base. The first pre-trained codec knowledge base and the first pre-trained distribution transformation network are set at the sender, and the second pre-trained codec knowledge base and the second pre-trained distribution transformation network are set at the receiver. The first and second pre-trained distribution transformation networks are obtained through meta-learning optimization training. In this invention, knowledge synchronization is guided to the receiver by sending synchronized images. Since the noise resistance of the images themselves is much higher than that of the precision-critical neural network weights, the noise resistance of cross-domain knowledge base synchronization is improved. Secondly, since synchronization is only performed after a preset number of encodings and only a portion of the images (the first synchronized image) are used as examples for knowledge synchronization, it consumes less bandwidth resources compared to transmitting a large number of model parameters. Finally, a meta-learning optimization strategy is used to train the distributed transformation network, which better preserves the knowledge bases of both synchronizing parties and improves the compatibility and scalability of cross-domain synchronization of the encoding and decoding knowledge bases.

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0056] Figure 1 A flowchart illustrating a cross-domain synchronization method for a generative image encoding / decoding knowledge base provided in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of a cross-domain synchronization system for a generative image encoding and decoding knowledge base provided in an embodiment of the present invention. Detailed Implementation

[0058] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0059] To improve the noise resistance, compatibility, and scalability of cross-domain synchronization of knowledge bases, while reducing bandwidth resource consumption, this invention provides a method for cross-domain synchronization of generative image encoding and decoding knowledge bases. Figure 1 This is a flowchart illustrating a cross-domain synchronization method for a generative image encoding / decoding knowledge base, provided as an embodiment of the present invention. Figure 1 As shown, it includes:

[0060] S101. Determine whether the preset synchronization condition has been reached by the number of times the sender encodes.

[0061] In this embodiment, the system determines whether the number of encoding operations performed by the sender has reached a preset encoding threshold. If the preset encoding threshold is reached, the preset synchronization condition is determined to have been met. For example, the preset encoding threshold can be 20 times, but the specific value can be flexibly set according to the synchronization requirements and device performance. This embodiment does not limit this value.

[0062] S102. When the preset synchronization condition is reached, the first synchronized image data is obtained through the first pre-trained distribution transformation network and the first pre-trained encoding and decoding knowledge base.

[0063] The first pre-trained encoder-decoder knowledge base includes: a first pre-trained encoder knowledge base and a first pre-trained decoder knowledge base;

[0064] The second pre-trained encoder-decoder knowledge base includes: a second pre-trained encoder knowledge base and a second pre-trained decoder knowledge base.

[0065] Both the first and second pre-trained distribution transformation networks include: a linear transformation layer, a sigmoid non-linear transformation layer, and a self-attention layer;

[0066] The linear transformation layer, the sigmoid nonlinear transformation layer, and the self-attention layer are connected in series; the self-attention layer includes multiple attention heads.

[0067] It should be noted that, in this embodiment, the dimension of the linear transformation layer in the distribution transformation network (first pre-trained distribution transformation network and second pre-trained distribution transformation network) needs to be the same as the input dimension in the encoding and decoding knowledge base.

[0068] In this embodiment, when the preset synchronization condition is not met, when the receiver receives the processing task from the sender, it can directly use the receiver's current second pre-trained codec knowledge base to process the task.

[0069] Furthermore, in this embodiment, the encoding / decoding modules in a variational autoencoder can be used as the encoding / decoding knowledge base for both the sender and receiver. Specifically, the encoding knowledge base may include: four sets of 3x3 convolutional kernels, a batch normalization layer, a non-linear activation function ReLU, and three sets of residual modules also with 3x3 convolutional kernels. A variational reparameterization module is also provided at the end of the encoding knowledge base. The decoding knowledge base may include: a set of parameterless self-attention mechanism layers (used to remap the decoded input), four sets of deconvolutional modules symmetrical to the encoding knowledge base, a batch normalization module, a non-linear activation function ReLU, and three sets of residual modules with deconvolution. Additionally, it should be noted that the first and second pre-trained encoding / decoding knowledge bases in this invention can be any neural network model with encoding and decoding capabilities; the specific model structure is not limited to the structure described in the above embodiment.

[0070] In addition, in this embodiment, the Xavier regularization initialization method can be used to reset the parameters of the distribution transformation network before training begins, so as to achieve the purpose of quickly training the distribution transformation network.

[0071] Step S102 may specifically include:

[0072] When the preset synchronization condition is reached, the first initial normal distribution data is sampled through the preset normal distribution sampling function in the first pre-trained distribution transformation network, and the first initial normal distribution data is input into the first pre-trained distribution transformation network to obtain the first normal distribution data;

[0073] The first normal distribution data is input into the first pre-trained decoding knowledge base to obtain the first synchronized image data.

[0074] S103. The first synchronized image data is sent to the receiver, and the second synchronized image data is obtained through the second pre-trained distribution transformation network and the second pre-trained encoding and decoding knowledge base.

[0075] S104. Merge the first synchronized image data and the second synchronized image data to obtain synchronized merged data.

[0076] S105. Randomly select a portion of samples from the synchronized merged data as synchronized sampling data.

[0077] S106. Input the synchronous sampling data into the second pre-trained encoder-decoder knowledge base to obtain synchronous prediction data.

[0078] S107. By synchronizing the sampled data and the predicted data, the parameters of the second pre-trained codec knowledge base are optimized to obtain the optimized codec knowledge base.

[0079] Step S107 may specifically include: using the mean squared error loss function (MSE) to calculate the loss value between the synchronously sampled data and the synchronously predicted data to obtain the predicted loss value; and optimizing the parameters of the second pre-trained codec knowledge base using the predicted loss value and the backpropagation algorithm to obtain the optimized codec knowledge base.

[0080] S108. Optimize the codec knowledge base to cover the second pre-trained codec knowledge base in order to complete the current cross-domain synchronization of the knowledge base.

[0081] The first pre-trained encoding / decoding knowledge base and the first pre-trained distribution transformation network are set at the sender, and the second pre-trained encoding / decoding knowledge base and the second pre-trained distribution transformation network are set at the receiver. The first pre-trained distribution transformation network and the second pre-trained distribution transformation network are obtained through meta-learning optimization training.

[0082] This invention provides a cross-domain synchronization method for a generative image encoding / decoding knowledge base, comprising: determining whether a preset synchronization condition has been reached based on the number of encoding attempts by the sender; when the preset synchronization condition is reached, acquiring first synchronized image data through a first pre-trained distribution transformation network and a first pre-trained encoding / decoding knowledge base; sending the first synchronized image data to a receiver, and obtaining second synchronized image data through a second pre-trained distribution transformation network and a second pre-trained encoding / decoding knowledge base; merging the first synchronized image data and the second synchronized image data to obtain synchronized merged data; and randomly selecting a portion of samples from the synchronized merged data as synchronized samples. The first pre-trained codec knowledge base and the first pre-trained distribution transformation network are set at the sender, and the second pre-trained codec knowledge base and the second pre-trained distribution transformation network are set at the receiver. The first pre-trained distribution transformation network and the second pre-trained distribution transformation network are obtained through meta-learning optimization training. In this embodiment of the invention, knowledge synchronization is guided to the receiver by sending synchronized images. Since the noise resistance of the images themselves is much higher than that of the neural network weights, which have stringent accuracy requirements, the noise resistance of cross-domain knowledge base synchronization is improved. Secondly, since synchronization is only performed after a preset number of encodings and only a portion of the images (the first synchronized image) are used as examples for knowledge synchronization, it consumes less bandwidth resources compared to transmitting a large number of model parameters. Finally, the training of the distributed transformation network is carried out using a meta-learning optimization strategy, which better preserves the knowledge bases of both synchronizing parties and improves the compatibility and scalability of cross-domain synchronization of the encoding and decoding knowledge bases.

[0083] The training process of the first pre-trained distribution transformation network includes:

[0084] S201. Obtain the first training dataset A and the second training dataset B, and merge A and B to form the third training dataset AB.

[0085] It should be noted that, in this embodiment, the first training dataset A and the second training dataset B can be datasets with significant differences. For example, the first training dataset A can be pedestrian image data, and the second training dataset B can be vehicle image data.

[0086] S202. Use A to train the first initial codec knowledge base to obtain the first pre-trained codec knowledge base; the first initial codec knowledge base includes: the first initial encoding knowledge base and the first initial decoding knowledge base.

[0087] S203. Use B to train the second initial codec knowledge base to obtain the second intermediate codec knowledge base.

[0088] The second intermediate encoding / decoding knowledge base includes: the second intermediate encoding knowledge base and the second intermediate decoding knowledge base.

[0089] S204. Obtain the normal distribution training samples (h) through the first initial distribution transformation network and the preset normal distribution sampling function inside the first initial distribution transformation network. 1 ,h 2 ,h 3 ).

[0090] Among them, h 1 h 2 and h 3 These represent normal distribution training sample one, normal distribution training sample two, and normal distribution training sample three, respectively.

[0091] It should be noted that the first initial distribution transformation network contains a preset normal distribution sampling interface, through which a preset number of standard normal distribution sample vectors can be sampled.

[0092] S205, h 1 Input the first pre-trained decoding knowledge base to obtain the first decoding training image a.

[0093] S206, h 2 Input the second intermediate decoding knowledge base to obtain the second decoding training image b.

[0094] S207. Add Gaussian noise to a and perform union with b to obtain the third training image ab.

[0095] S208. Randomly select a portion of samples from ab as the first sampled training image x.

[0096] S209. Input x into the second intermediate encoder-decoder knowledge base to obtain the prediction training image.

[0097] S210, Calculate x and using Mean Squared Error Loss (MSE) The training loss value between them.

[0098] S211. Optimize the parameters of the second intermediate codec knowledge base by training loss values ​​and backpropagation algorithm to obtain the second pre-trained codec knowledge base.

[0099] S212, h 3 Input the second pre-trained encoder-decoder knowledge base to obtain the test image TestAB.

[0100] S213. Using TestAB and AB, optimize the parameters of the first initial distribution transformation network to obtain the third loss value.

[0101] S214. Repeat the process S204-S213 until the third loss value is less than the preset loss threshold, and use the first initial distribution transformation network corresponding to the third loss value being less than the preset loss threshold as the first pre-trained distribution transformation network.

[0102] Step S202 may specifically include:

[0103] S301. Randomly sample a portion of the data from A to use as the second sampled training image.

[0104] S302. Input the second sampled training image into the first initial coding knowledge base to obtain the second training coded image.

[0105] S303. Input the second training encoded image into the first initial decoding knowledge base to obtain the second training decoded image.

[0106] S304. The mean square error loss function (MSE) is used to calculate the fourth loss value between the second sampled training image and the second trained decoded image, and the parameters of the first initial encoding / decoding knowledge base are optimized through the backpropagation algorithm and the fourth loss value.

[0107] S305. Repeat steps S301-S304 until the first initial codec knowledge base reaches the preset convergence condition, and use the first initial codec knowledge base that meets the preset convergence condition as the first pre-trained codec knowledge base.

[0108] Step S213 may specifically include: sampling image data from AB in an amount equal to that of TestAB, and using it as the third sampling training data;

[0109] The third loss value is obtained by calculating the loss between the third sampled training data and TestAB using LPIPS loss.

[0110] Based on the above implementation scheme of the present invention, the following beneficial effects can be achieved:

[0111] First, the method provided in this embodiment of the invention guides the receiver to perform knowledge synchronization by sending generated images. Only image data is transmitted, avoiding direct transmission of weights. The noise resistance of the image itself is much higher than that of the neural network weights, which have strict requirements for accuracy, thus significantly improving the noise resistance.

[0112] Secondly, the method provided in this embodiment of the invention can achieve encoding and decoding knowledge synchronization by transmitting only one example image (first synchronized image data) in scenarios where there is no restriction on the diversity of transmitted images. Even if multiple images are sent, the size of the compressed image still has lower bandwidth resource consumption compared to the large number of model parameters.

[0113] Third, the method provided in this embodiment of the invention better preserves the knowledge bases of both synchronizing parties. Specifically, existing methods cannot merge the knowledge bases of the sender and receiver; they can only retain the weight of one party while discarding the other. The meta-learning optimization strategy proposed in this embodiment of the invention can guide the sender to generate optimal data, and the receiver can retain its original encoding and decoding capabilities while adding encoding and decoding capabilities for unknown data domains during synchronization, thus exhibiting better compatibility and scalability.

[0114] To verify the reliability of the cross-domain synchronization method for generative image encoding / decoding knowledge base provided in this embodiment of the invention, experimental verification was also conducted, as follows:

[0115] 1. The test datasets used were the German road sign dataset GTSRB and the commonly used image classification dataset CIFAR. For the GTSRB dataset, classes with more than 1400 images (9 classes in total) were used, dividing it into dataset A (including classes (0, 1, 2, 3, 4, 5)) and dataset B (including classes (3, 4, 5, 6, 7, 8)). For the CIFAR dataset, only the test set containing 10,000 images was used. Images with class numbers (3, 6, 9) (corresponding to classes "cat", "frog", and "truck") were used as dataset A, and images of the remaining classes (including classes (0, 1, 4, 5, 7, 8)) were used as dataset B. End-to-end backpropagation was used to optimize and train the first and second pre-trained encoder-decoder knowledge bases on datasets A and B respectively. The optimizer used was Adam with a learning rate of 0.00045, and the loss function was the structural similarity index SSIM between the decoded and encoded images.

[0116] 2. Deploy the first pre-trained codec knowledge base on the sender, the second pre-trained codec knowledge base on the receiver, and deploy the distributed transformation network on both the sender and receiver.

[0117] 3. Sample K standard normal distributions of dimension L from the distribution transformation network and perform distribution transformation to h. l The sender's first pre-trained codec knowledge base utilizes h l Generate K samples Channel noise is added, and the receiver generates K samples from its second pre-trained encoder-decoder knowledge base. Will After merging the samples, the second pre-trained codec knowledge base is trained end-to-end through backpropagation with a training round of 1, resulting in the second optimized codec knowledge base.

[0118] 4. The sender transmits the encoding result from the first codec knowledge base, and the receiver uses the second optimized codec knowledge base to decode the received signal to obtain the decoding result, and measures the structural similarity index (SSIM) between the decoded result and the transmitted result. Table 1 shows a comparison between the synchronization result and the result of directly transmitting parameters in this example. When directly transmitting parameters, the SSIM index of all data distributions is also calculated and averaged.

[0119] Table 1. Synchronization results of the present invention on two datasets

[0120]

[0121]

[0122] Table 1 shows the data packet size, reflecting the different amounts of synchronized image data in the first synchronization. The sender-receiver SSIM represents the average SSIM of images across all categories after synchronization using the method of this invention. The direct parameter transmission SSIM represents the average SSIM of directly transmitted parameters (existing technology) across all categories. As can be seen from Table 1, this invention is compared with the direct parameter transmission method from three perspectives: bandwidth usage, noise immunity, and parameter fusion effect. The method of directly passing parameters is adopted. The encoding and decoding parameters trained on the GTSRB dataset are packaged and compressed using the xz compression algorithm to obtain a data packet with a bandwidth of 12.4MB. This data packet is transmitted in AWGN simulated channel environments with channel noise of 7dB, 10dB and 13dB respectively. After receiving the data packet in each channel environment, the receiver decompresses it and overwrites its own parameters. The SSIM scores of the generated images and real images in the decoding knowledge base are calculated on the nine image categories selected by GTSRB respectively. Finally, the average of the SSIM scores of the nine image categories is approximately 0.46, 0.53 and 0.53 respectively (it should be noted that the SSIM score must be calculated according to the category, that is, the generated images and data samples of the same category are calculated. Finally, GTSRB can calculate nine SSIM scores and calculate a weighted average of these nine SSIM scores). By controlling the size of the transmitted data, this invention achieves an average SSIM score of approximately 0.62 for both the generated and real images when transmitting only about 2.87MB of data (approximately 10 generated images). This saves 9.53MB of bandwidth compared to the method of directly transmitting parameters. Under three different channel noise environments, this invention achieves an average SSIM score of 0.80 when transmitting approximately 10.56MB of image data (approximately 50 generated images), resulting in relatively clear images. Similar results were obtained on the CIFAR dataset, where directly transmitting parameters consumed approximately 11.8MB of bandwidth, while this invention only requires approximately 2.65MB, saving approximately 9.15MB. Furthermore, the quality of the generated images is significantly higher than that of the method of directly transmitting parameters, as evidenced by the higher average SSIM score of the generated and real images after synchronization. In terms of bandwidth usage and noise resistance, the method of directly transmitting parameters is inferior to this invention. Moreover, this invention does not require overwriting the receiver's own knowledge base, resulting in better parameter fusion performance.

[0123] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0124] Based on the same inventive concept, embodiments of the present invention also provide a cross-domain synchronization system for generative image encoding and decoding knowledge bases. Figure 2This is a schematic diagram of a cross-domain synchronization system for a generative image encoding / decoding knowledge base provided in an embodiment of the present invention. It includes a processor 210, a storage medium 220, and a bus 230. The storage medium 220 stores machine-readable instructions executable by the processor 210. When the generative image encoding / decoding knowledge base cross-domain synchronization system is running, the processor 210 and the storage medium 220 communicate via the bus 230. The processor 210 executes the machine-readable instructions to perform the steps of the above-described method embodiment. The specific implementation and technical effects are similar and will not be described in detail here.

[0125] The storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the storage medium may also be at least one storage device located remotely from the aforementioned processor.

[0126] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0127] The present invention also provides a computer-readable storage medium. A computer program is stored in this computer-readable storage medium, and when executed by a processor, the computer program implements the steps of any of the above-described generative image encoding / decoding knowledge base cross-domain synchronization methods.

[0128] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0129] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0130] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0131] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for cross-domain synchronization of generative image encoding / decoding knowledge bases, characterized in that, include: The preset synchronization condition is determined by the number of times the sender encodes the code. When the preset synchronization condition is reached, the first synchronized image data is obtained through the first pre-trained distribution transformation network and the first pre-trained encoder-decoder knowledge base; The first synchronized image data is sent to the receiver, and the second synchronized image data is obtained through the second pre-trained distribution transformation network and the second pre-trained encoding and decoding knowledge base; The first synchronized image data and the second synchronized image data are merged to obtain synchronized merged data; Randomly select a portion of samples from the synchronized merged data as synchronized sampling data; The synchronous sampling data is input into the second pre-trained encoder-decoder knowledge base to obtain synchronous prediction data; By using the synchronous sampling data and the synchronous prediction data, the parameters of the second pre-trained encoder-decoder knowledge base are optimized to obtain an optimized encoder-decoder knowledge base; The optimized codec knowledge base is overlaid on the second pre-trained codec knowledge base to complete the current cross-domain synchronization of the knowledge base; The first pre-trained codec knowledge base and the first pre-trained distribution transformation network are set at the sender, and the second pre-trained codec knowledge base and the second pre-trained distribution transformation network are set at the receiver. The first pre-trained distribution transformation network and the second pre-trained distribution transformation network are obtained through meta-learning optimization training. The training process of the first pre-trained distribution transformation network includes: S201. Obtain the first training dataset A and the second training dataset B, and merge A and B to form a third training dataset AB; S202. The first initial codec knowledge base is trained using A to obtain the first pre-trained codec knowledge base; the first initial codec knowledge base includes: a first initial encoding knowledge base and a first initial decoding knowledge base. S203. The second initial encoding / decoding knowledge base is trained using the B to obtain the second intermediate encoding / decoding knowledge base; the second intermediate encoding / decoding knowledge base includes: a second intermediate encoding knowledge base and a second intermediate decoding knowledge base; S204. Obtain normal distribution training samples through the first initial distribution transformation network and the preset normal distribution sampling function inside the first initial distribution transformation network. ;in, , and These represent training samples 1, 2, and 3, respectively, which are normally distributed training samples. S205, will Input the first pre-trained decoding knowledge base to obtain the first decoding training image a; S206, will Input the second intermediate decoding knowledge base to obtain the second decoding training image b; S207. Add Gaussian noise to a and perform union processing with b to obtain the third training image ab; S208. Randomly select a portion of samples from the ab as the first sampling training images. ; S209, the above Input the second intermediate encoder-decoder knowledge base to obtain the prediction training image. ; S210, Calculate the following using the mean squared error loss (MSE) With the The training loss value between; S211. Optimize the parameters of the second intermediate codec knowledge base using the training loss value and the backpropagation algorithm to obtain the second pre-trained codec knowledge base; S212, the above Input the second pre-trained encoder-decoder knowledge base to obtain the test image TestAB; S213. Using the TestAB and the AB, optimize the parameters of the first initial distribution transformation network to obtain a third loss value; S214. Repeat the process S204-S213 until the third loss value is less than the preset loss threshold, and use the first initial distribution transformation network corresponding to the third loss value being less than the preset loss threshold as the first pre-trained distribution transformation network.

2. The method for cross-domain synchronization of a generative image encoding / decoding knowledge base according to claim 1, characterized in that, The first pre-trained encoder-decoder knowledge base includes: a first pre-trained encoder knowledge base and a first pre-trained decoder knowledge base; The second pre-trained encoder-decoder knowledge base includes: a second pre-trained encoder knowledge base and a second pre-trained decoder knowledge base.

3. The method for cross-domain synchronization of a generative image encoding / decoding knowledge base according to claim 2, characterized in that, When the preset synchronization condition is reached, the first synchronized image data is obtained through the first pre-trained distribution transformation network and the first pre-trained encoder-decoder knowledge base, including: When the preset synchronization condition is reached, the first initial normal distribution data is sampled through the preset normal distribution sampling function in the first pre-trained distribution transformation network, and the first initial normal distribution data is input into the first pre-trained distribution transformation network to obtain the first normal distribution data; The first normal distribution data is input into the first pre-trained decoding knowledge base to obtain the first synchronized image data.

4. The method for cross-domain synchronization of a generative image encoding / decoding knowledge base according to claim 2, characterized in that, The step of optimizing the parameters of the second pre-trained encoder-decoder knowledge base using the synchronous sampling data and the synchronous prediction data to obtain an optimized encoder-decoder knowledge base includes: The predicted loss value is obtained by calculating the loss value between the synchronous sampled data and the synchronous predicted data using the mean squared error loss function (MSE). The optimized codec knowledge base is obtained by optimizing the parameters of the second pre-trained codec knowledge base using the predicted loss value and the backpropagation algorithm.

5. The method for cross-domain synchronization of a generative image encoding / decoding knowledge base according to claim 1, characterized in that, Both the first pre-trained distribution transformation network and the second pre-trained distribution transformation network include: a linear transformation layer, a sigmoid nonlinear transformation layer, and a self-attention layer; The linear transformation layer, the sigmoid nonlinear transformation layer, and the self-attention layer are connected in series; the self-attention layer includes multiple attention heads.

6. The method for cross-domain synchronization of a generative image encoding / decoding knowledge base according to claim 1, characterized in that, The step of training the first initial codec knowledge base using A to obtain the first pre-trained codec knowledge base includes: S301. Randomly sample a portion of data from A to obtain a second sampled training image; S302. Input the second sampled training image into the first initial coding knowledge base to obtain the second training coded image; S303. Input the second training encoded image into the first initial decoding knowledge base to obtain the second training decoded image; S304. The mean square error loss function (MSE) is used to calculate the fourth loss value between the second sampled training image and the second training decoded image, and the parameters of the first initial encoding / decoding knowledge base are optimized by the backpropagation algorithm and the fourth loss value. S305. Repeat steps S301-S304 until the first initial codec knowledge base reaches the preset convergence condition, and use the first initial codec knowledge base that meets the preset convergence condition as the first pre-trained codec knowledge base.

7. The method for cross-domain synchronization of a generative image encoding / decoding knowledge base according to claim 1, characterized in that, The step of optimizing the parameters of the first initial distribution transformation network using the TestAB and the AB to obtain a third loss value includes: Image data of equal quantity to TestAB are sampled from AB and used as the third sampling training data; The third loss value is obtained by calculating the loss value between the third sampled training data and the TestAB using LPIPS loss.

8. A cross-domain synchronization system for a generative image encoding / decoding knowledge base, characterized in that, include: The system includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the generative image encoding / decoding knowledge base cross-domain synchronization system is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method as described in any one of claims 1-7.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, performs the steps of the method as described in any one of claims 1-7.

Images