Image coding method and system
By jointly using decoupled parametric semantics with textual semantic descriptions and semantic attribute value sets in semantic communication, the cross-modal and cross-domain generalization problem at extremely low bit rates is solved, achieving high-fidelity image reconstruction and improving the robustness of the communication system.
Patent Information
- Application Number
- CN202511583192.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-31
- Publication Date
- 2026-01-23
AI Technical Summary
Existing semantic communication methods lack cross-modal and cross-domain generalization capabilities at extremely low bit rates, making it difficult to adapt to packet loss and distortion in real-world communication. Furthermore, relying solely on textual semantic descriptions leads to the loss of image details, affecting reconstruction results.
By jointly using decoupled parametric semantics with textual semantic descriptions and semantic attribute value sets, high-quality semantic reconstruction of images is achieved through collaborative transmission and filling of structured latent space under limited channel conditions. Cross-modal semantic alignment technology is used to fuse and complete missing information at the receiving end.
Achieving high-fidelity semantic reconstruction of images at extremely low bit rates improves the robustness of communication systems and enables adaptive switching of coding methods.
Smart Images
Figure CN121392010A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, in particular to an image encoding method and system, and particularly relates to a cross-code rate domain progressive image encoding method and system based on hidden space cross-modal alignment, which is suitable for semantic image communication and reconstruction under extremely low code rate. BACKGROUND
[0002] Semantic communication, as a new communication paradigm with semantic level as the core, has attracted much attention in both academia and industry in recent years. Unlike traditional communication systems that aim to accurately transmit bit data, the core of semantic communication is to ensure accurate reconstruction of information semantic properties and correct execution of the receiving end task. Through in-depth understanding of information semantics and dynamic feedback optimization of receiving end task requirements, efficient encoding and transmission of key semantic information are achieved. Compared with traditional communication systems, semantic communication has unique advantages in reducing data redundancy, improving spectral efficiency and energy efficiency, and has broad application prospects in resource-scarce communication scenarios such as underwater communication and deep space communication.
[0003] Existing research shows that the method of semantic communication can reduce the communication code rate to the ultra-low code rate domain (0.1 > Bpp > 0.01). This method encodes the original image data into a hidden space vector and combines feature decoupling technology to achieve encoding and transmission that takes into account ultra-low compression rate and image semantic reconstruction quality. However, this method relies heavily on attribute extraction models in specific fields and lacks cross-modal and cross-domain generalization ability. At the same time, this method has high requirements for the integrity of the transmitted data, and it is difficult to adapt to situations such as packet loss and distortion that may occur in actual communication.
[0004] Text semantic description information has stronger abstraction and universality, and its data volume is much smaller than that of images and hidden space features, which helps to adapt to the demand for extremely low code rate transmission. However, relying solely on text semantic description information will result in the loss of image details and affect the reconstruction effect. Therefore, it is urgent to provide a cross-modal encoding strategy that combines structured semantics and text information to achieve a semantic high-fidelity and extremely low code rate domain image communication method. SUMMARY
[0005] In view of the above deficiencies in the prior art, the present application provides an image encoding method and system, which jointly uses decoupled parameter semantics and text semantic description and semantic attribute value set, and when the channel condition is limited, uses semantic attribute values and text information to cooperatively transmit and fill the structured hidden space, thereby realizing high-quality semantic reconstruction of images.
[0006] To achieve the above purpose, the present application is implemented by the following technical solutions: In a first aspect, the present application provides an image encoding method, which comprises the following steps: Design semantic attribute structured latent space; At the sending end, the original image data is encoded into corresponding text semantic description, decoupled parameter semantic description and semantic attribute value set, and the required transmission data is selected and sent according to the feedback of the channel parameter; After the receiving end receives the transmission data, the corresponding latent space data is obtained according to the different forms of semantic description information; after the required latent space of different semantic forms is obtained, the latent space of different semantic forms is fused into a complete latent space for image information reconstruction, and the image information reconstruction is performed according to the complete latent space.
[0007] In a second aspect, the application provides an image encoding system, the system comprising: A latent space design module for designing a semantic attribute structured latent space; A multi-state semantic information encoding module for encoding the original image data into corresponding text semantic description, decoupled parameter semantic description and semantic attribute value set at the sending end, and selecting and sending the required transmission data according to the feedback of the channel parameter; A multi-state semantic fusion decoding module for obtaining the corresponding latent space data according to the different forms of semantic description information after the receiving end receives the transmission data; after the required latent space of different semantic forms is obtained, the latent space of different semantic forms is fused into a complete latent space for image information reconstruction, and the image information reconstruction is performed according to the complete latent space.
[0008] The application proposes to use text semantic description, decoupled parameter semantic description and semantic attribute value to cooperatively encode and transmit, and to adaptively select the optimal encoding mode under different channel conditions, to realize image semantic compression transmission under extremely low code rate, by using the decoupled expression ability based on structured latent space. At the receiving end, the text and attribute semantics are mapped back to the structured latent space based on the cross-modal semantic alignment technology, and the missing information is fused and completed, to complete the image reconstruction. Therefore, by using the technical solution of the application, at least the following beneficial effects are achieved: 1. Image semantic high-fidelity reconstruction can be realized under extremely low code rate transmission conditions; 2. The cross-modal alignment mechanism can effectively improve the robustness of the communication system; 3. Adaptive switching from parameter semantic dominance to text semantic dominance is realized.
[0009] The above description is only a summary of the technical solution of the application, in order to more clearly understand the technical means of the application, the application can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the application more obvious and easy to understand, the specific embodiments of the application are described below. BRIEF DESCRIPTION OF DRAWINGS
[0010] Figure 1 is an execution flowchart of the image encoding method in the first embodiment of the application. Figure 2 A framework diagram for text semantic guided cross-modal aligned progressive image coding of the present application; Figure 3 A training flowchart for image-latent space encoder; Figure 4 A training flowchart for text-latent space encoder; Figure 5 A training flowchart for semantic attribute value-latent space encoder; Figure 6 A training flowchart for latent space-image decoder; Figure 7 A structural schematic diagram for image coding in embodiment one of the present application. DETAILED DESCRIPTION
[0011] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application. Embodiment one
[0012] The present embodiment provides an image coding method, as shown in Figure 1 and Figure 2 The method comprises the following steps: Step S1, designing a semantic attribute structured latent space; Step S2, at the sending end, encoding the original image data into corresponding text semantic description, decoupled parameter semantic description and semantic attribute value set, and selecting the required transmission data for sending according to the feedback of the channel parameters; Step S3, after receiving the transmission data, the receiving end obtains the corresponding latent space data according to different forms of semantic description information; after obtaining the required latent space of different semantic forms, the latent space of different semantic forms is fused into a complete latent space for image information reconstruction, and the image information reconstruction is performed according to the complete latent space.
[0013] The present application proposes to use three semantic forms of text semantic description, decoupled parameter semantic description and semantic attribute value for collaborative coding transmission based on the decoupled expression ability of the structured latent space, and to adaptively select the optimal coding mode under different channel conditions to realize image semantic compression transmission in the extremely low code rate domain. At the same time, the text and attribute semantics are mapped back to the structured latent space at the receiving end based on the cross-modal semantic alignment technology, and the missing information is fused and completed to complete the image reconstruction.
[0014] In some embodiments of the present application, the semantic attribute structured latent space of the present application is designed based on a contrastive loss function, and specifically, the design of the semantic attribute structured latent space is as follows: The single latent space is divided into subspaces, each corresponding to a specific semantic attribute, and the last subspace is used to encode other information not captured by the semantic attributes, so the single latent space is represented as: In the above formula, represents the th subspace of the latent space , and represents the number of semantic attributes of interest.
[0015] Further, a batch of latent spaces is represented as: In the above formula, represents the th latent space in a batch of data, represents the number of latent spaces in a batch.
[0016] In some embodiments of the present application, the encoding of the original image data into corresponding text semantic description, decoupled parameter semantic description and semantic attribute value set is as follows: The original image data is encoded into the corresponding text semantic description using an image-text encoder , the original image data is encoded into the corresponding decoupled parameter semantic description using an image-latent space encoder , and the original image data is encoded into the corresponding semantic attribute value set using an image-semantic attribute value encoder , respectively represented as: Among them: The image-text encoder is based on a pre-trained BLIP model, combined with a specific semantic attribute encoder for joint design, and the output is text data following a predetermined format, for example, when facing a face image, the statement used to describe the image is set to include English descriptions of gender, age, expression and hairstyle.
[0017] The image-implicit space encoder is designed based on the discriminator of the GAN, that is, the design of the image-implicit space encoder follows the commonly used method of inverse mapping of the GAN, and a residual calculation module is combined to jointly transmit residual data, so as to reduce the loss of features caused by dimension reduction; the goal of the image-implicit space encoder is to learn the mapping from the image to the structured implicit space of the pre-trained decoder, and the training process is as shown in Figure 3 The overall loss function of the image-implicit space encoder in the model training is composed of an encoder quality loss (that is, the image quality loss of the encoder) and an encoder semantic loss , and the input of the loss function is the original image data , so the formula of the overall loss function is: ; In the present application, the loss function of the image reconstruction quality uses the MSE to calculate the pixel loss of the reconstructed image, and uses the SSIM to calculate the structure loss of the reconstructed image; therefore, the image quality loss of the image-implicit space encoder is defined as follows: ; In the above formula, represents the mean minimization operation, represents the empirical mean calculation of a batch of data; is the mean square difference operation; is the structure similarity calculation; is the image decoder; is the image-implicit space encoder.
[0018] The semantic loss is defined as: ; In the above formula, represents the mean minimization operation, represents the empirical mean calculation of a batch of data; represents the normalization operation; represents the importance weight of the th semantic attribute, which can be adjusted according to the specific deployment scene and requirements; is the calculation of the semantic attribute feature distance of the input image; is the image decoder; is the image-implicit space encoder; represents the number of semantic attributes of interest.
[0019] In the image-implicit space encoder of the present application, the setting of the semantic loss guarantees the integrity of the semantic features during the training of the encoder, and through the double constraints of the image quality loss and the semantic loss, the guided training of the encoder can be reliably completed.
[0020] The image-semantic attribute value encoder uses a feature encoding pre-training model (for example, when used for face image data, the pre-training models such as ArcFace, ESR, R-Net, Dex, PSPNet, etc. are used to encode the semantic attribute values such as hairstyle and expression of face image) as the basis, and according to the different semantic attributes, the semantic attribute values have different dimensions.
[0021] In some embodiments of the present application, the transmission data required according to the feedback of the channel parameters is specifically: The mapping function (i.e. data selection module) based on the preset threshold is used to select the transmission data required according to the feedback of the channel parameters, and specifically includes: when the communication resources are rich, the decoupling parameter semantic description is transmitted in full; when the communication resources are limited, the decoupling parameter semantic description part of the subspace is dynamically adjusted to transmit, and at the same time, the text semantic description or the semantic attribute value set is used to replace another part of the subspace of the decoupling parameter semantic description for transmission; when the communication resources are rich or the communication resources are limited, the text semantic description is transmitted in full, and the semantic attribute value set is transmitted in full or only the relevant semantic attribute value according to the corresponding semantic attribute; it should be noted that since each word in the text information has a front and rear correlation, the text semantic description needs to be transmitted in full, and the semantic attribute value set may transmit only the relevant semantic attribute value according to the corresponding semantic attribute; Wherein, the feedback of the channel parameters is the channel capacity, SNR or other parameters representing the channel quality; the selected transmission data contains the corresponding semantic attribute index, that is, the serial number value of the dimension of the hidden space corresponding to each semantic attribute (for example, the expression attribute of the face image data corresponds to the 128th to 256th dimension in the hidden space), and the selected transmission data and the corresponding semantic attribute index are transmitted to the receiving end together, so that the receiving end can splice the hidden space according to the semantic attribute index.
[0022] In the specific implementation of the present application, after selecting the transmission data required according to the feedback of the channel parameters, the transmission data input into the channel can be represented as: ; In the above formula, represents the data selection module, which uses a preset threshold to select different transmission data for transmission; and representing the selected post-hidden space and semantic attribute value data; feedback for channel parameters.
[0023] In some embodiments of the present application, the obtaining of the corresponding hidden space data according to the semantic description information in different forms is specifically: When the receiving end receives the transmission data, according to the semantic description information in different forms, a text-hidden space encoder is used to encode the text semantic description of the receiving end ( representing the text semantic description received by the receiving end) to obtain the corresponding hidden space data , and a semantic attribute value-hidden space encoder is used to encode the semantic attribute value in the semantic attribute value set of the receiving end ( representing the semantic attribute value received by the receiving end) to obtain the corresponding hidden space data , which is specifically represented as: ; ; It should be noted that the hidden spaces corresponding to the three semantic forms in the present application have the same structure, and each subspace corresponds to the same semantic attribute. Among them: The text-hidden space encoder is composed of a semantic feature extractor BERT and a full connection network. The text data is first divided into sentences by the Tokenizer module of BERT, then the text features are extracted by the BERT model, and the extracted text features are input into the full connection network for dimension adjustment, so as to realize the mapping of text information to the hidden space. In the specific implementation of the present application, in order to improve the training efficiency, during model training, the image data set is preprocessed to make a paired data set of text-hidden space. Specifically, the pre-trained image-hidden space encoder and image-text encoder are used to encode the image data set into paired text data and hidden space data . During the whole training process, only low-dimensional text data and hidden space data need to be loaded, and the training process is as shown in Figure 4 ; Since the training of the text-hidden space encoder only involves the parameter update of the full connection network, the model of the text-hidden space encoder uses MSE as the loss function for model training, and the loss function is specifically represented as: ; In the above formula, , represents the original image data represents the hidden space data obtained by re-encoding after being encoded into text data first; represents the mean minimization operation, represents the empirical mean calculation on a batch of data; is the mean square error operation; is the image-text encoder; is the text-hidden space encoder; is the original hidden space data.
[0024] The model of the semantic attribute value-hidden space encoder is composed of 4 layers of full connection network, and each corresponding semantic attribute value-hidden space sub-encoder has different input dimensions according to the dimensions of different semantic attribute values, and the output dimension is the same as the dimension of the corresponding attribute subspace. The training process is as shown in Figure 5 The model of the semantic attribute value-hidden space encoder also uses MSE as the loss function for model training. For each semantic attribute value-hidden space encoding, the loss function is specifically represented as: In the above formula, , represents the original image data represents the hidden space data obtained by re-encoding after being encoded into semantic attribute value data first, represents the mean minimization operation, represents the empirical mean calculation on a batch of data; is the mean square error operation; is the image-semantic attribute value encoder; is the semantic attribute value-hidden space encoder; is the original hidden space data.
[0025] In some embodiments of the present application, the fusion of hidden spaces of different semantic forms into a complete hidden space for image information reconstruction is specifically: According to the semantic attribute index corresponding to different semantic attributes, the hidden spaces corresponding to different modalities are replaced and spliced to obtain a complete hidden space for image information reconstruction. Specifically, when the dimension of the hidden space is missing, the missing part can be confirmed according to the hidden space dimension index, and on this basis, the hidden space obtained from the text or semantic attribute value is used to fill the missing hidden space, and finally the hidden space after the supplement is used for image reconstruction.
[0026] In the specific implementation of the present application, after obtaining the required hidden spaces of different semantic forms, the received The complete latent space used for image information reconstruction can be represented as: ; In the above formula, represents the received latent space data, represents the latent space data obtained by the receiving end according to the text semantic description, represents the latent space data obtained by the receiving end according to the semantic attribute value.
[0027] In some embodiments of the present application, the image information reconstruction according to the complete latent space is specifically: The image information is reconstructed according to the complete latent space using the decoupled parameter semantic decoder, which can be specifically represented as: ; In the above formula, is the decoupled parameter semantic decoder, is the reconstructed image; The latent space-image decoder uses the generator structure of StyleGAN_v2 as the basis of the decoder, and updates the parameters of the generator and the discriminator simultaneously in the training process, as shown in Figure 6 ; Because the training of the decoupled parameter semantic decoder involves the iterative training of the generator and the discriminator , therefore, in the process of training, the overall loss of the decoupled parameter semantic decoder is composed of the generator loss and the discriminator loss , which are defined as follows: ; In the above formula, is the image data in the training model process, is the corresponding latent space data in the training model process; Wherein: in the training of the generator, the loss function includes the generator contrast loss and the generator discrimination loss , that is, the loss function of the generator in the training process is mainly composed of the above two parts; therefore, the training loss of the generator part can be represented as: ; According to the definition of the semantic attribute structured latent space, a single latent space is divided into subspaces, wherein the first subspaces correspond to semantic attributes respectively, and the last subspace corresponds to other semantic features; according to the idea of contrast learning, the goal of the structured latent space algorithm design is to train the first a semantic attribute has minimizing the feature distance of the first semantic attribute while maximizing the feature distance of other semantic attributes; therefore, the semantic attribute contrast training involves two optimization objectives, which can be represented as: ; ; In the above formula, is the calculation of the semantic attribute feature distance of the input image; denotes the output of the generator (i.e., the semantic decoder); and denote two different latent space vector data within the same batch during the training process, and denote the serial numbers of the two data in the batch latent space data; According to the optimization objective, the contrast loss function is defined as: ; In the above formula, is the index of the semantic attribute; in order to enhance the stability of the training, the present application sets as the threshold for determining the same degree of the first semantic attribute, and sets as the threshold for determining the different degree of the first semantic attribute; therefore, when the feature distance is less than , it can be determined that the semantic attributes of the input images are the same; when the feature distance is greater than , it is determined that the semantic attributes of the input images are different; In summary, the generator contrast loss function can be defined as: ; In the above formula, denotes the mean minimization operation, denotes the empirical mean calculation of a batch of data; denotes the normalization operation; denotes the importance weight of the first semantic attribute, which can be adjusted according to the specific deployment scene and requirements; denotes the number of semantic attributes of interest; For the generator discriminant loss of the generator during the training process, it is defined as: ; In the above formula, denotes the mean minimization operation, denotes the empirical mean calculation of a batch of data; is the output of the discriminator; is the logarithm operation; is the natural exponential operation; denotes the image reconstructed using the latent space; the loss function aims to evaluate the difference between the generated image and the real image through the output of the discriminator, so as to ensure the quality of the image generated by the generator.
[0028] In the training of the discriminator, the loss function includes a discriminator logistic loss and a discriminator regularization loss , that is, the loss function of the discriminator in the training process is mainly composed of the above two parts; therefore, the overall training loss of the discriminator can be defined as: ; wherein the definition of the discriminator logistic loss is: ; In the above formula, denotes the mean minimization operation, denotes the empirical mean calculation of a batch of data; is the output of the discriminator; is the logarithm operation; is the natural exponential operation; is the image data in the training model process, is the corresponding latent space data in the training model process; denotes the image reconstructed using the latent space; The discriminator regularization loss is defined as: ; wherein, is the gradient calculation, is the calculation with respect to the L2 norm; this loss function quantifies the gradient size of the real image in the discriminator, which is used to constrain the gradient norm of the discriminator training. The addition of this loss function enhances the stability of the model in the training process, which can effectively alleviate the problems of overfitting and mode collapse.
[0029] Based on the same inventive concept, the present application also provides a device corresponding to the method in Embodiment One, which is described in detail in Embodiment Two.
[0030] Embodiment Two In this embodiment, an image encoding system is provided, as shown in Figure 7 , the system comprises: A multi-modal semantic information encoding module is used to design a semantic attribute structured latent space; The multi-modal semantic information encoding module is used to encode the original image data into corresponding text semantic description, decoupled parameter semantic description and semantic attribute value set at the sending end, and transmit the required transmission data according to the feedback of the channel parameters; The multi-modal semantic fusion decoding module is used to obtain corresponding latent space data according to different modal semantic description information after receiving the transmission data at the receiving end, fuse the latent spaces in different semantic modalities into a complete latent space for image information reconstruction, and perform image information reconstruction according to the complete latent space.
[0031] The present application proposes to use text semantic description, decoupled parameter semantic description and semantic attribute value to cooperatively encode and transmit, and to adaptively select the optimal encoding mode under different channel conditions, to realize image semantic compression transmission in an extremely low code rate domain, by using the decoupled expression capability based on the structured latent space.
[0032] In some embodiments of the present application, the semantic attribute structured latent space of the present application is designed based on a contrast loss function, and specifically, the design of the semantic attribute structured latent space is specifically: The single latent space is divided into subspaces, each subspace corresponds to a specific semantic attribute, and the last subspace is used to encode other information not captured by the semantic attribute, so the single latent space is represented as: ; In the above formula, represents the i-th subspace of the latent space , represents the number of semantic attributes of interest.
[0033] Further, a batch of latent spaces is represented as: ; In the above formula, represents the i-th subspace of the latent space in a batch of data, represents the number of latent spaces in a batch.
[0034] In some embodiments of the present application, the encoding of the original image data into corresponding text semantic description, decoupled parameter semantic description and semantic attribute value set is specifically: An image-text encoder is used to encode the original image data into corresponding text semantic description, decoupled parameter semantic description and semantic attribute value set. original image data encoded as corresponding text semantic description by image-hidden space encoder original image data encoded as corresponding decoupled parameter semantic description by image-semantic attribute value encoder original image data encoded as corresponding semantic attribute value set respectively denoted as: ; ; ; wherein: The image-text encoder is based on a pre-trained BLIP model, combined with a specific semantic attribute encoder, and the output is text data following a preset format, for example, when facing a face image, the statement used to describe the image is set to include English descriptions of gender, age, expression, and hairstyle.
[0035] The image-hidden space encoder is designed based on the discriminator of GAN, that is, the design of the image-hidden space encoder follows the commonly used method of inverse mapping of GAN, and combines residual calculation modules to jointly transmit residual data, so as to reduce the feature loss caused by dimension reduction; the goal of the image-hidden space encoder is to learn the mapping from the image to the structured hidden space of the pre-trained decoder, and the training process is as shown in Figure 3 The overall loss function of the image-hidden space encoder in model training is composed of the encoder quality loss (i.e. the image quality loss of the encoder) and the encoder semantic loss , and the input of the loss function is the original image data , so the formula of the overall loss function is: ; In the present application, the loss function of the image reconstruction quality uses MSE to calculate the pixel loss of the reconstructed image, and uses SSIM to calculate the structure loss of the reconstructed image; therefore, the image quality loss of the image-hidden space encoder is defined as follows: ; In the above formula, represents the mean minimization operation, represents the empirical mean calculation of a batch of data; is the mean square error operation; is the structural similarity calculation; is the image decoder; is an image-implicit space encoder.
[0036] The semantic loss is defined as: ; In the above formula, represents a mean minimization operation, represents an empirical mean calculation on a batch of data; represents a normalization operation; represents the importance weight of the semantic attribute, which can be adjusted according to specific deployment scenarios and requirements; is the calculation of the semantic attribute feature distance of the input image; is an image decoder; is an image-implicit space encoder; represents the number of semantic attributes of interest.
[0037] In the image-implicit space encoder of the present application, the setting of the semantic loss guarantees the integrity of the semantic features during the training of the encoder. Through the dual constraints of image quality loss and semantic loss, the guided training of the encoder can be reliably completed.
[0038] The image-semantic attribute value encoder uses a feature encoding pre-training model for different semantic attributes (for example, when face image data is used, pre-training models such as ArcFace, ESR, R-Net, Dex, PSPNet, etc. are used to encode semantic attribute values such as hairstyle and expression of face images) as the basis. According to the different semantic attributes, the semantic attribute values have different dimensions.
[0039] In some embodiments of the present application, the required transmission data selected according to the feedback of the channel parameters for sending is specifically: Using a mapping function (i.e., a data selection module) based on a preset threshold to select the required transmission data for sending according to the feedback of the channel parameters, specifically including: when the communication resources are abundant, the decoupling parameter semantic description is transmitted in its entirety; when the communication resources are limited, a part of the subspace in the decoupling parameter semantic description is dynamically adjusted for transmission, while the semantic attribute values in the semantic attribute value set or the semantic attribute values in the semantic attribute value set are used to replace another part of the subspace in the decoupling parameter semantic description for transmission; when the communication resources are abundant or the communication resources are limited, the text semantic description is transmitted in its entirety, while the semantic attribute value set The complete transmission or only the relevant semantic attribute value according to the corresponding semantic attribute is transmitted. It should be noted that, due to the front and rear correlation of each word in the text information, the text semantic description needs to be completely transmitted, and the semantic attribute value set may transmit only the relevant semantic attribute value according to the corresponding semantic attribute. The feedback of the channel parameter is channel capacity, SNR or other parameters representing channel quality; the selected transmission data contains the corresponding semantic attribute index, that is, each semantic attribute corresponds to the serial number value of the dimension in the latent space (for example, the expression attribute of the face image data corresponds to the 128th to 256th dimension in the latent space), and the selected transmission data and the corresponding semantic attribute index are transmitted to the receiving end together, so that the receiving end splices the latent space according to the semantic attribute index.
[0040] In the specific implementation of the present application, after the required transmission data is selected according to the feedback of the channel parameter, the transmission data input into the channel can be represented as: ; In the above formula, represents a data selection module, which selects different transmission data for sending by using a preset threshold; and represent the selected latent space and semantic attribute value data; is the feedback of the channel parameter.
[0041] In some embodiments of the present application, the acquisition of the corresponding latent space data according to the semantic description information in different forms is specifically: After the receiving end receives the transmission data, the corresponding latent space data is acquired by using a text-latent space encoder on the text semantic description ( representing the text semantic description received by the receiving end) in different forms according to the semantic description information in different forms , and a semantic attribute value-latent space encoder is used to encode the semantic attribute value in the semantic attribute value set ( representing the semantic attribute value received by the receiving end) to acquire the corresponding latent space data , which is specifically represented as: ; ; It should be noted that the latent spaces corresponding to the three semantic forms in the present application have the same structure, and each subspace corresponds to the same semantic attribute. Among them: The text-hidden space encoder is composed of a semantic feature extractor BERT and a full connection network, text data is first divided into sentences by a Tokenizer module of the BERT, then text features are extracted by the BERT model, and the extracted text features are input into the full connection network for dimension adjustment, so as to realize the mapping of text information to the hidden space; in the specific implementation, in order to improve the training efficiency, in the model training, the image data set is preprocessed to make a text-hidden space paired data set, specifically, the pre-trained image-hidden space encoder and image-text encoder are used to encode the image data set into paired text data and hidden space data During the whole training process, only low-dimensional text data and hidden space data need to be loaded, and the training process is as shown in Figure 4 . Since the training of the text-hidden space encoder only involves the parameter update of the full connection network, the model of the text-hidden space encoder uses MSE as the loss function of the model training, and the loss function is specifically represented as: . In the above formula, , represents the hidden space data obtained by encoding the original image data into text data and then re-encoding; represents the mean minimization operation, represents the empirical mean calculation of a batch of data; is the mean square difference operation; is an image-text encoder; is a text-hidden space encoder; is original hidden space data.
[0042] The model of the semantic attribute value-hidden space encoder is composed of four full connection networks, according to the dimensions of different semantic attribute values, each corresponding semantic attribute value-hidden space sub-encoder has a different input dimension, and the output dimension is the same as the dimension of the corresponding attribute subspace, and the training process is as shown in Figure 5 ; the model of the semantic attribute value-hidden space encoder also uses MSE as the loss function of the model training, and for each semantic attribute value-hidden space encoding, the loss function is specifically represented as: . In the above formula, , represents the hidden space data obtained by encoding the original image data into semantic attribute value data and then re-encoding, This represents the mean minimization operation. This indicates the calculation of the empirical mean for a batch of data. This is for the calculation of mean squared error; An image-semantic attribute value encoder; Semantic attribute value - latent space encoder; This is the original latent space data.
[0043] In some embodiments of the present invention, the fusion of latent spaces with different semantic forms into a complete latent space for image information reconstruction specifically refers to: Based on the semantic attribute indices corresponding to different semantic attributes, the latent spaces corresponding to different modalities are replaced and concatenated to obtain a complete latent space for image information reconstruction. Specifically, when the latent space is missing dimensions, the missing part can be identified based on the latent space dimension index. Then, the latent space obtained from text or semantic attribute values is used to fill the missing latent space with the part of the same index dimension. Finally, the image is reconstructed using the supplemented latent space.
[0044] In a specific implementation of this invention, after obtaining the latent spaces of the different semantic forms required, it is necessary to process the received data according to the position of each subspace. Completed, the final latent space used for image information reconstruction can be represented as: ; In the above formula, This represents the received latent space data. This represents the latent space data obtained by the receiving end based on the semantic description of the text. This represents the implicit space data obtained by the receiving end based on the semantic attribute values.
[0045] In some embodiments of the present invention, the image information reconstruction based on the complete latent space specifically refers to: The image information is reconstructed using a decoupled parametric semantic decoder based on the complete latent space, which can be specifically represented as follows: ; In the above formula, To decouple the parametric semantic decoder, The reconstructed image; The latent space-based image decoder uses the generator structure of StyleGAN_v2 as the foundation of the decoder, updating the parameters of both the generator and discriminator simultaneously during training. Figure 6 As shown; because the training of the decoupled parameter semantic decoder involves the generator. and discriminator The training process involves iterative steps, therefore, during training, the overall loss of the decoupled parameter semantic decoder is determined by the generator loss. and discriminator loss together constitute, which are defined as follows: ; In the above formula, is the image data in the training model process, is the corresponding hidden space data in the training model process; Wherein: in the training of the generator, the loss function includes the generator contrast loss and the generator discriminant loss That is, the loss function of the generator in the training process is mainly composed of the above two parts; Therefore, the training loss of the generator part can be expressed as: ; According to the definition of the structured hidden space of semantic attributes, the single hidden space is divided into subspaces, wherein the first subspace corresponds to semantic attributes, and the last subspace corresponds to other semantic features; According to the idea of contrast learning, the goal of the structured hidden space algorithm design is to minimize the feature distance of the first semantic attribute when , and maximize the feature distance of other semantic attributes; Therefore, the semantic attribute contrast training involves two optimization objectives, which can be expressed as: ; ; In the above formula, is the calculation of the semantic attribute feature distance of the input image; denotes the output of the generator (i.e. semantic decoder); and denote two different hidden space vector data in the same batch in the training process, and denote the serial numbers of the two data in the batch hidden space data; According to the optimization objective, the contrast loss function is defined as: ; In the above formula, is the index of the semantic attribute; In order to enhance the stability of the training, the present application sets as the threshold for determining the same degree of the first semantic attribute, and sets as the threshold for determining the different degree of the first semantic attribute; Therefore, when the feature distance is less than When the semantic attributes of the input images are the same, it can be determined that they are identical; when the feature distance is greater than 1, the semantic attributes of the input images are identical. If the semantic attributes of the input images are different, then it is determined that they are different. In summary, the generator-contrast loss function can be defined as: ; In the above formula, This represents the mean minimization operation. This indicates the calculation of the empirical mean for a batch of data. This represents the normalization operation; Indicates the first The importance weights of each semantic attribute can be adjusted according to the specific deployment scenario and requirements. Indicates the number of semantic attributes of interest; The generator discrimination loss during the generator training process is defined as: ; In the above formula, This represents the mean minimization operation. This indicates the calculation of the empirical mean for a batch of data. This is the output of the discriminator; Logarithmic operations; For natural index calculations; This represents the image reconstructed using the latent space; the goal of this loss function is to evaluate the difference between the generated image and the real image through the output of the discriminator, thereby ensuring the quality of the image generated by the generator.
[0046] During the training of the discriminator, the loss function includes the discriminator logistic loss. and discriminator regularization loss In other words, the loss function of the discriminator during training mainly consists of the above two parts; therefore, the overall training loss of the discriminator can be defined as: ; The discriminant logistic loss is defined as follows: ; In the above formula, This represents the mean minimization operation. This indicates the calculation of the empirical mean for a batch of data. This is the output of the discriminator; Logarithmic operations; For natural index calculations; For image data in the training model process, For corresponding hidden space data in the training model process; Indicates the image reconstructed using the hidden space; The discriminator regularization loss is defined as: ; Wherein, is the gradient calculation, is the calculation about L2 norm; this loss function quantifies the gradient size of the real image in the discriminator, which is used to constrain the gradient norm of the discriminator training. The addition of this loss function enhances the stability of the model in the training process, which can effectively reduce the problems such as overfitting and pattern collapse.
[0047] Although the specific embodiments of the present application are described above, those skilled in the art should understand that the specific examples described are only illustrative, not for limiting the scope of the present application, and equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present application should be covered within the scope of the claims of the present application.
Claims
1. An image coding method characterized by, The method comprises the following steps: Designing a semantic attribute structured latent space; At the sending end, encoding the original image data into corresponding text semantic description, decoupled parameter semantic description and semantic attribute value set, and selecting the required transmission data for sending according to the feedback of the channel parameters; After receiving the transmission data, the receiving end obtains the corresponding latent space data according to different forms of semantic description information; after obtaining the required latent space of different semantic forms, the latent space of different semantic forms is fused into a complete latent space for image information reconstruction, and the image information is reconstructed according to the complete latent space.
2. The image coding method of claim 1, wherein The designed semantic attribute structured latent space is specifically: The single latent space is divided into subspaces, each corresponding to a specific semantic attribute, and a last subspace for encoding other information not captured by the semantic attributes, the single latent space being represented as: ； wherein, represents the latent space of the first subspace, represents the number of semantic attributes of interest; A batch of latent space is represented as: ； wherein, represents the i-th hidden space in a batch of data, represents the i-th hidden space in a batch of data, represents the number of hidden spaces in a batch.
3. The image coding method of claim 1, wherein The original image data is encoded into corresponding text semantic description, decoupled parameter semantic description and semantic attribute value set, specifically: The original image data is encoded into corresponding text semantic description by using an image-text encoder, into corresponding decoupled parameter semantic description by using an image-latent space encoder, and into corresponding semantic attribute value set by using an image-semantic attribute value encoder; wherein, The image-text encoder is based on a pre-trained BLIP model and is jointly designed with a specific semantic attribute encoder, and the output is text data following a preset format; The image-latent space encoder is designed based on the discriminator of GAN, and combines residual calculation module to jointly transmit residual data; the goal of the image-latent space encoder is to learn the mapping from image to structured latent space of pre-trained decoder, and in model training, the overall loss function of the image-latent space encoder is composed of encoder quality loss and encoder semantic loss; The image-semantic attribute value encoder uses a feature encoding pre-trained model for different semantic attributes as a basis, and the semantic attribute values have different dimensions according to different semantic attributes.
4. The image coding method of claim 1, wherein The transmission data required for sending according to the feedback of the channel parameters is specifically: A mapping function based on a preset threshold is used to select the transmission data required for sending according to the feedback of the channel parameters, specifically including: when the communication resources are rich, the decoupled parameter semantic description is transmitted completely; when the communication resources are limited, a part of the subspace in the decoupled parameter semantic description is dynamically adjusted and transmitted, and another part of the subspace in the decoupled parameter semantic description is replaced by the semantic attribute value in the text semantic description or the semantic attribute value set for transmission; when the communication resources are rich or limited, the text semantic description is transmitted completely, and the semantic attribute value set is transmitted completely or only the relevant semantic attribute value is transmitted according to the corresponding semantic attribute.
5. The image coding method of claim 4, wherein, The feedback of the channel parameters is channel capacity, SNR or other parameters representing channel quality; the selected transmission data contains corresponding semantic attribute index, and the selected transmission data and the corresponding semantic attribute index are transmitted to the receiving end together.
6. The image coding method of claim 1, wherein, The corresponding latent space data obtained according to different forms of semantic description information is specifically: According to different forms of semantic description information, the text semantic description is encoded using a text-hidden space encoder to obtain corresponding hidden space data, and the semantic attribute value in the semantic attribute value set is encoded using a semantic attribute value-hidden space encoder to obtain corresponding hidden space data; wherein, The text-hidden space encoder is composed of a semantic feature extractor BERT and a full connection network. The text data is first divided into sentences by the Tokenizer module of BERT, then the text features are extracted by the BERT model, and the extracted text features are input into the full connection network for dimension adjustment, so as to realize the mapping of text information to the hidden space. The model of the text-hidden space encoder uses MSE as the loss function for model training. The model of the semantic attribute value-hidden space encoder is composed of a 4-layer full connection network. According to the dimensions of different semantic attribute values, each corresponding semantic attribute value-hidden space sub-encoder has a different input dimension, and the output dimension is the same as the dimension of the corresponding attribute subspace. The model of the semantic attribute value-hidden space encoder also uses MSE as the loss function for model training.
7. The image coding method of claim 1, wherein The complete hidden space for image information reconstruction is obtained by replacing and splicing the hidden spaces corresponding to different modalities according to the semantic attribute indexes corresponding to different semantic attributes. The image information is reconstructed according to the complete hidden space.
8. The image coding method of claim 1, wherein, The image information is reconstructed according to the complete hidden space. The image information is reconstructed according to the complete hidden space.
9. An image coding system characterized by comprising: The image information is reconstructed according to the complete hidden space. The system comprises: A hidden space design module for designing a semantic attribute structured hidden space; A multi-state semantic information encoding module for encoding original image data into corresponding text semantic description, decoupled parameter semantic description and semantic attribute value set at the sending end, and selecting the required transmission data for transmission according to the feedback of the channel parameters; A multi-state semantic fusion decoding module for receiving the transmission data at the receiving end, obtaining corresponding hidden space data according to different forms of semantic description information, fusing different semantic forms of hidden space into a complete hidden space for image information reconstruction, and reconstructing the image information according to the complete hidden space.