Multi-mode guided high-fidelity image compression method and system and medium

Abstract

The invention discloses a multi-modal guided high-fidelity image compression method and system and a medium, and the method comprises the steps: firstly extracting text and audio features aligned with image features through a pre-trained text encoder and a pre-trained audio encoder, and then inputting the text and audio features and the image features into a multi-modal feature fusion module for fusion. The multi-modal feature fusion module extracts and fuses multi-modal features from two dimensions of space and channel by using an affine transformation method. A multi-modal feature fusion mode is used for guiding at an encoder end, so that the feature extraction and expression capability is enhanced, and the distribution of potential features can be better predicted. In addition, a discriminator is designed to carry out multi-modal guided generative adversarial training, so that a high-fidelity image is obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-modality guided high-fidelity image compression method, system and medium. Background Art

[0002] Image compression is very important for efficient data storage and transmission. Traditional image compression methods, such as JPEG, JPEG2000, and BPG, are widely used in various applications due to their mature standardized processes. In recent years, with the development of deep learning, many neural network image compression methods based on deep learning have achieved better performance than these traditional image compression methods. However, with the development of multimodal services, the synchronous transmission of multimedia data such as images, text, and audio has become a common situation. In multimodal services, multimodal signals transmitted at the same time are usually expressions of different aspects of the same thing, so they have inherent semantic associations. In order to give full play to the advantages of multimodal data, some researchers in the field of image compression have explored the use of multimodal-guided image compression and effectively improved the perceptual quality of compressed images.

[0003] Current multimodal image compression methods suffer from the following drawbacks:

[0004] First, most of the current multimodal guided image compression methods only use text and image modalities. However, for multimedia tasks, audio is often transmitted simultaneously with images, providing a more comprehensive understanding of multimedia tasks. For example, in telemedicine, doctors explain the image content through audio, and audio can help compression algorithms prioritize key medical details. Exploring whether audio can guide image compression is a necessary and untried area.

[0005] 2. In the text-guided image compression method, the multimodal feature fusion module only guides the image features by local text features in the spatial dimension, and ignores the global guiding effect of text features on image features in the channel dimension. Therefore, there is still a lot of room for optimization in the multimodal feature fusion module.

[0006] 3. When other modal information is introduced into existing text-guided image compression methods, the pixel fidelity (PSNR) of the reconstructed image usually decreases due to the insufficient feature extraction and expression capabilities of the feature fusion module. Summary of the invention

[0007] The main purpose of the present invention is to propose a multimodal guided high-fidelity image compression method, system and medium, aiming to enhance the ability to extract and express image features, better predict the distribution of potential features, and obtain high-fidelity images.

[0008] To achieve the above object, the present invention provides a multi-modality guided high-fidelity image compression method, comprising the following steps:

[0009] Step S100, pre-training a text and audio encoder: the text and audio encoder maps image features, text and audio features into a common semantic space by calculating the degree of matching with the image;

[0010] Step S200: Input the text t and audio a into the pre-trained text and audio encoder to extract the text feature f t With audio feature f a ;

[0011] Step S300: The input image x undergoes linear transformation through three learnable convolutional layers to obtain image features f I ;

[0012] Step S400, the fused features are linearly transformed through a learnable convolutional layer to obtain a new feature map, and then step S300 is repeated to linearly transform the features after fusion again through a learnable convolutional layer;

[0013] Step S500, inputting the feature map into the residual module to enhance the feature representation capability, optimize the training process, and improve the model performance;

[0014] Step S600: quantize the compact feature representation y and convert it into a binary code stream through a lossless encoding module. The binary code stream is transmitted through a signal and restored to a binary code stream through a lossless decoding module. The compact feature representation y is fed into the super encoder to predict the distribution of latent features;

[0015] Step S700: The features received through lossless decoding After a convolutional layer and a ResModule;

[0016] Step S800, reconstruct the feature representation through multiple upsampling convolutional layers, then pass through a ResBlock to prevent gradient vanishing or gradient explosion, and finally pass through a convolutional layer to obtain the reconstructed image

[0017] Step S900: Reconstructing the image and the quantified latent variables Input the discriminator for adversarial training.

[0018] A further technical solution of the present invention is that step S100 comprises:

[0019] Step S110, representing the input text t and audio a as low-dimensional embedding features, wherein the text encoder uses a bidirectional long short-term memory recurrent neural network, and the audio encoder uses a convolutional neural network and a recurrent neural network;

[0020] Step S120: using a pre-trained image encoder to represent the input image x as a low-dimensional embedded feature, and the image encoder uses a convolutional neural network to map the image to a semantic vector;

[0021] Step S130: training the text encoder and the image encoder in step S110 and step S120 by using a deep attention multimodal similarity model, wherein an attention-driven image-text matching score is used in the training process, and the matching degree of the image-sentence pair is measured by an attention model based on the image and the text, so that the trained text and image encoders can effectively map the text and image features to a common semantic space;

[0022] Step S140: train the audio encoder through the same steps, so that the audio and image encoders can effectively map the audio and image features to a common semantic space.

[0023] A further technical solution of the present invention is that step S300 comprises:

[0024] Step S310, designing a feature fusion module, wherein the feature fusion module only fuses two modalities, image and text or audio; the calculation formula used in the designed feature fusion module is:

[0025] F′ t / a =conv 3×3 (F t / a )

[0026] γ=σ(conv 3×3 (F′ t / a ))

[0027] β=conv 1×1 (g(F′ t / a ))

[0028] F′ image =γ·F image +β

[0029] Among them, F t / a represents text features or speech features, F′ t / a represents the text feature or speech feature after the 3×3 convolution operation, σ represents the Sigmoid activation function, g represents the average pooling operation, Y and β represent the scaling factor and offset factor respectively, and F image and F′ image Represent the image features before and after processing

[0030] Step S320, designing a multimodal feature fusion module, wherein the multimodal feature fusion module can meet the requirements of fusing three modalities while fusing two modal features; the formula used in designing the multimodal feature fusion module is:

[0031] F IT = fusion(F image , F text )

[0032] F IA = fusion(F image , F audio )

[0033] F ITA =con 1×1 (concat(F IT , F IA ))

[0034] Among them, fusion means using the feature fusion module to perform feature fusion operation, F image 、F text and F audio Represent image, text and speech features respectively, and F IT 、F IA and F ITA They represent image-text, image-audio, and image-text-audio fusion features respectively, and concat represents the concatenation operation in the channel dimension.

[0035] A further technical solution of the present invention is that step S310 includes:

[0036] Step S311: convert the text feature or voice feature F t / a Input to a 3×3 convolutional layer to transform it into the common semantic space of image features;

[0037] Step S312: Perform a 3×3 convolution operation on the feature map generated in the previous step to calculate the scaling factor Y of each position. Each position of the feature map generated in this step represents a scaling factor Y, which controls the specific adjustment of the image features. Then, the convolution output is passed through a Sigmoid activation function to generate a scaling factor in the range of [0, 1], which is used to control the scaling degree of the image features. Each element of the feature map can be enlarged or reduced according to its corresponding Y value, thereby adjusting the details of the feature map.

[0038] Step S313: First, perform global average pooling on the channel dimension of the feature map through the Average Pool operation to calculate the average value of each channel. This step aggregates the spatial information of each channel into a single value, thereby generating channel-level statistical information. Apply a 1×1 convolution to the pooled result to generate an offset factor β for each channel.

[0039] Step S314: Manipulate the input image feature F by scaling factor Y and offset factor β image .

[0040] A further technical solution of the present invention is that step S320 includes:

[0041] Step S321, inputting the image features and text features into the same feature fusion module, and inputting the image features and audio features into another feature fusion module, and the two modules output their respective fusion features;

[0042] Step S322: performing a concat operation on the channels of the two output fusion features to form a larger feature, providing rich information including three modal features for subsequent processing;

[0043] Step S323: After concatenating the features, a 1×1 convolution is performed to adjust the number of channels so that the final feature map has a suitable dimension.

[0044] A further technical solution of the present invention is that step S600 includes:

[0045] Step S610: After passing through two convolutional layers, the feature representation y is input into a multimodal feature fusion module together with the text and audio features. Since the distribution of latent features is closely related to the content of the image, and the text and audio descriptions can provide semantic information of the image, the multimodal feature fusion module is used to help predict the distribution of latent features. After that, z is obtained by passing through another convolutional layer.

[0046] Step S620: After z is quantized, it is converted into a binary code stream for transmission through a lossless encoding module, and then restored to After passing through the super decoder composed of three convolutional modules, the distribution of latent features is obtained and transmitted to the corresponding lossless codec module.

[0047] To achieve the above objectives, the present invention also proposes a multimodal guided high-fidelity image compression system, characterized in that the system includes a memory, a processor, and a multimodal guided high-fidelity image compression program stored on the processor, and the multimodal guided high-fidelity image compression program executes the steps of the method described above when run by the processor.

[0048] To achieve the above objectives, the present invention also proposes a computer-readable storage medium, which stores a multi-modality-guided high-fidelity image compression program, and when the multi-modality-guided high-fidelity image compression program is executed by the processor, the steps of the method described above are executed.

[0049] The beneficial effects of the multimodal guided high-fidelity image compression method, system and medium of the present invention are:

[0050] The present invention uses the modalities (text, audio) related to image semantics to guide high-fidelity image compression, and explores the guiding role of audio in the image compression process for the first time. In addition, the designed feature fusion module can not only guide local fine-grained image features in the spatial dimension, but also guide image features globally in the channel dimension. At the same time, the module has the advantages of a multi-scale structure, and while having good feature fusion capabilities, it can also enhance the nonlinear expression capabilities of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0052] Figure 1 It is a schematic diagram of the overall flow of a preferred embodiment of the multi-modality guided high-fidelity image compression method of the present invention;

[0053] Figure 2 It is an overall framework diagram involved in the multi-modality guided high-fidelity image compression method of the present invention;

[0054] Figure 3 is a detailed flow chart of step S100;

[0055] Figure 4 is a schematic diagram of a feature fusion module and a multimodal feature fusion module;

[0056] Figure 5 is a detailed flow chart of step S300;

[0057] Figure 6 is a detailed flow chart of step S310;

[0058] Figure 7 is a detailed flow chart of step S320;

[0059] Figure 8is a detailed flow chart of step S600;

[0060] Fig. 9 It is a system architecture diagram of the multi-modality guided high-fidelity image compression system of the present invention.

[0061] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The present invention proposes a multimodal guided high-fidelity image compression method. The technical solution adopted by the present invention is mainly to first extract text and audio features aligned with image features through a pre-trained text encoder and an audio encoder, and then input them and the image features into a multimodal feature fusion module for fusion. The multimodal feature fusion module uses an affine transformation method to extract and fuse multimodal features from two dimensions, space and channel. By guiding using a multimodal feature fusion method on the encoder side, not only the feature extraction and expression capabilities are enhanced, but also the distribution of potential features can be better predicted. In addition, the present invention designs a discriminator to perform multimodal guided generative adversarial training to obtain high-fidelity images.

[0064] For details, please refer to Figure 1 and Figure 2 The preferred embodiment of the multi-modality guided high-fidelity image compression method of the present invention comprises the following steps:

[0065] Step S100, pre-training text and audio encoders: the text and audio encoders map image features, text and audio features to a common semantic space by calculating the degree of matching with the image, so as to better perform multimodal feature fusion.

[0066] Please refer to Figure 3 , the step S100 specifically includes the following steps:

[0067] In step S110, the input text t and audio a are represented as low-dimensional embedding features (ranging from 128 to 1024), wherein the text encoder uses a bidirectional long short-term memory recurrent neural network (LSTM), and the audio encoder uses a convolutional neural network (CNN) and a recurrent neural network (RNN).

[0068] Step S120: Use a pre-trained image encoder to represent the input image x as a low-dimensional embedding feature (ranging from 512 to 1024). The image encoder may use a convolutional neural network (CNN) to map the image to a semantic vector.

[0069] Step S130: The text encoder and the image encoder in step S110 and step S120 are trained by a deep attention multimodal similarity model (DAMSM), wherein an attention-driven image-text matching score is used in the training process, and the matching degree of the image-sentence pair is measured by an attention model based on the image and text, so that the trained text and image encoders can effectively map the text and image features to a common semantic space.

[0070] Step S140: Similar to text, the audio encoder is trained through the same steps so that the audio and image encoders can effectively map audio and image features to a common semantic space.

[0071] Step S200: Input the text t and audio a into the pre-trained text and audio encoder to extract the text feature f t With audio feature f a .

[0072] Step S300: The input image x undergoes linear transformation through three learnable convolutional layers to obtain image features f I .

[0073] Image feature f I After the multimodal feature fusion module (such as Figure 4 As shown) and text feature f t With audio feature f a In this process, the input can be in the form of images and text, images and audio, or images, text and audio.

[0074] like Figure 5 As shown, the step S300 specifically includes the following steps:

[0075] Step S310, designing a feature fusion module, wherein the feature fusion module only fuses two modalities: image and text or audio.

[0076] The specific formula of the whole process of step S310 is:

[0077] F′ t / a =conv 3×3 (F t / a )

[0078] γ=σ(conv 3×3 (F′ t / a ))

[0079] β=conv 1×1 (g(F′ t / a ))

[0080] F' image =γ·F image +β

[0081] Among them, F t / a represents text features or speech features, F′ t / a represents the text feature or speech feature after the 3×3 convolution operation, σ represents the Sigmoid activation function, g represents the average pooling operation, Y and β represent the scaling factor and offset factor respectively, and F image and F′ image Represent the image features before and after processing respectively.

[0082] like Figure 6 As shown, the step S310 specifically includes the following steps:

[0083] Step S311: convert the text feature or voice feature F t / a Input to a 3×3 convolutional layer to transform it into the common semantic space of image features;

[0084] Step S312: Perform a 3×3 convolution operation on the feature map generated in the previous step to calculate the scaling factor Y at each position. Each position of the feature map generated in this step represents a scaling factor Y, which controls the specific adjustment of the image features. Then, the convolution output is passed through the Sigmoid activation function to generate a scaling factor in the range of [0, 1], which is used to control the scaling degree of the image features. Each element of the feature map can be enlarged or reduced according to its corresponding Y value, thereby adjusting the details of the feature map.

[0085] Step S313: First, perform global average pooling on the channel dimension of the feature map through the Average Pool operation to calculate the average value of each channel. This step aggregates the spatial information of each channel into a single value, thereby generating channel-level statistical information. Apply a 1×1 convolution to the pooled result to generate an offset factor β for each channel.

[0086] 1×1 convolution can effectively learn the channel-level offset factor to different channels. The generated channel-level offset factor β is used to adjust the feature map of each channel. In each channel, all pixels will be added with the corresponding β value to adjust the overall offset of the channel feature.

[0087] Step S314: Manipulate the input image feature F by scaling factor Y and offset factor β image .

[0088] Step S320, designing a multimodal feature fusion module, wherein the multimodal feature fusion module can meet the requirements of fusing three modalities while fusing two modal features; the formula used in designing the multimodal feature fusion module is:

[0089] F IT = fusion(F image , F text )

[0090] F IA = fusion(F image , F audio )

[0091] F ITA =con 1×1 (concat(F IT , F IA ))

[0092] Among them, fusion means using the feature fusion module to perform feature fusion operation, F image 、F text and F audio Represent image, text and speech features respectively, and F IT 、F IA and F ITA They represent image-text, image-audio, and image-text-audio fusion features respectively, and concat represents the concatenation operation in the channel dimension.

[0093] like Figure 7 As shown, the step S320 specifically includes:

[0094] Step S321, inputting the image features and text features into the same feature fusion module, and inputting the image features and audio features into another feature fusion module, and the two modules output their respective fusion features;

[0095] Step S322: performing a concat operation on the channels of the two output fusion features to form a larger feature, providing rich information including three modal features for subsequent processing;

[0096] Step S323: After the features are concatenated, a 1×1 convolution is performed to adjust the number of channels so that the final feature map has a suitable dimension. This helps to reduce the computational complexity and improve the effectiveness of feature expression, and finally outputs the fusion features of the three modalities.

[0097] It should be noted here that if it is detected that the input has only two modes, the following steps S322 and S323 will not be executed.

[0098] In step S400, the fused features are linearly transformed through a learnable convolutional layer to obtain a new feature map, and then step S300 is repeated to linearly transform the fused features again through a learnable convolutional layer.

[0099] Step S500: Input the feature map into the residual module (ResModule) to enhance the feature representation capability, optimize the training process, and improve the model performance. The feature map is further processed through the convolution layer to obtain a compact feature representation y.

[0100] Step S600, the compact feature representation y is quantized and converted into a binary code stream through a lossless encoding module. The binary code stream can be transmitted through a signal and restored to a binary code stream through a lossless decoding module. The compact feature representation y is input to the hyperencoder to predict the distribution of latent features.

[0101] like Figure 8 As shown, the step S600 specifically includes the following steps:

[0102] Step S610: After passing through two convolutional layers, the feature representation y is input into a multimodal feature fusion module together with the text and audio features. Since the distribution of latent features is closely related to the content of the image, and the text and audio descriptions can provide semantic information of the image, this module can help predict the distribution of latent features. After that, z is obtained by passing through another convolutional layer.

[0103] Step S620: After z is quantized, it is converted into a binary code stream for transmission through a lossless encoding module, and then restored to After passing through the super decoder composed of three convolutional modules, the distribution of latent features is obtained and transmitted to the corresponding lossless codec module.

[0104] Step S700: The features received through lossless decoding After a convolutional layer and a ResModule, the purpose is to process the received complex features, generate rich features and provide better training stability.

[0105] Step S800, reconstruct the feature representation through multiple upsampling convolutional layers, then pass through a ResBlock to prevent gradient vanishing or gradient explosion, and finally pass through a convolutional layer to obtain the reconstructed image

[0106] Step S900: Reconstructing the image and the quantified latent variables Input the discriminator for adversarial training.

[0107] The beneficial effects of the multi-modality guided high-fidelity image compression method of the present invention are:

[0108] The present invention uses the modalities (text, audio) related to image semantics to guide high-fidelity image compression, and explores the guiding role of audio in the image compression process for the first time. In addition, the designed feature fusion module can not only guide local fine-grained image features in the spatial dimension, but also guide image features globally in the channel dimension. At the same time, the module has the advantages of a multi-scale structure, and while having good feature fusion capabilities, it can also enhance the nonlinear expression capabilities of the network.

[0109] To achieve the above object, the present invention also proposes a multi-modality guided high-fidelity image compression system, such as Fig. 9 As shown, the system includes a processor 1001, a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002, and a multimodal guided high-fidelity image compression program stored on the processor, wherein the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0110] Those skilled in the art will understand that Fig. 9 The system structure shown in the figure does not constitute a limitation of the system, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0111] like Fig. 9 As shown, the memory 1005 as a computer storage medium may include an operating device, a network communication module, a user interface module, and a multi-modality guided high-fidelity image compression program.

[0112] exist Fig. 9 In the system shown, the network interface 1004 is mainly used to connect to the network server and communicate data with the network server; the user interface 1003 is mainly used to interact with the user terminal and receive instructions input by the user; and the processor 1001 can be used to call the multi-modal guided high-fidelity image compression program stored in the memory 1005.

[0113] To achieve the above objectives, the present invention also proposes a computer-readable storage medium, which stores a multi-modality-guided high-fidelity image compression program. When the multi-modality-guided high-fidelity image compression program is run by a processor, the steps of the method described above are executed, which will not be repeated here.

[0114] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A multi-modality guided high-fidelity image compression method, characterized in that: The following steps are involved: Step S100, pre-training a text and audio encoder: the text and audio encoder maps image features, text and audio features into a common semantic space by calculating the degree of matching with the image; Step S200: Input the text t and audio a into the pre-trained text and audio encoder to extract the text feature f t With the audio feature f a ; Step S300: The input image x undergoes linear transformation through three learnable convolutional layers to obtain image features f I ; Step S400, the fused features are linearly transformed through a learnable convolutional layer to obtain a new feature map, and then step S300 is repeated to linearly transform the features after fusion again through a learnable convolutional layer; Step S500, inputting the feature map into the residual module to enhance the feature representation capability, optimize the training process, and improve the model performance; Step S600: quantize the compact feature representation y and convert it into a binary code stream through a lossless encoding module. The binary code stream is transmitted through a signal and restored to a binary code stream through a lossless decoding module. The compact feature representation y is fed into the super encoder to predict the distribution of latent features; Step S700: The features received through lossless decoding After a convolutional layer and a ResModule; Step S800, reconstruct the feature representation through multiple upsampling convolutional layers, then pass through a ResBlock to prevent gradient vanishing or gradient explosion, and finally pass through a convolutional layer to obtain the reconstructed image Step S900: Reconstructing the image and the quantified latent variables Input the discriminator for adversarial training.

2. The multimodality guided high-fidelity image compression method according to claim 1, characterized in that: The step S100 includes: Step S110, representing the input text t and audio a as low-dimensional embedding features, wherein the text encoder uses a bidirectional long short-term memory recurrent neural network, and the audio encoder uses a convolutional neural network and a recurrent neural network; Step S120: using a pre-trained image encoder to represent the input image x as a low-dimensional embedded feature, and the image encoder uses a convolutional neural network to map the image to a semantic vector; Step S130: training the text encoder and the image encoder in step S110 and step S120 by using a deep attention multimodal similarity model, wherein an attention-driven image-text matching score is used in the training process, and the matching degree of the image-sentence pair is measured by an attention model based on the image and the text, so that the trained text and image encoders can effectively map the text and image features to a common semantic space; Step S140: train the audio encoder through the same steps, so that the audio and image encoders can effectively map the audio and image features to a common semantic space.

3. The multi-modality guided high-fidelity image compression method according to claim 2, characterized in that: The step S300 includes: Step S310, designing a feature fusion module, wherein the feature fusion module only fuses two modalities, image and text or audio; the calculation formula used in the designed feature fusion module is: F′ t / a =conv 3×3 (F t / a ) γ=σ(conv 3×3 (F′ t / a )) β=conv 1×1 (g(F′ t / a )) F' image =γ·F image +b Among them, F t / a Represents text features or speech features, F' t / a represents the text feature or speech feature after the 3×3 convolution operation, σ represents the Sigmoid activation function, g represents the average pooling operation, γ and β represent the scaling factor and offset factor respectively, and F image With F' image In step S320, the image features before and after processing are respectively represented, and a multimodal feature fusion module is designed. The multimodal feature fusion module can meet the requirements of fusing three modalities while fusing two modal features; the formula used in designing the multimodal feature fusion module is: F IT =fusion(F image ,F text ) F IA =fusion(F image ,F audio ) F ITA =con 1×1 (concat(F IT ,F IA )) Among them, fusion means using the feature fusion module to perform feature fusion operation, F image 、F text and F audio Represent image, text and speech features respectively, and F IT 、F IA and F ITA They represent image-text, image-audio, and image-text-audio fusion features respectively, and concat represents the concatenation operation in the channel dimension.

4. The multi-modality guided high-fidelity image compression method according to claim 3, characterized in that: The step S310 includes: Step S311: convert the text feature or voice feature F t / a Input to a 3×3 convolutional layer to transform it into the common semantic space of image features; Step S312: Perform a 3×3 convolution operation on the feature map generated in the previous step to calculate the scaling factor γ at each position. Each position of the feature map generated in this step represents a scaling factor γ, which controls the specific adjustment of the image features. Then, the convolution output is passed through a Sigmoid activation function to generate a scaling factor in the range of [0, 1], which is used to control the scaling degree of the image features. Each element of the feature map can be enlarged or reduced according to its corresponding γ value, thereby adjusting the details of the feature map. Step S313: First, perform global average pooling on the channel dimension of the feature map through the Average Pool operation to calculate the average value of each channel. This step aggregates the spatial information of each channel into a single value, thereby generating channel-level statistical information. Apply a 1×1 convolution to the pooled result to generate an offset factor β for each channel. Step S314: Manipulate the input image feature F by scaling factor γ and offset factor β image .

5. The multi-modality guided high-fidelity image compression method according to claim 4, characterized in that: The step S320 includes: Step S321, inputting the image features and text features into the same feature fusion module, and inputting the image features and audio features into another feature fusion module, and the two modules output their respective fusion features; Step S322: performing a concat operation on the channels of the two output fusion features to splice them into a larger feature, providing rich information containing three modal features for subsequent processing; Step S323: After concatenating the features, a 1×1 convolution is performed to adjust the number of channels so that the final feature map has a suitable dimension.

6. The multi-modality guided high-fidelity image compression method according to claim 5, characterized in that: The step S600 includes: Step S610: After passing through two convolutional layers, the feature representation y is input into a multimodal feature fusion module together with the text and audio features. Since the distribution of latent features is closely related to the content of the image, and the text and audio descriptions can provide semantic information of the image, the multimodal feature fusion module is used to help predict the distribution of latent features. After that, z is obtained by passing through another convolutional layer. Step S620: z is quantized and converted into a binary code stream for transmission through a lossless encoding module, and then restored to After passing through the super decoder composed of three convolutional modules, the distribution of latent features is obtained and transmitted to the corresponding lossless codec module.

7. A multi-modality guided high-fidelity image compression system, characterized in that: The system includes a memory, a processor, and a multi-modality-guided high-fidelity image compression program stored on the processor, wherein the multi-modality-guided high-fidelity image compression program is executed by the processor to perform the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multi-modality-guided high-fidelity image compression program, and the multi-modality-guided high-fidelity image compression program is executed by the processor to perform the steps of the method according to any one of claims 1 to 6.

Images