A multi-modal guided high-fidelity image compression method, system, and medium

Abstract

The application discloses a multi-modal guided high-fidelity image compression method and system and a medium. First, a pre-trained text encoder and an audio encoder are used to extract text and audio features aligned with image features, and then the text and audio features are input into a multi-modal feature fusion module together with the image features for fusion. The multi-modal feature fusion module uses an affine transformation method to extract and fuse multi-modal features from two dimensions of space and channels. By using the multi-modal feature fusion method for guidance at the encoder end, the feature extraction and expression capability is enhanced, and the distribution of potential features can be better predicted. In addition, a discriminator is designed for multi-modal guided generative adversarial training, so that high-fidelity images are obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal guided high-fidelity image compression method, system and medium. Background Art

[0002] Image compression is crucial for efficient data storage and transmission. Traditional image compression methods, such as JPEG, JPEG2000, and BPG, are widely used in various applications due to their mature, standardized processes. With the recent development of deep learning, many deep learning-based neural network image compression methods have achieved superior performance compared to these traditional methods. However, with the growth of multimodal services, the simultaneous transmission of multimedia data such as images, text, and audio has become commonplace. In multimodal services, multimodal signals transmitted simultaneously typically represent different aspects of the same subject, resulting in inherent semantic associations. To leverage the advantages of multimodal data, some researchers in the field of image compression have explored multimodal-guided image compression, effectively improving the perceptual quality of compressed images.

[0003] Current multimodal image compression methods suffer from the following drawbacks:

[0004] First, current multimodal guided image compression methods mostly utilize only text and image modalities. However, for multimedia tasks, audio is often transmitted simultaneously with images, providing a more comprehensive understanding of the task. For example, in telemedicine, doctors use audio to explain the image content, which can help compression algorithms prioritize key medical details. Exploring whether audio can guide image compression is a necessary and unexplored area.

[0005] 2. In the text-guided image compression method, the multimodal feature fusion module only guides the image features with local text features in the spatial dimension, ignoring the global guiding effect of text features on image features in the channel dimension. Therefore, there is still a lot of room for optimization in the multimodal feature fusion module.

[0006] 3. When existing text-guided image compression methods introduce other modal information, the pixel-wise reconstructed image fidelity (PSNR) usually decreases due to the insufficient feature extraction and expression capabilities of the feature fusion module. Summary of the Invention

[0007] The main purpose of the present invention is to propose a multimodal guided high-fidelity image compression method, system and medium, aiming to enhance the extraction and expression capabilities of image features, better predict the distribution of potential features, and obtain high-fidelity images.

[0008] To achieve the above object, the present invention provides a multimodal guided high-fidelity image compression method, comprising the following steps:

[0009] Step S100, pre-training a text and audio encoder: the text and audio encoder maps image features, text and audio features into a common semantic space by calculating the degree of matching with the image;

[0010] Step S200: Input the text t and audio a into the pre-trained text and audio encoder to extract the text feature f t With audio feature f a ;

[0011] Step S300: The input image x undergoes linear transformation through three learnable convolutional layers to obtain image features f I ;

[0012] In step S400, the fused features are linearly transformed through a learnable convolutional layer to obtain a new feature map, and then step S300 is repeated to linearly transform the fused features through a learnable convolutional layer.

[0013] Step S500: Input the feature map into the residual module to enhance feature representation capability, optimize the training process, and improve model performance;

[0014] Step S600: The compact feature representation y is quantized and converted into a binary code stream through a lossless encoding module. The binary code stream is transmitted through a signal and restored to a binary code stream through a lossless decoding module. The compact feature representation y is fed into the super-encoder to predict the distribution of latent features;

[0015] Step S700: The features received after lossless decoding After a convolutional layer and a ResModule;

[0016] Step S800: reconstruct the feature representation through multiple upsampling convolution layers, then pass through a ResBlock to prevent gradient vanishing or gradient explosion, and finally pass through a convolution layer to obtain the reconstructed image.

[0017] Step S900: Reconstructing the image and quantified latent variables Input the discriminator for adversarial training.

[0018] A further technical solution of the present invention is that step S100 includes:

[0019] Step S110: Represent the input text t and audio a as low-dimensional embedding features, where the text encoder uses a bidirectional long short-term memory recurrent neural network, and the audio encoder uses a convolutional neural network and a recurrent neural network;

[0020] Step S120: using a pre-trained image encoder to represent the input image x as a low-dimensional embedded feature, and the image encoder uses a convolutional neural network to map the image to a semantic vector;

[0021] Step S130: training the text encoder and image encoder in steps S110 and S120 using a deep attention multimodal similarity model, wherein an attention-driven image-text matching score is used during the training process to measure the matching degree of image-sentence pairs based on an attention model between image and text, so that the trained text and image encoders can effectively map text and image features to a common semantic space;

[0022] Step S140: Train the audio encoder through the same steps, so that the audio and image encoders can effectively map the audio and image features to a common semantic space.

[0023] A further technical solution of the present invention is that step S300 includes:

[0024] Step S310: Design a feature fusion module, wherein the feature fusion module only fuses the two modalities of image and text or audio; the calculation formula used by the designed feature fusion module is:

[0025] F′ t / a =conv 3×3 (F t / a )

[0026] γ=σ(conv 3×3 (F′ t / a ))

[0027] β=conv 1×1 (g(F′ t / a ))

[0028] F′ image =γ·F image +β

[0029] Among them, F t / a represents text features or speech features, F′ t / a represents the text features or speech features after the 3×3 convolution operation, σ represents the Sigmoid activation function, g represents the average pooling operation, Y and β represent the scaling factor and offset factor respectively, F image and F′ image Represents the image features before and after processing

[0030] Step S320: Design a multimodal feature fusion module. The multimodal feature fusion module can meet the requirements of fusing three modalities while fusing two modal features. The formula used in designing the multimodal feature fusion module is:

[0031] F IT =fusion(F image , F text )

[0032] F IA =fusion(F image , F audio )

[0033] F ITA =con 1×1 (concat(F IT , F IA ))

[0034] Among them, fusion means using feature fusion module to perform feature fusion operation, F image 、F text and F audio Represent image, text and speech features respectively, F IT 、F IA and F ITA They represent image-text, image-audio, and image-text-audio fusion features respectively, and concat represents the splicing operation in the channel dimension.

[0035] A further technical solution of the present invention is that step S310 includes:

[0036] Step S311: convert the text feature or voice feature F t / a The input is fed into a 3×3 convolutional layer to transform it into the common semantic space of image features;

[0037] Step S312: Perform a 3×3 convolution operation on the feature map generated in the previous step to calculate the scaling factor Y at each position. Each position of the feature map generated in this step represents a scaling factor Y, which controls the specific adjustment of the image features. Then, the convolution output is passed through the Sigmoid activation function to generate a scaling factor in the range of [0, 1], which is used to control the scaling degree of the image features. Each element of the feature map can be enlarged or reduced according to its corresponding Y value, thereby adjusting the details of the feature map.

[0038] Step S313: First, perform global average pooling on the channel dimension of the feature map through the Average Pool operation to calculate the average value of each channel. This step aggregates the spatial information of each channel into a single value, thereby generating channel-level statistical information. Apply a 1×1 convolution to the pooled result to generate the offset factor β for each channel;

[0039] Step S314: Manipulate the input image feature F by the scaling factor Y and the offset factor β image .

[0040] A further technical solution of the present invention is that step S320 includes:

[0041] Step S321: input the image features and text features into the same feature fusion module, and input the image features and audio features into another feature fusion module, and the two modules output their respective fusion features;

[0042] Step S322: Perform concat operation on the two output fusion features on the channel to splice them into a larger feature, providing rich information containing three modal features for subsequent processing;

[0043] Step S323: After concatenating the features, a 1×1 convolution is performed to adjust the number of channels so that the final feature map has an appropriate dimension.

[0044] A further technical solution of the present invention is that step S600 includes:

[0045] Step S610: Feature representation y passes through two convolutional layers and is then input into a multimodal feature fusion module along with text and audio features. Because the distribution of latent features is closely related to the image content, and text and audio descriptions can provide semantic information about the image, the multimodal feature fusion module helps predict the distribution of latent features. Afterwards, z is obtained by passing it through another convolutional layer.

[0046] Step S620: z is quantized and converted into a binary code stream for transmission through the lossless encoding module, and then restored to After passing through the super decoder composed of three convolutional modules, the distribution of latent features is obtained and transmitted to the corresponding lossless codec module.

[0047] To achieve the above objectives, the present invention also proposes a multimodal guided high-fidelity image compression system, characterized in that the system includes a memory, a processor, and a multimodal guided high-fidelity image compression program stored on the processor, and the multimodal guided high-fidelity image compression program executes the steps of the above-mentioned method when run by the processor.

[0048] To achieve the above object, the application further provides a computer readable storage medium, which stores a multi-modal guided high-fidelity image compression program, and the multi-modal guided high-fidelity image compression program performs the steps of the method as described above when executed by the processor.

[0049] The multi-modal guided high-fidelity image compression method, system and medium have the following advantages:

[0050] The application uses modalities (text, audio) related to image semantics to guide the implementation of high-fidelity image compression, and for the first time explores the guiding role of audio in the image compression process. In addition, the designed feature fusion module can not only perform local fine-grained image feature guidance in the spatial dimension, but also can globally guide image features in the channel dimension. At the same time, the module has the superiority of multi-scale structure, which can not only have good feature fusion capability, but also can enhance the nonlinear expression capability of the network. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from the structures shown in these drawings without creative labor.

[0052] Figure 1 is the overall flow schematic diagram of the preferred embodiment of the multi-modal guided high-fidelity image compression method of the application;

[0053] Figure 2 is the overall framework diagram involved in the multi-modal guided high-fidelity image compression method of the application;

[0054] Figure 3 is the detailed flow schematic diagram of step S100;

[0055] Figure 4 is the schematic diagram of the feature fusion module and the multi-modal feature fusion module;

[0056] Figure 5 is the detailed flow schematic diagram of step S300;

[0057] Figure 6 is the detailed flow schematic diagram of step S310;

[0058] Figure 7 is the detailed flow schematic diagram of step S320;

[0059] Figure 8is a detailed flowchart of step S600;

[0060] Figure 9 is a system architecture diagram of the multi-modal guided high-fidelity image compression system of the present application.

[0061] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments in conjunction with the accompanying drawings. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0063] The present application proposes a multi-modal guided high-fidelity image compression method. The technical solution adopted by the present application is mainly that text and audio features aligned with image features are extracted through a pre-trained text encoder and an audio encoder, and then they are input into a multi-modal feature fusion module for fusion. The multi-modal feature fusion module extracts and fuses multi-modal features from two dimensions of space and channel using an affine transformation method. By using the multi-modal feature fusion method for guidance at the encoder end, not only the feature extraction and expression ability is enhanced, but also the distribution of potential features can be better predicted. In addition, a discriminator is designed for multi-modal guided generative adversarial training, so as to obtain high-fidelity images.

[0064] Specifically, please refer to Figure 1 and Figure 2 The multi-modal guided high-fidelity image compression method of the present application preferably includes the following steps:

[0065] Step S100, pre-training text and audio encoders: the text and audio encoders map image features, text and audio features to a common semantic space by calculating the matching degree with the image, so as to better perform multi-modal feature fusion.

[0066] Please refer to Figure 3 The step S100 specifically includes the following steps:

[0067] Step S110, input text t and audio a are expressed as low-dimensional embedding features (the range can be between 128 and 1024), wherein the text encoder uses a bidirectional long short-term memory recurrent neural network (LSTM), and the audio encoder uses a convolutional neural network (CNN) and a recurrent neural network (RNN).

[0068] Step S120: Use a pre-trained image encoder to represent the input image x as a low-dimensional embedded feature (ranging from 512 to 1024). The image encoder can use a convolutional neural network (CNN) to map the image to a semantic vector.

[0069] Step S130: The text encoder and image encoder in step S110 and step S120 are trained by a deep attention multimodal similarity model (DAMSM), wherein an attention-driven image-text matching score is used in the training process, and the matching degree of the image-sentence pair is measured by an attention model based on the image and text, so that the trained text and image encoders can effectively map the text and image features to a common semantic space.

[0070] Step S140: Similar to text, the audio encoder is trained through the same steps so that the audio and image encoders can effectively map audio and image features to a common semantic space.

[0071] Step S200: Input the text t and audio a into the pre-trained text and audio encoder to extract the text feature f t With audio feature f a .

[0072] Step S300: The input image x undergoes linear transformation through three learnable convolutional layers to obtain image features f I .

[0073] Image feature f I After the multimodal feature fusion module (such as Figure 4 shown) and text feature f t With audio feature f a In this process, the input can be in the form of images and text, images and audio, or images, text and audio.

[0074] like Figure 5 As shown, the step S300 specifically includes the following steps:

[0075] Step S310: designing a feature fusion module, wherein the feature fusion module only fuses two modalities: image and text or audio.

[0076] The specific formula for the entire process of step S310 is:

[0077] F′ t / a =conv 3×3 (F t / a )

[0078] γ=σ(conv 3×3 (F′ t / a ))

[0079] β=conv 1×1 (g(F′ t / a ))

[0080] F' image =γ·F image +β

[0081] Among them, F t / a represents text features or speech features, F′ t / a represents the text features or speech features after the 3×3 convolution operation, σ represents the Sigmoid activation function, g represents the average pooling operation, Y and β represent the scaling factor and offset factor respectively, F image and F′ image Represent the image features before and after processing respectively.

[0082] like Figure 6 As shown, step S310 specifically includes the following steps:

[0083] Step S311: convert the text feature or voice feature F t / a The input is fed into a 3×3 convolutional layer to transform it into the common semantic space of image features;

[0084] Step S312: Perform a 3×3 convolution operation on the feature map generated in the previous step to calculate the scaling factor Y at each position. Each position in the feature map generated in this step represents a scaling factor Y, which controls the specific adjustment of the image features. The convolution output is then passed through a Sigmoid activation function to generate a scaling factor in the range of [0, 1], which is used to control the degree of scaling of the image features. Each element of the feature map can be enlarged or reduced according to its corresponding Y value, thereby adjusting the details of the feature map.

[0085] Step S313: First, perform global average pooling on the channel dimension of the feature map through the Average Pool operation to calculate the average value of each channel. This step aggregates the spatial information of each channel into a single value, thereby generating channel-level statistical information. Apply 1×1 convolution to the pooled result to generate the offset factor β for each channel.

[0086] 1×1 convolution effectively learns channel-level offset factors for different channels. The resulting channel-level offset factor β is used to adjust the feature map of each channel. In each channel, all pixels are added with the corresponding β value to adjust the overall offset of the channel features.

[0087] Step S314: Manipulate the input image feature F by the scaling factor Y and the offset factor β image .

[0088] Step S320: Design a multimodal feature fusion module. The multimodal feature fusion module can meet the requirements of fusing three modalities while fusing two modal features. The formula used in designing the multimodal feature fusion module is:

[0089] F IT =fusion(F image , F text )

[0090] F IA =fusion(F image , F audio )

[0091] F ITA =con 1×1 (concat(F IT , F IA ))

[0092] Among them, fusion means using feature fusion module to perform feature fusion operation, F image 、F text and F audio Represent image, text and speech features respectively, F IT 、F IA and F ITA They represent image-text, image-audio, and image-text-audio fusion features respectively, and concat represents the splicing operation in the channel dimension.

[0093] like Figure 7 As shown, the step S320 specifically includes:

[0094] Step S321: input the image features and text features into the same feature fusion module, and input the image features and audio features into another feature fusion module, and the two modules output their respective fusion features;

[0095] Step S322: Perform concat operation on the two output fusion features on the channel to splice them into a larger feature, providing rich information containing three modal features for subsequent processing;

[0096] Step S323: After concatenating the features, a 1×1 convolution is performed to adjust the number of channels so that the final feature map has an appropriate dimension. This helps reduce computational complexity while improving the effectiveness of feature expression. Finally, the fused features of the three modalities are output.

[0097] It should be noted here that if it is detected that the input has only two modes, the following steps S322 and S323 will not be executed.

[0098] Step S400, the fused features are linearly transformed through the learnable convolution layer to obtain new feature maps, and then the features after the second fusion are linearly transformed through the learnable convolution layer again.

[0099] Step S500, the feature maps are input into a residual module (ResModule) to enhance the feature representation capability, optimize the training process, and improve the model performance. The feature maps are further processed through a convolution layer to obtain compact feature representation y.

[0100] Step S600, after the compact feature representation y is quantized, it is converted into a binary code stream through a lossless coding module. The binary code stream can be transmitted through a signal transmission and restored to The compact feature representation y is input into the hyper-encoder to predict the distribution of the latent feature.

[0101] As shown in Figure 8 , the step S600 specifically includes the following steps:

[0102] Step S610, the feature representation y is input into a multi-modal feature fusion module together with the text and audio features after passing through two convolution layers. Since the distribution of the latent feature is closely related to the content of the image, and the text and audio description can provide semantic information of the image, the module can help predict the distribution of the latent feature, and then a convolution layer is used to obtain z.

[0103] Step S620: after z is quantized, it is converted into a binary code stream through a lossless coding module, and then restored to After passing through the hyper-decoder composed of three convolution modules, the distribution of the latent feature is obtained and transmitted to the corresponding lossless coding and decoding module.

[0104] Step S700, the feature received after lossless decoding is input into a convolution layer and a ResModule. The purpose is to process the received complex features, generate rich features, and provide better training stability.

[0105] Step S800, the feature representation is reconstructed through a plurality of up-sampling convolution layers, and then a ResBlock is used to prevent gradient vanishing or gradient explosion. Finally, a convolution layer is used to obtain the reconstructed image

[0106] Step S900, the reconstructed image and the quantized latent variable are input into the discriminator for adversarial training.

[0107] The beneficial effects of the multimodal guided high-fidelity image compression method of the present invention are:

[0108] This paper utilizes semantically relevant modalities (text, audio) to guide high-fidelity image compression, exploring for the first time the guiding role of audio in the image compression process. Furthermore, the designed feature fusion module not only provides local, fine-grained image feature guidance in the spatial dimension, but also globally guides image features in the channel dimension. Simultaneously, this module leverages the advantages of a multi-scale structure, enabling excellent feature fusion capabilities while also enhancing the network's nonlinear expression capabilities.

[0109] To achieve the above objectives, the present invention also proposes a multi-modality guided high-fidelity image compression system, such as Figure 9 As shown, the system includes a processor 1001, a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002, and a multimodal guided high-fidelity image compression program stored on the processor, wherein the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0110] Those skilled in the art will understand that Figure 9 The system structure shown in the figure does not constitute a limitation of the system, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0111] like Figure 9 As shown, the memory 1005 as a computer storage medium may include an operating device, a network communication module, a user interface module, and a multi-modality guided high-fidelity image compression program.

[0112] exist Figure 9 In the system shown, the network interface 1004 is mainly used to connect to the network server and communicate data with the network server; the user interface 1003 is mainly used to interact with the user terminal and receive instructions input by the user; and the processor 1001 can be used to call the multimodal guided high-fidelity image compression program stored in the memory 1005.

[0113] To achieve the above-mentioned objectives, the present invention also proposes a computer-readable storage medium, which stores a multi-modality-guided high-fidelity image compression program. When the multi-modality-guided high-fidelity image compression program is run by a processor, the steps of the method described above are executed, which will not be repeated here.

[0114] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present invention description and drawings under the concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A multimodality guided high-fidelity image compression method, characterized in that: The following steps are involved: Step S100, pre-training a text and audio encoder: the text and audio encoder maps image features, text and audio features into a common semantic space by calculating the degree of matching with the image; Step S200: Input the text t and audio a into the pre-trained text and audio encoder to extract the text feature f t With audio feature f a ; Step S300: The input image x undergoes linear transformation through three learnable convolutional layers to obtain image features f I ; Step S400: Fusion of text features f t , audio features f a With image feature f I , after a linear transformation through a learnable convolutional layer, a new feature map is obtained, and the new feature map is combined with the text feature f t With audio feature f a Fuse again to obtain the re-fused features, and perform linear transformation on the re-fused features through a learnable convolutional layer; Step S500: Input the linearly transformed feature map into the residual module to enhance the feature representation capability, optimize the training process, and improve the model performance; the linearly transformed feature map is further processed by the convolution layer to obtain a compact feature representation y; Step S600: The compact feature representation y is quantized and converted into a binary code stream through a lossless encoding module. The binary code stream is transmitted through a signal and restored to a binary code stream through a lossless decoding module. The compact feature representation y is fed into the super-encoder to predict the distribution of latent features; Step S700: restore the features obtained by lossless decoding Generate rich features through a convolutional layer and a residual module; Step S800: The rich features are passed through multiple upsampling convolution layers to reconstruct feature representation, and then passed through a residual block to prevent gradient disappearance or gradient explosion, and finally passed through a convolution layer to obtain a reconstructed image. Step S900: Reconstructing the image and features restored by lossless decoding Input the discriminator for adversarial training.

2. The multimodality-guided high-fidelity image compression method according to claim 1, characterized in that: The step S100 includes: Step S110: Represent the input text t and audio a as low-dimensional embedding features, where the text encoder uses a bidirectional long short-term memory recurrent neural network, and the audio encoder uses a convolutional neural network and a recurrent neural network; Step S120: using a pre-trained image encoder to represent the input image x as a low-dimensional embedded feature, and the image encoder uses a convolutional neural network to map the image into a semantic vector; Step S130: training the text encoder and image encoder in steps S110 and S120 using a deep attention multimodal similarity model, wherein an attention-driven image-text matching score is used during the training process to measure the matching degree of image-sentence pairs based on an attention model between image and text, so that the trained text and image encoders can effectively map text and image features to a common semantic space; Step S140: Train the audio encoder through the same steps, so that the audio and image encoders can effectively map the audio and image features to a common semantic space.

3. The multimodality-guided high-fidelity image compression method according to claim 2, characterized in that: The step S300 includes: Step S310: Design a feature fusion module, wherein the feature fusion module only fuses the two modalities of image and text or audio; the calculation formula used by the designed feature fusion module is: F′ t / a =conv3×3(F t / a ) γ=σ(conv 3×3 (F′ t / a )) β=conv 1×1 (g(F′ t / a )) F′ image =γ·F image +b Among them, F t / a represents text features or speech features, F′ t / a represents the text features or speech features after the 3×3 convolution operation, σ represents the Sigmoid activation function, g represents the average pooling operation, Y and β represent the scaling factor and offset factor respectively, F image and F′ image Represent the image features before and after processing respectively, and conv represents the convolution operation; Step S320: Design a multimodal feature fusion module. The multimodal feature fusion module can meet the requirements of fusing three modalities while fusing two modal features. The formula used in designing the multimodal feature fusion module is: F IT =fusion(F image ,F text ) F IA =fusion(F image ,F audio ) F ITA =conv 1×1 (concat(F IT ,F IA )) Among them, fusion means using feature fusion module to perform feature fusion operation, F image 、F text and F audio Represent image, text and speech features respectively, F IT 、F IA and F ITA They represent image-text, image-audio, and image-text-audio fusion features respectively. concat represents the concatenation operation in the channel dimension; conv represents the convolution operation.

4. The multimodality-guided high-fidelity image compression method according to claim 3, wherein: The step S310 includes: Step S311: convert the text feature or voice feature F t / a The input is fed into a 3×3 convolutional layer to transform it into the common semantic space of image features; Step S312: Perform a 3×3 convolution operation on the feature map generated in the previous step to calculate the scaling factor Y at each position. Each position of the feature map generated in this step represents a scaling factor Y, which controls the specific adjustment of the image features. Then, the convolution output is passed through the Sigmoid activation function to generate a scaling factor in the range of [0, 1], which is used to control the scaling degree of the image features. Each element of the feature map can be enlarged or reduced according to its corresponding Y value, thereby adjusting the details of the feature map. Step S313: First, perform global average pooling on the channel dimension of the feature map through the Average Pool operation to calculate the average value of each channel. This step aggregates the spatial information of each channel into a single value, thereby generating channel-level statistical information. Apply a 1×1 convolution to the pooled result to generate the offset factor β for each channel; Step S314: Manipulate the input image feature F by the scaling factor Y and the offset factor β image .

5. The multimodality-guided high-fidelity image compression method according to claim 4, characterized in that: The step S320 includes: Step S321: input the image features and text features into the same feature fusion module, and input the image features and audio features into another feature fusion module, and the two modules output their respective fusion features; Step S322: Perform concat operation on the two output fusion features on the channel to splice them into a larger feature, providing rich information containing three modal features for subsequent processing; Step S323: After concatenating the features, a 1×1 convolution is performed to adjust the number of channels so that the final feature map has an appropriate dimension.

6. The multimodality-guided high-fidelity image compression method according to claim 5, characterized in that: The step S600 includes: Step S610: The compact feature representation y passes through two convolutional layers and is then input into a multimodal feature fusion module along with the text and audio features. Because the distribution of latent features is closely related to the image content, and text and audio descriptions can provide semantic information about the image, the multimodal feature fusion module helps predict the distribution of latent features. Afterwards, z is obtained by passing it through another convolutional layer. Step S620: z is quantized and converted into a binary code stream for transmission through the lossless encoding module, and then restored to After passing through the super decoder composed of three convolutional modules, the distribution of latent features is obtained and transmitted to the corresponding lossless codec module.

7. A multimodality guided high-fidelity image compression system, characterized in that: The system includes a memory, a processor, and a multi-modality guided high-fidelity image compression program stored on the processor, wherein the multi-modality guided high-fidelity image compression program is executed by the processor to perform the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a multi-modality-guided high-fidelity image compression program, and the multi-modality-guided high-fidelity image compression program is executed by a processor to perform the steps of the method according to any one of claims 1 to 6.

Images