Image compression method and device

Abstract

The present application relates to the field of image processing technology, and specifically provides an image compression method and device, which aims to solve the problem that existing image compression is not suitable for the joint compression of visible light and infrared image pairs and cannot be encoded and decoded simultaneously. To this end, the image compression method of the present application includes: inputting the visible light image and thermal infrared image before compression into a pre-trained image compression model; using the image compression model to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and target thermal infrared image. By simultaneously compressing the visible light image and the thermal infrared image, the present application enables the visible light image and the thermal infrared image to be jointly compressed, significantly improving the compression performance and real-time performance of the image, and significantly improving the efficiency of the joint compression of the image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image compression method and device. Background Art

[0002] In recent years, the application of multimodal data has seen explosive growth in fields such as disaster monitoring, security surveillance, and military reconnaissance. In many practical applications, monitoring and reconnaissance tasks no longer rely solely on a single modality of data as input. Instead, multimodal data, such as visible light and thermal infrared imagery, is used in combination. This places higher demands on image compression, requiring higher compression efficiency and improved real-time performance. However, while multimodal data brings performance improvements, it also brings an exponential increase in data, creating unprecedented challenges for transmission bandwidth and storage space.

[0003] Existing joint compression methods for visible light and thermal infrared images often choose to compress images of one modality first and then the other modality, which results in low image compression efficiency and an exponential increase in encoding time, making it impossible to achieve good real-time processing.

[0004] There is modal redundancy between multimodal data captured in the same scene. Existing technologies provide cross-modal joint compression frameworks, which first compress images of one modality and use the compressed modality as prior information to assist in the compression of data from another modality, thereby reducing inter-modal redundancy. This solution can save transmission bandwidth and storage space pressure, and on the other hand, improve the average fidelity of multi-modal data. However, they also have the following problems: (1) Some frameworks are for visible light images paired with depth maps or hyperspectral images. Since the distribution of infrared images is different from that of depth maps or hyperspectral images, they are not suitable for joint compression of visible light-infrared image pairs. (2) Existing joint compression frameworks for visible light-thermal infrared image pairs require serial encoding and decoding of visible light and thermal infrared images, and cannot encode and decode simultaneously. Summary of the Invention

[0005] This application addresses the deficiencies of the above-mentioned prior art and proposes an image compression method and device.

[0006] In a first aspect, an embodiment of the present application provides an image compression method, comprising: inputting a visible light image and a thermal infrared image before compression into a pre-trained image compression model; and using the image compression model to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image.

[0007] In second aspect, an embodiment of the present application provides an image compression device, comprising: an input module for inputting a visible light image and a thermal infrared image before compression into a pre-trained image compression model; a compression module for using the image compression model to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image.

[0008] The technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0009] The embodiment of the present application inputs the visible light image and thermal infrared image before compression into a pre-trained image compression model, and uses the image compression model to simultaneously compress the visible light image and thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image, so that the visible light image and the thermal infrared image can be jointly compressed, which significantly improves the compression performance and real-time performance of the image, and significantly improves the joint compression efficiency of the image.

[0010] Additional aspects and advantages of the present application will be given in part in the description below and in part will become apparent from the description below or learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0012] Figure 1 A flowchart of an image compression method provided by an embodiment of the present application is shown;

[0013] Figure 2 A schematic diagram of the structure of the image compression model provided in an embodiment of the present application is shown;

[0014] Figure 3 A schematic diagram showing the structure of encoding a first image feature by a feature encoding module of an image compression model provided in an embodiment of the present application is shown;

[0015] Figure 4 A schematic diagram showing the structure of encoding the second image feature by the feature encoding module of the image compression model provided in an embodiment of the present application is shown;

[0016] Figure 5 A schematic diagram showing the structure of a feature reconstruction module of an image compression model provided in an embodiment of the present application is shown;

[0017] Figure 6A schematic structural diagram of an image compression device provided in an embodiment of the present application is shown;

[0018] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] The following describes exemplary embodiments of the present application in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0020] In recent years, the application of multimodal data has seen explosive growth in fields such as disaster monitoring, security surveillance, and military reconnaissance. In many practical applications, monitoring and reconnaissance tasks no longer rely solely on a single modality of data as input. Instead, multimodal data, such as visible light and thermal infrared imagery, is used in combination. This places higher demands on image compression, requiring higher compression efficiency and improved real-time performance. However, while multimodal data brings performance improvements, it also brings an exponential increase in data, creating unprecedented challenges for transmission bandwidth and storage space.

[0021] Existing joint compression methods for visible light and thermal infrared images often choose to compress images of one modality first and then the other modality, which results in low image compression efficiency and an exponential increase in encoding time, making it impossible to achieve good real-time processing.

[0022] There is modal redundancy between multimodal data captured in the same scene. Existing technologies provide cross-modal joint compression frameworks, which first compress images of one modality and use the compressed modality as prior information to assist in the compression of data from another modality, thereby reducing inter-modal redundancy. This solution can save transmission bandwidth and storage space pressure, and on the other hand, improve the average fidelity of multi-modal data. However, they also have the following problems: (1) Some frameworks are for visible light images paired with depth maps or hyperspectral images. Since the distribution of infrared images is different from that of depth maps or hyperspectral images, they are not suitable for joint compression of visible light-infrared image pairs. (2) Existing joint compression frameworks for visible light-thermal infrared image pairs require serial encoding and decoding of visible light and thermal infrared images, and cannot encode and decode simultaneously.

[0023] Based on this, an embodiment of the present application provides an image compression method. The solution of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0024] See also Figure 1A flowchart of an image compression method is shown, the method specifically comprising the following steps:

[0025] Step 101: Input the visible light image and thermal infrared image before compression into a pre-trained image compression model.

[0026] Step 102: Using an image compression model, the visible light image and the thermal infrared image are compressed simultaneously to obtain a compressed target visible light image and a target thermal infrared image.

[0027] See also Figure 2 The structural schematic diagram of the image compression model shown in FIG. 1 , in one embodiment, the image compression model may include a feature extraction module, a feature encoding module, and a feature reconstruction module. The feature extraction module may be used to perform feature extraction on the visible light image and the thermal infrared image respectively to obtain a first image feature corresponding to the visible light image and a second image feature corresponding to the thermal infrared image. The feature encoding module may then be used to calculate a first probability distribution of the first image feature and a second probability distribution corresponding to the second image feature based on the first image feature and the second image feature, and the first image feature may be encoded based on the first probability distribution and the second image feature may be encoded based on the second probability distribution. Finally, the feature reconstruction module may be used to perform reconstruction based on the encoded first image feature and the encoded second image feature to obtain a compressed target visible light image and a target thermal infrared image.

[0028] By using the feature coding module to simultaneously encode the first image feature and the second image feature, the problem of being unable to simultaneously encode and decode visible light and thermal infrared images is effectively overcome. At the same time, the speed of image compression can be significantly improved, with good compression performance and real-time performance.

[0029] Based on the above embodiments, in some modified embodiments, the feature extraction module may include a first feature extraction module and a second feature extraction module, and the feature extraction modules are used to perform feature extraction on the visible light image and the thermal infrared image respectively to obtain the first image feature corresponding to the visible light image and the second image feature corresponding to the thermal infrared image. Specifically, it may include: using the first feature extraction module to perform feature extraction on the visible light image and the thermal infrared image respectively to obtain the first candidate feature corresponding to the visible light image and the second candidate feature corresponding to the thermal infrared image, and then using the second feature extraction module to calculate the first image feature corresponding to the visible light image and the second image feature corresponding to the thermal infrared image based on the first candidate feature and the second candidate feature.

[0030] Based on the above embodiments, in some modified embodiments, the first feature extraction module may include a first residual network and a first self-attention unit, and the first feature extraction module is used to perform feature extraction on the visible light image to obtain the first candidate feature corresponding to the visible light image. Specifically, it may include: converting the color format of the visible light image into a preset format, the preset format of the visible light image includes multiple first channels, and then using the first residual network corresponding to each first channel to perform feature extraction on the pixel values ​​corresponding to each first channel to obtain multiple first sub-candidate features, and downsampling the first sub-candidate features corresponding to each pre-specified first channel to obtain multiple second sub-candidate features, and using the first self-attention unit corresponding to each first channel to perform feature extraction on each first sub-candidate feature that has not been downsampled and each second sub-candidate feature that has been downsampled to obtain multiple third sub-candidate features. Finally, the multiple third sub-candidate features are spliced ​​in the channel dimension to obtain the first candidate feature.

[0031] It should be noted that the color format of the visible light image may include RGB color mode, Bitmap mode, Grayscale mode, Duotone mode, Index color mode, CMYK color mode, Lab color mode, Multichannel mode, HSB color mode and other formats, which are not specifically limited in the embodiments of this application.

[0032] The preset format may refer to a YUV format, which is a color encoding method mainly used for data transmission in a compressed image process.

[0033] On the basis of the above embodiments, in some modified embodiments, the visible light image in the preset format includes a first channel Y, a first channel U, and a first channel V. Then, the first residual network corresponding to each first channel is used to perform feature extraction on the pixel values ​​corresponding to each first channel to obtain a plurality of first sub-candidate features, and the first sub-candidate features corresponding to each pre-specified first channel are downsampled to obtain a plurality of second sub-candidate features. The first self-attention unit corresponding to each first channel is used to calculate each first sub-candidate feature that has not been downsampled and each second sub-candidate feature that has been downsampled by the following formula (1) to obtain a plurality of third sub-candidate features:

[0034]

[0035] Among them, when F0 represents the third sub-candidate feature corresponding to the first channel Y, F0 represents the first sub-candidate feature corresponding to the first channel Y, Represents the weight matrix of the query vector Q0, key vector K0, and value vector V0 mapped to the first sub-candidate feature corresponding to the first channel Y; when F0 represents the third sub-candidate feature corresponding to the first channel U, F0 represents the first sub-candidate feature corresponding to the first channel U, Represents the weight matrix of the query vector Q0, key vector K0, and value vector V0 mapped to the first sub-candidate feature corresponding to the first channel U; when F0 represents the third sub-candidate feature corresponding to the first channel V, F0 represents the first sub-candidate feature corresponding to the first channel V, The weight matrix representing the query vector Q0, key vector K0, and value vector V0 to which the first candidate sub-feature corresponding to the first channel V is mapped; d k0 Represents the dimension of the key vector K0.

[0036] On the basis of the above-mentioned embodiments, in some modified embodiments, the first feature extraction module may further include a second residual network, a second self-attention unit and a first convolutional layer, and use the first feature extraction module to perform feature extraction on the thermal infrared image to obtain a second candidate feature corresponding to the thermal infrared image. Specifically, it may include: using the second residual network corresponding to the second channel of the thermal infrared image to perform feature extraction on the pixel value corresponding to the second channel to obtain a fourth sub-candidate feature, and then using the second self-attention unit corresponding to the second channel to perform feature extraction on the fourth sub-candidate feature to obtain a fifth sub-candidate feature, and finally using the first convolutional layer corresponding to the second channel to perform convolution processing on the fifth sub-candidate feature to obtain a second candidate feature.

[0037] On the basis of the above embodiment, in some modified embodiments, the fourth sub-candidate feature is extracted using the second self-attention unit corresponding to the second channel to obtain the fifth sub-candidate feature, which can also be calculated using the above formula (1), wherein when F0 represents the fifth sub-candidate feature corresponding to the second channel, F0 represents the fourth sub-candidate feature corresponding to the second channel, The weight matrix representing the query vector Q0, key vector K0, and value vector V0 to which the fourth sub-candidate feature corresponding to the second channel is mapped.

[0038] On the basis of the above embodiments, in some modified embodiments, the second feature extraction module may include a third residual network, a fourth residual network and a first cross-modal attention unit, a second cross-modal attention unit, and the second feature extraction module is used to respectively calculate the first image feature corresponding to the visible light image and the second image feature corresponding to the thermal infrared image based on the first candidate feature and the second candidate feature. Specifically, it may include: using the third residual network to downsample the first candidate feature to obtain a sixth sub-candidate feature, and using the fourth residual network to downsample the second candidate feature to obtain a seventh sub-candidate feature, and then using the first cross-modal attention unit to calculate the first image feature based on the sixth sub-candidate feature and the seventh sub-candidate feature by the following formula (2), and using the second cross-modal attention unit to calculate the second image feature based on the sixth sub-candidate feature and the seventh sub-candidate feature by the following formula (2):

[0039]

[0040] Among them, when F ′ When representing the first image feature, F1 represents the sixth sub-candidate feature, and F2 represents the seventh sub-candidate feature. Represents the weight matrix of the query vector Q1 to which the sixth child candidate feature is mapped, the key vector K1 to which the seventh child candidate feature is mapped, and the value vector V1; when F ′ When representing the second image feature, F1 represents the seventh sub-candidate feature, and F2 represents the sixth sub-candidate feature. The weight matrix representing the query vector Q1 to which the seventh sub-candidate feature is mapped, the key vector K1 to which the sixth sub-candidate feature is mapped, and the value vector V1; d k1 Indicates the dimension of the key vector K1.

[0041] By using the first cross-modal attention unit to calculate the first image features and using the second cross-modal attention unit to calculate the second image features, the correlation between the first image features corresponding to the visible light image and the second image features corresponding to the thermal infrared image can be effectively reduced, the redundancy between the first image features and the second image features can be reduced, and the most informative part of each feature can be captured more accurately, thereby effectively reducing the amount of data while maintaining the necessary information and improving the image compression ratio; in addition, the cross-modal attention mechanism of the first cross-modal attention unit and the second cross-modal attention unit helps to retain those features that have the greatest impact on visual perception, reduce blur or semantic distortion caused by compression, and thus improve the perceptual quality of the compressed image.

[0042] See also Figure 3 and 4As shown, based on the above-mentioned embodiments, in some modified embodiments, the feature encoding module may include a priori information acquisition unit, a low-frequency information acquisition unit and a probability distribution calculation unit, and utilize the feature encoding module to calculate the first probability distribution of the first image feature and the second probability distribution corresponding to the second image feature based on the first image feature and the second image feature. Specifically, it may include: utilizing the priori information acquisition unit to respectively acquire the first spatial domain prior information of the first image feature and the second spatial domain prior information of the second image feature, and then utilizing the low-frequency information acquisition unit to respectively acquire the global low-frequency fusion information of the first image feature and the global low-frequency information of the second image feature, and finally utilizing the probability distribution calculation unit to acquire the first probability distribution of the first image feature based on the first spatial domain prior information and the global low-frequency fusion information, and to acquire the second probability distribution of the second image feature based on the second spatial domain prior information and the global low-frequency information.

[0043] Based on the above-mentioned embodiments, in some modified embodiments, the first image feature may include a plurality of first sub-image features divided evenly into a preset order, the global low-frequency fusion information includes the global low-frequency fusion information corresponding to each first sub-image feature remaining except the first first sub-image feature, and the global low-frequency fusion information of the first image feature is obtained by using the low-frequency information acquisition unit. Specifically, the following may be included: all the first sub-image features before the second sub-image feature are obtained for decoding and splicing to obtain a first splicing feature, wherein the second sub-image feature is any one of the plurality of first sub-image features, and the second sub-image feature is not the first first sub-image feature, and the first first sub-image feature is the first sub-image feature sorted first according to the preset order, and the global low-frequency fusion information corresponding to the second sub-image feature is obtained by using the low-frequency information acquisition unit based on the second image feature and the first splicing feature.

[0044] It should be noted that the preset order refers to the order in which the first image features are segmented. The preset order can be from top to bottom and from left to right, or from bottom to top and from right to left, or from top to bottom and from right to left, or from bottom to top and from left to right, and is not specifically limited in the embodiments of the present application.

[0045] For example, assuming that the first image feature is evenly divided into five first sub-image features according to a preset order, it is necessary to obtain global low-frequency fusion information corresponding to the second first sub-image feature, the third first sub-image feature, the fourth first sub-image feature, and the fifth first sub-image feature. Acquiring global low-frequency fusion information corresponding to each first sub-image feature requires the corresponding low-frequency information acquisition unit and probability distribution calculation unit to perform corresponding operations, which will not be repeated here.

[0046] Based on the above embodiments, in some modified embodiments, the low-frequency information acquisition unit includes a first low-frequency information extraction unit, a second convolutional layer, a second low-frequency information extraction unit, and a low-frequency information fusion unit. The low-frequency information acquisition unit is used to obtain global low-frequency fusion information corresponding to the second sub-image feature based on the second image feature and the first splicing feature. Specifically, the method may include: using the first low-frequency information extraction unit to obtain first global low-frequency information based on the first splicing feature, using the second convolutional layer and the second low-frequency information extraction unit to obtain second global low-frequency information based on the second image feature, and then using the low-frequency information fusion unit to fuse the first global low-frequency information and the second global low-frequency information to obtain global low-frequency fusion information corresponding to the second sub-image feature.

[0047] For example, assuming that the global low-frequency fusion information corresponding to the third first sub-image feature is to be obtained, the first first sub-image feature, the second first sub-image feature and the third first sub-image feature are decoded and spliced ​​to obtain a first splicing feature. The first low-frequency information extraction unit is used to obtain the first global low-frequency information based on the first splicing feature. The second convolutional layer and the second low-frequency information extraction unit are used to obtain the second global low-frequency information based on the second image feature. Then, the low-frequency information fusion unit is used to fuse the first global low-frequency information and the second global low-frequency information to obtain the global low-frequency fusion information corresponding to the third first sub-image feature.

[0048] On the basis of the above embodiment, in some modified embodiments, the low-frequency information fusion unit may be used to calculate the global low-frequency fusion information corresponding to each first sub-image feature using the following formula (3):

[0049]

[0050] Among them, G fusionInfo Represents the global low-frequency fusion information corresponding to the N1th first sub-image feature, the G info Represents the global low-frequency information corresponding to the N1th first sub-image feature, V ′ represents a temporary variable, A represents a temporary matrix, F3 represents the first splicing feature, F4 represents the second image feature, The weight matrix d represents the query vector Q2 to which the first concatenated feature is mapped, the key vector K2 to which the second image feature is mapped, and the value vector V2. k2 represents the dimension of the key vector K2, d a Indicates the dimensions of the temporary matrix A.

[0051] Based on the above embodiments, in some modified embodiments, the low-frequency information acquisition unit may further include a third low-frequency information extraction unit. The low-frequency information acquisition unit is used to obtain global low-frequency information of the second image feature. Specifically, the second image feature may include a plurality of third sub-image features evenly divided into a plurality of third sub-image features according to a preset order, and all third sub-image features before the N2th third sub-image feature are obtained for decoding and splicing to obtain a second spliced ​​feature, wherein the N2th third sub-image feature is any one of the plurality of third sub-image features, and the N2th third sub-image feature is not the first third sub-image feature, and the first third sub-image feature is the third sub-image feature ranked first according to the preset order. The third low-frequency information extraction unit is used to obtain global low-frequency information of the second image feature based on the second image feature.

[0052] On the basis of the above embodiment, in some modified embodiments, the third low-frequency information extraction unit may be used to calculate the global low-frequency information of the second image feature using the following formula (4):

[0053]

[0054] Among them, G info Represents the global low-frequency information corresponding to the second image feature, V 1′ represents a temporary variable, A1 represents a temporary matrix, F5 represents the second image feature, The weight matrix d represents the query vector Q3, key vector K3 and value vector V3 to which the second image feature is mapped. k3 represents the dimension of the key vector K3, d a 1 represents the dimension of the temporary matrix A1.

[0055] On the basis of the above-mentioned embodiments, in some modified embodiments, a probability distribution calculation unit is used to obtain a first probability distribution of the first image feature based on the first spatial domain prior information and the global low-frequency fusion information. Specifically, the first probability distribution may include a third probability distribution corresponding to multiple first sub-image features. The third probability distribution corresponding to the first first sub-image feature may be obtained using the probability distribution calculation unit based on the first spatial domain prior information, and then the remaining multiple first sub-image features may be obtained using the probability distribution calculation unit based on the first spatial domain prior information and the global low-frequency fusion information corresponding to each first sub-image feature.

[0056] Furthermore, a probability distribution calculation unit is used to obtain a second probability distribution of the second image feature based on the second spatial domain prior information and the global low-frequency information.

[0057] Furthermore, the first probability distribution and the second probability distribution can be expressed as the following formula (5):

[0058]

[0059] in, represents the first probability distribution, represents the first spatial domain prior information, N1 represents the number of first sub-image features, represents the second probability distribution, represents the second spatial domain prior information, and N2 represents the number of third sub-image features.

[0060] See also Figure 5 As shown, based on the above embodiment, in some modified embodiments, a feature reconstruction module is used to perform reconstruction processing based on the encoded first image features and second image features to obtain a compressed target visible light image and target thermal infrared image. The feature reconstruction module and the feature extraction module have similar processes, mainly performing feature reconstruction on the encoded first image features and second image features, thereby converting the encoded first image features and second image features from the feature domain to the pixel domain to generate the final compressed image.

[0061] First, the encoded first image features are upsampled through the fifth residual network and vector mapped using the cross-modal attention mechanism of the third cross-modal attention unit. The encoded second image features are upsampled through the sixth residual network and vector mapped using the cross-modal attention mechanism of the fourth cross-modal attention unit. Subsequently, the feature map of the thermal infrared modality is subjected to a 1x1 convolution operation using the seventh convolutional layer. The feature map of the visible light modality is divided into feature maps of three channels, Y, U, and V, from the channel dimension. Finally, the feature maps of the four channels are respectively applied with the self-attention mechanism through the third self-attention unit and the fourth self-attention unit and upsampled through the seventh residual network and the eighth residual network, and finally the compressed visible light image and thermal infrared image are reconstructed. Among them, the calculation process of the self-attention mechanism and the cross-modal attention mechanism is the same as the calculation process of the self-attention mechanism and the cross-modal attention mechanism used in the feature extraction module in the above embodiment, and will not be repeated here.

[0062] On the basis of the above embodiment, in some modified embodiments, before the visible light image and the thermal infrared image before compression are input into the pre-trained image compression model, the image compression model can be trained. During the training process, the probability distribution of the visible light image to be trained and the probability distribution of the thermal infrared image to be trained are calculated using a probability distribution calculation unit. Then, the bit rate of the visible light image to be trained and the bit rate of the thermal infrared image to be trained are estimated based on the probability distribution of the visible light image to be trained and the probability distribution of the thermal infrared image to be trained. Then, the loss value of the image compression model is calculated using the loss function shown in the following formula (6). If the loss value is less than or equal to a preset value, the image compression model is obtained:

[0063] L=R ir +R r +λ(D ir +D r ). (5)

[0064] Among them, R r Indicates the bit rate of the visible light image to be trained, R ir represents the bit rate of the thermal infrared image to be trained, λ represents the hyperparameter, and D r Denotes the pixel-level mean square error between the visible light image to be trained and the compressed visible light image, D ir Represents the pixel-level mean square error between the thermal infrared image to be trained and the compressed thermal infrared image.

[0065] The embodiment of the present application inputs the visible light image and thermal infrared image before compression into a pre-trained image compression model, and uses the image compression model to simultaneously compress the visible light image and thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image, so that the visible light image and the thermal infrared image can be jointly compressed, which significantly improves the compression performance and real-time performance of the image, and significantly improves the joint compression efficiency of the image.

[0066] In order to verify the effectiveness of the image compression method in the embodiment of the present application for the joint compression of visible light and thermal infrared images, the image compression method in the embodiment of the present application is used to compare the rate-distortion performance of the best end-to-end single-modality image encoder Mlic++, the classic end-to-end image encoder Cheng2020, the traditional image encoder BPG, and the joint compression method proposed by this method on the LLVIP dataset. The delta rate (BD-Rate) is used as a metric to evaluate the rate-distortion performance of various image encoders. Notably, to ensure a fair comparison, we re-deployed and re-trained other end-to-end compression frameworks on the LLVIP dataset. Compared to other single-modal encoding frameworks, our proposed framework achieves significant improvements in BD-Rate performance. Specifically, our approach outperforms MLIC++ and BPG by 14.6% and 26.8%, respectively. These results clearly demonstrate that our proposed approach significantly outperforms other methods in terms of compression performance.

[0067] In addition, in order to verify the effectiveness of the image compression method in the embodiment of the present application for jointly compressing visible light and thermal infrared images, the image compression method in the embodiment of the present application introduced an ablation experiment. The results showed that the feature encoding module and the low-frequency information acquisition unit in the image compression model in the embodiment of the present application had BD-Rates that were 19.34% and 9.17% lower than the original model, respectively, which proves that these two modules effectively improved the overall encoding performance.

[0068] See also Figure 6 The present application also provides an image compression device, which is used to execute the image compression method described in the above embodiment. The device includes:

[0069] An input module 201 is used to input the visible light image and thermal infrared image before compression into a pre-trained image compression model;

[0070] The compression module 202 is configured to simultaneously compress the visible light image and the thermal infrared image using the image compression model to obtain a compressed target visible light image and a target thermal infrared image.

[0071] The image compression device provided in the embodiment of the present application and the image compression method provided in the above embodiment are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0072] The present application also provides an electronic device corresponding to the image compression method provided in the above embodiment. Figure 7 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 4 As shown, the electronic device 30 may include: a processor 300, a memory 301, a bus 302 and a communication interface 303, and the processor 300, the communication interface 303 and the memory 301 are connected via the bus 302; the memory 301 stores a computer program that can be run on the processor 300, and when the processor 300 runs the computer program, it executes the image compression method provided in any of the aforementioned embodiments of the present application.

[0073] Memory 301 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. Communication between the system network element and at least one other network element is achieved through at least one physical port 303 (which may be wired or wireless), and may use the Internet, a wide area network, a local area network, a metropolitan area network, etc.

[0074] The bus 302 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs. The processor 300 executes the programs upon receiving execution instructions. The image compression method disclosed in any of the aforementioned embodiments of the present application may be applied to the processor 300 or implemented by the processor 300.

[0075] The processor 300 may be an integrated circuit with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 300 or by software instructions. The above processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 301 , and the processor 300 reads the information in the memory 301 and completes the steps of the above method in combination with its hardware.

[0076] The electronic device provided in the embodiment of the present application and the image compression method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by them.

[0077] An embodiment of the present application also provides a computer-readable storage medium corresponding to the image compression method provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the image compression method provided in any of the aforementioned embodiments.

[0078] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0079] An embodiment of the present application further provides a computer program product corresponding to the image compression method provided in the aforementioned embodiment, including a computer program, which is executed by a processor to implement the image compression method provided in the aforementioned embodiments.

[0080] The computer-readable storage medium and computer program product provided in the above-mentioned embodiments of the present application are based on the same inventive concept as the image compression method provided in the embodiments of the present application, and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0081] It should be noted that:

[0082] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other device. Various general-purpose devices may also be used in conjunction with the teachings herein. Based on the above description, it is apparent that the structure required for constructing such devices is suitable. In addition, the present application is not directed to any specific programming language. It should be understood that various programming languages ​​may be utilized to implement the present application described herein, and the above description of specific languages ​​is provided for the purpose of disclosing the best mode of implementation of the present application.

[0083] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0084] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in fewer than all the features of the individual embodiments disclosed above. Accordingly, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim itself serving as a separate embodiment of the present application.

[0085] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0086] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0087] The various component embodiments of the present application can be implemented in hardware, or implemented in a software module running on one or more processors, or implemented in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the creation device of the virtual machine according to an embodiment of the present application. The application can also be implemented as a part or all of the equipment or device program (for example, computer program and computer program product) for performing the method described herein. Such a program realizing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0088] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0089] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image compression method, characterized in that: The method comprises: Inputting the visible light image and thermal infrared image before compression into a pre-trained image compression model; Using the image compression model, the visible light image and the thermal infrared image are simultaneously compressed to obtain a compressed target visible light image and a target thermal infrared image; The image compression model includes a feature extraction module and a feature encoding module. The image compression model is used to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image, including: Using the feature extraction module to perform feature extraction on the visible light image and the thermal infrared image respectively, to obtain a first image feature corresponding to the visible light image and a second image feature corresponding to the thermal infrared image; Utilizing the feature encoding module, based on the first image feature and the second image feature, calculate a first probability distribution of the first image feature and a second probability distribution corresponding to the second image feature, and encoding the first image feature based on the first probability distribution and encoding the second image feature based on the second probability distribution; The feature encoding module includes a priori information acquisition unit, a low-frequency information acquisition unit, and a probability distribution calculation unit. Calculating a first probability distribution of the first image feature and a second probability distribution corresponding to the second image feature based on the first image feature and the second image feature using the feature encoding module includes: utilizing the prior information acquisition unit to respectively acquire first spatial domain prior information of the first image feature and second spatial domain prior information of the second image feature; utilizing the low-frequency information acquisition unit to respectively acquire global low-frequency fusion information of the first image feature and global low-frequency information of the second image feature; The probability distribution calculation unit is used to obtain the first probability distribution of the first image feature based on the first spatial domain prior information and the global low-frequency fusion information, and the second probability distribution of the second image feature is obtained based on the second spatial domain prior information and the global low-frequency information.

2. The image compression method according to claim 1, wherein: The image compression model further includes a feature reconstruction module, which uses the image compression model to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image, including: The feature reconstruction module is used to perform reconstruction processing based on the encoded first image features and second image features to obtain a compressed target visible light image and a target thermal infrared image.

3. The image compression method according to claim 2, wherein: The feature extraction module includes a first feature extraction module and a second feature extraction module, and the feature extraction modules are used to extract features from the visible light image and the thermal infrared image respectively to obtain a first image feature corresponding to the visible light image and a second image feature corresponding to the thermal infrared image, including: Using the first feature extraction module to perform feature extraction on the visible light image and the thermal infrared image respectively, to obtain a first candidate feature corresponding to the visible light image and a second candidate feature corresponding to the thermal infrared image; The second feature extraction module is used to calculate, based on the first candidate feature and the second candidate feature, a first image feature corresponding to the visible light image and a second image feature corresponding to the thermal infrared image.

4. The image compression method according to claim 3, wherein: The first feature extraction module includes a first residual network and a first self-attention unit, and the first feature extraction module is used to extract features from the visible light image to obtain a first candidate feature corresponding to the visible light image, including: converting a color format of the visible light image into a preset format, wherein the visible light image in the preset format includes a plurality of first channels; Using the first residual network corresponding to each of the first channels, respectively, to perform feature extraction on the pixel values ​​corresponding to each of the first channels to obtain a plurality of first sub-candidate features, and respectively, performing downsampling processing on the first sub-candidate features corresponding to each of the pre-specified first channels to obtain a plurality of second sub-candidate features; Using the first self-attention units corresponding to the first channels, respectively, to perform feature extraction on each of the first sub-candidate features that have not been downsampled and each of the second sub-candidate features that have been downsampled, to obtain a plurality of third sub-candidate features; Multiple third sub-candidate features are concatenated in the channel dimension to obtain the first candidate feature.

5. The image compression method according to claim 3, wherein: The first feature extraction module includes a second residual network, a second self-attention unit, and a first convolutional layer. The first feature extraction module is used to extract features from the thermal infrared image to obtain a second candidate feature corresponding to the thermal infrared image, including: performing feature extraction on pixel values ​​corresponding to the second channel of the thermal infrared image using a second residual network corresponding to the second channel to obtain a fourth sub-candidate feature; performing feature extraction on the fourth sub-candidate feature using the second self-attention unit corresponding to the second channel to obtain a fifth sub-candidate feature; Perform convolution processing on the fifth sub-candidate feature using the first convolution layer corresponding to the second channel to obtain the second candidate feature.

6. The image compression method according to claim 3, wherein: The second feature extraction module includes a third residual network, a fourth residual network, a first cross-modal attention unit, and a second cross-modal attention unit. The second feature extraction module is used to calculate the first image feature corresponding to the visible light image and the second image feature corresponding to the thermal infrared image based on the first candidate feature and the second candidate feature, respectively, including: Downsampling the first candidate feature using the third residual network to obtain a sixth sub-candidate feature, and downsampling the second candidate feature using the fourth residual network to obtain a seventh sub-candidate feature; Using the first cross-modal attention unit, based on the sixth sub-candidate feature and the seventh sub-candidate feature, the first image feature is calculated using the following formula; and using the second cross-modal attention unit, based on the sixth sub-candidate feature and the seventh sub-candidate feature, the second image feature is calculated using the following formula: Among them, when F ′ When representing the first image feature, F1 represents the sixth sub-candidate feature, and F2 represents the seventh sub-candidate feature. The weight matrix representing the query vector Q1 to which the sixth sub-candidate feature is mapped, the key vector K1 and the value vector V1 to which the seventh sub-candidate feature is mapped; when F ′ When representing the second image feature, F1 represents the seventh sub-candidate feature, and F2 represents the sixth sub-candidate feature. A weight matrix representing the query vector Q1 to which the seventh sub-candidate feature is mapped, the key vector K1 and the value vector V1 to which the sixth sub-candidate feature is mapped; d k1 Indicates the dimension of the key vector K1.

7. The image compression method according to claim 1, wherein: The first image feature includes a plurality of first sub-image features that are evenly divided according to a preset order, the global low-frequency fusion information includes global low-frequency fusion information corresponding to each of the first sub-image features except the first first sub-image feature, and the acquiring of the global low-frequency fusion information of the first image feature by the low-frequency information acquiring unit includes: Obtaining all first sub-image features preceding a second sub-image feature, decoding and concatenating them to obtain a first concatenated feature, wherein the second sub-image feature is any one of the plurality of first sub-image features, and the second sub-image feature is not the first first sub-image feature, and the first first sub-image feature is the first sub-image feature that is ranked first according to the preset order; The low-frequency information acquisition unit is used to acquire global low-frequency fusion information corresponding to the second sub-image feature based on the second image feature and the first splicing feature.

8. The image compression method according to claim 7, wherein: The low-frequency information acquisition unit includes a first low-frequency information extraction unit, a second convolutional layer, a second low-frequency information extraction unit, and a low-frequency information fusion unit. The low-frequency information acquisition unit is used to acquire global low-frequency fusion information corresponding to the second sub-image feature based on the second image feature and the first splicing feature, including: Obtaining first global low-frequency information based on the first splicing feature using the first low-frequency information extraction unit; Obtaining second global low-frequency information based on the second image features using the second convolutional layer and the second low-frequency information extraction unit; The first global low-frequency information and the second global low-frequency information are fused by using the low-frequency information fusion unit to obtain global low-frequency fusion information corresponding to the second sub-image feature.

9. An image compression device, characterized in that: include: An input module, used to input the visible light image and thermal infrared image before compression into a pre-trained image compression model; a compression module, configured to simultaneously compress the visible light image and the thermal infrared image using the image compression model to obtain a compressed target visible light image and a target thermal infrared image; The image compression model includes a feature extraction module and a feature encoding module. The image compression model is used to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image, including: Using the feature extraction module to perform feature extraction on the visible light image and the thermal infrared image respectively, to obtain a first image feature corresponding to the visible light image and a second image feature corresponding to the thermal infrared image; Utilizing the feature encoding module, based on the first image feature and the second image feature, calculate a first probability distribution of the first image feature and a second probability distribution corresponding to the second image feature, and encoding the first image feature based on the first probability distribution and encoding the second image feature based on the second probability distribution; The feature encoding module includes a priori information acquisition unit, a low-frequency information acquisition unit, and a probability distribution calculation unit. Calculating a first probability distribution of the first image feature and a second probability distribution corresponding to the second image feature based on the first image feature and the second image feature using the feature encoding module includes: utilizing the prior information acquisition unit to respectively acquire first spatial domain prior information of the first image feature and second spatial domain prior information of the second image feature; utilizing the low-frequency information acquisition unit to respectively acquire global low-frequency fusion information of the first image feature and global low-frequency information of the second image feature; The probability distribution calculation unit is used to obtain the first probability distribution of the first image feature based on the first spatial domain prior information and the global low-frequency fusion information, and the second probability distribution of the second image feature is obtained based on the second spatial domain prior information and the global low-frequency information.

Images