Image compression method and device

Abstract

The invention relates to the technical field of image processing, particularly provides an image compression method and device, and aims to solve the problem that existing image compression is not suitable for joint compression of visible light-infrared image pairs and cannot be coded and decoded at the same time. In order to achieve the purpose, the image compression method comprises the steps that a visible light image and a thermal infrared image before compression are input into a pre-trained image compression model; and simultaneously compressing the visible light image and the thermal infrared image by using the image compression model to obtain a compressed target visible light image and a compressed target thermal infrared image. According to the method, the visible light image and the thermal infrared image are compressed at the same time, so that the visible light image and the thermal infrared image can be jointly compressed, the compression performance and the real-time performance of the image are remarkably improved, and the joint compression efficiency of the image is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image compression method and device. Background Art

[0002] In recent years, the application of multi-modal data has shown explosive growth, for example, in the fields of disaster monitoring, security monitoring, military reconnaissance, etc. In many practical applications, data of one modality is no longer used alone as input for tasks such as monitoring and reconnaissance. Instead, multi-modal data such as visible light and thermal infrared images are used together, which places higher requirements on image compression: higher compression efficiency and better real-time performance are required. However, while multi-modal data brings performance improvements, it also brings about an exponential increase in data, which brings unprecedented challenges to transmission bandwidth and storage space.

[0003] Existing visible light and thermal infrared image joint compression methods often choose to compress images of one modality first and then compress the other modality, which leads to low image compression efficiency and exponentially increases the encoding time, and cannot achieve good real-time processing.

[0004] There is modal redundancy between multimodal data captured in the same scene. The existing technology provides a cross-modal joint compression framework, that is, first compressing the image of one modality, and using the compressed modality as prior information to assist the compression of the other modality data, thereby reducing the redundancy between modalities. This solution can save transmission bandwidth and storage space pressure, and on the other hand, it also improves the average fidelity of multi-modal data. However, they also have the following problems: (1) Some frameworks are for visible light images paired with depth maps or hyperspectral images. Since the distribution of infrared images is different from that of depth maps or hyperspectral images, they are not suitable for joint compression of visible light-infrared image pairs. (2) The existing joint compression framework for visible light-thermal infrared image pairs requires serial encoding and decoding of visible light and thermal infrared images, and cannot encode and decode simultaneously. Summary of the invention

[0005] The present application proposes an image compression method and device to address the deficiencies of the above-mentioned prior art.

[0006] In a first aspect, an embodiment of the present application provides an image compression method, comprising: inputting a visible light image and a thermal infrared image before compression into a pre-trained image compression model; and using the image compression model to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image.

[0007] In a second aspect, an embodiment of the present application provides an image compression device, comprising: an input module for inputting a visible light image and a thermal infrared image before compression into a pre-trained image compression model; a compression module for using the image compression model to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image.

[0008] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0009] The embodiment of the present application inputs the visible light image and thermal infrared image before compression into a pre-trained image compression model, and uses the image compression model to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image, so that the visible light image and the thermal infrared image can be compressed jointly, which significantly improves the compression performance and real-time performance of the image, and significantly improves the joint compression efficiency of the image.

[0010] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0012] Figure 1 A flowchart of an image compression method provided by an embodiment of the present application is shown;

[0013] Figure 2 A schematic diagram of the structure of an image compression model provided in an embodiment of the present application is shown;

[0014] Figure 3 A schematic diagram showing a structure in which a feature encoding module of an image compression model provided in an embodiment of the present application encodes a first image feature;

[0015] Figure 4 A schematic diagram showing a structure in which a feature encoding module of an image compression model provided in an embodiment of the present application encodes a second image feature;

[0016] Figure 5 A schematic diagram showing the structure of a feature reconstruction module of an image compression model provided in an embodiment of the present application is shown;

[0017] Figure 6A schematic diagram of the structure of an image compression device provided in an embodiment of the present application is shown;

[0018] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0020] In recent years, the application of multi-modal data has shown explosive growth, for example, in the fields of disaster monitoring, security monitoring, military reconnaissance, etc. In many practical applications, data of one modality is no longer used alone as input for tasks such as monitoring and reconnaissance. Instead, multi-modal data such as visible light and thermal infrared images are used together, which places higher requirements on image compression: higher compression efficiency and better real-time performance are required. However, while multi-modal data brings performance improvements, it also brings about an exponential increase in data, which brings unprecedented challenges to transmission bandwidth and storage space.

[0021] Existing visible light and thermal infrared image joint compression methods often choose to compress images of one modality first and then compress the other modality, which leads to low image compression efficiency and exponentially increases the encoding time, and cannot achieve good real-time processing.

[0022] There is modal redundancy between multimodal data captured in the same scene. The existing technology provides a cross-modal joint compression framework, that is, first compressing the image of one modality, and using the compressed modality as prior information to assist the compression of the other modality data, thereby reducing the redundancy between modalities. This solution can save transmission bandwidth and storage space pressure, and on the other hand, it also improves the average fidelity of multi-modal data. However, they also have the following problems: (1) Some frameworks are for visible light images paired with depth maps or hyperspectral images. Since the distribution of infrared images is different from that of depth maps or hyperspectral images, they are not suitable for joint compression of visible light-infrared image pairs. (2) The existing joint compression framework for visible light-thermal infrared image pairs requires serial encoding and decoding of visible light and thermal infrared images, and cannot encode and decode simultaneously.

[0023] Based on this, the embodiment of the present application provides an image compression method. The solution of the embodiment of the present application is specifically described below in conjunction with the accompanying drawings.

[0024] See also Figure 1A flowchart of an image compression method is shown, the method specifically comprising the following steps:

[0025] Step 101: Input the visible light image and thermal infrared image before compression into a pre-trained image compression model.

[0026] Step 102: Using an image compression model, the visible light image and the thermal infrared image are compressed simultaneously to obtain a compressed target visible light image and a target thermal infrared image.

[0027] See also Figure 2 The structural schematic diagram of the image compression model shown in FIG. 1 , in one embodiment, the image compression model may include a feature extraction module, a feature encoding module and a feature reconstruction module. The feature extraction module may be used to perform feature extraction on the visible light image and the thermal infrared image, respectively, to obtain a first image feature corresponding to the visible light image and a second image feature corresponding to the thermal infrared image. Then, the feature encoding module may be used to calculate a first probability distribution of the first image feature and a second probability distribution corresponding to the second image feature based on the first image feature and the second image feature, and the first image feature may be encoded based on the first probability distribution and the second image feature may be encoded based on the second probability distribution. Finally, the feature reconstruction module may be used to perform reconstruction based on the encoded first image feature and the encoded second image feature to obtain a compressed target visible light image and a target thermal infrared image.

[0028] By using the feature encoding module to encode the first image feature and the second image feature at the same time, the problem of being unable to encode and decode visible light and thermal infrared images at the same time is effectively overcome. At the same time, the speed of image compression can be significantly improved, and it has good compression performance and real-time performance.

[0029] On the basis of the above-mentioned implementation manner, in some changed implementation manners, the feature extraction module may include a first feature extraction module and a second feature extraction module, and the feature extraction modules are used to perform feature extraction on the visible light image and the thermal infrared image respectively to obtain a first image feature corresponding to the visible light image, and a second image feature corresponding to the thermal infrared image. Specifically, it may include: using the first feature extraction module to perform feature extraction on the visible light image and the thermal infrared image respectively to obtain a first candidate feature corresponding to the visible light image, and a second candidate feature corresponding to the thermal infrared image, and then, using the second feature extraction module, based on the first candidate feature and the second candidate feature, respectively calculate the first image feature corresponding to the visible light image, and the second image feature corresponding to the thermal infrared image.

[0030] On the basis of the above-mentioned implementation manner, in some modified implementation manners, the first feature extraction module may include a first residual network and a first self-attention unit, and the first feature extraction module is used to perform feature extraction on the visible light image to obtain a first candidate feature corresponding to the visible light image. Specifically, it may include: converting the color format of the visible light image into a preset format, the preset format of the visible light image includes multiple first channels, and then using the first residual network corresponding to each first channel to perform feature extraction on the pixel values ​​corresponding to each first channel to obtain multiple first sub-candidate features, and downsampling the first sub-candidate features corresponding to each pre-specified first channel to obtain multiple second sub-candidate features, and using the first self-attention unit corresponding to each first channel to perform feature extraction on each first sub-candidate feature that has not been downsampled and each second sub-candidate feature that has been downsampled to obtain multiple third sub-candidate features, and finally, splicing the multiple third sub-candidate features in the channel dimension to obtain the first candidate feature.

[0031] It should be noted that the color format of the visible light image may include RGB color mode, Bitmap mode, Grayscale mode, Duotone mode, Index color mode, CMYK color mode, Lab color mode, Multichannel mode, HSB color mode and other formats, which are not specifically limited in the embodiments of the present application.

[0032] The preset format may refer to a YUV format, which is a color encoding method mainly used for data transmission in a compressed image process.

[0033] On the basis of the above implementation, in some modified implementations, the visible light image in the preset format includes a first channel Y, a first channel U and a first channel V, and then, the first residual network corresponding to each first channel is used to extract features of the pixel values ​​corresponding to each first channel to obtain a plurality of first sub-candidate features, and the first sub-candidate features corresponding to each pre-specified first channel are down-sampled to obtain a plurality of second sub-candidate features, and the first self-attention unit corresponding to each first channel is used to calculate each first sub-candidate feature that has not been down-sampled and each second sub-candidate feature that has been down-sampled by the following formula (1) to obtain a plurality of third sub-candidate features:

[0034]

[0035] Among them, when F0 represents the third sub-candidate feature corresponding to the first channel Y, F0 represents the first sub-candidate feature corresponding to the first channel Y, represents the weight matrix of the query vector Q0, key vector K0 and value vector V0 mapped to the first sub-candidate feature corresponding to the first channel Y; when F0 represents the third sub-candidate feature corresponding to the first channel U, F0 represents the first sub-candidate feature corresponding to the first channel U, represents the weight matrix of the query vector Q0, key vector K0 and value vector V0 mapped to the first sub-candidate feature corresponding to the first channel U; when F0 represents the third sub-candidate feature corresponding to the first channel V, F0 represents the first sub-candidate feature corresponding to the first channel V, The weight matrix representing the query vector Q0, key vector K0 and value vector V0 to which the first sub-candidate feature corresponding to the first channel V is mapped; d k0 Represents the dimension of the key vector K0.

[0036] On the basis of the above implementation manner, in some changed implementation manners, the first feature extraction module may also include a second residual network, a second self-attention unit and a first convolutional layer, and use the first feature extraction module to perform feature extraction on the thermal infrared image to obtain a second candidate feature corresponding to the thermal infrared image. Specifically, it may include: using the second residual network corresponding to the second channel of the thermal infrared image to perform feature extraction on the pixel value corresponding to the second channel to obtain a fourth sub-candidate feature, and then using the second self-attention unit corresponding to the second channel to perform feature extraction on the fourth sub-candidate feature to obtain a fifth sub-candidate feature, and finally using the first convolutional layer corresponding to the second channel to perform convolution processing on the fifth sub-candidate feature to obtain a second candidate feature.

[0037] On the basis of the above implementation, in some modified implementations, the fourth sub-candidate feature is extracted using the second self-attention unit corresponding to the second channel to obtain the fifth sub-candidate feature, which can also be calculated using the above formula (1), wherein when F0 represents the fifth sub-candidate feature corresponding to the second channel, F0 represents the fourth sub-candidate feature corresponding to the second channel, The weight matrix representing the query vector Q0, key vector K0, and value vector V0 to which the fourth sub-candidate feature corresponding to the second channel is mapped.

[0038] On the basis of the above implementation, in some modified implementations, the second feature extraction module may include a third residual network, a fourth residual network and a first cross-modal attention unit, a second cross-modal attention unit, and the second feature extraction module is used to respectively calculate the first image feature corresponding to the visible light image and the second image feature corresponding to the thermal infrared image based on the first candidate feature and the second candidate feature. Specifically, it may include: using the third residual network to downsample the first candidate feature to obtain a sixth sub-candidate feature, and using the fourth residual network to downsample the second candidate feature to obtain a seventh sub-candidate feature, and then using the first cross-modal attention unit to calculate the first image feature based on the sixth sub-candidate feature and the seventh sub-candidate feature by the following formula (2), and using the second cross-modal attention unit to calculate the second image feature based on the sixth sub-candidate feature and the seventh sub-candidate feature by the following formula (2):

[0039]

[0040] Among them, when F ′ When F1 represents the first image feature, F2 represents the sixth sub-candidate feature, and F1 represents the seventh sub-candidate feature. represents the weight matrix of the query vector Q1 to which the sixth sub-candidate feature is mapped, the key vector K1 to which the seventh sub-candidate feature is mapped, and the value vector V1; when F ′ When representing the second image feature, F1 represents the seventh sub-candidate feature, and F2 represents the sixth sub-candidate feature. The weight matrix representing the query vector Q1 to which the seventh sub-candidate feature is mapped, the key vector K1 to which the sixth sub-candidate feature is mapped, and the value vector V1; d k1 Represents the dimension of the key vector K1.

[0041] By using the first cross-modal attention unit to calculate the first image feature and using the second cross-modal attention unit to calculate the second image feature, the correlation between the first image feature corresponding to the visible light image and the second image feature corresponding to the thermal infrared image can be effectively reduced, the redundancy between the first image feature and the second image feature can be reduced, and the most informative part of each feature can be captured more accurately, thereby effectively reducing the amount of data and improving the compression ratio of the image while maintaining necessary information; in addition, the cross-modal attention mechanism of the first cross-modal attention unit and the second cross-modal attention unit helps to retain those features that have the greatest impact on visual perception, reduce blur or semantic distortion caused by compression, and thus improve the perceptual quality of the compressed image.

[0042] See also Figure 3 and 4As shown, on the basis of the above-mentioned implementation manner, in some modified implementation manners, the feature encoding module may include a prior information acquisition unit, a low-frequency information acquisition unit and a probability distribution calculation unit. The feature encoding module is used to calculate the first probability distribution of the first image feature and the second probability distribution corresponding to the second image feature based on the first image feature and the second image feature. Specifically, it may include: using the prior information acquisition unit to respectively acquire the first spatial domain prior information of the first image feature and the second spatial domain prior information of the second image feature, and then using the low-frequency information acquisition unit to respectively acquire the global low-frequency fusion information of the first image feature and the global low-frequency information of the second image feature, and finally using the probability distribution calculation unit to acquire the first probability distribution of the first image feature based on the first spatial domain prior information and the global low-frequency fusion information, and to acquire the second probability distribution of the second image feature based on the second spatial domain prior information and the global low-frequency information.

[0043] On the basis of the above implementation manner, in some modified implementation manners, the first image feature may include first sub-image features that are evenly divided into multiple first sub-image features according to a preset order, the global low-frequency fusion information includes global low-frequency fusion information corresponding to each first sub-image feature remaining except the first first sub-image feature, and the global low-frequency fusion information of the first image feature is obtained by using the low-frequency information acquisition unit, which may specifically include: obtaining all first sub-image features before the second sub-image feature, decoding and splicing them to obtain a first splicing feature, wherein the second sub-image feature is any one of the multiple first sub-image features, and the second sub-image feature is not the first first sub-image feature, and the first first sub-image feature is the first sub-image feature that is sorted in the first place according to the preset order, and using the low-frequency information acquisition unit, based on the second image feature and the first splicing feature, the global low-frequency fusion information corresponding to the second sub-image feature is obtained.

[0044] It should be noted that the preset order refers to the order in which the first image feature is segmented. The preset order may be from top to bottom and from left to right, or from bottom to top and from right to left, or from top to bottom and from right to left, or from bottom to top and from left to right, which is not specifically limited in the embodiments of the present application.

[0045] For example, assuming that the first image feature is evenly divided into five first sub-image features according to a preset order, it is necessary to obtain the global low-frequency fusion information corresponding to the second first sub-image feature, the third first sub-image feature, the fourth first sub-image feature, and the fifth first sub-image feature. When obtaining the global low-frequency fusion information corresponding to each first sub-image feature, the corresponding low-frequency information acquisition unit and the probability distribution calculation unit need to perform corresponding operations, which will not be repeated here.

[0046] On the basis of the above implementation manner, in some modified implementation manners, the low-frequency information acquisition unit includes a first low-frequency information extraction unit, a second convolutional layer, a second low-frequency information extraction unit and a low-frequency information fusion unit. The low-frequency information acquisition unit is used to obtain global low-frequency fusion information corresponding to the second sub-image feature based on the second image feature and the first splicing feature. Specifically, it may include: using the first low-frequency information extraction unit to obtain first global low-frequency information based on the first splicing feature, using the second convolutional layer and the second low-frequency information extraction unit to obtain second global low-frequency information based on the second image feature, and then using the low-frequency information fusion unit to fuse the first global low-frequency information and the second global low-frequency information to obtain global low-frequency fusion information corresponding to the second sub-image feature.

[0047] For example, assuming that the global low-frequency fusion information corresponding to the third first sub-image feature is to be obtained, the first first sub-image feature, the second first sub-image feature and the third first sub-image feature are decoded and spliced ​​to obtain a first splicing feature, and the first low-frequency information extraction unit is used to obtain the first global low-frequency information based on the first splicing feature. The second convolutional layer and the second low-frequency information extraction unit are used to obtain the second global low-frequency information based on the second image feature. Then, the low-frequency information fusion unit is used to fuse the first global low-frequency information and the second global low-frequency information to obtain the global low-frequency fusion information corresponding to the third first sub-image feature.

[0048] On the basis of the above implementation, in some modified implementations, the low-frequency information fusion unit may be used to calculate the global low-frequency fusion information corresponding to each first sub-image feature by using the following formula (3):

[0049]

[0050] Among them, G fusionInfo represents the global low-frequency fusion information corresponding to the N1th first sub-image feature, the G info represents the global low-frequency information corresponding to the N1th first sub-image feature, V ′ represents a temporary variable, A represents a temporary matrix, F3 represents the first splicing feature, F4 represents the second image feature, The weight matrix representing the query vector Q2 to which the first concatenated feature is mapped, the key vector K2 to which the second image feature is mapped, and the value vector V2, d k2 represents the dimension of the key vector K2, d a Represents the dimensions of the temporary matrix A.

[0051] On the basis of the above implementation manner, in some changed implementation manners, the low-frequency information acquisition unit may further include a third low-frequency information extraction unit. The low-frequency information acquisition unit is used to obtain the global low-frequency information of the second image feature, which may specifically include: the second image feature may include a plurality of third sub-image features equally divided into a plurality of third sub-image features according to a preset order, all third sub-image features before the N2th third sub-image feature are obtained for decoding and splicing to obtain a second splicing feature, wherein the N2th third sub-image feature is any one of the plurality of third sub-image features, and the N2th third sub-image feature is not the first third sub-image feature, and the first third sub-image feature is the third sub-image feature ranked first according to the preset order, and the third low-frequency information extraction unit is used to obtain the global low-frequency information of the second image feature based on the second image feature.

[0052] On the basis of the above implementation, in some modified implementations, the third low-frequency information extraction unit may be used to calculate the global low-frequency information of the second image feature by using the following formula (4):

[0053]

[0054] Among them, G info represents the global low-frequency information corresponding to the second image feature, V 1′ represents a temporary variable, A1 represents a temporary matrix, F5 represents the second image feature, The weight matrix representing the query vector Q3, key vector K3 and value vector V3 to which the second image feature is mapped, d k3 represents the dimension of the key vector K3, d a 1 represents the dimension of the temporary matrix A1.

[0055] On the basis of the above-mentioned implementation manner, in some changed implementation manner, a probability distribution calculation unit is used to obtain a first probability distribution of a first image feature based on first spatial domain prior information and global low-frequency fusion information. Specifically, the first probability distribution may include third probability distributions corresponding to multiple first sub-image features. The third probability distribution corresponding to the first first sub-image feature may be obtained by the probability distribution calculation unit based on the first spatial domain prior information, and then the remaining multiple first sub-image features may be obtained by the probability distribution calculation unit based on the first spatial domain prior information and the global low-frequency fusion information corresponding to each first sub-image feature.

[0056] Furthermore, a probability distribution calculation unit is used to obtain a second probability distribution of the second image feature based on the second spatial domain prior information and the global low-frequency information.

[0057] Furthermore, the first probability distribution and the second probability distribution can be expressed as the following formula (5):

[0058]

[0059] in, represents the first probability distribution, represents the first spatial domain prior information, N1 represents the number of first sub-image features, represents the second probability distribution, represents the second spatial domain prior information, and N2 represents the number of third sub-image features.

[0060] See also Figure 5 As shown, based on the above implementation, in some modified implementations, a feature reconstruction module is used to perform reconstruction processing based on the encoded first image features and second image features to obtain a compressed target visible light image and target thermal infrared image. Among them, the feature reconstruction module and the feature extraction module have similar processes, mainly for feature reconstruction of the encoded first image features and second image features, thereby converting the encoded first image features and second image features from the feature domain to the pixel domain to generate the final compressed image.

[0061] First, the encoded first image features are upsampled through the fifth residual network and vector mapped using the cross-modal attention mechanism of the third cross-modal attention unit. The encoded second image features are upsampled through the sixth residual network and vector mapped using the cross-modal attention mechanism of the fourth cross-modal attention unit. Then, the feature map of the thermal infrared modality is subjected to a 1x1 convolution operation using the seventh convolutional layer, and the feature map of the visible light modality is divided into feature maps of three channels, Y, U, and V, from the channel dimension. Finally, the feature maps of the four channels are respectively applied with the self-attention mechanism through the third self-attention unit and the fourth self-attention unit and upsampled through the seventh residual network and the eighth residual network, and finally the compressed visible light image and thermal infrared image are reconstructed. Among them, the calculation process of the self-attention mechanism and the cross-modal attention mechanism is the same as the calculation process of the self-attention mechanism and the cross-modal attention mechanism used in the feature extraction module in the above embodiment, and will not be repeated here.

[0062] On the basis of the above implementation, in some modified implementations, before the visible light image and the thermal infrared image before compression are input into the pre-trained image compression model, the image compression model can be trained. During the training process, the probability distribution of the visible light image to be trained and the probability distribution of the thermal infrared image to be trained are calculated using a probability distribution calculation unit, and then the bit rate of the visible light image to be trained and the bit rate of the thermal infrared image to be trained are estimated according to the probability distribution of the visible light image to be trained and the probability distribution of the thermal infrared image to be trained. Then, the loss value of the image compression model is calculated by the loss function shown in the following formula (6). If the loss value is less than or equal to the preset value, the image compression model is obtained:

[0063] L=R ir +R r +λ(D ir +D r ). (5)

[0064] Among them, R r Represents the bit rate of the visible light image to be trained, R ir represents the bit rate of the thermal infrared image to be trained, λ represents the hyperparameter, and D r represents the pixel-level mean square error between the visible light image to be trained and the compressed visible light image, D ir Represents the pixel-level mean square error between the thermal infrared image to be trained and the compressed thermal infrared image.

[0065] The embodiment of the present application inputs the visible light image and thermal infrared image before compression into a pre-trained image compression model, and uses the image compression model to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image, so that the visible light image and the thermal infrared image can be compressed jointly, which significantly improves the compression performance and real-time performance of the image, and significantly improves the joint compression efficiency of the image.

[0066] In order to verify the effectiveness of the image compression method in the embodiment of the present application for the joint compression of visible light and thermal infrared images, the image compression method in the embodiment of the present application is used to compare the rate-distortion performance of the best end-to-end single-modal image encoder Mlic++, the classic end-to-end image encoder Cheng2020, and the traditional image encoder BPG on the Kodak dataset with the joint compression method proposed by the present method on the LLVIP dataset. The delta rate (BD-Rate) is used as a scale to evaluate the rate-distortion performance of each image encoder. It is worth noting that in order to ensure a fair comparison, we redeployed and retrained other end-to-end compression frameworks on the LLVIP dataset. Compared with other single-modal encoding frameworks, our proposed framework has a significant improvement in BD-Rate performance. Specifically, our method is 14.6% and 26.8% better than MLIC++ and BPG, respectively. The results clearly show that our proposed method significantly outperforms other methods in compression performance.

[0067] In addition, in order to verify the effectiveness of the image compression method in the embodiment of the present application for jointly compressing visible light and thermal infrared images, the image compression method in the embodiment of the present application introduced an ablation experiment. The results showed that the feature encoding module and the low-frequency information acquisition unit in the image compression model in the embodiment of the present application were 19.34% and 9.17% lower in BD-Rate than the original model, respectively, which proves that these two modules effectively improve the overall encoding performance.

[0068] See also Figure 6 The present application also provides an image compression device, which is used to execute the image compression method described in the above embodiment, and the device includes:

[0069] An input module 201, used to input the visible light image and the thermal infrared image before compression into a pre-trained image compression model;

[0070] The compression module 202 is used to compress the visible light image and the thermal infrared image simultaneously using the image compression model to obtain a compressed target visible light image and a target thermal infrared image.

[0071] The image compression device provided in the embodiment of the present application and the image compression method provided in the above embodiment are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0072] The present application also provides an electronic device corresponding to the image compression method provided in the above embodiment. Figure 7 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 4 As shown, the electronic device 30 may include: a processor 300, a memory 301, a bus 302 and a communication interface 303, wherein the processor 300, the communication interface 303 and the memory 301 are connected via the bus 302; the memory 301 stores a computer program that can be run on the processor 300, and when the processor 300 runs the computer program, the image compression method provided in any of the aforementioned embodiments of the present application is executed.

[0073] The memory 301 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port 303 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0074] The bus 302 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs, and the processor 300 executes the programs after receiving the execution instruction. The image compression method disclosed in any implementation of the embodiment of the present application may be applied to the processor 300, or implemented by the processor 300.

[0075] The processor 300 may be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 300 or an instruction in the form of software. The above processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 301, and the processor 300 reads the information in the memory 301 and completes the steps of the above method in combination with its hardware.

[0076] The electronic device provided in the embodiment of the present application and the image compression method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0077] An embodiment of the present application also provides a computer-readable storage medium corresponding to the image compression method provided in the aforementioned embodiment, on which a computer program (ie, a program product) is stored. When the computer program is executed by a processor, the image compression method provided in any of the aforementioned embodiments is executed.

[0078] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0079] An embodiment of the present application also provides a computer program product corresponding to the image compression method provided in the aforementioned embodiment, including a computer program, which is executed by a processor to implement the image compression method provided in the aforementioned embodiments.

[0080] The computer-readable storage medium and computer program product provided in the above-mentioned embodiments of the present application are based on the same inventive concept as the image compression method provided in the embodiments of the present application, and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0081] It should be noted that:

[0082] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language. It should be understood that the content of the application described herein can be realized using various programming languages, and the description of the specific language above is to disclose the best mode of implementation of the application.

[0083] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0084] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0085] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0086] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0087] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the creation device of the virtual machine according to the embodiment of the present application. The application can also be implemented as a device or device program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0088] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0089] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An image compression method, characterized in that: The method comprises: Inputting the visible light image and the thermal infrared image before compression into a pre-trained image compression model; The image compression model is used to simultaneously compress the visible light image and the thermal infrared image to obtain a compressed target visible light image and a target thermal infrared image.

2. The image compression method according to claim 1, characterized in that: The image compression model includes a feature extraction module, a feature encoding module and a feature reconstruction module. The image compression model is used to compress the visible light image and the thermal infrared image at the same time to obtain a compressed target visible light image and a target thermal infrared image, including: Using the feature extraction module to extract features from the visible light image and the thermal infrared image respectively, to obtain a first image feature corresponding to the visible light image and a second image feature corresponding to the thermal infrared image; Using the feature encoding module, based on the first image feature and the second image feature, calculate a first probability distribution of the first image feature and a second probability distribution corresponding to the second image feature, and encode the first image feature based on the first probability distribution, and encode the second image feature based on the second probability distribution; The feature reconstruction module is used to perform reconstruction processing based on the encoded first image features and second image features to obtain a compressed target visible light image and a target thermal infrared image.

3. The image compression method according to claim 2, characterized in that: The feature extraction module includes a first feature extraction module and a second feature extraction module, and the feature extraction modules are used to extract features from the visible light image and the thermal infrared image respectively to obtain a first image feature corresponding to the visible light image and a second image feature corresponding to the thermal infrared image, including: Using the first feature extraction module to perform feature extraction on the visible light image and the thermal infrared image respectively, to obtain a first candidate feature corresponding to the visible light image and a second candidate feature corresponding to the thermal infrared image; The second feature extraction module is used to calculate first image features corresponding to the visible light image and second image features corresponding to the thermal infrared image based on the first candidate features and the second candidate features.

4. The image compression method according to claim 3, characterized in that: The first feature extraction module includes a first residual network and a first self-attention unit, and the first feature extraction module is used to extract features from the visible light image to obtain a first candidate feature corresponding to the visible light image, including: Converting the color format of the visible light image into a preset format, wherein the visible light image in the preset format includes a plurality of first channels; Using the first residual network corresponding to each of the first channels, respectively, to extract features from the pixel values ​​corresponding to each of the first channels, to obtain a plurality of first sub-candidate features, and respectively, to downsample the first sub-candidate features corresponding to each of the first channels, to obtain a plurality of second sub-candidate features; Using the first self-attention units corresponding to the first channels, respectively, to extract features from the first sub-candidate features that have not been downsampled and the second sub-candidate features that have been downsampled, to obtain a plurality of third sub-candidate features; The plurality of third sub-candidate features are concatenated in the channel dimension to obtain the first candidate feature.

5. The image compression method according to claim 3, characterized in that: The first feature extraction module includes a second residual network, a second self-attention unit and a first convolutional layer. The first feature extraction module is used to extract features from the thermal infrared image to obtain a second candidate feature corresponding to the thermal infrared image, including: Using a second residual network corresponding to a second channel of the thermal infrared image, extracting features from pixel values ​​corresponding to the second channel to obtain a fourth sub-candidate feature; Using the second self-attention unit corresponding to the second channel to perform feature extraction on the fourth sub-candidate feature to obtain a fifth sub-candidate feature; The fifth sub-candidate feature is convolved using the first convolution layer corresponding to the second channel to obtain the second candidate feature.

6. The image compression method according to claim 3, characterized in that: The second feature extraction module includes a third residual network, a fourth residual network, a first cross-modal attention unit, and a second cross-modal attention unit. The second feature extraction module is used to calculate the first image feature corresponding to the visible light image and the second image feature corresponding to the thermal infrared image based on the first candidate feature and the second candidate feature, respectively, including: Downsampling the first candidate feature using the third residual network to obtain a sixth sub-candidate feature, and downsampling the second candidate feature using the fourth residual network to obtain a seventh sub-candidate feature; Using the first cross-modal attention unit, based on the sixth sub-candidate feature and the seventh sub-candidate feature, the first image feature is calculated by the following formula, and using the second cross-modal attention unit, based on the sixth sub-candidate feature and the seventh sub-candidate feature, the second image feature is calculated by the following formula: Among them, when F ′ When representing the first image feature, F1 represents the sixth sub-candidate feature, and F2 represents the seventh sub-candidate feature. represents the weight matrix of the query vector Q1 to which the sixth sub-candidate feature is mapped, the key vector K1 and the value vector V1 to which the seventh sub-candidate feature is mapped; when F ′ When representing the second image feature, F1 represents the seventh sub-candidate feature, and F2 represents the sixth sub-candidate feature. A weight matrix representing the query vector Q1 to which the seventh sub-candidate feature is mapped, the key vector K1 and the value vector V1 to which the sixth sub-candidate feature is mapped; d k1 Represents the dimension of the key vector K1.

7. The image compression method according to claim 2, characterized in that: The feature encoding module includes a priori information acquisition unit, a low-frequency information acquisition unit and a probability distribution calculation unit. The feature encoding module is used to calculate a first probability distribution of the first image feature and a second probability distribution corresponding to the second image feature based on the first image feature and the second image feature, including: Using the prior information acquisition unit to respectively acquire first spatial domain prior information of the first image feature and second spatial domain prior information of the second image feature; Using the low-frequency information acquisition unit to respectively acquire the global low-frequency fusion information of the first image feature and the global low-frequency information of the second image feature; The probability distribution calculation unit is used to obtain the first probability distribution of the first image feature based on the first spatial domain prior information and the global low-frequency fusion information, and the second probability distribution of the second image feature is obtained based on the second spatial domain prior information and the global low-frequency information.

8. The image compression method according to claim 7, characterized in that: The first image feature includes a plurality of first sub-image features that are evenly divided into a plurality of first sub-image features according to a preset order, the global low-frequency fusion information includes global low-frequency fusion information corresponding to each of the first sub-image features remaining except the first first sub-image feature, and the step of acquiring the global low-frequency fusion information of the first image feature by using the low-frequency information acquisition unit includes: Obtaining all first sub-image features before a second sub-image feature for decoding and splicing to obtain a first splicing feature, wherein the second sub-image feature is any one of a plurality of the first sub-image features, and the second sub-image feature is not the first first sub-image feature, and the first first sub-image feature is the first sub-image feature that is sorted first according to the preset order; The low-frequency information acquisition unit is used to acquire global low-frequency fusion information corresponding to the second sub-image feature based on the second image feature and the first splicing feature.

9. The image compression method according to claim 8, characterized in that: The low-frequency information acquisition unit includes a first low-frequency information extraction unit, a second convolution layer, a second low-frequency information extraction unit and a low-frequency information fusion unit. The low-frequency information acquisition unit is used to acquire global low-frequency fusion information corresponding to the second sub-image feature based on the second image feature and the first splicing feature, including: Using the first low-frequency information extraction unit, based on the first splicing feature, obtain first global low-frequency information; Using the second convolutional layer and the second low-frequency information extraction unit, based on the second image feature, obtain second global low-frequency information; The first global low-frequency information and the second global low-frequency information are fused by using the low-frequency information fusion unit to obtain global low-frequency fusion information corresponding to the second sub-image feature.

10. An image compression device, characterized in that: include: An input module, used for inputting the visible light image and the thermal infrared image before compression into a pre-trained image compression model; The compression module is used to compress the visible light image and the thermal infrared image simultaneously using the image compression model to obtain a compressed target visible light image and a target thermal infrared image.

Images