A facial image compression method and system

Abstract

The embodiment of the present application discloses an image compression method and system, the method comprising: inputting a style encoder and a content encoder from an original face image to extract style features and structural features; performing probability estimation and entropy coding respectively to obtain a style coding bit stream corresponding to the style features and a structural coding bit stream corresponding to the structural features, inputting the bit streams into a decoder and a multi-task analysis network; the decoder reconstructs the image of the style coding bit stream and the structural coding bit stream and outputs a reconstructed image; the multi-task analysis network performs semantic understanding analysis on the style coding bit stream and the structural coding bit stream and outputs the semantic information of the image. Under the condition of extremely high compression efficiency, the high subjective visual evaluation quality of the reconstructed image is maintained, and decoding time and resource overhead are saved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of digital signal processing technology, and specifically to a facial image compression method and system. Background Art

[0002] Image / video compression methods based on neural networks have developed rapidly in recent years, and the quality of compressed and reconstructed images has exceeded the new generation video coding standard VVC in objective indicators such as PSNR and MS-SSIM. The compression framework based on the generative model can greatly improve the compression ratio without affecting the relevant evaluation indicators that directly reflect the viewing effect of the human eye.

[0003] Currently, in various studies, end-to-end image coding based on neural networks faces two major problems: first, the representation mechanism for the input original image signal is limited, and there is a lack of support for the currently widely used computer vision processing tasks; second, the resources at the signal receiving end are limited and insufficient to support the neural network model with a large number of parameters. Summary of the invention

[0004] To this end, the embodiments of the present application provide an image compression method and system, which maintain high subjective visual evaluation quality of the reconstructed image with extremely high compression efficiency and save decoding time and resource overhead.

[0005] In order to achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a face image compression method is provided, the method comprising:

[0007] Input the style encoder and content encoder from the original face image to extract style features and structural features;

[0008] Probability estimation and entropy coding are performed respectively to obtain the style coding bit stream corresponding to the style features and the structure coding bit stream corresponding to the structure features, which are input into the decoder and the multi-task analysis network;

[0009] The decoder reconstructs the image of the style coding bit stream and the structure coding bit stream, and outputs a reconstructed image; the multi-task analysis network performs semantic understanding analysis on the style coding bit stream and the structure coding bit stream, and outputs semantic information of the image.

[0010] Optionally, probability estimation and entropy coding are performed respectively to obtain a style coding bit stream corresponding to the style feature and a structure coding bit stream corresponding to the structure feature, including:

[0011] Quantifying the style features and the structural features respectively to obtain quantified style features and structural features;

[0012] The quantized style features and structural features are entropy encoded according to the probability estimation results calculated by the probability estimation model to obtain a style encoding bit stream corresponding to the style features and a structural encoding bit stream corresponding to the structural features.

[0013] Optionally, the decoder reconstructs the image of the style coded bit stream and the structure coded bit stream, and outputs a reconstructed image, including:

[0014] The style encoding bitstream and the structure encoding bitstream are fused through the fusion module in the decoder and processed through the multi-layer perception MLP to learn the mean and variance of the convolutional layer in the residual block;

[0015] The generator in the decoder performs an image compression task on the fused coded bit stream to obtain a compressed reconstructed image;

[0016] The compressed reconstructed image is discriminated by the discriminator to obtain a loss optimization function; and the generator is trained according to the loss optimization function.

[0017] Optionally, the loss optimization function is according to the following formula:

[0018]

[0019]

[0020] Among them, D is the discriminator, E is the content encoder and style encoder, G is the generator, P is the probability estimation model, x is the original face image, To reconstruct the image, are the quantified style features and structural features, p is the probability estimation result, and λ and β are hyperparameters.

[0021] Optionally, the multi-task analysis network performs semantic understanding analysis on the style coding bit stream and the structure coding bit stream, and outputs semantic information of the image, including:

[0022] The style coding bit stream and the structure coding bit stream are input into the multi-task analysis network, the coding bit streams are fused through the fusion module, and then the multi-task analysis network is trained according to the multi-task analysis loss function to obtain the corresponding task results as the output of the semantic information of the image.

[0023] Optionally, the multi-task analysis loss function L multi Calculate according to the following formula:

[0024] L multi =λ cls l cls +λ seg l seg

[0025] Among them, l cls , l seg They are the loss functions of the classification task and the segmentation task, λ cls , seg is the corresponding weight hyperparameter.

[0026] Optionally, the method further includes: training parameters in the multi-task analysis model by optimizing the multi-task analysis loss function to obtain a global optimal solution; wherein the total loss function applied in the training of the multi-task analysis model is in accordance with the following formula:

[0027] L=L EGP +L D +γL multi

[0028] Among them, γ is a hyperparameter.

[0029] According to a second aspect of an embodiment of the present application, a face image compression system is provided, the system comprising:

[0030] A feature extraction module, used to input a style encoder and a content encoder from an original face image to extract style features and structural features;

[0031] The encoding module is used to perform probability estimation and entropy encoding respectively to obtain a style encoding bit stream corresponding to the style feature and a structure encoding bit stream corresponding to the structure feature, which are input into a decoder and a multi-task analysis network;

[0032] A compression decoding module, used for a decoder to reconstruct an image of the style coded bit stream and the structure coded bit stream, and output a reconstructed image;

[0033] The multi-task analysis module is used for the multi-task analysis network to perform semantic understanding analysis on the style coding bit stream and the structure coding bit stream, and output semantic information of the image.

[0034] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0035] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described in the first aspect above.

[0036] In summary, the embodiment of the present application provides an image compression method and system, which extracts style features and structural features by inputting a style encoder and a content encoder from the original face image; performing probability estimation and entropy coding respectively to obtain a style coding bit stream corresponding to the style features and a structural coding bit stream corresponding to the structural features, and inputting them into a decoder and a multi-task analysis network; the decoder reconstructs the image of the style coding bit stream and the structural coding bit stream, and outputs a reconstructed image; the multi-task analysis network performs semantic understanding analysis on the style coding bit stream and the structural coding bit stream, and outputs the semantic information of the image. With extremely high compression efficiency, the high subjective visual evaluation quality of the reconstructed image is maintained, and decoding time and resource overhead are saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0038] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0039] Figure 1 A schematic diagram of a facial image compression method provided in an embodiment of the present application;

[0040] Figure 2 A flowchart of the technical solution provided for the embodiment of the present application;

[0041] Figure 3 A diagram of a multi-task analysis network structure provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram showing the quality index of a compressed reconstructed image provided in an embodiment of the present application;

[0043] Figure 5 A face image compression system provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown;

[0045] Figure 7 A schematic diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The image compression method provided in the embodiment of the present application is ahead of the existing technology in multiple evaluation indicators simulating human visual perception under the premise of the same bit rate; at the same time, the compressed data is used as the input of the downstream visual analysis task, with very little loss of analysis accuracy compared to the original image, but brings about a large decoding end storage resource overhead and network transmission bandwidth.

[0048] Figure 1 The image compression method provided by the embodiment of the present application is shown, and the method comprises the following steps:

[0049] Step 101: Input a style encoder and a content encoder from the original face image to extract style features and structural features;

[0050] Step 102: Probability estimation and entropy coding are performed respectively to obtain a style coding bit stream corresponding to the style feature and a structure coding bit stream corresponding to the structure feature, and input them into a decoder and a multi-task analysis network;

[0051] Step 103: The decoder reconstructs the image of the style coding bit stream and the structure coding bit stream, and outputs a reconstructed image; the multi-task analysis network performs semantic understanding analysis on the style coding bit stream and the structure coding bit stream, and outputs semantic information of the image.

[0052] In a possible implementation, in step 102, probability estimation and entropy coding are performed respectively to obtain a style coding bit stream corresponding to the style feature and a structure coding bit stream corresponding to the structure feature, including:

[0053] The style features and the structural features are quantized respectively to obtain quantized style features and structural features; the quantized style features and structural features are entropy encoded respectively according to the probability estimation results calculated by the probability estimation model to obtain a style encoding bit stream corresponding to the style features and a structural encoding bit stream corresponding to the structural features.

[0054] In a possible implementation, in step 103, the decoder reconstructs the image of the style coding bit stream and the structure coding bit stream and outputs a reconstructed image, including:

[0055] The style coding bit stream and the structure coding bit stream are fused through a fusion module in the decoder, and processed through a multi-layer perception MLP to learn the mean and variance of the convolution layer in the residual block; the generator in the decoder performs an image compression task on the fused coding bit stream to obtain a compressed reconstructed image; the discriminator discriminates the compressed reconstructed image to obtain a loss optimization function; and the generator is trained according to the loss optimization function.

[0056] In a possible implementation, the loss optimization function is in accordance with the following formulas (1) and (2):

[0057]

[0058]

[0059] Among them, D is the discriminator, E is the content encoder and style encoder, G is the generator, P is the probability estimation model, x is the original face image, To reconstruct the image, are the quantified style features and structural features, p is the probability estimation result, and λ and β are hyperparameters.

[0060] In a possible implementation, in step 103, the multi-task analysis network performs semantic understanding analysis on the style coding bit stream and the structure coding bit stream, and outputs semantic information of the image, including:

[0061] The style coding bit stream and the structure coding bit stream are input into the multi-task analysis network, the coding bit streams are fused through the fusion module, and then the multi-task analysis network is trained according to the multi-task analysis loss function to obtain the corresponding task results as the output of the semantic information of the image.

[0062] In the compression framework, the intermediate compression results are used directly as input for multiple analysis tasks without decoding to obtain the semantic information of the original image signal. The multi-task here refers to multiple visual semantics-related tasks, such as recognition, detection, segmentation, etc.

[0063] In a possible implementation, the multi-task analysis loss function L multi Calculate according to the following formula (3):

[0064] L multi =λ cls l cls +λ seg l segFormula (3)

[0065] Among them, l cls , l seg They are the loss functions of the classification task and the segmentation task, λ cls , seg is the corresponding weight hyperparameter.

[0066] In a possible implementation, the method further includes: training parameters in the multi-task analysis model by optimizing the multi-task analysis loss function to obtain a global optimal solution; wherein the total loss function applied in the training of the multi-task analysis model is in accordance with the following formula (4):

[0067] L=L EGP +L D +γL multi Formula (4)

[0068] Among them, γ is a hyperparameter.

[0069] The face image compression method provided in the embodiment of the present application will apply the current idea of ​​constructing a generative model for hierarchical representation of image signals to map the original input image into style features and structural features, thereby performing further quantization and entropy coding on the corresponding feature expression distribution. In addition, the embodiment of the present application will directly use compressed data in the compression domain as input for subsequent multiple visual tasks. Since compressed data is an efficient and compact form of data expression, the embodiment of the present application proposes a multi-task analysis network model to obtain the semantic information of the original image at a very low computational cost without decompression. At the same time, the rate-distortion loss function and the machine analysis target loss function are jointly optimized to obtain a common solution for the image compression task and multiple machine vision analysis tasks.

[0070] Figure 2 A model architecture diagram applicable to the face image compression method for multi-visual analysis tasks proposed in an embodiment of the present application is shown, which mainly includes a compression model and a multi-task analysis model.

[0071] The compression model mainly consists of four main parts: encoder, generator, discriminator and probability estimation model. The encoder includes content encoder and style encoder.

[0072] Given an original image x, the encoder first encodes it into y = E(x) and then quantizes it Where Q is the quantization function. After that, the probability estimation result given by the probability estimation model will be Entropy coding is used to encode losslessly into a bit stream. At the decoder end, there is in To reconstruct the image.

[0073] The original image to be compressed is mapped to the visual semantic feature domain, and then decomposed into style features and structural features. The decomposed style features and structural features are fitted with probability distribution using independent probability estimation methods, and the entropy value of the fitted probability is used as the bit rate value obtained by actual encoding.

[0074] The original image signal is input and decomposed into content features and style features at the semantic feature level. Two independent encoders are used to encode the input image x into content representation and style representation, which are E1 and E2 respectively. The content feature and style feature are y1=E1(x) and y2=E2(x) respectively. Then quantization is performed using the quantization function Q to obtain and

[0075] Since the decoupled features have independent data distributions, a probability estimation model P is set for each layer, namely p1(y1|z1) and p2(y2|z2). According to the probability estimation results p1(y1|z1) and p2(y2|z2) given by the probability estimation model, and The entropy coding method is used to losslessly encode the bit stream, and the entropy value of the fitted probability is used as the bit rate value obtained by actual coding.

[0076] Entropy coding is used to further compress features. The input of entropy coding requires the probability distribution of elements in the feature, so a probability estimation model is used to estimate the probability distribution of each element that appears. Entropy coding methods include but are not limited to Huffman coding, arithmetic coding, and context-based binary coding.

[0077] The coded bitstream here is referred to as bitstream. The binary file obtained by compressing the feature through the entropy encoder. Figure 2 The bitstream 1 and bitstream 2 in are the entropy coding results of content features and style features respectively.

[0078] The embodiment of the present application also proposes an encoder based on semantic layering, and the decoder part mainly includes a generator and a discriminator. The embodiment of the present application designs a fusion module at the decoding end. The fusion module is based on the Adaptive Instance Normalization (AdaIN) residual block, in which the content features are directly used as the input of the AdaIN module; the style features are processed by multi-layer perceptron (MLP) as the mean and variance of the residual network convolution layer in the AdaIN module to learn the mean and variance of the convolution layer in the residual block.

[0079] On the decoder side, there is in To reconstruct the image.

[0080] The embodiment of the present application optimizes the compression model according to the rate-distortion theory. There are two reference indicators in the distortion loss metric: the mean average error (MAE) loss at the pixel level d MAE And the SSIM loss of the overall structure evaluation d SSIM At the same time, considering the subjective perceived quality, the perceptual distortion loss d p , by extracting high-order features from the pre-trained convolutional neural network VGG16 to simulate the human eye perception characteristics.

[0081] The method for calculating the total distortion loss using the compression model proposed in the embodiment of the present application is in accordance with the following formula (5):

[0082] d=λ MAE d MAE +λ SSIM d SSIM +λ p d p Formula (5)

[0083] Among them, λ MAE , SSIM and λ p is a hyperparameter. MAE is the pixel-level mean average error (MAE) loss, d SSIM is the SSIM loss of the overall structure evaluation, d p is the perceptual distortion loss.

[0084] Based on the definition of this loss function, the modules corresponding to the embodiments of the present application (referring to the modules corresponding to the L subscript) Figure 2 The loss optimization function of each module, E refers to the content and style encoder, G refers to the generator, P refers to the probability estimation model, and D refers to the discriminator) can be defined as formula (1) and formula (2).

[0085] In the embodiment of the present application, the expected code stream transmission rate is changed by changing the number of feature channels when training the compression model, so as to more effectively obtain an extremely high compression ratio model.

[0086] During the training process, uniform noise is added to avoid the non-differentiable gradient of the quantization operation during back propagation.

[0087] The probability distribution fitting methods used during training include but are not limited to Gaussian models and mixed Gaussian models.

[0088] Figure 3The schematic diagram of the multi-task analysis network structure provided by the embodiment of the present application is shown. In terms of the multi-task analysis network model, the compressed domain multi-task analysis network refers to the classification task and the semantic segmentation task, and adopts the corresponding network structure design and the corresponding loss function. ASPP refers to Atrous Spatial Pyramid Pooling.

[0089] The style features and structural features are input into the multi-task analysis network, first through the fusion module, and then the task network branches are designed according to the characteristics of different tasks, and the corresponding results are output according to different task settings.

[0090] In the embodiments of the present application, classification tasks and segmentation tasks that have been widely studied are taken as examples because they are the most representative tasks in visual analysis.

[0091] The balance between tasks is controlled by setting hyperparameters, so the multi-task analysis loss L multi It can be expressed as formula (6):

[0092] L multi =λ cls l cls +λ seg l seg Formula (6)

[0093] Among them l cls , l seg They are the loss functions of the classification task and the segmentation task, λ cls , seg are their corresponding weight hyperparameters.

[0094] To further explore the relationship between compression and visual analysis, two analysis network training methods are verified: separate training and joint training.

[0095] The training process of separate training involves a small number of parameters and is easier to train. In the separate training setting, the embodiment of the present application fixes the compression model and only trains the multi-task analysis model.

[0096] The training process of joint training is to directly train the balanced relationship among compression, reconstruction and analysis, which makes it easier to find the global optimal point and achieve better analysis results.

[0097] For the joint training method, the compression model and the multi-task analysis model are jointly optimized, and the total loss function is defined as formula (4), where the hyperparameter γ is used to balance the weight between the compression task and the visual analysis task.

[0098] The embodiment of the present application jointly optimizes the rate-distortion loss function and the machine analysis target loss function to obtain a common solution for the image compression task and various machine vision analysis tasks.

[0099] The effects of the reconstructed image corresponding to the compression model in the embodiment of the present application on the four perception-based image evaluation indicators are as follows: Figure 4 As shown, it can be seen that various image quality indicators significantly exceed the existing traditional encoding methods and end-to-end compression methods based on deep learning.

[0100] The face image compression method provided in the embodiment of the present application decouples the original image signal into style features and structural features in the visual feature domain, applies independent probability estimation models to perform distribution fitting on the features respectively, and then obtains the encoded bit stream through an entropy encoder; in order to solve the problem of directly analyzing compressed data to obtain semantic information, a multi-task analysis model is proposed.

[0101] The method proposed in the embodiment of the present application can maintain high subjective visual evaluation quality of the reconstructed image under extremely high compression efficiency, and can receive the code stream without decoding at the decoding end, and use a multi-task analysis network acting on compressed data to obtain the semantic information of the original image, saving decoding time and resource overhead.

[0102] In summary, the embodiment of the present application provides an image compression method, which extracts style features and structural features by inputting a style encoder and a content encoder from an original face image; performing probability estimation and entropy coding respectively to obtain a style coding bit stream corresponding to the style features and a structural coding bit stream corresponding to the structural features, and inputting them into a decoder and a multi-task analysis network; the decoder reconstructs the image of the style coding bit stream and the structural coding bit stream, and outputs a reconstructed image; the multi-task analysis network performs semantic understanding analysis on the style coding bit stream and the structural coding bit stream, and outputs the semantic information of the image. With extremely high compression efficiency, the high subjective visual evaluation quality of the reconstructed image is maintained, and decoding time and resource overhead are saved.

[0103] Based on the same technical concept, the embodiment of the present application also provides a face image compression system. Figure 5 As shown, the system comprises:

[0104] The feature extraction module 501 is used to input the style encoder and the content encoder from the original face image to extract the style features and the structural features;

[0105] The coding module 502 is used to perform probability estimation and entropy coding respectively to obtain a style coding bit stream corresponding to the style feature and a structure coding bit stream corresponding to the structure feature, and input them into a decoder and a multi-task analysis network;

[0106] A compression decoding module 503, used for a decoder to reconstruct an image of the style coded bit stream and the structure coded bit stream, and output a reconstructed image;

[0107] The multi-task analysis module 504 is used for the multi-task analysis network to perform semantic understanding analysis on the style coding bit stream and the structure coding bit stream, and output semantic information of the image.

[0108] The present application also provides an electronic device corresponding to the method provided in the above embodiment. Figure 6 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and the processor 200 executes the method provided by any of the aforementioned embodiments of the present application when running the computer program.

[0109] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0110] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving execution instructions. The method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.

[0111] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0112] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0113] The present application also provides a computer-readable storage medium corresponding to the method provided in the above embodiment. Figure 7 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided in any of the aforementioned embodiments is executed.

[0114] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0115] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0116] It should be noted that:

[0117] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language. It should be understood that the content of the application described herein can be realized using various programming languages, and the description of the specific language above is to disclose the best mode of implementation of the application.

[0118] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0119] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0120] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0121] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0122] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the creation device of the virtual machine according to the embodiment of the present application. The application can also be implemented as a device or device program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0123] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0124] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A facial image compression method, characterized in that: The method comprises: Input the original face image into the style encoder and content encoder to extract style features and structural features; Probability estimation and entropy coding are performed respectively to obtain the style coding bit stream corresponding to the style features and the structure coding bit stream corresponding to the structure features, which are input into the decoder and the multi-task analysis network; The decoder reconstructs the image of the style coded bit stream and the structure coded bit stream, and outputs a reconstructed image; The style coding bitstream and the structure coding bitstream are input into the multi-task analysis network, the coding bitstreams are fused through the fusion module, and then the multi-task analysis network is trained according to the multi-task analysis loss function to obtain the corresponding task results as the output of the semantic information of the image; the multi-task analysis loss function L multi Calculate according to the following formula: L multi =λ cls l cls +λ seg l seg Among them, l cls , l seg They are the loss functions of the classification task and the segmentation task, λ cls , seg is the corresponding weight hyperparameter.

2. The method according to claim 1, characterized in that The performing probability estimation and entropy coding respectively to obtain a style coding bit stream corresponding to the style feature and a structure coding bit stream corresponding to the structure feature includes: Quantifying the style features and the structural features respectively to obtain quantified style features and structural features; The quantized style features and structural features are entropy encoded according to the probability estimation results calculated by the probability estimation model to obtain a style encoding bit stream corresponding to the style features and a structural encoding bit stream corresponding to the structural features.

3. The method according to claim 1, characterized in that The decoder reconstructs an image of the style coded bit stream and the structure coded bit stream, including: The style encoding bitstream and the structure encoding bitstream are fused through the fusion module in the decoder and processed through the multi-layer perception MLP to learn the mean and variance of the convolutional layer in the residual block; The generator in the decoder performs an image compression task on the fused coded bit stream to obtain a compressed reconstructed image; The compressed reconstructed image is discriminated by a discriminator to obtain a loss optimization function; and the generator is trained according to the loss optimization function.

4. The method according to claim 3, characterized in that The loss optimization function is as follows: Among them, D is the discriminator, E is the content encoder and style encoder, G is the generator, P is the probability estimation model, x is the original face image, To reconstruct the image, are the quantized style features and structural features, p is the probability estimation result, λ and β are hyperparameters, and L EGP is the training loss function of the encoder, generator and probability estimation model, L D is the training loss function of the discriminator, E x～px It is the conditional probability estimation function under the condition that x follows the px distribution, and d is the distortion measure function.

5. The method according to any one of claims 1 to 4, characterized in that: The method further includes: training the parameters in the multi-task analysis model by optimizing the multi-task analysis loss function to obtain a global optimal solution; wherein the total loss function used in the training of the multi-task analysis model is in accordance with the following formula: L=L EGP +L D +γL multi Among them, γ is a hyperparameter, L D The training loss function of the discriminator.

6. A face image compression system, characterized in that: The system comprises: A feature extraction module, used for inputting the original face image into the style encoder and the content encoder to extract style features and structural features; The encoding module is used to perform probability estimation and entropy encoding respectively to obtain a style encoding bit stream corresponding to the style feature and a structure encoding bit stream corresponding to the structure feature, which are input into a decoder and a multi-task analysis network; A compression decoding module, used for a decoder to reconstruct an image of the style coded bit stream and the structure coded bit stream, and output a reconstructed image; The multi-task analysis module is used to input the style coding bit stream and the structure coding bit stream into the multi-task analysis network, fuse the coding bit streams through the fusion module, and then train the multi-task analysis network according to the multi-task analysis loss function to obtain the corresponding task results as the output of the semantic information of the image; the multi-task analysis loss function L multi Calculate according to the following formula: L multi =λ cls l cls +λ seg l seg Among them, l cls , l seg They are the loss functions of the classification task and the segmentation task, λ cls , seg is the corresponding weight hyperparameter.

7. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1-5.

Images