Image compression method, device, equipment and storage medium based on parallel adapter
By storing the residuals between INRs in parallel adapters in the implicit neural representation network, the problem that the INR network cannot efficiently store associated images is solved, the flexibility and efficiency of image compression is improved, and storage and computing resources are saved Effect.
Patent Information
- Application Number
- CN202411142878.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-20
AI Technical Summary
Implicit neural representation (INR) network cannot efficiently store associated images, resulting in inflexible compression methods, long encoding time, high computing resources consumption, and the existing INR network corresponds one by one with image and video storage, which lacks flexibility.
Multiple parallel adapters are connected in the network structure represented by implicit neural, and the residuals between different INRs are stored using parallel adapters. The model is micro-adjusted to the storage space to improve the flexibility of image compression.
It effectively alleviates the problem of INR redundant storage, improves the flexibility and efficiency of image compression, reduces the demand for storage and computing resources, and achieves efficient image compression.
Smart Images

Figure CN118657842B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a parallel adapter-based image compression method, device, equipment and storage medium. Background Art
[0002] Our visual world can be represented continuously, which is a basic principle in fields such as computer vision. Since the digitization of television signals was proposed in 1948, people have begun to study image compression coding. The first "Image Coding Conference" held in the United States in 1969 marked the birth of image coding as an independent discipline. Traditional methods of encoding input signals into representations usually follow an explicit paradigm, where the input space is discretized or divided into individual pixels. However, in recent years, another representation method, Implicit Neural Representation (INR), has become popular due to its efficient use of continuous storage. Unlike traditional discrete representation methods, INR uses a neural network to map pixel coordinates to RGB values, where data can be stored in a continuous form. However, due to the uninterpretability of backpropagation, it is not possible to efficiently store associated images through INR. Summary of the invention
[0003] The main purpose of the present application is to provide an image compression method, device, equipment and storage medium based on a parallel adapter, aiming to solve the technical problems in the prior art that the INR network corresponds one-to-one with the image, the associated images cannot be stored efficiently, and the compression method is inflexible.
[0004] To achieve the above object, the present application proposes an image compression method based on a parallel adapter, and the image compression method based on a parallel adapter includes:
[0005] Acquire an original image, a plurality of edited images corresponding to the original image, and an implicit neural representation network of the original image;
[0006] Performing network parallel connection based on the implicit neural representation network of the original image and a plurality of parallel adapters to generate a compression network to be trained;
[0007] Training the compression network to be trained according to the multiple edited images corresponding to the original image to obtain a target image compression network;
[0008] The image compression results of the original image and each edited image are obtained according to the target image compression network.
[0009] In one embodiment, the step of performing network parallel connection based on the implicit neural representation network of the original image and a plurality of parallel adapters to generate a compression network to be trained comprises:
[0010] Determining an initial parameter rank and an initial scaling factor according to image information of a plurality of edited images corresponding to the original image;
[0011] Defining an adapter structure according to the initial parameter rank and the initial scaling factor to obtain a plurality of parallel adapters;
[0012] Each parallel adapter is connected in parallel with each perception layer in the implicit neural representation network of the original image to generate a compression network to be trained.
[0013] In one embodiment, the step of training the compression network to be trained according to the multiple edited images corresponding to the original image to obtain the target image compression network includes:
[0014] Get the regularization parameter;
[0015] Constructing a target loss function according to the regularization parameter and the target regularization method;
[0016] The compression network to be trained is trained using a plurality of edited images corresponding to the original image and the target loss function to obtain a target image compression network.
[0017] In one embodiment, the step of training the compression network to be trained using the multiple edited images corresponding to the original image and the target loss function to obtain the target image compression network includes:
[0018] Determining an adapter to be trained in the compression network to be trained;
[0019] Freezing weights of the network structure except the adapter to be trained in the compression network to be trained;
[0020] When the weight freezing operation is completed, updating the parameters of the adapter to be trained by back propagation, the multiple edited images corresponding to the original image, and the target loss function to obtain a target adapter;
[0021] A target image compression network is obtained according to the target adapter.
[0022] In one embodiment, the step of training the compression network to be trained using the multiple edited images corresponding to the original image and the target loss function to obtain the target image compression network includes:
[0023] Constructing a network training set according to the original image and a plurality of edited images corresponding to the original image;
[0024] Training the compressed network to be trained by using a hypernetwork method, the target loss function and the network training set to obtain an initial training network;
[0025] Performing performance measurement on the initial training network to obtain a performance measurement result of the initial training network;
[0026] When the performance measurement result is a performance-reaching result, a target image compression network is obtained according to the initial training network.
[0027] In one embodiment, the step of performing performance measurement on the initial training network to obtain a performance measurement result of the initial training network includes:
[0028] Constructing a network test set according to a plurality of variant images corresponding to the original image;
[0029] Testing the initial training network according to the network test set to obtain a peak signal-to-noise ratio and a storage index of each test image;
[0030] The performance measurement result of the initial training network is obtained according to the peak signal-to-noise ratio and storage index of each test image.
[0031] In one embodiment, after the step of obtaining the image compression results of the original image and each edited image according to the target image compression network, the method further includes:
[0032] Performing coordinate conversion on the target image coordinates to determine the target position code corresponding to the target image coordinates;
[0033] The target position code and the target image requirement are input into the target image compression network to obtain the target output image.
[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes an image compression device based on a parallel adapter, and the image compression device based on a parallel adapter includes:
[0035] An acquisition module, used to acquire an original image, a plurality of edited images corresponding to the original image, and an implicit neural representation network of the original image;
[0036] A generation module, used for performing network parallel connection based on the implicit neural representation network of the original image and a plurality of parallel adapters to generate a compression network to be trained;
[0037] A training module, used for training the compression network to be trained according to a plurality of edited images corresponding to the original image, to obtain a target image compression network;
[0038] A processing module is used to obtain the image compression results of the original image and each edited image according to the target image compression network.
[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes an image compression device based on a parallel adapter, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the image compression method based on a parallel adapter as described above.
[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image compression method based on the parallel adapter as described above are implemented.
[0041] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the parallel adapter-based image compression method as described above are implemented.
[0042] This application obtains an original image, multiple edited images corresponding to the original image, and an implicit neural representation network of the original image; performs network parallel connection based on the implicit neural representation network of the original image and multiple parallel adapters to generate a compression network to be trained; trains the compression network to be trained based on multiple edited images corresponding to the original image to obtain a target image compression network; obtains the image compression results of the original image and each edited image based on the target image compression network. In the above manner, multiple parallel adapters are connected in parallel in the network structure of the implicit neural representation, and parallel adapters are used to store the residuals between different INRs, thereby saving the storage space required for multiple INRs, and the use of model fine-tuning effectively alleviates the problem of redundant INR storage and improves the flexibility of image compression. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0045] Figure 1 A flowchart diagram of the first embodiment of the image compression method based on the parallel adapter of the present application;
[0046] Figure 2A schematic diagram of the training of the traditional INR of the image compression method based on the parallel adapter provided in the first embodiment of the present application;
[0047] Figure 3 A flowchart diagram of the second embodiment of the image compression method based on parallel adapters provided in the present application;
[0048] Figure 4 A schematic diagram of training an improved PA-INR of the parallel adapter-based image compression method provided in the second embodiment of the present application;
[0049] Figure 5 A schematic diagram of PA-INR weight generation by a hypernetwork of an image compression method based on a parallel adapter provided in Example 2 of the present application;
[0050] Figure 6 A schematic diagram of the PA structure of the image compression method based on the parallel adapter provided in the second embodiment of the present application;
[0051] Figure 7 A schematic diagram of a brief flow chart of an image compression method based on a parallel adapter provided in the second embodiment of the present application;
[0052] Figure 8 A schematic diagram showing comparison of results of the image compression method based on a parallel adapter provided in the second embodiment of the present application;
[0053] Fig. 9 This is a schematic diagram of the module structure of an image compression device based on a parallel adapter according to an embodiment of the present application;
[0054] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the image compression method based on the parallel adapter in the embodiment of the present application.
[0055] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0056] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0057] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0058] The main solution of the embodiment of the present application is: obtaining an original image, multiple edited images corresponding to the original image, and an implicit neural representation network of the original image; performing network parallel connection based on the implicit neural representation network of the original image and multiple parallel adapters to generate a compression network to be trained; training the compression network to be trained based on the multiple edited images corresponding to the original image to obtain a target image compression network; and obtaining image compression results of the original image and each edited image based on the target image compression network.
[0059] INR uses a neural network to map pixel coordinates to RGB values, where the data can be stored in a continuous form. At this time, the image is stored as a corresponding neural network weight, and the mapping relationship is shown in the function . In the discrete representation method of the image, the information contained in each pixel has an interpretable meaning. By adjusting the corresponding value of each pixel, the entire image can be edited. However, for implicit neural representations, it is unrealistic to accurately decouple the weights in order to adapt to the edited image because the update of the weights of each layer is unknown. In 3D INR editing, most methods use image guidance and text guidance for style transfer of 3D data. In recent years, some methods have attempted to decouple the weights of the network, hoping to make the image editing process based on INR representation interpretable and increase the generalization ability of the training process. Another approach is to use methods such as Transformer as a meta-learner to improve the generalization ability and effectiveness of INRs. These methods have promoted the development of INR editing, but their processes are cumbersome or rely on the learning of the Transformer architecture, which requires a lot of computing resources. In addition, they focus on high-level editing of images, such as style transfer operations. However, low-level editing based on INR has received less attention.
[0060] INR is a network, but due to the unexplainable nature of back-propagation, it is not possible to efficiently store associated images. This makes the storage space for associated images consistent with that for unrelated images, and the edited image cannot utilize the information of the original image. It is believed that there is a lot of redundancy, which leads to the following problems with the current method. 1. The encoding time of INR is usually higher than other methods. 2. The overfitting process of INR can be regarded as the encoding process of implicit neural representation. Since back-propagation requires a huge amount of computation to update network parameters, this process is time-consuming and requires high computing power of the device. 3. The existing INR network corresponds one-to-one with storage media such as images and videos, and the compression method is not flexible.
[0061] The present application provides a solution by connecting multiple parallel adapters in parallel in the network structure of implicit neural representation, and using parallel adapters to store the residuals between different INRs, thereby saving the storage space required for multiple INRs. The problem of redundant INR storage is effectively alleviated by using model fine-tuning, and the flexibility of image compression is improved.
[0062] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an image compression device based on a parallel adapter that can realize the above functions, etc. The following takes the image compression device based on a parallel adapter as the execution subject as an example to illustrate this embodiment and the following embodiments.
[0063] Based on this, the embodiment of the present application provides an image compression method based on a parallel adapter, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the image compression method based on parallel adapters of the present application.
[0064] In this embodiment, the image compression method based on the parallel adapter includes steps S10 to S40:
[0065] Step S10, obtaining an original image, a plurality of edited images corresponding to the original image, and an implicit neural representation network of the original image.
[0066] It should be noted that the multiple edited images corresponding to the original image refer to images associated with the original image, which are multiple versions of images generated by making different modifications or transformations to the original image. For example, the multiple images obtained after color adjustment and adding filter effects to the original image are the multiple edited images corresponding to the original image. In this embodiment, the original image can be a picture, a video, or other media that needs to be compressed and stored. This embodiment does not limit the specific content of the original image.
[0067] It can be understood that INR uses a neural network to map pixel coordinates to RGB values, where the data can be stored in a continuous form. At this time, the original image is stored as a corresponding neural network weight, and the mapping relationship is shown in the function .
[0068] Step S20, performing network parallel connection based on the implicit neural representation network of the original image and a plurality of parallel adapters to generate a compression network to be trained.
[0069] It should be noted that a parallel adapter refers to a series of small networks or modules that run in parallel with the backbone network. In a parallel adapter (PA), a pair of and To calculate the modification vector, the weight of the model becomes ,in represents the original model weight, Zoom. Typically, The parameter count is much smaller than The parameter count of , making it more efficient in terms of parameter utilization. At present, a large model fine-tuning method LoRA (Low-Rank Adaptation) can be regarded as a ScaledPA, in which and The activation functions in the matrix are discarded. Based on the specified rank , through the singular value decomposition method, Parallel Adapter learns a low-rank matrix and obtains the fine-tuning result by adding the weights of the original network and the weights of the low-rank matrix. With fewer parameters introduced, this method reduces the demand for storage and computing resources, making it feasible to fine-tune the model in a resource-constrained environment. In other words, if the INR can be fine-tuned using the model fine-tuning method, then a smaller model fine-tuning structure can be used quickly and efficiently, such as and The two parameters are much smaller than the matrix structure of the original network, storing the residual of the two images, and then using an INR network to store multiple images with editing relationships. During training, the network is sometimes enlarged to accommodate more parameters to improve performance. On the contrary, during inference, smaller structures are usually preferred to reduce the computational burden. Structural reparameterization involves building a series of training structures and equivalently converting their parameters to another set. Structural reparameterization simplifies the process of adjusting the size of the network while maintaining or improving its effect.
[0070] It is understandable that the traditional INR image compression method uses a complete multilayer perceptron (MLP) or hybrid network to represent the image, such as Figure 2 As shown, in this embodiment, a PA structure is connected in parallel to each layer of the MLP to form a PA-INR. In this embodiment, the compression network to be trained refers to an untrained PA-INR.
[0071] In a feasible implementation, step S20 may include steps A11 to A13:
[0072] Step A11, determining an initial parameter rank and an initial scaling factor according to image information of a plurality of edited images corresponding to the original image.
[0073] It should be noted that the image type and the amount of image information of each edited image are determined based on the multiple edited images corresponding to the original image. In this embodiment, the image type refers to the image type of different types of edited images, including various edited versions of the original image, such as color adjustment, geometric transformation, and image synthesis; the image information includes but is not limited to the amount of image information and the image type of each edited image.
[0074] It can be understood that the appropriate PA parameter rank is selected according to the image information of each edited image. and scaling factor , for the PA structure, where the rank It is used to control the PA network parameters, corresponding to the size of the residual storage space, that is, the size of the additional space required to store one more image. Used to control the zoom factor, specifically , which is used to control the influence weight of the residual on the output result. According to the storage difficulty of the image to be compressed, it can be flexibly adjusted to adapt to different tasks. In this embodiment, the initial parameter rank determined according to the image information refers to the selected parameter , the initial zoom factor determined based on the image information refers to the selected zoom factor In addition, the weight of PA is expressed as .
[0075] Step A12, defining an adapter structure according to the initial parameter rank and the initial scaling factor to obtain a plurality of parallel adapters.
[0076] Step A13, connecting each parallel adapter in parallel with each perception layer in the implicit neural representation network of the original image to generate a compression network to be trained.
[0077] It should be noted that the parameters of the PA structure are initially set based on the selected initial parameter rank and initial scaling factor to obtain a plurality of parallel adapters with set parameters, and a PA structure is connected in parallel to each layer of the MLP in the INR network to obtain the compression network to be trained. In this embodiment, the following five groups of original INR structures of different sizes can be used to connect the PA structure in parallel to obtain the compressed network to be trained, namely: 1. Preset0, 0.11bpp, Params:2.66 k, hidden layers:5, width:20; 2. Preset1, 0.20bpp, Params:4.89 k, hidden layers:5, width:30; 3. Preset2, 0.34bpp, Params:8.46 k, hidden layers:10, width:28; 4. Preset3,0.65bpp, Params:15.92 k, hidden layers:10, width:40; 5. Preset4, 1.24bpp, Params:30.38 k, hidden layers:13, width:49. The five parameters contained in each group of data represent the image size, the parameter amount for the INR network, the number of layers of the MLP, and the width of each layer of the MLP.
[0078] Step S30, training the compression network to be trained according to the multiple edited images corresponding to the original image to obtain a target image compression network.
[0079] It should be noted that the compression network to be trained is trained based on the original image and multiple edited images of the original image. In this embodiment, the network training can be performed by back propagation or by super network method. The trained compression network to be trained is the target image compression network.
[0080] Step S40, obtaining image compression results of the original image and each edited image according to the target image compression network.
[0081] It should be noted that the compression of the original image and each edited image is completed through the target image compression network, thereby obtaining the compression result. The backbone INR network in the target image compression network represents the original image, and each PA structure in the target image compression network represents an edited image.
[0082] In a feasible implementation manner, after step S40, steps B11 to B12 may be included:
[0083] Step B11, performing coordinate conversion on the target image coordinates to determine the target position code corresponding to the target image coordinates.
[0084] Step B12, inputting the target position code and target image requirements into the target image compression network to obtain a target output image.
[0085] It should be noted that when the original image or an edited image needs to be restored, the coordinates of the target image are transformed using a position encoding method, specifically through the following formula: , the target image coordinates are represented as high-dimensional position codes, thereby obtaining the target position codes corresponding to the target image coordinates.
[0086] It can be understood that the target image requirement refers to the image that the user expects to restore, the target position code and the target image requirement are input to the target image compression network, the other networks in the target image compression network are shielded by the target image requirement, the network structure corresponding to the image expected to be restored is determined, and the image that the user expects to restore can be obtained through the network structure and the target position coordinates. In this embodiment, the target output image refers to the image that the user expects to restore output by the target image compression network.
[0087] In the specific implementation, the target image compression network uses a parallel PA structure to store the residuals between different images. When PA is shielded for the neural network forward propagation process, PA-INR can output the original image corresponding to the original INR. When PA participates in the neural network forward propagation process, PA-INR outputs another edited image. The above process can be repeated multiple times, using one PA-INR to flexibly store multiple images.
[0088] This embodiment obtains an original image, multiple edited images corresponding to the original image, and an implicit neural representation network of the original image; connects the implicit neural representation network of the original image and multiple parallel adapters in parallel to generate a compression network to be trained; trains the compression network to be trained according to the multiple edited images corresponding to the original image to obtain a target image compression network; and obtains image compression results of the original image and each edited image according to the target image compression network. In the above manner, multiple parallel adapters are connected in parallel in the network structure of the implicit neural representation, and parallel adapters are used to store the residuals between different INRs, thereby saving the storage space required for multiple INRs. The problem of redundant INR storage is effectively alleviated by using model fine-tuning, thereby improving the flexibility of image compression.
[0089] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 In the parallel adapter-based image compression method, step S30 further includes steps S31 to S33:
[0090] Step S31, obtaining a regularization parameter.
[0091] Step S32, constructing a target loss function according to the regularization parameter and the target regularization method.
[0092] It should be noted that the regularization parameter refers to the parameter used to construct the parameter loss function. , the target regularization method refers to the Lasso regularization technique. Lasso regularization is a regularization technique commonly used in machine learning and statistical modeling. It is mainly used to control the complexity of the model, prevent overfitting, and promote the sparsity of the model. Overfitting refers to the phenomenon that the model performs very well on the training data, but has poor generalization ability on new data that has not been seen. Regularization is a common method to solve the overfitting problem by adding a penalty term to the loss function to limit the size or number of model parameters.
[0093] It can be understood that the loss function of this embodiment is the mean square error between the restored image and the original image. After the original INR is encoded using position, the loss function of this embodiment is Transformed into In the above formula, Indicates the coordinate number, represents the value of the target image, and θ generally represents an MLP network. It is important to note that since the method of this embodiment does not impose additional constraints on the composition of the INR weights, it is currently necessary to use traditional methods to convert the original image into the target editing function. Subsequently, the regularization parameter and the target regularization method are applied to convert the target function into the following form: At this time, the sum of the PA parameters and the weight of the original INR is θ, which is the pre-trained weight \[W\in R\], which can be combined with the incremental weight and recorded as , combined, the incremental weight is a parameter contained in the adapter structure. Assuming the input is set to x and the output is set to h, there is an equation: ,in is the original INR weight, is the weight of PA, and the objective loss function can be further expressed as .
[0094] In the specific implementation, for each image coordinate , using the position encoding method, using the position encoding method, the following formula: , representing it as a high-dimensional positional encoding as the input of the network , the above method is more suitable for overfitting of INR.
[0095] Step S33, training the compression network to be trained using the multiple edited images corresponding to the original image and the target loss function to obtain a target image compression network.
[0096] It should be noted that the compression network to be trained is trained by using the original image and multiple edited images corresponding to the original image and the target loss function, thereby obtaining the target image compression network.
[0097] In this embodiment, in order to verify the compatibility of the method of this embodiment with the INR method in the prior art, experiments were conducted on the INR generation method based on back propagation and the INR generation method based on the hypernetwork. For the method based on back propagation: determine whether the training has reached the set maximum number of training times. If not, continue the cycle and update the network parameters according to the goal of minimizing the loss function. Use the Adam optimizer to adjust the learning rate. The initial learning rate is 0.0001, and the Reduce LR On Plateau scheme is used to adjust the learning rate. Specifically, if the indicator does not improve within a certain number of training cycles (patience), then multiply the learning rate by a factor of 0.5 to reduce the learning rate.
[0098] In a feasible implementation, step S33 may include steps C11 to C14:
[0099] Step C11, determining an adapter to be trained in the compression network to be trained.
[0100] Step C12, freezing the weights of the network structure except the adapter to be trained in the compression network to be trained.
[0101] It should be noted that each PA structure in the compression network to be trained is used as an adapter to be trained in turn. When updating the parameters of each adapter to be trained, the weights of the INR network and other PA structures of the original image need to be frozen, such as Figure 4 As shown in the figure, frozen means frozen and overfit means overfitting.
[0102] Step C13, when the weight freezing operation is completed, the parameters of the adapter to be trained are updated through back propagation, multiple edited images corresponding to the original image and the target loss function to obtain a target adapter.
[0103] It should be noted that after the weights of the INR network and other PA structures of the original image are frozen, the weight freezing operation is completed. At this time, through back propagation, according to the INR training method, combined with the target loss function, PA-INR is directly used to overfit an edited image corresponding to the original image, and the parameters of each adapter to be trained are updated respectively. The PA structure after the parameter update is the target adapter. In this embodiment, based on the back propagation method, the initial value of the learning rate can be set to 0.0001, and the decay coefficient of the learning rate of the Reduce LR On Plateau method can be set to 0.5. In this embodiment, the restored image is calculated. The MSE with L1 regularization between the original image, , to measure the quality of stored images.
[0104] Step C14, obtaining a target image compression network according to the target adapter.
[0105] It should be noted that after updating the parameters of each PA structure, multiple target adapters will be obtained, and the multiple target adapters connected in parallel to the INR network of the original image are the target image compression network.
[0106] In a feasible implementation, step S33 may include steps D11 to D14:
[0107] Step D11, constructing a network training set according to the original image and a plurality of edited images corresponding to the original image.
[0108] It should be noted that a data set is constructed based on the original image, multiple edited images corresponding to the original image, and multiple variant images corresponding to the original image that have not appeared before, and is divided into a network test set and a network training set in a ratio of 7:3. Multiple variant images should be in the network test set.
[0109] Step D12, training the compressed network to be trained by using a hypernetwork method, the target loss function and the network training set to obtain an initial training network.
[0110] It should be noted that the compression network to be trained is trained through the hypernetwork method, the target loss function and the network training set, and the compression network to be trained completed through the hypernetwork method is the initial training network. In this embodiment, based on the hypernetwork method, the initial value of the learning rate can be set to 0.001, the batch size can be set to 32, the image can be cropped to 178×178 and 1 pixel of padding can be added. In the hypernetwork used, the input size of the token can be set to 178, the segmentation size of the image can be set to 9×9, and the padding value can be set to 1. For the implementation of the hypernetwork, this embodiment adopts a structure similar to the Vision Transformer. The size of the input vector can be set to 768, the encoding and the number of layers can be set to 6, the number of heads of the multi-head attention mechanism can be set to 12, the dimension of the attention head can be set to 64, and the dimension of the feedforward network can be set to 3072. The preset size of the INR network is 5 layers, and the width of each layer is 256. The training process based on the hypernetwork method is as follows Figure 5 As shown in the figure, FC1~FC3 in the figure are multiple fully connected layers respectively; Transformer Encoder is the transformer encoder; Original Weight tokens are the original weight tokens; Adapter Weight tokens are the adapter weight tokens; Data tokens are the data tokens.
[0111] Step D13, performing performance measurement on the initial training network to obtain a performance measurement result of the initial training network.
[0112] It should be noted that the performance of the initial training network is measured through the network test set, and the peak signal-to-noise ratio (PSNR) of the test image and the original image is calculated, so that the quality of the image can be quantitatively judged. For the overall INR network or PA-INR, the ratio of the total size of the storage network file to the number of pixels, that is, bits per pixel (BBP), is used as an indicator for quantitatively measuring the storage size. The performance measurement results of the initial training network are obtained by combining the PSNR and BBP of each test image with the PSNR threshold and the BBP threshold. In this embodiment, when the percentage of the number of images that meet the requirements of the PSNR and BBP of the test image reaches the preset percentage requirement, the performance measurement results of the initial training network can be determined as performance compliance results.
[0113] Step D14, when the performance measurement result is a performance-reaching result, a target image compression network is obtained according to the initial training network.
[0114] It should be noted that when the performance measurement result of the initial training network is a performance-compliant result, the initial training network is used as the target image compression network. In this embodiment, the PA structure in the target image compression network is as follows: Figure 6 As shown in the figure, Input and Output are input and output respectively, and merge is the merge weight.
[0115] It can be understood that the forward propagation process in this embodiment is specifically as follows: Entering the INR network, the forward propagation of each layer of the network can be summarized as the formula , Indicates the current network The input of the layer, Indicates the current network The output of the layer, Indicates the frozen original INR network weight, represents the network weight of PA. The current formula can be shown in more detail as ,in and is the matrix weight of the PA structure, represents the SINE periodic activation function. During the forward propagation process, is frozen, In and is the actual trainable parameter. The final output tensor The restored image.
[0116] In the specific implementation, the Adam optimizer is used to train the network, and the learning rate decay strategy adopts the Reduce LROn Plateau method with a learning rate decay coefficient of 0.5.
[0117] In a feasible implementation, step D13 may include steps E11 to E13:
[0118] Step E11, constructing a network test set according to a plurality of variant images corresponding to the original image.
[0119] Step E12, testing the initial training network according to the network test set to obtain the peak signal-to-noise ratio and storage index of each test image.
[0120] Step E13, obtaining the performance measurement result of the initial training network according to the peak signal-to-noise ratio and storage index of each test image.
[0121] It should be noted that the initial training network is tested according to the network test set, each test image is input into the initial training network to obtain a restored image, the peak signal-to-noise ratio (PSNR) of multiple restored images and the original image is calculated to measure the image quality, and the bit-per-pixel size between the images, i.e., the BBP, is calculated to measure the storage size required for the image. In this embodiment, the peak signal-to-noise ratio of each test image refers to the PSNR corresponding to the restored image output by the initial training network after each image in the network test set is input; the storage index of each test image refers to the BBP corresponding to the restored image output by the initial training network after each image in the network test set is input.
[0122] It can be understood that the qualification of each test image is determined based on the peak signal-to-noise ratio and storage index of each test image, and the percentage of qualified images is obtained. When the percentage of qualified images reaches the preset percentage requirement, the performance measurement result of the initial training network can be determined as the performance standard result.
[0123] In the specific implementation, the coordinate points are numbered as Positional encoding Input the target image compression network to obtain the color information of the corresponding coordinate points Due to the characteristics of neural networks, this process can be calculated in parallel, so that the color information of all coordinate points can be quickly obtained, and a complete image can be restored by combining them.
[0124] This embodiment obtains a regularization parameter; constructs a target loss function according to the regularization parameter and the target regularization method; trains the compression network to be trained using multiple edited images corresponding to the original image and the target loss function to obtain a target image compression network. In this way, the performance of the target image compression network is guaranteed.
[0125] For example, to help understand the implementation process of the image compression method based on the parallel adapter obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 7 , Figure 7 A brief schematic diagram of an image compression method based on a parallel adapter is provided, specifically:
[0126] The method of this embodiment uses a target image compression network to store multiple images. On the KODAK and ILSVRC2012 datasets, three methods for generating implicit neural representation (INR) parameters are compared. These three methods include: retraining INR from scratch (i.e., INR (Full Retraining)), fully fine-tuning the parameters of the INR network using the original image weights (i.e., INR (Full Fine-Tuning)), and the method proposed in this embodiment (i.e., Ours). Figure 8The average bit rate and performance comparison of each method on the entire KODAK dataset are shown. The area of the circular area represents the bit rate size of each method.
[0127] Through comparison and display, it is found that the method of this embodiment has the following advantages: 1. A method for effectively alleviating the redundant storage problem of INR by using model fine-tuning. Various existing compression methods for INR always ignore the characteristics of using INR as a neural network model, and simply use traditional quantization or pruning methods to try to compress INR. 2. Use parallel adapters to store residuals between different INRs, thereby saving the storage space required for multiple INRs. Based on structural reparameterization, the editing history can be stored and removed in an efficient and flexible manner. 3. By innovatively introducing the SINE cycle activation function into the parallel adapter structure, it is found through comparison that this improvement always brings a positive impact, and this impact is more obvious when the original INR network parameters are large. 4. The method of this embodiment is compatible with image and video representations and various existing INR training methods. Experiments have proved that the method of this embodiment has good stability and strong compatibility. It has shown good performance on INRs of different sizes and different types of test sets, achieving a satisfactory trade-off between image quality and storage size.
[0128] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image compression method based on the parallel adapter of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.
[0129] The present application also provides an image compression device based on a parallel adapter, please refer to Fig. 9 , the parallel adapter-based image compression device comprises:
[0130] The acquisition module 10 is used to acquire an original image, a plurality of edited images corresponding to the original image, and an implicit neural representation network of the original image.
[0131] The generation module 20 is used to perform network parallel connection based on the implicit neural representation network of the original image and multiple parallel adapters to generate a compression network to be trained.
[0132] The training module 30 is used to train the compression network to be trained according to the multiple edited images corresponding to the original image to obtain a target image compression network.
[0133] The processing module 40 is used to obtain the image compression results of the original image and each edited image according to the target image compression network.
[0134] Optionally, the generating module 20 is further used for:
[0135] The initial parameter rank and the initial scaling factor are determined according to the image information of the multiple edited images corresponding to the original image; the adapter structure is defined according to the initial parameter rank and the initial scaling factor to obtain multiple parallel adapters; each parallel adapter is connected in parallel with each perception layer in the implicit neural representation network of the original image to generate a compression network to be trained.
[0136] Optionally, the training module 30 is further used to:
[0137] Obtaining a regularization parameter; constructing a target loss function according to the regularization parameter and the target regularization method; training the compression network to be trained by using a plurality of edited images corresponding to the original image and the target loss function to obtain a target image compression network.
[0138] Optionally, the training module 30 is further used to:
[0139] Determine an adapter to be trained in the compression network to be trained; freeze the weights of the network structure except the adapter to be trained in the compression network to be trained; when the weight freezing operation is completed, update the parameters of the adapter to be trained by back propagation, multiple edited images corresponding to the original image and the target loss function to obtain a target adapter; and obtain a target image compression network according to the target adapter.
[0140] Optionally, the training module 30 is further used to:
[0141] A network training set is constructed according to the original image and multiple edited images corresponding to the original image; the compression network to be trained is trained by a hypernetwork method, the target loss function and the network training set to obtain an initial training network; a performance measurement is performed on the initial training network to obtain a performance measurement result of the initial training network; when the performance measurement result is a performance-compliant result, a target image compression network is obtained according to the initial training network.
[0142] Optionally, the training module 30 is further used to:
[0143] A network test set is constructed according to a plurality of variant images corresponding to the original image; the initial training network is tested according to the network test set to obtain a peak signal-to-noise ratio and a storage index of each test image; and a performance measurement result of the initial training network is obtained according to the peak signal-to-noise ratio and the storage index of each test image.
[0144] Optionally, the processing module 40 is further configured to:
[0145] The target image coordinates are transformed to determine the target position code corresponding to the target image coordinates; the target position code and the target image requirements are input into the target image compression network to obtain the target output image.
[0146] The parallel adapter-based image compression device provided by the present application adopts the parallel adapter-based image compression method in the above embodiment, which can solve the technical problems in the prior art that the INR network corresponds to the image one-to-one, the associated images cannot be stored efficiently, and the compression method is inflexible. Compared with the prior art, the beneficial effects of the parallel adapter-based image compression device provided by the present application are the same as the beneficial effects of the parallel adapter-based image compression method provided by the above embodiment, and the other technical features of the parallel adapter-based image compression device are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0147] The present application provides an image compression device based on a parallel adapter, and the image compression device based on a parallel adapter includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image compression method based on the parallel adapter in the above-mentioned embodiment 1.
[0148] Reference below Fig.10 , which shows a schematic diagram of the structure of a parallel adapter-based image compression device suitable for implementing an embodiment of the present application. The parallel adapter-based image compression device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The parallel adapter-based image compression device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0149] like Fig.10As shown, the parallel adapter-based image compression device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the parallel adapter-based image compression device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the parallel adapter-based image compression device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an image compression device based on a parallel adapter with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0150] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0151] The parallel adapter-based image compression device provided by the present application adopts the parallel adapter-based image compression method in the above embodiment, which can solve the technical problems in the prior art that the INR network corresponds to the image one-to-one, the associated images cannot be stored efficiently, and the compression method is inflexible. Compared with the prior art, the beneficial effects of the parallel adapter-based image compression device provided by the present application are the same as the beneficial effects of the parallel adapter-based image compression method provided by the above embodiment, and the other technical features of the parallel adapter-based image compression device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.
[0152] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0154] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the parallel adapter-based image compression method in the above-mentioned embodiment.
[0155] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.
[0156] The computer-readable storage medium may be included in the image compression device based on the parallel adapter; or may exist independently without being assembled into the image compression device based on the parallel adapter.
[0157] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an image compression device based on a parallel adapter, the image compression device based on a parallel adapter: obtains an original image, multiple edited images corresponding to the original image, and an implicit neural representation network of the original image; performs network parallel connection based on the implicit neural representation network of the original image and multiple parallel adapters to generate a compression network to be trained; trains the compression network to be trained based on the multiple edited images corresponding to the original image to obtain a target image compression network; and obtains image compression results of the original image and each edited image based on the target image compression network.
[0158] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0159] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0160] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0161] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned parallel adapter-based image compression method, and can solve the technical problems in the prior art that the INR network corresponds to the image one-to-one, the associated images cannot be stored efficiently, and the compression method is inflexible. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the parallel adapter-based image compression method provided by the above-mentioned embodiment, and will not be repeated here.
[0162] The present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned parallel adapter-based image compression method are implemented.
[0163] The computer program product provided by the present application can solve the technical problems in the prior art that the INR network corresponds to the image one-to-one, the associated images cannot be stored efficiently, and the compression method is inflexible. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the image compression method based on the parallel adapter provided in the above embodiment, which will not be described in detail here.
[0164] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A parallel adapter-based image compression method, characterized in that: The parallel adapter-based image compression method comprises: Acquire an original image, a plurality of edited images corresponding to the original image, and an implicit neural representation network of the original image; Performing network parallel connection based on the implicit neural representation network of the original image and a plurality of parallel adapters to generate a compression network to be trained; Training the compression network to be trained according to the multiple edited images corresponding to the original image to obtain a target image compression network; Obtaining image compression results of the original image and each edited image according to the target image compression network; The step of performing network parallel connection based on the implicit neural representation network of the original image and a plurality of parallel adapters to generate a compression network to be trained comprises: Determining an initial parameter rank and an initial scaling factor according to image information of a plurality of edited images corresponding to the original image; Defining an adapter structure according to the initial parameter rank and the initial scaling factor to obtain a plurality of parallel adapters; Connecting each parallel adapter in parallel with each perception layer in the implicit neural representation network of the original image to generate a compression network to be trained; The step of training the compression network to be trained according to the multiple edited images corresponding to the original image to obtain the target image compression network includes: Get the regularization parameter; Constructing a target loss function according to the regularization parameter and the target regularization method; The compression network to be trained is trained using a plurality of edited images corresponding to the original image and the target loss function to obtain a target image compression network; The step of training the compression network to be trained by using the multiple edited images corresponding to the original image and the target loss function to obtain the target image compression network includes: Determining an adapter to be trained in the compression network to be trained; Freezing weights of the network structure except the adapter to be trained in the compression network to be trained; When the weight freezing operation is completed, updating the parameters of the adapter to be trained by back propagation, the multiple edited images corresponding to the original image, and the target loss function to obtain a target adapter; A target image compression network is obtained according to the target adapter.
2. The method according to claim 1, characterized in that The step of training the compression network to be trained by using the multiple edited images corresponding to the original image and the target loss function to obtain the target image compression network comprises: Constructing a network training set according to the original image and a plurality of edited images corresponding to the original image; Training the compressed network to be trained by using a hypernetwork method, the target loss function and the network training set to obtain an initial training network; Performing performance measurement on the initial training network to obtain a performance measurement result of the initial training network; When the performance measurement result is a performance-reaching result, a target image compression network is obtained according to the initial training network.
3. The method according to claim 2, characterized in that The step of measuring the performance of the initial training network to obtain the performance measurement result of the initial training network comprises: Constructing a network test set according to a plurality of variant images corresponding to the original image; Testing the initial training network according to the network test set to obtain a peak signal-to-noise ratio and a storage index of each test image; The performance measurement result of the initial training network is obtained according to the peak signal-to-noise ratio and storage index of each test image.
4. The method according to any one of claims 1 to 3, characterized in that After the step of obtaining the image compression results of the original image and each edited image according to the target image compression network, the method further includes: Performing coordinate conversion on the target image coordinates to determine the target position code corresponding to the target image coordinates; The target position code and the target image requirement are input into the target image compression network to obtain the target output image.
5. An image compression device based on a parallel adapter, characterized in that: The parallel adapter-based image compression device comprises: An acquisition module, used to acquire an original image, a plurality of edited images corresponding to the original image, and an implicit neural representation network of the original image; A generation module, used for performing network parallel connection based on the implicit neural representation network of the original image and a plurality of parallel adapters to generate a compression network to be trained; A training module, used for training the compression network to be trained according to a plurality of edited images corresponding to the original image, to obtain a target image compression network; A processing module, used for obtaining image compression results of the original image and each edited image according to the target image compression network; The generation module is further used to determine an initial parameter rank and an initial scaling factor according to image information of multiple edited images corresponding to the original image; define an adapter structure according to the initial parameter rank and the initial scaling factor to obtain multiple parallel adapters; connect each parallel adapter in parallel with each perception layer in the implicit neural representation network of the original image to generate a compression network to be trained; The training module is further used to obtain a regularization parameter; construct a target loss function according to the regularization parameter and the target regularization method; train the compression network to be trained by using a plurality of edited images corresponding to the original image and the target loss function to obtain a target image compression network; The training module is also used to determine the adapter to be trained in the compression network to be trained; freeze the weights of the network structure other than the adapter to be trained in the compression network to be trained; when the weight freezing operation is completed, update the parameters of the adapter to be trained through back propagation, multiple edited images corresponding to the original image and the target loss function to obtain a target adapter; and obtain a target image compression network based on the target adapter.
6. An image compression device based on a parallel adapter, characterized in that: The parallel adapter-based image compression device includes: a memory, a processor, and a parallel adapter-based image compression program stored in the memory and executable on the processor, wherein the parallel adapter-based image compression program is configured to implement the parallel adapter-based image compression method as described in any one of claims 1 to 4.
7. A storage medium, characterized in that: The storage medium stores a parallel adapter-based image compression program, which, when executed by a processor, implements the parallel adapter-based image compression method according to any one of claims 1 to 4.
Citation Information
Patent Citations
Image compression method and device, equipment and storage medium
CN115834890A
Neural representation video compression method based on time domain feature enhancement
CN117750015A
Image compression method and device capable of realizing fluidization implicit neural representation, and medium
CN118096905A
Image and video compression using learned dictionary of implicit neural representations
WO2024078892A1
