Image processing model training method and image processing model application method

Abstract

The invention provides an image processing model training method and an image processing model application method, and the method comprises the steps: inputting a first potential feature into a second visual task model for the first potential feature, and obtaining a second task execution result; calculating a first loss function value based on first task execution results and second task execution results corresponding to the plurality of visual task models, so as to ensure that potential features learned by each encoder learn features of other visual task models, so as to improve the generalization ability of each encoder, and based on the plurality of potential features and the sample images, calculating a second loss function value; a second loss function value is calculated to optimize reconstruction quality within a given bitrate constraint.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image processing model training method and an image processing model application method. Background Art

[0002] In the current digital age, in order to increase the transmission rate of the Internet or save space on storage devices, images to be recognized are compressed before performing visual tasks. However, traditional image compression methods mainly focus on retaining the accuracy of low-level visual signals (such as pixel values) to reduce the visual quality loss caused by compression. These methods are often unable to cope with machine vision tasks that require high-level semantic understanding. For example, when performing object detection or image segmentation after compressing the image, the low-level pixel information retained by traditional methods cannot support these tasks well, resulting in a significant performance degradation.

[0003] And if the image is used for multiple visual tasks, the potential representation features of the image will be obtained and segmented to obtain multiple segmentation features. The multiple segmentation features are used for different visual tasks respectively. The multiple segmentation features are further compressed and decoded to be input into different visual tasks for processing. However, the coupling between the segmentation features and the visual tasks is too strong. If new visual tasks need to be processed, the corresponding encoder needs to be reset to compress the image. Summary of the invention

[0004] The present application proposes an image processing model training method and an image processing model application method, which can solve the technical problem that the current image features after image compression cannot meet the requirements of visual tasks.

[0005] The first embodiment of the present application proposes an image processing model training method, comprising:

[0006] Input the sample image into multiple encoders of the multi-task collaborative model to obtain multiple latent features;

[0007] Inputting the plurality of potential features into corresponding visual task models in the multi-task collaboration model respectively to obtain a plurality of first task execution results;

[0008] For a first latent feature, input the first latent feature into a second visual task model to obtain a second task execution result, where the second visual task model is a visual task model corresponding to the second latent feature, the first latent feature and the second latent feature are any two latent features among the multiple latent features, and the first latent feature and the second latent feature are different;

[0009] Calculate a first loss function value based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models;

[0010] Calculating a second loss function value based on the plurality of potential features and the sample image;

[0011] Based on the first loss function value and the second loss function value, the model parameters of the multiple encoders are adjusted until a preset first training completion condition is met, thereby obtaining multiple target encoders.

[0012] The second aspect of the present application provides an image processing model application method, including:

[0013] Inputting the target images into multiple target encoders of the multi-task collaborative model respectively to obtain multiple target potential features;

[0014] Fusing the multiple target latent features to obtain a target fused latent feature;

[0015] The target fusion potential features are respectively input into multiple visual task models of the multi-task collaboration model to obtain multiple target task execution results.

[0016] The third aspect of the present application provides an image processing model training method and device, including:

[0017] A first input module, used for inputting a sample image into multiple encoders of a multi-task collaborative model to obtain multiple potential features;

[0018] The first input module is further used to input the multiple potential features into corresponding visual task models in the multi-task collaboration model respectively to obtain multiple first task execution results;

[0019] The first input module is further used for inputting the first potential feature into a second visual task model to obtain a second task execution result, wherein the second visual task model is a visual task model corresponding to the second potential feature, the first potential feature and the second potential feature are any two potential features among the multiple potential features, and the first potential feature and the second potential feature are different;

[0020] A calculation module, configured to calculate a first loss function value based on the first task execution results and the second task execution results corresponding to each of the plurality of visual task models;

[0021] The calculation module is further used to calculate a second loss function value based on the multiple potential features and the sample image;

[0022] An adjustment module is used to adjust the model parameters of the multiple encoders based on the first loss function value and the second loss function value until a preset first training completion condition is met to obtain multiple target encoders.

[0023] The fourth aspect of the present application provides an image processing model application device, including:

[0024] A second input module is used to input the target image into multiple target encoders of the multi-task collaborative model respectively to obtain multiple target potential features;

[0025] A fusion module, used for fusing the multiple target potential features to obtain a target fused potential feature;

[0026] The second input module is also used to input the target fusion potential features into multiple visual task models of the multi-task collaboration model respectively to obtain multiple target task execution results.

[0027] An embodiment of the fifth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect or the second aspect above.

[0028] The embodiment of the sixth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect or the second aspect above.

[0029] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0030] The present application proposes an image processing model training method, an image processing model training method, a device and a storage medium, the method comprising: inputting a sample image into multiple encoders of a multi-task collaborative model to obtain multiple potential features; respectively inputting the multiple potential features into corresponding visual task models in the multi-task collaborative model to obtain multiple first task execution results; for the first potential feature, inputting the first potential feature into a second visual task model to obtain a second task execution result, the second visual task model is a visual task model corresponding to the second potential feature, the first potential feature and the second potential feature are any two potential features among the multiple potential features, and the first potential feature and the second potential feature are different; based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models, calculating a first loss function value; based on the multiple potential features and the sample image, calculating a second loss function value; based on the first loss function value and the second loss function value, adjusting the model parameters of the multiple encoders until the preset first training completion condition is met to obtain multiple target encoders. In the embodiment of the present application, for a first latent feature, the first latent feature is input into a second visual task model to obtain a second task execution result; based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models, a first loss function value is calculated to ensure that the latent features learned by each encoder learn the features of other visual task models to improve the generalization ability of each encoder, and based on multiple latent features and sample images, a second loss function value is calculated to optimize the reconstruction quality within a given bit rate constraint.

[0031] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0033] In the attached picture:

[0034] Figure 1 A flow chart of an image processing model training method provided by an embodiment of the present application is shown;

[0035] Figure 2 A schematic diagram of a model structure of a multi-task collaboration model provided in an embodiment of the present application is shown;

[0036] Figure 3 A schematic diagram of the model structure of a denoising model provided in an embodiment of the present application is shown;

[0037] Figure 4 A flow chart of an image processing model application method provided by an embodiment of the present application is shown;

[0038] Figure 5 A schematic diagram of a model structure of another multi-task collaboration model provided in an embodiment of the present application is shown;

[0039] Figure 6 A schematic diagram of the structure of an image processing model training device provided by an embodiment of the present application is shown;

[0040] Figure 7 A schematic diagram of the structure of an image processing model application device provided by an embodiment of the present application is shown;

[0041] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0042] Fig. 9 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0044] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0045] Based on the above background technology, the current related image compression technology mainly has the following problems:

[0046] Image compression and computer vision goals are inconsistent: Traditional image compression and computer vision have different goals. The former focuses on preserving low-level pixel signals to optimize storage and transmission efficiency, while the latter focuses on extracting semantic information from visual data. Existing image compression methods will lead to insufficient semantic embedding representation of reconstructed images at low bit rates, resulting in a significant drop in the performance of downstream machine vision tasks.

[0047] 2 Limitations of task-driven image compression: The image feature stream is decomposed into N sub-image feature streams, and machine vision constraints are introduced in the codec training to bind them to the N parts respectively. The compression performance is optimized through the task-driven loss function, but these sub-image feature streams are tightly coupled with specific models or tasks and are difficult to adapt to other tasks or models.

[0048] In order to solve the above technical problems, the present application proposes an image processing model training method, an image processing model training method, a device and a storage medium, the method comprising: inputting a sample image into multiple encoders of a multi-task collaborative model to obtain multiple potential features; respectively inputting the multiple potential features into the corresponding visual task models in the multi-task collaborative model to obtain multiple first task execution results; for the first potential feature, inputting the first potential feature into the second visual task model to obtain a second task execution result, the second visual task model is a visual task model corresponding to the second potential feature, the first potential feature and the second potential feature are any two potential features among the multiple potential features, and the first potential feature and the second potential feature are different; based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models, calculating a first loss function value; based on the multiple potential features and the sample image, calculating a second loss function value; adjusting the model parameters of the multiple encoders based on the first loss function value and the second loss function value until the preset first training completion condition is met to obtain multiple target encoders. In the embodiment of the present application, for a first latent feature, the first latent feature is input into a second visual task model to obtain a second task execution result; based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models, a first loss function value is calculated to ensure that the latent features learned by each encoder learn the features of other visual task models to improve the generalization ability of each encoder, and based on multiple latent features and sample images, a second loss function value is calculated to optimize the reconstruction quality within a given bit rate constraint.

[0049] The following describes an image processing model training method proposed according to an embodiment of the present application in conjunction with the accompanying drawings.

[0050] See also Figure 1 , the method specifically comprises the following steps:

[0051] S101. Input a sample image into multiple encoders of a multi-task collaborative model to obtain multiple potential features.

[0052] The encoder is used to compress the sample image and obtain the compressed potential features.

[0053] The multiple latent features correspond to different visual task models. Since the visual task models have different emphases on sample images, the information amounts of the multiple latent features are different.

[0054] In some embodiments, the process of obtaining multiple potential features using an encoder can be implemented as follows:

[0055] Receive the original image as input; use structures such as convolutional neural networks to extract image features, which are gradually compressed in space; reduce the resolution of the feature map through pooling layers or convolution layers while retaining important information; feature encoding: further encode the extracted features into a more compact representation, which usually includes linear transformation of the features or application of more complex transformations; quantize and encode the encoded features to generate compressed image features, i.e. latent features.

[0056] S102: Input the plurality of potential features into corresponding visual task models in the multi-task collaboration model respectively to obtain a plurality of first task execution results.

[0057] Each latent feature corresponds to a visual task model, that is, the visual task model can output an accurate execution result based on the corresponding latent feature, that is, the first task execution result.

[0058] It can be understood that multiple potential features are compressed image features. Therefore, the visual task model may include a decoder and a task processor. The encoder is used to reconstruct the sample image based on the potential features to obtain a reconstructed image, and the task processor is used to perform the corresponding visual task based on the reconstructed image.

[0059] In some embodiments, the processing of the encoder and the decoder can be implemented as follows: Assume that E k (·) is the encoder of task k, which converts the original image Mapping to embedding space And D k (·) is the decoder, which embeds the space y k The feature maps in are then reconstructed

[0060] S103: For the first latent feature, input the first latent feature into a second visual task model to obtain a second task execution result.

[0061] S104: Calculate a first loss function value based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models.

[0062] The second visual task model is a visual task model corresponding to the second latent feature, the first latent feature and the second latent feature are any two latent features among the multiple latent features, and the first latent feature and the second latent feature are different.

[0063] Among them, the first loss function value can be a cross-task loss function.

[0064] Since the multi-task collaboration model includes multiple visual task models, and each visual task model corresponds to a potential feature, the multi-task collaboration model includes multiple task model groups, and each task model group includes an encoder and a visual task model.

[0065] In some embodiments, the model structure of the multi-task collaboration model is as follows: Figure 2 As shown, the multi-task collaboration model includes multiple task model groups, each task model group includes an encoder, a decoder and a task processor, the sample image passes through multiple encoders to obtain potential features including different information, the multiple potential features pass through corresponding decoders respectively to obtain different reconstructed images, and different reconstructed images pass through the corresponding task processors to obtain the corresponding first task execution results.

[0066] In some embodiments, the first potential feature can be input into the visual task model of other task model groups, such as the second visual task model, and the second task execution result output by the second visual task model based on the first potential feature is obtained. Similarly, each visual task model can output a corresponding task execution result based on the potential feature corresponding to other task model groups, so as to obtain the first task execution result and the second task execution result corresponding to each of the multiple visual task models, wherein the first task execution result is the task execution result output by the visual task model based on the corresponding potential feature, and the second task execution result is the task execution result output by the visual task model based on the potential feature corresponding to other task model groups.

[0067] It can be understood that in order to determine the accuracy of multiple visual task models, the visual task models can be trained based on the first task execution results of each visual task model and the true value corresponding to each visual task, and based on the difference between the first task execution results and the second task execution results corresponding to each visual task model, the first loss function value of the multi-task collaboration model is calculated to train multiple encoders.

[0068] Among them, the first loss function can be expressed by formula (1):

[0069]

[0070] in, is the first loss function, Y k is the true value of the visual task label, is the predicted label result of the reconstructed image, that is, the first task execution result of the visual task model, ‖·‖2 is norm, The second task execution result of the visual task model. τ is a hyperparameter used to balance the learning ability of specific tasks and the generalization ability of multiple encoders.

[0071] S105. Calculate a second loss function value based on multiple potential features and sample images.

[0072] The second loss function value is a rate-distortion loss function value, which is used to optimize the reconstruction quality within a given bit rate constraint.

[0073] A reconstructed image may be obtained based on a plurality of potential features, and distortion rates corresponding to respective encoders may be determined based on the sample image and the reconstructed image.

[0074] In some embodiments, the bit rates of multiple encoders may be determined based on multiple potential features, so that the multiple encoders may be trained based on the distortion rate and the bit rate to optimize the reconstruction quality under a given bit rate constraint.

[0075] In some embodiments, the bit rates of multiple potential images can be determined based on an entropy model estimation method, that is, the entropy model calculates the probability distribution of each symbol in the potential feature, and calculates the optimal coding length based on this. In this way, the entropy model can determine how many bits are required to represent each pixel on average in lossless compression, thereby determining the bit rate of image compression.

[0076] In some embodiments, multiple potential features may be fused to obtain fused potential features, and the bit rates of the multiple potential images may be determined based on entropy model estimation.

[0077] In some embodiments, a second loss function value is calculated based on multiple potential features and a sample image, including: fusing multiple potential features to obtain a fused potential feature; inputting the fused potential feature into a decoder of any visual task model to obtain a reconstructed image; calculating a distortion loss function value based on the reconstructed image and the sample image; calculating a probability distribution of statistical features of the fused potential feature; and calculating a second loss function value based on the distortion loss function value and the probability distribution of statistical features of the fused potential feature.

[0078] In some embodiments, the second loss function can be expressed by formula (2):

[0079]

[0080] in, is the second loss function, Bit rate loss function, is the bit rate loss function, and λ is the trade-off parameter that controls the trade-off between the bit rate loss function and the bit rate loss function. The probability distribution of the statistical features of the fused latent features, represents the average amount of information per symbol in the fused latent features, is the reconstructed image, x is the sample image.

[0081] S106. Adjust model parameters of multiple encoders based on the first loss function value and the second loss function value until a preset first training completion condition is met, thereby obtaining multiple target encoders.

[0082] In some embodiments, a first target loss function value can be determined based on the first loss function value and the second loss function value, and model parameters of multiple encoders can be adjusted based on the first target loss function value until a preset first training completion condition is met, thereby obtaining multiple target encoders.

[0083] The first objective loss function can be expressed as:

[0084]

[0085] in, is the first objective loss function, is the second loss function, is the first loss function, μ is a hyperparameter used in and Make a trade-off between.

[0086] The first training completion condition may be that the number of training times reaches a training times threshold, or the target loss function value reaches a preset value, etc.

[0087] In the embodiment of the present application, for a first latent feature, the first latent feature is input into a second visual task model to obtain a second task execution result; based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models, a first loss function value is calculated to ensure that the latent features learned by each encoder learn the features of other visual task models to improve the generalization ability of each encoder, and based on multiple latent features and sample images, a second loss function value is calculated to optimize the reconstruction quality within a given bit rate constraint.

[0088] In some embodiments, the above method also includes: fusing multiple target potential features output by multiple target encoders to obtain target fused potential features; converting the target fused potential features into aligned potential features based on multiple representative features in the memory bank; inputting the aligned potential features into the denoising model of the multi-task collaborative model to obtain generalized potential features; calculating the third loss function value based on the aligned potential features and the target fused potential features; calculating the fourth loss function value based on the generalized potential features and the aligned potential features; adjusting the representative features of the memory bank and the model parameters of the denoising model based on the third loss function value and the fourth loss function value, and continuing training until the preset second training completion condition is met to obtain the target memory bank and the target denoising model.

[0089] In some embodiments, the process of fusing multiple target potential features can be implemented as follows: Assume that N target potential features are The target fusion potential features are generated by the aggregation function, where the target fusion potential features can be expressed by formula (4):

[0090]

[0091] in,, is the k-th task-specific embedding representation, is the target fusion latent feature, which integrates the information of all tasks.

[0092] Among them, the memory bank is used to store representative features of the latent space. The representative features can be obtained by encoding and transforming the input data, reflecting the core information and structure of the data, and any latent feature can be represented by multiple representative features.

[0093] Therefore, the target fused latent features can be transformed into aligned latent features based on multiple representative features in the memory bank to alleviate the overfitting problem and help to acquire invariant knowledge applicable to open set scenarios.

[0094] It is understandable that the multiple representative features in the memory bank need to be updated in time to improve the accuracy of the aligned latent features converted by Baocheng. Therefore, the third loss function value can be calculated based on the difference between the aligned latent features and the target fusion latent features, and the representative features of the memory bank can be updated in time based on the third loss function value.

[0095] The third loss function can be expressed as:

[0096]

[0097] in, is the third loss function, To align latent features, Fusion of latent features for the target.

[0098] It can be understood that since multiple target encoders are trained under a given bit rate constraint, there is distortion and noise in the target latent features output by the multiple target encoders. Therefore, it is necessary to input the aligned latent features into the denoising model of the multi-task collaborative model to remove the noise in the aligned latent noise and restore the information of the aligned latent features.

[0099] Furthermore, the fourth loss function value can be calculated based on the generalized latent features and the aligned latent features, so that the representative features of the memory bank and the model parameters of the denoising model can be jointly adjusted based on the third loss function value and the fourth loss function value until the preset second training completion condition is met to obtain the target memory bank and the target denoising model.

[0100] Among them, the second training completion condition may be that the number of training times reaches a training times threshold, or the target loss function value reaches a preset value, etc.

[0101] In some embodiments, jointly adjusting the representative features of the memory bank and the model parameters of the denoising model based on the third loss function value and the fourth loss function value can be implemented as follows:

[0102] A second target loss function is generated based on the third loss function and the fourth loss function, and representative features of the memory bank and model parameters of the denoising model are adjusted based on the second target loss function.

[0103] The second objective loss function can be expressed as:

[0104]

[0105] in, is the second objective loss function, is the third loss function, is the fourth loss function, η is a hyperparameter used in and Make a trade-off between.

[0106] In some embodiments, based on multiple representative features in a memory library, a target fusion potential feature is converted into an alignment potential feature, including: calculating distance measurement values ​​between the target fusion potential feature and multiple representative features respectively; normalizing the multiple distance measurement values ​​to obtain multiple normalized measurement values; describing the target fusion potential feature based on multiple representative features and the normalized measurement values ​​corresponding to each of the multiple representative features to obtain an alignment potential feature.

[0107] The distance measurement value may be a measure of the proximity or matching degree between the target fusion potential feature and the plurality of representative features stored in the memory library.

[0108] The distance metric can be expressed as:

[0109]

[0110] Among them, a j is the distance metric value, Dist(·,·) is the distance metric function, Fusion of latent features m for the target jThere are many representative features.

[0111] After determining the distance metric value, in order to facilitate calculation, multiple distance metric values ​​can be normalized to obtain multiple normalized metric values, where the softmax function can be used to normalize the L distance metric values ​​a. j Normalize and use the normalized distance metric a j and all representative features in the memory bank The reconstruction target fuses the latent features to obtain aligned latent features.

[0112] In some embodiments, the process of obtaining the aligned latent features can be represented by the function group (8):

[0113]

[0114] Among them, a j ’ is the normalized metric value, is the alignment latent feature.

[0115] Different from traditional image compression techniques that rely on embedded representations, aligning target fused latent features with multiple representative features can alleviate the overfitting problem and help to acquire invariant knowledge applicable to open set scenarios.

[0116] In some embodiments, the aligned latent features are input into a denoising model to obtain a generalized latent feature, including: inputting noise and the aligned latent features into a modulation subnetwork of the denoising model; inputting noise into a diffusion model of the denoising model; and using the modulation subnetwork to learn the denoising capability of the diffusion model so as to output the generalized latent features corresponding to the aligned latent features through the denoising model.

[0117] In some embodiments, the denoising model may include a pre-trained diffusion model and a modulation sub-network.

[0118] The diffusion model construction process can be implemented as follows:

[0119] Add Gaussian noise with variance βt∈(0,1) at time t to the encoded latent space y=E(x) to produce a noisy latent space: Among them, ∈~(0,I), αt=1-βt, and When t is large enough, the potential z t It is almost a standard Gaussian distribution. The diffusion process is the reverse process of the above process, that is, from z t The process to z, that is, removing z t The process of removing the noise ∈ in and obtaining the noise-free feature z.

[0120] The denoising ability of the diffusion model is further learned by utilizing the modulation sub-network to align the generalized latent features corresponding to the latent features output by the denoising model.

[0121] In some embodiments, the model structure of the denoising model is as follows: Figure 3 As shown in Fig. 1, the denoising model includes a diffusion model and a modulation subnetwork parallel to the diffusion model. The output of the modulation subnetwork is connected to the diffusion model, and the input of the modulation subnetwork is the alignment potential feature and the noise z t , the input of the diffusion model is the noise z t , the output of the diffusion model is the output generalization potential feature of the denoising model.

[0122] In some embodiments, calculating a fourth loss function value based on the generalized potential feature and the aligned potential feature includes: calculating a noise difference between the generalized potential feature and the aligned potential feature; and calculating the fourth loss function value based on a preset noise difference and the noise difference.

[0123] It can be understood that the denoising model may include multiple denoising sub-models, each denoising sub-model can remove part of the noise of the aligned latent feature, each denoising sub-model has a corresponding preset sub-noise difference, and each time the aligned latent feature passes through a denoising sub-model, a process latent feature will be obtained. The corresponding sub-noise difference can be calculated based on the output feature of each denoising sub-model and the input feature of the denoising sub-model, and the fourth loss function value can be calculated based on the sub-noise differences corresponding to multiple denoising sub-models and the preset sub-noise differences.

[0124] The fourth loss function can be expressed by formula (9):

[0125]

[0126] in, is the fourth loss function, c is the conditional information, which may be the label or context information related to the generation task, t is the time step, which is usually used to control the noise addition and denoising process in the diffusion model. To align the potential features, ∈ is the preset noise difference, is the noise difference predicted by the model.

[0127] It should be noted that the loss function represented by formula (9) is the case where the denoising model has only one denoising sub-model. If there are multiple denoising sub-models, it is necessary to add multiple preset noise differences and model predicted noise differences to formula (9) to obtain four loss functions.

[0128] During the adjustment process, the conditional information is empty and the denoising model is optimized by minimizing the fourth loss function.

[0129] The following is a description of an image processing model training method proposed in accordance with an embodiment of the present application in conjunction with the accompanying drawings. The method is applied to Figure 1 The multi-task collaborative model obtained by the image processing model training method shown in Figure 4 , the method specifically comprises the following steps:

[0130] S401. Input the target image into multiple target encoders of the multi-task collaborative model respectively to obtain multiple target potential features.

[0131] S402: Fuse multiple target latent features to obtain target fused latent features.

[0132] S403, inputting the target fusion potential features into multiple visual task models of the multi-task collaboration model respectively to obtain multiple target task execution results.

[0133] In some embodiments, the target fusion latent features are respectively input into multiple visual task models to obtain multiple target task execution results, including: based on multiple representative features in a memory bank, converting the target fusion latent features into aligned latent features; inputting the aligned latent features into a denoising model of a multi-task collaboration model to obtain generalized latent features; and inputting the generalized latent features into multiple visual task models to obtain multiple target task execution results.

[0134] In some embodiments, the structure of the multi-task collaboration model is as follows: Figure 5 As shown, the multi-task collaborative model includes multiple decoders, a fusion processor, a memory bank, a denoising model, multiple decoders and multiple task processors.

[0135] Multiple decoders output multiple potential features based on the sample image. The multiple potential features are fused to obtain the target fused potential features. The target fused potential features are transformed through multiple representative features in the memory bank to obtain the aligned potential features. The noise of the aligned potential features is removed by the denoising model and the information hidden in the aligned potential features is restored to obtain the generalized potential features. The generalized potential features are passed through any decoder to obtain the reconstructed image. Multiple task processors perform corresponding tasks based on the reconstructed image to obtain multiple task execution results.

[0136] The present application also provides an image processing model training device, which is used to perform the above Figure 1 The image processing model training method provided in the embodiment. Figure 6 As shown, the device includes: a first input module 601, a calculation module 602 and an adjustment module 603.

[0137] A first input module 601 is used to input a sample image into multiple encoders of a multi-task collaborative model to obtain multiple potential features;

[0138] The first input module 601 is further used to input the multiple potential features into the corresponding visual task models in the multi-task collaboration model respectively to obtain multiple first task execution results;

[0139] The first input module 601 is further used for inputting the first potential feature into a second visual task model to obtain a second task execution result, wherein the second visual task model is a visual task model corresponding to the second potential feature, the first potential feature and the second potential feature are any two potential features among the multiple potential features, and the first potential feature and the second potential feature are different;

[0140] A calculation module 602 is used to calculate a first loss function value based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models;

[0141] The calculation module 602 is further used to calculate a second loss function value based on the multiple potential features and the sample image;

[0142] The adjustment module 603 is used to adjust the model parameters of the multiple encoders based on the first loss function value and the second loss function value until a preset first training completion condition is met to obtain multiple target encoders.

[0143] The present application proposes an image processing model training device, which obtains multiple potential features by inputting a sample image into multiple encoders of a multi-task collaborative model; respectively inputting the multiple potential features into the corresponding visual task models in the multi-task collaborative model to obtain multiple first task execution results; for the first potential feature, inputting the first potential feature into the second visual task model to obtain a second task execution result, the second visual task model is the visual task model corresponding to the second potential feature, the first potential feature and the second potential feature are any two potential features among the multiple potential features, and the first potential feature and the second potential feature are different; based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models, a first loss function value is calculated; based on the multiple potential features and the sample image, a second loss function value is calculated; based on the first loss function value and the second loss function value, the model parameters of the multiple encoders are adjusted until a preset first training completion condition is met to obtain multiple target encoders. In the embodiment of the present application, for a first latent feature, the first latent feature is input into a second visual task model to obtain a second task execution result; based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models, a first loss function value is calculated to ensure that the latent features learned by each encoder learn the features of other visual task models to improve the generalization ability of each encoder, and based on multiple latent features and sample images, a second loss function value is calculated to optimize the reconstruction quality within a given bit rate constraint.

[0144] In some embodiments, the above-mentioned device further includes: a fusion module and a conversion module;

[0145] A fusion module, used for fusing the multiple target potential features output by the multiple target encoders to obtain a target fused potential feature;

[0146] a conversion module, for converting the target fused latent features into aligned latent features based on a plurality of representative features in a memory bank;

[0147] The first input module 601 is further used to input the aligned latent features into the denoising model of the multi-task collaboration model to obtain generalized latent features;

[0148] The calculation module 602 is further used to calculate a third loss function value based on the alignment potential feature and the target fusion potential feature;

[0149] The calculation module 602 is further used to calculate a fourth loss function value based on the generalized potential feature and the aligned potential feature;

[0150] The adjustment module 603 is also used to adjust the representative features of the memory bank and the model parameters of the denoising model based on the third loss function value and the fourth loss function value, and continue training until the preset second training completion condition is met to obtain the target memory bank and the target denoising model.

[0151] In some embodiments, the calculation module 602 is specifically used to:

[0152] Fusing the multiple latent features to obtain a fused latent feature;

[0153] Inputting the fused latent features into a decoder of any visual task model to obtain a reconstructed image;

[0154] Calculating a distortion loss function value based on the reconstructed image and the sample image;

[0155] Calculating the probability distribution of the statistical features of the fused latent features;

[0156] A second loss function value is calculated based on the distortion loss function value and the probability distribution of the statistical features of the fused latent features.

[0157] In some embodiments, the conversion module is specifically used to:

[0158] Calculating distance measurement values ​​between the target fusion potential feature and the plurality of representative features respectively;

[0159] Normalizing multiple distance metric values ​​to obtain multiple normalized metric values;

[0160] The target fusion potential feature is described based on the multiple representative features and the normalized measurement values ​​corresponding to each of the multiple representative features to obtain an aligned potential feature.

[0161] In some embodiments, the first input module 601 is specifically used to:

[0162] Inputting the noise and the aligned latent features into a modulation subnetwork of a denoising model;

[0163] Inputting the noise into a diffusion model of the denoising model;

[0164] The modulation sub-network is used to learn the denoising capability of the diffusion model, so as to output the generalized latent feature corresponding to the aligned latent feature through the denoising model.

[0165] In some embodiments, the calculation module 603 is specifically configured to:

[0166] Calculating a noise difference between the generalized latent feature and the aligned latent feature;

[0167] A fourth loss function value is calculated based on a preset noise difference value and the noise difference value.

[0168] The present application also provides an image processing model application device, which is used to perform the above Figure 4 The image processing model application method provided in the embodiment. Figure 7 As shown, the device includes: a second input module 701 and a fusion module 702.

[0169] The second input module 701 is used to input the target image into multiple target encoders of the multi-task collaborative model respectively to obtain multiple target potential features;

[0170] A fusion module 702 is used to fuse the multiple target potential features to obtain a target fused potential feature;

[0171] The second input module 701 is further used to input the target fusion potential features into multiple visual task models of the multi-task collaboration model respectively to obtain multiple target task execution results.

[0172] In some embodiments, the second input module 701 is specifically used to:

[0173] Based on a plurality of representative features in a memory bank, converting the target fused latent features into aligned latent features;

[0174] Inputting the aligned latent features into a denoising model of the multi-task collaborative model to obtain a generalized latent feature;

[0175] The generalized latent features are respectively input into multiple visual task models to obtain multiple target task execution results.

[0176] The present application also provides an electronic device to perform the above-mentioned image processing model training method. Figure 8 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 8 As shown, the electronic device 7 includes: a processor 700, a memory 701, a bus 702 and a communication interface 703, and the processor 700, the communication interface 703 and the memory 701 are connected via the bus 702; the memory 701 stores a computer program that can be run on the processor 700, and when the processor 700 runs the computer program, it executes the image processing model training method or the image processing model application method provided in any of the aforementioned embodiments of the present application.

[0177] The memory 701 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 703 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0178] The bus 702 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 701 is used to store programs, and the processor 700 executes the program after receiving the execution instruction. The image processing model training method or the image processing model application method disclosed in any implementation of the embodiment of the present application may be applied to the processor 700, or implemented by the processor 700.

[0179] The processor 700 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 700. The above processor 700 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 701, and the processor 700 reads the information in the memory 701 and completes the steps of the above method in combination with its hardware.

[0180] The electronic device provided in the embodiment of the present application and the image processing model training method or image processing model application method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0181] The present application also provides a computer-readable storage medium corresponding to the image processing model training method provided in the above embodiment. Fig. 9 The computer-readable storage medium shown is a CD 30 on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the image processing model training method or image processing model application method provided by any of the aforementioned embodiments.

[0182] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0183] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the image processing model training method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0184] It should be noted that:

[0185] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0186] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0187] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0188] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for training an image processing model, characterized in that: include: Input the sample image into multiple encoders of the multi-task collaborative model to obtain multiple latent features; Inputting the plurality of potential features into corresponding visual task models in the multi-task collaboration model respectively to obtain a plurality of first task execution results; For a first latent feature, input the first latent feature into a second visual task model to obtain a second task execution result, where the second visual task model is a visual task model corresponding to the second latent feature, the first latent feature and the second latent feature are any two latent features among the multiple latent features, and the first latent feature and the second latent feature are different; Calculate a first loss function value based on the first task execution results and the second task execution results corresponding to each of the multiple visual task models; Calculating a second loss function value based on the plurality of potential features and the sample image; Based on the first loss function value and the second loss function value, the model parameters of the multiple encoders are adjusted until a preset first training completion condition is met, thereby obtaining multiple target encoders.

2. The method according to claim 1, characterized in that The method further comprises: Fusing multiple target latent features output by the multiple target encoders to obtain a target fused latent feature; Based on a plurality of representative features in a memory bank, converting the target fused latent features into aligned latent features; Inputting the aligned latent features into a denoising model of the multi-task collaborative model to obtain a generalized latent feature; Calculating a third loss function value based on the aligned latent features and the target fused latent features; Calculating a fourth loss function value based on the generalized latent feature and the aligned latent feature; Based on the third loss function value and the fourth loss function value, the representative features of the memory bank and the model parameters of the denoising model are adjusted, and training is continued until a preset second training completion condition is met to obtain a target memory bank and a target denoising model.

3. The method according to claim 1, characterized in that The calculating a second loss function value based on the plurality of potential features and the sample image comprises: Fusing the multiple latent features to obtain a fused latent feature; Inputting the fused latent features into a decoder of any visual task model to obtain a reconstructed image; Calculating a distortion loss function value based on the reconstructed image and the sample image; Calculating the probability distribution of the statistical features of the fused latent features; A second loss function value is calculated based on the distortion loss function value and the probability distribution of the statistical features of the fused latent features.

4. The method according to claim 2, characterized in that: The method of converting the target fusion potential feature into an alignment potential feature based on a plurality of representative features in the memory bank comprises: Calculating distance measurement values ​​between the target fusion potential feature and the plurality of representative features respectively; Normalizing multiple distance metric values ​​to obtain multiple normalized metric values; The target fusion potential feature is described based on the multiple representative features and the normalized measurement values ​​corresponding to each of the multiple representative features to obtain an aligned potential feature.

5. The method according to claim 2, characterized in that: The step of inputting the aligned latent features into a denoising model to obtain a generalized latent feature comprises: Inputting the noise and the aligned latent features into a modulation subnetwork of a denoising model; Inputting the noise into a diffusion model of the denoising model; The modulation sub-network is used to learn the denoising capability of the diffusion model, so as to output the generalized latent feature corresponding to the aligned latent feature through the denoising model.

6. The method according to claim 5, characterized in that The calculating a fourth loss function value based on the generalized potential feature and the aligned potential feature comprises: Calculating a noise difference between the generalized latent feature and the aligned latent feature; A fourth loss function value is calculated based on a preset noise difference value and the noise difference value.

7. A method for applying an image processing model, characterized in that: A multi-task collaboration model obtained by the image processing model training method according to any one of claims 1 to 6, the method comprising: Inputting the target images into multiple target encoders of the multi-task collaborative model respectively to obtain multiple target potential features; Fusing the multiple target latent features to obtain a target fused latent feature; The target fusion potential features are respectively input into multiple visual task models of the multi-task collaboration model to obtain multiple target task execution results.

8. The method according to claim 7, characterized in that The step of inputting the target fusion potential features into the multiple visual task models of the multi-task collaboration model to obtain multiple target task execution results includes: Based on a plurality of representative features in a memory bank, converting the target fused latent features into aligned latent features; Inputting the aligned latent features into a denoising model of the multi-task collaborative model to obtain a generalized latent feature; The generalized latent features are respectively input into multiple visual task models to obtain multiple target task execution results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images