Image processing model training method and image processing model application method

Abstract

The application provides an image processing model training method and an image processing model application method. Embodiments of the application input a first latent feature to a second visual task model to obtain a second task execution result. A first loss function value is calculated based on first task execution results and second task execution results corresponding to a plurality of visual task models, to ensure that each encoder learns a feature of other visual task models, to improve the generalization ability of each encoder. A second loss function value is calculated based on a plurality of latent features and sample images, to optimize the reconstruction quality within a given bit rate constraint.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to an image processing model training method and an image processing model application method. BACKGROUND

[0002] In the current digital era, in order to improve the transmission rate of the Internet or save the space of the storage device, image compression will be performed on the image to be recognized before performing a visual task. However, the traditional image compression method mainly focuses on preserving the accuracy of low-level visual signals (such as pixel values) to reduce the visual quality loss caused by compression. These methods often fall short when dealing with machine vision tasks that require high-level semantic understanding. For example, when performing target detection or image segmentation after compressing the image, the low-level pixel information preserved by the traditional method cannot well support these tasks, resulting in a significant decrease in performance.

[0003] And if the image is used for multiple visual tasks, the latent representation features of the image are obtained, and the latent representation features are segmented to obtain multiple segmentation features, and the multiple segmentation features are respectively used for different visual tasks. Further, the multiple segmentation features are compressed and decoded to be input to different visual tasks for processing, but the coupling between the segmentation features and the visual tasks is too strong, and if a new visual task needs to be processed, the corresponding encoder needs to be re-set to compress the image. SUMMARY

[0004] The present application proposes an image processing model training method and an image processing model application method, which can solve the technical problem that the image features after image compression cannot meet the demand of the visual task.

[0005] The first aspect embodiment of the present application proposes an image processing model training method, comprising:

[0006] inputting a sample image into multiple encoders of a multi-task cooperation model to obtain multiple latent features;

[0007] inputting the multiple latent features into corresponding visual task models in the multi-task cooperation model respectively to obtain multiple first task execution results;

[0008] for a first latent feature, inputting the first latent feature into a second visual task model to obtain a second task execution result, the second visual task model being a visual task model corresponding to a second latent feature, the first latent feature and the second latent feature being any two of the multiple latent features, and the first latent feature and the second latent feature being different;

[0009] compute a first loss function value based on the first task execution result and the second task execution result corresponding to each of the plurality of visual task models;

[0010] compute a second loss function value based on the plurality of latent features and the sample image;

[0011] adjust model parameters of the plurality of encoders based on the first loss function value and the second loss function value until a preset first training completion condition is met, to obtain a plurality of target encoders.

[0012] The second aspect embodiment of the present application provides an image processing model application method, comprising:

[0013] input the target image into the plurality of target encoders of the multi-task cooperation model respectively, to obtain a plurality of target latent features;

[0014] fuse the plurality of target latent features to obtain a target fusion latent feature;

[0015] input the target fusion latent feature into the plurality of visual task models of the multi-task cooperation model respectively, to obtain a plurality of target task execution results.

[0016] The third aspect embodiment of the present application provides an image processing model training method and device, comprising:

[0017] a first input module configured to input a sample image into a plurality of encoders of a multi-task cooperation model, to obtain a plurality of latent features;

[0018] The first input module is further configured to input the plurality of latent features into corresponding visual task models of the multi-task cooperation model respectively, to obtain a plurality of first task execution results;

[0019] The first input module is further configured to, for a first latent feature, input the first latent feature into a second visual task model to obtain a second task execution result, the second visual task model being a visual task model corresponding to a second latent feature, the first latent feature and the second latent feature being any two of the plurality of latent features, and the first latent feature and the second latent feature being different;

[0020] a computing module configured to compute a first loss function value based on the first task execution result and the second task execution result corresponding to each of the plurality of visual task models;

[0021] The computing module is further configured to compute a second loss function value based on the plurality of latent features and the sample image;

[0022] The adjusting module is configured to adjust model parameters of the plurality of encoders based on the first loss function value and the second loss function value until a preset first training completion condition is met, to obtain a plurality of target encoders.

[0023] Embodiments of the fourth aspect of the application provide an image processing model application device, comprising:

[0024] The second input module is configured to input a target image into a plurality of target encoders of the multi-task cooperation model respectively, to obtain a plurality of target latent features.

[0025] The fusion module is configured to fuse the plurality of target latent features, to obtain a target fusion latent feature.

[0026] The second input module is further configured to input the target fusion latent feature into a plurality of visual task models of the multi-task cooperation model respectively, to obtain a plurality of target task execution results.

[0027] Embodiments of the fifth aspect of the application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect or the second aspect.

[0028] Embodiments of the sixth aspect of the application provide a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method of the first aspect or the second aspect.

[0029] The technical solutions provided in the embodiments of the application have at least the following technical effects or advantages:

[0030] The application provides an image processing model training method, an image processing model training method, equipment and a storage medium. The method comprises the following steps: inputting a sample image into multiple encoders of a multi-task cooperation model to obtain multiple latent features; inputting the multiple latent features into corresponding visual task models in the multi-task cooperation model respectively to obtain multiple first task execution results; inputting a first latent feature into a second visual task model to obtain a second task execution result, the second visual task model being a visual task model corresponding to a second latent feature, the first latent feature and the second latent feature being any two of the multiple latent features, and the first latent feature and the second latent feature being different; calculating a first loss function value based on the first task execution results and the second task execution results of the multiple visual task models corresponding to each other; calculating a second loss function value based on the multiple latent features and the sample image; and adjusting model parameters of the multiple encoders based on the first loss function value and the second loss function value until a preset first training completion condition is met, and multiple target encoders are obtained. According to the embodiment of the application, the first latent feature is input into the second visual task model to obtain the second task execution result; the first loss function value is calculated based on the first task execution results and the second task execution results of the multiple visual task models corresponding to each other, so that the latent feature learned by each encoder learns the features of other visual task models, the generalization ability of each encoder is improved, the second loss function value is calculated based on the multiple latent features and the sample image, and the reconstruction quality is optimized within the given bit rate constraint.

[0031] Additional aspects and advantages will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and the appended claims or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the application. Furthermore, the drawings are not necessarily drawn to scale.

[0033] In the drawings:

[0034] Figure 1 A flow chart of an image processing model training method provided by an embodiment of the application is shown;

[0035] Figure 2 A model structure schematic diagram of a multi-task cooperation model provided by an embodiment of the application is shown;

[0036] Figure 3 A model structure schematic diagram of a de-noising model provided by an embodiment of the application is shown;

[0037] Figure 4 A flow chart of an image processing model application method provided by an embodiment of the present application is shown;

[0038] Figure 5 A model structure diagram of another multi-task cooperation model provided by an embodiment of the present application is shown;

[0039] Figure 6 A structure diagram of an image processing model training device provided by an embodiment of the present application is shown;

[0040] Figure 7 A structure diagram of an image processing model application device provided by an embodiment of the present application is shown;

[0041] Figure 8 A structure diagram of an electronic device provided by an embodiment of the present application is shown;

[0042] Figure 9 A schematic diagram of a storage medium provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be accurately conveyed to those skilled in the art.

[0044] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should be understood as their common meanings by those skilled in the art to which the present application belongs.

[0045] In view of the above background, the current related image compression technology mainly has the following problems:

[0046] Image compression and computer vision goals are inconsistent: the goals of traditional image compression and computer vision are different, the former mainly focuses on the preservation of low-level pixel signals to optimize storage and transmission efficiency, and the latter focuses on extracting semantic information from visual data. Existing image compression methods will result in insufficient semantic embedding representation of the reconstructed image at low bit rates, thereby significantly reducing the performance of downstream machine vision tasks.

[0047] 2Limitations of task-driven image compression: decompose the image feature stream into N partial image feature streams, introduce machine vision constraints in codec training, respectively bind N parts, and optimize compression performance through task-driven loss function, but these sub-image feature streams are tightly coupled with specific models or tasks, and are difficult to adapt to other tasks or models.

[0048] To solve the above technical problems, the present application provides an image processing model training method, an image processing model training method, an equipment and a storage medium. The method comprises the following steps: inputting a sample image into a plurality of encoders of a multi-task cooperation model to obtain a plurality of latent features; inputting the plurality of latent features into corresponding visual task models of the multi-task cooperation model respectively to obtain a plurality of first task execution results; inputting a first latent feature into a second visual task model to obtain a second task execution result, the second visual task model being a visual task model corresponding to a second latent feature, the first latent feature and the second latent feature being any two of the plurality of latent features, and the first latent feature and the second latent feature being different; calculating a first loss function value based on the first task execution results and the second task execution results of the plurality of visual task models respectively; calculating a second loss function value based on the plurality of latent features and the sample image; adjusting model parameters of the plurality of encoders based on the first loss function value and the second loss function value until a preset first training completion condition is met, and obtaining a plurality of target encoders. The embodiments of the present application input a first latent feature into a second visual task model to obtain a second task execution result; calculate a first loss function value based on the first task execution results and the second task execution results of the plurality of visual task models respectively, to ensure that each encoder learns the features of other visual task models, to improve the generalization ability of each encoder; calculate a second loss function value based on the plurality of latent features and the sample image, to optimize the reconstruction quality within the given bit rate constraint.

[0049] An image processing model training method according to an embodiment of the present application will be described below in conjunction with the accompanying drawings.

[0050] Referring to Figure 1 , the method specifically comprises the following steps:

[0051] S101, inputting a sample image into a plurality of encoders of a multi-task cooperation model to obtain a plurality of latent features.

[0052] Among them, the encoder is used for compressing the sample image and obtaining the compressed latent feature.

[0053] The plurality of latent features correspond to different visual task models. Since the visual task models have different focuses on the sample image, the information amount of the plurality of latent features is different.

[0054] In some embodiments, the process of obtaining multiple latent features using an encoder can be implemented as follows:

[0055] Receiving an original image as input; extracting image features using structures such as convolutional neural networks, which are gradually compressed in space; reducing the resolution of feature maps while preserving important information through pooling layers or convolutional layers; feature encoding: further encoding the extracted features into a more compact representation, usually including linear transformation of features or applying more complex transformations; quantizing and encoding the encoded features to generate compressed image features, i.e. latent features.

[0056] S102, inputting multiple latent features into corresponding visual task models of the multi-task collaborative model respectively to obtain multiple first task execution results.

[0057] Each latent feature corresponds to a visual task model, i.e. the visual task model can output accurate execution results, i.e. first task execution results, based on the corresponding latent feature.

[0058] It can be understood that the multiple latent features are compressed image features, therefore, the visual task model can include a decoder and a task processor, the encoder is used to reconstruct the sample image based on the latent feature to obtain a reconstructed image, and the task processor is used to execute the corresponding visual task based on the reconstructed image.

[0059] In some embodiments, the processing process of the encoder and the decoder can be implemented as follows: assuming E k (·) is the encoder of task k, which maps the original image to the embedding space And D k (·) is the decoder, which maps the features in the embedding space y k back to the reconstructed

[0060] S103, for the first latent feature, inputting the first latent feature into the second visual task model to obtain a second task execution result.

[0061] S104, based on the first task execution result and the second task execution result corresponding to each of the multiple visual task models, calculating a first loss function value.

[0062] The second visual task model is the visual task model corresponding to the second latent feature, the first latent feature and the second latent feature are any two latent features in the multiple latent features, and the first latent feature and the second latent feature are different.

[0063] The first loss function value can be a cross-task loss function.

[0064] Since the multi-task cooperation model includes multiple visual task models, and each visual task model corresponds to a latent feature, the multi-task cooperation model includes multiple task model groups, and each task model group includes an encoder and a visual task model.

[0065] In some embodiments, the model structure of the multi-task cooperation model is as shown in Figure 2 The multi-task cooperation model includes multiple task model groups, each of which includes an encoder, a decoder, and a task processor. The sample image passes through multiple encoders to obtain latent features including different information. The multiple latent features pass through corresponding decoders to obtain different reconstructed images. The different reconstructed images pass through corresponding task processors to obtain corresponding first task execution results.

[0066] In some embodiments, the first latent feature can be input to the visual task model of another task model group, such as a second visual task model, and the second task execution result output by the second visual task model based on the first latent feature can be obtained. By analogy, each visual task model can output a corresponding task execution result based on the latent feature corresponding to another task model group, to obtain the first task execution result and the second task execution result corresponding to each visual task model, wherein the first task execution result is the task execution result output by the visual task model based on the corresponding latent feature, and the second task execution result is the task execution result output by the visual task model based on the latent feature corresponding to another task model group.

[0067] It can be understood that, in order to determine the accuracy of the multiple visual task models, each visual task model can be trained based on the first task execution result of each visual task model and the true value corresponding to each visual task, and the first loss function value of the multi-task cooperation model can be calculated based on the difference between the first task execution result and the second task execution result corresponding to each visual task model, and the multiple encoders can be trained.

[0068] The first loss function can be represented by formula (1):

[0069]

[0070] wherein, is the first loss function, Y k is the true value of the visual task label, is the predicted label result of the reconstructed image, i.e., the first task execution result of the visual task model, and ‖·‖2 is the L2 norm. norm, is the second task execution result of the visual task model. τ is a hyperparameter for balancing the learning ability of a specific task and the generalization ability of the multiple encoders.

[0071] S105, calculating a second loss function value based on the plurality of latent features and the sample image.

[0072] The second loss function value is a rate-distortion loss function value, which is used to optimize the reconstruction quality under a given bit rate constraint.

[0073] The reconstructed image can be obtained based on the plurality of latent features, and the distortion rate of each of the plurality of encoders can be determined based on the sample image and the reconstructed image.

[0074] In some embodiments, the rate of the plurality of encoders can be determined based on the plurality of latent features, and the plurality of encoders can be trained based on the distortion rate and the rate to optimize the reconstruction quality under a given bit rate constraint.

[0075] In some embodiments, the rate of the plurality of latent images can be determined based on the way of estimating the entropy model, i.e., the entropy model calculates the probability distribution of each symbol in the latent feature, and calculates the optimal encoding length based on this. In this way, the entropy model can determine how many bits are needed on average to represent each pixel in lossless compression, thereby determining the rate of image compression.

[0076] In some embodiments, the plurality of latent features can be fused to obtain fused latent features, and the rate of the plurality of latent images can be determined based on the way of estimating the entropy model.

[0077] In some embodiments, the second loss function value can be calculated based on the plurality of latent features and the sample image, including: fusing the plurality of latent features to obtain fused latent features; inputting the fused latent features into a decoder of any visual task model to obtain a reconstructed image; calculating a distortion loss function value based on the reconstructed image and the sample image; calculating a probability distribution of statistical features of the fused latent features; and calculating the second loss function value based on the distortion loss function value and the probability distribution of the statistical features of the fused latent features.

[0078] In some embodiments, the second loss function can be represented by formula (2):

[0079]

[0080] wherein, is the second loss function, is a rate loss function, is a rate loss function, and λ is a trade-off parameter that controls the trade-off between the rate loss function and the rate loss function. is the probability distribution of the statistical features of the fused latent features, represents the average amount of information of each symbol in the fused latent features, is the reconstructed image, and x is the sample image.

[0081] S106, adjust the model parameters of the plurality of encoders based on the first loss function value and the second loss function value until a preset first training completion condition is met, to obtain a plurality of target encoders.

[0082] In some embodiments, a first target loss function value can be determined based on the first loss function value and the second loss function value, and the model parameters of the plurality of encoders can be adjusted based on the first target loss function value until a preset first training completion condition is met, to obtain a plurality of target encoders.

[0083] The first target loss function can be represented by formula (3):

[0084]

[0085] wherein, is the first target loss function, is the second loss function, is the first loss function, μ is a hyperparameter, and is used to balance between and .

[0086] The first training completion condition can be that the number of training times reaches a training time threshold, or that the target loss function value reaches a preset value, etc.

[0087] Embodiments of the present application input the first latent feature into the second visual task model to obtain a second task execution result; based on the first task execution result and the second task execution result of each of the plurality of visual task models, a first loss function value is calculated to ensure that each encoder learns the features of other visual task models, to improve the generalization ability of each encoder, and based on the plurality of latent features and the sample image, a second loss function value is calculated to optimize the reconstruction quality within a given bit rate constraint.

[0088] In some embodiments, the above method further comprises: fusing a plurality of target latent features output by the plurality of target encoders to obtain a target fusion latent feature; converting the target fusion latent feature into an aligned latent feature based on the plurality of representative features in the memory bank; inputting the aligned latent feature into a denoising model of the multi-task collaboration model to obtain a generalized latent feature; calculating a third loss function value based on the aligned latent feature and the target fusion latent feature; calculating a fourth loss function value based on the generalized latent feature and the aligned latent feature; adjusting the representative features of the memory bank and the model parameters of the denoising model based on the third loss function value and the fourth loss function value, and continuing training until a preset second training completion condition is met, to obtain a target memory bank and a target denoising model.

[0089] In some embodiments, the process of fusing the plurality of target latent features can be implemented as follows: assuming that the N target latent features are generating a target fusion latent feature by the aggregation function, where the target fusion latent feature can be represented by equation (4):

[0090]

[0091] wherein, is the kth specific task embedding representation, is the target fusion latent feature, which integrates the information of all tasks.

[0092] wherein, the representative features in the memory bank for storing latent spaces, the representative features can be obtained by encoding and converting the input data, reflecting the core information and structure of the data, and any latent feature can be represented by multiple representative features.

[0093] Therefore, the target fusion latent feature can be converted into an aligned latent feature based on the plurality of representative features in the memory bank, so as to alleviate the overfitting problem and help to obtain invariant knowledge suitable for open set scenarios.

[0094] It can be understood that the plurality of representative features in the memory bank need to be updated in time to improve the accuracy of the converted aligned latent feature. Therefore, a third loss function value can be calculated based on the difference between the aligned latent feature and the target fusion latent feature, and the representative features of the memory bank can be updated in time based on the third loss function value.

[0095] The third loss function can be represented by equation (5):

[0096]

[0097] wherein, is the third loss function, is the aligned latent feature, is the target fusion latent feature.

[0098] It can be understood that since the plurality of target encoders are trained under a given bit rate constraint, there is distortion and noise in the target latent features output by the plurality of target encoders, and therefore, the aligned latent feature needs to be input into a denoising model of the multi-task collaborative model to remove the noise in the aligned latent noise and restore the information of the aligned latent feature.

[0099] Further, a fourth loss function value can be calculated based on the generalized latent feature and the aligned latent feature, so that the representative features of the memory library and the model parameters of the denoising model can be jointly adjusted based on the third loss function value and the fourth loss function value until a preset second training completion condition is met, to obtain a target memory library and a target denoising model.

[0100] The second training completion condition can be that the number of training times reaches a training time threshold, or that the target loss function value reaches a preset value, etc.

[0101] In some embodiments, jointly adjusting the representative features of the memory library and the model parameters of the denoising model based on the third loss function value and the fourth loss function value can be implemented as follows:

[0102] Generating a second target loss function based on the third loss function and the fourth loss function, and adjusting the representative features of the memory library and the model parameters of the denoising model based on the second target loss function.

[0103] The second target loss function can be represented by formula (6):

[0104]

[0105] wherein, is the second target loss function, is the third loss function, is the fourth loss function, and η is a hyperparameter for balancing and .

[0106] In some embodiments, converting the target fusion latent feature into the aligned latent feature based on the plurality of representative features in the memory library includes: calculating distance metric values of the target fusion latent feature with respect to the plurality of representative features; performing normalization processing on the plurality of distance metric values to obtain a plurality of normalized metric values; and describing the target fusion latent feature based on the plurality of representative features and the plurality of normalized metric values corresponding to the plurality of representative features, to obtain the aligned latent feature.

[0107] The distance metric value can be a measure of proximity or matching degree between the target fusion latent feature and the plurality of representative features stored in the memory library.

[0108] The distance metric value can be represented by formula (7):

[0109]

[0110] wherein, a j is the distance metric value, Dist(·,·) is a distance metric function, is the target fusion latent feature m jfor a plurality of representative features.

[0111] After determining the distance metric values, for the convenience of calculation, the plurality of distance metric values can be normalized to obtain a plurality of normalized metric values, wherein the L distance metric values a j are normalized, and the normalized distance metric values a j and all representative features in the memory library are used to reconstruct the target fusion latent feature to obtain the aligned latent feature.

[0112] In some embodiments, the process of obtaining the aligned latent feature can be represented by a function group (8):

[0113]

[0114] wherein a j ’ is the normalized metric value, is the aligned latent feature.

[0115] Unlike traditional image compression techniques that rely on embedding representations, aligning the target fusion latent feature with the plurality of representative features can alleviate the overfitting problem and help obtain invariant knowledge suitable for open set scenarios.

[0116] In some embodiments, the aligned latent feature is input into a denoising model to obtain a generalized latent feature, including: inputting noise and the aligned latent feature into a modulation subnetwork of the denoising model; inputting the noise into a diffusion model of the denoising model; and learning the denoising capability of the diffusion model by the modulation subnetwork to output a generalized latent feature corresponding to the aligned latent feature through the denoising model.

[0117] In some embodiments, the denoising model can include a pre-trained diffusion model and a modulation subnetwork.

[0118] wherein the diffusion model construction process can be implemented as:

[0119] Gaussian noise with variance βt∈(0,1) is added to the encoded latent space y=E(x) at time t to produce a noisy latent space: wherein ∈~(0,I), αt=1-βt, and When t is large enough, the latent z t is almost a standard Gaussian distribution, and the diffusion process is the inverse process of the above process, i.e., the process from z t to z, i.e., the process of removing the noise ∈ in z t to obtain the noise-free feature z.

[0120] ​Further, the denoising ability of the diffusion model is learned by using the modulation subnetwork to output a generalization latent feature corresponding to the aligned latent feature through the denoising model.

[0121] In some embodiments, the model structure of the denoising model is as shown in Figure 3 The denoising model includes a diffusion model and a modulation subnetwork in parallel with the diffusion model, the output of the modulation subnetwork is connected to the diffusion model, and the input of the modulation subnetwork is the aligned latent feature and the noise z t The input of the diffusion model is the noise z t The output of the diffusion model is the output generalization latent feature of the denoising model.

[0122] In some embodiments, based on the generalization latent feature and the aligned latent feature, a fourth loss function value is calculated, including: calculating a noise difference value of the generalization latent feature and the aligned latent feature; and calculating the fourth loss function value based on a preset noise difference value and the noise difference value.

[0123] It can be understood that the denoising model can include multiple denoising submodels, each denoising submodel can remove part of the noise of the aligned latent feature, each denoising submodel has a corresponding preset sub-noise difference value, and the aligned latent feature will obtain a process latent feature after passing through each denoising submodel. The corresponding sub-noise difference value can be calculated based on the output feature of each denoising submodel and the input feature of the denoising submodel, and the fourth loss function value can be calculated based on the corresponding sub-noise difference values of the multiple denoising submodels and the preset sub-noise difference values.

[0124] The fourth loss function can be represented by formula (9):

[0125]

[0126] Wherein, is the fourth loss function, c is the conditional information, which can be a label or context information related to the generation task, t is the time step, which is usually used to control the addition of noise and the denoising process in the diffusion model, is the aligned latent feature, ∈ is the preset noise difference value, is the model predicted noise difference value.

[0127] It should be noted that the loss function represented by formula (9) is for the case where the denoising model has only one denoising submodel. If there are multiple denoising submodels, multiple preset noise difference values and model predicted noise difference values need to be added to formula (9) to obtain the fourth loss function.

[0128] During the adjustment process, the conditional information is empty, and the denoising model is optimized by minimizing the fourth loss function.

[0129] An image processing model training method according to an embodiment of the present application is described below in conjunction with the accompanying drawings. The method is applied to Figure 1 The multi-task cooperation model obtained by the image processing model training method shown in Figure 4 The method specifically includes the following steps:

[0130] S401, input the target image into multiple target encoders of the multi-task cooperation model respectively to obtain multiple target latent features.

[0131] S402, fuse the multiple target latent features to obtain a target fusion latent feature.

[0132] S403, input the target fusion latent feature into multiple visual task models of the multi-task cooperation model respectively to obtain multiple target task execution results.

[0133] In some embodiments, inputting the target fusion latent feature into the multiple visual task models respectively to obtain the multiple target task execution results includes: converting the target fusion latent feature into an aligned latent feature based on multiple representative features in a memory bank; inputting the aligned latent feature into a denoising model of the multi-task cooperation model to obtain a generalized latent feature; and inputting the generalized latent feature into the multiple visual task models respectively to obtain the multiple target task execution results.

[0134] In some embodiments, the structure of the multi-task cooperation model is as shown in Figure 5 The multi-task cooperation model includes multiple decoders, a fusion processor, a memory bank, a denoising model, multiple decoders, and multiple task processors.

[0135] The multiple decoders output multiple latent features based on the sample image. The multiple latent features are fused to obtain a target fusion latent feature. The target fusion latent feature is converted by multiple representative features in the memory bank to obtain an aligned latent feature. The aligned latent feature is denoised and the hidden information of the aligned latent feature is recovered by the denoising model to obtain a generalized latent feature. The generalized latent feature is input into any decoder to obtain a reconstructed image. The multiple task processors execute corresponding tasks based on the reconstructed image to obtain multiple task execution results.

[0136] The present application also provides an image processing model training device. The device is used to execute the image processing model training method provided in the above Figure 1 embodiments. As shown in Figure 6 The device includes a first input module 601, a calculation module 602, and an adjustment module 603.

[0137] The first input module 601 is configured to input a sample image into multiple encoders of a multi-task cooperation model to obtain multiple latent features.

[0138] The first input module 601 is further configured to input the plurality of potential features into corresponding visual task models of the multi-task cooperative model respectively to obtain a plurality of first task execution results.

[0139] The first input module 601 is further configured to, for a first potential feature, input the first potential feature into a second visual task model to obtain a second task execution result, the second visual task model being a visual task model corresponding to a second potential feature, the first potential feature and the second potential feature being any two potential features in the plurality of potential features, and the first potential feature being different from the second potential feature.

[0140] The calculation module 602 is configured to calculate a first loss function value based on the first task execution result and the second task execution result of each of the plurality of visual task models.

[0141] The calculation module 602 is further configured to calculate a second loss function value based on the plurality of potential features and the sample image.

[0142] The adjustment module 603 is configured to adjust model parameters of the plurality of encoders based on the first loss function value and the second loss function value until a preset first training completion condition is met to obtain a plurality of target encoders.

[0143] The application provides an image processing model training device. A plurality of latent features are obtained by inputting a sample image into a plurality of encoders of a multi-task cooperation model. A plurality of first task execution results are obtained by inputting the plurality of latent features into corresponding visual task models of the multi-task cooperation model. A second task execution result is obtained by inputting a first latent feature into a second visual task model, the second visual task model being a visual task model corresponding to a second latent feature, the first latent feature and the second latent feature being any two of the plurality of latent features, and the first latent feature and the second latent feature being different. A first loss function value is calculated based on the first task execution results and the second task execution results of the plurality of visual task models. A second loss function value is calculated based on the plurality of latent features and the sample image. The model parameters of the plurality of encoders are adjusted based on the first loss function value and the second loss function value until a preset first training completion condition is met, and a plurality of target encoders are obtained. In the embodiment, the first latent feature is input into the second visual task model to obtain the second task execution result. The first loss function value is calculated based on the first task execution results and the second task execution results of the plurality of visual task models, so that the latent feature learned by each encoder learns the features of other visual task models, the generalization ability of each encoder is improved, the second loss function value is calculated based on the plurality of latent features and the sample image, and the reconstruction quality is optimized within a given bit rate constraint.

[0144] In some embodiments, the device further comprises a fusion module and a conversion module.

[0145] The fusion module is configured to fuse a plurality of target latent features output by the plurality of target encoders to obtain a target fusion latent feature.

[0146] The conversion module is configured to convert the target fusion latent feature into an aligned latent feature based on a plurality of representative features in a memory bank.

[0147] The first input module 601 is further configured to input the aligned latent feature into a denoising model of the multi-task cooperation model to obtain a generalization latent feature.

[0148] The calculation module 602 is further configured to calculate a third loss function value based on the aligned latent feature and the target fusion latent feature.

[0149] The calculation module 602 is further configured to calculate a fourth loss function value based on the generalization latent feature and the aligned latent feature.

[0150] The adjusting module 603 is further configured to adjust the representative features of the memory bank and the model parameters of the denoising model based on the third loss function value and the fourth loss function value, continue training until a preset second training completion condition is met, and obtain a target memory bank and a target denoising model.

[0151] In some embodiments, the calculating module 602 is specifically configured to:

[0152] fuse the plurality of latent features to obtain fused latent features;

[0153] input the fused latent features into a decoder of any visual task model to obtain a reconstructed image;

[0154] calculate a distortion loss function value based on the reconstructed image and the sample image;

[0155] calculate a probability distribution of a statistical feature of the fused latent features;

[0156] calculate a second loss function value based on the distortion loss function value and the probability distribution of the statistical feature of the fused latent features.

[0157] In some embodiments, the converting module is specifically configured to:

[0158] calculate distance measurement values between the target fused latent features and the plurality of representative features, respectively;

[0159] normalize the plurality of distance measurement values to obtain a plurality of normalized measurement values;

[0160] describe the target fused latent features based on the plurality of representative features and the plurality of normalized measurement values corresponding to the plurality of representative features, respectively, to obtain aligned latent features.

[0161] In some embodiments, the first input module 601 is specifically configured to:

[0162] input noise and the aligned latent features into a modulation subnetwork of a denoising model;

[0163] input the noise into a diffusion model of the denoising model;

[0164] learn a denoising capability of the diffusion model by using the modulation subnetwork, so as to output a generalization latent feature corresponding to the aligned latent feature by the denoising model.

[0165] In some embodiments, the calculating module 603 is specifically configured to:

[0166] calculate a noise difference value between the generalization latent feature and the aligned latent feature;

[0167] The fourth loss function value is calculated based on the preset noise difference value and the noise difference value.

[0168] The embodiments of the present application also provide an image processing model application device, which is used for executing the image processing model training method and the image processing model application method. Figure 4 The embodiments provide an image processing model application method. As shown in Figure 7 The device includes a second input module 701 and a fusion module 702.

[0169] The second input module 701 is configured to input a target image into a plurality of target encoders of the multi-task cooperative model respectively to obtain a plurality of target latent features.

[0170] The fusion module 702 is configured to fuse the plurality of target latent features to obtain a target fusion latent feature.

[0171] The second input module 701 is further configured to input the target fusion latent feature into a plurality of visual task models of the multi-task cooperative model respectively to obtain a plurality of target task execution results.

[0172] In some embodiments, the second input module 701 is specifically configured to:

[0173] convert the target fusion latent feature into an aligned latent feature based on a plurality of representative features in a memory bank;

[0174] input the aligned latent feature into a denoising model of the multi-task cooperative model to obtain a generalized latent feature;

[0175] input the generalized latent feature into the plurality of visual task models respectively to obtain the plurality of target task execution results.

[0176] The embodiments of the present application also provide an electronic device for executing the image processing model training method and the image processing model application method. Please refer to Figure 8 which shows a schematic diagram of an electronic device provided by some embodiments of the present application. As shown in Figure 8 The electronic device 7 includes a processor 700, a memory 701, a bus 702 and a communication interface 703, the processor 700, the communication interface 703 and the memory 701 are connected through the bus 702; the memory 701 stores a computer program which can run on the processor 700, and the processor 700 runs the computer program to execute the image processing model training method or the image processing model application method provided by any one of the preceding embodiments of the present application.

[0177] The memory 701 can include a random access memory (RAM) and can further include a non-volatile memory, such as at least one disk memory. The communication connection between the apparatus network element and at least one other network element is realized through at least one communication interface 703 (which can be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. can be used.

[0178] The bus 702 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 701 is used to store programs, and the processor 700 executes the programs after receiving execution instructions. The image processing model training method or the image processing model application method disclosed in any of the embodiments of the present application can be applied to the processor 700 or implemented by the processor 700.

[0179] The processor 700 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by integrated logic circuits or instructions in the form of software in the processor 700. The processor 700 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-program gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 701, and the processor 700 reads the information in the memory 701, and combines the hardware to complete the steps of the above method.

[0180] The electronic device provided by the embodiments of the present application and the image processing model training method or image processing model application method provided by the embodiments of the present application have the same beneficial effects as the methods they use, run or implement.

[0181] The embodiment of the present application further provides a computer readable storage medium corresponding to the image processing model training method provided in the foregoing embodiment, please refer to Figure 9 The computer readable storage medium shown in the embodiment of the present application is an optical disc 30, and a computer program (i.e., a program product) is stored on the optical disc 30. When the computer program is run by a processor, the image processing model training method or the image processing model application method provided in any of the foregoing embodiments is executed.

[0182] It should be noted that examples of the computer readable storage medium can further include, but are not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a flash memory, or other optical or magnetic storage medium, which are not listed here.

[0183] The computer readable storage medium provided in the foregoing embodiment of the present application and the image processing model training method provided in the embodiment of the present application are based on the same inventive concept, and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0184] It should be noted that:

[0185] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some examples, well-known structures and techniques are not shown in detail in order not to obscure the understanding of the present specification.

[0186] Similarly, it should be understood that, in order to simplify the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be interpreted as reflecting an intention that the claimed application requires more features than those explicitly recited in each claim. Rather, inventive aspects are based on less than all of the features of the single embodiments disclosed above, as reflected in the claims below. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim is a separate embodiment of the present application.

[0187] Furthermore, those skilled in the art will recognize that, in the following claims, the singular Forms "a," "an," and "the" include plural Forms unless the context clearly dictates otherwise. Also, the term "another" means "at least a second" or "at least a third," unless the context clearly dictates otherwise.

[0188] The above description is only the preferred specific embodiments of the application, but the protection scope of the application is not limited to this. Any changes or replacements within the technical scope disclosed by the application can be easily thought by those skilled in the art, and should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. An image processing model training method, characterized in that, The method comprises the following steps: inputting a sample image into multiple encoders of a multi-task collaborative model to obtain multiple latent features; inputting the multiple latent features into corresponding visual task models of the multi-task collaborative model respectively to obtain multiple first task execution results; for a first latent feature, inputting the first latent feature into a second visual task model to obtain a second task execution result, the second visual task model being a visual task model corresponding to a second latent feature, the first latent feature and the second latent feature being any two of the multiple latent features, and the first latent feature and the second latent feature being different; calculating a first loss function value based on the first task execution results and the second task execution results of the multiple visual task models respectively; calculating a second loss function value based on the multiple latent features and the sample image; adjusting model parameters of the multiple encoders based on the first loss function value and the second loss function value until a preset first training completion condition is met to obtain multiple target encoders; the method further comprises the following steps: fusing the multiple target latent features output by the multiple target encoders to obtain a target fused latent feature; converting the target fused latent feature into an aligned latent feature based on multiple representative features in a memory bank; inputting the aligned latent feature into a denoising model of the multi-task collaborative model to obtain a generalized latent feature; calculating a third loss function value based on the aligned latent feature and the target fused latent feature; calculating a fourth loss function value based on the generalized latent feature and the aligned latent feature; 2. The method of claim 1, wherein, adjusting the representative features of the memory bank and the model parameters of the denoising model based on the third loss function value and the fourth loss function value, and continuing training until a preset second training completion condition is met to obtain a target memory bank and a target denoising model. the method further comprises the following steps: calculating distance measurement values of the target fused latent feature and the multiple representative features respectively; normalizing the multiple distance measurement values to obtain multiple normalized measurement values; describing the target fused latent feature based on the multiple representative features and the multiple normalized measurement values corresponding to the multiple representative features respectively to obtain an aligned latent feature. the method further comprises the following steps: inputting noise and the aligned latent feature into a modulation subnetwork of the denoising model; 3. The method of claim 2, wherein, ​ ​ ​ ​ 4. The method of claim 2, wherein, ​ ​ inputting the noise into a denoising model of the diffusion model; learning a denoising capability of the diffusion model by using the modulation sub-network, so as to output a generalization latent feature corresponding to the alignment latent feature by the denoising model.

5. The method of claim 4, wherein, The fourth loss function value is calculated based on the generalization latent feature and the alignment latent feature, including: calculating a noise difference value of the generalization latent feature and the alignment latent feature; calculating a fourth loss function value based on a preset noise difference value and the noise difference value.

6. An image processing model application method characterized by comprising: a step of applying a model to an image; and a step of outputting a result of the application of the model. The multi-task cooperation model obtained by applying the image processing model training method of any one of claims 1-5, the method comprising: inputting a target image into a plurality of target encoders of the multi-task cooperation model respectively, to obtain a plurality of target latent features; fusing the plurality of target latent features to obtain a target fusion latent feature; inputting the target fusion latent feature into a plurality of visual task models of the multi-task cooperation model respectively, to obtain a plurality of target task execution results.

7. The method of claim 6, wherein, The method further comprises: based on a plurality of representative features in a memory bank, converting the target fusion latent feature into an alignment latent feature; inputting the alignment latent feature into a denoising model of the multi-task cooperation model to obtain a generalization latent feature; inputting the generalization latent feature into a plurality of visual task models respectively, to obtain a plurality of target task execution results.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor runs the computer program to implement the method of any one of claims 1-7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-7.

Images