Training method and device of coding model and image processing method and device

Abstract

The application relates to a training method and device of a coding and decoding model, an image processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: inputting a training sample image into a coding and decoding model to obtain an encoding code stream and a decoding feature; inputting the decoding feature into a machine vision task network to obtain first feature information output by a backbone network, and inputting the training sample image into the machine vision task network to obtain second feature information output by the backbone network; iteratively training the coding and decoding model according to the first feature information, the second feature information, the encoding code stream, the decoding feature, the training sample image and a target loss function to obtain a trained coding and decoding model; and the target loss function comprises a task feature loss function used for representing the difference between the first feature information and the second feature information. The coding and decoding model obtained by the method can improve the execution accuracy of a machine vision task after processing an image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a training method and device of a coding and decoding model, an image processing method and device, a computer device, a storage medium and a computer program product. BACKGROUND

[0002] To meet the requirements of data transmission bandwidth, latency or reduce storage space, an encoder is usually used to encode image or video data to compress data, and then a decoder is used to decode the encoded code stream to reconstruct the image or video data. With the development of artificial intelligence technology, machine vision task networks appear, which are used to perform related visual analysis tasks (such as target detection, target segmentation, target tracking, etc.) according to images or videos.

[0003] However, in order to achieve the expected data compression rate, the decoded reconstructed image can only meet the visual effect, which has certain difference from the original image, resulting in that the accuracy of the task execution result obtained by the machine vision task network when performing intelligent analysis task according to the decoded reconstructed image is low. SUMMARY

[0004] Therefore, it is necessary to provide a training method, device, computer device, computer readable storage medium and computer program product of a coding and decoding model capable of improving the execution accuracy of machine vision tasks to solve the above technical problems.

[0005] In a first aspect, the present application provides a training method of a coding and decoding model. The coding and decoding model comprises an encoder and a decoder, and the method comprises:

[0006] inputting a training sample image into the coding and decoding model to obtain an encoded code stream and a decoding feature of the encoded code stream;

[0007] inputting the decoding feature into a machine vision task network to obtain first feature information output by a backbone network in the machine vision task network, and inputting the training sample image into the machine vision task network to obtain second feature information output by the backbone network;

[0008] iteratively training the coding and decoding model according to the first feature information, the second feature information, the encoded code stream, the decoding feature, the training sample image and a target loss function until a training end condition is reached to obtain a trained coding and decoding model; the target loss function comprises a task feature loss function and a coding and decoding model loss function, and the task feature loss function is used to represent the difference between the first feature information and the second feature information.

[0009] In one of the embodiments, the codec model loss function comprises an encoding bitstream loss function and a decoding feature loss function; the iterative training of the codec model according to the first feature information, the second feature information, the encoding bitstream, the decoding feature, the training sample image and the target loss function until a training end condition is reached to obtain a trained codec model comprises:

[0010] calculating a task feature loss value according to the first feature information, the second feature information and the task feature loss function;

[0011] calculating an encoding bitstream loss value according to the encoding bitstream, the training sample image and the encoding bitstream loss function;

[0012] calculating a decoding feature loss value according to the decoding feature, the training sample image and the decoding feature loss function;

[0013] performing weighted summation of the task feature loss value, the encoding bitstream loss value and the decoding feature loss value to obtain a total loss value;

[0014] iteratively training the codec model according to the total loss value until a training end condition is reached to obtain a trained codec model.

[0015] In one of the embodiments, the number of machine vision task networks is multiple, and the calculation of the task feature loss value according to the first feature information, the second feature information and the task feature loss function comprises:

[0016] for each machine vision task network, calculating a task feature loss value corresponding to the machine vision task network according to the first feature information, the second feature information and the task feature loss function corresponding to the machine vision task network;

[0017] the weighted summation of the task feature loss value, the encoding bitstream loss value and the decoding feature loss value to obtain a total loss value comprises:

[0018] performing weighted summation of the task feature loss value corresponding to each machine vision task network, the encoding bitstream loss value and the decoding feature loss value to obtain a total loss value.

[0019] In one of the embodiments, the backbone network in the machine vision task network comprises a target number of feature output layers for outputting the target number of first feature information and the target number of second feature information; and the calculation of the task feature loss value according to the first feature information, the second feature information and the task feature loss function comprises:

[0020] For each of the feature output layers, a task feature sub-loss value corresponding to the feature output layer is calculated according to the first feature information, the second feature information output by the feature output layer, and the task feature loss function.

[0021] The task feature sub-loss values corresponding to the feature output layers are weighted and averaged to obtain a task feature loss value.

[0022] In one of the embodiments, the weighted sum of the task feature loss value, the encoding code stream loss value, and the decoding feature loss value to obtain a total loss value comprises:

[0023] According to the historical task feature loss value and the historical decoding feature loss value corresponding to the historical iteration number, a task feature loss training speed and a decoding feature loss training speed are determined.

[0024] According to the task feature loss training speed, the decoding feature loss training speed, the historical task feature loss value, and the historical decoding feature loss value, a task feature loss weight, a decoding feature loss weight, and an encoding code stream loss weight are determined.

[0025] Based on the task feature loss weight, the decoding feature loss weight, and the encoding code stream loss weight, the weighted sum of the task feature loss value, the encoding code stream loss value, and the decoding feature loss value is performed to obtain a total loss value.

[0026] In one of the embodiments, according to the historical task feature loss value and the historical decoding feature loss value corresponding to the historical iteration number, a task feature loss training speed and a decoding feature loss training speed are determined, which comprises:

[0027] The first historical task feature loss value and the first historical decoding feature loss value corresponding to the last iteration number of the current iteration number, and the second historical task feature loss value and the second historical decoding feature loss value corresponding to the last iteration number of the last iteration number are determined.

[0028] According to the ratio of the first historical task feature loss value and the second historical task feature loss value, a task feature loss training speed is determined, and according to the ratio of the first historical decoding feature loss value and the second historical decoding feature loss value, a decoding feature loss training speed is determined.

[0029] In one of the embodiments, according to the task feature loss training speed, the decoding feature loss training speed, the historical task feature loss value, and the historical decoding feature loss value, a task feature loss weight, a decoding feature loss weight, and an encoding code stream loss weight are determined, which comprises:

[0030] a sum value of the task feature loss training speed and the decoding feature loss training speed is calculated, and a mean value is calculated according to a first historical task feature loss value and a first historical decoding feature loss value corresponding to a last iteration number of the current iteration number;

[0031] a task feature loss sub-weight is calculated according to the task feature loss training speed, the first historical task feature loss value, the sum value and the mean value, and a decoding feature loss sub-weight is calculated according to the decoding feature loss training speed, the first historical decoding feature loss value, the sum value and the mean value;

[0032] a task feature loss sub-weight is calculated according to the task feature loss training speed, the first historical task feature loss value, the sum value and the mean value, and a decoding feature loss sub-weight is calculated according to the decoding feature loss training speed, the first historical decoding feature loss value, the sum value and the mean value;

[0033] In a second aspect, the present application further provides an image processing method. The method comprises:

[0034] inputting a to-be-processed image into an encoder in a trained coding-decoding model to obtain an encoding code stream;

[0035] sending the encoding code stream to a receiving device;

[0036] The coding-decoding model is trained according to the training method of the coding-decoding model in the first aspect.

[0037] In a third aspect, the present application further provides another image processing method. The method comprises:

[0038] receiving an encoding code stream of a to-be-processed image, the encoding code stream being obtained by inputting the to-be-processed image into an encoder in a trained coding-decoding model;

[0039] inputting the encoding code stream into a decoder in the trained coding-decoding model to obtain a decoding feature, the decoding feature being used for a machine vision task network to perform a vision task;

[0040] The coding-decoding model is trained according to the training method of the coding-decoding model in the first aspect.

[0041] In a fourth aspect, the present application further provides a training device of a coding-decoding model. The device comprises:

[0042] a first input module configured to input a training sample image into the coding-decoding model to obtain an encoding code stream and a decoding feature of the encoding code stream;

[0043] a second input module configured to input the decoded feature into the machine vision task network to obtain first feature information output by a backbone network in the machine vision task network, and input the training sample image into the machine vision task network to obtain second feature information output by the backbone network;

[0044] a training module configured to iteratively train the codec model according to the first feature information, the second feature information, the encoding code stream, the decoded feature, the training sample image, and a target loss function until a training end condition is reached to obtain a trained codec model; the target loss function includes a task feature loss function and a codec model loss function, and the task feature loss function is configured to represent a difference between the first feature information and the second feature information.

[0045] In one of the embodiments, the codec model loss function includes an encoding code stream loss function and a decoded feature loss function; and the training module is specifically configured to:

[0046] calculate a task feature loss value according to the first feature information, the second feature information, and the task feature loss function; calculate an encoding code stream loss value according to the encoding code stream, the training sample image, and the encoding code stream loss function; calculate a decoded feature loss value according to the decoded feature, the training sample image, and the decoded feature loss function; and perform weighted summation on the task feature loss value, the encoding code stream loss value, and the decoded feature loss value to obtain a total loss value; and iteratively train the codec model according to the total loss value until a training end condition is reached to obtain a trained codec model.

[0047] In one of the embodiments, the number of the machine vision task networks is multiple, and the training module is specifically configured to:

[0048] for each of the machine vision task networks, calculate a task feature loss value corresponding to the machine vision task network according to the first feature information, the second feature information, and the task feature loss function corresponding to the machine vision task network; and perform weighted summation on the task feature loss values corresponding to the machine vision task networks, the encoding code stream loss value, and the decoded feature loss value to obtain a total loss value.

[0049] In one of the embodiments, the backbone network in the machine vision task network includes a target number of feature output layers configured to output the target number of first feature information and the target number of second feature information; and the training module is specifically configured to:

[0050] For each of the feature output layers, a task feature sub-loss value corresponding to the feature output layer is calculated according to the first feature information, the second feature information output by the feature output layer, and the task feature loss function; and the task feature sub-loss values corresponding to the feature output layers are weighted and averaged to obtain the task feature loss value.

[0051] In one of the embodiments, the training module is specifically configured to:

[0052] The task feature loss training speed and the decoding feature loss training speed are determined according to the historical task feature loss value and the historical decoding feature loss value corresponding to the historical iteration number; the task feature loss weight, the decoding feature loss weight, and the encoding code stream loss weight are determined according to the task feature loss training speed, the decoding feature loss training speed, the historical task feature loss value, and the historical decoding feature loss value; and the task feature loss value, the encoding code stream loss value, and the decoding feature loss value are weighted and summed based on the task feature loss weight, the decoding feature loss weight, and the encoding code stream loss weight to obtain the total loss value.

[0053] In one of the embodiments, the training module is specifically configured to:

[0054] The first historical task feature loss value and the first historical decoding feature loss value corresponding to the last iteration number of the last iteration number of the current iteration number, and the second historical task feature loss value and the second historical decoding feature loss value corresponding to the last iteration number of the last iteration number of the last iteration number are determined; the task feature loss training speed is determined according to the ratio of the first historical task feature loss value and the second historical task feature loss value, and the decoding feature loss training speed is determined according to the ratio of the first historical decoding feature loss value and the second historical decoding feature loss value.

[0055] In one of the embodiments, the training module is specifically configured to:

[0056] The sum of the task feature loss training speed and the decoding feature loss training speed is calculated, and the mean of the first historical task feature loss value and the first historical decoding feature loss value corresponding to the last iteration number of the last iteration number of the current iteration number is calculated; the task feature loss sub-weight is calculated according to the task feature loss training speed, the first historical task feature loss value, the sum, and the mean, and the decoding feature loss sub-weight is calculated according to the decoding feature loss training speed, the first historical decoding feature loss value, the sum, and the mean; and the task feature loss weight, the decoding feature loss weight, and the encoding code stream loss weight are determined according to the task feature loss sub-weight, the decoding feature loss sub-weight, and the adjustable weight parameter.

[0057] In a fifth aspect, the present application provides an image processing device. The device comprises:

[0058] An input module configured to input an image to be processed into an encoder of a trained codec model to obtain an encoded bitstream, wherein the trained codec model is trained according to the training method of the codec model of the first aspect.

[0059] A sending module configured to send the encoded bitstream to a receiving device.

[0060] In a sixth aspect, the present application provides another image processing device. The device comprises:

[0061] A receiving module configured to receive an encoded bitstream of an image to be processed, wherein the encoded bitstream is obtained by inputting the image to be processed into an encoder of a trained codec model.

[0062] An input module configured to input the encoded bitstream into a decoder of the trained codec model to obtain decoded features, wherein the decoded features are used for a machine vision task network to perform a vision task, and wherein the trained codec model is trained according to the training method of the codec model of the first aspect.

[0063] In a seventh aspect, the present application provides a computer device. The computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of the first aspect, the second aspect or the third aspect when executing the computer program.

[0064] In an eighth aspect, the present application provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program implements the steps of the method of the first aspect, the second aspect or the third aspect when executed by a processor.

[0065] In a ninth aspect, the present application provides a computer program product. The computer program product comprises a computer program, and the computer program implements the steps of the method of the first aspect, the second aspect or the third aspect when executed by a processor.

[0066] The training method and device of the coding and decoding model, the image processing method and device, the computer device, the storage medium and the computer program product, by inputting the decoding feature output by the coding and decoding model after coding and decoding processing of the original image (training sample image) into the machine vision task network, obtaining the first feature information corresponding to the decoding feature output by the backbone network of the machine vision task network, and inputting the original image before coding and decoding processing into the machine vision task network, obtaining the second feature information corresponding to the original image, then according to the difference between the first feature information and the second feature information, combining the coding and decoding model loss (the difference between the coding bitstream BPP and the decoding feature and the original image), jointly training the coding and decoding model until the training end condition is reached to obtain the trained coding and decoding model. In the method, the coding and decoding model is jointly trained according to the difference between the first feature information extracted by the machine vision task network from the decoding feature and the second feature information extracted from the original image, not only considering the coding and decoding model loss, but also reducing the loss of the decoding feature after coding and decoding and the feature information required by the machine vision task network in the original image, effectively preserving the necessary features required by the machine vision task network to execute the vision task. Therefore, the coding and decoding model trained by the method can improve the execution accuracy of the machine vision task in the application scenario of the machine vision task. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 A flowchart of a training method of a coding and decoding model in an embodiment is shown;

[0068] Figure 2 A flowchart of a training process of a coding and decoding model in an embodiment is shown;

[0069] Figure 3 A flowchart of a training method of a coding and decoding model in an embodiment is shown;

[0070] Figure 4 A flowchart of a training process of a coding and decoding model in an embodiment is shown;

[0071] Figure 5 A flowchart of a training process of a coding and decoding model in an embodiment is shown;

[0072] Figure 6 A flowchart of a training process of a coding and decoding model in an embodiment is shown;

[0073] Figure 7 A flowchart of a training process of a coding and decoding model in an embodiment is shown;

[0074] Figure 8 A flowchart of a training process of a coding and decoding model in an embodiment is shown;

[0075] Figure 9 A flowchart of a training process of a coding and decoding model in an embodiment is shown;

[0076] Figure 10 a structural block diagram of a training device of a coding and decoding model in an embodiment;

[0077] Figure 11 a structural block diagram of an image processing device in an embodiment;

[0078] Figure 12 a structural block diagram of an image processing device in another embodiment;

[0079] Figure 13 an internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0080] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0081] First, before specifically introducing the technical solutions of the embodiments of the present application, the technical background or technical evolution context based on which the embodiments of the present application are introduced. With the development of technology, more and more application scenarios (such as automatic driving, smart city, industrial quality inspection, etc.) involve using machine vision task networks to perform intelligent analysis (such as target detection, target segmentation, target tracking, etc.) on images or videos. In order to meet the requirements of data transmission bandwidth, delay or reduce storage space, etc., the coding bitstream of the image usually needs to achieve an expected (relatively high) compression rate to reduce the amount of data transmission. However, the decoded reconstructed image (compressed image) decoded from the coding bitstream with a relatively high compression rate usually only meets the visual effect (facing human vision) and has certain differences (certain information loss) from the original image. Using the decoded reconstructed image for the machine vision task network to perform intelligent analysis tasks is likely to result in a lower accuracy of the task execution result.

[0082] Based on this background, the applicant proposes a training method of a coding and decoding model through long-term research and development and experimental verification. By combining the differences between the first feature information extracted from the decoded features by the machine vision task network and the second feature information extracted from the original image to jointly train the coding and decoding model, the loss of the decoded features after coding and decoding and the feature information required by the machine vision task network in the original image can be reduced, and the necessary features required by the machine vision task network to perform visual tasks are effectively retained. Therefore, the coding and decoding model trained by the present method can improve the execution accuracy of the machine vision task in the application scenario facing the machine vision task.

[0083] It should be noted that the beneficial effects brought about by the embodiments of the present application or the technical problems solved are not limited to this, but also other implicit or related problems, which can be seen from the description of the following embodiments.

[0084] Before introducing the specific embodiments of the present application, the professional terms involved in the present application are explained:

[0085] Compression: reduce the data size by some technical means.

[0086] Decompression: restore the data of reduced data size to the original form and size by some technical means.

[0087] Encoding: a specific technical means of compression.

[0088] Decoding: a specific means of decompression.

[0089] Encoding bit rate (BPP, Bits Per Pixel) can be calculated by dividing the number of bits occupied by the compressed code stream (encoding code stream) by the number of pixel points of the original image (image height * width (H*W)), or by multiplying the number of bytes occupied by the compressed code stream (file size) by 8, and then dividing by the number of pixel points of the original image (H*W). The encoding bit rate can reflect the compression ratio.

[0090] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0091] In one embodiment, as shown in Figure 1 A method for training a codec model is provided, which can be applied to a computer device. The computer device can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, portable wearable devices, servers, etc. In this embodiment, the method includes the following steps:

[0092] Step 101, input the training sample image into the codec model to obtain the encoding code stream and the decoding feature of the encoding code stream.

[0093] In implementation, the training sample image can be an image related to a machine vision intelligent analysis task, such as an image / video data set that can be established from a machine vision task such as target detection, target segmentation, target tracking, etc. The training sample image can also be other images or video frames collected. It should be noted that the training sample image can be an image related to a visual task performed by the machine vision task network, but it is not used to train the machine vision task network, and therefore does not need to contain the annotation information required to train the machine vision task network.

[0094] The coding and decoding model includes an encoder and a decoder, and can adopt an end-to-end artificial intelligence (AI) model, such as an end-to-end AI coding and decoding model such as Cheng2020, Mbt2018-mean, etc. The batch size during training of the model can be set to 16.

[0095] The computer device can input the training sample image to the coding and decoding model to be trained, encode the training sample image (original image input) through the encoder, and output an encoded code stream. The encoded code stream output by the encoder can then be decoded through the decoder to output a decoding feature. Thus, the computer device can obtain the encoded code stream and the decoding feature of the encoded code stream. It can be understood that the decoding feature can be used to generate a decoding reconstruction image.

[0096] Step 102, input the decoding feature to the machine vision task network to obtain first feature information output by a backbone network in the machine vision task network, and input the training sample image to the machine vision task network to obtain second feature information output by the backbone network.

[0097] In implementation, the machine vision task network refers to a network (or model) used to perform visual intelligent analysis tasks according to images or videos, which can be but is not limited to a target detection task network, a target segmentation task network, and a target tracking task network, etc. The machine vision task network can adopt a convolutional neural network such as LeNet, AlexNet, VGGNet, ResNet, etc., and generally includes a backbone network and a head network. Optionally, the machine vision task network can be a network that has been trained for a visual task.

[0098] The computer device can input the decoding feature corresponding to the training sample image into the machine vision task network, perform feature extraction on the decoding feature through a backbone network in the machine vision task network, and output first feature information. Thus, the computer device can obtain the first feature information corresponding to the decoding feature. In addition, the computer device can input the training sample image into the machine vision task network, perform feature extraction on the training sample image through the backbone network, and obtain second feature information output by the backbone network.

[0099] It can be understood that the machine vision task network can first perform preprocessing on the input training sample image, and then input the preprocessed data into the backbone network for feature extraction to output the second feature information.

[0100] It can be understood that the feature information output by the backbone network can be used for further processing by other network structures (such as a neck network, a head network, etc.) in the machine vision task network to output a vision task execution result.

[0101] Step 103: iteratively training the coding and decoding model according to the first feature information, the second feature information, the encoding code stream, the decoding feature, the training sample image, and a target loss function until a training end condition is reached to obtain a trained coding and decoding model.

[0102] The target loss function includes a task feature loss function and a coding and decoding model loss function. The task feature loss function is used to represent the difference between the first feature information and the second feature information. The coding and decoding model loss function can be the same as the loss function used in the traditional coding and decoding model training process, and can generally include an encoding code stream loss function representing the encoding code stream bit rate BPP, and a decoding feature loss function representing the difference between the decoding feature of the input image (the training sample image) and the input image.

[0103] In implementation, the computer device can calculate a loss value according to the first feature information, the second feature information, the encoding code stream, the decoding feature, the training sample image, and the target loss function. The loss value can reflect the comprehensive loss of the difference between the first feature information and the second feature information, the encoding code stream bit rate BPP, and the difference between the decoding feature and the input image. Then, the computer device can update the network parameters (network weights) of the coding and decoding model according to the loss value by using a gradient back propagation algorithm, so that the loss value calculated in the next iteration training is reduced. The iteration training is performed in this way until a training end condition (such as the target loss function converges or the number of iterations reaches a preset value) is reached, and then the training is stopped to obtain the trained coding and decoding model. It should be noted that in the process of training the coding and decoding model, only the network parameters of the coding and decoding model are updated, and the parameters of the machine vision task network are not updated.

[0104] In the training method of the coding and decoding model, the decoding feature output by the coding and decoding model after the coding and decoding processing of the original image (training sample image) is input into the machine vision task network to obtain the first feature information corresponding to the decoding feature output by the backbone network of the machine vision task network, and the original image before the coding and decoding processing is input into the machine vision task network to obtain the second feature information corresponding to the original image. Then, the coding and decoding model is jointly trained according to the difference between the first feature information and the second feature information, combined with the coding and decoding model loss (the encoding code stream BPP and the difference between the decoding feature and the original image), until a trained coding and decoding model is obtained when the training end condition is reached. In this method, the coding and decoding model is jointly trained by combining the difference between the first feature information extracted by the machine vision task network from the decoding feature and the second feature information extracted by the machine vision task network from the original image. Not only the coding and decoding model loss is considered, but also the loss of the decoding feature (or the decoding reconstructed image obtained further) in the machine vision task is considered, that is, the performance of the decoding feature (or the decoding reconstructed image obtained further) in the machine vision task is considered, and the necessary features required by the machine vision task network to perform the vision task are effectively preserved. Therefore, the coding and decoding model trained by the method can achieve the effect of considering both high compression rate and execution accuracy of the machine vision task in the application scenario of the machine vision task.

[0105] In addition, in this method, the decoding feature is input into the machine vision task network for jointly training the coding and decoding model. Compared with inputting the decoding reconstructed image into the machine vision task network to jointly train the coding and decoding model, the decoding feature can record the training process information, the back propagation in the training process can be continuous, and the gradient descent direction of the coding and decoding model is affected by the loss of the output feature of the machine vision task network, which is more conducive to improving the model training effect, including improving the training efficiency, improving the adaptability of the coding and decoding model and the machine vision task network, and the like.

[0106] It should be noted that the coding and decoding model trained by the method can improve the accuracy of the machine vision task network in performing a visual analysis task under the same image compression rate, but the coding and decoding model trained by the method is not limited to being applied to the machine vision scene, and can also be applied to the human vision or human-machine hybrid vision scene, and the application does not limit the application scene.

[0107] In one embodiment, the coding and decoding model loss function includes an encoding code stream loss function and a decoding feature loss function. As shown in Figure 2 The process of training the coding and decoding model in step 103 specifically includes the following steps:

[0108] Step 201, according to the first feature information, the second feature information and the task feature loss function, the task feature loss value is calculated.

[0109] In implementation, the task feature loss function can adopt a mean squared error loss function (MSE). The computer device can calculate the mean squared error of the first feature information and the second feature information to obtain the task feature loss value (which can be recorded as MSE LossG).

[0110] Step 202, according to the encoding code stream, the training sample image and the encoding code stream loss function, the encoding code stream loss value is calculated.

[0111] In implementation, the encoding code stream loss function can represent the encoding code stream bit rate BPP. The computer device can calculate the encoding code stream BPP according to the encoding code stream and the training sample image as the encoding code stream loss value (which can be recorded as Bpp Loss).

[0112] Step 203, according to the decoding feature, the training sample image and the decoding feature loss function, the decoding feature loss value is calculated.

[0113] In implementation, the decoding feature loss function can adopt a mean squared error loss function MSE. The computer device can calculate the mean squared error of the decoding feature and the training sample image to obtain the decoding feature loss value (which can be recorded as MSE Loss1)

[0114] Step 204, the task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain the total loss value.

[0115] In implementation, the computer device can weight and sum the task feature loss value MSE LossG, the encoding code stream loss value BppLoss and the decoding feature loss value MSE Loss1 to obtain the total loss value (which can be recorded as L). The weights corresponding to each loss value can be pre-set or calculated according to other methods.

[0116] In step 205, the codec model is iteratively trained according to the total loss value until a training end condition is reached to obtain the trained codec model.

[0117] In implementation, the computer device can update the network parameters of the codec model using the gradient back propagation algorithm according to the loss value L, and iteratively train until a training end condition (such as the target loss function converges or the number of iterations reaches a preset value) is reached, and then stop training to obtain the trained codec model.

[0118] In this embodiment, the total loss value obtained according to the task feature loss value, the encoding code stream loss value and the decoding feature loss value is used to train the codec model. The total loss value reflects the difference between the features extracted by the machine vision task network from the decoding features and the features extracted from the original image, and the comprehensive situation of the encoding code stream BPP (reflecting the compression rate) of the codec model and the difference between the decoding features and the original image, which is used to jointly train the codec model. The codec model can retain the key information for machine vision tasks after encoding and decoding, and thus can balance high compression rate and accurate execution result of machine vision tasks.

[0119] In one embodiment, the number of machine vision task networks is multiple. The process of calculating the task feature loss value in step 201 specifically includes: for each machine vision task network, calculating the task feature loss value corresponding to the machine vision task network according to the first feature information, the second feature information and the task feature loss function corresponding to the machine vision task network. Correspondingly, the specific process of calculating the total loss value in step 204 includes: weighting and summing the task feature loss values corresponding to each machine vision task network, the encoding code stream loss value and the decoding feature loss value to obtain the total loss value.

[0120] In implementation, in order to make the codec model applicable to multiple machine vision task application scenarios, the task feature loss values corresponding to multiple machine vision task networks can be jointly trained to jointly train the codec model. As shown in Figure 3 The decoding features output by the decoder (the dimension can be (B, 3, H, W), B is the batch size of training, and (H, W) is the size of the input image) can be input into multiple machine vision task networks respectively to obtain the first feature information output by the backbone network of each machine vision task network, and each training sample image (original image) can be input into multiple machine vision task networks respectively to obtain the second feature information output by the backbone network of each machine vision task network. Figure 3It is understood that in other examples, the types and number of machine vision task networks can be different, and the present application does not limit the types and number of machine vision task networks. Then, for each machine vision task network, the computer device can calculate a task feature loss value (which can be denoted as MSE LossGi, i represents the i-th machine vision task network) corresponding to the machine vision task network according to the first feature information, the second feature information and the task feature loss function (such as the mean square error loss function MSE) corresponding to the machine vision task network. Subsequently, the computer device can perform weighted summation on the task feature loss values MSE LossGi (i = 1, 2, …, n, n is the total number of machine vision task networks) corresponding to each machine vision task network, the encoding code stream loss value Bpp Loss and the decoding feature loss value MSE Loss1 to obtain a total loss value Loss.

[0121] In the present embodiment, the task feature loss of multiple machine vision task networks can be combined to jointly train the encoding and decoding model, so that the encoding and decoding model obtained by training has high adaptability to different intelligent vision tasks and is more widely applicable.

[0122] In one embodiment, the backbone network in the machine vision task network includes a target number of feature output layers for outputting a target number of first feature information and a target number of second feature information. As shown in Figure 4 As shown in FIG. 4, the process of calculating the task feature loss value in step 401 specifically includes the following steps:

[0123] Step 401, for each feature output layer, a task feature sub-loss value corresponding to the feature output layer is calculated according to the first feature information, the second feature information and the task feature loss function output by the feature output layer.

[0124] In implementation, the backbone network backbone of part of the machine vision task networks can contain one feature output layer, such as the machine vision task network using the ResNet50 network structure. The backbone network backbone of part of the machine vision task networks can contain multiple feature output layers, such as Figure 5The machine vision task network (only the backbone part is shown in the figure) shown using the ResNet50+FPN (feature pyramid networks) network structure includes five feature output layers. In the case of including multiple feature output layers, after the decoding feature is input into the machine vision task network, the first feature information (which can be denoted as Pj, j represents the jth feature output layer) output by each feature output layer of the backbone network can be obtained. After the training sample image is input into the machine vision task network, the second feature information (which can be denoted as P’j) output by each feature output layer of the backbone network can be obtained. For each feature output layer, the computer device can calculate the task feature sub-loss value (which can be denoted as MSE LossGj) corresponding to each feature output layer according to the first feature information Pj, the second feature information P’j output by the feature output layer, and the task feature loss function.

[0125] Step 402, the task feature sub-loss value corresponding to each feature output layer is weighted and averaged to obtain the task feature loss value.

[0126] In implementation, the computer device can weight and average the task feature sub-loss values MSE LossGj corresponding to each feature output layer to obtain the task feature loss value MSE LossG. In one example, the calculation formula of the task feature loss value MSE LossG is as follows:

[0127]

[0128] Wherein, m represents the total number of feature output layers, j = 1, 2, …, m.

[0129] In this embodiment, if the backbone network in the machine vision task network includes multiple feature output layers, the task feature sub-loss value corresponding to each feature output layer can be calculated respectively, and then weighted and averaged to obtain the task feature loss value corresponding to the machine vision task network. When the encoding and decoding model is jointly trained based on the loss value, the image processed by the encoding and decoding model can take into account both the high compression ratio and the execution performance of the machine vision task, and the encoding and decoding model can be applied to machine vision task networks of various network structures, and the application scenarios are widely applicable.

[0130] In one embodiment, as Figure 6 shown, the process of calculating the total loss value in step 204 specifically includes the following steps:

[0131] Step 601, determining the task feature loss training speed and the decoding feature loss training speed according to the historical task feature loss value and the historical decoding feature loss value corresponding to the historical iteration number.

[0132] In implementation, when calculating the total loss value (denoted as L(t)) of the current iteration training (denoted as the tth iteration), the computer device can obtain the historical task feature loss value and the historical decoding feature loss value corresponding to the historical iteration number, such as the task feature loss value (denoted as MSE LossG(t-1)) and the decoding feature loss value (denoted as MSE Loss1(t-1)) calculated in the previous iteration. The task feature loss training speed can be used to indicate the descending speed of the task feature loss value. The decoding feature loss training speed can be used to indicate the descending speed of the decoding feature loss value. The computer device can calculate the descending speed of the task feature loss value according to the historical task feature loss value corresponding to the historical iteration number, and take the descending speed of the task feature loss value as the task feature loss training speed (denoted as r G (), which represents the training speed used in the tth iteration). In addition, the computer device can calculate the descending speed of the decoding feature loss value according to the historical decoding feature loss value corresponding to the historical iteration number, and take the descending speed of the decoding feature loss value as the decoding feature loss training speed (denoted as r1()).

[0133] It can be understood that, in each iteration training, the task feature loss training speed and the decoding feature loss training speed are calculated according to the historical task feature loss value and the historical decoding feature loss value corresponding to the historical iteration number before the current iteration number, and thus the task feature loss training speed and the decoding feature loss training speed used in each iteration can be different.

[0134] It can be understood that, if the number of machine vision task networks is multiple, the task feature loss training speed (denoted as r Gi () corresponding to each machine vision task network can be calculated according to the historical task feature loss value MSE LossGi(t-x) corresponding to each machine vision task network, where i represents the ith machine vision task network.

[0135] Step 602, determining the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight according to the task feature loss training speed, the decoding feature loss training speed, the historical task feature loss value and the historical decoding feature loss value.

[0136] In implementation, the computer device can calculate the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight according to the task feature loss training speed, the decoding feature loss training speed, the historical task feature loss value (such as the task feature loss value MSE LossGi(t-1) calculated in the last iteration) and the historical decoding feature loss value (such as the decoding feature loss value MSE Loss1(t-1) calculated in the last iteration). The task feature loss weight is negatively correlated with the task feature loss training speed and the historical task feature loss value, that is, the faster the task feature loss training speed (the faster the task feature loss value decreases), the smaller the task feature loss weight, and the larger the historical task feature loss value, the smaller the task feature loss weight. The decoding feature loss weight is negatively correlated with the decoding feature loss training speed and the historical decoding feature loss value. The encoding code stream loss weight can be pre-set to a variable interval or a variable range, and in each training, the task feature loss weight and the decoding feature loss weight are adaptively adjusted together.

[0137] In step 603, based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight, the task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value.

[0138] In implementation, the computer device can calculate the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight according to the task feature loss training speed, the decoding feature loss training speed, the historical task feature loss value (such as the task feature loss value MSE LossGi(t-1) calculated in the last iteration) and the historical decoding feature loss value (such as the decoding feature loss value MSE Loss1(t-1) calculated in the last iteration). The task feature loss weight is negatively correlated with the task feature loss training speed and the historical task feature loss value, that is, the faster the task feature loss training speed (the faster the task feature loss value decreases), the smaller the task feature loss weight, and the larger the historical task feature loss value, the smaller the task feature loss weight. The decoding feature loss weight is negatively correlated with the decoding feature loss training speed and the historical decoding feature loss value. The encoding code stream loss weight can be pre-set to a variable interval or a variable range, and in each training, the task feature loss weight and the decoding feature loss weight are adaptively adjusted together.

[0139] In this embodiment, the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight corresponding to the current iteration number are calculated by training the task feature loss training speed, the task feature loss training speed, the decoding feature loss training speed, the historical task feature loss value and the historical decoding feature loss value according to the task feature loss, so as to calculate the total loss value. Thus, the weight of each loss value is adaptively adjusted in each iteration, so that the weight corresponding to the loss value with a faster loss reduction speed is smaller, and the weight corresponding to the loss value with a slower loss reduction speed is larger in the task feature loss and the decoding feature loss of each machine vision task network, and the orders of magnitude of each weight are the same (to avoid affecting the training effect due to a large difference in orders of magnitude), so that the parameter adjustment based on the total loss value can make the loss reduction speed more balanced, that is, the training speed of each task (including the feature loss of the machine vision task and the encoding and decoding task) is balanced, so that the performance of the trained encoding and decoding model in each machine vision task and encoding and decoding task is balanced, and thus the model can be applied to various machine vision, human vision or human-machine hybrid vision application scenarios, and has stronger universality.

[0140] In one embodiment, as shown in FIG. 6, Figure 7 the process of calculating the task feature loss training speed and the decoding feature loss training speed in step 601 specifically includes the following steps:

[0141] In step 701, the first historical task feature loss value and the first historical decoding feature loss value corresponding to the last iteration number of the current iteration number, and the second historical task feature loss value and the second historical decoding feature loss value corresponding to the last iteration number of the last iteration number are determined.

[0142] In implementation, the computer device can store the task feature loss value and the decoding feature loss value calculated in each iteration in the database. Thus, the computer device can obtain the first historical task feature loss value and the first historical decoding feature loss value corresponding to the last iteration number (the t-1th iteration), and the second historical task feature loss value and the second historical decoding feature loss value corresponding to the last iteration number (the t-2th iteration) from the database.

[0143] In step 702, the task feature loss training speed is determined according to the ratio of the first historical task feature loss value and the second historical task feature loss value, and the decoding feature loss training speed is determined according to the ratio of the first historical decoding feature loss value and the second historical decoding feature loss value.

[0144] In implementation, the computer device can calculate the ratio of the first historical task feature loss value and the second historical task feature loss value to obtain the task feature loss training speed, and calculate the ratio of the first historical decoding feature loss value and the second historical decoding feature loss value to obtain the decoding feature loss training speed.

[0145] In one example, when there are multiple machine vision task networks, the calculation formula of the task feature loss training speed corresponding to each machine vision task network is as follows:

[0146]

[0147] wherein rGi(t) represents the task feature loss training speed corresponding to the i-th machine vision task network using the current iteration number (the t-th time). MSE LossGi(t-1) represents the task feature loss value corresponding to the i-th machine vision task network calculated in the (t-1)-th (last) iteration. MSE LossGi(t-2) represents the task feature loss value corresponding to the i-th machine vision task network calculated in the (t-2)-th (last but one) iteration. It can be known that the smaller r is, the faster the training speed is.

[0148] In one example, the calculation formula of the decoding feature loss training speed is as follows:

[0149]

[0150] wherein r1(t) represents the decoding feature loss training speed using the current iteration number (the t-th time). MSE Loss1(t-1) represents the decoding feature loss value calculated in the (t-1)-th (last) iteration. MSE LossGi(t-2) represents the decoding feature loss value calculated in the (t-2)-th (last but one) iteration.

[0151] In the present embodiment, the ratio of the task feature loss values of the last iteration and the last but one iteration and the ratio of the decoding feature loss values can reflect the task feature loss descending speed and the decoding feature loss descending speed, and thus they are respectively taken as the task feature loss training speed and the decoding feature loss training speed for calculating the weights of various loss values, which can make the descending speeds of various loss values more balanced after adjusting the network parameters based on the total loss value, so that the performance of the trained encoding and decoding model in various machine vision tasks is more balanced, and the applicable scenarios are more extensive.

[0152] In an embodiment, the process of determining the loss weights of the categories in step 602 specifically includes the following steps: calculating a sum value of the task feature loss training speed and the decoding feature loss training speed, and calculating a mean value according to a first historical task feature loss value corresponding to a last iteration number of the current iteration number and a first historical decoding feature loss value; calculating a task feature loss sub-weight according to the task feature loss training speed, the first historical task feature loss value, the sum value and the mean value, and calculating a decoding feature loss sub-weight according to the decoding feature loss training speed, the first historical decoding feature loss value, the sum value and the mean value; and determining the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight according to the task feature loss sub-weight, the decoding feature loss sub-weight and an adjustable weight parameter.

[0153] In implementation, the computer device can calculate a sum value of the task feature loss training speed and the decoding feature loss training speed. Wherein, the sum value can be directly calculated according to the task feature loss training speed and the decoding feature loss training speed, or the sum value can be calculated according to an exponential of the task feature loss training speed and an exponential of the decoding feature loss training speed.

[0154] In an example, the calculation formula of the sum value (which can be denoted as sumR) of the task feature loss training speed and the decoding feature loss training speed can be as follows:

[0155]

[0156] Wherein, wherein rGi(t) represents a task feature loss training speed corresponding to the i th machine vision task network used at the current iteration number (t th time), and r1(t) represents a decoding feature loss training speed used at the current iteration number (t th time).

[0157] And the computer device can calculate a sum value according to a first historical task feature loss value (which can be a first historical task feature loss value corresponding to each of the plurality of machine vision task networks) corresponding to a last iteration number of the current iteration number and a first historical decoding feature loss value, and divide the sum value by a total number of the first historical task feature loss value (which can be a plurality) and the first historical decoding feature loss value to obtain a mean value.

[0158] In an example, the calculation formula of the mean value (which can be denoted as meanL) of the first historical task feature loss value and the first historical decoding feature loss value can be as follows:

[0159]

[0160] Wherein, i represents the i th machine vision task network, i = 1, 2, …, n, and n represents the number of machine vision task networks (i.e. the number of first historical task feature loss values).

[0161] Then, the computer device can calculate the task feature loss sub-weight (which can be denoted as W Gi (t)) according to the task feature loss training speed, the first historical task feature loss value, the sum R and the mean L, and calculate the decoding feature loss sub-weight (which can be denoted as W1(t)) according to the decoding feature loss training speed, the first historical decoding feature loss value, the sum R and the mean L.

[0162] In one example, the task feature loss sub-weight W Gi (t) can be calculated by the following formula:

[0163]

[0164] The decoding feature loss sub-weight W1(t) can be calculated by the following formula:

[0165]

[0166] Wherein, T and K are variable constants.

[0167] After that, the computer device can determine the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight according to the task feature loss sub-weight, the decoding feature loss sub-weight and the adjustable weight parameter. Then, the total loss value Loss(t) can be calculated based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight.

[0168] In one example, the calculation formula of the total loss value Loss(t) is as follows:

[0169] Loss(t) = λ(W Gi (t) × MSE LossGi(t) + W1(t) × MSE Loss1(t)) + Bpp Loss(t)

[0170] Wherein, W Gi (t) represents the task feature loss sub-weight corresponding to the i-th machine vision task network, i = 1, 2, …, n, and n is the total number of machine vision task networks; MSE LossGi(t) represents the task feature loss value corresponding to the i-th machine vision task network, ∑W Gi (t) × MSE LossGi(t) represents the weighted sum of the task feature loss values corresponding to each machine vision task network. W1(t) represents the decoding feature loss sub-weight. λ represents the adjustable weight parameter, which can reflect the encoding code stream loss weight, and the size of λ will affect the model compression ratio. λ × W Gi (t) can reflect the task feature loss weight corresponding to the i-th machine vision task network. λ × W1(t) can reflect the decoding feature loss weight.

[0171] The application further provides an application method of the coding and decoding model trained by the above embodiments. In one embodiment, as shown in Figure 8 The application further provides an image processing method, which can be applied to a computer device (or an electronic device) and includes the following steps:

[0172] Step 801: inputting a to-be-processed image into an encoder in the trained coding and decoding model to obtain an encoding code stream.

[0173] In implementation, the to-be-processed image can be an image collected by a computer device in real time through a shooting device, and can be used for a receiving device (such as a cloud server) or other computer devices to perform a machine vision task. The coding and decoding model can be trained according to the training method of the coding and decoding model provided in the above embodiments. The computer device can input the collected to-be-processed image into the encoder in the trained coding and decoding model, encode the to-be-processed image through the encoder, and output an encoding code stream.

[0174] Step 802: sending the encoding code stream to a receiving device.

[0175] In implementation, the computer device can send the encoding code stream corresponding to the to-be-processed image to the receiving device. The receiving device can be a cloud server or other computer devices, and can communicate with the computer device to receive the encoding code stream of the to-be-processed image sent by the computer device. The receiving device can input the encoding code stream into a decoder in the trained coding and decoding model to obtain a decoding feature (which can further obtain a decoding reconstructed image), and the decoding feature or the decoding reconstructed image can be used for a machine vision task network to perform a machine vision task, or can be used for a human vision or a human-machine hybrid vision task.

[0176] In the above image processing method, the to-be-processed image is encoded by the coding and decoding model to obtain an encoding code stream, and then the encoding code stream is sent to a receiving device. In the training process of the coding and decoding model, the differences between the features extracted from the decoding feature and the original image by the machine vision task network are combined to jointly train, so that the compressed image data (including the encoding code stream, the decoding feature, or the decoding reconstructed image) can effectively retain the necessary features in the original image for the machine vision task network to perform a vision task. Therefore, the to-be-processed image is encoded, which can meet the data transmission bandwidth and latency requirements, and the encoded data can improve the accuracy of the execution result of the vision task when used to perform the machine vision task.

[0177] In one embodiment, as shown in Figure 9 The application further provides another image processing method, which can be applied to a receiving device, and the receiving device can be a computer device. In this embodiment, the method includes the following steps:

[0178] Step 901, receiving an encoding code stream of a to-be-processed image.

[0179] In implementation, other computer devices or electronic devices can input the to-be-processed image into the encoder of the trained coding-decoding model to obtain the encoding code stream, and then send the encoding code stream to the receiving device. Thus, the receiving device can receive the encoding code stream of the to-be-processed image. The coding-decoding model is trained according to the training method of the coding-decoding model provided in the above embodiments.

[0180] Step 902, inputting the encoding code stream into the decoder of the trained coding-decoding model to obtain decoding features.

[0181] In implementation, the receiving device can input the encoding code stream into the decoder of the trained coding-decoding model, decode the encoding code stream by the decoder, and obtain decoding features. The receiving device can directly input the decoding features into the machine vision task network to execute the vision task, or obtain a decoding reconstructed image according to the decoding features, and execute the vision task by the machine vision task network.

[0182] In the above image processing method, after the receiving device receives the encoding code stream of the to-be-processed image, the to-be-processed image can be decoded by the decoder of the coding-decoding model to obtain decoding features, which are used for the machine vision task network to execute the vision task. In the training process of the coding-decoding model, the difference between the features extracted from the decoding features and the original image by the machine vision task network is combined for joint training, so that the compressed image data (including the encoding code stream, the decoding features, or the decoding reconstructed image) can effectively retain the necessary features in the original image for the machine vision task network to execute the vision task. Therefore, the to-be-processed image is encoded, which can meet the requirements of data transmission bandwidth and time delay, and then the encoding code stream is decoded to obtain the decoding features, which are used for executing the machine vision task, and the accuracy of the execution result of the vision task can be improved.

[0183] It should be understood that, although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0184] Based on the same inventive concept, the embodiment of the present application also provides a training device for the coding and decoding model of the training method of the coding and decoding model. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more coding and decoding model training device embodiments provided below can refer to the limitations of the coding and decoding model training method in the foregoing, which will not be repeated here.

[0185] In one embodiment, as shown in Figure 10 A training device 1000 for a coding and decoding model is provided, comprising: a first input module 1001, a second input module 1002, and a training module 1003, wherein:

[0186] The first input module 1001 is configured to input a training sample image into the coding and decoding model to obtain an encoded code stream and a decoding feature of the encoded code stream.

[0187] The second input module 1002 is configured to input the decoding feature into a machine vision task network to obtain first feature information output by a backbone network in the machine vision task network, and input the training sample image into the machine vision task network to obtain second feature information output by the backbone network.

[0188] The training module 1003 is configured to iteratively train the coding and decoding model according to the first feature information, the second feature information, the encoded code stream, the decoding feature, the training sample image, and a target loss function until a training end condition is reached to obtain a trained coding and decoding model; the target loss function comprises a task feature loss function and a coding and decoding model loss function, and the task feature loss function is used to represent the difference between the first feature information and the second feature information.

[0189] In one embodiment, the coding and decoding model loss function comprises an encoded code stream loss function and a decoding feature loss function. The training module 1003 is specifically configured to: calculate a task feature loss value according to the first feature information, the second feature information, and the task feature loss function; calculate an encoded code stream loss value according to the encoded code stream, the training sample image, and the encoded code stream loss function; calculate a decoding feature loss value according to the decoding feature, the training sample image, and the decoding feature loss function; perform weighted summation on the task feature loss value, the encoded code stream loss value, and the decoding feature loss value to obtain a total loss value; and iteratively train the coding and decoding model according to the total loss value until a training end condition is reached to obtain a trained coding and decoding model.

[0190] In one of the embodiments, the number of machine vision task networks is multiple. The training module 1003 is specifically configured to: for each machine vision task network, calculate a task feature loss value corresponding to the machine vision task network according to the first feature information, the second feature information corresponding to the machine vision task network, and the task feature loss function; and perform weighted summation on the task feature loss values corresponding to the machine vision task networks, the encoding code stream loss value, and the decoding feature loss value to obtain a total loss value.

[0191] In one of the embodiments, the backbone network in the machine vision task network includes a target number of feature output layers for outputting a target number of first feature information and a target number of second feature information. The training module 1003 is specifically configured to: for each feature output layer, calculate a task feature sub-loss value corresponding to the feature output layer according to the first feature information, the second feature information output by the feature output layer, and the task feature loss function; and perform weighted averaging on the task feature sub-loss values corresponding to the feature output layers to obtain the task feature loss value.

[0192] In one of the embodiments, the training module 1003 is specifically configured to: determine a task feature loss training speed and a decoding feature loss training speed according to a historical task feature loss value and a historical decoding feature loss value corresponding to a historical iteration number; determine a task feature loss weight, a decoding feature loss weight, and an encoding code stream loss weight according to the task feature loss training speed, the decoding feature loss training speed, the historical task feature loss value, and the historical decoding feature loss value; and perform weighted summation on the task feature loss value, the encoding code stream loss value, and the decoding feature loss value based on the task feature loss weight, the decoding feature loss weight, and the encoding code stream loss weight to obtain the total loss value.

[0193] In one of the embodiments, the training module 1003 is specifically configured to: determine a first historical task feature loss value and a first historical decoding feature loss value corresponding to a last iteration number of a last iteration number of a current iteration number, and a second historical task feature loss value and a second historical decoding feature loss value corresponding to a last iteration number of a last iteration number of the last iteration number; determine a task feature loss training speed according to a ratio of the first historical task feature loss value and the second historical task feature loss value, and determine a decoding feature loss training speed according to a ratio of the first historical decoding feature loss value and the second historical decoding feature loss value.

[0194] In one of the embodiments, the training module 1003 is specifically configured to: calculate a sum value of a task feature loss training speed and a decoding feature loss training speed, and calculate a mean value of a first historical task feature loss value and a first historical decoding feature loss value corresponding to a last iteration number of the current iteration number; calculate a task feature loss sub-weight according to the task feature loss training speed, the first historical task feature loss value, the sum value and the mean value, and calculate a decoding feature loss sub-weight according to the decoding feature loss training speed, the first historical decoding feature loss value, the sum value and the mean value; and determine the task feature loss weight, the decoding feature loss weight and the encoding bitstream loss weight according to the task feature loss sub-weight, the decoding feature loss sub-weight and an adjustable weight parameter.

[0195] Based on the same inventive concept, the embodiments of the present application also provide an image processing device for implementing the image processing method described above. The implementation scheme of the device for solving the problem is similar to the implementation scheme described in the above method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations of the image processing method in the above text, which will not be repeated here.

[0196] In one embodiment, as shown in Figure 11 , an image processing device 1100 is provided, comprising an input module 1101 and a sending module 1102, wherein:

[0197] The input module 1101 is configured to input a to-be-processed image into an encoder in a trained codec model to obtain an encoding bitstream; wherein the codec model is trained according to the training method of the codec model provided in the above embodiments.

[0198] The sending module 1102 is configured to send the encoding bitstream to a receiving device.

[0199] In one embodiment, as shown in Figure 12 , another image processing device 1200 is also provided, comprising a receiving module 1201 and an input module 1202, wherein:

[0200] The receiving module 1201 is configured to receive an encoding bitstream of a to-be-processed image, and the encoding bitstream is obtained by inputting the to-be-processed image into an encoder in a trained codec model.

[0201] The input module 1202 is configured to input the encoding bitstream into a decoder in the trained codec model to obtain decoding features, and the decoding features are used for a machine vision task network to perform a vision task; wherein the codec model is trained according to the training method of the codec model provided in the above embodiments.

[0202] The modules in the training apparatus of the coding and decoding model or the image processing apparatus can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules.

[0203] In an embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in FIG. 1. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved by WIFI, mobile cellular network, NFC (near field communication), or other technologies. The computer program is executed by the processor to implement a training method of a coding and decoding model or an image processing method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc. Figure 13

[0204] Those skilled in the art can understand that the structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. Specifically, the computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. Figure 13

[0205] In an embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the steps in the above method embodiments.

[0206] In an embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0207] In an embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0208] ​​It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0209] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0210] The technical features of the above embodiments can be combined in any way. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present disclosure.

[0211] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for training a coding model, comprising: The codec model comprises an encoder and a decoder, and the method comprises: inputting a training sample image into the codec model to obtain an encoded code stream and a decoding feature of the encoded code stream; inputting the decoding feature into a machine vision task network to obtain first feature information output by a backbone network in the machine vision task network, and inputting the training sample image into the machine vision task network to obtain second feature information output by the backbone network; iteratively training the codec model according to the first feature information, the second feature information, the encoded code stream, the decoding feature, the training sample image and a target loss function until a training end condition is reached to obtain a trained codec model; the target loss function comprises a task feature loss function and a codec model loss function, and the task feature loss function is used to represent the difference between the first feature information and the second feature information; the codec model loss function comprises an encoded code stream loss function and a decoding feature loss function; the iteratively training the codec model according to the first feature information, the second feature information, the encoded code stream, the decoding feature, the training sample image and the target loss function until the training end condition is reached to obtain the trained codec model comprises: calculating a task feature loss value according to the first feature information, the second feature information and the task feature loss function; calculating an encoded code stream loss value according to the encoded code stream, the training sample image and the encoded code stream loss function; calculating a decoding feature loss value according to the decoding feature, the training sample image and the decoding feature loss function; performing weighted summation on the task feature loss value, the encoded code stream loss value and the decoding feature loss value to obtain a total loss value; iteratively training the codec model according to the total loss value until the training end condition is reached to obtain the trained codec model.

2. The method of claim 1, wherein, The number of the machine vision task networks is multiple, and the calculating a task feature loss value according to the first feature information, the second feature information and the task feature loss function comprises: for each machine vision task network, calculating a task feature loss value corresponding to the machine vision task network according to the first feature information, the second feature information and the task feature loss function corresponding to the machine vision task network; the performing weighted summation on the task feature loss value, the encoded code stream loss value and the decoding feature loss value to obtain a total loss value comprises: performing weighted summation on the task feature loss value corresponding to each machine vision task network, the encoded code stream loss value and the decoding feature loss value to obtain a total loss value.

3. The method of claim 1, wherein, The backbone network in the machine vision task network comprises a target number of feature output layers for outputting the target number of first feature information and the target number of second feature information; the calculating a task feature loss value according to the first feature information, the second feature information and the task feature loss function comprises: For each of the feature output layers, a task feature sub-loss value corresponding to the feature output layer is calculated according to the first feature information, the second feature information output by the feature output layer and the task feature loss function; The task feature sub-loss values corresponding to the feature output layers are weighted and averaged to obtain the task feature loss value.

4. The method of claim 1, wherein, The task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value. According to the historical task feature loss value and the historical decoding feature loss value corresponding to the historical iteration number, a task feature loss training speed and a decoding feature loss training speed are determined. According to the task feature loss training speed, the decoding feature loss training speed, the historical task feature loss value and the historical decoding feature loss value, a task feature loss weight, a decoding feature loss weight and an encoding code stream loss weight are determined. The task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight.

5. The method of claim 4, wherein, The task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight. The task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight. The task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight.

6. The method of claim 4, wherein, The task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight. The task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight. The task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight. The task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight.

7. An image processing method characterized by, The task feature loss value, the encoding code stream loss value and the decoding feature loss value are weighted and summed to obtain a total loss value based on the task feature loss weight, the decoding feature loss weight and the encoding code stream loss weight. The method comprises: inputting an image to be processed into an encoder in the trained encoding and decoding model to obtain an encoding code stream; sending the encoding code stream to a receiving device; The codec model is trained according to the method in any one of claims 1-6.

8. An image processing method characterized by, The method comprises: receiving an encoding code stream of a to-be-processed image, the encoding code stream being obtained by inputting the to-be-processed image into an encoder in the trained codec model; inputting the encoding code stream into a decoder in the trained codec model to obtain decoding features, the decoding features being used for a machine vision task network to perform a vision task; The codec model is trained according to the method in any one of claims 1-6.

9. An apparatus for training a coding model, the apparatus comprising: The device comprises: a first input module configured to input a training sample image into the codec model to obtain an encoding code stream and decoding features of the encoding code stream; a second input module configured to input the decoding features into a machine vision task network to obtain first feature information output by a backbone network in the machine vision task network, and input the training sample image into the machine vision task network to obtain second feature information output by the backbone network; a training module configured to iteratively train the codec model according to the first feature information, the second feature information, the encoding code stream, the decoding features, the training sample image, and a target loss function until a training end condition is reached to obtain a trained codec model; the target loss function comprises a task feature loss function and a codec model loss function, and the task feature loss function is used to represent a difference between the first feature information and the second feature information; The codec model loss function comprises an encoding code stream loss function and a decoding feature loss function; and the training module is specifically configured to: calculate a task feature loss value according to the first feature information, the second feature information, and the task feature loss function; calculate an encoding code stream loss value according to the encoding code stream, the training sample image, and the encoding code stream loss function; calculate a decoding feature loss value according to the decoding features, the training sample image, and the decoding feature loss function; and perform weighted summation on the task feature loss value, the encoding code stream loss value, and the decoding feature loss value to obtain a total loss value; and iteratively train the codec model according to the total loss value until the trained codec model is obtained when the training end condition is reached. 10.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-9. The processor executes the computer program to implement the steps of the method in any one of claims 1-8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method in any one of claims 1-8.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1-8. The computer program is executed by the processor to implement the steps of the method in any one of claims 1-8.

Images