Model processing method and device

Abstract

An embodiment of the present application provides a model processing method and device, wherein the model processing method includes: inputting an initial image into an image processing model for feature screening to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image, determining a first loss value of the image processing model based on the initial image and the intermediate image, compressing the intermediate image, and determining a second loss value of the image processing model based on the compression result, and adjusting model parameters of the image processing model according to the first loss value and the second loss value.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and more particularly to a model processing method. One or more embodiments of the present application also relate to a model processing device, a computing device, and a computer-readable storage medium. Background Art

[0002] Video encoding and decoding are widely used in digital video applications today, such as broadcast digital television, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVD and Blu-ray discs, video content acquisition and editing systems, and security applications in camcorders.

[0003] Even in the case of short films, a large amount of video data needs to be described, which may cause difficulties when the data is sent or otherwise transmitted in a network with limited bandwidth capacity. Therefore, video data is usually compressed before being transmitted in modern telecommunications networks. Since memory resources may be limited, the size of the video may also become an issue when storing the video on a storage device. Video compression devices usually use software and / or hardware on the source side to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received by a video decompression device. With limited network resources and the growing demand for higher video quality, there is a need to improve compression and decompression technology, that is, how to improve the compression rate while ensuring the quality of the compressed video in the process of compressing the video frames in the video has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a model processing method. One or more embodiments of the present application also relate to a model processing device, a computing device, and a computer-readable storage medium to solve the technical defect in the prior art that in the process of compressing a video, if the video compression rate needs to be increased, the quality of the compressed video cannot be guaranteed.

[0005] According to a first aspect of an embodiment of the present application, a model processing method is provided, including:

[0006] Inputting the initial image into the image processing model for feature screening to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image;

[0007] Determining a first loss value of the image processing model based on the initial image and the intermediate image;

[0008] compressing the intermediate image, and determining a second loss value of the image processing model based on the compression result;

[0009] According to the first loss value and the second loss value, the model parameters of the image processing model are adjusted.

[0010] According to a second aspect of an embodiment of the present application, a model processing device is provided, including:

[0011] A screening module is configured to input the initial image into the image processing model for feature screening to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image;

[0012] A first determination module is configured to determine a first loss value of the image processing model based on the initial image and the intermediate image;

[0013] A second determination module is configured to perform compression processing on the intermediate image and determine a second loss value of the image processing model based on the compression result;

[0014] An adjustment module is configured to adjust the model parameters of the image processing model according to the first loss value and the second loss value.

[0015] According to a third aspect of an embodiment of the present application, a computing device is provided, including:

[0016] Memory and processor;

[0017] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the processor implements the steps of the model processing method when executing the computer-executable instructions.

[0018] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the model processing method are implemented.

[0019] An embodiment of the present application implements a model processing method and device, wherein the model processing method includes inputting an initial image into an image processing model for feature screening to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image, determining a first loss value of the image processing model based on the initial image and the intermediate image, compressing the intermediate image, and determining a second loss value of the image processing model based on the compression result, and adjusting model parameters of the image processing model according to the first loss value and the second loss value.

[0020] When adjusting the model parameters of the image processing model, the embodiment of the present application first uses the intermediate image output by the image processing model and the initial image to jointly calculate the first loss value of the image processing model, and then compresses the intermediate image to determine the second loss value of the image processing model based on the compression result. Then, based on the first loss value and the second loss value, the model parameters of the image processing model are adjusted together, so that the adjusted image processing model can improve the compression rate while ensuring the quality of the compressed video. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is an architecture diagram of a model processing method provided by an embodiment of the present application;

[0022] Figure 2 is a flow chart of a model processing method provided by an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a video encoding process provided by an embodiment of the present application;

[0024] Figure 4 This is a specific flow chart of the application of the model processing method in a video compression scenario provided by an embodiment of the present application;

[0025] Figure 5 is a structural schematic diagram of a model processing device provided by an embodiment of the present application;

[0026] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0028] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0030] First, the terms involved in one or more embodiments of the present application are explained.

[0031] DCT: Discrete Cosine Transform, discrete cosine transform.

[0032] L1 Loss: Mean absolute error loss function.

[0033] Low Rank DCT Loss: Low rank expression loss function based on DCT.

[0034] With the development of mobile communication technology and Internet technology, more and more multimedia content based on images or videos has emerged. As an important carrier of information, images or videos have many characteristics such as intuitive and easy to understand, vivid and vivid. They are an efficient means of information dissemination and are widely used in various fields such as life entertainment, road monitoring, media, and medical treatment.

[0035] By encoding images or videos, the information contained in the images or videos can be represented with fewer bits while meeting certain image or video quality requirements, thereby reducing the bandwidth resources required to transmit the images or videos and reducing the storage resources required to store the images or videos. Transformation, quantization, entropy coding and other technologies can usually be used to eliminate redundant information in the image, achieve image encoding, store and transmit high-quality images or videos with a lower amount of data, and effectively transmit information.

[0036] Therefore, in order to meet users' growing demands for image or video transmission and storage, how to improve the compression rate while ensuring the quality of the compressed video has become an urgent problem to be solved.

[0037] At present, most traditional filtering methods such as guided filtering are used to reduce the filtering strength in areas with obvious edge features of the image; on the contrary, in areas with unclear texture features, the filtering strength is increased in flat areas of the image to filter out the weak texture in the flat areas. At this time, the processed video is encoded, so that the encoder does not need to consume a lot of bit rate in the flat texture, and more bit rate will be allocated to the edge texture.

[0038] Existing methods use traditional filtering methods to save bitrate by filtering small intensity textures at the edge of the image and large intensity textures in the flat area. However, this processing method cannot optimize filtering more specifically for different texture features, resulting in a decrease in image quality despite saving bitrate.

[0039] This application provides a deep learning algorithm, which designs a low-rank expression loss function based on the DCT frequency domain, uses some high-fidelity image data to train the convolutional neural network, and retains some important texture details while weakening some secondary textures through adaptive learning in the frequency domain (DCT transformation of the image, eliminating some frequencies with small frequency domain component coefficients, and retaining some frequencies with large frequency domain component coefficients). In this way, some important texture details can be better protected for different texture features, and the picture quality can be guaranteed while saving bit rate.

[0040] In the present application, a model processing method is provided. One or more embodiments of the present application also relate to a model processing device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0041] In specific implementation, the images or videos of the embodiments of the present application can be presented on large-scale video playback devices, game consoles, desktop computers, smart phones, tablet computers, MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) players, MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers, e-book readers and other display terminals and other clients.

[0042] See also Figure 1 , Figure 1 The figure shows an architecture diagram of a model processing method provided according to an embodiment of the present application.

[0043] Figure 1 In the method, the initial image, i.e., the RGB image, is first input into the image processing model for feature screening to generate an intermediate image, i.e., the predicted RGB image. Since the image processing model is used to perform feature screening on the initial image to reduce the bit rate of the initial image, the bit rate of the intermediate image output by the image processing model is lower than that of the initial image. The initial image may be a video frame obtained by decoding a video.

[0044] After the image processing model outputs the predicted RGB image, the first loss value of the image processing model can be calculated based on the RGB image, the predicted RGB image and the first loss function, and then the second loss value of the image processing model can be calculated based on the predicted RGB image and the second loss function. The model parameters of the image processing model are adjusted together with the first loss value and the second loss value.

[0045] When calculating the second loss value of the image processing model based on the predicted RGB image and the second loss function, the predicted RGB image can be compressed first, for example, the predicted RGB image is subjected to discrete cosine transform, and the second loss value of the image processing model is determined based on the compression result (transformation result).

[0046] When adjusting the model parameters of the image processing model, the embodiment of the present application first uses the intermediate image output by the image processing model and the initial image to jointly calculate the first loss value of the image processing model, and then compresses the intermediate image to determine the second loss value of the image processing model based on the compression result. Then, based on the first loss value and the second loss value, the model parameters of the image processing model are adjusted together, so that the adjusted image processing model can improve the compression rate while ensuring the quality of the compressed video.

[0047] See also Figure 2 , Figure 2 A flow chart of a model processing method provided according to an embodiment of the present application is shown, comprising the following steps:

[0048] Step 202: Input the initial image into an image processing model for feature screening to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image.

[0049] Specifically, the initial image and the intermediate image can be RGB images. When the input of the image processing model is a tensor of height*width*3, the output intermediate image is a three-channel tensor (height*width*3) of the same size as the input.

[0050] In practical applications, the initial image may be a video frame obtained by decoding the video to be processed. Since the image processing model is used to perform feature screening on the initial image to reduce the bit rate of the initial image, the bit rate of the intermediate image output by the image processing model is lower than the bit rate of the initial image.

[0051] Additionally, the image processing model may be a convolutional neural network.

[0052] Step 204: Determine a first loss value of the image processing model based on the initial image and the intermediate image.

[0053] Specifically, after the image processing model outputs the intermediate image corresponding to the initial image, the first loss value of the image processing model can be calculated based on the initial image and the intermediate image. The first loss value can be used to adjust the model parameters of the image processing model to achieve model optimization.

[0054] In practical applications, the first loss value can be calculated using a first preset loss function, and the first preset loss function can be an L1 loss function, that is, a mean absolute error loss function (MAE), also known as a minimum absolute deviation loss function (LAD) or an absolute value loss function (LAE). It is the sum of the absolute values ​​of the difference between the target value and the predicted value, representing the average error margin of the predicted value, and the calculation formula is shown in formula (1):

[0055]

[0056] Among them, y i represents the target value, f(x i ) represents the predicted value, both of which are vectors or tensors.

[0057] In a specific implementation, determining a first loss value of the image processing model based on the initial image and the intermediate image includes:

[0058] Based on the pixel value corresponding to each pixel point in the initial image and the pixel value corresponding to each pixel point in the intermediate image, a first loss value of the image processing model is determined.

[0059] Specifically, when calculating the first loss value of the image processing model based on the initial image and the intermediate image, the first loss value can be calculated by calculating the difference between the pixel values ​​corresponding to each pixel in the initial image and the intermediate image. When the first loss value is calculated using formula (1), the pixel value corresponding to each pixel in the initial image and the pixel value corresponding to each pixel in the intermediate image can be input into formula (1) to calculate the corresponding first loss value L. In this case, i is the pixel points contained in the initial image or the intermediate image, n is the total number of pixel points contained in the initial image or the intermediate image, and y is the number of pixels contained in the initial image or the intermediate image. i That is, it represents the pixel value corresponding to each pixel in the initial image, f(x i ) represents the pixel value corresponding to each pixel in the intermediate image.

[0060] Step 206: compress the intermediate image and determine a second loss value of the image processing model based on the compression result.

[0061] Specifically, after the image processing model performs feature screening on the initial image and outputs an intermediate image, in addition to determining the first loss value of the image processing model based on the intermediate image and the initial image to adjust the model parameters of the image processing model using the first loss value, the intermediate image can also be compressed to determine the second loss value of the image processing model based on the compression result, and the model parameters of the image intermediate model are adjusted in combination with the first loss value and the second loss value, so as to ensure the accuracy of the output results of the image processing model, and through the adjusted image processing model, feature screening is performed on the video frames in the processed video to reduce the video, which can not only improve the compression rate of the video frames, but also ensure the quality of the compressed video.

[0062] In a specific implementation, compressing the intermediate image includes:

[0063] Performing image segmentation on the intermediate image to generate at least two image units;

[0064] The at least two image units are compressed using a preset image compression algorithm to generate corresponding compression results.

[0065] Furthermore, compressing the at least two image units using a preset image compression algorithm to generate corresponding compression results includes:

[0066] Determine the position coordinates of a target image unit in the intermediate image, and determine a pixel value corresponding to the target image unit, wherein the target image unit is any one of the at least two image units;

[0067] According to a preset image compression algorithm, the number of units of the at least two image units, the position coordinates and the pixel values, the frequency domain information corresponding to the target image unit is determined, and the frequency domain information is determined as a compression result corresponding to the target image unit.

[0068] Specifically, when compressing the intermediate image, the intermediate image can be first divided into at least two image units, and each image unit can be compressed using a preset image compression algorithm to generate a corresponding compression result. When compressing each image unit, the position coordinates of each image unit in the intermediate image can be determined, and the pixel value corresponding to each image unit can be determined. Then, based on the preset image compression algorithm, the number of image units, the position coordinates and the pixel value, the frequency domain information corresponding to each image unit can be determined, and the frequency domain information can be determined as the corresponding compression result.

[0069] In a specific implementation, the image compression algorithm includes a discrete cosine transform algorithm.

[0070] Specifically, the embodiments of the present specification may use a discrete cosine transform algorithm, i.e., a DCT algorithm, to compress the intermediate image. Specifically, the intermediate image may be first segmented to generate at least two image units. In practical applications, the intermediate image may be usually segmented into 8*8, i.e., 64 image units. Then, the 64 image units are subjected to a DCT transform to obtain corresponding transform coefficients, i.e., a compression result. The specific transform formula used is shown in formula (2):

[0071]

[0072] Wherein, N represents the total number of image units, x and y represent the position information of each image unit in the intermediate image, x is the horizontal coordinate of the image unit in the intermediate image, y is the vertical coordinate of the image unit in the intermediate image, and f(x, y) is the pixel value corresponding to the image unit.

[0073] Therefore, the discrete cosine transform algorithm is used to compress at least two image units to generate corresponding compression results. Specifically, the number of image units, the position coordinates of each image unit and the pixel value corresponding to each image unit are input into formula (2) for calculation to obtain the corresponding calculation result, which is the transformation coefficient, that is, the compression result corresponding to the intermediate image.

[0074] In addition, the compression result corresponding to the intermediate image includes low-frequency coefficients and high-frequency coefficients;

[0075] Accordingly, determining the second loss value of the image processing model based on the compression result includes:

[0076] Determine a coefficient mean corresponding to the high frequency coefficient;

[0077] Screening the high-frequency coefficients according to the coefficient mean to obtain target high-frequency coefficients;

[0078] A second loss value of the image processing model is determined based on the target high-frequency coefficients.

[0079] Further, the high-frequency coefficients are screened according to the coefficient mean to obtain target high-frequency coefficients, including:

[0080] Screening the high-frequency coefficients according to the coefficient mean to obtain target high-frequency coefficients that are smaller than the coefficient mean;

[0081] Accordingly, determining the second loss value of the image processing model according to the target high-frequency coefficient includes:

[0082] A reference value corresponding to the target loss function is determined, and a second loss value of the image processing model is determined based on a difference between the target high-frequency coefficient and the reference value.

[0083] Specifically, if the discrete cosine transform algorithm is used to compress the intermediate image, the compression result obtained by the processing includes AC coefficients (alternating current components, also called high-frequency coefficients or high-frequency components) and DC coefficients (direct current components, also called low-frequency coefficients or low-frequency components). Among them, the high-frequency coefficients generally store the information of the boundary area of ​​the image, the boundary area has a large mutation degree and rich texture; the low-frequency coefficients mainly store the information of the flat area in the image, the flat area has a small mutation degree and low texture richness.

[0084] For example, if the intermediate image is divided into 64 image units, and DCT transformation is performed on each of the 64 image units to obtain the corresponding compression result F(u, v), then the compression result F(u, v) is composed of 64 transformation coefficients, among which, the coefficient F corresponding to u=0 and v=0 is called the DC coefficient, and the other 63 coefficients are called AC coefficients. Since the grayscale or color difference signal between pixels in an image changes slowly, the correlation between pixels in the 8*8 image units is very strong. Therefore, after DCT transformation processing, coefficients with large values ​​are concentrated in the low-frequency range of spatial frequency, which makes data compression possible. The high-frequency coefficients far away from the low-frequency coefficients are mostly 0 or close to 0.

[0085] Therefore, in an embodiment of the present application, when determining the second loss value of the image processing model based on the compression result, the coefficient mean corresponding to each high-frequency coefficient can be first calculated, and each high-frequency coefficient can be screened according to the coefficient mean to obtain the target high-frequency coefficient, and then the second loss value of the image processing model can be determined based on the target high-frequency coefficient.

[0086] The formula for determining the mean coefficient is shown in formula (3):

[0087] Threshold=Average((F1(u,v)), F1(u,v)∈{highfrequency component} Formula (3)

[0088] Among them, F1(u, v) is the high-frequency coefficient in F(u, v).

[0089] After determining the coefficient mean of the high-frequency coefficient, the high-frequency coefficient is screened according to the coefficient mean, specifically, the target high-frequency coefficient that is smaller than the coefficient mean is screened, and the second loss value of the image processing model is determined according to the target high-frequency coefficient. The formula for determining the second loss value is shown in formula (4):

[0090] Loss=Mean Absolute Error(F2(u, v), 0), F2(u, v)∈{F1(u, v)<Threshold} Formula (4)

[0091] Among them, F2(u, v) is the target high-frequency coefficient that is less than the coefficient mean among the high-frequency coefficients, and the second loss value of the image processing model is determined according to the target high-frequency coefficient, that is, the sum of the absolute values ​​of the differences between each target high-frequency coefficient and the target value is calculated. In the embodiment of the present application, the target value (reference value) can be selected as 0. Therefore, the second loss value of the image processing model is determined according to the target high-frequency coefficient, that is, the sum of each target high-frequency coefficient is determined.

[0092] In addition, in the embodiment of the present application, formula (2), formula (3) and formula (4) together constitute a low-rank expression DCT loss function.

[0093] Step 208: Adjust the model parameters of the image processing model according to the first loss value and the second loss value.

[0094] Specifically, after determining the first loss value and the second loss value, the first loss value and the second loss value can be used together to adjust the model parameters of the image processing model to optimize the image processing model.

[0095] In a specific implementation, adjusting the model parameters of the image compression model according to the first loss value and the second loss value includes:

[0096] Determining a target loss value of the image processing model according to the first loss value and the second loss value;

[0097] According to the target loss value, the model parameters of the image processing model are adjusted.

[0098] Further, determining a target loss value of the image processing model according to the first loss value and the second loss value includes:

[0099] Determine a first weight corresponding to the first loss value and a second weight corresponding to the second loss value;

[0100] A target loss value of the image processing model is determined according to the first loss value, the first weight, the second loss value and the second weight.

[0101] Specifically, after determining the first loss value and the second loss value, the target loss value of the image processing model can be determined according to the first loss value and the second loss value, so as to adjust the model parameters of the image processing model according to the target loss value.

[0102] Among them, when the model parameters of the image processing model are adjusted by using the target loss value, it can be realized by the Adam optimization algorithm. The Adam algorithm is a collection of the advantages of two extended forms of stochastic gradient descent, namely the adaptive gradient algorithm (AdaGrad) and the root mean square propagation algorithm (RMSProp). Among them, the adaptive gradient algorithm (AdaGrad) calculates the adaptive parameter learning rate based on the first-order moment mean, and retains a learning rate for each parameter to improve the performance on sparse gradients (i.e., natural language and computer vision problems); the root mean square propagation algorithm (RMSProp) adaptively retains the learning rate for each parameter based on the second-order moment mean of the weight gradient; therefore, when the model parameters of the image processing model are reversely adjusted by using the Adam algorithm, the Adam algorithm designs independent adaptive learning rates for different parameters by calculating the first-order moment estimate and the second-order moment estimate of the gradient. In this way, it is beneficial to make the output result of the adjusted generated model more accurate and to improve the adjustment efficiency of the model parameters.

[0103] Among them, since the first loss value and the second loss value are obtained by calculating different loss functions, and different loss functions may have different effects, when using the first loss value and the second loss value to calculate the target loss value of the image processing model, different weights can be set for different loss values ​​to reflect the importance of different loss values ​​to the model parameter adjustment results.

[0104] In practical applications, since the second loss value is generated by the transformation result obtained by performing DCT transformation on the intermediate image, and the transformation coefficient generated by the DCT transformation can characterize the richness of the texture information it contains, when the second loss value is needed to adjust the model parameters of the image processing model so that the adjusted image processing model can not only improve the image compression rate but also ensure the quality of the compressed video, when setting weights for the first loss value and the second loss value, the embodiment of the present application can set the second weight corresponding to the second loss value to be greater than the first weight corresponding to the first loss value, and the ratio between the first weight and the second weight can be 1:10, and then the first loss value and the second loss value can be weightedly summed based on the first weight and the second weight to obtain the target loss value of the image processing model, and the model parameters of the image processing model can be adjusted using the target loss value.

[0105] In the embodiment of the present application, the loss between the intermediate image and the initial image output by the image processing model is calculated through the L1 loss function, and then the loss of the intermediate image is calculated in combination with the low-rank expression DCT loss function. The image processing model is optimized based on the joint participation of the two. Through multiple rounds of training iterations, the image processing model converges and achieves the expected effect of saving bit rate and improving image compression rate.

[0106] In a specific implementation, the model parameters of the image processing model are adjusted, and after the adjusted image processing model is obtained, a video to be processed can be obtained, and the video to be processed is decoded to obtain an initial video frame contained in the video to be processed;

[0107] Inputting the initial video frame into the image processing model for feature screening processing to generate a target video frame;

[0108] The target video frame is encoded according to the target bit rate to generate a target video.

[0109] Specifically, after adjusting the model parameters of the image processing model and obtaining the adjusted image processing model, the video to be processed can be obtained, and the video to be processed is decoded to obtain the initial video frame contained in the video to be processed, and the initial video frame is input into the adjusted image processing model for feature screening (compression) processing to generate a target video frame, and then the target video frame is encoded according to the target bit rate to generate the target video.

[0110] The target video frame can be encoded by HEVC or AVC at different preset bit rates to obtain multiple target videos with different resolutions. Figure 3 As shown, Figure 3 In the pre-processing part, the image processing model used is embedded in the decoded video before the secondary encoding. The input of the image processing model is the decoded original video frame, and the output is the target video frame calculated by the model. It is then sent to the encoder to complete the video encoding and generate the target video.

[0111] This application uses deep learning methods to pre-process the video to be processed, saving bit rate while ensuring picture quality. In addition, since the image processing model uses a low-rank expression loss function based on the DCT frequency domain during training (adjusting model parameters), the model learns the ability to target different texture features and protect some important texture details during training.

[0112] An embodiment of the present application implements a model processing method and device, wherein the model processing method includes inputting an initial image into an image processing model for feature screening processing to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image, determining a first loss value of the image processing model based on the initial image and the intermediate image, compressing the intermediate image, and determining a second loss value of the image processing model based on the compression result, and adjusting model parameters of the image processing model according to the first loss value and the second loss value.

[0113] When adjusting the model parameters of the image processing model, the embodiment of the present application first uses the intermediate image output by the image processing model and the initial image to jointly calculate the first loss value of the image processing model, and then compresses the intermediate image to determine the second loss value of the image processing model based on the compression result. Then, based on the first loss value and the second loss value, the model parameters of the image processing model are adjusted together, so that the adjusted image processing model can improve the image compression rate while ensuring the quality of the compressed video.

[0114] See also Figure 4 , Figure 4 It is a specific flow chart of the application of the model processing method in a video compression scenario provided by an embodiment of the present application.

[0115] Step 402: decode the video to be processed to obtain an initial image contained in the video to be processed.

[0116] Step 404: input the initial image into an image processing model for feature screening to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image.

[0117] Step 406, determining a first loss value of the image processing model based on the pixel value corresponding to each pixel point in the initial image and the pixel value corresponding to each pixel point in the intermediate image.

[0118] Step 408: segment the intermediate image to generate at least two image units.

[0119] Step 410: compress at least two image units according to a discrete cosine transform algorithm to generate corresponding compression results, where the compression results include low-frequency coefficients and high-frequency coefficients.

[0120] Step 412, determine the coefficient mean corresponding to the high-frequency coefficient, screen the high-frequency coefficient according to the coefficient mean, and obtain the target high-frequency coefficient that is smaller than the coefficient mean.

[0121] Step 414, determine a reference value corresponding to the target loss function, and determine a second loss value of the image processing model based on the difference between the target high-frequency coefficient and the reference value.

[0122] Step 416, determining a first weight corresponding to the first loss value and a second weight corresponding to the second loss value.

[0123] Step 418, determining a target loss value of the image processing model based on the first loss value, the first weight, the second loss value and the second weight.

[0124] Step 420, adjusting the model parameters of the image processing model according to the target loss value to generate an adjusted image processing model.

[0125] Step 422, input the initial image into the adjusted image processing model for compression processing to generate a target image.

[0126] Step 424, encoding the target image according to the target bit rate to generate a target video.

[0127] When adjusting the model parameters of the image processing model, the embodiment of the present application first uses the intermediate image output by the image processing model and the initial image to jointly calculate the first loss value of the image processing model, and then compresses the intermediate image to determine the second loss value of the image processing model based on the compression result. Then, based on the first loss value and the second loss value, the model parameters of the image processing model are adjusted together, so that the adjusted image processing model can improve the compression rate while ensuring the quality of the compressed video.

[0128] Corresponding to the above method embodiment, the present application also provides a model processing device embodiment, Figure 5 FIG. 1 is a schematic diagram showing the structure of a model processing device provided by an embodiment of the present application. Figure 5 As shown, the device comprises:

[0129] A screening module 502 is configured to input the initial image into an image processing model for feature screening to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image;

[0130] A first determination module 504 is configured to determine a first loss value of the image processing model based on the initial image and the intermediate image;

[0131] A second determination module 506 is configured to perform compression processing on the intermediate image and determine a second loss value of the image processing model based on the compression result;

[0132] The adjustment module 508 is configured to adjust the model parameters of the image processing model according to the first loss value and the second loss value.

[0133] Optionally, the first determining module 504 is further configured to:

[0134] Based on the pixel value corresponding to each pixel point in the initial image and the pixel value corresponding to each pixel point in the intermediate image, a first loss value of the image processing model is determined.

[0135] Optionally, the second determining module 506 is further configured to:

[0136] Performing image segmentation on the intermediate image to generate at least two image units;

[0137] The at least two image units are compressed using a preset image compression algorithm to generate corresponding compression results.

[0138] Optionally, the second determining module 506 is further configured to:

[0139] Determine the position coordinates of a target image unit in the intermediate image, and determine a pixel value corresponding to the target image unit, wherein the target image unit is any one of the at least two image units;

[0140] According to a preset image compression algorithm, the number of units of the at least two image units, the position coordinates and the pixel values, the frequency domain information corresponding to the target image unit is determined, and the frequency domain information is determined as a compression result corresponding to the target image unit.

[0141] Optionally, the compression result corresponding to the intermediate image includes low-frequency coefficients and high-frequency coefficients;

[0142] Accordingly, the second determining module 506 is further configured to:

[0143] Determine a coefficient mean corresponding to the high frequency coefficient;

[0144] Screening the high-frequency coefficients according to the coefficient mean to obtain target high-frequency coefficients;

[0145] A second loss value of the image processing model is determined based on the target high-frequency coefficients.

[0146] Optionally, the second determining module 506 is further configured to:

[0147] Screening the high-frequency coefficients according to the coefficient mean to obtain target high-frequency coefficients that are smaller than the coefficient mean;

[0148] A reference value corresponding to the target loss function is determined, and a second loss value of the image processing model is determined based on a difference between the target high-frequency coefficient and the reference value.

[0149] Optionally, the adjustment module 508 is further configured to:

[0150] Determining a target loss value of the image processing model according to the first loss value and the second loss value;

[0151] According to the target loss value, the model parameters of the image processing model are adjusted.

[0152] Optionally, the adjustment module 508 is further configured to:

[0153] Determine a first weight corresponding to the first loss value and a second weight corresponding to the second loss value;

[0154] A target loss value of the image processing model is determined according to the first loss value, the first weight, the second loss value and the second weight.

[0155] Optionally, the image compression algorithm comprises a discrete cosine transform algorithm.

[0156] Optionally, the model processing device further includes a processing module configured to:

[0157] Obtaining a video to be processed, and decoding the video to be processed to obtain an initial video frame contained in the video to be processed;

[0158] Inputting the initial video frame into the image processing model for feature screening processing to generate a target video frame;

[0159] The target video frame is encoded according to the target bit rate to generate a target video.

[0160] An embodiment of the present application implements a model processing method and device, wherein the model processing device inputs an initial image into an image processing model for feature screening through a screening module to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image; determines a first loss value of the image processing model based on the initial image and the intermediate image through a first determination module; compresses the intermediate image through a second determination module, and determines a second loss value of the image processing model based on the compression result; and adjusts model parameters of the image processing model according to the first loss value and the second loss value through an adjustment module.

[0161] When adjusting the model parameters of the image processing model, the embodiment of the present application first uses the intermediate image output by the image processing model and the initial image to jointly calculate the first loss value of the image processing model, and then compresses the intermediate image to determine the second loss value of the image processing model based on the compression result. Then, based on the first loss value and the second loss value, the model parameters of the image processing model are adjusted together, so that the adjusted image processing model can improve the compression rate while ensuring the quality of the compressed video.

[0162] The above is a schematic scheme of a model processing device of this embodiment. It should be noted that the technical scheme of the model processing device and the technical scheme of the above-mentioned model processing method belong to the same concept, and the details not described in detail in the technical scheme of the model processing device can be referred to the description of the technical scheme of the above-mentioned model processing method.

[0163] Figure 6 The block diagram of a computing device 600 according to an embodiment of the present application is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0164] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0165] In one embodiment of the present application, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0166] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 600 may also be a mobile or stationary server.

[0167] The processor 620 is used to execute the following computer executable instructions, wherein the processor is used to execute the computer executable instructions, wherein the processor implements the steps of the model processing method when executing the computer executable instructions.

[0168] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above model processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above model processing method.

[0169] An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the model processing method when executed by a processor.

[0170] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned model processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned model processing method.

[0171] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0172] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0173] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described action sequence, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present application.

[0174] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0175] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A model processing method, characterized in that: include: Inputting the initial image into the image processing model for feature screening to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image; Determining a first loss value of the image processing model based on the initial image and the intermediate image; compressing the intermediate image, and determining a second loss value of the image processing model based on the compression result; Adjusting model parameters of the image processing model according to the first loss value and the second loss value; The step of compressing the intermediate image includes: Performing image segmentation on the intermediate image to generate at least two image units; Using a preset image compression algorithm to compress the at least two image units to generate corresponding compression results; The compression result corresponding to the intermediate image includes low-frequency coefficients and high-frequency coefficients; Accordingly, determining the second loss value of the image processing model based on the compression result includes: Determine a coefficient mean corresponding to the high frequency coefficient; Screening the high-frequency coefficients according to the coefficient mean to obtain target high-frequency coefficients; A second loss value of the image processing model is determined based on the target high-frequency coefficients.

2. The model processing method according to claim 1, characterized in that: The determining, based on the initial image and the intermediate image, a first loss value of the image processing model comprises: Based on the pixel value corresponding to each pixel point in the initial image and the pixel value corresponding to each pixel point in the intermediate image, a first loss value of the image processing model is determined.

3. The model processing method according to claim 1, characterized in that: The compressing the at least two image units using a preset image compression algorithm to generate corresponding compression results includes: Determine the position coordinates of a target image unit in the intermediate image, and determine a pixel value corresponding to the target image unit, wherein the target image unit is any one of the at least two image units; According to a preset image compression algorithm, the number of units of the at least two image units, the position coordinates and the pixel values, the frequency domain information corresponding to the target image unit is determined, and the frequency domain information is determined as a compression result corresponding to the target image unit.

4. The model processing method according to claim 1, characterized in that: The step of screening the high frequency coefficients according to the coefficient mean to obtain target high frequency coefficients includes: Screening the high-frequency coefficients according to the coefficient mean to obtain target high-frequency coefficients that are smaller than the coefficient mean; Accordingly, determining the second loss value of the image processing model according to the target high-frequency coefficient includes: A reference value corresponding to the target loss function is determined, and a second loss value of the image processing model is determined based on a difference between the target high-frequency coefficient and the reference value.

5. The model processing method according to claim 1, characterized in that: The adjusting the model parameters of the image processing model according to the first loss value and the second loss value includes: Determining a target loss value of the image processing model according to the first loss value and the second loss value; According to the target loss value, the model parameters of the image processing model are adjusted.

6. The model processing method according to claim 5, characterized in that: Determining a target loss value of the image processing model according to the first loss value and the second loss value includes: Determine a first weight corresponding to the first loss value and a second weight corresponding to the second loss value; A target loss value of the image processing model is determined according to the first loss value, the first weight, the second loss value and the second weight.

7. The model processing method according to claim 1 or 3, characterized in that: The image compression algorithm includes a discrete cosine transform algorithm.

8. The model processing method according to claim 1, characterized in that: Also includes: Obtaining a video to be processed, and decoding the video to be processed to obtain an initial video frame contained in the video to be processed; Inputting the initial video frame into the image processing model for feature screening processing to generate a target video frame; The target video frame is encoded according to the target bit rate to generate a target video.

9. A model processing device, characterized in that: include: A screening module is configured to input the initial image into the image processing model for feature screening to generate an intermediate image, wherein the bit rate of the intermediate image is lower than the bit rate of the initial image; A first determination module is configured to determine a first loss value of the image processing model based on the initial image and the intermediate image; A second determination module is configured to perform compression processing on the intermediate image and determine a second loss value of the image processing model based on the compression result; an adjustment module, configured to adjust a model parameter of the image processing model according to the first loss value and the second loss value; The step of compressing the intermediate image includes: Performing image segmentation on the intermediate image to generate at least two image units; Using a preset image compression algorithm to compress the at least two image units to generate corresponding compression results; The compression result corresponding to the intermediate image includes low-frequency coefficients and high-frequency coefficients; Accordingly, determining the second loss value of the image processing model based on the compression result includes: Determine a coefficient mean corresponding to the high frequency coefficient; Screening the high-frequency coefficients according to the coefficient mean to obtain target high-frequency coefficients; A second loss value of the image processing model is determined based on the target high-frequency coefficients.

10. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the processor implements the steps of the model processing method described in any one of claims 1 to 8 when executing the computer-executable instructions.

11. A computer-readable storage medium, characterized in that: It stores computer instructions, which, when executed by a processor, implement the steps of the model processing method described in any one of claims 1 to 8.

12. A computer program product comprising computer instructions, characterized in that: When the computer instruction is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

Images