Video data processing method, device, storage medium and electronic device

Abstract

The present application discloses a video data processing method, device, storage medium and electronic device, the method comprising: reading the video data to be processed frame by frame to determine the video frame image to be processed; performing color space conversion on the video frame image to be processed, and selecting a brightness channel image as the input frame image; inputting the input frame image into a trained data processing model for processing to obtain an output frame image; performing channel merging on the output frame image, and converting the channel-merged image into an image of three primary color channels through the color space; merging the images into a video sequence according to the video frame order of the video data to be processed to obtain target video data; the method can achieve more natural and significant image quality enhancement, and the processing process link is simple, the computing power requirement is low, and the processing time is short.

Description

Technical Field

[0001] The present application relates to the field of digital media technology, and more particularly to a method and apparatus for processing video data. The present application also relates to a method and apparatus for training a data processing model, as well as a computer storage medium and electronic equipment. Background Art

[0002] With the popularization of smart phones and mobile networks, video, as an emerging form of communication, has become an important form of entertainment in daily life.

[0003] From upload to user presentation, videos require processing such as video editing, encoding / decoding, transcoding, and compression. As users' expectations for a better viewing experience continue to rise, HD and UHD video have become mainstream. During the upload and playback process, video quality is meticulously processed. For example, video compression technology is employed to minimize file size while maintaining image quality, thereby reducing storage and transmission costs. Furthermore, the appropriate video version or format for playback is selected based on the device and network environment to ensure a superior viewing experience. Summary of the Invention

[0004] The present application provides a training method for a data processing model to solve the problem in the prior art that video data has poor playback picture quality due to pre-playback processing.

[0005] This application provides a data processing model training method, including:

[0006] Get the image set to be trained;

[0007] Converting the images in the to-be-trained image set into a luminance and chrominance space, and selecting luminance channel images as training samples;

[0008] The training samples are input into a data processing model for iterative training, wherein the loss function of the iterative training includes at least a component determined by the following method:

[0009] determining a negatively optimized portion of the output image based on a difference between an output image of a current iteration and an input image;

[0010] The difference between the image including the negative optimization part and the standard image corresponding to the input image is determined as one of the components of the loss function.

[0011] In some embodiments, determining the negatively optimized portion of the output image according to the difference between the output image of the current iteration and the input image comprises:

[0012] Comparing the output image with the input image pixel by pixel, marking a region where the contrast ratio of the output image is smaller than that of the input image pixel by pixel, and marking a region where the contrast ratio of the output image is greater than or equal to that of the input image pixel by pixel;

[0013] determining an image including the first mark and the second mark as a mask image;

[0014] The area of ​​the first mark in the mask image is determined as a negatively optimized portion of the output image.

[0015] In some embodiments, determining the difference between the image including the negatively optimized portion and a standard image corresponding to the input image as one of the components of the loss function comprises:

[0016] Determining a first pixel value of the negative optimized portion in the standard image by comparing the image of the negative optimized portion with the standard image pixel by pixel;

[0017] Determining a second pixel value of the negative optimized portion in the output image by comparing the image of the negative optimized portion with the output image pixel by pixel;

[0018] An average value of the difference between the first pixel value and the second pixel value is determined as a pixel difference component of the loss function.

[0019] In some embodiments, determining the difference between the image including the negatively optimized portion and a standard image corresponding to the input image as one of the components of the loss function comprises:

[0020] Determining a first eigenvalue of the negative optimized portion in the output image by performing feature matching on the image of the negative optimized portion and the output image;

[0021] Determining a second eigenvalue of the negative optimized portion in the standard image by performing feature matching on the image of the negative optimized portion and the standard image;

[0022] An average value of the difference between the first eigenvalue and the second eigenvalue is determined as a feature difference component of the loss function.

[0023] In some embodiments, determining the difference between the image including the negatively optimized portion and a standard image corresponding to the input image as one of the components of the loss function comprises:

[0024] Determining a first pixel value of the negative optimized portion in the standard image by comparing the image of the negative optimized portion with the standard image pixel by pixel;

[0025] Determining a second pixel value of the negative optimized portion in the output image by comparing the image of the negative optimized portion with the output image pixel by pixel;

[0026] determining an average value of the difference between the first pixel value and the second pixel value as a pixel difference component of the loss function;

[0027] Determining a first eigenvalue of the negative optimized portion in the output image by performing feature matching on the image of the negative optimized portion and the output image;

[0028] Determining a second eigenvalue of the negative optimized portion in the standard image by performing feature matching on the image of the negative optimized portion and the standard image;

[0029] Determine the difference between the first eigenvalue and the second eigenvalue as a characteristic difference component of the loss function;

[0030] The pixel difference component and the feature difference component are respectively determined as one of the components of the loss function.

[0031] In some embodiments, further comprising:

[0032] When the components of the loss function are the pixel difference component and / or the feature difference component, the weight ratio of the pixel difference component and / or the feature difference component is greater than the weight ratio of the third component in the loss function.

[0033] In some embodiments, further comprising:

[0034] The weights of the pixel difference component and the feature difference component are determined according to respective proportions of the pixel difference component and the feature difference component in the standard image.

[0035] In some embodiments, when the image set to be trained is derived from reference video data, obtaining the image set to be trained includes:

[0036] performing a first degradation process on the reference video data, and determining the video data after the first degradation process as the first video data;

[0037] performing a second degradation process on the frame image in the first video data, and converting the frame image after the second degradation process into second video data;

[0038] Acquire the image set to be trained according to the second video data;

[0039] When the image set to be trained is derived from a reference image set, obtaining the image set to be trained includes:

[0040] performing degradation processing on the reference image set, and determining the image set after the degradation processing as a degraded image set;

[0041] The to-be-trained image set is acquired according to the degraded image set.

[0042] In some embodiments, determining the to-be-trained image set based on the second video data includes:

[0043] Randomly extracting a frame image from the second video data;

[0044] The frame image is determined as the image set to be trained.

[0045] The present application also provides a data processing model training device, comprising:

[0046] An acquisition unit, used for acquiring an image set to be trained;

[0047] A conversion unit, configured to convert images in the image set to be trained into a luminance and chrominance space, and select luminance channel images as training samples;

[0048] A training unit is configured to input the training samples into a data processing model for iterative training, wherein the loss function of the iterative training includes at least a component determined by:

[0049] a first determining subunit, configured to determine a negatively optimized portion of the output image according to a difference between the output image and the input image of a current iteration;

[0050] The second determining subunit is configured to determine the difference between the image including the negative optimized part and the standard image corresponding to the input image as one of the components of the loss function.

[0051] This application also provides a video data processing method, including:

[0052] Read the video data to be processed frame by frame to determine the video frame image to be processed;

[0053] Performing color space conversion on the video frame image to be processed, and selecting a brightness channel image as an input frame image;

[0054] Inputting the input frame image into the model trained according to the training method of the above-mentioned data processing model for processing to obtain an output frame image;

[0055] Performing channel merging on the output frame image, and converting the channel-merged image into an image of three primary color channels through the color space;

[0056] The images are merged into a video sequence according to the video frame order of the video data to be processed to obtain target video data.

[0057] The present application also provides a video data processing device, comprising:

[0058] A determination unit, configured to read the video data to be processed frame by frame and determine the video frame image to be processed;

[0059] A first conversion unit is configured to perform color space conversion on the video frame image to be processed and select a luminance channel image as an input frame image;

[0060] a processing unit, configured to input the input frame image into a model trained according to the above-mentioned data processing model training method for processing, to obtain an output frame image;

[0061] A second conversion unit is configured to perform channel merging on the output frame image, and convert the channel-merged image into an image of three primary color channels through the color space;

[0062] The obtaining unit is used to merge the images into a video sequence according to the video frame sequence of the video data to be processed to obtain target video data.

[0063] The present application also provides a computer storage medium, characterized in that it includes a computer program, which, when run on an electronic device, enables the electronic device to execute the steps in the training method of the data processing model as described above; or, execute the steps in the video data processing method as described above.

[0064] The present application also provides an electronic device, comprising:

[0065] processor;

[0066] A memory for storing a program for processing data generated by an electronic device, wherein when the program is read and executed by the processor, the program executes the steps in the training method of the data processing model as described above; or, the program executes the steps in the video data processing method as described above.

[0067] Compared with the prior art, this application has the following advantages:

[0068] The present application provides a data processing model training method that can, during iterative training of the data processing model, determine the difference between an input image and an output image by comparing the two, and based on the difference, determine the portion of the output image that needs to be re-optimized during the iterative process, i.e., the negatively optimized portion present during the iterative process. The difference obtained by comparing the image including the negatively optimized portion with a standard image corresponding to the input image is determined as one of the components of the loss function. This method can automatically detect areas of insufficient training optimization during the iterative training process, so that these areas can be enhanced, thereby ensuring that detailed areas that are easily overlooked during the iterative training of the model are fully enhanced.

[0069] The data processing model training method provided in this application can also simulate various image quality defect problems to obtain a rich and diverse set of images to be trained, and can complete the training of complex image quality defect problems through a separate video data processing model. Since the training link is simple, the training computing power cost and computing time can be effectively reduced, and there is no need to use multiple detection models to detect image quality defect types in order to deal with complex image quality problems.

[0070] A video data processing method provided in the present application can obtain frame images with clearer image quality by inputting frame images in the video data into a trained data processing model, especially for the details, edges and other areas of the frame image, which can achieve more natural and significant image quality enhancement. In addition, the processing process link is simple, the computing power requirement is low, and the processing time is short. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a flowchart of a method for training a data processing model provided in this application.

[0072] Figure 2 This is a schematic diagram of an embodiment of upsampling and downsampling in a training method for a data processing model provided in this application.

[0073] Figure 3 This is a schematic diagram of an embodiment of determining difference loss data in a model process in a training method of a data processing model provided in the present application.

[0074] Figure 4 This is a structural diagram of a training device for a data processing model provided in this application.

[0075] Figure 5 This is a flowchart of a video data processing method provided by this application.

[0076] Figure 6 This is a structural diagram of a video data processing device provided by this application.

[0077] Figure 7 This is a structural diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0078] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present application. Therefore, the present application is not limited to the specific implementations disclosed below.

[0079] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. Descriptive terms such as "a," "a," "a first," and "a second," etc., used in this application and the appended claims, are not intended to limit quantity or sequence, but are used to distinguish information of the same type from one another.

[0080] Based on the above background technology, it can be seen that the invention of this application is based on the demand for video data viewing experience. In the field of digital media technology, video is widely used in various scenarios as a means of information dissemination. From uploading to presentation, videos usually need to go through video editing, video encoding and decoding and other related processing, aiming to enhance the viewing and practicality of the video. During the above processing, there will be processing flows that affect the image quality, and the video will also be transmitted multiple times. During this process, the video is compressed multiple times, resulting in a decrease in video quality, resulting in a poor viewing experience for users.

[0081] Although there are some technical solutions to poor image quality in the existing technology, there are still problems such as low processing efficiency, high computing power costs, and complex processing links. This application provides a training method for a data processing model that can improve image clarity in practical applications, especially in easily overlooked detail areas, without requiring high computing power.

[0082] The following describes a training method for a data processing model provided in this application.

[0083] like Figure 1 As shown, Figure 1 This is a flowchart of a method for training a data processing model provided by the present application, the method comprising:

[0084] Step S101: obtaining an image set to be trained;

[0085] Step S102: converting the images in the image set to be trained into luminance and chromaticity space, and selecting luminance channel images as training samples;

[0086] Step S103: Input the training samples into a data processing model for iterative training, wherein the loss function of the iterative training includes at least components determined in the following manner:

[0087] Step S103-1: determining a negatively optimized portion of the output image according to a difference between the output image and the input image of the current iteration;

[0088] Step S103 - 2 : determining the difference between the image including the negatively optimized portion and the standard image corresponding to the input image as one of the components of the loss function.

[0089] Before describing the above steps S101 to S103 in detail, a general description of the relevant technical terms involved in the technical solution of this application is first given.

[0090] Deep learning: An artificial intelligence technique that uses multi-layered neural networks to perform complex data processing and learning.

[0091] CNN, short for Convolutional Neural Network, is a neural network model used in deep learning algorithms. A CNN is an artificial neural network that mimics human visual perception. It extracts key features from data through a layered process of operations like convolution and pooling, enabling classification, recognition, or prediction. Its core lies in the convolution operation, which extracts local features by sliding a filter (also known as a convolution kernel) over the input data.

[0092] Perceptual loss: A loss function used in computer vision and image processing that aims to evaluate the similarity between two images in the feature domain to improve visual perception.

[0093] GAN loss: A loss function in Generative Adversarial Networks (GANs) that evaluates the difference between generated images and real images.

[0094] VGG: A classic convolutional neural network structure.

[0095] Degradation operation: refers to the operation of reducing the quality of a high-quality image to a low-quality image.

[0096] Real-BasicVSR: A video super-resolution solution that provides a comprehensive set of degradation schemes to simulate low-quality video frames in real-world scenarios. To improve model generalization and training efficiency, Real-BasicVSR introduces a randomized degradation mechanism. This mechanism generates different combinations of degradations (such as Gaussian blur, Poisson noise, and JPEG compression) for supervised training, enabling the model to generalize to real-world scenarios.

[0097] Crf (Constant Rate Factor): A method of allocating bitrate in the FFmpeg encoder that ensures "a certain quality" and intelligently allocates bitrate, including bitrate allocation within the same frame and bitrate allocation between frames.

[0098] Regarding step S101: obtaining a set of images to be trained.

[0099] The purpose of step S101 is to obtain a training image set before training a data processing model (e.g., a neural network model; in this embodiment, a convolutional neural network model is used as an example). In this embodiment, the training image set can be obtained from video data or image data. In this embodiment, the images in the training image set to be trained are images with image quality issues, such as unclearness. Accordingly, the training process requires reference images or standard images corresponding to the problematic images in the training image set. Therefore, images with image quality issues can be obtained by simulating various image quality defects on the reference images. When the training image set is obtained from video data, the reference data is reference video data (i.e., clear video). When the training image set is obtained from image data, the reference data (also known as standard data) is standard images. Therefore, the specific implementation process of step S102 can include two methods.

[0100] In a first approach, when the image set to be trained is derived from reference video data, obtaining the image set to be trained includes:

[0101] Step S101-11: performing a first degradation process on the reference video data, and determining the video data after the first degradation process as the first video data;

[0102] Step S101-12: performing a second degradation process on the frame image in the first video data, and converting the frame image after the second degradation process into second video data;

[0103] Step S101 - 13 : Determine the image set to be trained based on the second video data.

[0104] In this embodiment, the first degradation processing may include compression, scaling, blurring, and other processing on the reference video data, with the processed video data being the first video data. The second degradation processing may include degradation operations such as adding noise, sharpening, and white edges to the frame image. Of course, degradation operations such as adding noise may also be performed on the reference video data. This results in video data with reduced image quality.

[0105] The specific implementation process of step S101-13 may include: randomly extracting frame images from the second video data; and determining the frame images as the image set to be trained. The randomly extracted frame images correspond to reference images (i.e., standard images) extracted from the reference video, and the two may be in the form of data pairs.

[0106] It can be understood that in this embodiment, during the degradation processing of the reference video data and / or frame image, the parameters of the relevant degradation operations involved can also be randomly set, and the degradation operation can also be performed multiple times. The specific operation or process of simulating image quality problems can be set according to needs. The above is only an example.

[0107] In a second approach, when the image set to be trained is derived from a reference image set, obtaining the image set to be trained includes:

[0108] Step S101-21: performing degradation processing on the reference image set, and determining the image set after degradation processing as a degraded image set;

[0109] Step S101 - 22 : Acquire the image set to be trained based on the degraded image set.

[0110] In the second approach, the degradation processing performed on the reference image set may also include using one or more operations such as compression, noise reduction, white edge enhancement, and sharpening to reduce image quality. The degradation operation parameters may be randomly selected, and the degradation operation may be performed multiple times. The same image may be subjected to multiple different or the same degradation operations. The specific degradation operation process may be tailored to actual needs and is not limited to the above example.

[0111] In addition, if the generation of image quality problems in different video data is related to the type of video data generation, the following operations can also be used in some other implementation methods:

[0112] Determining, based on the generation type of the video data, a method for simulating multiple image quality defects on the reference video data;

[0113] performing degradation processing on the reference video data according to the simulation method, and determining the frame image after the degradation processing as the simulated frame image;

[0114] The training image set is determined according to the simulated frame image.

[0115] The types of video data generated may include: original self-shot videos (i.e., original videos shot and produced by users themselves), reposted videos (i.e., videos reposted between platforms), original edited videos (i.e., videos edited from non-original videos and accompanied by original audio), etc. Videos of different generation types may have the same or different image quality defects. For example, original self-shot videos may have shooting technical issues such as blurring and camera noise during the shooting process; reposted videos may have compression distortion caused by multiple transcoding on multiple platforms; original edited videos may have problems caused by both shooting and compression, as well as blurring and oversharpening introduced during the editing process. Of course, the above three video types are only examples, and in fact, video generation types are not limited to the above examples. The above examples are only used to illustrate the causes of image quality defects, that is, different video generation types may have the same or different image quality defects. Whether the image quality defects are caused by the video generation type or by other reasons, they can all be included in the scope of simulation. Image quality defects include the above-mentioned out-of-focus blur, camera noise, compression distortion, editing blur, editing oversharpening, etc., thereby determining a simulation method for image quality defects that can be performed on the reference video data. Therefore, after performing a degradation operation on the video data, a degradation operation can be performed on the image in the video after the degradation operation to obtain video data with image quality defects.

[0116] The degradation processing or degradation operation in this embodiment can be understood as one or more operations such as scaling, blurring, adding noise, adding sharp white edges, and compression, and can be performed on both the video and the images within the video. In this embodiment, the degradation processing can be performed multiple times, in a random order, and with random parameters, to obtain an image with reduced image quality and image quality defects after the degradation processing.

[0117] In another implementation, the degradation operation can also be performed based on the identification of the generation type of the reference video data. For example, when the reference video data is an original self-shot video (i.e., an original video shot and produced by the user), degradation operations such as blurring and noise can be performed; when the reference video data is a reposted video (i.e., a video reposted between platforms), compression degradation operations can be performed; when the reference video data is an original edited video (i.e., a video edited from a non-original video and accompanied by original audio), degradation operations such as blurring, noise, compression, and sharpening white edges can be performed. Of course, the video type is not limited to the above examples, and the specific degradation operation can also be selected based on the video data processing requirements. The purpose of the degradation operation is to obtain an image with image quality problems, which is the training image required for training in this embodiment.

[0118] It is understandable that, in order to improve the generalization of the data processing model, this embodiment may not consider the simulation of image quality defects achieved by the generation type of reference video data. However, this does not rule out the possibility of such implementation in specific scenarios.

[0119] It should be noted that other possible implementation methods for simulating various image quality defects on reference video data or reference image data include: determining the simulation method of image quality defects through viewing feedback data of video data or image data, and then implementing corresponding processing of image quality defects.

[0120] Based on the above-mentioned degradation processing of the reference video data or reference image data, different image quality defects in the video data or image data in the simulated application scenario can be obtained, so that the image set to be trained can include a variety of rich and complex image quality defect information without relying on additional image quality detection operations, thereby providing a basis for the accuracy of subsequent model training, and also providing a basis for improving the effect and efficiency of video data or image data processing in actual application scenarios.

[0121] When training the convolutional neural network model, the reference image and the image in the image set to be trained are images with a corresponding relationship. The corresponding relationship can be that the two images are completely identical in expression content, and of course, the local content can also be the same. For example, when the training image is mainly trained for the edge, the reference image and the training image can be images with the same edge.

[0122] Based on the above, step S101 can be understood as constructing a set of images to be trained, that is, a data set with image quality defects. The reference image can be understood as an image whose image quality meets the requirements (such as a high-quality image). The reference image is used as the standard answer, and the images in the set of images to be trained are used as images that need to be trained and processed.

[0123] Regarding step S102: converting the images in the to-be-trained image set into the luminance and chromaticity space, and selecting the luminance channel image as the training sample.

[0124] The purpose of step S102 is to pre-process the images in the image set to be trained.

[0125] In this embodiment, the preprocessing may be performing color space conversion on the images in the to-be-trained image set. In this embodiment, the color space conversion may be converting the images in the to-be-trained image set from RGB color channels to YUV color channels, and splitting the converted YUV color channels into Y, U, and V channels. Y represents luminance (Luma), U and V represent chrominance (Chroma), representing blue difference and red difference, respectively. RGB colors include the three primary colors of red, green, and blue, where R represents red, G represents green, and B represents blue.

[0126] After channel conversion, the pixel values ​​of the image are normalized, that is, the pixel range is converted from integers between 0-255 to floating-point numbers between 0-1, which is beneficial to improving calculation accuracy, accelerating model training speed, and enhancing model performance. Accordingly, the brightness channel image after the convolutional neural network model training needs to be denormalized, that is, the pixel range is converted from floating-point numbers between 0-1 to integers between 0-255; then channel merging is performed, that is, the enhanced Y channel, the U channel after color space conversion, and the V channel after color space conversion are merged, and color space conversion is performed again, that is, the YUV channel is converted into RGB channels, so as to obtain the output image.

[0127] Color space conversion can be performed in a variety of ways, such as using a conversion formula, image processing software, or machine learning. Color space conversion is a well-known technique, and examples are not provided here.

[0128] In order to avoid the problem of color deviation in the iterative processing of the image after conversion, in this embodiment, the Y channel image, that is, the brightness channel image, is used as a training sample of the convolutional neural network model.

[0129] Regarding the step S103: inputting the training sample into the convolutional neural network model for iterative training;

[0130] The purpose of step S103 is to iteratively train the model based on the training sample images, and the loss function will be calculated during the iterative training process. In this embodiment, the components in the iterative training loss function can be implemented using steps S103-1 to S103-2. The iterative process in step S103 is described below.

[0131] like Figure 2 As shown, Figure 2 This is a schematic diagram of an embodiment of upsampling and downsampling in a training method for a video data processing model provided in this application.

[0132] Step S103-a: downsampling the training samples according to the set downsampling requirements, determining downsampled images corresponding to the downsampling requirements in sequence, and extracting downsampled image feature data from the downsampled images;

[0133] Step S103-b: upsampling the downsampled image according to an upsampling requirement corresponding to the downsampled requirement, determining an upsampled image corresponding to the upsampling requirement, and merging upsampled image feature data extracted from the upsampled image with the matched downsampled image features to determine a merged image;

[0134] Step S103-c: Determine the merged image as the output image of the convolutional neural network model.

[0135] The purpose of step S103-a is to downsample the training image. In this embodiment, the downsampling can be accomplished by an encoder in a CNN network model (convolutional neural network model). The encoder is located on the left side of the CNN network and is responsible for extracting image feature information at different resolutions. In this embodiment, image feature extraction is performed using two 3×3 convolutional layers as an example, and the extraction process is to extract image features from different downsampled images. The downsampling requirements in this embodiment may be requirements for sampling resolution, for example, downsampling from the original resolution to a resolution of one-half (1 / 2), one-quarter (1 / 4), and one-eighth (1 / 8) in sequence. Of course, this is not limited to the requirements for sampling resolution, and may also be based on size requirements, etc.

[0136] like Figure 2 As shown on the left, the specific implementation process of step S103-a may include:

[0137] Step S103-a1: downsample the training image according to the set first downsampling requirement, determine the first downsampled image, and extract the first downsampled image feature data; in this embodiment, the first downsampling requirement can be half of the original resolution of the training image, that is, the training image is downsampled according to the requirement of half resolution to obtain the first downsampled image, and the first downsampled image feature data is extracted from the first downsampled image.

[0138] Step S103-a2: downsample the first downsampled image according to the set second downsampled requirement, determine the second downsampled image, and extract the second downsampled image feature data; in this embodiment, the second downsampled requirement may be downsampling the first sampling image at one-quarter of the resolution to obtain the second downsampled image, and extract the second downsampled image feature data from the second downsampled image.

[0139] Step S103-a3: downsample the second downsampled image according to the set third downsampled requirement, determine the third downsampled image, and extract the feature data of the third downsampled image; in this embodiment, the third downsampled requirement may be to downsample the second downsampled image by one-eighth to obtain the third downsampled image, and extract the feature data of the third downsampled image from the third downsampled image.

[0140] It should be noted that downsampling refers to reducing the resolution or sampling rate of data by reducing the number of data samples. In image processing, it is used to reduce the image, reduce the amount of data or perform feature extraction. Downsampling methods include: simple averaging, averaging multiple pixel values ​​to obtain the downsampled pixel value. Maximum pooling, selecting the maximum value of multiple pixel values ​​as the downsampled pixel value, which is often used for feature extraction. Minimum pooling, selecting the minimum value of multiple pixel values ​​as the downsampled pixel value, this method may be useful in certain specific applications. Convolution downsampling, downsampling is achieved through convolution operation, and features are extracted at the same time. In this embodiment, downsampling is achieved through convolution, but the specific method used for downsampling is not limited to this method, and other sampling methods that can be performed according to the downsampling requirements may also be included.

[0141] Regarding step S103-b: according to the upsampling requirement corresponding to the downsampling requirement, the downsampled image is upsampled, the upsampled image corresponding to the upsampling requirement is determined, and the upsampled image feature data extracted from the upsampled image is merged with the matching downsampled image feature to determine a merged image.

[0142] The purpose of step S103-b is to upsample the training image. The upsampling can be accomplished by the decoder in the CNN network model. The encoder is on the right side of the CNN network, which is responsible for restoring images of different resolutions and extracting image feature information. Similar to downsampling, in this embodiment, two 3×3 convolutional layers are used as an example for upsampling and image feature extraction. The extraction process is to extract image features from different upsampled images. The upsampling requirements in this embodiment correspond to the downsampling requirements, that is, upsampling from 1 / 8 to 1 / 4, 1 / 2, and original resolution in sequence.

[0143] The specific implementation process of step S103-b includes at least two implementation methods.

[0144] Method 1 specifically includes:

[0145] Step S103-b-11: Upsample the third downsampled image according to the set first upsampling requirement to determine the first upsampled image corresponding to the second downsampled image; the first upsampling requirement may be upsampling according to one-fourth of the resolution of the third downsampled image (one-eighth of the resolution), that is, upsampling starts from the third downsampled image according to the requirement of one-fourth of the resolution to obtain the first upsampled image with a resolution of one-fourth, and the sampling resolution of the second downsampled image is the same as the sampling resolution of the first upsampled image, therefore, the first upsampled image and the second downsampled image correspond to each other.

[0146] Step S103-b-12: Merge the first up-sampled image feature data extracted from the first up-sampled image with the second down-sampled image feature data to determine first merged image feature data of the first merged image.

[0147] Step S103-b-13: Upsample the first merged image according to the set second upsampling requirement to determine a second upsampled image corresponding to the first downsampled image; in this embodiment, the second upsampling requirement may be to upsample the first merged image at half the resolution to obtain a second upsampled image, and the upsampling resolution of the second upsampled image is the same as the downsampling resolution of the first downsampled image, therefore, the second upsampled image corresponds to the first downsampled image.

[0148] Step S103-b-14: Merge the extracted second up-sampled image feature data in the second up-sampled image with the first down-sampled image feature data to determine second merged image feature data of the second merged image.

[0149] Step S103-b-15: Upsample the second merged image according to the set third upsampling requirement to determine the third upsampled image corresponding to the training image; in this embodiment, the resolution of the third upsampling requirement corresponds to the original resolution of the training image, therefore, upsample the second merged image and use the original resolution of the training image to obtain the third upsampled image.

[0150] Step S103-b-16: Merge the extracted third up-sampled image feature data in the third up-sampled image with the training image feature data of the training image to determine a target merged image.

[0151] Method 2 specifically includes:

[0152] Step S103-b-21: upsample the third downsampled image according to the set first upsampling requirement, and determine the first upsampled image corresponding to the second downsampled image; the first upsampling requirement in the second method is the same as the first upsampling requirement in the first method, and the first upsampling requirement may be upsampling according to one-fourth of the resolution of the third downsampled image (one-eighth of the resolution), that is, upsampling starts from the third downsampled image according to the requirement of one-fourth of the resolution to obtain the first upsampled image with a resolution of one-fourth, and the sampling resolution of the second downsampled image is the same as the sampling resolution of the first upsampled image, therefore, the first upsampled image and the second downsampled image correspond to each other.

[0153] Step S103-b-22: Upsample the first upsampled image according to the set second upsampling requirement to determine a second upsampled image corresponding to the first downsampled image; the second upsampling requirement in the second method is the same as the second upsampling requirement in the first method. In this embodiment, the second upsampling requirement can be upsampling the first upsampled image in the step S103-2-21 at half the resolution to obtain a second upsampled image. The upsampling resolution of the second upsampled image is the same as the downsampling resolution of the first downsampled image. Therefore, the second upsampled image corresponds to the first downsampled image.

[0154] Step S103-b-23: Upsample the second upsampled image according to the set third upsampling requirement to determine the third upsampled image corresponding to the training image; the third upsampling requirement in the second method is the same as the third upsampling requirement in the first method. In this embodiment, the resolution of the third upsampling requirement corresponds to the original resolution of the training image. Therefore, the second upsampled image is upsampled with the original resolution of the training image to obtain the third upsampled image.

[0155] Step S103-b-24: merging the first up-sampled image feature data extracted from the first up-sampled image and the second down-sampled image feature data extracted from the second down-sampled image to determine first merged image feature data;

[0156] Step S103-b-25: merging the extracted second up-sampled image feature data of the second up-sampled image with the extracted first down-sampled image feature data of the first down-sampled image to determine second merged image feature data;

[0157] Step S103-b-26: merging the extracted third up-sampled image feature data of the third up-sampled image with the extracted training image feature data of the training image to determine third merged image feature data;

[0158] Step S103-b-27: Merge the first merged image feature data, the second merged image feature data, and the third merged image feature data to determine the target merged image.

[0159] Regardless of method one or method two, the merging can be to merge the image feature data based on the number of channels in the feature space, for example: adding the number of channels of the two feature maps, that is, merging the three-dimensional feature map of size (channel number C1, height H, width W) with the three-dimensional feature map of size (channel number C2, height H, width W) into a three-dimensional feature map of size (C1+C2, H, W).

[0160] It should be noted that upsampling refers to increasing the resolution or sampling rate of data by increasing the number of data samples. In image processing, it is used to enlarge the image or improve the details of the image. The upsampling methods may include: nearest neighbor interpolation: select the known pixel value closest to the point to be interpolated as the interpolation result. Bilinear interpolation: based on the four known pixel values ​​around the point to be interpolated, the interpolation result is obtained by linear interpolation. Bicubic interpolation: considering the 16 known pixel values ​​around the point to be interpolated, the interpolation result is obtained by cubic polynomial interpolation. Transposed convolution (also called deconvolution): This is a method of achieving upsampling through convolution operation, which is commonly used in image generation tasks in deep learning. In this embodiment, upsampling is achieved through convolution, but the specific method used for upsampling is not limited to this method, and other sampling methods that can be performed according to the upsampling requirements may also be included.

[0161] The above describes the process of learning and training the input image during the iterative process of step S103, thereby obtaining an output image corresponding to the input image, namely, a merged image (the target merged image in the above specific embodiment). The convolutional neural network model needs to determine the loss function for the merged image (output image) and the standard image, namely, the execution of steps S103-1 and S103-2.

[0162] Regarding step S103 - 1 : determining the negative optimization portion of the output image according to the difference between the output image and the input image of the current iteration.

[0163] The purpose of step S103-1 is to determine the negative optimization part of the current iterative output image of the data processing model. The negative optimization part can be understood as: the area where the current iterative output image has image quality defects compared with the input image. For example, the clarity of the upper right corner of the output image is lower than that of the upper right corner of the input image, indicating that the iteration does not improve the image quality of the upper right corner area but reduces it. Therefore, this part of the area belongs to the negative optimization part.

[0164] In this embodiment, the input image can be the image that is first input to the data processing model, that is, the negative optimization part is determined by the difference between the first input image and the output image of the current iteration; or the output image of the previous iteration is used as the input image of the model, and the negative optimization part is determined according to the difference between the input image of the previous iteration and the output image of the current iteration, and so on.

[0165] like Figure 3 As shown, Figure 3 This is a schematic diagram of an embodiment of determining difference loss data in a model process in a training method for a data processing model provided by the present application. The specific implementation process of step S103-1 may include:

[0166] Step S103-11: Compare the output image and the input image pixel by pixel, mark the area where the contrast of the output image pixel is less than that of the input image pixel as a first mark, and mark the area where the contrast of the output image pixel is greater than or equal to that of the input image pixel as a second mark; for example, the first mark is 1 and the second mark is 0; this can be represented by the following formula:

[0167]

[0168] in, Represents a binary mask image; the area with decreased contrast compared to the x-1 iteration is set to 1, and the other areas are set to 0; when x=0, is a mask image with all 0s, Represents the weight ratio of iterative loss; Represents a standard image; Represents the output image of the current iteration.

[0169] Step S103-12: Determine the image including the first mark and the second mark as a mask image; that is, obtain a mask image including 0 and 1;

[0170] Step S103 - 13 : Determine the area of ​​the first mark in the mask image as the negative optimization part of the output image.

[0171] It is understood that in this embodiment, the negative optimization portion in the output image is detected by comparing the input image and the output image pixel by pixel. Alternatively, the input image and the output image can be compared using frequency domain analysis or image gradient information. In order to better train the convolutional neural network model for the negative optimization portion during the iterative training process, step S103-2 can be performed.

[0172] Regarding step S103 - 2 : determining the difference between the image including the negative optimized part and the standard image corresponding to the input image as one of the components of the loss function.

[0173] The purpose of step S103-2 is to determine the components of the loss function, that is, the loss function of the convolutional neural network model in this embodiment may include multiple components of the loss function, which can be determined by the difference between the image including the negative optimization part and the standard image corresponding to the input image. The specific implementation process may include multiple methods.

[0174] Method 1:

[0175] Step S103-2-11: Determine a first pixel value of the negative optimized portion in the standard image by comparing the image of the negative optimized portion with the standard image pixel by pixel;

[0176] Step S103-2-12: Determine a second pixel value of the negative optimized portion in the output image by comparing the image of the negative optimized portion with the output image pixel by pixel;

[0177] Step S103-2-13: Determine the average value of the difference between the first pixel value and the second pixel value as the pixel difference component of the loss function.

[0178] When pixel difference is used as a component of the loss function, the following formula can be used:

[0179]

[0180] in, represents the pixel difference loss component; represents the output image, represents a standard image, represents the image including the negative optimization part, m represents the total number of pixels, Represents the input image.

[0181] Assume that the output image of the convolutional neural network model at iteration x-1 is Ox-1, and the output image at iteration x (i.e., the current iteration) is Ox. By comparing the pixel contrast between Ox-1 and Ox, we can determine which areas of Ox have lower clarity than Ox-1, which areas of Ox have lower detail than Ox-1, and so on. These areas can be labeled as 1, meaning that areas where Ox's image quality is lower than Ox-1 are labeled as 1, and areas where Ox's image quality is higher than Ox-1 are labeled as 0. This allows us to identify areas of the output image that require targeted optimization in the current iteration, resulting in an image (diff image) that includes the negatively optimized portion. Assuming that the areas with a value of 1 in the difference image (i.e., areas where Ox's image quality is lower than Ox-1) are the top left and bottom right corners, we can calculate the pixel difference loss between the top left corner of Ox and the top left corner of the standard image, and the pixel difference loss between the bottom right corner of Ox and the bottom right corner of the standard image. During training, the CNN model learns how to minimize the difference between the output image and the standard image (i.e., the smaller the loss between the output image and the standard image, the higher the output image quality and the clearer the image). By comparing pixels, the CNN model can determine which negative optimization areas should be prioritized in the xth iteration. The model then calculates the pixel difference loss between the image containing the negative optimization areas and the standard image. By making the pixel difference loss values ​​larger than those of other areas, the CNN model autonomously learns that areas with large loss values ​​are areas of focus, thereby improving the clarity of these negative optimization areas during training. This ensures that the number of CNN model iterations is proportional to the quality of the output image, ensuring the stability of the CNN model's output results. This process can also be applied to the following processes related to feature difference loss.

[0182] The above is the pixel difference loss component of the output image at the current iteration determined at the pixel level.

[0183] Method 2:

[0184] Step S103-2-21: Determining a first eigenvalue of the negative optimized portion in the output image by performing feature matching on the image of the negative optimized portion and the output image; the feature matching may be a pixel-by-pixel comparison to extract image features of the negative optimized portion, thereby determining the first eigenvalue of the negative optimized portion in the output image;

[0185] Step S103-2-22: Determine a second eigenvalue of the negative optimized portion in the standard image by performing feature matching on the image of the negative optimized portion and the standard image; the feature matching may be a pixel-by-pixel comparison to extract image features of the negative optimized portion, thereby determining the second eigenvalue of the negative optimized portion in the standard image;

[0186] Step S103-2-23: Determine the average value of the difference between the first eigenvalue and the second eigenvalue as the characteristic difference component of the loss function.

[0187] In this embodiment, the feature difference component of the loss function can be expressed by the following formula:

[0188]

[0189] in, Represents the feature difference loss component; Represents the features in the feature space of the standard image extracted by the VGG network; represents the features in the feature space of the output image extracted by the VGG network.

[0190] In this embodiment, the calculation process of the feature difference loss (ie, the feature difference loss of the output image) may refer to the calculation process of the pixel difference loss, which will not be described in detail here.

[0191] Method 3:

[0192] Step S103-2-31: Determine a first pixel value of the negative optimized portion in the standard image by comparing the image of the negative optimized portion with the standard image pixel by pixel;

[0193] Step S103-2-32: Determine a second pixel value of the negative optimized portion in the output image by comparing the image of the negative optimized portion with the output image pixel by pixel;

[0194] Step S103-2-33: Determine the average value of the difference between the first pixel value and the second pixel value as the pixel difference component of the loss function;

[0195] Step S103-2-34: Determining a first eigenvalue of the negative optimized portion in the output image by performing feature matching on the image of the negative optimized portion and the output image; the feature matching may be a pixel-by-pixel comparison to extract image features of the negative optimized portion, thereby determining the first eigenvalue of the negative optimized portion in the output image;

[0196] Step S103-2-35: Determining a second eigenvalue of the negative optimized portion in the standard image by performing feature matching on the image of the negative optimized portion and the standard image; the feature matching may be a pixel-by-pixel comparison to extract image features of the negative optimized portion, thereby determining the second eigenvalue of the negative optimized portion in the standard image;

[0197] Step S103-2-36: Determine the difference between the first eigenvalue and the second eigenvalue as the characteristic difference component of the loss function;

[0198] Step S103-2-37: Determine the pixel difference component and the feature difference component as components of the loss function.

[0199] The third approach mainly uses both the first and second approaches as components of the loss function. In the first approach, the loss function component is the pixel difference loss component, while in the second approach, the loss function component is the feature difference loss component. However, both approaches can be used as components of the loss function.

[0200] In this embodiment, the components of the loss function may further include: one or more components of pixel loss, feature loss, and discrimination loss, which can be obtained by comparing the output image with the standard image. The loss function can be expressed by the following formula:

[0201] ;

[0202] ;

[0203] ;

[0204] The above formulas are representations of three embodiments and are not limited to the above representations. The components of the loss function can be adjusted according to needs.

[0205] in, (pixel loss component), (feature loss component) and (Discrimination loss component) is also a component of the loss function, Represents weight.

[0206] About the pixel loss in the loss function The specific implementation process of the component may include:

[0207] Comparing the output image and the standard image pixel by pixel to determine pixel differences between the output image and the standard image;

[0208] A pixel loss of the output image is determined based on the pixel differences.

[0209] In this embodiment, the pixel loss component can be expressed by the following formula:

[0210] .

[0211] About the feature loss in the loss function The specific implementation process of the component may include:

[0212] comparing the feature data of the output image with the feature data of the standard image to determine feature differences between the feature data of the output image and the standard image;

[0213] A feature loss of the output image is determined according to the feature difference.

[0214] In this embodiment, the characteristic loss component can be expressed by the following formula:

[0215] .

[0216] About the discriminant loss in the loss function The specific implementation process of the component may include:

[0217] Inputting the output image into a discriminator model for discrimination and determining a discrimination result, wherein the discriminator model is a model that has been trained in advance;

[0218] Determining the discrimination loss according to the discrimination result;

[0219] In this embodiment, the discrimination loss component can be expressed by the following formula:

[0220]

[0221] in, The discriminator model can discriminate images that meet the high quality requirements of the image (high-quality images) as 1, and images that do not meet the image quality requirements (low-quality images) as 0. When training the convolutional neural network model, the output image can be measured by the discriminant loss. Is it a high-quality image? In this embodiment, the discriminator model It can be a trained model. The loss function of the discriminator model can be expressed as follows:

[0222]

[0223] in, is the discriminant loss function during the training of the discriminator model, The discriminator model is required to classify the standard image that meets the image quality requirements as 1. The discriminator model is required to classify the output image processed by CNN as 0 if the image quality is lower than the standard image.

[0224] It should be noted that in this embodiment The discriminant loss function is determined by discriminating the output image based on the trained discriminator model.

[0225] Based on the above content, it can be seen that when determining the loss function according to any of the above loss function components When , any component can be weighted, or each component can be weighted. The specific process may include:

[0226] When the components of the loss function are the pixel difference component and / or the feature difference component, the weight ratio of the pixel difference component and / or the feature difference component is greater than the weight ratio of the third component in the loss function. The third component can be a pixel loss component, a feature loss component, a discrimination loss component, etc., that is: and / or Can be greater than 、 、 .

[0227] The weight ratio between the pixel difference component and the feature difference component can be determined based on the respective proportions of the pixel difference component and the feature difference component in the output image. For example, if the proportion of the pixel difference component in the output image is greater than the proportion of the feature difference component in the output image, the weight of the pixel difference component can be set greater than the weight of the feature difference component, so that the CNN model pays more attention to the pixel difference area corresponding to the pixel difference component.

[0228] It is understood that the weights of any of the above loss function components can be adjusted in real time based on the needs of model training and the input and output images, and are not limited to the above examples. Therefore, the weight setting can be a dynamic weight that is dynamically adjusted based on the number of iterations, output results, etc.

[0229] In this embodiment, after obtaining the components of the above-mentioned loss function and thus determining the loss function value, the CNN model can be adjusted so that the image quality of the output image output by the model training process gradually approaches (or is equal to) the image quality of the standard image.

[0230] The above is a description of an embodiment of a training method for a data processing model provided by the present application. The method performs color space conversion on images in a training image set and inputs the converted luminance channel images as training samples into a convolutional neural network model for iterative training, thereby avoiding image color deviation during the training process and ensuring the accuracy of image color. In addition, in order to avoid the situation where the image quality does not improve or even deteriorates with the increase in the number of iterations, causing the model to ignore these negative optimization areas, thereby causing the model output results to fail to meet higher requirements, the method can determine the negative optimization part included in the output image by the difference between the current iteration output image and the current iteration input image, determine the components of the loss function for the image including the negative optimization part and the standard image, so that the model can self-learn the areas of these negative optimization parts, thereby improving the quality of the output image and further enhancing the model's ability to retain image details.

[0231] In addition, this training method can obtain a rich and diverse set of images to be trained by simulating various image quality defect problems, and can complete the training of more complex image quality defect problems through a separate data processing model of this application. Since the training link is simple, the training computing power cost and computing time can be effectively reduced, and there is no need to use multiple detection models to detect image quality defect types in order to deal with complex image quality problems.

[0232] The above is a detailed description of an embodiment of a training method for a data processing model provided by this application. Corresponding to the embodiment of a training method for a data processing model provided above, this application also discloses an embodiment of a training device for a video data processing model. Please refer to Figure 4 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0233] like Figure 4 As stated, Figure 4 This is a schematic diagram of the structure of a data processing model training device provided by the present application, which includes:

[0234] An acquisition unit 401 is used to acquire an image set to be trained;

[0235] The conversion unit 402 is used to convert the images in the image set to be trained into the luminance and chrominance space, and select the luminance channel image as the training sample;

[0236] The training unit 403 is configured to input the training samples into a convolutional neural network model for iterative training, wherein the loss function of the iterative training includes at least a component determined as follows:

[0237] A first determining subunit 403-1 is configured to determine a negatively optimized portion of the output image according to a difference between the output image of a current iteration and the input image of the current iteration;

[0238] The second determining subunit 403 - 2 is configured to determine the difference between the image including the negative optimized portion and the standard image corresponding to the input image as one of the components of the loss function.

[0239] When the image set to be trained is derived from reference video data, the acquisition unit 401 includes:

[0240] a first processing subunit, configured to perform a first degradation process on the reference video data, and determine the video data after the first degradation process as the first video data;

[0241] a second processing subunit, configured to perform a second degradation process on the frame image in the first video data, and convert the frame image after the second degradation process into second video data;

[0242] An acquisition subunit is used to acquire the image set to be trained based on the second video data; specifically, it may include: an extraction subunit is used to randomly extract frame images from the second video data; and a determination subunit is used to determine the frame images as the image set to be trained.

[0243] When the image set to be trained is derived from a reference image set, obtaining the image set to be trained includes:

[0244] an exit processing subunit, configured to perform degradation processing on the reference image set, and determine the image set after degradation processing as a degraded image set;

[0245] The acquisition subunit is configured to acquire the to-be-trained image set based on the degraded image set.

[0246] The first determining subunit 403-1 may include:

[0247] a marking subunit, configured to compare the output image with the input image pixel by pixel, mark a region where the pixel contrast of the output image is smaller than the pixel contrast of the input image with a first mark, and mark a region where the pixel contrast of the output image is greater than or equal to the contrast of the input image with a second mark;

[0248] a mask determination subunit, configured to determine the image including the first mark and the second mark as a mask image;

[0249] The negative optimization determination subunit is configured to determine the area of ​​the first mark in the mask image as the negative optimization part of the output image.

[0250] The second determining subunit 403-2 may include at least three of the following methods:

[0251] Method 1 includes:

[0252] a first pixel value determining subunit, configured to determine a first pixel value of the negative optimized portion in the standard image by comparing the image of the negative optimized portion with the standard image pixel by pixel;

[0253] a second pixel value determining subunit, configured to determine a second pixel value of the negative optimized portion in the output image based on a pixel-by-pixel comparison of the image of the negative optimized portion and the output image;

[0254] The pixel difference determining subunit is configured to determine an average value of the difference between the first pixel value and the second pixel value as a pixel difference component of the loss function.

[0255] Method 2 includes:

[0256] a first eigenvalue determining subunit, configured to determine a first eigenvalue of the negative optimized portion in the output image by performing feature matching between the image of the negative optimized portion and the output image;

[0257] a second eigenvalue determining subunit, configured to determine a second eigenvalue of the negative optimized portion in the standard image by performing feature matching on the image of the negative optimized portion and the standard image;

[0258] The feature difference determination subunit is used to determine the average value of the difference between the first eigenvalue and the second eigenvalue as the feature difference component of the loss function.

[0259] Method three includes:

[0260] a first pixel value determining subunit, configured to determine a first pixel value of the negative optimized portion in the standard image by comparing the image of the negative optimized portion with the standard image pixel by pixel;

[0261] a second pixel value determining subunit, configured to determine a second pixel value of the negative optimized portion in the output image based on a pixel-by-pixel comparison of the image of the negative optimized portion and the output image;

[0262] a pixel difference determining subunit, configured to determine an average value of a difference between the first pixel value and the second pixel value as a pixel difference component of the loss function;

[0263] a first eigenvalue determining subunit, configured to determine a first eigenvalue of the negative optimized portion in the output image by performing feature matching between the image of the negative optimized portion and the output image;

[0264] a second eigenvalue determining subunit, configured to determine a second eigenvalue of the negative optimized portion in the standard image by performing feature matching on the image of the negative optimized portion and the standard image;

[0265] a feature difference determining subunit, configured to determine an average value of a difference between the first eigenvalue and the second eigenvalue as a feature difference component of the loss function;

[0266] The component determination subunit is configured to determine the pixel difference component and the feature difference component as one of the components of the loss function respectively.

[0267] The method further includes: a first weight setting unit, configured to, when the components of the loss function are the pixel difference component and / or the feature difference component, increase the weight of the pixel difference component and / or the feature difference component to be greater than the weight of the third component in the loss function.

[0268] The method further includes: a second weight setting unit, configured to determine the weights of the pixel difference component and the feature difference component according to the respective proportions of the pixel difference component and the feature difference component in the standard image.

[0269] The above is a description of a data processing model training device provided in this application. For the specific content of the device, please refer to the above-mentioned training method embodiment and will not be described in detail here.

[0270] Based on the above, this application also provides a video data processing method, such as Figure 5 As shown, Figure 5 This is a flow chart of a video data processing method provided by the present application, the method comprising:

[0271] Step S501: Read the video data to be processed frame by frame to determine the video frame image to be processed;

[0272] Step S502: performing color space conversion on the video frame image to be processed, and selecting a luminance channel image as an input frame image;

[0273] Step S503: inputting the input frame image into the model trained according to the training method of the data processing model for processing to obtain an output frame image;

[0274] Step S504: performing channel merging on the output frame image, and converting the channel merging image into an image of three primary color channels using the color space;

[0275] Step S505: merging the images into a video sequence according to the video frame sequence of the video data to be processed to obtain target video data.

[0276] The video data processing method can retain more details, so that the target video data has a clearer and more natural picture enhancement effect.

[0277] The specific contents of the above steps S501 to S505 can be referred to the relevant contents in the training method of the data processing model, which will not be described in detail here.

[0278] Based on the above content, this application also provides a video data processing device, such as Figure 6 As shown, Figure 6 : is a structural diagram of a video data processing device provided by the present application, the device comprising:

[0279] A determination unit 601 is configured to read the video data to be processed frame by frame to determine a video frame image to be processed;

[0280] The first conversion unit 602 is configured to perform color space conversion on the video frame image to be processed and select a luminance channel image as an input frame image;

[0281] The processing unit 603 is used to input the input frame image into the model trained according to the training method of the data processing model to process it and obtain an output frame image;

[0282] A second conversion unit 604 is configured to perform channel merging on the output frame image, and convert the channel-merged image into an image of three primary color channels using the color space;

[0283] The obtaining unit 605 is configured to merge the images into a video sequence according to the video frame sequence of the video data to be processed, so as to obtain target video data.

[0284] Based on the above content, the present application also provides a computer storage medium, including a computer program. When the computer program is run on an electronic device, the electronic device executes the steps in the training method of the data processing model; or, executes the steps in the video data processing method.

[0285] Based on the above content, the present application also provides an electronic device, such as Figure 7 As shown, Figure 7 The structural diagram of an electronic device provided by this application includes:

[0286] Processor 701;

[0287] Memory 702 is used to store a program for processing data generated by an electronic device. When the program is read and executed by the processor, it executes the steps in the training method of the data processing model; or, it executes the steps in the video data processing method.

[0288] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0289] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0290] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0291] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0292] 2. Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0293] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A method for training a data processing model, characterized in that: include: Acquire a set of images to be trained, wherein the training image set includes a variety of image quality defect information; Converting the images in the to-be-trained image set into a luminance and chrominance space, and selecting luminance channel images as training samples; The training samples are input into a data processing model for iterative training, wherein the loss function of the iterative training includes at least a component determined by the following method: determining a negatively optimized portion of the output image based on a difference between an output image of a current iteration and an input image; The difference between the image including the negative optimization part and the standard image corresponding to the input image is determined as one of the components of the loss function, wherein the difference between the image including the negative optimization part and the standard image corresponding to the input image includes: performing pixel comparison and / or feature matching between the image of the negative optimization part and the standard image, and performing pixel comparison and / or feature matching between the image of the negative optimization part and the output image to determine the difference.

2. The data processing model training method according to claim 1, characterized in that: The step of determining the negative optimization portion of the output image according to the difference between the output image and the input image of the current iteration includes: Comparing the output image with the input image pixel by pixel, marking a region where the contrast ratio of the output image is smaller than that of the input image pixel by pixel, and marking a region where the contrast ratio of the output image is greater than or equal to that of the input image pixel by pixel; determining an image including the first mark and the second mark as a mask image; The area of ​​the first mark in the mask image is determined as a negatively optimized portion of the output image.

3. The data processing model training method according to claim 1, characterized in that: Determining the difference between the image including the negative optimization part and the standard image corresponding to the input image as one of the components of the loss function includes: Determining a first pixel value of the negative optimized portion in the standard image by comparing the image of the negative optimized portion with the standard image pixel by pixel; Determining a second pixel value of the negative optimized portion in the output image by comparing the image of the negative optimized portion with the output image pixel by pixel; An average value of the difference between the first pixel value and the second pixel value is determined as a pixel difference component of the loss function.

4. The data processing model training method according to claim 1, characterized in that: Determining the difference between the image including the negative optimization part and the standard image corresponding to the input image as one of the components of the loss function includes: Determining a first eigenvalue of the negative optimized portion in the output image by performing feature matching on the image of the negative optimized portion and the output image; Determining a second eigenvalue of the negative optimized portion in the standard image by performing feature matching on the image of the negative optimized portion and the standard image; An average value of the difference between the first eigenvalue and the second eigenvalue is determined as a feature difference component of the loss function.

5. The data processing model training method according to claim 1, characterized in that: Determining the difference between the image including the negative optimization part and the standard image corresponding to the input image as one of the components of the loss function includes: Determining a first pixel value of the negative optimized portion in the standard image by comparing the image of the negative optimized portion with the standard image pixel by pixel; Determining a second pixel value of the negative optimized portion in the output image by comparing the image of the negative optimized portion with the output image pixel by pixel; determining an average value of the difference between the first pixel value and the second pixel value as a pixel difference component of the loss function; Determining a first eigenvalue of the negative optimized portion in the output image by performing feature matching on the image of the negative optimized portion and the output image; Determining a second eigenvalue of the negative optimized portion in the standard image by performing feature matching on the image of the negative optimized portion and the standard image; Determine the difference between the first eigenvalue and the second eigenvalue as a characteristic difference component of the loss function; The pixel difference component and the feature difference component are respectively determined as one of the components of the loss function.

6. The data processing model training method according to claim 4 or 5, characterized in that: Also includes: When the components of the loss function are the pixel difference component and / or the feature difference component, the weight ratio of the pixel difference component and / or the feature difference component is greater than the weight ratio of the third component in the loss function.

7. The data processing model training method according to claim 4 or 5, characterized in that: Also includes: The weights of the pixel difference component and the feature difference component are determined according to respective proportions of the pixel difference component and the feature difference component in the standard image.

8. The data processing model training method according to claim 1, characterized in that: When the image set to be trained is derived from reference video data, obtaining the image set to be trained includes: performing a first degradation process on the reference video data, and determining the video data after the first degradation process as the first video data; performing a second degradation process on the frame image in the first video data, and converting the frame image after the second degradation process into second video data; Acquire the image set to be trained according to the second video data; When the image set to be trained is derived from a reference image set, obtaining the image set to be trained includes: performing degradation processing on the reference image set, and determining the image set after the degradation processing as a degraded image set; The to-be-trained image set is acquired according to the degraded image set.

9. The data processing model training method according to claim 8, characterized in that: The step of determining the to-be-trained image set according to the second video data includes: Randomly extracting a frame image from the second video data; The frame image is determined as the image set to be trained.

10. A data processing model training device, characterized in that: include: An acquisition unit, configured to acquire a set of images to be trained, wherein the training image set includes a variety of image quality defect information; A conversion unit, configured to convert images in the image set to be trained into a luminance and chrominance space, and select luminance channel images as training samples; A training unit is configured to input the training samples into a data processing model for iterative training, wherein the loss function of the iterative training includes at least a component determined by: a first determining subunit, configured to determine a negatively optimized portion of the output image according to a difference between the output image and the input image of a current iteration; The second determination subunit is used to determine the difference between the image including the negative optimization part and the standard image corresponding to the input image as one of the components of the loss function, wherein the difference between the image including the negative optimization part and the standard image corresponding to the input image includes: performing pixel comparison and / or feature matching between the image of the negative optimization part and the standard image, and performing pixel comparison and / or feature matching between the image of the negative optimization part and the output image to determine the difference.

11. A video data processing method, characterized in that: include: Read the video data to be processed frame by frame to determine the video frame image to be processed; Performing color space conversion on the video frame image to be processed, and selecting a brightness channel image as an input frame image; Inputting the input frame image into a model trained by the data processing model training method according to any one of claims 1 to 9 for processing to obtain an output frame image; Performing channel merging on the output frame image, and converting the channel-merged image into an image of three primary color channels through the color space; The images are merged into a video sequence according to the video frame order of the video data to be processed to obtain target video data.

12. A video data processing device, characterized in that: include: A determination unit, configured to read the video data to be processed frame by frame and determine the video frame image to be processed; A first conversion unit is configured to perform color space conversion on the video frame image to be processed and select a luminance channel image as an input frame image; a processing unit, configured to input the input frame image into a model trained according to the data processing model training method according to any one of claims 1 to 9 for processing, to obtain an output frame image; A second conversion unit is configured to perform channel merging on the output frame image, and convert the channel-merged image into an image of three primary color channels through the color space; The obtaining unit is used to merge the images into a video sequence according to the video frame sequence of the video data to be processed to obtain target video data.

13. A computer storage medium, characterized in that The method comprises a computer program, which, when executed on an electronic device, causes the electronic device to execute the steps of the method for training a data processing model according to any one of claims 1 to 9; Alternatively, execute the steps in the video data processing method as claimed in claim 11.

14. An electronic device, characterized in that: include: processor; a memory, configured to store a program for processing data generated by an electronic device, wherein when the program is read and executed by the processor, the program executes the steps of the method for training a data processing model according to any one of claims 1 to 9; Alternatively, execute the steps in the video data processing method as claimed in claim 11.

Images