3D lookup table training methods and video enhancement methods

Abstract

This application relates to a three-dimensional lookup table training method and a video enhancement method. The three-dimensional lookup table training method includes: acquiring a training image set, a validation image set, and an initial three-dimensional lookup table, wherein the training images in the training image set carry scene category labels; extracting image features from each training image to obtain image feature information corresponding to the training images; classifying the training images into scenes based on the image feature information to obtain predicted scene classification results, and inputting the image feature information into the initial three-dimensional lookup table to obtain predicted enhanced images; obtaining a loss function based on the scene category labels, predicted scene classification results, validation images, and predicted enhanced images; and returning to the step of extracting image features from each training image to obtain image feature information corresponding to the training images, until the loss function converges, resulting in the target three-dimensional lookup table. This scheme can improve video enhancement effects and real-time performance.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a three-dimensional lookup table training method and a video enhancement method. Background Technology

[0002] With the development of image processing technology, image enhancement technology has emerged. Image enhancement technology can enhance the useful information in an image, with the aim of improving the visual effect and increasing the image's clarity. Image enhancement technology can be applied to video enhancement.

[0003] In traditional techniques, video enhancement involves first extracting frames from the video to obtain corresponding video frames. These frames are then sequentially input into a pre-trained convolutional neural network (CNN) to enhance the video. Finally, the enhanced video frames are aggregated to achieve video enhancement. Specifically, when using a pre-trained CNN to enhance video frames, the main approach is to downsample the full-resolution input image using the CNN, extract features from the downsampled low-resolution image to learn a 3D look-up table (3D LUT), and then transform each input pixel of the input image according to the input-output pixel mapping relationship in the 3D LUT to obtain the enhanced result.

[0004] However, in traditional methods, the 3D lookup table is learned by downsampling video frames. This lookup table corresponds to a lower resolution, resulting in a significant loss of enhancement when applied to the original resolution video frames, leading to poor video enhancement. If a convolutional neural network is directly used to process the entire video frame at full resolution, the processing speed decreases with increasing frame resolution. This results in poor real-time video enhancement when the frame resolution is too high. Summary of the Invention

[0005] Therefore, it is necessary to provide a three-dimensional lookup table training method, as well as a video enhancement method, apparatus, computer equipment, and storage medium that can improve the video enhancement effect and real-time performance, in response to the above-mentioned technical problems.

[0006] A method for training a three-dimensional lookup table, the method comprising:

[0007] Obtain the training image set, the validation image set, and the initial 3D lookup table. The training images in the training image set carry scene category labels. The validation images in the validation image set correspond one-to-one with the training images in the training image set. The validation images are the augmented images corresponding to the training images.

[0008] Image features are extracted from the training images to obtain the image feature information corresponding to the training images;

[0009] Based on image feature information, the training image is classified into scenes to obtain the predicted scene classification result. The image feature information is then input into an initial three-dimensional lookup table to obtain the predicted enhanced image.

[0010] The loss function is obtained based on the scene category label, the predicted scene classification result, the validation image, and the predicted augmented image.

[0011] Return to the steps of extracting image features from each training image to obtain the image feature information corresponding to the training image, until the loss function converges and the target 3D lookup table is obtained.

[0012] In one embodiment, the initial three-dimensional lookup table includes three-dimensional sub-lookup tables, the number of which corresponds to the dimension of the image feature information.

[0013] In one embodiment, inputting image feature information into an initial 3D lookup table to obtain a predicted enhanced image includes:

[0014] Based on the feature information of each dimension in the image feature information, determine the lookup table weight of the three-dimensional sub-lookup table;

[0015] The three-dimensional sub-lookup table is weighted according to the lookup table weights to obtain the weighted three-dimensional lookup table;

[0016] The training images are input into a weighted 3D lookup table to obtain the predicted enhanced images.

[0017] In one embodiment, the loss function obtained based on the scene category label, the predicted scene classification result, the verification image, and the predicted augmented image includes:

[0018] The first loss function is calculated based on the scene category label and the predicted scene classification result, and the second loss function is calculated based on the verification image and the predicted augmented image.

[0019] The loss function is obtained based on the first loss function and the second loss function.

[0020] In one embodiment, calculating the second loss function based on the verification image and the predicted augmented image includes:

[0021] Determine the pixel correspondence between the first pixel in the verification image and the second pixel in the predicted augmented image;

[0022] Based on the pixel correspondence, determine the pixel difference between the first pixel and the corresponding second pixel;

[0023] The second loss function is calculated based on the pixel differences.

[0024] A three-dimensional lookup table training device, the device comprising:

[0025] The acquisition module is used to acquire the training image set, the validation image set, and the initial 3D lookup table. The training images in the training image set carry scene category labels. The validation images in the validation image set correspond one-to-one with the training images in the training image set. The validation images are the augmented images corresponding to the training images.

[0026] The feature extraction module is used to extract image features from the training images to obtain image feature information corresponding to the training images;

[0027] The first processing module is used to classify the training image into a scene based on the image feature information, obtain the predicted scene classification result, and input the image feature information into an initial three-dimensional lookup table to obtain the predicted enhanced image.

[0028] The loss function calculation module is used to obtain the loss function based on the scene category label, the predicted scene classification result, the verification image, and the predicted augmented image;

[0029] The second processing module is used to return the steps of extracting image features from the training images to obtain the image feature information corresponding to the training images, until the loss function converges and the target three-dimensional lookup table is obtained.

[0030] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0031] Obtain the training image set, the validation image set, and the initial 3D lookup table. The training images in the training image set carry scene category labels. The validation images in the validation image set correspond one-to-one with the training images in the training image set. The validation images are the augmented images corresponding to the training images.

[0032] Image features are extracted from the training images to obtain the image feature information corresponding to the training images;

[0033] Based on image feature information, the training image is classified into scenes to obtain the predicted scene classification result. The image feature information is then input into an initial three-dimensional lookup table to obtain the predicted enhanced image.

[0034] The loss function is obtained based on the scene category label, the predicted scene classification result, the validation image, and the predicted augmented image.

[0035] Return to the steps of extracting image features from each training image to obtain the image feature information corresponding to the training image, until the loss function converges and the target 3D lookup table is obtained.

[0036] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0037] Obtain the training image set, the validation image set, and the initial 3D lookup table. The training images in the training image set carry scene category labels. The validation images in the validation image set correspond one-to-one with the training images in the training image set. The validation images are the augmented images corresponding to the training images.

[0038] Image features are extracted from the training images to obtain the image feature information corresponding to the training images;

[0039] Based on image feature information, the training image is classified into scenes to obtain the predicted scene classification result. The image feature information is then input into an initial three-dimensional lookup table to obtain the predicted enhanced image.

[0040] The loss function is obtained based on the scene category label, the predicted scene classification result, the validation image, and the predicted augmented image.

[0041] Return to the steps of extracting image features from each training image to obtain the image feature information corresponding to the training image, until the loss function converges and the target 3D lookup table is obtained.

[0042] A video enhancement method, the method comprising:

[0043] Acquire the video data to be processed;

[0044] Extract frames from the video data to be processed to obtain a set of video frames corresponding to the video data to be processed.

[0045] The video frames in the video frame set are sequentially input into the target 3D lookup table to obtain the enhanced video frames corresponding to the video frames. The target 3D lookup table is obtained through the above 3D lookup table training method.

[0046] Enhanced video data is obtained from the enhanced video frames.

[0047] In one embodiment, the video frames in the video frame set are sequentially input into a target 3D lookup table to obtain the enhanced video frames corresponding to the video frames, including:

[0048] The video frames in the video frame set are sequentially input into the target 3D lookup table to obtain the lookup output results corresponding to the video frames.

[0049] When the video frame is not the first frame of the video, retrieve the previous video frame corresponding to the video frame and look up the output result.

[0050] The target search output result is obtained by performing a moving average on the search output result of the previous video frame and the search output result.

[0051] Based on the target search output, the video frames are enhanced and adjusted to obtain the enhanced video frames corresponding to the original video frames.

[0052] A video enhancement device, the device comprising:

[0053] The video acquisition module is used to acquire video data to be processed.

[0054] The frame extraction module is used to extract frames from the video data to be processed, and obtain a set of video frames corresponding to the video data to be processed.

[0055] The enhancement module is used to input the video frames in the video frame set into the target three-dimensional lookup table in sequence to obtain the enhanced video frames corresponding to the video frames. The target three-dimensional lookup table is obtained through the three-dimensional lookup table training method described above.

[0056] The frame processing module is used to obtain enhanced video data based on the enhanced video frames.

[0057] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0058] Acquire the video data to be processed;

[0059] Extract frames from the video data to be processed to obtain a set of video frames corresponding to the video data to be processed.

[0060] The video frames in the video frame set are sequentially input into the target 3D lookup table to obtain the enhanced video frames corresponding to the video frames. The target 3D lookup table is obtained through the above 3D lookup table training method.

[0061] Enhanced video data is obtained from the enhanced video frames.

[0062] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0063] Acquire the video data to be processed;

[0064] Extract frames from the video data to be processed to obtain a set of video frames corresponding to the video data to be processed.

[0065] The video frames in the video frame set are sequentially input into the target 3D lookup table to obtain the enhanced video frames corresponding to the video frames. The target 3D lookup table is obtained through the above 3D lookup table training method.

[0066] Enhanced video data is obtained from the enhanced video frames.

[0067] The aforementioned 3D lookup table training method, apparatus, computer equipment, and storage medium, after acquiring a training image set, a validation image set, and an initial 3D lookup table, extract image features from each training image to obtain image feature information corresponding to the training image. Based on the image feature information, scene classification is performed on the training images to obtain predicted scene classification results. The image feature information is then input into the initial 3D lookup table to obtain predicted enhanced images, thus achieving preliminary training of the initial 3D lookup table. By obtaining a loss function based on scene category labels, predicted scene classification results, validation images, and predicted enhanced images, the process of extracting image features from each training image is repeated until the loss function converges, resulting in the target 3D lookup table. This method, combined with scene classification training, yields a target 3D lookup table that can accurately achieve enhanced color grading, improving video enhancement effects. Furthermore, by using the target 3D lookup table to process video frames in the video data, the correspondence between the pixel values ​​of video frames and the pixel values ​​of enhanced video frames can be quickly obtained, accelerating video frame processing speed and improving the real-time performance of video enhancement.

[0068] The aforementioned video enhancement method, apparatus, computer equipment, and storage medium acquire video data to be processed, extract frames from the video data to obtain a set of video frames corresponding to the video data to be processed, input the video frames in the set of video frames sequentially into a target three-dimensional lookup table to obtain the enhanced video frames corresponding to the video frames, obtain enhanced video data based on the enhanced video frames, and process the video frames in the video data using the target three-dimensional lookup table. This can improve the video enhancement effect, quickly obtain the correspondence between the pixel values ​​of the video frames and the pixel values ​​of the enhanced video frames, speed up the processing speed of video frames, and improve the real-time performance of video enhancement. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating a three-dimensional lookup table training method in one embodiment;

[0070] Figure 2 This is a schematic diagram of a neural network in one embodiment;

[0071] Figure 3 This is a schematic diagram of a scene prediction network in one embodiment;

[0072] Figure 4 This is a flowchart illustrating a video enhancement method in one embodiment;

[0073] Figure 5 This is a flowchart illustrating the three-dimensional lookup table training method in another embodiment;

[0074] Figure 6 This is a structural block diagram of a three-dimensional lookup table training device in one embodiment;

[0075] Figure 7 This is a structural block diagram of a video enhancement device in one embodiment;

[0076] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0078] In one embodiment, such as Figure 1 As shown, a three-dimensional lookup table training method is provided. This embodiment illustrates the application of this method to a server. It is understood that this method can also be applied to terminals, and to systems including terminals and servers, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, portable wearable devices, panoramic cameras, action cameras, etc., and the server can be a standalone server or a server cluster composed of multiple servers. In this embodiment, the method includes the following steps:

[0079] Step 102: Obtain the training image set, the validation image set, and the initial 3D lookup table. The training images in the training image set carry scene category labels. The validation images in the validation image set correspond one-to-one with the training images in the training image set. The validation images are the augmented images corresponding to the training images.

[0080] The training image set includes training images, which are unprocessed images captured by cameras, mobile phones, or other imaging devices. For example, a training image could be a photograph taken with such a device, or a video frame extracted from a video recording. The validation image set includes validation images, which are augmented versions of the training images, obtained by color-correcting the training images. A 3D Look-Up Table (3D LUT) is a dataset structure that allows direct lookup of corresponding output data from input data. For example, based on the RGB (red, green, blue) values ​​of a training image, the corresponding output RGB values ​​can be found in the 3D lookup table. Scene category labels represent the scene category corresponding to the training image. For example, a scene category could be distinguished by brightness, such as daytime or nighttime.

[0081] Specifically, when training the 3D lookup table, the server retrieves the training image set, the validation image set, and the initial 3D lookup table from a pre-set database.

[0082] Step 104: Extract image features from the training images to obtain image feature information corresponding to the training images.

[0083] Image feature information refers to the information used to characterize image features. For example, image feature information can specifically refer to an N-dimensional vector used to characterize image features.

[0084] Specifically, the server obtains a neural network and uses it to extract image features from the training images, obtaining image feature information corresponding to the training images. The neural network consists of a downsampling layer, a convolutional layer, and an output layer. The downsampling layer is used to downsample the training images to reduce their size and mitigate the impact of training image resolution on training speed. The specific network architecture of the neural network can be configured as needed; this embodiment does not impose specific limitations here. For example, the network architecture of the neural network can be as follows: Figure 2 As shown, it includes a downsampling layer, a convolutional layer consisting of multiple convolutional networks (Conv+ReLU), and an output layer (Dropout+Conv).

[0085] Step 106: Based on the image feature information, classify the training image into a scene to obtain the predicted scene classification result, and input the image feature information into the initial three-dimensional lookup table to obtain the predicted enhanced image.

[0086] Scene classification refers to classifying the training images according to their corresponding scenes. For example, scene classification could specifically classify a training image as belonging to daytime or nighttime. The initial 3D lookup table refers to a 3D lookup table without parameter adjustments.

[0087] Specifically, the server can use image feature information to perform scene classification on training images, obtaining predicted scene classification results. During scene classification, the server can acquire a scene prediction network and use it to classify the training images. The scene prediction network includes fully connected layers and softmax layers. For example... Figure 3 As shown, the scene prediction network can be connected after the neural network. After the neural network outputs image feature information (f1, f2, f3), the image feature information can be directly input into the scene prediction network. Through the fully connected layer and the softmax layer, the probability (p1, p2) of the training image belonging to each preset scene can be obtained. The probability of the training image belonging to each preset scene is the predicted scene classification result.

[0088] Specifically, the initial 3D lookup table includes a number of 3D sub-lookup tables corresponding to the dimensions of the image feature information. After obtaining the image feature information, the server uses the feature information of each dimension in the image feature information to determine the lookup table weight of each 3D sub-lookup table. Then, based on the lookup table weights, the 3D sub-lookup tables are weighted to obtain a weighted 3D lookup table. By inputting the training image into the weighted 3D lookup table, the corresponding predicted enhanced image can be obtained. Specifically, after inputting the training image into the weighted 3D lookup table, for the parameter value of each pixel in the training image, the corresponding output parameter value can be found in the weighted 3D lookup table. By adjusting the parameter value of each pixel in the training image based on the output parameter value of each pixel, the predicted enhanced image can be obtained. The parameter value can specifically be RGB values, YUV values, etc., which are not specifically limited in this embodiment.

[0089] Step 108: Obtain the loss function based on the scene category label, the predicted scene classification result, the verification image, and the predicted augmented image.

[0090] In the process of training a neural network, the loss function is used to evaluate the degree to which the predicted values ​​of the neural network differ from the true values. A smaller loss function generally indicates better performance of the neural network. In this embodiment, the loss function is used to evaluate the degree to which the predicted values ​​of scene classification and image enhancement differ from the true values. A smaller loss function indicates higher accuracy in scene classification and better image enhancement effects.

[0091] Specifically, the server calculates a first loss function based on the scene category label and the predicted scene classification result, and calculates a second loss function based on the verification image and the predicted augmented image. The final loss function is obtained by combining the first and second loss functions.

[0092] Step 110: Return to the step of extracting image features from the training images respectively to obtain the image feature information corresponding to the training images, until the loss function converges and the target three-dimensional lookup table is obtained.

[0093] Specifically, when the loss function fails to converge, the server returns to the steps of extracting image features from each training image to obtain image feature information corresponding to the training image. It then adjusts the weight matrices of the neural network and the scene prediction network, as well as the parameters of the 3D sub-lookup table in the initial 3D lookup table, to obtain a new neural network, a new scene prediction network, and a new 3D lookup table. The new neural network is used to extract image features from the training images to obtain new image feature information. Based on this new image feature information and the new scene prediction network, the training images are then classified into scenes to obtain predicted scene classification results. The new image feature information is then input into the new 3D lookup table to obtain the predicted augmented image. Based on the scene category label, the predicted scene classification result, the validation image, and the predicted augmented image, the loss function is recalculated to determine if it has converged. If the loss function still fails to converge, the process returns to the steps of extracting image features from each training image to obtain image feature information corresponding to the training image, until the loss function converges and the target 3D lookup table is obtained.

[0094] The aforementioned 3D lookup table training method, after acquiring a training image set, a validation image set, and an initial 3D lookup table, extracts image features from each training image to obtain image feature information corresponding to the training image. Based on the image feature information, it performs scene classification on the training images to obtain predicted scene classification results. The image feature information is then input into the initial 3D lookup table to obtain predicted enhanced images, thus achieving preliminary training of the initial 3D lookup table. A loss function is obtained based on scene category labels, predicted scene classification results, validation images, and predicted enhanced images. This process of extracting image features from each training image is repeated until the loss function converges, resulting in the target 3D lookup table. This method, combined with scene classification training, yields a target 3D lookup table that accurately achieves enhanced color grading, improving video enhancement effects. Furthermore, using the target 3D lookup table to process video frames in the video data allows for rapid determination of the correspondence between pixel values ​​of video frames and pixel values ​​of enhanced video frames, accelerating video frame processing and improving the real-time performance of video enhancement.

[0095] In one embodiment, the initial three-dimensional lookup table includes three-dimensional sub-lookup tables, the number of which corresponds to the dimension of the image feature information.

[0096] Specifically, the initial three-dimensional lookup table includes three-dimensional sub-lookup tables, and the number of three-dimensional sub-lookup tables corresponds to the dimension of the image feature information. For example, when the image feature information is an N-dimensional vector, the number of three-dimensional sub-lookup tables is N.

[0097] In this embodiment, by corresponding the number of three-dimensional sub-lookup tables to the dimensions of image feature information, richer semantic information can be learned through multiple three-dimensional sub-lookup tables, so as to enhance the training image by utilizing semantic information.

[0098] In one embodiment, inputting image feature information into an initial 3D lookup table to obtain a predicted enhanced image includes:

[0099] Based on the feature information of each dimension in the image feature information, determine the lookup table weight of the three-dimensional sub-lookup table;

[0100] The three-dimensional sub-lookup table is weighted according to the lookup table weights to obtain the weighted three-dimensional lookup table;

[0101] The training images are input into a weighted 3D lookup table to obtain the predicted enhanced images.

[0102] Specifically, the server uses the feature information of each dimension in the image feature information as the lookup table weights of the three-dimensional sub-lookup table. The lookup table weights are multiplied by the corresponding three-dimensional sub-lookup tables, and the multiplied three-dimensional sub-lookup tables are superimposed to obtain a weighted three-dimensional lookup table. The training image is input into the weighted three-dimensional lookup table, and the output parameter value corresponding to the parameter value of each pixel in the training image is found through the weighted three-dimensional lookup table. By adjusting the parameter value of each pixel in the training image according to the output parameter value of each pixel, the predicted enhanced image can be obtained. The parameter value can specifically be RGB value, YUV value, etc., which is not specifically limited in this embodiment.

[0103] In this embodiment, the lookup table weights of the three-dimensional sub-lookup table are determined based on the feature information of each dimension in the image feature information. The three-dimensional sub-lookup table is then weighted according to the lookup table weights to obtain a weighted three-dimensional lookup table. The training image is then input into the weighted three-dimensional lookup table to obtain a predicted enhanced image. This method can learn richer semantic information using multiple three-dimensional sub-lookup tables, thereby enhancing the training image using semantic information.

[0104] In one embodiment, the loss function obtained based on the scene category label, the predicted scene classification result, the verification image, and the predicted augmented image includes:

[0105] The first loss function is calculated based on the scene category label and the predicted scene classification result, and the second loss function is calculated based on the verification image and the predicted augmented image.

[0106] The loss function is obtained based on the first loss function and the second loss function.

[0107] Specifically, the predicted scene classification result is the probability that a training image belongs to each preset scene. The server, based on the scene category labels and the probabilities of training images belonging to each preset scene, can obtain the cross-entropy loss function, i.e., the first loss function. For example, training images can be divided into daytime scenes and nighttime scenes based on brightness, with corresponding scene category labels c: daytime label and nighttime label. i ∈(0, 1), if the probability of a training image belonging to each preset scene is p i Then the cross-entropy loss function can be obtained as:

[0108]

[0109] Where n is the number of training images in the training image set, and c is the scene category label of the training images. i You can use either daytime label 0 and nighttime label 1, or daytime label 1 and nighttime label 0.

[0110] Specifically, the server also calculates a second loss function based on the verification image and the predicted augmented image. This second loss function primarily calculates the pixel difference between pixels at the same location in the verification image and the predicted augmented image. For example, the second loss function could be: Where n represents the number of training images in the training image set, B i Represents the verification image, A i Indicates the predicted augmented image, ||B i -A i ‖1 represents the loss function corresponding to a single training image, which is the pixel difference between pixels at the same location in the validation image and the predicted augmented image. For example, the loss function for a single training image can be expressed by the formula: The calculation shows that h refers to the height of the verification image and the predicted augmented image, w refers to the width of the verification image and the predicted augmented image, and (x,y) is used to represent the pixels at the same position in the verification image and the predicted augmented image.

[0111] Specifically, after obtaining the first loss function and the second loss function, the server can obtain the final loss function by superimposing the first and second loss functions. For example, the loss function can be:

[0112]

[0113] In this embodiment, the first loss function is calculated based on the scene category label and the predicted scene classification result, and the second loss function is calculated based on the verification image and the predicted enhancement image. Then, the loss function is obtained based on the first loss function and the second loss function, which enables the calculation of the loss function.

[0114] In one embodiment, calculating the second loss function based on the verification image and the predicted augmented image includes:

[0115] Determine the pixel correspondence between the first pixel in the verification image and the second pixel in the predicted augmented image;

[0116] Based on the pixel correspondence, determine the pixel difference between the first pixel and the corresponding second pixel;

[0117] The second loss function is calculated based on the pixel differences.

[0118] Specifically, the server determines the pixel correspondence between the first pixel in the verification image and the second pixel in the predicted augmented image based on the position of the first pixel in the verification image and the position of the second pixel in the predicted augmented image. After obtaining the pixel correspondence, the server calculates the pixel difference between the first pixel and the corresponding second pixel. Based on this pixel difference, the server determines the loss function corresponding to a single training image, and then calculates the second loss function based on the loss function corresponding to a single training image. The corresponding second pixel refers to the pixel located at the same position as the first pixel. The pixel difference mainly reflects the magnitude of the difference between the first and second pixels. It can be the difference in pixel value, the difference in brightness, or other differences, or a combination of multiple differences. This embodiment does not limit this.

[0119] For example, it can be illustrated by formulas. The loss function corresponding to a single training image is obtained, where h refers to the height of the validation image and the predicted augmentation image, w refers to the width of the validation image and the predicted augmentation image, and (x,y) is used to represent the pixels at the same position in the validation image and the predicted augmentation image.

[0120] In this embodiment, by determining the pixel correspondence, the pixel difference between the first pixel and the corresponding second pixel is determined based on the pixel correspondence, and the second loss function can be calculated based on the pixel difference.

[0121] In one embodiment, such as Figure 4As shown, a video enhancement method is provided. This embodiment illustrates the method by applying it to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0122] Step 402: Obtain the video data to be processed.

[0123] Among them, the video data to be processed refers to the video data that needs to be enhanced.

[0124] Specifically, when video data needs to be enhanced, the server will obtain the video data to be processed.

[0125] Step 404: Extract frames from the video data to be processed to obtain a set of video frames corresponding to the video data to be processed.

[0126] Specifically, after receiving the video data to be processed, the server will extract frames from the video data to obtain a set of video frames corresponding to the video data to be processed. The method of extracting frames from the video data to be processed can be uniform frame extraction, that is, extracting frames according to a preset frame extraction interval. In this embodiment, the frame extraction method is not limited, and the preset frame extraction interval can be set as needed.

[0127] Step 406: Input the video frames in the video frame set into the target 3D lookup table in sequence to obtain the enhanced video frames corresponding to the video frames. The target 3D lookup table is obtained through the above-described 3D lookup table training method.

[0128] Specifically, the server sequentially inputs the video frames from the video frame set into a target 3D lookup table. Using this table, it finds the output parameter values ​​corresponding to the parameter values ​​of pixels within the video frames. By using the output parameter values ​​for each pixel, the lookup output result corresponding to the video frame is obtained. After obtaining the lookup output result corresponding to the video frame, the server combines it with the lookup output result of the previous video frame to obtain a target lookup output result used for enhancing and adjusting the video frame. This target lookup output result is then used to enhance and adjust the video frame, resulting in an enhanced video frame. The parameter values ​​can specifically be RGB values, YUV values, etc., but this embodiment does not specify a particular value.

[0129] Step 408: Obtain enhanced video data based on the enhanced video frames.

[0130] Specifically, after obtaining the enhanced video frames, the server can perform video reconstruction based on the enhanced video frames to obtain enhanced video data.

[0131] The aforementioned video enhancement method acquires the video data to be processed, extracts frames from the video data to obtain a set of video frames corresponding to the video data to be processed, inputs the video frames in the set into a target three-dimensional lookup table in sequence to obtain the enhanced video frames corresponding to the video frames, and obtains the enhanced video data based on the enhanced video frames. By using the target three-dimensional lookup table to process the video frames in the video data, the video enhancement effect can be improved, and the correspondence between the pixel values ​​of the video frames and the pixel values ​​of the enhanced video frames can be quickly obtained, thereby speeding up the processing speed of video frames and improving the real-time performance of video enhancement.

[0132] In one embodiment, the video frames in the video frame set are sequentially input into a target 3D lookup table to obtain the enhanced video frames corresponding to the video frames, including:

[0133] The video frames in the video frame set are sequentially input into the target 3D lookup table to obtain the lookup output results corresponding to the video frames.

[0134] When the video frame is not the first frame of the video, retrieve the previous video frame corresponding to the video frame and look up the output result.

[0135] The target search output result is obtained by performing a moving average on the search output result of the previous video frame and the search output result.

[0136] Based on the target search output, the video frames are enhanced and adjusted to obtain the enhanced video frames corresponding to the original video frames.

[0137] The lookup output result refers to the output parameter value of each pixel in the video frame output by the target 3D lookup table, which corresponds to the parameter value of each pixel.

[0138] Specifically, since the enhancement process mainly processes individual video frames, and the processing of video frames is independent, the relationship between video frames is not linked, which will cause discontinuous jitter and affect the visual quality of the result. Therefore, when obtaining the enhanced video frame, the server needs to apply the moving average method to solve the jitter problem and obtain a time-domain stable result.

[0139] Specifically, the server sequentially inputs the video frames from the video frame set into the target 3D lookup table to obtain the lookup output results corresponding to the video frames. It then determines whether the video frame is the first frame of the video. If the video frame is not the first frame, it obtains the lookup output result of the previous video frame corresponding to the video frame. It performs a moving average on the lookup output result of the previous video frame and the current lookup output result to obtain the target lookup output result. Based on the target parameter value of each pixel in the target lookup output result, it enhances and adjusts the parameter value of each corresponding pixel in the video frame to obtain the enhanced video frame corresponding to the video frame.

[0140] For example, the target search output result can be obtained using a pre-set moving average calculation formula. The moving average calculation formula can be as follows:

[0141] moving_average_lut_new=γ*output_lut+(1-γ)*moving_average_lut,

[0142] Where output_lut is the search output result, moving_average_lut is the search output result of the previous video frame, moving_average_lut_new is the target search output result, and γ is a coefficient between 0 and 1 that can be set as needed.

[0143] In this embodiment, after obtaining the search output result corresponding to the video frame, the search output result of the previous video frame corresponding to the video frame is obtained. The search output result of the previous video frame and the search output result are then averaged to obtain the target search output result. Based on the target search output result, the video frame is enhanced and adjusted to obtain the enhanced video frame corresponding to the video frame. The problem of jitter between video frames can be solved by using the moving average.

[0144] like Figure 5 As shown, this application also provides a flowchart to illustrate the three-dimensional lookup table training method of this application. Specifically, the three-dimensional lookup table training method includes the following steps:

[0145] The server obtains a training image set, a validation image set, and an initial 3D lookup table. The training images in the training image set carry scene category labels. The validation images in the validation image set correspond one-to-one with the training images in the training image set. The validation images are augmented images corresponding to the training images. The initial 3D lookup table includes 3D sub-lookup tables, and the number of 3D sub-lookup tables corresponds to the dimension of the image feature information.

[0146] Use the training image (input image A) i The image is input into a neural network for image feature extraction, yielding image feature information corresponding to the training image (i.e., ...). Figure 5The N-dimensional weights in the image are used to classify the training image based on image feature information, thus obtaining the predicted scene classification result (i.e., Figure 5 In this process, N-dimensional weights are input into a fully connected layer. After passing through the fully connected layer and softmax, the predicted probability p is obtained. i ), and based on the feature information of each dimension in the image feature information, determine the three-dimensional sub-lookup table (i.e. Figure 5 The lookup table weights of multiple parallel 3DLUTs (multiple N-dimensional weights) are then used to weight the 3D sub-lookup tables, resulting in a weighted 3D lookup table (i.e., ...). Figure 5 The process involves aggregating multiple parallel 3D LUTs to obtain a single 3D LUT. The training image is then input into a weighted 3D lookup table to obtain the predicted enhanced image (output image B). i Based on the scene category label and the predicted scene classification result, the first loss function is calculated, and based on the verification image and the predicted enhanced image, the second loss function is calculated. Based on the first loss function and the second loss function, the loss function is obtained. The process of extracting image features from the training images and obtaining the image feature information corresponding to the training images is repeated until the loss function converges, and the target 3D lookup table is obtained.

[0147] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0148] In one embodiment, such as Figure 6 As shown, a three-dimensional lookup table training device is provided, including: an acquisition module 602, a feature extraction module 604, a first processing module 606, a loss function calculation module 608, and a second processing module 610, wherein:

[0149] The acquisition module 602 is used to acquire the training image set, the verification image set, and the initial 3D lookup table. The training images in the training image set carry scene category labels. The verification images in the verification image set correspond one-to-one with the training images in the training image set. The verification images are the augmented images corresponding to the training images.

[0150] The feature extraction module 604 is used to extract image features from the training images respectively to obtain image feature information corresponding to the training images;

[0151] The first processing module 606 is used to classify the training image into a scene based on the image feature information, obtain the predicted scene classification result, and input the image feature information into an initial three-dimensional lookup table to obtain the predicted enhanced image.

[0152] The loss function calculation module 608 is used to obtain the loss function based on the scene category label, the predicted scene classification result, the verification image, and the predicted augmented image;

[0153] The second processing module 610 is used to return the steps of extracting image features from the training images to obtain image feature information corresponding to the training images, until the loss function converges and the target three-dimensional lookup table is obtained.

[0154] The aforementioned 3D lookup table training method, after acquiring a training image set, a validation image set, and an initial 3D lookup table, extracts image features from each training image to obtain image feature information corresponding to the training image. Based on the image feature information, it performs scene classification on the training images to obtain predicted scene classification results. The image feature information is then input into the initial 3D lookup table to obtain predicted enhanced images, thus achieving preliminary training of the initial 3D lookup table. A loss function is obtained based on scene category labels, predicted scene classification results, validation images, and predicted enhanced images. This process of extracting image features from each training image is repeated until the loss function converges, resulting in the target 3D lookup table. This method, combined with scene classification training, yields a target 3D lookup table that accurately achieves enhanced color grading, improving video enhancement effects. Furthermore, using the target 3D lookup table to process video frames in the video data allows for rapid determination of the correspondence between pixel values ​​of video frames and pixel values ​​of enhanced video frames, accelerating video frame processing and improving the real-time performance of video enhancement.

[0155] In one embodiment, the initial three-dimensional lookup table includes three-dimensional sub-lookup tables, the number of which corresponds to the dimension of the image feature information.

[0156] In one embodiment, the first processing module is further configured to determine the lookup table weights of the three-dimensional sub-lookup table based on the feature information of each dimension in the image feature information, weight the three-dimensional sub-lookup table according to the lookup table weights to obtain a weighted three-dimensional lookup table, input the training image into the weighted three-dimensional lookup table to obtain the predicted enhanced image.

[0157] In one embodiment, the loss function calculation module is further configured to calculate a first loss function based on the scene category label and the predicted scene classification result, and calculate a second loss function based on the verification image and the predicted augmented image, and obtain the loss function based on the first loss function and the second loss function.

[0158] In one embodiment, the loss function calculation module is further configured to determine the pixel correspondence between the first pixel in the verification image and the second pixel in the predicted augmented image, determine the pixel difference between the first pixel and the corresponding second pixel based on the pixel correspondence, and calculate the second loss function based on the pixel difference.

[0159] In one embodiment, such as Figure 7 As shown, a video enhancement device is provided, including: a video acquisition module 702, a frame extraction module 704, an enhancement module 706, and a frame processing module 708, wherein:

[0160] The video acquisition module 702 is used to acquire video data to be processed;

[0161] The frame extraction module 704 is used to extract frames from the video data to be processed, and obtain a set of video frames corresponding to the video data to be processed.

[0162] The enhancement module 706 is used to input the video frames in the video frame set into the target three-dimensional lookup table in sequence to obtain the enhanced video frames corresponding to the video frames. The target three-dimensional lookup table is obtained by the above-mentioned three-dimensional lookup table training method.

[0163] The frame processing module 708 is used to obtain enhanced video data based on the enhanced video frames.

[0164] The aforementioned video enhancement method acquires the video data to be processed, extracts frames from the video data to obtain a set of video frames corresponding to the video data to be processed, inputs the video frames in the set into a target three-dimensional lookup table in sequence to obtain the enhanced video frames corresponding to the video frames, and obtains the enhanced video data based on the enhanced video frames. By using the target three-dimensional lookup table to process the video frames in the video data, the video enhancement effect can be improved, and the correspondence between the pixel values ​​of the video frames and the pixel values ​​of the enhanced video frames can be quickly obtained, thereby speeding up the processing speed of video frames and improving the real-time performance of video enhancement.

[0165] In one embodiment, the enhancement module is further configured to sequentially input video frames from the video frame set into a target three-dimensional lookup table to obtain a lookup output result corresponding to the video frame. When the video frame is not the first video frame, the module obtains the lookup output result of the previous video frame corresponding to the video frame, performs a sliding average on the lookup output result of the previous video frame and the lookup output result to obtain the target lookup output result, and enhances and adjusts the video frame according to the target lookup output result to obtain the enhanced video frame corresponding to the video frame.

[0166] Specific embodiments of the 3D lookup table training device and the video enhancement device can be found in the embodiments of the 3D lookup table training method and the video enhancement method described above, and will not be repeated here. Each module in the above-mentioned 3D lookup table training device and video enhancement device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0167] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as training image sets and validation image sets. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a three-dimensional lookup table training method and a video enhancement method.

[0168] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0169] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0170] Obtain the training image set, the validation image set, and the initial 3D lookup table. The training images in the training image set carry scene category labels. The validation images in the validation image set correspond one-to-one with the training images in the training image set. The validation images are the augmented images corresponding to the training images.

[0171] Image features are extracted from the training images to obtain the image feature information corresponding to the training images;

[0172] Based on image feature information, the training image is classified into scenes to obtain the predicted scene classification result. The image feature information is then input into an initial three-dimensional lookup table to obtain the predicted enhanced image.

[0173] The loss function is obtained based on the scene category label, the predicted scene classification result, the validation image, and the predicted augmented image.

[0174] Return to the steps of extracting image features from each training image to obtain the image feature information corresponding to the training image, until the loss function converges and the target 3D lookup table is obtained.

[0175] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the lookup table weights of the three-dimensional sub-lookup table based on the feature information of each dimension in the image feature information; weighting the three-dimensional sub-lookup table according to the lookup table weights to obtain a weighted three-dimensional lookup table; and inputting the training image into the weighted three-dimensional lookup table to obtain the predicted enhanced image.

[0176] In one embodiment, when the processor executes the computer program, it further performs the following steps: calculating a first loss function based on the scene category label and the predicted scene classification result, and calculating a second loss function based on the verification image and the predicted augmented image, and obtaining a loss function based on the first loss function and the second loss function.

[0177] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the pixel correspondence between a first pixel in the verification image and a second pixel in the predicted augmented image; determining the pixel difference between the first pixel and the corresponding second pixel based on the pixel correspondence; and calculating a second loss function based on the pixel difference.

[0178] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0179] Acquire the video data to be processed;

[0180] Extract frames from the video data to be processed to obtain a set of video frames corresponding to the video data to be processed.

[0181] The video frames in the video frame set are sequentially input into the target 3D lookup table to obtain the enhanced video frames corresponding to the video frames. The target 3D lookup table is obtained through the above 3D lookup table training method.

[0182] Enhanced video data is obtained from the enhanced video frames.

[0183] In one embodiment, when the processor executes the computer program, it further implements the following steps: sequentially inputting video frames from the video frame set into a target three-dimensional lookup table to obtain a lookup output result corresponding to the video frame; when the video frame is not the first video frame, obtaining the lookup output result of the previous video frame corresponding to the video frame; performing a sliding average on the lookup output result of the previous video frame and the lookup output result to obtain the target lookup output result; and enhancing and adjusting the video frame according to the target lookup output result to obtain an enhanced video frame corresponding to the video frame.

[0184] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0185] Obtain the training image set, the validation image set, and the initial 3D lookup table. The training images in the training image set carry scene category labels. The validation images in the validation image set correspond one-to-one with the training images in the training image set. The validation images are the augmented images corresponding to the training images.

[0186] Image features are extracted from the training images to obtain the image feature information corresponding to the training images;

[0187] Based on image feature information, the training image is classified into scenes to obtain the predicted scene classification result. The image feature information is then input into an initial three-dimensional lookup table to obtain the predicted enhanced image.

[0188] The loss function is obtained based on the scene category label, the predicted scene classification result, the validation image, and the predicted augmented image.

[0189] Return to the steps of extracting image features from each training image to obtain the image feature information corresponding to the training image, until the loss function converges and the target 3D lookup table is obtained.

[0190] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the lookup table weights of the three-dimensional sub-lookup table based on the feature information of each dimension in the image feature information; weighting the three-dimensional sub-lookup table according to the lookup table weights to obtain a weighted three-dimensional lookup table; and inputting the training image into the weighted three-dimensional lookup table to obtain the predicted enhanced image.

[0191] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: calculating a first loss function based on the scene category label and the predicted scene classification result, and calculating a second loss function based on the verification image and the predicted augmented image, and obtaining a loss function based on the first loss function and the second loss function.

[0192] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the pixel correspondence between a first pixel in the verification image and a second pixel in the predicted augmented image; determining the pixel difference between the first pixel and the corresponding second pixel based on the pixel correspondence; and calculating a second loss function based on the pixel difference.

[0193] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0194] Acquire the video data to be processed;

[0195] Extract frames from the video data to be processed to obtain a set of video frames corresponding to the video data to be processed.

[0196] The video frames in the video frame set are sequentially input into the target 3D lookup table to obtain the enhanced video frames corresponding to the video frames. The target 3D lookup table is obtained through the above 3D lookup table training method.

[0197] Enhanced video data is obtained from the enhanced video frames.

[0198] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: sequentially inputting the video frames in the video frame set into the target three-dimensional lookup table to obtain the lookup output result corresponding to the video frame; when the video frame is not the first video frame, obtaining the lookup output result of the previous video frame corresponding to the video frame; performing a sliding average on the lookup output result of the previous video frame and the lookup output result to obtain the target lookup output result; and enhancing and adjusting the video frame according to the target lookup output result to obtain the enhanced video frame corresponding to the video frame.

[0199] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0200] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0201] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for training a three-dimensional lookup table, characterized in that, The method includes: A training image set, a validation image set, and an initial 3D lookup table are obtained. The training images in the training image set carry scene category labels. The validation images in the validation image set correspond one-to-one with the training images in the training image set. The validation images are augmented images corresponding to the training images. The initial 3D lookup table includes 3D sub-lookup tables. The number of 3D sub-lookup tables corresponds to the dimension of the image feature information. Image features are extracted from the training images to obtain image feature information corresponding to the training images; Based on the image feature information, the training image is classified into scenes to obtain a predicted scene classification result. The feature information of each dimension in the image feature information is used as the lookup table weight of the three-dimensional sub-lookup table. The three-dimensional sub-lookup table is weighted according to the lookup table weight to obtain a weighted three-dimensional lookup table. The training image is input into the weighted three-dimensional lookup table to obtain a predicted enhanced image. The loss function is obtained based on the scene category label, the predicted scene classification result, the verification image, and the predicted augmented image; Return to the step of extracting image features from the training images to obtain image feature information corresponding to the training images, until the loss function converges and the target three-dimensional lookup table is obtained.

2. The method according to claim 1, characterized in that, The loss function obtained based on the scene category label, the predicted scene classification result, the verification image, and the predicted augmented image includes: A first loss function is calculated based on the scene category label and the predicted scene classification result, and a second loss function is calculated based on the verification image and the predicted enhanced image. The loss function is obtained based on the first loss function and the second loss function.

3. The method according to claim 2, characterized in that, The step of calculating the second loss function based on the verification image and the predicted enhanced image includes: Determine the pixel correspondence between the first pixel in the verification image and the second pixel in the predicted enhanced image; Based on the pixel correspondence, determine the pixel difference between the first pixel and the corresponding second pixel; The second loss function is calculated based on the pixel differences.

4. A video enhancement method, characterized in that, The method includes: Acquire the video data to be processed; Frames are extracted from the video data to be processed to obtain a set of video frames corresponding to the video data to be processed; The video frames in the video frame set are sequentially input into the target three-dimensional lookup table to obtain the enhanced video frame corresponding to the video frame. The target three-dimensional lookup table is obtained by the method described in any one of claims 1-3. Enhanced video data is obtained based on the enhanced video frames.

5. The method according to claim 4, characterized in that, The step of sequentially inputting the video frames in the video frame set into the target 3D lookup table to obtain the enhanced video frame corresponding to the video frame includes: The video frames in the video frame set are sequentially input into the target three-dimensional lookup table to obtain the lookup output results corresponding to the video frames. When the video frame is not the first video frame, obtain the previous video frame lookup output result corresponding to the video frame; The target search output result is obtained by performing a moving average on the search output result of the previous video frame and the search output result. Based on the target lookup output, the video frame is enhanced and adjusted to obtain an enhanced video frame corresponding to the video frame.

6. A three-dimensional lookup table training device, characterized in that, The device includes: The acquisition module is used to acquire a training image set, a validation image set, and an initial 3D lookup table. The training images in the training image set carry scene category labels, and the validation images in the validation image set correspond one-to-one with the training images in the training image set. The validation images are augmented images corresponding to the training images. The initial 3D lookup table includes 3D sub-lookup tables, and the number of 3D sub-lookup tables corresponds to the dimension of the image feature information. The feature extraction module is used to extract image features from the training images respectively to obtain image feature information corresponding to the training images; The first processing module is used to classify the training image into scenes based on the image feature information to obtain a predicted scene classification result, and to use the feature information of each dimension in the image feature information as the lookup table weight of the three-dimensional sub-lookup table, to weight the three-dimensional sub-lookup table according to the lookup table weight to obtain a weighted three-dimensional lookup table, and to input the training image into the weighted three-dimensional lookup table to obtain a predicted enhanced image. The loss function calculation module is used to obtain a loss function based on the scene category label, the predicted scene classification result, the verification image, and the predicted augmented image; the loss function is used to evaluate the degree to which the predicted values ​​of scene classification and image augmentation differ from the true values. The second processing module is used to return to the step of extracting image features from the training images to obtain image feature information corresponding to the training images, until the loss function converges and the target three-dimensional lookup table is obtained.

7. The apparatus according to claim 6, characterized in that, The loss function calculation module is further configured to calculate a first loss function based on the scene category label and the predicted scene classification result, and calculate a second loss function based on the verification image and the predicted augmented image, and obtain the loss function based on the first loss function and the second loss function.

8. The apparatus according to claim 7, characterized in that, The loss function calculation module is further configured to determine the pixel correspondence between the first pixel in the verification image and the second pixel in the predicted enhancement image, determine the pixel difference between the first pixel and the corresponding second pixel based on the pixel correspondence, and calculate the second loss function based on the pixel difference.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images