An intelligent video color adjustment method, device, storage medium and electronic equipment based on LUT copying, merging and evaluation

Abstract

The application provides an intelligent video color grading method and device based on LUT replication, merging and evaluation, a storage medium and an electronic device, relates to the field of computer vision and computer graphics, and the method comprises the following steps: performing shot segmentation on a video before and after color grading and extracting key frames; extracting feature vectors based on the key frames and merging the same type of shots; replicating the LUT of the same type of shots; training a LUT evaluation network model based on the key frames before color grading and the feature vectors of the LUT images applied to the key frames before color grading; and recommending LUT for a new video shot by using the trained LUT evaluation network model. The application realizes the whole process of LUT replication, merging and evaluation, provides a solution of intelligent LUT recommendation and color grading for the same type of video shots, and reduces the labor cost of video production.

Description

Technical Field

[0001] This application relates to the fields of computer vision and computer graphics technology, and more specifically, to an intelligent video color correction method, apparatus, storage medium, and electronic device based on LUT replication, merging, and evaluation. Background Technology

[0002] Video color grading is a core technology in film and television production, short video creation, and other fields. By adjusting the colors of video images, it can enhance the visual effects of the video, create a specific atmosphere, and give the video a unique artistic style. LUTs (Look-Up Tables), as an important tool for video color grading, can apply preset color mapping relationships to video pixels, achieving fast and standardized color adjustments. They are widely used in various professional video color grading software and production workflows.

[0003] Currently, color grading for batches of similar shots with repetitive styles often relies on professional colorists to determine the style and adjust the colors for each shot. This process is time-consuming and labor-intensive, and the differences in style among different colorists make it difficult to achieve standardized color grading results. A common solution is to use LUTs to improve efficiency, but this introduces numerous problems. For example, how to determine the number of color grading styles; how to replicate the original color grading effect based on the before-and-after results; and how to quickly select the most suitable LUT from multiple options when faced with new video shots. These problems typically require manual intervention, wasting the human and time resources of video production, and lacking intelligent LUT recommendations and color grading solutions.

[0004] Although some existing technologies use neural network-based color transfer and LUT generation methods that can generate LUTs for single images or video segments, they have not yet achieved a fully integrated intelligent color grading process that combines LUT replication, merging, and evaluation. They cannot intelligently recommend LUTs and color grade based on the characteristics of video shots, making it difficult to meet the needs of efficient and standardized color grading in batch video production. Summary of the Invention

[0005] The embodiments of this application provide an intelligent video color grading method, apparatus, storage medium, and electronic device based on LUT replication, merging, and evaluation, to solve the problems of high LUT production cost, low efficiency of manual color grading, and lack of intelligent recommendation mechanism in existing video color grading.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0007] According to a first aspect of the embodiments of this application, a smart video color grading method based on LUT replication, merging, and evaluation is provided, comprising: Perform shot segmentation and extract keyframes from the video before and after color grading; Feature vectors are extracted from the keyframes and similar shots are merged. LUTs that replicate similar lenses; The LUT evaluation network model is trained based on the keyframes of the pre-color grading shot and the feature vectors of the LUT images applied to the keyframes of the pre-color grading shot. Use the trained LUT to evaluate the network model and recommend LUTs for new video footage.

[0008] In some embodiments of this application, based on the foregoing scheme, the step of segmenting the video before and after color grading and extracting keyframes includes: The probability score of each frame as a shot cut point is calculated using a deep learning model. Set a score threshold, and perform shot segmentation based on the score threshold; For each shot after segmentation, extract several frames at equal intervals as keyframes for that shot.

[0009] In some embodiments of this application, based on the foregoing scheme, the step of extracting feature vectors based on the keyframes and merging similar shots includes: Extract the feature vectors of keyframes; The K-means algorithm is used to cluster all shots into several classes based on feature vectors; Replicate the LUT for each lens category; For each shot within each category, the PSNR value of the keyframe before color grading is calculated and then the corresponding LUT is applied sequentially to the keyframe after color grading. Based on the PSNR value, the lenses are clustered again within each lens category to obtain the final clustering and merging result.

[0010] In some embodiments of this application, based on the foregoing scheme, the LUT for replicating similar lenses includes: Collect all pixels of all keyframes for all shots in each category and establish a color mapping table before and after color grading. Construct a KD-tree to perform nearest neighbor search and generate a LUT; Smooth the LUT.

[0011] In some embodiments of this application, based on the foregoing scheme, training the LUT evaluation network model based on the feature vectors of the LUT image applied to the pre-color grading keyframe and the pre-color grading keyframe LUT image includes: Extract the keyframes of the shot before color grading and the feature vectors of the LUT images of the keyframes of the shot before color grading. Construct feature vector samples for training LUT classification models and scene classification models; Set the loss function and training parameters; Train and save the LUT classification model and the scene classification model.

[0012] In some embodiments of this application, based on the foregoing scheme, the loss function adopts InfoNCE loss, and the calculation steps include: Calculate the cosine similarity between the anchor sample and the feature vectors of the positive and negative samples; Divide the cosine similarity result by the temperature coefficient to obtain the positive sample similarity and negative sample similarity; The similarity scores of positive samples are concatenated with the similarity scores of all negative samples along the channel dimension to form a comparison vector; The positive sample class is used as the target label, and the cross-entropy loss is used to complete the optimization.

[0013] In some embodiments of this application, based on the foregoing scheme, the step of using a trained LUT to evaluate the network model and recommend LUTs for new video shots includes: Extract keyframes from new shots, and then apply each LUT to the keyframes sequentially. For each LUT applied, extract the keyframe feature vectors before and after application and input them into the scene classification model to obtain a scene classification score. If the scene classification score is lower than the threshold, the LUT is not recommended. If it is higher than the threshold, it is input into the LUT classification model to obtain a LUT classification score. Finally, sort all LUT classification scores and select the N LUTs with the highest classification scores as recommended LUTs.

[0014] According to a second aspect of the embodiments of this application, a smart video color grading apparatus based on LUT replication, merging, and evaluation is provided, comprising: The extraction unit is used to segment the video before and after color grading and extract keyframes. A merging unit is used to extract feature vectors based on the keyframes and merge similar shots. Replica unit, used to replicate the LUT of the same type of lens; Training unit, used to train LUT evaluation network model based on key frames of the pre-color grading shot and feature vectors of the LUT image applied to the key frames of the pre-color grading shot; The recommendation unit is used to evaluate the network model using a trained LUT and recommend LUTs for new video shots.

[0015] According to a third aspect of the embodiments of this application, a computer-readable storage medium is provided, the storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect.

[0016] According to a fourth aspect of the embodiments of this application, an electronic device is provided, including: a memory and a processor; The memory is used to store computer instructions; The processor is configured to invoke computer instructions stored in the memory, causing the electronic device to execute the method described in the first aspect.

[0017] The technical solution of this application solves the problems of high cost of manual LUT production, low efficiency of manual color grading, and lack of intelligent recommendation mechanism in the existing video color grading process. It realizes the intelligentization of the entire process of LUT replication, merging and evaluation, reduces the labor cost of video production, and improves the efficiency and standardization of video color grading.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 A flowchart illustrating an intelligent video color grading method based on LUT replication, merging, and evaluation according to an embodiment of this application is shown. Figure 2 A block diagram of an intelligent video color grading apparatus based on LUT replication, merging, and evaluation according to one embodiment of this application is shown; Figure 3 A block diagram of an electronic device according to one embodiment of this application is shown; Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0020] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0021] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0022] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] The following detailed description of some embodiments of this application will be provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0026] To address the shortcomings of existing technologies, this application provides an intelligent video color grading method based on LUT replication, merging, and evaluation. This method aims to solve the problems of high manual LUT creation costs, low efficiency of manual color grading, and lack of intelligent recommendation mechanisms in existing video color grading processes. This method achieves intelligent processing of the entire LUT replication, merging, and evaluation process, providing intelligent LUT recommendations and color grading solutions for similar video shots, reducing the labor costs of video production, and improving the efficiency and standardization of video color grading.

[0027] For details, see Figure 1The diagram illustrates a flowchart of an intelligent video color grading method based on LUT replication, merging, and evaluation according to an embodiment of this application.

[0028] like Figure 1 As shown, an intelligent video color grading method based on LUT replication, merging, and evaluation is demonstrated, specifically including steps S100 to S500.

[0029] refer to Figure 1 Step S100: Perform shot segmentation and extract keyframes from the video before and after color grading.

[0030] In some feasible embodiments, based on the foregoing scheme, the step of segmenting the video before and after color grading and extracting keyframes includes: The probability score of each frame as a shot cut point is calculated using a deep learning model. Set a score threshold, and perform shot segmentation based on the score threshold; For each shot after segmentation, extract several frames at equal intervals as keyframes for that shot.

[0031] Continue to refer to Figure 1 Step S200: Extract feature vectors based on the keyframes and merge similar shots.

[0032] In some feasible embodiments, based on the foregoing scheme, the step of extracting feature vectors based on the keyframes and merging similar shots includes: Extract the feature vectors of keyframes; The K-means algorithm is used to cluster all shots into several classes based on feature vectors; Replicate the LUT for each lens category; For each shot within each category, the PSNR value of the keyframe before color grading is calculated and then the corresponding LUT is applied sequentially to the keyframe after color grading. Based on the PSNR value, the lenses are clustered again within each lens category to obtain the final clustering and merging result.

[0033] Understandably, during the re-clustering process, it can be further divided into 1 to 5 categories to achieve further subdivision of the categories.

[0034] It should be noted that in this embodiment, the extracted keyframe feature vectors include two types: one is the feature extracted from a single image, such as the mean, standard deviation, and skewness of each channel in the RGB / LAB / HSV / grayscale color space; the other is the feature extracted from image pairs, including the difference in the mean and standard deviation of each color space between images, histogram similarity, color moment similarity, etc. When merging similar shots, the clustering uses the features extracted from each keyframe image before color grading.

[0035] Continue to refer to Figure 1 Step S300: Replicate the LUT of the same type of lens.

[0036] In some feasible embodiments, based on the foregoing scheme, the LUT for replicating similar lenses includes: Collect all pixels of all keyframes for all shots in each category and establish a color mapping table before and after color grading. Construct a KD-tree to perform nearest neighbor search and generate a LUT; Smooth the LUT.

[0037] It should be noted that the detailed process of "replicating the LUT for each lens category" in sub-step S200 is roughly the same as the entire process of this replication step. The specific replication process can be referred to in this step.

[0038] Understandably, in the process of constructing a KD-tree for nearest neighbor search to generate a LUT, a KD-tree (K-dimension tree) is a tree-like data structure that stores instance points in K-dimensional space for fast retrieval. A KD-tree is a binary tree representing a partition of K-dimensional space. Constructing a KD-tree is equivalent to continuously dividing the K-dimensional space using hyperplanes perpendicular to the coordinate axes, forming a series of K-dimensional hyperrectangular regions. Each node in the KD-tree corresponds to a K-dimensional hyperrectangular region. Using a KD-tree can eliminate the need to search for most data points, thus reducing the computational load. The specific method for LUT replication is as follows: for the color value of each grid point in a 33×33×33 LUT, use the KD-tree to find the nearest pre-adjustment color values ​​in the statistical color mapping pairs, and then use the median or mean of the corresponding post-adjustment color values ​​as the color value at the corresponding position in the LUT.

[0039] It should be noted that, in this embodiment, the specific process of smoothing the LUT is as follows: Gaussian smoothing is applied to the R, G, and B channels of the LUT to avoid color spot problems caused by abrupt changes in color values.

[0040] Continue to refer to Figure 1 Step S400: Train the LUT evaluation network model based on the key frame of the pre-color grading shot and the feature vector of the LUT image applied to the key frame of the pre-color grading shot.

[0041] It should be noted that in this embodiment, training the LUT evaluation network model specifically requires training two models: a LUT classification model and a scene classification model. The training method is contrastive learning training, which constructs positive and negative samples and optimizes the network using a loss function, enabling the two network models to output the color correction effect and target effect of each LUT, as well as a comparison score for each scene category. A higher score indicates a higher matching degree.

[0042] In some feasible embodiments, based on the foregoing scheme, the step of training the LUT evaluation network model based on the feature vectors of the LUT image applied to the pre-color-grading keyframe and the pre-color-grading keyframe LUT image includes: Extract the keyframes of the shot before color grading and the feature vectors of the LUT images of the keyframes of the shot before color grading. Construct feature vector samples for training LUT classification models and scene classification models; Set the loss function and training parameters; Train and save the LUT classification model and the scene classification model.

[0043] It should be noted that, in this embodiment, the training samples for the LUT classification model include all feature vectors calculated using single keyframe images and image pairs before and after color grading, while the training samples for the scene classification model only include feature vectors extracted from single keyframe images before color grading.

[0044] It should be noted that, in this embodiment, the trained network structure is a projection head network constructed using a multilayer perceptron, including multiple fully connected layers, batch normalization layers, activation function layers, and dropout layers. This network maps the input feature vectors to a low-dimensional compact feature space through nonlinear transformations to improve the discriminativeness and robustness of the feature vectors, thereby better supporting the model evaluation task.

[0045] In some feasible embodiments, based on the aforementioned scheme, the loss function adopts the InfoNCE loss, and the calculation steps include: Calculate the cosine similarity between the anchor sample and the feature vectors of the positive and negative samples; Divide the cosine similarity result by the temperature coefficient to obtain the positive sample similarity and negative sample similarity; The similarity scores of positive samples are concatenated with the similarity scores of all negative samples along the channel dimension to form a comparison vector; The positive sample class is used as the target label, and the cross-entropy loss is used to complete the optimization.

[0046] For example, the calculation steps specifically include: First, calculate the cosine similarity between the anchor point sample and the feature vectors of the positive and negative samples. The anchor point sample is the mean of the feature vectors of all keyframes in a class of shots, while the positive and negative samples are the feature vectors of a single keyframe. The formula for calculating the cosine similarity is: ; in These are the feature vectors of the anchor sample, positive sample, and negative sample, respectively. The calculated... The value range is [0,1].

[0047] Then divide the above similarity results by the temperature coefficient. To amplify the similarity differences between feature vectors, the formula is: ; ; in The temperature coefficient is set to an empirical value of 0.1. and The similarity between the anchor sample and the positive and negative samples after scaling is calculated respectively.

[0048] Then, the similarity scores of positive samples are concatenated with the similarity scores of all negative samples along the channel dimension to form a comparison vector: ; in The number of negative samples. Let K be the comparison vector of length 1+K.

[0049] Finally, using the positive sample class as the target label, cross-entropy loss is employed for optimization. The InfoNCE loss calculation formula is as follows: ; in The numerator represents the number of samples within the batch, the denominator is the exponentially normalized similarity value of the positive samples, and the denominator is the sum of the exponentially normalized similarity values ​​of the positive samples and all negative samples. This loss is used to bring the anchor point closer to the feature vector of the positive samples and to increase the distance between the anchor point and the feature vector of the negative samples.

[0050] Continue to refer to Figure 1 Step S500: Use the trained LUT to evaluate the network model and recommend LUTs for new video shots.

[0051] In some feasible embodiments, based on the foregoing scheme, the step of using a trained LUT to evaluate the network model and recommend LUTs for new video shots includes: Extract keyframes from new shots, and then apply each LUT to the keyframes sequentially. For each LUT applied, extract the keyframe feature vectors before and after application and input them into the scene classification model to obtain a scene classification score. If the scene classification score is lower than the threshold, the LUT is not recommended. If it is higher than the threshold, it is input into the LUT classification model to obtain a LUT classification score. Finally, sort all LUT classification scores and select the N LUTs with the highest classification scores as recommended LUTs.

[0052] The following describes an apparatus embodiment of this application, which can be used to execute an intelligent video color grading method based on LUT replication, merging, and evaluation as described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in the above applications.

[0053] Reference Figure 2 As shown, a smart video color grading apparatus 200 based on LUT replication, merging, and evaluation according to an embodiment of this application includes: Extraction unit 201 is used to perform shot segmentation and extract keyframes from the video before and after color grading. Merging unit 202 is used to extract feature vectors based on the keyframes and merge similar shots; Replication unit 203 is used to replicate LUTs of the same type of lens; Training unit 204 is used to train a LUT evaluation network model based on the key frames of the pre-color grading shot and the feature vectors of the LUT image applied to the key frames of the pre-color grading shot. Recommendation unit 205 is used to evaluate the network model using a trained LUT to recommend LUTs for new video shots.

[0054] like Figure 3 As shown, this application embodiment also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, it implements the steps of the above-described intelligent video color grading method based on LUT replication, merging, and evaluation.

[0055] Since the electronic device described in this embodiment is the device used to implement the intelligent video color grading device based on LUT replication, merging and evaluation in the embodiments of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiments of this application is within the scope of protection of this application.

[0056] In practice, when the computer program 311 is executed by the processor, it can implement any of the embodiments corresponding to the first aspect.

[0057] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0058] It should be noted that, Figure 4The computer system 400 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0059] like Figure 4 As shown, the computer system 400 includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 402 or programs loaded from storage portion 408 into Random Access Memory (RAM) 403, such as performing the methods described in the above embodiments. The RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An Input / Output (I / O) interface 405 is also connected to the bus 404.

[0060] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0061] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs various functions defined in the system of this application.

[0062] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0063] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0064] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0065] In another aspect, this application also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the intelligent video color grading method based on LUT replication, merging, and evaluation described in the above embodiments.

[0066] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to implement the intelligent video color grading method based on LUT replication, merging, and evaluation described in the above embodiments.

[0067] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0068] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0069] Other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A smart video color grading method based on LUT replication, merging, and evaluation, characterized in that, include: Perform shot segmentation and extract keyframes from the video before and after color grading; Feature vectors are extracted from the keyframes and similar shots are merged. LUTs that replicate similar lenses; The LUT evaluation network model is trained based on the keyframes of the pre-color grading shot and the feature vectors of the LUT images applied to the keyframes of the pre-color grading shot. Use the trained LUT to evaluate the network model and recommend LUTs for new video footage.

2. The method according to claim 1, characterized in that, The process of segmenting the video before and after color grading and extracting keyframes includes: The probability score of each frame as a shot cut point is calculated using a deep learning model. Set a score threshold, and perform shot segmentation based on the score threshold; For each shot after segmentation, extract several frames at equal intervals as keyframes for that shot.

3. The method according to claim 1, characterized in that, The step of extracting feature vectors based on the keyframes and merging similar shots includes: Extract the feature vectors of keyframes; The K-means algorithm is used to cluster all shots into several classes based on feature vectors; Replicate the LUT for each lens category; For each shot within each category, the PSNR value of the keyframe before color grading is calculated and then the corresponding LUT is applied sequentially to the keyframe after color grading. Based on the PSNR value, the lenses are clustered again within each lens category to obtain the final clustering and merging result.

4. The method according to claim 3, characterized in that, The LUT for replicating similar lenses includes: Collect all pixels of all keyframes for all shots in each category and establish a color mapping table before and after color grading. Construct a KD-tree to perform nearest neighbor search and generate a LUT; Smooth the LUT.

5. The method according to claim 1, characterized in that, The method of training a LUT evaluation network model based on the feature vectors of the LUT image applied to the pre-color grading keyframe and the pre-color grading keyframe LUT image includes: Extract the keyframes of the shot before color grading and the feature vectors of the LUT images of the keyframes of the shot before color grading. Construct feature vector samples for training LUT classification models and scene classification models; Set the loss function and training parameters; Train and save the LUT classification model and the scene classification model.

6. The method according to claim 5, characterized in that, The loss function uses InfoNCE loss, and the calculation steps include: Calculate the cosine similarity between the anchor sample and the feature vectors of the positive and negative samples; Divide the cosine similarity result by the temperature coefficient to obtain the positive sample similarity and negative sample similarity; The similarity scores of positive samples are concatenated with the similarity scores of all negative samples along the channel dimension to form a comparison vector; The positive sample class is used as the target label, and the cross-entropy loss is used to complete the optimization.

7. The method according to claim 5, characterized in that, The process of using a trained LUT to evaluate the network model for recommending LUTs for new video shots includes: Extract keyframes from new shots, and then apply each LUT to the keyframes sequentially. For each LUT applied, extract the keyframe feature vectors before and after application and input them into the scene classification model to obtain a scene classification score. If the scene classification score is lower than the threshold, the LUT is not recommended. If it is higher than the threshold, it is input into the LUT classification model to obtain a LUT classification score. Finally, sort all LUT classification scores and select the N LUTs with the highest classification scores as recommended LUTs.

8. A smart video color grading device based on LUT replication, merging, and evaluation, characterized in that, include: The extraction unit is used to segment the video before and after color grading and extract keyframes. A merging unit is used to extract feature vectors based on the keyframes and merge similar shots. Replica unit, used to replicate the LUT of the same type of lens; Training unit, used to train LUT evaluation network model based on key frames of the pre-color grading shot and feature vectors of the LUT image applied to the key frames of the pre-color grading shot; The recommendation unit is used to evaluate the network model using a trained LUT and recommend LUTs for new video shots.

9. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer instructions; The processor is configured to invoke computer instructions stored in the memory, causing the electronic device to perform the method as described in any one of claims 1-7.

Images