Video color processing method, device, medium and product for smart terminals

Abstract

The embodiment of the present application provides a video color processing method, device, medium and product for smart terminals. The method includes: determining a target frame in a video to be processed, and adjacent frames in the video to be processed that have a time sequence relationship with the target frame; inputting the target frame and the adjacent frames into the ST module in the color correction model for spatiotemporal feature extraction processing to obtain spatiotemporal features; inputting the spatiotemporal features into the predicted illumination module in the color correction model for prediction processing to obtain predicted illumination values; performing color correction processing on the target frame in the video to be processed according to the predicted illumination value to obtain a corrected target frame; adjusting the video to be processed according to the corrected target frame, and displaying the adjusted video to be processed. The method is used to achieve an improved correction effect on the video color.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video color processing method, device, medium and product for smart terminals. Background Art

[0002] In digital media processing, video color correction is an important technology used to adjust the color, contrast and hue of video images to improve video quality and visual experience. Currently, video color correction plays a key role in broadcasting, film production, monitoring systems and other fields.

[0003] In order to ensure the fidelity and temporal stability of the corrected video color, the temporal automatic white balance method needs to flexibly adapt to changes in illumination or content between frames and ensure that stable illumination estimates are maintained during these changes. Currently, some existing methods such as RCCNet (Recurrent color constancy Network) and TCCNet (Temporal color constancy Network) use a network framework based on long-short term memory (LSTM) to obtain the illumination value shared by a video frame by inputting the entire video frame sequence. This approach enables the system to better understand the timing and illumination changes in video data, thereby achieving more accurate white balance adjustments and improving video quality and visual experience.

[0004] However, the existing methods have the problem of poor effect when correcting video color. Summary of the invention

[0005] The embodiments of the present application provide a video color processing method, device, medium and product for smart terminals, so as to improve the correction effect of video color.

[0006] In a first aspect, an embodiment of the present application provides a video color processing method for a smart terminal, comprising:

[0007] Determine a target frame in the video to be processed and adjacent frames in the video to be processed that have a temporal relationship with the target frame;

[0008] The target frame and the adjacent frames are input into the ST module in the color correction model for spatiotemporal feature extraction to obtain spatiotemporal features;

[0009] The spatiotemporal features are input into the predicted illumination module in the color correction model for prediction processing to obtain the predicted illumination value;

[0010] According to the predicted illumination value, color correction processing is performed on the target frame in the processed video to obtain a corrected target frame;

[0011] According to the correction target frame, the video to be processed is adjusted, and the adjusted video to be processed is displayed.

[0012] In a possible implementation, the target frame and the adjacent frames are input into the ST module in the color correction model to extract the spatiotemporal features to obtain the spatiotemporal features, including:

[0013] According to the basic module units in the ST module, the target intermediate features of the target frame and the adjacent intermediate features of the adjacent frames are extracted;

[0014] According to the spatial feature extraction unit in the ST module and the target intermediate features, the spatial features are obtained;

[0015] The temporal features are obtained according to the temporal feature extraction unit in the ST module, the target intermediate features of the target frame, and the adjacent intermediate features of the adjacent frames.

[0016] In a possible implementation, after obtaining the spatiotemporal features according to the spatial features and the temporal features, the spatiotemporal features are input into the predicted illumination module in the color correction model for prediction processing, and before obtaining the predicted illumination value, the method further includes:

[0017] Perform dimensionality reduction processing on spatiotemporal features to obtain dimensionality reduction intermediate features;

[0018] Performing spatial feature extraction processing and temporal feature extraction processing on the dimensionality reduction intermediate features respectively to obtain dimensionality reduction spatial features and dimensionality reduction temporal features;

[0019] According to the reduced-dimensional space features and reduced-dimensional time series features, the target spatiotemporal features are obtained;

[0020] The target spatiotemporal features are used as spatiotemporal features, and the step of repeatedly performing dimensionality reduction processing on the spatiotemporal features to obtain the reduced dimensionality intermediate features is repeated until the number of repeated executions meets the preset number of executions. The obtained target spatiotemporal features are used as spatiotemporal features, and the spatiotemporal features are input into the predicted illumination module in the color correction model for prediction processing to obtain the predicted illumination value.

[0021] In a possible implementation, the spatiotemporal features are subjected to dimensionality reduction processing to obtain dimensionality-reduced intermediate features, including:

[0022] According to the intra-frame attention layer unit in the ST module, the spatial information of spatiotemporal features is extracted;

[0023] According to the first convolution layer, the second convolution layer and the activation normalization layer in the ST module, the spatial information of the spatiotemporal features is reduced in dimension to obtain the reduced-dimensional intermediate features, wherein the first convolution layer is a 3×3 depth convolution layer and the second convolution layer is a 1×1 convolution layer.

[0024] In a possible implementation, spatial feature extraction processing and temporal feature extraction processing are performed on the dimensionality reduction intermediate features respectively to obtain dimensionality reduction spatial features and dimensionality reduction temporal features, including:

[0025] According to the token mixer in the ST module, the dimension reduction intermediate features are transformed to obtain the initial dimension reduction space features of the dimension reduction intermediate features;

[0026] According to the channel mixer in the ST module, the initial reduced-dimensionality space features are transformed to obtain reduced-dimensionality space features;

[0027] According to the inter-frame attention layer in the ST module, the reduced-dimensionality intermediate features are processed for temporal feature extraction to obtain reduced-dimensionality temporal features.

[0028] In a possible implementation, according to the inter-frame attention layer in the ST module, the reduced-dimensionality intermediate features are subjected to temporal feature extraction processing to obtain reduced-dimensionality temporal features, including:

[0029] According to the inter-frame attention layer in the ST module, the Q information, K information and V information in the initial dimensionality reduction space feature are determined; the Q information and K information are normalized to obtain SoftMax(Q, K);

[0030] The SoftMax (Q, K) and V information are normalized to obtain the reduced-dimensional time series features that represent the attention information.

[0031] In a possible implementation, adjusting the video to be processed according to the correction target frame, and displaying the adjusted video to be processed includes:

[0032] According to the correction target frame, a global correction target frame of the video to be processed is obtained;

[0033] According to the global correction target frame, the video to be processed is adjusted to obtain a processed video;

[0034] Display the processed video.

[0035] In a second aspect, an embodiment of the present application provides a video color processing device for a smart terminal, comprising:

[0036] A determination module, used to determine a target frame in the video to be processed and adjacent frames in the video to be processed that have a time sequence relationship with the target frame;

[0037] A module is obtained, which is used to input the target frame and the adjacent frames into the ST module in the color correction model to perform spatiotemporal feature extraction processing to obtain spatiotemporal features;

[0038] A prediction module is used to input the spatiotemporal features into the prediction illumination module in the color correction model for prediction processing to obtain a predicted illumination value;

[0039] A color correction module is used to perform color correction processing on a target frame in the processed video according to the predicted illumination value to obtain a corrected target frame;

[0040] The display module is used to adjust the video to be processed according to the correction target frame, and to display the adjusted video to be processed.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0042] Memory stores computer-executable instructions;

[0043] The processor executes the computer execution instructions stored in the memory, so that the processor executes the method of the embodiment of the present application.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method of the embodiment of the present application.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that it includes a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0046] The video color processing method, device, medium and product for smart terminals provided in the embodiments of the present application obtain spatiotemporal features that characterize the spatiotemporal relationship between the target frame and the adjacent frames by inputting the target frame in the processed video and the adjacent frames having a temporal relationship with the target frame into a lightweight ST module in a color correction model, thereby reducing the amount of data processed by the color correction model when performing color correction and the parameter amount of the model parameters in the color correction model. The spatiotemporal features are then input into the predicted illumination module in the color correction model for prediction, so that the obtained predicted illumination value has better fidelity and temporal stability. Finally, color correction is performed on the processed video according to the predicted illumination value, and the video to be processed after color correction is displayed, thereby achieving the purpose of improving the correction effect on the video color. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 A schematic diagram of a scenario of a video color processing method for a smart terminal provided by the present application;

[0049] Figure 2A flowchart of a video color processing method for smart terminals provided in this application;

[0050] Figure 3a Another video color processing method for smart terminals provided by this application is shown in FIG. Figure 1 ;

[0051] Figure 3b Another video color processing method for smart terminals provided by this application is shown in FIG. Figure 2 ;

[0052] Figure 4 A schematic diagram of the structure of a video color processing device for smart terminals provided in this application;

[0053] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0054] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0056] First, the terms involved in this application are explained:

[0057] Temporal automatic white balance is a technology that aims to automatically adjust the white balance of video frames by analyzing the color information in the video sequence and its changes over time to eliminate color deviation caused by changes in light sources or other factors.

[0058] Color cast refers to the deviation or distortion of colors in an image or video due to factors such as light sources, photographic equipment or displays. Color cast may appear as certain colors in the image deviating from their true values ​​or changing due to ambient lighting conditions. Usually, color cast can be corrected by adjusting white balance, color correction, etc. to make the colors in the image or video closer to the real scene.

[0059] Inter-frame attention is a module that focuses on the correlation between video frames and is used to assist in extracting temporal features. RepViT (Revisiting Vision Transformer) is a visual processing architecture based on the intra-frame attention model, which is used to extract spatial features and process image data. The ST (Spatial-temporal) module is a spatiotemporal feature extraction module that combines RepViT's intra-frame attention and inter-frame attention. This module can be used to process input data containing spatiotemporal information, perform visual feature extraction and processing, to better capture spatiotemporal correlations and perform more efficient visual task processing. This can help process complex tasks that have both spatiotemporal information and visual information.

[0060] Transformer models excel in processing spatiotemporal data because they are able to capture long-range dependencies and complex features.

[0061] Inter-frame attention is a method for modeling the relationship and dependency between different frames in a video sequence. The inter-frame attention mechanism allows the model to focus on the information interaction at different time steps in the video sequence, thereby better capturing the long-range dependencies within the video.

[0062] Intra-frame attention is an attention mechanism used to model the relationships and dependencies between different regions or elements within a single video frame.

[0063] The gating mechanism refers to the structure in the LSTM network that controls the way information flows. It consists of a forget gate, an input gate, and an output gate, each of which contains a sigmoid activation function to adjust the information flow.

[0064] In order to ensure the fidelity and temporal stability of the corrected video color, the temporal automatic white balance method needs to flexibly adapt to changes in illumination or content between frames to maintain a stable illumination estimation. At present, existing methods such as RCCNet and TCCNet use a LSTM-based network framework to obtain a shared illumination value for a video frame by inputting the entire video frame sequence, and then use this shared illumination value to perform color correction on the entire video. However, the existing methods have the following two problems: First, the key to LSTM's extraction of spatiotemporal features lies in the gating mechanism, but its parameters are fixed. This makes it difficult for existing methods to adapt to changes in input frame content or illumination, and it is difficult to ensure the temporal stability of the corrected video color. Second, due to the Recurrent Neural Network (RNN) structure, the number of LSTM parameters is huge, making it difficult for existing methods to be used in common electronic devices such as mobile phones and SLRs. Therefore, there is an urgent need for a temporal automatic white balance method that guarantees color stability and is mobile device-friendly for video white balance correction on mobile devices.

[0065] The embodiment of the present application can integrate the modules of inter-frame attention and intra-frame attention to design an ST module for spatiotemporal feature extraction, thereby adjusting each target frame by processing the auxiliary effect provided by the adjacent frames of the target frame in the video, thereby ensuring the continuous stability of the corrected video color. At the same time, based on the lightweight ST module architecture, the number of parameters required for the model is reduced, so that it can be deployed on mobile devices with limited storage and computing power for operation and use. Therefore, while combining the effect of reducing the number of parameters by processing only one target frame each time, the accuracy and reasoning speed of the model are also improved, so that the video effect of color correction through the video color processing method for smart terminals is better.

[0066] Figure 1 A schematic diagram of a scenario of a video color processing method for a smart terminal provided in this application, such as Figure 1 As shown, the execution subject of the specific application scenario of the present application can be a server with a time-series automatic white balance model, wherein the server can be a mobile phone, a tablet, a computer and other devices. This embodiment does not impose any special restrictions on the implementation method of the execution subject, as long as the execution subject can determine the target frame in the video to be processed, and the adjacent frames in the video to be processed that have a time-series relationship with the target frame; input the target frame and the adjacent frame into the ST module in the color correction model for spatiotemporal feature extraction processing to obtain spatiotemporal features; input the spatiotemporal features into the predicted illumination module in the color correction model for prediction processing to obtain predicted illumination values; according to the predicted illumination values, color correction processing is performed on the target frame in the video to be processed to obtain a corrected target frame; according to the corrected target frame, the video to be processed is adjusted, and the adjusted video to be processed is displayed.

[0067] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] Figure 2 This is a flow chart of a video color processing method for smart terminals provided in this application. The execution subject of this method can be a terminal device with a display function or a storage function. This embodiment is not particularly limited here. Figure 2 As shown, the method includes:

[0069] S201, determining a target frame in a video to be processed and adjacent frames in the video to be processed that have a time sequence relationship with the target frame.

[0070] Among them, the video to be processed may refer to a video that may have color cast problems. In an embodiment of the present application, the video to be processed may be a video shot by the current device, or a video transmitted to the current device by other devices through network transmission or other methods.

[0071] The target frame may refer to an image frame in the video to be processed, wherein the image frame may be extracted from the video to be processed through a video library or software, for example, the image frame may be extracted using the FFmpeg command line tool. In an embodiment of the present application, after obtaining the video to be processed, the current device may extract the image frame in the video to be processed through the image processing module in the current device. After the image frame is extracted, the extracted image frame may be saved in a specified location in JPEG format, and any image frame in the image frame may be selected as the target frame.

[0072] The adjacent frames may refer to image frames having a temporal relationship with the target frame, wherein the temporal relationship may characterize the sequential relationship between the image frames. In some embodiments, the adjacent frames may include image frames selected from the upper N frames of the target frame and image frames selected from the lower M frames of the target frame, where M and N are both integers greater than 0. In the embodiment of the present application, the adjacent frames may be the upper image frame and the lower image frame of the target frame.

[0073] S202: Input the target frame and the adjacent frames into the ST module in the color correction model to perform spatiotemporal feature extraction processing to obtain spatiotemporal features.

[0074] Among them, the spatiotemporal feature extraction process refers to extracting spatiotemporal features that can describe its spatiotemporal changes from the target frame and adjacent frames. These spatiotemporal features can not only include spatial information (such as shape, color, texture, etc.), but also temporal changes (such as motion, speed, trajectory, etc.). In an embodiment of the present application, the ST module can be based on the Transformer architecture, and combined with self-attention mechanism, multi-head self-attention mechanism and re-parameterization technology, so as to efficiently capture and represent complex spatiotemporal relationships. And through multi-level feature extraction and fusion, an efficient spatiotemporal feature representation is generated.

[0075] In the embodiment of the present application, the target frame and the adjacent frames are input into the ST module in the color correction model to extract the spatiotemporal features. The method for obtaining the spatiotemporal features may include:

[0076] According to the basic module units in the ST module, the target intermediate features of the target frame and the adjacent intermediate features of the adjacent frames are extracted;

[0077] According to the spatial feature extraction unit in the ST module and the target intermediate features, the spatial features are obtained;

[0078] The temporal features are obtained according to the temporal feature extraction unit in the ST module, the target intermediate features of the target frame, and the adjacent intermediate features of the adjacent frames.

[0079] The color correction model may be a temporal automatic white balance model, which is a model for processing color deviation in a video. The temporal automatic white balance model may combine timing information and automatic white balance technology to correct color distortion problems in video frames caused by different light sources, photographic equipment, time changes of photographic equipment during shooting, or other factors.

[0080] When extracting intermediate features, the target frame and the adjacent frame can be input into the basic module unit in the ST module, and the target intermediate features of the target frame and the adjacent intermediate features of the adjacent frames can be extracted through the ViT encoder and the reparameterization operation in the basic module unit. In an embodiment of the present application, the basic module may include 2 3×3 convolutional layers and 1 activation normalization layer. When using the basic module unit for extraction, the target frame and the adjacent frame can be input into the first 3×3 convolutional layer, and then the output of the first 3×3 convolutional layer is applied to increase the nonlinear characteristics. After adjusting the activation value through the activation normalization layer to improve the stability and generalization ability of the model, the output of the first 3×3 convolutional layer is input into the second 3×3 convolutional layer, and then the activation function is applied to the output of the second 3×3 convolutional layer. Finally, the features are integrated and output to obtain the intermediate features.

[0081] When extracting spatial features, the spatial feature extraction unit can perform operations such as pooling, convolution, and intra-frame attention layer (token mixer + channel mixer) on the target intermediate features to capture the spatial information in the target frame and obtain spatial features. When extracting temporal features, the token mixer, inter-frame attention layer, and channel mixer in the temporal feature extraction unit can be used to obtain spatial features and temporal features, and finally the convolution layer and activation normalization layer are used to obtain spatiotemporal features.

[0082] Among them, in order to reduce the dimension and complexity of the data in the spatiotemporal features and speed up the data processing speed, in the embodiment of the present application, after the spatiotemporal features are obtained according to the spatial features and the temporal features, the spatiotemporal features are input into the predicted illumination module in the color correction model for prediction processing, and the method before obtaining the predicted illumination value may also include:

[0083] Perform dimensionality reduction processing on spatiotemporal features to obtain dimensionality reduction intermediate features;

[0084] Performing spatial feature extraction processing and temporal feature extraction processing on the dimensionality reduction intermediate features respectively to obtain dimensionality reduction spatial features and dimensionality reduction temporal features;

[0085] According to the reduced-dimensional space features and reduced-dimensional time series features, the target spatiotemporal features are obtained;

[0086] The target spatiotemporal features are used as spatiotemporal features, and the step of repeatedly performing dimensionality reduction processing on the spatiotemporal features to obtain the reduced dimensionality intermediate features is repeated until the number of repeated executions meets the preset number of executions. The obtained target spatiotemporal features are used as spatiotemporal features, and the spatiotemporal features are input into the predicted illumination module in the color correction model for prediction processing to obtain the predicted illumination value.

[0087] Among them, dimensionality reduction processing can refer to the process of mapping high-dimensional data to low-dimensional space. In this process, the most important information in the original data can be retained, while removing redundancy and noise in the data as much as possible to more effectively represent the data and improve the performance of the algorithm. In an embodiment of the present application, a downsampling layer can be used to perform dimensionality reduction processing on spatiotemporal features to obtain dimensionality reduction intermediate features.

[0088] Performing spatial feature extraction processing on the dimensionality reduction intermediate features may refer to performing spatial feature extraction on the spatial features related to the target frame in the dimensionality reduction intermediate features, thereby obtaining the dimensionality reduction spatial features related to the target frame. Performing temporal feature extraction processing on the dimensionality reduction intermediate features may refer to performing temporal feature extraction processing on the features related to the target frame and the temporal features related to the adjacent frames in the dimensionality reduction intermediate features, thereby obtaining the dimensionality reduction temporal features.

[0089] The preset number of executions may refer to the number of dimensionality reductions for spatiotemporal features. When the number of dimensionality reductions is larger, the data dimension can be reduced, thereby improving computational efficiency, but this may result in some information loss. When the number of dimensionality reductions is smaller, common electronic devices such as mobile phones and SLRs cannot withstand large amounts of data due to their small storage and poor data processing capabilities, resulting in poor video color processing effects. Therefore, specific situations are needed to evaluate and select the appropriate number of dimensionality reductions to retain sufficient information while reducing data dimensions and improving computational efficiency.

[0090] In an embodiment of the present application, the preset number of executions may be three times. When the preset number of executions is three times, the method for obtaining spatiotemporal features may include: a. reducing the dimension of the spatiotemporal features through a downsampling layer to obtain a reduced-dimension intermediate feature; b. extracting the temporal and spatial features of the reduced-dimension intermediate feature through a spatiotemporal feature layer to obtain a target spatiotemporal feature; c. using the target spatiotemporal feature as the spatiotemporal feature, repeating steps a and b, and obtaining the target spatiotemporal feature again; e. using the target spatiotemporal feature as the spatiotemporal feature, repeating steps a and b again, and finally obtaining a high-dimensional spatiotemporal feature, and using the high-dimensional spatiotemporal feature as the spatiotemporal feature after the number of repeated executions meets the preset number of executions. Among them, the downsampling layer may include 1 intra-frame attention layer, 1 3×3 deep convolution layer, 3 1×1 convolution layers, and 1 activation normalization layer.

[0091] Among them, in the embodiment of the present application, the method of performing dimensionality reduction processing on the spatiotemporal features to obtain the dimensionality reduction intermediate features may include:

[0092] According to the intra-frame attention layer unit in the ST module, the spatial information of spatiotemporal features is extracted;

[0093] According to the first convolution layer, the second convolution layer and the activation normalization layer in the ST module, the spatial information of the spatiotemporal features is subjected to dimensionality reduction processing to obtain dimensionality reduction intermediate features, wherein the first convolution layer is a 3×3 depth convolution layer and the second convolution layer is a 1×1 convolution layer. In the embodiment of the present application, the spatial feature extraction processing and the temporal feature extraction processing are respectively performed on the dimensionality reduction intermediate features to obtain the reduced dimensionality spatial features and the reduced dimensionality temporal features. The method may include:

[0094] According to the token mixer in the ST module, the dimension reduction intermediate features are transformed to obtain the initial dimension reduction space features of the dimension reduction intermediate features;

[0095] According to the channel mixer in the ST module, the initial reduced-dimensionality space features are transformed to obtain reduced-dimensionality space features;

[0096] According to the inter-frame attention layer in the ST module, the reduced-dimensionality intermediate features are processed for temporal feature extraction to obtain reduced-dimensionality temporal features.

[0097] The token mixer may be a 3×3 depth convolutional layer, and the channel mixer may be a 1×1 depth convolutional layer.

[0098] Among them, in the embodiment of the present application, according to the inter-frame attention layer in the ST module, the method for extracting the time series features of the reduced-dimensional intermediate features to obtain the reduced-dimensional time series features may include:

[0099] According to the inter-frame attention layer in the ST module, the Q information, K information and V information in the initial dimensionality reduction space feature are determined; the Q information and K information are normalized to obtain SoftMax(Q, K);

[0100] The SoftMax (Q, K) and V information are normalized to obtain the reduced-dimensional time series features that represent the attention information.

[0101] Among them, Q information can refer to Query, which is a representation of the item currently being processed in the attention mechanism. For example, if you are translating a sentence, the query may be a representation of the target language word you are currently trying to predict. The query interacts with the key to determine the focus of attention. In an embodiment of the present application, the Q information can represent the search for information related to the current position of the target frame.

[0102] K information may refer to a Key, where a key is part of a mechanism used by the model to associate importance with different parts of the input data. Each key is associated with an input item. The model uses a query to compare against all keys to determine the degree of "attention" or relevance that each input part should receive. V information may refer to a Value, where a value is also associated with an input item. Once the model determines the degree of attention for each input part, these attention weights are used to weight the values. The weighted values ​​are then summed to produce the output of the attention mechanism, which is then fed into a deeper level of the model or used to generate predictions. In an embodiment of the present application, K information and V information may characterize a search for adjacent frames that are similar to or contain content from a target frame.

[0103] S203 , inputting the spatiotemporal features into a predicted illumination module in a color correction model for prediction processing to obtain a predicted illumination value.

[0104] Among them, the predicted illumination module may refer to a model or algorithm for estimating the illumination conditions of a scene in an image or video. This module can analyze the input data and predict the illumination value in the target frame. In an embodiment of the present application, the predicted illumination module can use deep learning techniques, such as convolutional neural networks (CNN) or recurrent neural networks (RNN), combined with spatial and temporal features extracted to predict the illumination value. These modules can infer the illumination information of the target frame by learning the relationship between the input frames and the relationship between the input frames and the target frames.

[0105] S204: Perform color correction processing on the target frame in the video to be processed according to the predicted illumination value to obtain a corrected target frame.

[0106] The color correction process may refer to a process of adjusting the illumination value of the target frame according to the predicted illumination value.

[0107] S205: adjusting the video to be processed according to the correction target frame, and displaying the adjusted video to be processed.

[0108] Among them, according to the correction target frame, adjusting the video to be processed can refer to adjusting one frame image in the video to be processed, or it can refer to adjusting multiple frame images or all frame images in the video to be processed. When adjusting multiple frame images and all frame images, steps S201 to S204 can be repeated continuously to obtain correction target frames of multiple different frame images. Then, the video to be processed can be adjusted according to the correction target frames of multiple different frame images.

[0109] Displaying the adjusted video to be processed may refer to a method in which the current device generates a file of the video to be processed or plays the video to be processed for display.

[0110] In the embodiment of the present application, the method of adjusting the video to be processed and displaying the adjusted video to be processed according to the correction target frame may include:

[0111] According to the correction target frame, a global correction target frame of the video to be processed is obtained;

[0112] According to the global correction target frame, the video to be processed is adjusted to obtain a processed video;

[0113] Display the processed video.

[0114] The global correction target frame may include the correction target frames of all image frames in the video to be processed.

[0115] The video color processing method for smart terminals provided in the embodiment of the present application obtains spatiotemporal features characterizing the spatiotemporal relationship between the target frame and the adjacent frames by inputting the target frame in the processed video and the adjacent frames having a temporal relationship with the target frame into a lightweight ST module in a color correction model, thereby reducing the amount of data processed by the color correction model when performing color correction and the parameter amount of the model parameters in the color correction model. The spatiotemporal features are then input into the predicted illumination module in the color correction model for prediction, so that the obtained predicted illumination value has better fidelity and temporal stability. Finally, the processed video is color corrected according to the predicted illumination value, and the video to be processed after the color correction is displayed, thereby achieving the purpose of improving the correction effect on the video color.

[0116] Figure 3a Another video color processing method for smart terminals provided by this application is shown in FIG. Figure 1 , Figure 3b Another video color processing method for smart terminals provided by this application is shown in FIG. Figure 2 ,like Figure 3a and 3b As shown, the method includes:

[0117] S301: Determine a target frame of an input color-shifted video and its preceding and succeeding adjacent frames.

[0118] The size of the color-shifted video target frame and the adjacent frames can be expressed as 3×C×H×W. C is the number of channels of the image, H is the height of the image, which can be expressed by the number of pixels of the image in the vertical dimension; W is the width of the image, which can be expressed by the number of pixels of the image in the horizontal dimension. In some embodiments, the size of the color-shifted video target frame and the adjacent frames can be 3×3×224×224, indicating that there are 3 images input, each image has three channels of red (Red, R), green (Green, G), and blue (Blue, B), with a height of 224px and a width of 224px.

[0119] Optionally, the color-biased video target frame can be processed by the following feasible implementation method: feature extraction is performed on the color-biased video target frame through a spatiotemporal feature extraction module to obtain temporal and spatial features; the temporal and spatial features are processed through a predicted illumination module to obtain predicted illumination values; the illumination values ​​are applied to the color-biased video frame to obtain a video frame without color deviation.

[0120] S302: Input the target frame and the preceding and following adjacent frames into a spatiotemporal feature extraction module to obtain corresponding spatiotemporal features.

[0121] Among them, the spatiotemporal feature extraction module may include a basic module, a first spatiotemporal feature layer, a second spatiotemporal feature layer, a third spatiotemporal feature layer, a fourth spatiotemporal feature layer and three downsampling layers.

[0122] Among them, the basic module includes 2 3×3 convolutional layers and 1 activation normalization layer. The first spatiotemporal feature layer, the second spatiotemporal feature layer, the third spatiotemporal feature layer, and the fourth spatiotemporal feature layer may include 1 spatiotemporal attention layer, 1 3×3 convolutional layer, and 1 activation normalization layer. The downsampling layer may include 1 intra-frame attention layer, 1 3×3 deep convolutional layer, 3 1×1 convolutional layers, and 1 activation normalization layer, wherein the inter-frame attention in the spatiotemporal attention layer may be based on the formula:

[0123]

[0124] Q=W Q v i ;

[0125] K′=W K [v i-1 ;v i+1 ];

[0126] V′=W V [v i-1 ;v i+1 ];

[0127] Among them, Q is the information from the target frame, and K′, V′ are the information from the adjacent frames.

[0128] Among them, when the size of the color-biased video target frame and the preceding and following adjacent frames is 3×3×224×224, after the basic module, the spatial feature can be preliminarily obtained, and its size can be 3×32×56×56, indicating that the number of channels of the image block is 32, the height is 56px, and the width is 56px; then, the first spatiotemporal feature layer can perform a feature extraction operation on the image block to obtain a first spatiotemporal feature, and its size can be 3×32×56×56, indicating that the number of channels of the first spatiotemporal feature is 32, the height is 56px, and the width is 56px; then, after the downsampling layer, the first intermediate feature block can be obtained, and its size can be 3×64×28×28, indicating that the number of channels of the first intermediate feature block is 64, the height is 28px, and the width is 28px; further, the second spatiotemporal feature layer can perform a feature extraction operation on the first intermediate feature block to obtain a second spatiotemporal feature, and its size is 3×64×28×28, indicating that the number of channels of the second spatiotemporal feature is 64, the height is 28px, and the width is 28px. is 28px and 28px wide; then, after the downsampling layer, the second intermediate feature block can be obtained, whose size can be 3×128×14×14, indicating that the number of channels of the second intermediate feature block is 128, the height is 14px, and the width is 14px; further, the third spatiotemporal feature layer can perform a feature extraction operation on the second intermediate feature block to obtain a third spatiotemporal feature, whose size is 3×128×14×14, indicating that the number of channels of the third spatiotemporal feature is 128, the height is 14px, and the width is 14px; then, after the downsampling layer, the third intermediate feature block can be obtained, whose size can be 3×256×7×7, indicating that the number of channels of the third intermediate feature block is 256, the height is 7px, and the width is 7px; finally, the fourth spatiotemporal feature layer can perform a feature extraction operation on the third intermediate feature block to obtain a fourth spatiotemporal feature, whose size is 3×256×7×7, indicating that the number of channels of the third spatiotemporal feature is 256, the height is 7px, and the width is 7px.

[0129] S303: Input the corresponding spatiotemporal features into a predicted illumination module to obtain an illumination value for white balance.

[0130] The predicted illumination module may include an averaging operation, two linear layers, and two activation normalization layers.

[0131] Among them, if the size of the spatiotemporal feature F is 3×256×7×7, then after the averaging operation, the size becomes 3×256×1×1, indicating that the number of channels is 256, the height is 1px, and the width is 1px; after the first linear layer and the activated normalization layer, the size is 3×128×1×1, indicating that the number of channels is 128, the height is 1px, and the width is 1px; after the second linear layer and the activated normalization layer, the size is 3×3×1×1, indicating that the number of channels is 3, the height is 1px, and the width is 1px; take the predicted value of the target frame, and the illumination value is 3×1×1, indicating that the number of channels is 3, which are red (Red, R), green (Green, G), and blue (Blue, B).

[0132] S304: adjusting the color-biased video target frame according to the illumination value of the white balance to obtain a white-balanced target video frame, and obtaining a video without color bias according to the white-balanced target video frame.

[0133] Among them, by multiplying the predicted illumination value obtained by the time-series automatic white balance model with the normalized value of the target video frame, the target video frame after white balance can be obtained, and then the target video frame without color deviation can be obtained after gamma processing. This correction operation is performed on all video frames in sequence to obtain a video without color deviation.

[0134] Among them, the temporal automatic white balance model can be obtained by model training, wherein the training samples may include sample color-biased video frames and corresponding sample color-free video frames. In an embodiment of the present application, the sample color-biased video frames may be cropped and / or data enhanced to obtain a large number of sample color-biased video frames to enrich the training samples. For example, the video frame can be cropped to 3×224×224. Then, the illumination value output by the temporal automatic white balance model and the actual illumination value can be input into the network training supervision function to calculate the loss Loss value, and then the model parameters of the temporal automatic white balance model are updated according to the Loss value until the Loss value is stable to obtain a trained temporal automatic white balance model.

[0135] Another video color processing method for smart terminals provided in an embodiment of the present application can integrate the modules of inter-frame attention and intra-frame attention, design a spatiotemporal feature extraction module, improve the correction effect of color-biased video white balance and inter-frame stability, and reduce the number of parameters required for the method based on a lightweight module, thereby improving the accuracy and reasoning speed of the model. It can be applicable to mobile devices, thereby significantly reducing the number of parameters while improving the fidelity and timing stability of the corrected video color, thereby reducing the color deviation error by 3%, improving the stability by 33%, reducing the number of parameters by 80%, and speeding up the processing speed by 80%, thereby achieving the effect of improving video color processing.

[0136] Figure 4The schematic diagram of the structure of the video color processing device for smart terminals provided in this application is as follows: Figure 4 As shown, the video color processing device 40 for smart terminals provided in this embodiment includes: a determination module 401, an acquisition module 402, a prediction module 403, a color correction module 404 and a display module 405. Among them:

[0137] A determination module 401 is used to determine a target frame in the video to be processed and adjacent frames in the video to be processed that have a time sequence relationship with the target frame;

[0138] Obtaining module 402, for inputting the target frame and the adjacent frames into the ST module in the color correction model for spatiotemporal feature extraction processing to obtain spatiotemporal features;

[0139] Prediction module 403, used for inputting the spatiotemporal features into the prediction illumination module in the color correction model for prediction processing to obtain a predicted illumination value;

[0140] A color correction module 404 is used to perform color correction processing on a target frame in the video to be processed according to the predicted illumination value to obtain a corrected target frame;

[0141] The display module 405 is used to adjust the video to be processed according to the correction target frame, and display the adjusted video to be processed.

[0142] In the embodiment of the present application, the obtaining module 402 may also be specifically used for:

[0143] According to the basic module units in the ST module, the target intermediate features of the target frame and the adjacent intermediate features of the adjacent frames are extracted;

[0144] According to the spatial feature extraction unit in the ST module and the target intermediate features, the spatial features are obtained;

[0145] The temporal features are obtained according to the temporal feature extraction unit in the ST module, the target intermediate features of the target frame, and the adjacent intermediate features of the adjacent frames.

[0146] In the embodiment of the present application, the obtaining module 402 may also be used for:

[0147] Perform dimensionality reduction processing on spatiotemporal features to obtain dimensionality reduction intermediate features;

[0148] Performing spatial feature extraction processing and temporal feature extraction processing on the dimensionality reduction intermediate features respectively to obtain dimensionality reduction spatial features and dimensionality reduction temporal features;

[0149] According to the reduced-dimensional space features and reduced-dimensional time series features, the target spatiotemporal features are obtained;

[0150] The target spatiotemporal features are used as spatiotemporal features, and the step of repeatedly performing dimensionality reduction processing on the spatiotemporal features to obtain the reduced dimensionality intermediate features is repeated until the number of repeated executions meets the preset number of executions. The obtained target spatiotemporal features are used as spatiotemporal features, and the spatiotemporal features are input into the predicted illumination module in the color correction model for prediction processing to obtain the predicted illumination value.

[0151] In the embodiment of the present application, the obtaining module 402 may also be specifically used for:

[0152] According to the intra-frame attention layer unit in the ST module, the spatial information of spatiotemporal features is extracted;

[0153] According to the first convolution layer, the second convolution layer and the activation normalization layer in the ST module, the spatial information of the spatiotemporal features is reduced in dimension to obtain the reduced-dimensional intermediate features, wherein the first convolution layer is a 3×3 depth convolution layer and the second convolution layer is a 1×1 convolution layer.

[0154] In the embodiment of the present application, the obtaining module 402 may also be specifically used for:

[0155] According to the token mixer in the ST module, the dimension reduction intermediate features are transformed to obtain the initial dimension reduction space features of the dimension reduction intermediate features;

[0156] According to the channel mixer in the ST module, the initial reduced-dimensionality space features are transformed to obtain reduced-dimensionality space features;

[0157] According to the inter-frame attention layer in the ST module, the reduced-dimensionality intermediate features are processed for temporal feature extraction to obtain reduced-dimensionality temporal features.

[0158] In the embodiment of the present application, the obtaining module 402 may also be specifically used for:

[0159] According to the inter-frame attention layer in the ST module, the Q information, K information and V information in the initial dimensionality reduction space feature are determined; the Q information and K information are normalized to obtain SoftMax(Q, K);

[0160] The SoftMax (Q, K) and V information are normalized to obtain the reduced-dimensional time series features that represent the attention information.

[0161] In the embodiment of the present application, the display module 405 may also be specifically used for:

[0162] According to the correction target frame, a global correction target frame of the video to be processed is obtained;

[0163] According to the global correction target frame, the video to be processed is adjusted to generate an adjusted video to be processed;

[0164] Display the adjusted video to be processed.

[0165] The video color processing device for smart terminals provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0166] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0167] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0168] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0169] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0170] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0171] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0172] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0173] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0174] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0175] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0176] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0179] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0180] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0181] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A video color processing method for intelligent terminals, characterized in that: include: Determine a target frame in a video to be processed, and adjacent frames in the video to be processed that have a temporal relationship with the target frame; The target frame and the adjacent frames are input into the ST module in the color correction model for spatiotemporal feature extraction processing to obtain spatiotemporal features, including: extracting the target intermediate features of the target frame and the adjacent intermediate features of the adjacent frames according to the basic module unit in the ST module; obtaining spatial features according to the spatial feature extraction unit in the ST module and the target intermediate features; obtaining temporal features according to the temporal feature extraction unit in the ST module, the target intermediate features of the target frame, and the adjacent intermediate features of the adjacent frames; obtaining spatiotemporal features according to the spatial features and the temporal features; Inputting the spatiotemporal features into a predicted illumination module in a color correction model for prediction processing to obtain a predicted illumination value; According to the predicted illumination value, color correction processing is performed on the target frame in the to-be-processed video to obtain a corrected target frame; According to the correction target frame, the video to be processed is adjusted, and the adjusted video to be processed is displayed.

2. The method according to claim 1, characterized in that After obtaining the spatiotemporal features according to the spatial features and the temporal features, the spatiotemporal features are input into a predicted illumination module in a color correction model for prediction processing, and before obtaining the predicted illumination value, the method further includes: Performing dimensionality reduction processing on the spatiotemporal features to obtain dimensionality reduction intermediate features; Respectively performing spatial feature extraction processing and temporal feature extraction processing on the dimensionality reduction intermediate features to obtain dimensionality reduction spatial features and dimensionality reduction temporal features; Obtaining target spatiotemporal features according to the reduced-dimensionality spatial features and the reduced-dimensionality temporal features; The target spatiotemporal features are used as the spatiotemporal features, and the step of performing dimensionality reduction processing on the spatiotemporal features to obtain dimensionality reduction intermediate features is repeatedly performed until the number of repeated executions meets the preset number of executions, and the obtained target spatiotemporal features are used as the spatiotemporal features, and the step of inputting the spatiotemporal features into the predicted illumination module in the color correction model for prediction processing to obtain predicted illumination values ​​is performed.

3. The method according to claim 2, characterized in that The performing dimensionality reduction processing on the spatiotemporal features to obtain dimensionality reduction intermediate features includes: Extracting spatial information of spatiotemporal features according to the intra-frame attention layer unit in the ST module; According to the first convolution layer, the second convolution layer and the activation normalization layer in the ST module, the spatial information of the spatiotemporal features is subjected to dimensionality reduction processing to obtain dimensionality reduced intermediate features, wherein the first convolution layer is a 3×3 deep convolution layer and the second convolution layer is a 1×1 convolution layer.

4. The method according to claim 2, characterized in that: The step of performing spatial feature extraction processing and temporal feature extraction processing on the dimensionality reduction intermediate features to obtain dimensionality reduction spatial features and dimensionality reduction temporal features comprises: According to the token mixer in the ST module, the dimension reduction intermediate feature is converted to obtain the initial dimension reduction space feature of the dimension reduction intermediate feature; According to the channel mixer in the ST module, the initial reduced-dimensionality space feature is converted to obtain a reduced-dimensionality space feature; According to the inter-frame attention layer in the ST module, the reduced-dimensional intermediate features are subjected to temporal feature extraction processing to obtain reduced-dimensional temporal features.

5. The method according to claim 4, characterized in that The step of performing temporal feature extraction processing on the dimension reduction intermediate feature according to the inter-frame attention layer in the ST module to obtain the dimension reduction temporal feature includes: According to the inter-frame attention layer in the ST module, the Q information, the K information and the V information in the initial dimensionality reduction space feature are determined; the Q information and the K information are normalized to obtain SoftMax (Q, K); The SoftMax (Q, K) and the V information are normalized to obtain a reduced-dimensional time series feature representing the attention information.

6. The method according to claim 1, characterized in that The step of adjusting the video to be processed according to the correction target frame, and displaying the adjusted video to be processed includes: According to the correction target frame, a global correction target frame of the video to be processed is obtained; According to the global correction target frame, the video to be processed is adjusted to obtain a processed video; The processed video is displayed.

7. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

9. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images