Video processing method and device, electronic device, computer readable storage medium

Abstract

The present disclosure relates to a video processing method and device, electronic equipment and storage medium, the video processing method comprising: obtaining a video frame of a video to be processed; performing color adjustment on the video frame by using at least one color enhancement model in a color enhancement model pool to obtain a video output frame after color adjustment; wherein the color enhancement model pool contains a model set for color adjustment of videos with different style requirements for multiple video content types.

Description

Technical Field

[0001] This disclosure relates to image processing technology and artificial intelligence, specifically to video processing methods and apparatus, electronic devices, and computer-readable storage media. Background Technology

[0002] Color enhancement is a crucial technique in image processing. It adjusts the colors of images and videos to make them richer and more realistic, significantly improving video quality. This technique can be applied to computer vision scenarios such as video color creation and source image quality enhancement. Summary of the Invention

[0003] This disclosure provides a video processing method and apparatus, an electronic device, and a computer-readable storage medium.

[0004] In a first aspect, embodiments of this disclosure provide a video processing method, comprising: acquiring video frames of a video to be processed; adjusting the colors of the video frames using at least one color enhancement model from a color enhancement model pool to obtain a color-adjusted video output frame; wherein the color enhancement model pool comprises a set of models for color adjustment of videos with different style requirements for various video content types.

[0005] The step of using at least one color enhancement model from the color enhancement model pool to perform color adjustment on the video frame to obtain a color-adjusted video output frame includes:

[0006] Receive information on the video content type and style requirements of the video frame; select a color enhancement model from the color enhancement model pool that corresponds to the video content type and style requirements; process the video frame using the selected color enhancement model to obtain the color-adjusted video output frame.

[0007] The step of adjusting the video frame using at least one color enhancement model from the color enhancement model pool to obtain a color-adjusted video output frame includes: processing the video frame using each color enhancement model from the color enhancement model pool to obtain a color-adjusted video frame output by each color enhancement model; processing the video frame based on a content analysis model to generate a weight value corresponding to each color enhancement model and the video frame; wherein the content analysis model is configured to determine the weight value of each color enhancement model in the color enhancement model pool based on the video frame; and using the weight values ​​corresponding to each color enhancement model and the video frame to perform weighted fusion processing on the color-adjusted video frames output by each color enhancement model to generate the video output frame.

[0008] The color enhancement model includes an image reconstruction branch model and a color enhancement branch model; the step of processing the video frame using the selected color enhancement model to obtain the color-adjusted video output frame includes:

[0009] Feature extraction is performed on the video frame to obtain a first image feature; the video frame is filtered, and feature extraction is performed on the filtered video frame to obtain a second image feature; the first image feature is input into the image reconstruction branch model, and the second image feature is input into the color enhancement branch model, so as to perform image reconstruction and color adjustment based on the first image feature and the second image feature; feature fusion processing is performed on the output results of the image reconstruction branch model and the output results of the color enhancement branch model to obtain a fusion result; based on the fusion result, a color-adjusted video output frame is generated.

[0010] The image reconstruction branch model and the color enhancement branch model both include N dense residual connection network modules (RDB); the color enhancement model also includes N fully connected layers; where N is an integer greater than or equal to 1; the step of inputting the first image feature into the image reconstruction branch model and the second image feature into the color enhancement branch model to perform image reconstruction and color adjustment based on the first image feature and the second image feature includes:

[0011] The first RDB in the image reconstruction branch model and the first RDB in the color enhancement branch model are obtained as the current RDB of the first branch model and the current RDB of the second branch model, respectively. The first image features are processed using the current RDB of the first branch model, and the second image features are processed using the current RDB of the second branch model, resulting in a feature map output by the current RDB of the first branch model and low-frequency color features output by the current RDB of the second branch model. The low-frequency color features output by the current RDB of the second branch model are processed through a fully connected layer to obtain a feature vector. This feature vector is then compared with the feature map output by the current RDB of the first branch model. The corresponding images are multiplied to obtain a feature map adjusted for local color features. The feature map adjusted for local color features is used as a new first image feature, and the low-frequency color features output by the current RDB of the second branch model are used as a new second image feature. The next RDB in the image reconstruction branch model and the next RDB in the color enhancement branch model are obtained as the current RDB of the new first branch model and the current RDB of the new second branch model, respectively, until the number of times is equal to N. The newly obtained feature map adjusted for local color features is used as the output result of the image reconstruction branch model, and the newly obtained low-frequency color features are used as the output result of the color enhancement branch model.

[0012] The RDB includes a densely connected network layer and a local feature fusion layer. The local feature fusion layer is used to add the input features of the RDB to the output features of the densely connected network layer using local residual learning to obtain the output features of the RDB. The densely connected layer includes M densely connected modules and 1 connection module, where M is an integer greater than or equal to 1.

[0013] The input to the first dense connection module is the input feature of the RDB; the input feature of the i-th dense connection module is the input feature of the RDB and the output feature of each dense connection module preceding the i-th dense connection module, where i is an integer greater than 1 and less than or equal to M; the input to the connection module is the input feature of the RDB and the output feature of each dense connection module.

[0014] Each color enhancement model in the color enhancement model pool has the same model structure; the color enhancement models in the color enhancement model pool are pre-trained color enhancement models obtained by training an initial color enhancement model with different color enhancement data pairs for different video content types; the color enhancement data pairs include: uncolored image blocks and colored image blocks with the same image size selected from the same image position in the uncolored image and the first colored image, wherein the first colored image is an image obtained in advance after color correction of the uncolored image based on the image color style features.

[0015] The training of the initial color enhancement model includes: processing uncolored image patches selected from the uncolored image using the initial color enhancement model to obtain color adjustment results for the uncolored image patches, and obtaining colored image patches corresponding to the uncolored image patches from the color enhancement data pairs as color adjustment reference results; constructing a loss function for the initial color enhancement model based on the color adjustment results and the color adjustment reference results; adjusting the model parameters of the initial color enhancement model using the loss function to obtain an updated color enhancement model; using the updated color enhancement model as a new initial color enhancement model, and processing uncolored image patches selected from the uncolored image using the new initial color enhancement model, until the training process of the initial color enhancement model meets a first training termination condition, thereby obtaining the trained color enhancement model; wherein, the first training termination condition includes at least one of the following conditions: the training loss calculated according to the loss function of the initial color enhancement model is less than or equal to a set first loss threshold, and the number of training iterations of the initial color enhancement model reaches a set first number threshold.

[0016] The step of constructing the loss function of the initial color enhancement model based on the color adjustment result and the color adjustment reference result includes: calculating the minimum squared error between the color adjustment result and the color adjustment reference result to obtain the minimum squared error loss; calculating the Gaussian filtered output result of the color adjustment result and the Gaussian filtered output result of the color adjustment reference result to obtain the color loss; and performing a weighted summation of the minimum squared error loss and the color loss to generate the loss function of the initial color enhancement model.

[0017] The content analysis model includes: a convolutional network with K sequentially connected convolutional structures and two fully connected layers. Each of the first K-1 convolutional structures includes a convolutional layer, an activation layer and a pooling layer. The Kth convolutional structure includes a convolutional layer and an activation layer.

[0018] The content analysis model is a pre-trained content analysis model obtained by training the initial content analysis model with image data pairs of different styles, based on each color enhancement model in the pre-trained color enhancement model pool. The image data pairs include: uncolored image blocks and colored image blocks of the same size selected from the same image position in the uncolored image and the second colored image. The second colored image is an image obtained in advance after color correction of the uncolored image based on different video content types.

[0019] The training of the initial content analysis model includes:

[0020] Using each color enhancement model in the pre-trained color enhancement model pool, the uncolored image patch selected from the uncolored image is processed to obtain the color adjustment result of each color enhancement model for the uncolored image patch; the uncolored image patch is processed using an initial content analysis model to obtain a predetermined number of weight values, the predetermined number being the same as the number of color enhancement models in the color enhancement model pool;

[0021] According to the predetermined number of weight values, the color adjustment results of each color enhancement model on the uncolored image patch are weighted and fused to obtain the color enhancement result of the uncolored image patch. A color-corrected image patch corresponding to the uncolored image patch is obtained from the image data pair as the color enhancement reference result. Based on the color enhancement result of the uncolored image patch and the color enhancement reference result, a loss function of the initial content analysis model is constructed. The model parameters of the content analysis model are adjusted using the loss function of the initial content analysis model to obtain an updated content analysis model. The updated content analysis model is used as a new initial content analysis model, and the uncolored image patch is processed using the new initial content analysis model until the training process of the initial content analysis model meets the second training termination condition, resulting in the trained content analysis model. The second training termination condition includes at least one of the following conditions: the training loss calculated according to the loss function of the initial content analysis model is less than or equal to a set second loss threshold; the number of training iterations of the initial content analysis model reaches a set second iteration threshold.

[0022] The loss function of the initial content analysis model is a loss function constructed by minimizing the squared error of the color enhancement result of the uncolored image patch and the color enhancement reference result.

[0023] Secondly, embodiments of this disclosure provide a video processing apparatus, comprising: an acquisition module for acquiring a video frame to be processed; and a processing module for adjusting the color of the video frame using at least one color enhancement model from a color enhancement model pool to obtain a color-adjusted video output frame; wherein the color enhancement model pool includes a set of models for color adjustment of videos with different style requirements for various video content types.

[0024] Thirdly, embodiments of this disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any one of the video processing methods of this disclosure.

[0025] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the video processing methods of this disclosure.

[0026] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0027] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0028] Figure 1 This is a flowchart of a video processing method in some embodiments of this disclosure;

[0029] Figure 2 This is a flowchart illustrating color adjustment processing in some embodiments of this disclosure;

[0030] Figure 3 This is a flowchart illustrating color adjustment processing in some embodiments of this disclosure;

[0031] Figure 4a This is a schematic diagram of the structure of a video processing system according to some embodiments of the present disclosure;

[0032] Figure 4b This is a schematic diagram of the structure of a video processing system according to some embodiments of the present disclosure;

[0033] Figure 5 This is a schematic diagram of the specific architecture of the color enhancement model according to an embodiment of the present disclosure;

[0034] Figure 6 This is a flowchart illustrating the specific process of adjusting the color of video frames using a color enhancement model in an embodiment of this disclosure.

[0035] Figure 7 This is a flowchart illustrating the specific process of image reconstruction and color adjustment in this embodiment of the disclosure;

[0036] Figure 8 This is a detailed architecture diagram of the dense residual connection network module in the embodiments of this disclosure;

[0037] Figure 9 This is a schematic diagram of the specific architecture of the content analysis model in the embodiments of this disclosure;

[0038] Figure 10 This is a schematic diagram of the structure of the video processing apparatus according to an embodiment of the present disclosure;

[0039] Figure 11 A block diagram of an electronic device for implementing the video processing method of the embodiments of this disclosure. Detailed Implementation

[0040] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0041] In practical applications, with the development of image display technology, users have placed higher demands on viewing experience, and video quality enhancement display applications, such as ultra-high-definition (UHD) displays, are gradually appearing in daily life. Currently, my country's UHD display industry chain is also continuously improving. To obtain better video quality, higher requirements are placed on both video image quality and the hardware capabilities of display terminals. From UHD cameras at the acquisition end, to UHD resolution large screens and High Dynamic Range (HDR) televisions at the display end, to next-generation broadband mobile communication technology connecting networks at the transmission end, and UHD television stations, numerous enterprises and institutions are investing in these areas, driving the rapid development of image enhancement display technologies such as UHD displays.

[0042] In the embodiments of this disclosure, the resolution of the display terminal may include various display formats such as Standard Definition (SD), High Definition (HD), Full High Definition (FHD), and Ultra High Definition (Ultra HD).

[0043] For example, standard definition resolution can be 480×320 pixels or 640×480 pixels, high definition resolution can be 1024×720 pixels, full high definition resolution can be 1920×1080 pixels, and ultra-high definition resolution can be 3840×2160 pixels, that is, 3840 pixels can be displayed horizontally and 2160 pixels can be displayed vertically on the entire display screen. Since ultra-high definition resolution can reach four times that of full high definition resolution, ultra-high definition resolution can also be called 4K resolution.

[0044] For example, 8K resolution can be 7680×4320 pixels, meaning that 7680 pixels can be displayed horizontally and 4320 pixels vertically on the entire display screen. In this embodiment of the disclosure, 8K resolution can also be referred to as ultra-high definition resolution.

[0045] For example, the next-generation broadband mobile communication technology connection network can be a communication network using 5G or a higher network standard. With advancements in display and mobile communication technologies, the high-speed and low-latency communication networks provided by next-generation broadband mobile communication technologies offer better support for the widespread adoption of ultra-high-definition video.

[0046] For example, dynamic range refers to the ratio of the brightest to the darkest areas in an image, that is, the number of gray levels in the image from the brightest to the darkest. The larger the dynamic range, the richer the levels it can represent, and the wider the color space it includes. High dynamic range refers to a ratio from the brightest to the darkest that can reach a very high level (exceeding a predetermined threshold). Through grayscale quantization, high dynamic range images (i.e., HDR images) can provide more dynamic range and image detail. HDR TVs are devices that support HDR technology. The use of HDR technology can increase the detail in the picture, improve picture quality, and maintain the realism of image colors.

[0047] In some application scenarios, the technology of ultra-high-definition (UHD) equipment platforms is developing rapidly, while the production of UHD video content is lagging behind. The existing stock of 4K / 8K content often cannot meet the demands of UHD playback, but there is a large inventory of standard-definition (SD) and high-definition (HD) videos that cannot be played on UHD display platforms. Therefore, remastering SD and HD content into UHD format is the fastest and most direct way to solve the shortage of UHD content.

[0048] Because ultra-high-definition (UHD) video adopts the color space definition in the television display standards of the new generation of UHD video production and display systems—for example, 4K / 8K video can use the BT.2020 color gamut—which defines the largest color gamut achievable using the three primary colors to date, it provides greater scope for color creation. However, current video color creation is often done manually, and the UHD remastering of low-definition sources often relies on manual processing, which cannot be automated. This results in long source production cycles, high labor costs, and low processing efficiency.

[0049] This disclosure provides a video processing method and apparatus, an electronic device, and a computer-readable storage medium. The method can be executed by an electronic device such as a terminal device or a server. The terminal device may include, but is not limited to, personal computers, smartphones, tablet computers, personal digital assistants (PDAs), servers, etc. The video processing method can be implemented by a processor calling computer-readable program instructions stored in memory; or, the method can be executed by a server.

[0050] To better understand this disclosure, a video processing method according to embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that these embodiments are not intended to limit the scope of this disclosure.

[0051] Figure 1 This is a flowchart illustrating a video processing method in some embodiments of this disclosure. For example... Figure 1 As shown, the video processing method includes the following steps.

[0052] S110, acquire video frames of the video to be processed.

[0053] In this step, the execution device can receive the video to be processed via a communication network, read the video to be processed from its internal or external storage device, download the video to be processed from the Internet using a search engine on the execution device, capture the video to be processed using a shooting device on the execution device, such as a camera or webcam, or obtain video frames of the video to be processed carried in a video processing instruction. It should be understood that the method of obtaining the video to be processed in this embodiment can also be other methods, and the specific details are not limited here.

[0054] S120, use at least one color enhancement model from the color enhancement model pool to perform color adjustment on the video frame to obtain the color-adjusted video output frame.

[0055] The color enhancement model pool contains a collection of models that adjust the color of videos for different style requirements across various video content types.

[0056] In some embodiments, the color enhancement model pool may include at least one color enhancement model, each of which can be used to adjust the color of a video frame for a style requirement of a video content type.

[0057] As an example, video content types can include at least one of the following: TV series, documentaries, movies, animation, sports, news, drama, entertainment, and education; in practical applications, video content types can also be other types, which are not specifically limited here.

[0058] As an example, style requirements refer to the requirements for the style characteristics of the video to be processed. For a certain type of video content, the video content it includes generally has relatively uniform style characteristics, which may include at least one of the following features: image style, image quality, color tone, and brightness.

[0059] Here, "style" refers to the overall audiovisual experience and visual style that a video evokes in the viewer; "image quality" refers to the quality of the picture, including indicators such as sharpness, clarity, lens distortion, chromatic aberration, resolution, color gamut, color purity (or color saturation), and color balance; "hue" refers to the relative brightness and darkness of the image; and "brightness" refers to the luminance of the image. In practical applications, the stylistic characteristics of video content may also include other features, which are not specifically limited here.

[0060] According to the video processing method of this disclosure, the color enhancement model pool includes a set of models that adjust the video for the style requirements of various video types. The video frames of the acquired video to be processed are color-adjusted according to at least one color enhancement model in the model pool. The color grading effect corresponding to the video content type and style requirements of the video frame can be automatically generated. The video processing process can be automated by the execution device calling the model, thereby automating video color creation, automating the ultra-high definition remastering of low-definition sources, shortening the source production cycle, saving labor costs, and improving video processing efficiency.

[0061] Figure 2 A flowchart illustrating color adjustment processing in some embodiments of this disclosure is shown. For example... Figure 2 As shown, in some embodiments, step S120 may specifically include the following steps.

[0062] S21, receive information including the video content type and style requirements of the video frames.

[0063] In some embodiments, upon receiving a video frame of the video to be processed, information containing the video content type and style requirements of the video frame can also be received, for use in selecting a corresponding model from the color enhancement model pool based on the video content type and style requirements. Exemplarily, step S21 can be executed before or after step S110, or step S21 can be executed synchronously with step S110. That is, the executing device can simultaneously receive information containing the video content type and style requirements of the video frame when receiving the video frame of the video to be processed. In practical applications, as long as the video frame and video content type and style requirements of the video to be processed are obtained before selecting a color enhancement model, the execution order of steps S110 and S21 can be flexibly set according to actual needs.

[0064] S22, Select a color enhancement model from the color enhancement model pool that matches the video content type and style requirements.

[0065] In some embodiments, different video content types may have different style requirements, and the style requirements for the same video content type may also be different; therefore, based on the received information on the video content type and style requirements containing video frames, a color enhancement model that matches the video content type and style requirements can be selected from the color enhancement model pool.

[0066] S23, use the selected color enhancement model to process the video frame to obtain the color-adjusted video output frame.

[0067] In some embodiments, based on the received information about the video content type and style requirements containing the video frame, a color enhancement model that matches the video content type and style requirements can be selected from the color enhancement model pool. The video frame can then be processed according to the matching color enhancement model to obtain a color-adjusted video output frame.

[0068] In other embodiments, if at least two color enhancement models are selected from the color enhancement model pool, and one of the at least two color enhancement models is a model that matches the video content type and style requirements, then each of the at least two color enhancement models can be used to process the video frame to obtain the input result of each selected color enhancement model. Then, according to the preset weight value of each color enhancement model and the video frame, the input result of each selected color enhancement model is weighted and fused to generate a video output frame.

[0069] In this embodiment of the disclosure, when adjusting the color of a video frame using at least one color enhancement model in the color enhancement model pool, for the video frame of the video to be processed, a matching color enhancement model can be selected from the color enhancement model pool based on the received information on the video content type and style requirements of the video frame. Then, the selected color enhancement model is used to process the video frame of the video to be processed, generating a color grading effect corresponding to the received video content type and style requirements. This achieves automated adjustment of the video to be processed, meets the color style requirements of the corresponding video content type, and improves the data processing efficiency of video color adjustment.

[0070] Figure 3 A flowchart illustrating color adjustment processing in some embodiments of this disclosure is shown. For example... Figure 3 As shown, in some embodiments, step S120 may specifically include the following steps.

[0071] S31, process the video frame using each color enhancement model in the color enhancement model pool to obtain the color-adjusted video frame output by each color enhancement model.

[0072] In this step, each color enhancement model in the color enhancement model pool processes the video frames of the video to be processed, and obtains the color adjustment result of each color enhancement model on the video frame according to the style requirements of its corresponding video content type.

[0073] S32, based on the content analysis model, process video frames to generate weight values ​​for each color enhancement model and the corresponding video frame; wherein, the content analysis model is configured to determine the weight value of each color enhancement model in the color enhancement model pool based on the video frame.

[0074] In this step, the content analysis model can analyze the video content type of the video frame and assign weight values ​​(also known as weight coefficients) to each color enhancement model in the color enhancement model pool. Thus, the weight values ​​of each color enhancement model output by the content analysis model and the corresponding video frame are used to weight the output results of each model in the model pool.

[0075] S33 uses the weight values ​​corresponding to each color enhancement model and the video frame to perform weighted fusion processing on the color-adjusted video frames output by each color enhancement model, generating video output frames.

[0076] In this step, the weighted fusion process includes: multiplying the weight value corresponding to each color enhancement model and the color-adjusted video frame output by the corresponding color enhancement model by matrix; and then summing the results of each matrix multiplication operation to obtain a matrix-form video output frame.

[0077] In the video processing method of this disclosure embodiment, the weights of each color enhancement model in the color enhancement model pool can be assigned by the weight values ​​of each color enhancement model output by the content analysis model and the corresponding video frame. Finally, the color adjustment result after the fusion of the processing results of each color enhancement model in the model pool is output. The color adjustment result is based on the video content type analysis of the video to be processed, and a video color adjustment result that is more in line with the video content type is obtained.

[0078] The following is combined with Figure 4a and Figure 4b This disclosure describes a video processing system and a corresponding video processing method that are exemplary embodiments of the present disclosure. Figure 4a This invention discloses a schematic diagram of the structure of a video processing system according to some embodiments of the present disclosure; Figure 4b This invention discloses a schematic diagram of the structure of a video processing system according to some embodiments of the present disclosure; Figure 4a and Figure 4b The same labels can represent the same structure.

[0079] like Figure 4a As shown, in some embodiments, the video processing system includes: a video frame input module 41, a color enhancement model pool processing module 42, a content analysis model processing module 43, a weight value output module 44, a fusion processing module 45, and a video frame output module 46.

[0080] In some embodiments, the video frame input module 41 can be used to acquire input video frames.

[0081] In some embodiments, the execution device may obtain the video frame carried in the received video processing instruction.

[0082] The color enhancement model pool processing module 42 is used to perform color adjustment on the video frame using at least one color enhancement model in the color enhancement model pool to obtain the color-adjusted video output frame.

[0083] In some embodiments, the color enhancement model pool processing module may be simply referred to as the color enhancement model pool.

[0084] In some embodiments, when acquiring an input video frame, if information containing the video content type and style requirements of the video frame is also received, a color enhancement model corresponding to the video content type and style requirements can be selected from the color enhancement model pool, and the selected color enhancement model can be used to process the video frame to obtain a color-adjusted video output frame.

[0085] In some embodiments, regardless of whether information regarding the video content type and style requirements containing the video frame is received, the video frame can be processed using each color enhancement model in the color enhancement model pool to obtain a color-adjusted video frame output by each color enhancement model.

[0086] The content analysis model processing module 43 is used to process video frames based on the content analysis model and generate weight values ​​for each color enhancement model and the corresponding video frame.

[0087] In some embodiments, the weight values ​​of each color enhancement model in the color enhancement model pool corresponding to the video frame can be preset; for example, when processing a large number of videos with the same style requirements of the same video content type, the weight values ​​of each color enhancement model in the color enhancement model pool can be preset based on multiple test results or practical experience.

[0088] In other embodiments, video frames are processed based on content analysis models, which can automatically generate weight values ​​for each color enhancement model and the corresponding video frame, reducing the involvement of manual processing, facilitating the automation of color creation in videos, significantly reducing video remastering costs, and improving video processing efficiency.

[0089] The weight value output module 44 is used to output the weight value corresponding to each color enhancement model and video frame.

[0090] The fusion processing module 45 is used to perform weighted fusion processing on the color-adjusted video frames output by each color enhancement model using the weight values ​​corresponding to each color enhancement model and the video frame, so as to obtain the fusion result.

[0091] The video frame output module 46 is used to generate color-adjusted video output frames based on the fusion result.

[0092] In this embodiment, the acquired video frame can be color-adjusted according to at least one color enhancement model in the color enhancement model pool, automatically generating a color grading effect corresponding to the video content type and style requirements of the video frame; and the weights of each color enhancement model in the color enhancement model pool can be assigned by the weight values ​​of each color enhancement model and the video frame output by the content analysis model, and the color adjustment result after the fusion of the processing results of each color enhancement model is output, resulting in a video color adjustment result that better meets the style requirements of the video content type, realizing automated adjustment of video color, thereby reducing video remastering costs and improving video processing efficiency.

[0093] like Figure 4b As shown, this video processing system is... Figure 4a The video processing systems are basically the same, the difference being... Figure 4b The image shows several color enhancement models in the color enhancement model pool.

[0094] In Figure 4, the color enhancement model pool includes at least one set of color enhancement models, each set of color enhancement models corresponds to a video content type; each set of color enhancement models includes at least one color enhancement model, and each color enhancement model corresponds to a style requirement.

[0095] For example, the first set of color enhancement models corresponds to TV drama videos, and includes N1 models, such as TV drama style model 1, TV drama style model 2, ..., TV drama style model N1; each model in this set can correspond to different style requirements of TV drama videos. The second set of color enhancement models corresponds to documentary videos, and includes N2 models, such as documentary style model 1, documentary style model 2, ..., documentary style model N2; each model in this set can correspond to different style requirements of documentary videos; ...; the third set of color enhancement models corresponds to movie videos, and includes N3 models, such as movie style model 1, movie style model 2, ..., movie style model N3; each model in this set can correspond to different style requirements of movie videos. N1, N2, and N3 are all integers greater than or equal to 1.

[0096] It should be understood that the color enhancement model pool may also include other model groups corresponding to at least one style requirement for other video content types; the specific settings can be made according to the actual situation, and this disclosure does not impose specific limitations.

[0097] Continue to refer to Figure 4b , Figure 4b The diagram shows the specific processing units within the fusion module, where the symbols... The symbol ∑ represents matrix multiplication, and the symbol ∑ represents matrix addition. In some embodiments, the above-mentioned fusion module can be specifically used to: perform matrix multiplication on the weight values ​​corresponding to each color enhancement model and video frame output by the content analysis model and the color-adjusted video frame output by the corresponding color enhancement model; and perform matrix addition on the results of each matrix multiplication to obtain a matrix addition result, which is used to generate a video output frame based on the matrix addition result.

[0098] In this embodiment, the video processing method can be automated by an execution device using an artificial intelligence (AI) model. The AI ​​model can be based on a convolutional neural network and can be used in image processing fields such as image detection, image classification, and image content analysis. The color enhancement model and content analysis model in this embodiment are both AI models.

[0099] In some embodiments, the models in the color enhancement model pool have the same model structure. The following is an example... Figures 5-8 It describes in detail the specific structure and working principle of the color enhancement model.

[0100] Figure 5 A schematic diagram illustrating the specific architecture of the color enhancement model according to an embodiment of this disclosure is shown.

[0101] exist Figure 5 In this model, the color enhancement model includes 2N Residual Dense Blocks (RDBs) 51 and N Fully Connected Layers 52, where N is an integer greater than or equal to 1. For example, the color enhancement model includes at least two Residual Dense Blocks (RDBs) 51 and at least one Fully Connected Layer 52.

[0102] exist Figure 5 In the network structure of the color enhancement model shown, the above 2N RDBs can be set as two task branch models: the image reconstruction (IR) branch model and the color enhancement (Color Enhancement) branch model, each of which includes N RDBs.

[0103] Continue to refer to Figure 5 The color enhancement model may also include: a first convolutional layer 53, a filtering unit 54, a second convolutional layer 55, a third convolutional layer 56, and a fourth convolutional layer 57.

[0104] The following is through Figure 6 and Figure 7 Introducing how to... Figure 5 The color enhancement model in the video is used to adjust the color of the video frames. Figure 6 This diagram illustrates a specific flowchart of color adjustment of video frames using a color enhancement model in an embodiment of the present disclosure. Figure 7 The following is a flowchart illustrating the specific process of image reconstruction and color adjustment in an embodiment of this disclosure.

[0105] like Figure 6 As shown, step S23 above, which involves processing the video frame using the selected color enhancement model to obtain the color-adjusted video output frame, may specifically include the following steps.

[0106] S601, extract features from video frames to obtain the first image features.

[0107] In this step, the first image features can be obtained by extracting features from the input color enhancement model video frames through the first convolutional layer 53 in the color enhancement model.

[0108] S602, the video frame is filtered, and the filtered video frame is used to extract features to obtain the second image features.

[0109] In this step, the video frames of the input color enhancement model can be filtered by the filtering unit 54, and the filtered video frames can be feature extracted by the second convolutional layer 55 to obtain the second image features.

[0110] In some embodiments, the filtering unit 54 may be one of a Gaussian filtering unit, a mean filtering unit, and a median filtering unit. The Gaussian filtering unit is used to perform Gaussian filtering on the video frames; the mean filtering unit is used to perform mean filtering on the video frames; and the median filtering unit is used to perform median filtering on the video frames.

[0111] Specifically, Gaussian filtering is a linear smoothing filter suitable for eliminating Gaussian noise. In some scenarios, Gaussian smoothing filters are very effective at suppressing noise that follows a normal distribution. Mean filtering can use the pixel mean of the local image information of the image pixels in a video frame as the processed pixel value. The filtering algorithm is simple and the calculation speed is fast. Median filtering can arrange the gray values ​​of the image pixels in the video frame in descending order and use the median gray value as the gray value of the image pixel. In some scenarios, median filtering is beneficial for better preserving the image boundaries when denoising images.

[0112] In this embodiment of the disclosure, filtering the video frame by the filtering unit not only helps to remove noise from the video frame and achieve a smooth transition of images in the video frame, but also helps to enhance the low-frequency features of the images in the video frame and obtain the low-frequency information of the images in the video frame.

[0113] In some embodiments, if the pixel values ​​of adjacent pixels in a region of an image differ little (less than or equal to a pixel value threshold), then the information in that region is low-frequency information; if the pixel values ​​of adjacent pixels in a region of an image differ much (more than a pixel value threshold), then the information in that region is high-frequency information.

[0114] In some embodiments, low-frequency information may be color information, grayscale information, etc., of the image, while high-frequency information may refer to texture information, edge information, etc., of the image. After filtering by the filtering unit, the image features extracted subsequently are less affected by high-frequency information such as image noise and image texture, thus retaining more low-frequency color features (also known as low-frequency color features).

[0115] It should be understood that the filtering unit can also be other types of filtering units to perform other types of filtering processing on video frames; the specific type of filtering unit can be selected according to actual needs, and the embodiments disclosed herein do not impose specific limitations.

[0116] S603, input the first image features into the image reconstruction branch model and the second image features into the color enhancement branch model, so as to perform image reconstruction and color adjustment based on the first image features and the second image features.

[0117] Image reconstruction (IR) is a technique that processes existing low-resolution (LR) images and transforms them into high-resolution (HR) images.

[0118] In this step, the image reconstruction branch model extracts features from the first image features of the video frame using RDB and performs coarse-precision reconstruction on the output image. The coarse-precision reconstruction uses the video frame of the video to be processed as the initial image. After feature extraction of the initial image through the convolutional layer, the feature information of the initial image is obtained. The feature information of the initial image is extracted using RDB. The image reconstructed based on the feature information of the initial image can be called the coarse-precision reconstructed image.

[0119] In this step, the second image features (i.e., low-frequency color features) obtained after filtering by the filtering unit are analyzed and extracted for subsequent color adjustment based on the analyzed and extracted features.

[0120] S604 performs feature fusion processing on the output results of the image reconstruction branch model and the color enhancement branch model to obtain the fusion result.

[0121] In this step, the outputs of the image reconstruction branch model and the color enhancement branch model are multiplied by their corresponding feature channels, and then the feature channel multiplication results are added together to obtain the fusion result.

[0122] S605 generates color-adjusted video output frames based on the fusion results.

[0123] In this step, the global color adjustment output of the input video frame is obtained based on the fusion result.

[0124] For example, the third convolutional layer 56 is used to extract features from the output of the image reconstruction branch model, and the fourth convolutional layer 57 is used to extract features from the image re-branch model; the feature extraction results of the third convolutional layer 56 and the feature extraction results of the fourth convolutional layer 57 are multiplied by the corresponding feature channels, and the features of the two task branch models are fused to obtain the color adjustment output result of the color enhancement model.

[0125] In this embodiment of the disclosure, when processing video frames using a color enhancement model, the input video frames can be reconstructed and color adjusted by an image reconstruction branch model and a color enhancement branch model. The output results of the image reconstruction branch model and the color enhancement branch model are then subjected to feature fusion processing to obtain the color adjustment output result of the color enhancement model.

[0126] exist Figure 5 In each RDB51, the size of the convolutional kernel (i.e., width and height) is the same. For example, the size of the convolutional kernel in RDB51 is 3×3, and the number of output feature channels is 64. The size of the filter kernel in the filter unit 54 is 21×21. The size of the first convolutional layer 53 is the same as that of the second convolutional layer 55. For example, both of them are 3×64. The size of the third convolutional layer 56 and the fourth convolutional layer 57 are the same. For example, both of them are 64×3.

[0127] It should be noted that the number and size of the convolutional kernels used for feature extraction, the number and size of the convolutional kernels in each RDB51, and the number and size of the filtering kernels in the filtering unit 54 can all be set as needed, and this embodiment does not impose any specific limitations.

[0128] In some embodiments, for Figure 5The color enhancement model shown has two task branches: an image reconstruction branch and a color enhancement branch. Each branch can include N dense residual connection network modules (RDBs). The color enhancement model also includes N fully connected layers, where N is an integer greater than or equal to 1. The image reconstruction and color adjustment processes based on the model architecture of these two task branches are described below through specific embodiments.

[0129] like Figure 7 As shown, in step S603 above, the first image feature is input into the image reconstruction branch model and the second image feature is input into the color enhancement branch model to perform image reconstruction and color adjustment based on the first image feature and the second image feature. Specifically, this step may include the following steps.

[0130] S701, obtain the first RDB in the image reconstruction branch model and the first RDB in the color enhancement branch model, and use them as the current RDB of the first branch model and the current RDB of the second branch model.

[0131] S702, the first image features are processed by the current RDB of the first branch model, and the second image features are processed by the current RDB of the second branch model, to obtain the feature map output by the current RDB of the first branch model and the low-frequency color features output by the current RDB of the second branch model.

[0132] S703: For the low-frequency color features of the current RDB output of the second branch model, a feature vector is obtained after processing by a fully connected layer. The feature vector is then multiplied with the corresponding feature map of the current RDB output of the first branch model to obtain a feature map adjusted for local color features.

[0133] In this step, image features can be extracted using the current RDB (e.g., a 3×3 kernel with 64 output feature channels) in the image reconstruction branch model; low-frequency color features of the image can be analyzed and extracted using the current RDB (e.g., a 3×3 kernel with 64 output feature channels) in the color enhancement branch model; and the extracted low-frequency color features are converted into feature vectors with a width of 1 (e.g., a 1×64 feature vector) through a fully connected layer. Each element of the feature vector output by the fully connected layer is multiplied by the corresponding feature channel of the feature map output by the current RDB in the image reconstruction branch model, thereby achieving fine-tuning of local color features.

[0134] S704 uses the feature map adjusted by local color features as the new first image feature and the low-frequency color features of the current RDB output of the second branch model as the new second image feature.

[0135] S705, obtain the next RDB in the image reconstruction branch model and the next RDB in the color enhancement branch model, and use them as the current RDB of the new first branch model and the current RDB of the new second branch model, until the number of acquisitions equals N. Use the newly obtained feature map adjusted by local color features as the output result of the image reconstruction branch model, and use the newly obtained low-frequency color features as the output result of the color enhancement branch model.

[0136] Through the above steps S701-S705, the low-frequency color features extracted by the last RDB in the color enhancement branch model are obtained. These low-frequency color features are then converted into a feature vector with a width of 1 through the last fully connected layer. Each element of the feature vector output by the last fully connected layer is multiplied by the corresponding feature channel of the feature map output by the last RDB in the image reconstruction branch model to obtain the output result of the image reconstruction branch model in the color enhancement model. The low-frequency color features extracted by the last RDB in the color enhancement branch model are used as the output result of the color enhancement branch model in the color enhancement model. Based on the output result of the image reconstruction branch model and the input result of the color enhancement branch model, feature fusion processing is performed to obtain the color adjustment output result of the color enhancement model, thereby achieving global color style enhancement.

[0137] Figure 8 This diagram illustrates the specific architecture of the dense residual connection network module in an embodiment of this disclosure. Figure 8 As shown, in some embodiments, each RDB includes a densely connected network layer 511 and a local feature fusion layer 512.

[0138] In this embodiment, the local feature fusion layer 512 is used to: use local residual learning to perform feature fusion processing on the input features of RDB and the output features of the densely connected network layer 511 to obtain the output features of RDB.

[0139] In some embodiments, the densely connected network layer 511 includes: M densely connected modules 5111, where M is an integer greater than or equal to 1; wherein the input of the first densely connected module 5111 is the input feature of RDB; the input feature of the i-th densely connected module 5111 is: the input feature of RDB and the output feature of each densely connected module 5111 preceding the i-th densely connected module 5111, where i is an integer greater than 1 and less than or equal to M.

[0140] exist Figure 8 The diagram shows three dense connection modules 5111 included in the dense connection network layer 511 when M=3. In practical applications, the number of dense connection modules 5111 can be set according to actual needs.

[0141] Continue to refer to Figure 8 The input to the local feature fusion layer 512 includes: the input features of the RDB (e.g., which can be denoted as F). d-1 ) and the output characteristics of each densely connected module 5111 (e.g., can be denoted as F) d,1 F d,C The local feature fusion layer 512 may include a concat layer and a 1×1 convolutional layer; wherein, the concat layer fuses the features input to the concat layer in a connected manner, and the 1×1 convolutional layer is used to adaptively fuse a series of features at different levels to generate the output feature of local feature fusion (e.g., it can be denoted as F). d,LF The output features of the local feature fusion are added to the input features of the RDB for the corresponding channels to generate the global feature fusion result (e.g., it can be denoted as F). d The result of the global feature fusion is used as the output feature of the RDB.

[0142] In this embodiment, RDB is a network structure that combines residual networks and densely connected networks. Through the combined effects of dense connections, feature fusion, and residual learning, it achieves information sharing by preserving low-level information and adaptively acquires information from different convolutional layers via feature fusion, resulting in good performance in image processing. Specifically, the intermediate layers of RDB can use the dense connection mode of the dense connection module 5121, and the connection module 5122 of the last layer outputs the processing result of the RDB through residual jumpers.

[0143] In this embodiment, each densely connected module in the RDB can pass the feature information to be saved to the subsequent densely connected modules. This network structure can fully guarantee the transmission and flow of feature information between each layer, reduce the number of parameters while realizing feature reuse, and ensure better processing speed even when the number of network parameters is large.

[0144] In some embodiments, each color enhancement model in the color enhancement model pool has the same model structure; the color enhancement models in the color enhancement model pool are pre-trained color enhancement models obtained by training the initial color enhancement model with different color enhancement data pairs for different video content types; the color enhancement data pairs include: uncolored image blocks and colored image blocks with the same image size selected from the same image position in the uncolored image and the first colored image, and the first colored image is an image obtained in advance after color adjustment of the uncolored image based on the image color style features.

[0145] In this embodiment, the initial color enhancement model has the same model structure as each color enhancement model in the color enhancement model pool; the initial values ​​of the model parameters of the initial color enhancement model can be random values ​​of the model parameters or empirical values ​​of the model parameters; the first color-corrected image can be an image that has been pre-acquired and color-adjusted by a colorist; the training data of the color enhancement network model includes image patches with the same image size selected from the same image position in the uncorrected image and the first color-corrected image. For example, during the training process of each color enhancement model, a 256×256 image patch at a certain position in the image can be randomly selected and input into the network for training.

[0146] It should be understood that the size of the image patches required during training can be selected according to actual needs, and this disclosure does not impose specific limitations.

[0147] In some embodiments, training the initial color enhancement model, i.e., the color enhancement model, may include: processing uncolored image patches selected from the uncolored images using the initial color enhancement model to obtain color adjustment results for the uncolored image patches, and obtaining colored image patches corresponding to the uncolored image patches from the color enhancement data pairs as color adjustment reference results; constructing a loss function for the initial color enhancement model based on the color adjustment results and the color adjustment reference results; adjusting the model parameters of the initial color enhancement model using the loss function of the initial color enhancement model to obtain an updated color enhancement model; using the updated color enhancement model as a new initial color enhancement model, and processing the uncolored image patches selected from the uncolored images using the new initial color enhancement model, until the training process of the initial color enhancement model meets the first training termination condition, thereby obtaining a trained color enhancement model.

[0148] The first training termination condition includes at least one of the following conditions: the training loss calculated based on the loss function of the initial color enhancement model is less than or equal to a set first loss threshold, or the number of training iterations of the initial color enhancement model reaches a set first number threshold.

[0149] In this embodiment of the disclosure, the loss function can be used to estimate the gap between the result of each model training and the training target; for the initial color enhancement model, the result of each model training is the color adjustment result of the unadjusted image patch output by the initial color enhancement model, and the training target is the corresponding color adjustment reference result; the first loss threshold and the first number threshold can be customized according to the actual training requirements.

[0150] In some embodiments, the step of constructing the loss function of the initial color enhancement model based on the color adjustment result and the color adjustment reference result during the training process of the color enhancement model may specifically include: calculating the minimum squared error between the color adjustment result and the color adjustment reference result to obtain the minimum squared error loss; calculating the Gaussian filtered output result of the color adjustment result and the Gaussian filtered output result of the color adjustment reference result to obtain the color loss; and performing a weighted summation of the minimum squared error loss and the color loss to generate the loss function of the initial color enhancement model.

[0151] For example, the loss function of the initial color enhancement model can be expressed as the following expression (1):

[0152]

[0153] In the above expression (1), Loss1 is the loss function of the initial color enhancement model. Color adjustment result, I gt1 As a reference result for color adjustment, Gauss() is a function that calculates the output of a Gaussian filter. The size of this filter kernel is the same as the filter kernel size of the filter unit in the trained color enhancement model. This is the L2 norm of the difference between the color adjustment result and the color adjustment reference result, which is the L2 loss of the image features in each training iteration of the initial color enhancement model. The L2 paradigm is the difference between the Gaussian filtered output of the color adjustment result and the Gaussian filtered output of the color adjustment reference result for each training iteration of the initial color enhancement model, i.e., the color loss for each training iteration; a is the weight of this L2 loss, and b is the weight of this color loss; a and b can be preset according to actual training needs or empirical values.

[0154] In some embodiments, during the training process of the model, a network optimizer such as the Adam optimizer can be used to ensure that the output result (color adjustment result) of each model training is closer to the model training target (color adjustment reference result).

[0155] In this embodiment of the disclosure, different color enhancement data pairs can be used to train the color enhancement model to meet the style requirements of different video content types. This results in a set of models that adjust the color of video frames according to the different style requirements of different video content types, thereby generating different color grading effects. This forms a color enhancement model pool, which is used to automatically adjust the color of video frames of the video to be processed based on the color enhancement model pool, thereby improving the efficiency of color adjustment.

[0156] Figure 9 This diagram illustrates the specific architecture of the content analysis model in an embodiment of this disclosure. For example... Figure 9As shown, in some embodiments, the content analysis model includes: a convolutional network of K sequentially connected stacked convolutional structures 91 and two fully connected layers.

[0157] Each of the first K-1 stacked convolutional structures 91 includes a convolutional layer 911, an activation layer (The Rectified Linear Unit, ReLU) 912, and a pooling layer (Max Pooling) 913. The Kth stacked convolutional structure 92 includes a convolutional layer 911 and an activation layer 912. The two fully connected layers are a fully connected layer 93 and a fully connected layer 94.

[0158] In this embodiment of the disclosure, the activation layer 912 can be used to perform nonlinear mapping on the output results of the corresponding convolutional layer to accelerate the convergence speed of the model; the pooling layer 913 can be used to perform image downsampling, which can reduce the dimension of image features, improve accuracy and avoid overfitting.

[0159] exist Figure 9 In the example, the input image size is 256×256×3 (length, width, and height). In each stacked convolutional structure 91 of the content analysis model, the size of the convolutional layer can be 3×3. Taking K=5 as an example, for five sequentially connected stacked convolutional structures 91, the first stacked convolutional structure 91 can scale the input image size to 128×128×32, the second stacked convolutional structure 91 can scale the input image size to 64×64×64, the third stacked convolutional structure 91 can scale the input image size to 32×32×128, the fourth stacked convolutional structure 91 can scale the input image size to 16×16×256, and the fifth stacked convolutional structure 91 can scale the input image size to 8×8×512.

[0160] Continue to refer to Figure 9 The fully connected layer 93 converts the output of the last stacked convolutional structure 91 into a one-dimensional vector of a specified dimension (e.g., 1×1024). The fully connected layer 94 is used to convert this one-dimensional vector into a one-dimensional vector of dimension L, where L is the total number of models in the color enhancement model pool.

[0161] In some embodiments, Figure 9 The color enhancement model pool in the above embodiments is combined with Figure 4a and Figure 4b The described color enhancement model pool has the same model structure.

[0162] In this embodiment, the layered convolutional network structure in the content analysis model can analyze and extract image features through layered convolutional layers. Feature extraction is performed through convolutional layers, and the final output of the convolutional layer is converted into L weight values ​​through two fully connected layers, where L is an integer greater than or equal to 1. Each weight value is multiplied by the output of the corresponding color enhancement model, and the results of each multiplication operation are summed by the features of the corresponding feature channels to generate the final color enhancement result. Thus, by analyzing the content of video frames, different weights are assigned to different color enhancement models to achieve the purpose of adaptive color adjustment based on the content.

[0163] In some embodiments, the content analysis model is a trained content analysis model obtained by training an initial content analysis model with image data pairs of different styles, based on each color enhancement model in a pre-trained color enhancement model pool; wherein, the image data pairs include: uncolored image blocks and colored image blocks of the same size selected from the same image position in the uncolored image and the second colored image, and the second colored image is an image obtained in advance after color correction of the uncolored image based on different video content types.

[0164] In this embodiment, the model structure of the initial content analysis model is the same as that of the content analysis model; the initial values ​​of the model parameters of the initial content analysis model can be random values ​​of the model parameters or empirical values ​​of the model parameters; the second color-corrected image can be an image that has been pre-acquired and color-adjusted by a colorist; the second color-corrected image is different from the first color-corrected image in the above embodiment.

[0165] The training data for the content analysis model includes image patches of the same size selected from the same location in both the uncolored and colored images. For example, during the training of each color enhancement model, a 256×256 image patch can be randomly selected from a specific location in the image and input into the network for training. This random sampling method ensures that the model does not overfit to a particular style.

[0166] It should be understood that the size of the image patches required during training can be selected according to actual needs, and this disclosure does not impose specific limitations.

[0167] In some embodiments, training the initial content analysis model includes: processing uncolored image patches selected from the uncolored image using each color enhancement model in a pre-trained color enhancement model pool to obtain the color adjustment result of each color enhancement model for the uncolored image patches; processing the uncolored image patches using the initial content analysis model to obtain a predetermined number of weight values, the predetermined number being the same as the number of color enhancement models in the color enhancement model pool; and performing weighted fusion processing on the color adjustment results of each color enhancement model for the uncolored image patches according to the predetermined number of weight values ​​to obtain the color enhancement result of the uncolored image patches. The process involves obtaining the color-corrected image patch corresponding to the uncorrected image patch from the image data pair, using it as a color enhancement reference result; constructing the loss function of the initial content analysis model based on the color enhancement result of the uncorrected image patch and the color enhancement reference result; adjusting the model parameters of the initial content analysis model using the loss function of the initial content analysis model to obtain an updated content analysis model; using the updated content analysis model as the new initial content analysis model, and processing the uncorrected image patch using the new initial content analysis model until the training process of the initial content analysis model meets the second training termination condition, thus obtaining the trained content analysis model.

[0168] The second training termination condition includes at least one of the following conditions: the training loss calculated based on the loss function of the initial content analysis model is less than or equal to the set second loss threshold, or the number of training iterations of the initial content analysis model reaches the set second number threshold.

[0169] In this embodiment, for the initial content analysis model, the result of each model training is the color enhancement result of the uncolored image patch output by the content analysis model, and the training objective is the corresponding color enhancement reference result; and the loss function of the initial content analysis model is constructed based on the results of each model training and the training objective; it should be understood that the second loss threshold and the second number threshold can be customized according to actual training requirements.

[0170] In some embodiments, during the training of the content analysis model, the loss function of the initial content analysis model is a loss function constructed by minimizing the squared error of the color enhancement result and the color enhancement reference result of the uncolored image patch.

[0171] For example, the loss function of the initial content analysis model can be expressed as the following expression (2):

[0172]

[0173] In the above expression (2), Loss2 is the loss function of the initial content analysis model. For color enhancement results, I gt1For color enhancement reference results, The L2 norm of the difference between the color enhancement result and the color enhancement reference result is the L2 loss of the image features for each training iteration of the initial content analysis model.

[0174] In some embodiments, during the training process of the model, a network optimizer such as the Adam optimizer may be used to ensure that the output result (color enhancement result) of each model training is closer to the model training target (color enhancement reference result).

[0175] During the training process of the color enhancement model and content analysis model disclosed herein, the learning rate of the model training can be set. The learning rate determines whether the loss function can converge to a local minimum and when it converges to the minimum. As an example, the learning rate can be the same for the training process of the color enhancement model and the content analysis model, for example, both can be 1e-4. In other examples, the learning rates of the two models can also be different during the model training process. The specific value of the learning rate can be set according to the actual situation.

[0176] In this embodiment of the disclosure, different color enhancement data pairs can be used to train the initial content analysis model according to the style requirements of different video content types. The trained content analysis model can output the weight value of each color enhancement model in the color enhancement model pool and the corresponding video frame. The weight value of each color enhancement model in the color enhancement model pool and the corresponding video frame and the output result of the corresponding color enhancement model are weighted and fused to realize adaptive color style enhancement processing of video frames, thereby improving the efficiency of video color enhancement.

[0177] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0178] In addition, this disclosure also provides video processing apparatus, electronic equipment, and computer-readable storage medium, all of which can be used to implement any of the video processing methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section, and will not be repeated here.

[0179] This disclosure provides a video processing apparatus, which is a corresponding device for implementing the video processing method provided in the above embodiments of this disclosure. The apparatus can be implemented in software and / or hardware and is generally integrated into an electronic device.

[0180] Figure 10This is a schematic diagram of the structure of a video processing apparatus according to an embodiment of the present disclosure.

[0181] Reference Figure 10 The video processing apparatus 1000 provided in this embodiment includes the following modules.

[0182] The acquisition module 1010 is used to acquire the video frames to be processed.

[0183] The processing module 1020 is used to perform color adjustment on the video frame using at least one color enhancement model in the color enhancement model pool to obtain a color-adjusted video output frame.

[0184] The color enhancement model pool contains a collection of models that adjust the color of videos for different style requirements across various video content types.

[0185] According to the video processing apparatus of this disclosure, the video frames of the acquired video to be processed are color-adjusted according to at least one color enhancement model in the model pool. The color adjustment effect corresponding to the video content type and style requirements of the video frame can be automatically generated. The video processing process can be automatically implemented by the execution device calling the model, thereby automating the video color creation, saving labor costs, and improving video processing efficiency.

[0186] In some embodiments, the processing module 1020 may specifically include: a receiving unit for receiving information on the video content type and style requirements of the video frame; a selection unit for selecting a color enhancement model corresponding to the video content type and style requirements from a color enhancement model pool; and a model processing unit for processing the video frame using the selected color enhancement model to obtain a color-adjusted video output frame.

[0187] In some embodiments, the processing module 1020 may specifically include: a color adjustment unit, configured to process video frames using each color enhancement model in the color enhancement model pool to obtain color-adjusted video frames output by each color enhancement model; a content analysis unit, configured to process video frames based on a content analysis model to generate weight values ​​corresponding to each color enhancement model and the video frames; wherein the content analysis model is configured to determine the weight values ​​of each color enhancement model in the color enhancement model pool based on the video frames; and a fusion processing unit, configured to perform weighted fusion processing on the color-adjusted video frames output by each color enhancement model using the weight values ​​corresponding to each color enhancement model and the video frames to generate video output frames.

[0188] In some embodiments, the color enhancement model includes an image reconstruction branch model and a color enhancement branch model; the model processing unit may specifically include: an extraction subunit for extracting features from video frames to obtain first image features; a filtering subunit for filtering video frames and extracting features from the filtered video frames to obtain second image features; a branch model processing subunit for inputting the first image features into the image reconstruction branch model and the second image features into the color enhancement branch model to perform image reconstruction and color adjustment based on the first image features and the second image features; a result fusion subunit for performing feature fusion processing on the output results of the image reconstruction branch model and the output results of the color enhancement branch model to obtain a fusion result; and a result generation subunit for generating a color-adjusted video output frame based on the fusion result.

[0189] In some embodiments, both the image reconstruction branch model and the color enhancement branch model include: N dense residual connection network modules (RDBs); the color enhancement model also includes N fully connected layers; where N is an integer greater than or equal to 1; the branch model processing subunit is specifically used to: obtain the first RDB in the image reconstruction branch model and the first RDB in the color enhancement branch model, as the current RDB of the first branch model and the current RDB of the second branch model; process the first image features through the current RDB of the first branch model, and process the second image features through the current RDB of the second branch model, to obtain the feature map output by the current RDB of the first branch model and the low-frequency color features output by the current RDB of the second branch model; for the low-frequency color features output by the current RDB of the second branch model... The first image model obtains a feature vector after processing through a fully connected layer. This feature vector is then multiplied by the corresponding feature map output by the current RDB of the first branch model to obtain a feature map adjusted for local color features. This adjusted feature map is used as the new first image feature, and the low-frequency color features output by the current RDB of the second branch model are used as the new second image feature. The next RDB in the image reconstruction branch model and the next RDB in the color enhancement branch model are then obtained and used as the new current RDB of the first branch model and the new current RDB of the second branch model, respectively. This process is repeated until the number of times the number of times equals N. The newly obtained feature map adjusted for local color features is then used as the output of the image reconstruction branch model, and the newly obtained low-frequency color features are then used as the output of the color enhancement branch model.

[0190] In some embodiments, the RDB includes a densely connected network layer and a local feature fusion layer. The local feature fusion layer is used to: add the input features of the RDB to the output features of the densely connected network layer using local residual learning to obtain the output features of the RDB. The densely connected layer includes: M densely connected modules and 1 connection module, where M is an integer greater than or equal to 1. The input of the first densely connected module is the input feature of the RDB. The input feature of the i-th densely connected module is: the input feature of the RDB and the output feature of each densely connected module before the i-th densely connected module, where i is an integer greater than 1 and less than or equal to M. The input of the connection module is: the input feature of the RDB and the output feature of each densely connected module.

[0191] In some embodiments, each color enhancement model in the color enhancement model pool has the same model structure; the color enhancement models in the color enhancement model pool are pre-trained color enhancement models obtained by training the initial color enhancement model with different color enhancement data pairs for different video content types; the color enhancement data pairs include: uncolored image blocks and colored image blocks with the same image size selected from the same image position in the uncolored image and the first colored image, and the first colored image is an image obtained in advance after color adjustment of the uncolored image based on the image color style features.

[0192] In some embodiments, the video processing apparatus 1000 further includes a color enhancement model training module for training an initial color enhancement model; the color enhancement training module includes: a first adjustment result acquisition unit, used to process an uncolored image patch selected from an uncolored image using the initial color enhancement model to obtain a color adjustment result for the uncolored image patch; a first reference result acquisition unit, used to acquire a colored image patch corresponding to the uncolored image patch from the color enhancement data pair as a color adjustment reference result; a first loss function construction unit, used to construct a loss function of the initial color enhancement model based on the color adjustment result and the color adjustment reference result; and a first parameter adjustment unit, used to utilize the initial... The loss function of the initial color enhancement model is used to adjust the model parameters of the initial color enhancement model to obtain an updated color enhancement model. The first iterative training unit is used to use the updated color enhancement model as a new initial color enhancement model and to process the uncolored image patches selected from the uncolored images using the new initial color enhancement model until the training process of the initial color enhancement model meets the first training termination condition, thereby obtaining a trained color enhancement model. The first training termination condition includes at least one of the following conditions: the training loss calculated according to the loss function of the initial color enhancement model is less than or equal to a set first loss threshold, and the number of training iterations of the initial color enhancement model reaches a set first number threshold.

[0193] In some embodiments, the color enhancement training module, when constructing the loss function of the initial color enhancement model based on the color adjustment result and the color adjustment reference result, specifically performs the following steps: calculating the minimum squared error between the color adjustment result and the color adjustment reference result to obtain the minimum squared error loss; calculating the Gaussian filtered output of the color adjustment result and the Gaussian filtered output of the color adjustment reference result to obtain the color loss; and performing a weighted summation of the minimum squared error loss and the color loss to generate the loss function of the initial color enhancement model.

[0194] In some embodiments, the content analysis model includes: a convolutional network of K sequentially connected convolutional structures and two fully connected layers. Each of the first K-1 convolutional structures includes a convolutional layer, an activation layer and a pooling layer, and the Kth convolutional structure includes a convolutional layer and an activation layer.

[0195] In some embodiments, the content analysis model is a pre-trained content analysis model obtained by training an initial content analysis model with image data pairs of different styles, based on each color enhancement model in a pre-trained color enhancement model pool; wherein, the image data pairs include: uncolored image blocks and colored image blocks of the same size selected from the same image position in the uncolored image and the second colored image, and the second colored image is an image obtained in advance after color correction of the uncolored image based on different video content types.

[0196] In some embodiments, the video processing apparatus 1000 further includes a content analysis model training module for training an initial content analysis model. The content analysis model training module includes: a second adjustment result acquisition unit, used to process uncolored image blocks selected from the uncolored image using each color enhancement model in a pre-trained color enhancement model pool, to obtain the color adjustment result of each color enhancement model for the uncolored image blocks; a weight value acquisition unit, used to process the uncolored image blocks using the initial content analysis model to obtain a predetermined number of weight values, the predetermined number being the same as the number of color enhancement models in the color enhancement model pool; an enhancement result acquisition unit, used to perform weighted fusion processing on the color adjustment results of each color enhancement model for the uncolored image blocks according to the predetermined number of weight values, to obtain the color enhancement result of the uncolored image blocks; and a second reference result acquisition unit, used to acquire a reference result from the image data pair that corresponds to the uncolored image blocks. The corresponding color-corrected image patch is used as a color enhancement reference result; the second loss function construction unit is used to construct the loss function of the initial content analysis model based on the color enhancement result and the color enhancement reference result of the uncorrected image patch; the second parameter adjustment unit is used to adjust the model parameters of the initial content analysis model using the loss function of the initial content analysis model to obtain an updated content analysis model; the second iterative training unit is used to use the updated content analysis model as a new initial content analysis model and use the new initial content analysis model to process the uncorrected image patch until the training process of the initial content analysis model meets the second training termination condition to obtain a trained content analysis model; wherein, the second training termination condition includes at least one of the following conditions: the training loss calculated according to the loss function of the initial content analysis model is less than or equal to the set second loss threshold, and the number of training iterations of the initial content analysis model reaches the set second number threshold.

[0197] In some embodiments, the loss function of the initial content analysis model is a loss function constructed by minimizing the squared error of the color enhancement result and the color enhancement reference result of the uncolored image patch.

[0198] According to the video processing apparatus of this disclosure, the input video frames can be processed using color enhancement models and content analysis models in a color enhancement model pool, respectively. The weight values ​​corresponding to each color enhancement model in the color enhancement model pool, output by the content analysis model, can be used to assign weights to each color enhancement model in the pool corresponding to the video frame. Finally, a weighted fusion of the processing results of each color enhancement model in the color enhancement model pool is output as a color adjustment result. This color adjustment result is based on the video content type analysis of the video to be processed, resulting in a video color adjustment result that better matches the video content type. Therefore, the automatic adjustment of the video to be processed based on the weight allocation of the content analysis model satisfies the color style requirements of the corresponding video content type, improving the data processing efficiency of video color adjustment.

[0199] It should be clarified that this disclosure is not limited to the specific configurations and processes described in the foregoing embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here, and the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0200] Figure 11 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0201] Reference Figure 11 This disclosure provides an electronic device, which includes: at least one processor 1101; at least one memory 1102; and one or more I / O interfaces 1103 connected between the processor 1101 and the memory 1102; wherein the memory 1102 stores one or more computer programs that can be executed by the at least one processor 1101, and the one or more computer programs are executed by the at least one processor 1101 to enable the at least one processor 1101 to perform the video processing method described above.

[0202] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

[0203] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processing core, implements the video processing method described above. The computer-readable storage medium may be volatile or non-volatile.

[0204] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described video processing method.

[0205] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0206] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0207] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0208] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0209] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0210] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0211] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0212] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0213] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0214] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A video processing method, comprising: Obtain video frames from the video to be processed; The video frame is color-adjusted using at least one color enhancement model from the color enhancement model pool to obtain a color-adjusted video output frame; wherein, the color enhancement model pool contains a set of models for color adjustment of videos with different style requirements for various video content types. The step of adjusting the video frame using at least one color enhancement model from the color enhancement model pool to obtain a color-adjusted video output frame includes: Receive information containing the video content type and style requirements of the video frame; select a color enhancement model from the color enhancement model pool that corresponds to the video content type and style requirements; process the video frame using the selected color enhancement model to obtain the color-adjusted video output frame; The color enhancement model includes an image reconstruction branch model and a color enhancement branch model; both the image reconstruction branch model and the color enhancement branch model include N dense residual connection network modules (RDBs); the color enhancement model also includes N fully connected layers; where N is an integer greater than or equal to 1; the step of processing the video frame using the selected color enhancement model to obtain the color-adjusted video output frame includes: Feature extraction is performed on the video frame to obtain a first image feature; the video frame is filtered, and feature extraction is performed on the filtered video frame to obtain a second image feature; the first image feature is input into the image reconstruction branch model, and the second image feature is input into the color enhancement branch model, so as to perform image reconstruction and color adjustment based on the first image feature and the second image feature; feature fusion processing is performed on the output results of the image reconstruction branch model and the output results of the color enhancement branch model to obtain a fusion result; based on the fusion result, a color-adjusted video output frame is generated. The step of inputting the first image feature into the image reconstruction branch model and the second image feature into the color enhancement branch model to perform image reconstruction and color adjustment based on the first image feature and the second image feature includes: The first RDB in the image reconstruction branch model and the first RDB in the color enhancement branch model are obtained as the current RDB of the first branch model and the current RDB of the second branch model, respectively. The first image features are processed using the current RDB of the first branch model, and the second image features are processed using the current RDB of the second branch model, resulting in a feature map output by the current RDB of the first branch model and low-frequency color features output by the current RDB of the second branch model. The low-frequency color features output by the current RDB of the second branch model are processed through a fully connected layer to obtain a feature vector. This feature vector is then compared with the feature map output by the current RDB of the first branch model. The corresponding images are multiplied to obtain a feature map adjusted for local color features. The feature map adjusted for local color features is used as a new first image feature, and the low-frequency color features output by the current RDB of the second branch model are used as a new second image feature. The next RDB in the image reconstruction branch model and the next RDB in the color enhancement branch model are obtained as the current RDB of the new first branch model and the current RDB of the new second branch model, respectively, until the number of times is equal to N. The newly obtained feature map adjusted for local color features is used as the output result of the image reconstruction branch model, and the newly obtained low-frequency color features are used as the output result of the color enhancement branch model.

2. The method according to claim 1, wherein, The RDB includes a densely connected network layer and a local feature fusion layer; The local feature fusion layer is used to: add the input features of the RDB to the output features of the densely connected network layer using local residual learning to obtain the output features of the RDB; The densely connected network layer comprises: M densely connected modules and 1 connection module, where M is an integer greater than or equal to 1; wherein, The input to the first densely connected module is the input feature of the RDB; The input features of the i-th dense connection module are: the input features of the RDB and the output features of each dense connection module before the i-th dense connection module, where i is an integer greater than 1 and less than or equal to M; The inputs to the connection module are: the input characteristics of the RDB and the output characteristics of each dense connection module.

3. The method according to claim 1 or 2, wherein, Each of the color enhancement models in the color enhancement model pool has the same model structure; The color enhancement models in the color enhancement model pool are pre-trained color enhancement models obtained by training the initial color enhancement model with different color enhancement data pairs for different video content types and color styles. The color enhancement data pair includes: uncolored image blocks and colored image blocks of the same size selected from the same image position in the uncolored image and the first colored image, wherein the first colored image is an image obtained in advance after color correction of the uncolored image based on image color style features.

4. The method according to claim 3, wherein, The training of the initial color enhancement model includes: The initial color enhancement model is used to process the uncolored image block selected from the uncolored image to obtain the color adjustment result of the uncolored image block, and the color-adjusted image block corresponding to the uncolored image block is obtained from the color enhancement data pair as the color adjustment reference result; Based on the color adjustment results and the color adjustment reference results, a loss function for the initial color enhancement model is constructed. The model parameters of the initial color enhancement model are adjusted using the loss function of the initial color enhancement model to obtain an updated color enhancement model; The updated color enhancement model is used as the new initial color enhancement model, and the new initial color enhancement model is used to process the uncolored image patches selected from the uncolored image until the training process of the initial color enhancement model meets the first training termination condition, thus obtaining the trained color enhancement model; wherein, The first training termination condition includes at least one of the following conditions: the training loss calculated based on the loss function of the initial color enhancement model is less than or equal to a set first loss threshold, or the number of training iterations of the initial color enhancement model reaches a set first number threshold.

5. The method according to claim 4, wherein, The loss function for constructing the initial color enhancement model based on the color adjustment result and the color adjustment reference result includes: Calculate the minimum squared error between the color adjustment result and the color adjustment reference result to obtain the minimum squared error loss; The color loss is obtained by calculating the Gaussian filtered output of the color adjustment result and the Gaussian filtered output of the color adjustment reference result. The loss function of the initial color enhancement model is generated by weighted summation of the minimum squared error loss and the color loss.

6. A video processing apparatus, comprising: The acquisition module is used to acquire the video frames to be processed. The processing module is used to perform color adjustment on the video frame using at least one color enhancement model in the color enhancement model pool to obtain a color-adjusted video output frame. The color enhancement model pool contains a set of models for color adjustment of videos with different style requirements for various video content types; the color enhancement model adopts the color enhancement model of any one of claims 1-5.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein... When the computer program is executed by the processor, it implements the video processing method according to any one of claims 1 to 5.

8. A non-transient computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the computer program implements the video processing method according to any one of claims 1 to 5.

Images