Video processing methods, devices and related equipment

Abstract

This disclosure provides a video processing method, apparatus, and related equipment, relating to the field of video processing technology. The method involves: acquiring a video frame to be processed and its processing instruction information, which indicates time steps with correlations; during the processing of each preset time step, determining, based on the processing instruction information, whether there exists a first correlated time step preceding the current time step; if so, obtaining the image semantic features of the current time step based on the processing instruction information and cache information, and using these image semantic features to perform repair processing on the video frame to be processed, obtaining the repair result of the current time step; and obtaining the target video frame based on the repair result obtained from the last time step. This method effectively utilizes the correlations between time steps, reduces unnecessary computation, lowers computational resource consumption, and shortens inference time, making it suitable for real-time or near-real-time video processing scenarios.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a video processing method, video processing apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In the field of video processing, diffusion models, with their progressive generation characteristics, excel in tasks such as video restoration, denoising, super-resolution, and image quality enhancement, improving the visual experience. However, the inference process of diffusion models is complex, gradually restoring clear and high-quality content through multi-step iterative inference. This results in high computational resource consumption and long inference time, making diffusion models difficult to apply in real-time or near-real-time video processing scenarios.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] This disclosure provides a video processing method, video processing apparatus, electronic device, computer-readable storage medium, and computer program product to overcome or at least partially solve the above-mentioned problems.

[0005] According to one aspect of the present disclosure, a video processing method is provided, the method comprising: acquiring a video frame to be processed and processing instruction information of the video frame to be processed; the processing instruction information being used to indicate time steps with a correlation; during the processing of each time step of a preset time step, determining, according to the processing instruction information, whether there exists a first related time step related to the current time step; the first related time step being a time step preceding the current time step; if the first related time step exists, obtaining image semantic features of the current time step according to the processing instruction information and cache information; performing repair processing on the video frame to be processed according to the image semantic features of the current time step to obtain a repair result of the current time step; and obtaining a target video frame of the video frame to be processed according to the repair result obtained from the processing of the last time step.

[0006] In some embodiments of this disclosure, the method further includes: if the first relevant time step does not exist, then by using the state control branch of the video processing model, performing feature extraction on the video frame to be processed to obtain the image semantic features of the current time step.

[0007] In some embodiments of this disclosure, the state control branch includes one or more network layers; wherein, obtaining the image semantic features of the current time step according to the processing instruction information and cache information includes: determining the first relevant network layer corresponding to the current time step and the first relevant time step according to the processing instruction information; the processing instruction information is used to indicate the relevant network layer corresponding to the time step with the correlation; for each network layer, if the network layer is the first relevant network layer, then extracting the image semantic features of the network layer during the processing of the first relevant time step from the cache information, and using the extracted result as the image semantic features of the network layer during the processing of the current time step; if the network layer is not the first relevant network layer, performing feature extraction on the video frame to be processed through the network layer to obtain the image semantic features of the network layer during the processing of the current time step.

[0008] In some embodiments of this disclosure, the method further includes: determining, according to the processing instruction information, whether there exists a second related time step associated with the current time step; the second related time step is a time step after the current time step; if the second related time step exists, determining, according to the processing instruction information, a second related network layer corresponding to the current time step and the second related time step; and placing the image semantic features of the second related network layer during the processing of the current time step into the cache information.

[0009] In some embodiments of this disclosure, the step of repairing the video frame to be processed based on the image semantic features of the current time step to obtain the repair result of the current time step includes: fusing the image semantic features of the current time step and the repair result of the previous time step through the main model branch of the video processing model to obtain the repair result of the current time step; wherein, if the current time step is the first time step, the repair result of the previous time step is noise data.

[0010] In some embodiments of this disclosure, the processing instruction information is pre-generated according to the following process: acquiring sample video frames; the scene information of the sample video frames is the same as the scene information of the video frame to be processed; according to the preset time step, performing feature extraction on the sample video frames through each network layer in the state control branch included in the video processing model to obtain sample features of each network layer in each time step processing; for each network layer, calculating the similarity between the sample features of the network layer in different time steps processing based on the obtained sample features, determining the time steps with correlation and the relevant network layers corresponding to the time steps with correlation based on the calculated similarity; generating the processing instruction information based on the time steps with correlation and the relevant network layers corresponding to the time steps with correlation.

[0011] In some embodiments of this disclosure, the step of calculating the similarity between sample features of the network layer at different time steps based on the obtained sample features, and determining the time steps with a correlation and the corresponding network layers based on the calculated similarity, includes: for each network layer, selecting sample features of the network layer at any two time steps from the obtained sample features, and calculating the similarity between the two selected sample features; if the calculated similarity is greater than a similarity threshold, then determining the time steps corresponding to the two selected sample features as the time steps with a correlation, and designating the network layer as the corresponding network layer of the time steps with a correlation.

[0012] According to another aspect of the present disclosure, a video processing apparatus is provided, the apparatus comprising: an acquisition module configured to acquire a video frame to be processed and processing instruction information of the video frame to be processed; the processing instruction information being used to indicate time steps with a correlation; an iteration module configured to, during the processing of each time step of a preset time step, determine, according to the processing instruction information, whether there exists a first related time step related to the current time step; the first related time step being a time step preceding the current time step; if the first related time step exists, obtaining image semantic features of the current time step according to the processing instruction information and cache information; performing repair processing on the video frame to be processed according to the image semantic features of the current time step to obtain a repair result of the current time step; and a processing module configured to obtain a target video frame of the video frame to be processed based on the repair result obtained from the processing of the last time step.

[0013] According to another aspect of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the video processing method described above.

[0014] According to another aspect of the present disclosure, a computer-readable storage medium is provided that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the video processing method described above.

[0015] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the video processing method described above.

[0016] The video processing method provided in this disclosure introduces processing instruction information, which indicates time steps with correlations. During the processing of each time step of a preset time step, if it is determined according to the processing instruction information that there is a first related time step that is related to the current time step and precedes the current time step, the image semantic features of the current time step are directly obtained according to the processing instruction information and cache information. Then, the image semantic features are used to perform repair processing on the video frame to be processed at the current time step. This method can effectively utilize the correlation between time steps during iterative repair processing, reduce unnecessary computation, thereby reducing the consumption of computing resources and shortening the inference time. Therefore, this method can generate high-quality videos faster, improve the user's visual experience, and is suitable for real-time or near-real-time video processing scenarios.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] Figure 1 A schematic diagram of an exemplary system architecture to which the video processing method of the present disclosure embodiments can be applied is shown;

[0020] Figure 2 A flowchart of a video processing method according to an embodiment of the present disclosure is shown;

[0021] Figure 3 An architecture diagram of a video processing model according to an embodiment of this disclosure is shown;

[0022] Figure 4 A diagram illustrating the process of generating instruction information in the video processing method of this disclosure embodiment is shown.

[0023] Figure 5 This diagram illustrates the process of obtaining the image semantic features at the current time step based on processing instruction information and cache information in the video processing method of this embodiment.

[0024] Figure 6 This illustration shows a schematic diagram of video processing based on cache information according to an embodiment of the present disclosure;

[0025] Figure 7 A block diagram of a video processing apparatus according to an embodiment of the present disclosure is shown;

[0026] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0027] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0028] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0029] The collection, updating, analysis, processing, use, transmission, and storage of user personal information disclosed herein comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0030] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different network and / or processor devices and / or microcontroller devices.

[0031] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0032] In this specification, the terms “a,” “one,” “the,” “the,” and “at least one” are used to indicate the presence of at least one element / component / etc.; the term “multiple” refers to two or more; the terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion meaning and that other elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms “first,” “second,” and “third,” etc., are used only as markings and are not a limitation on the number of objects.

[0033] In the field of video processing, diffusion models have become an important technological direction. However, the complex multi-step iterative inference process of diffusion models leads to high computational resource consumption and long inference time, limiting their application in real-time or near-real-time video processing scenarios. Related technologies employ acceleration techniques such as quantization and pruning to reduce the computational cost of the model; however, for video processing applications requiring multiple iterations to generate high-quality images, these acceleration techniques have limited effectiveness in speeding up diffusion models and are insufficient to meet the demands of real-time or near-real-time video processing.

[0034] To address the aforementioned issues, this disclosure provides a video processing method that introduces processing instruction information. This information identifies time steps with relevant relationships. By utilizing this processing instruction information to guide the iterative process, unnecessary computation is reduced, thereby lowering the consumption of computing resources and shortening inference time. This method can generate high-quality videos faster, improving the user's visual experience, and is suitable for real-time or near-real-time video processing scenarios.

[0035] Figure 1 A schematic diagram of an exemplary system architecture to which the video processing method of the present disclosure can be applied is shown.

[0036] like Figure 1 As shown, the system architecture may include server 101, network 102, and terminal device 103. Network 102 serves as the medium for providing a communication link between terminal device 103 and server 101. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0037] In an exemplary embodiment, the terminal device 103 that transmits data with the server 101 may include, but is not limited to, mobile devices such as smartphones, tablets, and laptops, as well as terminal devices with specific functions or forms such as smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, and smart wearable devices. Alternatively, the terminal device 103 may also be a personal computer, such as a laptop computer or a desktop computer. Optionally, the operating system running on the electronic device may include, but is not limited to, Android, iOS, Linux, and Windows.

[0038] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some practical applications, server 101 can also be a server for a network platform, such as a trading platform, live streaming platform, social platform, or audio platform, etc., which is not limited in this embodiment. The server can be a single server or a cluster of multiple servers; the specific architecture of the server is not limited in this disclosure.

[0039] In some embodiments of this disclosure, the process of the server 101 for video processing may include: acquiring a video frame to be processed and processing instruction information of the video frame to be processed; the processing instruction information is used to indicate time steps with correlation; during the processing of each time step of a preset time step, determining whether there is a first related time step related to the current time step according to the processing instruction information; the first related time step is a time step before the current time step; if there is a first related time step, obtaining the image semantic features of the current time step according to the processing instruction information and cache information; performing repair processing on the video frame to be processed according to the image semantic features of the current time step to obtain the repair result of the current time step; and obtaining the target video frame of the video frame to be processed according to the repair result obtained from the last time step.

[0040] In addition, it should be noted that, Figure 1 The example shown is merely one application environment of the image enhancement model training method or image enhancement method provided in this disclosure. Figure 1 The number of terminal devices 103, networks 102 and servers 101 in the diagram is merely illustrative. Depending on actual needs, there can be any number of terminal devices, networks and servers.

[0041] Figure 2 A flowchart of a video processing method according to an embodiment of this disclosure is shown. Figure 2 The execution subject of the method provided in the embodiments can be any electronic device, such as... Figure 1 The terminal device 101 in the embodiment, for example Figure 1 Terminal device 103, for example Figure 1 In this embodiment, server 101 and terminal device 103 jointly implement the video processing method, but this disclosure is not limited thereto. (See also...) Figure 2 The video processing method provided in this disclosure includes the following steps.

[0042] Step S210: Obtain the video frame to be processed and the processing instruction information of the video frame to be processed; the processing instruction information is used to indicate the time steps where there is a correlation.

[0043] In this embodiment of the disclosure, the video frame to be processed refers to a video frame that needs to be processed or repaired. This frame may be used to improve quality due to reasons such as low resolution, noise, blurriness, or damage. During the video processing, the video to be processed is acquired, which includes multiple video frames to be processed. These video frames to be processed can be acquired frame by frame, and each video frame to be processed can be processed subsequently.

[0044] The processing instruction information is associated with the video frame to be processed, providing specific guidance on how to process or repair it. In this embodiment, the processing instruction information is used to indicate time steps with correlations, or in other words, it indicates which time steps correspond to video frame features that are related or similar.

[0045] Step S220: During the processing of each time step of the preset time step, determine whether there is a first related time step that is related to the current time step according to the processing instruction information; the first related time step is the time step before the current time step.

[0046] In this embodiment, the preset time step is used to indicate the number of times the video frame to be repaired is processed. The preset time step can be set to T, where T is a positive integer. The video processing model performs T repairs on the video frame to be processed. In this embodiment, the video frame to be processed is iteratively repaired according to the preset time step. During each time step processing, that is, during each repair processing of the video frame to be processed, the processing indication information is used to determine whether there is a first related time step associated with the current time step. That is, it is determined whether there are any time steps in the time steps preceding the current time step that are related to the current time step.

[0047] Step S230: If a first relevant time step exists, the image semantic features of the current time step are obtained based on the processing instruction information and cache information.

[0048] In this embodiment of the disclosure, the cache information is used to store the image semantic features of the target time step during processing. The target time step is an earlier time step among the time steps that are related. For example, if time step t and time step t+1 are related time steps, and time step t is the previous time step of time step t+1, then time step t is the target time step.

[0049] In this embodiment of the present disclosure, if it is determined that there is a first related time step in the current time step during the processing of the current time step, it means that the image semantic features of the current time step and the image semantic features of the first related time step are highly similar, and the first related time step is before the current time step. Then, the image semantic features of the current time step processing can be obtained according to the processing instruction information and cache information.

[0050] Taking the current time step t+1 as an example, since time step t and time step t+1 are related, this means that the semantic features of the image at time step t are highly similar to those at time step t+1, and time step t is earlier than time step t+1. During the processing of time step t, the semantic features of the image at time step t are stored in the cache. Thus, during the processing of time step t+1, the semantic features of the image at time step t+1 are obtained based on the processing instructions and the cached information.

[0051] Step S240: Repair the video frame to be processed based on the image semantic features of the current time step to obtain the repair result of the current time step.

[0052] In this embodiment of the disclosure, after obtaining the image semantic features of the current time step, the image semantic features of the current time step are used to perform the repair processing of the current time step to obtain the repair result of the current time step.

[0053] Step S250: Based on the repair result obtained from the last time step, obtain the target video frame of the video frame to be processed.

[0054] In this embodiment of the disclosure, for the video frame to be processed, iterative repair processing is performed according to a preset time step T. In each time step, if it is determined that there is a first related time step according to the processing instruction information, the image semantic features of the current time step are directly obtained according to the cache information and the processing instruction information. Then, the repair processing of the current time step is performed using the image semantic features of the current time step to obtain the repair result of the current time step. Finally, at the end of the last time step (i.e., the Tth time step), a repair result after multiple iterations is obtained, and a high-quality target video frame is obtained using the repair result.

[0055] The video processing method provided in this disclosure introduces processing instruction information, which indicates time steps with correlations. During the processing of each time step of a preset time step, if it is determined according to the processing instruction information that there is a first related time step that is related to the current time step and precedes the current time step, the image semantic features of the current time step are directly obtained according to the processing instruction information and cache information. Then, the image semantic features are used to perform repair processing on the video frame to be processed at the current time step. This method can effectively utilize the correlation between time steps during iterative repair processing, reduce unnecessary computation, thereby reducing the consumption of computing resources and shortening the inference time. Therefore, this method can generate high-quality videos faster, improve the user's visual experience, and is suitable for real-time or near-real-time video processing scenarios.

[0056] In some embodiments of this disclosure, the video processing method further includes: if there is no first relevant time step, then through the state control branch of the video processing model, performing feature extraction on the video frame to be processed to obtain the image semantic features of the current time step.

[0057] In this embodiment, the video processing model is a pre-trained diffusion model. The diffusion model is a probability-based generative model capable of generating high-quality images or videos by progressively reducing noise. The video processing model utilizes the diffusion process to learn how to generate clear image or video frames from noise, progressively reducing noise and recovering high-quality image content during the restoration process.

[0058] To facilitate understanding, the video processing model of the present disclosure embodiment will be described first. Figure 3 An architecture diagram of a video processing model according to an embodiment of this disclosure is shown. Figure 3 As shown, the architecture includes an encoder 310, a video processing model 320, and a decoder 330.

[0059] The encoder 310 is a Variational Autoencoder (VAE) encoder, and the decoder 330 is a VAE decoder. The encoder 310 is used to encode the video frame to be processed to obtain the encoded features of the video frame to be processed, and the decoder 330 is used to decode the repair result of the video processing model 320 to finally generate a high-quality video frame.

[0060] The video processing model 320 is the core of the entire architecture, responsible for executing iterative repair tasks on the video frames to be processed, such as... Figure 3 As shown, the video processing model 320 includes a state control branch 321 and a backbone model branch 322. The video processing model 320 performs an iterative repair process over T time steps; that is, the state control branch 321 and the backbone model branch 322 collaboratively perform iterative repair over T time steps.

[0061] In this embodiment of the present disclosure, during the processing of the current time step, if it is determined that there is a first related time step in the current time step according to the processing instruction information, the image and speech features of the current time step are obtained according to the processing instruction information and the cache information; if it is determined that there is no first related time step in the current time step according to the processing instruction information, the encoded features of the video frame to be processed are extracted through the state control branch 321 to generate features containing rich image semantic information, that is, the image semantic features of the current time step are obtained.

[0062] In the video processing method provided in this embodiment, during the processing of each time step, if it is determined that there is no first relevant time step at the current time step, the image semantic features of the current time step are generated through the state control branch of the video processing model. The generated image semantic features are then used for repair processing, which helps to more accurately restore the damaged or missing image parts during the repair process.

[0063] In some embodiments of this disclosure, the video frame to be processed is repaired based on the image semantic features of the current time step to obtain the repair result of the current time step, including: fusing the image semantic features of the current time step and the repair result of the previous time step through the backbone model branch of the video processing model to obtain the repair result of the current time step; wherein, if the current time step is the first time step, the repair result of the previous time step is noise data.

[0064] In this embodiment of the disclosure, after obtaining the image semantic features at the current time step, these features are fused into the backbone model branch 322 as supervision information to significantly improve the fidelity and authenticity of the generated content. The backbone model branch 322 receives noise (at the first time step of the repair process) or the repair result of the previous time step as input, and combines the supervision information provided by the state control branch 321 (i.e., the image semantic features at the current time step) to perform repair processing and obtain the repair result at the current time step.

[0065] from Figure 3 As can be seen, state control branch 321 can combine category information C for feature extraction. Here, category information C refers to the category label of the video frame or video sequence, such as "animal," "landscape," or "movement scene." During feature extraction, combining category information C helps the model more accurately capture category-related features, thereby improving the accuracy of subsequent processing.

[0066] In the video processing method provided in this disclosure, the repair result of the previous time step is used as a reference during the processing of the current time step, which can enhance the stability of the repair of the current time step, help to more accurately restore the image content, and reduce the amount of computation and resource consumption required for the processing of the current time step to a certain extent.

[0067] Figure 4 A diagram illustrating the process of generating instruction information in a video processing method according to an embodiment of this disclosure is shown. (Refer to...) Figure 4 In this embodiment of the disclosure, processing instruction information can be generated in advance according to the following steps.

[0068] Step S410: Obtain sample video frames; the scene information of the sample video frames is the same as the scene information of the video frames to be processed.

[0069] In this embodiment of the disclosure, the sample video frame refers to the video frame used to generate processing instruction information. The sample video frame and the video frame to be processed have the same scene information, meaning the application scenario of the sample video frame and the video frame to be processed are the same. For example, if the application scenario of the video frame to be processed is a live streaming scenario, then the sample video frame is a video frame from the live streaming scenario. This ensures that video processing requirements (such as real-time or near real-time performance) are met under the same application scenario.

[0070] Step S420: According to the preset time step, feature extraction is performed on the sample video frame through each network layer in the state control branch included in the video processing model to obtain the sample features of each network layer in each time step processing.

[0071] In this embodiment of the disclosure, the state control branch includes one or more network layers, to Figure 3Taking the state control branch 321 as an example, it includes three network layers.

[0072] In this embodiment, the preset time step is used to indicate the number of times the video frame to be repaired is processed. The preset time step can be set to T, where T is a positive integer. To improve the accuracy of the processing indication information, feature extraction is performed on the sample video frame through each network layer of the state control branch according to the preset time step T, so as to obtain the sample features of each network layer in each time step processing.

[0073] Step S430: For each network layer, based on the obtained sample features, calculate the similarity between sample features processed at different time steps of the network layer, and determine the time steps with correlation and the corresponding network layers based on the calculated similarity.

[0074] In this embodiment of the disclosure, for each network layer in the state control branch, similarity is calculated using the sample features obtained at different time steps. This calculated similarity is used to evaluate the degree of similarity of sample features of the same network layer at different time steps. Cosine similarity, Euclidean distance, or other similarity calculation methods can be used, and this embodiment of the disclosure does not limit the specific methods used.

[0075] Based on the calculated similarity values, it is possible to identify which time steps have significantly similar sample features, and then determine the time steps with correlations and the corresponding network layers.

[0076] In some embodiments of this disclosure, for each network layer, based on the obtained sample features, the similarity between sample features processed at different time steps of the network layer is calculated. Based on the calculated similarity, the time steps with a correlation and the corresponding network layers of the time steps with a correlation are determined. This includes: for each network layer, selecting sample features of the network layer processed at any two time steps from the obtained sample features, and calculating the similarity between the two selected sample features; if the calculated similarity is greater than a similarity threshold, then the time steps corresponding to the two selected sample features are determined to be time steps with a correlation, and the network layer is designated as the corresponding network layer of the time step with a correlation.

[0077] In this embodiment of the disclosure, for each network layer in the state control branch, sample features from any two time steps are selected from the sample features obtained by the network layer at different time steps for similarity calculation. The specific calculation formula is as follows:

[0078]

[0079] In formula (1), This represents the sample features of the i-th layer in the state control branch at time step t. This represents the sample features of the i-th layer in the state control branch at time step t+1. This represents the calculated cosine similarity.

[0080] In this embodiment of the disclosure, a similarity threshold θ can be preset. This indicates the sample features of the i-th layer in the state control branch during time step t. The state control branch processes the sample features of the i-th layer at time step t+1. If the similarity between them is high, then time step t and time step t+1 are determined to be time steps with a correlation, and network layer i is taken as the corresponding network layer for time step t and time step t+1.

[0081] In this embodiment of the disclosure, a balance between processing efficiency and processing effect can be achieved by controlling the size of the similarity threshold θ. For scenarios with high requirements for video processing efficiency, the similarity threshold θ can be set smaller, such as setting the similarity threshold θ to 0.9. For scenarios with high requirements for video processing image quality, the similarity threshold can be set larger, such as setting the similarity threshold θ to 0.95.

[0082] For each network layer in the state control branch, after performing similarity analysis on the sample features of any two time steps among the sample features obtained by the network layer at different time steps, all time steps corresponding to the network layer that have correlations are obtained.

[0083] Step S440: Generate processing instruction information based on the time steps where there is a correlation and the corresponding network layers of the time steps where there is a correlation.

[0084] In this embodiment of the disclosure, after comprehensively analyzing the sample features of all network layers at different time steps, all time steps with correlations within a preset time step can be obtained, as well as the relevant network layers corresponding to the time steps with correlations, and finally, processing instruction information is generated. Therefore, the processing instruction information is used to indicate the time steps with correlations and the relevant network layers corresponding to the time steps with correlations.

[0085] In the video processing method of this disclosure, a sample video frame with the same scene information as the video frame to be processed is obtained, and processing instruction information is generated using the sample video frame, ensuring that the processing instruction information matches the scene information of the video frame to be processed; and, by using each network layer in the state control branch of the video processing model to extract features from the sample video frame, subtle changes in the video frame at different time steps and different network layers can be captured, and then by calculating the similarity between sample features processed at different time steps, the time steps with correlation and the corresponding related network layers can be accurately identified, providing strong support for generating accurate processing instruction information.

[0086] Furthermore, the similarity between sample features processed by each network layer at different time steps is calculated, and a similarity threshold is set to determine the time steps and corresponding network layers with a correlation. This method can accurately identify those time steps and network layers with significant correlation, thereby ensuring the accuracy and reliability of the processing indication information.

[0087] In this embodiment of the present disclosure, during each time step processing, the state control branch of the video processing model extracts features from the video frame to be processed according to the processing instruction information to obtain the image semantic features of the current time step. Figure 5 This diagram illustrates the process of obtaining the image semantic features at the current time step based on processing instruction information and cache information in a video processing method according to an embodiment of this disclosure. (Refer to...) Figure 5 Obtaining the image semantic features at the current time step based on processing instruction information and cache information may include the following steps.

[0088] Step S510: Based on the processing instruction information, determine the first related network layer corresponding to the current time step and the first related time step; the processing instruction information is used to indicate the related network layer corresponding to the time step with a correlation relationship.

[0089] Step S520: For each network layer, if the network layer is the first relevant network layer, extract the image semantic features of the network layer during the first relevant time step processing from the cached information, and use the extracted results as the image semantic features of the network layer during the current time step processing.

[0090] Step S530: For each network layer, if the network layer is not the first relevant network layer, feature extraction is performed on the video frame to be processed through the network layer to obtain the image semantic features of the network layer during the current time step processing.

[0091] In this embodiment of the disclosure, the processing indication information is used not only to indicate the time steps where a correlation exists, but also to indicate the relevant network layer corresponding to the time steps where a correlation exists.

[0092] In this embodiment of the disclosure, during the processing of the current time step, if there is a first related time step that is related to the current time step, it is determined that the image semantic features of the current time step are similar to the image semantic features of the first related time step. However, this does not mean that the image semantic features of each network layer are similar. Therefore, the first related network layer is further determined according to the control instruction information, that is, it is determined which network layer the image semantic features of the current time step and the first related time step are similar to.

[0093] Each network layer is analyzed. If the network layer is the first relevant network layer, it means that the image semantic features of this network layer in the current time step are highly similar to the image semantic features in the first relevant time step, and the first relevant time step is before the current time step. This indicates that the cached information stores the image semantic features of this network layer in the first relevant time step, and these features can be directly obtained and used as the image semantic features of this network layer in the current time step. If the network layer is not the first relevant network layer, feature extraction is performed on the video frame to be processed through this network layer to obtain the image semantic features of this network layer in the current time step.

[0094] Figure 6 A schematic diagram illustrating video processing based on cache information according to an embodiment of the present disclosure is shown. Figure 6 Taking the current time step t+1 as an example, time step t and time step t+1 are related time steps; that is, time t is the first related time step of time step t+1, and the corresponding first related network layer is the three network layers of the state control branch. This means that the image semantic features of each network layer during the processing of time step t have a high similarity to the image semantic features during the processing of time step t+1. During the processing of time step t, the image semantic features of each network layer in the state control branch are stored in the cache information. Thus, during the processing of time step t+1, the image semantic features of each network layer during the processing of time step t are retrieved from the cache information and used as the image semantic features of the corresponding network layer during the processing of time step t+1, without the need for feature extraction through the state control branch.

[0095] In the video processing method provided in this embodiment, if there is a first related time step during the processing of the current time step, further, the first related network layer corresponding to the current time step and the first related time step can be accurately determined through the indication information. That is, it can be accurately determined which network layer features can be reused between the current time step and the first related time step. By using the image semantic features of the first related time step in the first related network layer in the cache to determine the image semantic features of the current time step in the first related network layer, unnecessary computation is reduced, thereby speeding up the video processing speed. This is especially important for real-time video processing or application scenarios that require fast response.

[0096] In some embodiments of this disclosure, the video processing method further includes: determining, according to processing instruction information, whether there is a second related time step associated with the current time step; the second related time step is a time step after the current time step; if there is a second related time step, determining, according to processing instruction information, the second related network layer corresponding to the current time step and the second related time step; and placing the image semantic features of the second related network layer during the processing of the current time step into cache information.

[0097] In this embodiment of the disclosure, during the processing of the current time step, it is determined whether there is a second related time step that is related to the current time step according to the processing instruction information, that is, it is determined whether there is a time step that is related to the current time step in the time steps after the current time step.

[0098] If a second relevant time step is identified at the current time step, it indicates a high similarity between the image semantic features of the current time step and the image semantic features of the second relevant time step. Further, the second relevant network layer corresponding to the current time step and the second relevant time step is determined, i.e., determining which specific network layer in which the image semantic features of the current time step and the second relevant time step are similar. Thus, after obtaining the image semantic features of the second relevant network layer during the current time step processing, these features are placed in the cache information. This allows the feature to be directly retrieved from the cache information during the second relevant time step processing and used as the image semantic feature of the second relevant network layer during the second relevant time step processing.

[0099] Taking the current time step t+2 as an example, time steps t+2 and t+3 are related, and the corresponding second related network layer is network layer 2. This means that the image semantic features of network layer 2 during the processing of time step t+2 have a high similarity to the image semantic features during the processing of time step t+3. During the processing of time step t+2, the image semantic features of network layer 2 are stored in the cache information. Then, during the processing of time step t+3, this feature is retrieved from the cache information and used as the image semantic features of network layer 2 during the processing of time step t+3. Furthermore, since network layers 1 and 3 are not the second related network layers corresponding to time steps t+2 and t+3, during the processing of time step t+3, feature extraction is performed using network layers 1 and 3 respectively to obtain the image semantic features of network layers 1 and 3 during the processing of time step t+3.

[0100] In the video processing method provided in this embodiment, if there is a second related time step during the processing of the current time step, the second related network layer corresponding to the current time step and the second related time step can be accurately determined through indication information. That is, it can accurately determine which network layer features can be reused between the current time step and the second related time step. The image semantic features of the second related network layer at the current time step are stored in the cache. This reduces the need to recalculate these features in the subsequent second time step processing, reduces unnecessary computation, and thus significantly improves the efficiency of video processing.

[0101] In summary, the video processing method provided in this disclosure acquires the video frame to be processed and its processing instruction information, then uses a video processing model to iteratively repair the video frame to be processed at preset time steps. In each time step, the repair result is updated based on the processing instruction information and the repair result of the previous time step (the first step is noise data). Finally, a high-quality target video frame is determined based on the repair result obtained in the last time step. It is evident that this method introduces processing instruction information, which indicates the time steps with correlations. This allows the video processing model to effectively utilize the correlations between time steps during iterative repair processing, reducing unnecessary computation and thus lowering the consumption of computing resources and shortening inference time. Therefore, this method can generate high-quality video faster, improve the user's visual experience, and is suitable for real-time or near-real-time video processing scenarios.

[0102] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0103] Figure 7 A block diagram of a video processing apparatus according to an embodiment of the present disclosure is shown. Figure 7 As shown, the video processing device 700 includes: an acquisition module 710, an iteration module 720, and a processing module 730.

[0104] The acquisition module 710 is configured to acquire the video frame to be processed and its processing instruction information; wherein the processing instruction information indicates the time steps where a correlation exists. The iteration module 720 is configured to: during the processing of each time step of a preset time step, determine whether a first related time step exists based on the processing instruction information, wherein the first related time step is a time step preceding the current time step; if a first related time step exists, obtain the image semantic features of the current time step based on the processing instruction information and cache information; perform repair processing on the video frame to be processed based on the image semantic features of the current time step to obtain the repair result of the current time step. The processing module 730 is configured to: obtain the target video frame of the video frame to be processed based on the repair result obtained from the last time step.

[0105] In some embodiments of this disclosure, the iteration module 720 is further configured to: if there is no first relevant time step, extract features from the video frame to be processed through the state control branch of the video processing model to obtain the image semantic features of the current time step.

[0106] In some embodiments of this disclosure, the state control branch includes one or more network layers. The iteration module 720 is further configured to: determine the first relevant network layer corresponding to the current time step and the first relevant time step based on processing instruction information; the processing instruction information is used to indicate the relevant network layer corresponding to the time step with a correlation; for each network layer, if the network layer is the first relevant network layer, extract the image semantic features of the network layer during the processing of the first relevant time step from the cache information, and use the extracted result as the image semantic features of the network layer during the processing of the current time step; if the network layer is not the first relevant network layer, perform feature extraction on the video frame to be processed through the network layer to obtain the image semantic features of the network layer during the processing of the current time step.

[0107] In some embodiments of this disclosure, the iteration module 720 is further configured to: determine, according to the processing instruction information, whether there is a second related time step associated with the current time step; the second related time step is a time step after the current time step; if there is a second related time step, determine, according to the processing instruction information, the second related network layer corresponding to the current time step and the second related time step; and put the image semantic features of the second related network layer during the processing of the current time step into the cache information.

[0108] In some embodiments of this disclosure, the iteration module 720 is further configured to: fuse the image semantic features of the current time step and the repair result of the previous time step through the main model branch of the video processing model to obtain the repair result of the current time step; wherein, if the current time step is the first time step, the repair result of the previous time step is noise data.

[0109] In some embodiments of this disclosure, the apparatus 700 further includes an instruction generation module 740, configured to pre-generate processing instruction information according to the following process: acquiring sample video frames, wherein the scene information of the sample video frames is the same as the scene information of the video frames to be processed; extracting features from the sample video frames through each network layer in the state control branch included in the video processing model according to a preset time step, thereby obtaining sample features of each network layer in each time step processing; for each network layer, calculating the similarity between sample features of the network layer in different time steps processing based on the obtained sample features, determining the time steps with correlation and the corresponding network layers based on the calculated similarity; and generating processing instruction information based on the time steps with correlation and the corresponding network layers.

[0110] In some embodiments of this disclosure, the instruction generation module 740 is further configured to: for each network layer, select sample features of the network layer in any two time steps from the obtained sample features, calculate the similarity between the two selected sample features; if the calculated similarity is greater than the similarity threshold, determine the time steps corresponding to the two selected sample features as time steps with a correlation relationship, and use the network layer as the relevant network layer corresponding to the time step with a correlation relationship.

[0111] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0112] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. It should be noted that... Figure 8 The illustrated electronic device 800 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0113] like Figure 8 As shown, the electronic device 800 is manifested in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including storage unit 820 and processing unit 810).

[0114] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 810 can perform, as follows: Figure 2 The steps are shown in the figure.

[0115] Storage unit 820 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include a read-only memory (ROM) 8203.

[0116] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0117] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0118] Electronic device 800 can also communicate with one or more external devices 900 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 840. As shown, network adapter 840 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0120] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0121] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0122] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0123] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0124] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0125] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0126] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0127] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0128] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered illustrative only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0129] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that, The method includes: Obtain the video frame to be processed and the processing instruction information of the video frame to be processed; the processing instruction information is used to indicate the time steps where there is a correlation. During the processing of each time step of the preset time step, it is determined whether there is a first related time step that is related to the current time step according to the processing instruction information; the first related time step is the time step before the current time step; If the first relevant time step exists, the image semantic features of the current time step are obtained according to the processing instruction information and cache information; The video frame to be processed is repaired based on the image semantic features at the current time step to obtain the repair result at the current time step; Based on the repair result obtained from the last time step, the target video frame of the video frame to be processed is obtained.

2. The method according to claim 1, characterized in that, The method further includes: If the first relevant time step does not exist, the video processing model's state control branch is used to extract features from the video frame to be processed, thereby obtaining the image semantic features of the current time step.

3. The method according to claim 2, characterized in that, The state control branch includes one or more network layers; wherein, obtaining the image semantic features of the current time step based on the processing instruction information and cache information includes: Based on the processing instruction information, the first relevant network layer corresponding to the current time step and the first relevant time step is determined; the processing instruction information is used to indicate the relevant network layer corresponding to the time step with the relevant relationship. For each network layer, if the network layer is the first relevant network layer, then the image semantic features of the network layer during the first relevant time step processing are extracted from the cache information, and the extracted results are used as the image semantic features of the network layer during the current time step processing. If the network layer is not the first relevant network layer, feature extraction is performed on the video frame to be processed through the network layer to obtain the image semantic features of the network layer during the current time step processing.

4. The method according to claim 3, characterized in that, The method further includes: Based on the processing instruction information, it is determined whether there is a second related time step associated with the current time step; the second related time step is a time step after the current time step; If the second relevant time step exists, then the second relevant network layer corresponding to the current time step and the second relevant time step is determined according to the processing instruction information; The image semantic features of the second related network layer during the current time step processing will be placed into the cache information.

5. The method according to claim 2, characterized in that, The step of repairing the video frame to be processed based on the image semantic features at the current time step to obtain the repair result at the current time step includes: The image semantic features at the current time step and the repair result at the previous time step are fused through the main branch of the video processing model to obtain the repair result at the current time step; wherein, if the current time step is the first time step, the repair result at the previous time step is noise data.

6. The method according to any one of claims 1 to 5, characterized in that, The processing instruction information is pre-generated according to the following process: Obtain sample video frames; the scene information of the sample video frames is the same as the scene information of the video frames to be processed; According to the preset time step, feature extraction is performed on the sample video frame through each network layer in the state control branch of the video processing model to obtain the sample features of each network layer in each time step processing. For each network layer, based on the obtained sample features, the similarity between the sample features processed by the network layer at different time steps is calculated. Based on the calculated similarity, the time steps with correlation and the corresponding network layers of the time steps with correlation are determined. The processing instruction information is generated based on the time steps where the correlation exists and the corresponding network layers of the time steps where the correlation exists.

7. The method according to claim 6, characterized in that, For each network layer, based on the obtained sample features, the similarity between the sample features processed by the network layer at different time steps is calculated. Based on the calculated similarity, the time steps with correlation and the corresponding network layers at the time steps with correlation are determined, including: For each network layer, select sample features from the obtained sample features processed at any two time steps of the network layer, and calculate the similarity between the two selected sample features. If the calculated similarity is greater than the similarity threshold, then the time step corresponding to the selected two sample features is determined as the time step where there is a correlation, and the network layer is used as the relevant network layer corresponding to the time step where there is a correlation.

8. A video processing apparatus, characterized in that, The device includes: The acquisition module is configured to acquire a video frame to be processed and processing indication information of the video frame to be processed; the processing indication information is used to indicate time steps where there is a correlation. The iteration module is configured to, during the processing of each time step of a preset time step, determine whether there is a first related time step associated with the current time step based on the processing instruction information; the first related time step is a time step preceding the current time step; if the first related time step exists, then obtain the image semantic features of the current time step based on the processing instruction information and cache information; and perform repair processing on the video frame to be processed based on the image semantic features of the current time step to obtain the repair result of the current time step. The processing module is configured to obtain the target video frame of the video frame to be processed based on the repair result obtained from the last time step.

9. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the video processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the video processing method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the video processing method as described in any one of claims 1 to 7.

Images