Video processing method and device

Abstract

The invention provides a video processing method and device. The video processing method comprises the following steps: acquiring a to-be-processed original video frame sequence; constructing a pyramid-type multi-layer video frame sequence from bottom to top based on the original video frame sequence; and processing the multi-layer video frame sequence from top to bottom to obtain a target video frame sequence corresponding to the original video frame sequence.

Description

Video processing methods and apparatus Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a video processing method, video processing apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of video generation technology, advanced video diffusion models (such as the DiT (DiffusionTransformer) model) can now generate videos with rich and realistic details. Thanks to the powerful generation capabilities of video diffusion models, generative restoration of low-quality videos can now restore high-quality results. However, limited by the temporal window size of a single inference, long video restoration based on video diffusion models often faces problems such as abrupt changes in objects in the scene and the cumulative collapse of errors in static areas of the scene, resulting in poor temporal consistency of the video. Summary of the Invention

[0003] According to a first aspect of the present disclosure, a video processing method is provided, the method comprising: acquiring an original video frame sequence to be processed; constructing a pyramid-shaped multi-layer video frame sequence from bottom to top based on the original video frame sequence; and processing the multi-layer video frame sequence from top to bottom to obtain a target video frame sequence corresponding to the original video frame sequence.

[0004] Optionally, constructing a pyramid-shaped multi-layer video frame sequence from bottom to top based on the original video frame sequence includes: using the original video frame sequence as the bottom layer video frame sequence of the multi-layer video frame sequence; for each layer video frame sequence in the other layers of the multi-layer video frame sequence, extracting video frames from the next layer video frame sequence at preset intervals, and using the extracted video frames as the video frame sequence of that layer.

[0005] Optionally, processing the multi-layer video frame sequence from top to bottom includes: for the topmost video frame sequence of the multi-layer video frame sequence, processing the topmost video frame sequence using an artificial intelligence model; for each of the other video frame sequences in the multi-layer video frame sequence, processing the video frame sequence of that layer using the artificial intelligence model based on the target video frame in the processed previous layer video frame sequence of that layer.

[0006] Optionally, the target video frame is obtained based on a preset interval used in constructing the multi-layer video frame sequence.

[0007] Optionally, for the topmost video frame sequence of the multi-layer video frame sequence, an artificial intelligence model is used to process the topmost video frame sequence, including: for video frames to be processed in a preset window starting from the first video frame of the topmost video frame sequence, the artificial intelligence model is used to process the video frames to be processed; for video frames to be processed in the preset window that do not start from the first video frame, the artificial intelligence model is used to process the video frames to be processed based on the processed video frames before the preset window.

[0008] Optionally, for each layer of video frame sequence in the other layers of the multi-layer video frame sequence, the artificial intelligence model is used to process the video frame sequence of that layer based on the target video frame in the processed previous layer of video frame sequence. This includes: for video frames to be processed in a preset window starting from the first video frame of that layer, the artificial intelligence model is used to process the video frames to be processed based on the target video frames in the target video frames that correspond to the video frames to be processed; for video frames to be processed in the preset window that do not start from the first video frame, the artificial intelligence model is used to process the video frames to be processed based on the target video frames in the target video frames that correspond to the video frames to be processed and the processed video frames before the preset window.

[0009] According to a second aspect of the present disclosure, a video processing apparatus is provided, the apparatus comprising: an acquisition module configured to acquire an original video frame sequence to be processed; a video construction module configured to construct a pyramid-shaped multi-layer video frame sequence from bottom to top based on the original video frame sequence; and a video processing module configured to process the multi-layer video frame sequence from top to bottom to obtain a target video frame sequence corresponding to the original video frame sequence.

[0010] Optionally, the video construction module is configured to: use the original video frame sequence as the bottom layer video frame sequence of the multi-layer video frame sequence; for each layer video frame sequence in the other layers of the multi-layer video frame sequence, extract video frames from the next layer video frame sequence at a preset interval, and use the extracted video frames as the video frame sequence of that layer.

[0011] Optionally, the video processing module is configured to: process the topmost video frame sequence of the multi-layer video frame sequence using an artificial intelligence model; and process each of the other video frame sequences in the multi-layer video frame sequence using the artificial intelligence model based on the target video frame in the processed previous layer video frame sequence of that layer.

[0012] Optionally, the target video frame is obtained based on a preset interval used in constructing the multi-layer video frame sequence.

[0013] Optionally, the video processing module is configured to: process video frames to be processed using the artificial intelligence model within a preset window starting from the first video frame of the top-level video frame sequence; and process video frames to be processed using the artificial intelligence model based on previously processed video frames before the preset window for video frames that do not start from the first video frame.

[0014] Optionally, the video processing module is configured to: for each layer of video frame sequence in the other layers of the multi-layer video frame sequence, for a video frame to be processed in a preset window starting from the first video frame of that layer, use the artificial intelligence model to process the video frame to be processed based on the target video frame in the target video frame that corresponds to the video frame to be processed; for a video frame to be processed in the preset window that does not start from the first video frame, use the artificial intelligence model to process the video frame to be processed based on the target video frame in the target video frame that corresponds to the video frame to be processed and the processed video frames before the preset window.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device may include: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the video processing method as described above.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores instructions which, when executed by at least one processor, cause the at least one processor to perform the video processing method as described above.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, wherein instructions in the computer program product are executed by at least one processor in an electronic device to perform the video processing method as described above.

[0018] The technical solutions provided by the embodiments of this disclosure offer at least the following beneficial effects: by introducing a pyramid inference mechanism, and through top-down pyramid layer-by-layer reasoning, the temporal consistency of video diffusion repair is improved. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0020] Figure 1 is a flowchart of a video processing method according to an embodiment of the present disclosure.

[0021] Figure 2 illustrates a schematic diagram of obtaining a multi-layer video frame sequence according to an embodiment of the present disclosure.

[0022] Figure 3 shows a schematic diagram of video frames within an inference time window according to an embodiment of the present disclosure.

[0023] Figure 4 is a block diagram of a video processing apparatus according to an embodiment of the present disclosure.

[0024] Figure 5 is a block diagram of a video processing apparatus according to an embodiment of the present disclosure.

[0025] Figure 6 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0026] Throughout the accompanying drawings, it should be noted that the same reference numerals are used to denote the same or similar elements, features, and structures. Detailed Implementation

[0027] The following description, provided with reference to the accompanying drawings, is intended to aid in a full understanding of embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to aid understanding, but these details are to be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures are omitted.

[0028] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0029] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0030] Currently, video restoration schemes based on the DiT model typically use low-quality video and text descriptions as model input. They leverage the model's powerful generative capabilities to perform generative image restoration, resulting in a restored high-quality video. Adjacent inference time windows usually have a small number of overlapping frames to improve temporal consistency.

[0031] For example, any video restoration diffusion model (such as the DiT model) can be derived from... express, This represents the low-quality video frames from frame i to frame j. For high-quality repaired video frames from frame i to frame j, These are the other required input conditions for the DiT model. The variables mentioned above have the following relationship: or in, This represents the temporal concatenation of high-quality restored video frames from frame i to frame j and low-quality video frames from frame j+1 to frame k. The two relationships described above correspond to the two cases of the DiT model: one with a high-quality prefix and the other with a high-quality prefix frame reference.

[0032] However, in the above scheme, during long video restoration, except for the first inference time window (i.e., the inference time window starting from the first video frame in the video frame sequence), the last output frame of the previous inference time window is used as the start of the next inference time window. Although the above scheme can pass the inference result of the previous inference time window to the next inference time window, errors accumulate during multiple rounds of passing in the long video restoration process. After multiple passings, the image quality will deteriorate significantly, severely affecting the video restoration effect.

[0033] In response to the above-mentioned solutions, this disclosure proposes a technical solution for improving the temporal consistency of images in long video diffusion repair. The technical concept of this disclosure will be described in detail below.

[0034] Figure 1 is a flowchart of a video processing method according to an embodiment of the present disclosure. The method shown in Figure 1 can be performed by an electronic device with video processing capabilities.

[0035] In step S101, the original video frame sequence to be processed is obtained. For example, the original video frame sequence may be a long video with low-quality images, or a video frame sequence that needs to be repaired. The above examples are merely illustrative, and this disclosure is not limited thereto.

[0036] In step S102, a pyramid-shaped multi-layer video frame sequence is constructed from the bottom up based on the original video frame sequence.

[0037] According to the embodiment, the original video frame sequence can be used as the bottom layer video frame sequence of a multi-layer video frame sequence. For each layer video frame sequence in the other layers of the multi-layer video frame sequence, video frames are extracted from the next layer video frame sequence at preset intervals, and the extracted video frames are used as the video frame sequence of that layer. Here, the next layer can be understood as the layer below and adjacent to the current layer.

[0038] As an example, a multi-level video structure can be built from the bottom up. For instance, suppose we want to build a shared... A pyramid-shaped video structure, the first The number of frames per layer is expressed as The original video frame sequence can be used as the first-level video frame sequence in the video structure (i.e., the lowest-level video frame sequence, which is represented as...). The subsequent... Layered video frame sequence It can be obtained from the first Each interval in the video frame sequence of the layer Extract a frame to obtain the following expression (1).

[0039] (1) FIG2 shows a schematic diagram of obtaining a multi-layer video frame sequence according to an embodiment of the present disclosure.

[0040] Referring to Figure 2, assume the original video frame sequence to be repaired contains 100 frames, and the interval between frame extractions is 5 frames. When constructing a pyramid-shaped video structure, the original video frame sequence can be used as the bottom-level sequence. Every 5 frames from the original video frame sequence, one frame is extracted, forming a new video frame sequence, which is then used as the second-level (counting from the bottom up) video frame sequence. Then, every 5 frames from the second-level sequence, one frame is extracted, forming a new video frame sequence, which is then used as the third-level sequence. This process continues until a preset requirement is met. For example, if the preset is L levels, extraction can stop after obtaining the L-th level video frame sequence; or, for example, extraction can stop after the top-level frame count reaches a preset number of frames. The above examples are merely illustrative, and this disclosure is not limited thereto.

[0041] In step S103, the multi-layer video frame sequence is processed from top to bottom to obtain a target video frame sequence corresponding to the original video frame sequence. The target video frame sequence can represent a high-quality video frame sequence after the original video frame sequence has been repaired.

[0042] According to embodiments of this disclosure, for the topmost video frame sequence of a multi-layer video frame sequence, an artificial intelligence model can be used to process the topmost video frame sequence. For each of the other layers of the multi-layer video frame sequence, the artificial intelligence model processes the layer's video frame sequence based on the target video frame in the previously processed upper-layer video frame sequence. For example, the artificial intelligence model can be the aforementioned DiT model, but this disclosure is not limited to this; it can also be any video diffusion model. Here, the upper layer of the current layer can be understood as the layer above and adjacent to the current layer.

[0043] As an example, for the common The video structure can be divided into layers, starting from the top layer (the first layer). Reasoning proceeds layer by layer from the first layer (level 1) to the bottom layer (level 2). Except for the first... Outside the first layer, the second During inference, the layers need to be sequentially spliced ​​together from the first layer. The scene frame (i.e. the target video frame) of the layer is used as a reference.

[0044] According to the embodiment, in the case of the top-level video frame sequence, for video frames to be processed within a preset window starting from the first video frame of the top-level video frame sequence, an artificial intelligence model is used to process the video frames to be processed; for video frames to be processed within the preset window that do not start from the first video frame, the artificial intelligence model is used to process the video frames to be processed based on the video frames already processed before the preset window. Here, the preset window may refer to the aforementioned inference time window, and the size of the window can be set differently.

[0045] As an example, that is, for the first Layer, DiT model The inference method for repairing the i-th to j-th frames in a certain inference time window can be the same as the above method, as shown in the following expression (2).

[0046] (2) Among them, This represents the i-th to j-th frames of the (low-quality) video frame sequence of layer L. Other required inputs to the model, This represents the i-th to j-th frames of the processed (high-quality) video frame sequence of layer L. This indicates the previous processed frame of the i-th frame, i.e., the (high-quality) video frame that was not included in the current inference time window and was processed before the current inference time window. This situation can be understood as processing the i-th to j-th frames within the inference time window starting from the first video frame. This situation can be understood as processing frames i through j within an inference time window that does not begin with the first video frame. In this case, when performing inference on video frames within the current inference time window, the output at the end of the previous inference time window (i.e., ...) can be combined with the output at the end of the previous inference time window. (representing processed video frames) performs inference on the video frames in the current inference time window.

[0047] Figure 3 shows a schematic diagram of video frames within an inference time window according to an embodiment of the present disclosure.

[0048] Referring to Figure 3, Figure 3(a) shows the above. Figure 3(b) illustrates the above situation. In the case of inference in the video diffusion model, the current video frame to be processed is obtained through an inference time window. The first video frame in the inference time window containing the video frame sequence can be understood as the first inference time window, and the first video frame in the inference time window that does not contain the video frame sequence can be understood as a certain inference time window after the first inference time window. Figure 3 is only an example, and this disclosure is not limited thereto.

[0049] According to the embodiment, in the case of a non-top-level video frame sequence, for the video frames to be processed in a preset window starting from the first video frame of that layer's video frame sequence, an artificial intelligence model is used to process the video frames to be processed based on the target video frame corresponding to the video frames to be processed; for the video frames to be processed in a preset window that does not start from the first video frame, an artificial intelligence model is used to process the video frames to be processed based on the target video frame corresponding to the video frames to be processed and the video frames that have been processed before the preset window.

[0050] As an example, for the non-topmost... Layers, during reasoning, can be sequentially spliced ​​together from the first layer. Scene frames of the layer For reference, the reasoning method can be shown in the following expression (3).

[0051] (3) Among them, situation and The situation can be referred to in Figure 3, and will not be repeated here.

[0052] According to an embodiment, a target video frame can be obtained based on a preset interval used in constructing a multi-layer video frame sequence.

[0053] As an example, the frame number of the scene frame can be calculated according to the following equation (4).

[0054] (4) No. The scene frame of a layer can be selected as the one whose timestamp is earlier than and closest to the current inference time window. The output of the layer.

[0055] This disclosure introduces a scene frame pyramid inference method, which extracts frames from the video at intervals to form a multi-layer pyramid. Except for the top layer, when inferring each layer of the pyramid, the scene frame of the previous layer that guides the current inference window is added for inference, so that the inference follows the top-down temporal consistency.

[0056] Through top-down pyramid-style layer-by-layer reasoning, higher layers provide lower layers with reference scene frames that have a more global temporal perspective. These frames can serve as temporal constraints during lower-level reasoning, reduce error accumulation, and improve the temporal consistency of repair results during lower-level reasoning.

[0057] Figure 4 is a block diagram of a video processing apparatus according to an embodiment of the present disclosure.

[0058] Referring to Figure 4, the video processing apparatus 400 may include an acquisition module 401, a video construction module 402, and a video processing module 403. The number and names of the modules shown in Figure 4 are merely exemplary, and this disclosure is not limited thereto. Modules may be added, reduced, merged, or split as needed.

[0059] According to an embodiment, the acquisition module 401 can acquire the original video frame sequence to be processed. For example, the original video frame sequence may be a long video with low-quality images, or a video frame sequence that needs to be repaired. The above examples are merely illustrative, and this disclosure is not limited thereto.

[0060] The video construction module 402 can construct a pyramid-shaped multi-layer video frame sequence from the bottom up based on the original video frame sequence.

[0061] According to an embodiment, the video construction module 402 can use the original video frame sequence as the bottom layer video frame sequence of the multi-layer video frame sequence; for each layer video frame sequence in the other layers of the multi-layer video frame sequence, video frames are extracted from the next layer video frame sequence at a preset interval, and the extracted video frames are used as the video frame sequence of that layer. For example, the video structure of this disclosure can be constructed with reference to the above expression (1) and Figure 2.

[0062] The video processing module 403 can process the multi-layer video frame sequence from top to bottom to obtain a target video frame sequence corresponding to the original video frame sequence.

[0063] According to the embodiment, for the topmost video frame sequence of the multi-layer video frame sequence, the video processing module 403 can use an artificial intelligence model to process the topmost video frame sequence; for each of the other layers of the multi-layer video frame sequence, the artificial intelligence model can be used to process the video frame sequence based on the target video frame in the processed previous layer of the video frame sequence.

[0064] According to an embodiment, the target video frame is obtained based on a preset interval used in constructing the multi-layer video frame sequence. For example, the preset interval can be obtained by referring to equation (4) above.

[0065] According to the embodiment, for a video frame to be processed within a preset window starting from the first video frame of the top-level video frame sequence, the video processing module 403 can use the artificial intelligence model to process the video frame to be processed; for a video frame to be processed within the preset window that does not start from the first video frame, the artificial intelligence model can be used to process the video frame to be processed based on the processed video frames before the preset window. For example, the video frame can be processed with reference to the expression (2) above.

[0066] According to an embodiment, the video processing module 403 can process video frames in each of the other layers of the multi-layer video frame sequence, for video frames to be processed within a preset window starting from the first video frame of that layer, using the artificial intelligence model based on the target video frame corresponding to the video frame to be processed in the target video frame; for video frames to be processed within the preset window that do not start from the first video frame, using the artificial intelligence model based on the target video frame corresponding to the video frame to be processed in the target video frame and the processed video frames before the preset window. For example, video frames can be processed with reference to the expression (3) above.

[0067] By introducing a scene pyramid reasoning mechanism, the higher levels provide the lower levels with reference scene frames that have a more global temporal perspective through top-down pyramid reasoning. These frames can serve as temporal constraints during lower-level reasoning and reduce error accumulation, thereby improving the temporal consistency of the repair results during lower-level reasoning.

[0068] Figure 5 is a schematic diagram of the structure of a video processing device in the hardware operating environment of an embodiment of this disclosure. The video processing device described in Figure 5 can execute a video processing method according to an embodiment of this disclosure.

[0069] As shown in Figure 5, the video processing device 1000 may include: a processing component 1001, a communication bus 1002, a network interface 1003, an input / output interface 1004, a memory 1005, and a power supply component 1006. The communication bus 1002 is used to enable communication between these components. The input / output interface 1004 may include a video display (such as a liquid crystal display), a microphone and speaker, and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). Optionally, the input / output interface 1004 may also include standard wired interfaces and wireless interfaces. The network interface 1003 may optionally include standard wired interfaces and wireless interfaces (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory or a stable non-volatile memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processing component 1001.

[0070] Those skilled in the art will understand that the structure shown in FIG5 does not constitute a limitation on the video processing device 1000, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0071] As shown in Figure 5, the memory 1005, which serves as a storage medium, may include an operating system (such as a MAC operating system), a data storage module, a network communication module, a user interface module, a program implementing this disclosure, and a database.

[0072] In the video processing device 1000 shown in Figure 5, the network interface 1003 is mainly used for data communication with external electronic devices / terminals; the input / output interface 1004 is mainly used for data interaction with users; the processing component 1001 and the memory 1005 in the video processing device 1000 can be disposed in the video processing device 1000. The video processing device 1000 calls the program stored in the memory 1005 and various APIs provided by the operating system through the processing component 1001 to execute the video processing method provided in the embodiments of this disclosure.

[0073] Processing component 1001 may include at least one processor, and memory 1005 stores a set of computer-executable instructions. When the set of computer-executable instructions is executed by at least one processor, a video processing method according to embodiments of the present disclosure is performed. However, the above examples are merely exemplary, and the present disclosure is not limited thereto.

[0074] The processing component 1001 can control the components included in the video processing device 1000 by executing a program.

[0075] As an example, the video processing device 1000 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, the video processing device 1000 is not necessarily a single electronic device, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The video processing device 1000 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0076] In the video processing device 1000, the processing component 1001 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processing component 1001 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0077] The processing component 1001 can execute instructions or code stored in memory, wherein memory 1005 can also store data. Instructions and data can also be sent and received over a network via network interface 1003, wherein network interface 1003 can employ any known transport protocol.

[0078] The memory 1005 can be integrated with the processing component 1001, for example, by arranging RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 1005 can include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory and the processing component 1001 can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processing component 1001 to read data stored in the memory 1005.

[0079] According to embodiments of the present disclosure, an electronic device may be provided. FIG6 is a block diagram of an electronic device 1100 according to an embodiment of the present disclosure. The electronic device 1100 may include at least one memory 1102 and at least one processor 1101. The at least one memory 1102 stores a set of computer-executable instructions. When the set of computer-executable instructions is executed by the at least one processor 1101, a video processing method according to an embodiment of the present disclosure is performed.

[0080] Processor 1101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 1101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0081] The memory 1102, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, a program for executing the methods of this disclosure, and a database.

[0082] The memory 1102 may be integrated with the processor 1101; for example, RAM or flash memory may be arranged within an integrated circuit microprocessor. Alternatively, the memory 1102 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory and processor may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the memory.

[0083] In addition, electronic device 1100 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of electronic device 1100 can be interconnected via a bus and / or network.

[0084] As an example, electronic device 1100 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 1100 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 1100 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0085] Those skilled in the art will understand that the structure shown in FIG6 does not constitute a limitation on the structure and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0086] According to embodiments of this disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, they cause at least one processor to perform a video processing method according to this disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0087] According to embodiments of this disclosure, a computer program product may also be provided, wherein the instructions in the computer program product can be executed by the processor of a computer device to perform the above-described video processing method.

[0088] According to embodiments of this disclosure, the video inference process is expanded from linear inference of single-layer video frames (inferring video frames from front to back) to top-down inference of a multi-layer pyramid with scene frames. The scene pyramid is used for video restoration inference, leveraging the broader temporal perspective of higher layers during inference to achieve temporal consistency of restoration results over a larger time span. Simultaneously, the restoration results of the previous layer serve as reference frames during the inference of the next layer, guiding the restoration direction of the next layer and resulting in better temporal consistency of similar segments within the same inference layer.

[0089] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0090] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video processing method, comprising: Obtain the original video frame sequence to be processed; construct a pyramid-shaped multi-layer video frame sequence from bottom to top based on the original video frame sequence; and process the multi-layer video frame sequence from top to bottom to obtain the target video frame sequence corresponding to the original video frame sequence.

2. The method according to claim 1, wherein, Constructing a pyramid-shaped multi-layer video frame sequence from bottom to top based on the original video frame sequence includes: using the original video frame sequence as the bottom layer video frame sequence of the multi-layer video frame sequence; for each layer video frame sequence in the other layers of the multi-layer video frame sequence, extracting video frames from the next layer video frame sequence at preset intervals, and using the extracted video frames as the video frame sequence of that layer.

3. The method according to claim 1, wherein, The top-down processing of the multi-layer video frame sequence includes: for the top-level video frame sequence of the multi-layer video frame sequence, processing the top-level video frame sequence using an artificial intelligence model; for each of the other layers of the multi-layer video frame sequence, processing the layer of video frame sequence using the artificial intelligence model based on the target video frame in the processed previous layer of video frame sequence.

4. The method according to claim 3, wherein, The target video frame is obtained based on a preset interval used in constructing the multi-layer video frame sequence.

5. The method according to claim 3, wherein, For the topmost video frame sequence of the multi-layer video frame sequence, the topmost video frame sequence is processed using an artificial intelligence model, including: for video frames to be processed in a preset window starting from the first video frame of the topmost video frame sequence, the artificial intelligence model is used to process the video frames to be processed; for video frames to be processed in the preset window that do not start from the first video frame, the artificial intelligence model is used to process the video frames to be processed based on the video frames that have been processed before the preset window.

6. The method according to claim 3, wherein, For each layer of video frame sequence in the other layers of the multi-layer video frame sequence, the artificial intelligence model is used to process the video frame sequence of that layer based on the target video frames in the processed previous layer of video frame sequence. This includes: for video frames to be processed in a preset window starting from the first video frame of that layer, the artificial intelligence model is used to process the video frames to be processed based on the target video frames in the target video frames that correspond to the video frames to be processed; for video frames to be processed in the preset window that do not start from the first video frame, the artificial intelligence model is used to process the video frames to be processed based on the target video frames in the target video frames that correspond to the video frames to be processed and the processed video frames before the preset window.

7. A video processing apparatus, comprising: The acquisition module is configured to acquire the raw video frame sequence to be processed; A video construction module is configured to construct a pyramid-shaped multi-layer video frame sequence from bottom to top based on the original video frame sequence; and a video processing module is configured to process the multi-layer video frame sequence from top to bottom to obtain a target video frame sequence corresponding to the original video frame sequence.

8. The apparatus according to claim 7, wherein, The video construction module is configured to: use the original video frame sequence as the bottom layer video frame sequence of the multi-layer video frame sequence; for each layer video frame sequence in the other layers of the multi-layer video frame sequence, extract video frames from the next layer video frame sequence at a preset interval, and use the extracted video frames as the video frame sequence of that layer.

9. The apparatus according to claim 7, wherein, The video processing module is configured to: process the topmost video frame sequence of the multi-layer video frame sequence using an artificial intelligence model; and process each video frame sequence of other layers of the multi-layer video frame sequence using the artificial intelligence model based on the target video frame in the processed previous layer video frame sequence of that layer.

10. The apparatus according to claim 9, wherein, The target video frame is obtained based on a preset interval used in constructing the multi-layer video frame sequence.

11. The apparatus according to claim 9, wherein, The video processing module is configured to: process video frames to be processed using the artificial intelligence model within a preset window starting from the first video frame of the top-level video frame sequence; and process video frames to be processed using the artificial intelligence model based on previously processed video frames before the preset window for video frames that do not start from the first video frame.

12. The apparatus according to claim 9, wherein, The video processing module is configured to: for each layer of video frame sequence in the other layers of the multi-layer video frame sequence, for a video frame to be processed in a preset window starting from the first video frame of that layer, use the artificial intelligence model to process the video frame to be processed based on the target video frame in the target video frame that corresponds to the video frame to be processed; for a video frame to be processed in the preset window that does not start from the first video frame, use the artificial intelligence model to process the video frame to be processed based on the target video frame in the target video frame that corresponds to the video frame to be processed and the processed video frames before the preset window.

13. An electronic device, characterized in that, It includes: at least one processor; at least one memory storing computer-executable instructions, wherein, when executed by the at least one processor, the computer-executable instructions cause the at least one processor to perform the video processing method as described in any one of claims 1 to 6.

14. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one processor, it causes the at least one processor to perform the video processing method as described in any one of claims 1 to 6.

15. A computer program product, wherein instructions in the computer program product are executed by at least one processor in an electronic device to perform the video processing method as claimed in any one of claims 1 to 6.

Images