A video processing device, method, electronic device, storage medium, and computer program product

Abstract

The present disclosure relates to the field of computer technologies, and in particular, to a video processing apparatus, method, electronic device, storage medium, and computer program product. The video processing apparatus includes: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module includes: a downsampling unit configured to perform downsampling processing on a video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed; a shared computing unit configured to, when the motion and optical flow fusion processing module is in an optical flow calculation mode, perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain an optical flow vector of the video to be processed; and the shared computing unit configured to, when the motion and optical flow fusion processing module is in a motion estimation mode, perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain a motion vector of the video to be processed. According to the embodiments of the present disclosure, the hardware utilization rate can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a video processing apparatus, method, electronic device, storage medium, and computer program product. Background Art

[0002] In the video encoding scenario, motion estimation and optical flow calculation are two completely independent processes. Motion estimation is performed during video compression encoding based on minimum residual estimation for motion search, which is exclusive to the video encoder. Optical flow calculation is based on motion search for the best-matching pixels, usually calculated using a general module, with relatively high search accuracy and long time consumption. Summary of the Invention

[0003] In view of this, the present disclosure provides a video processing apparatus, method, electronic device, storage medium, and computer program product.

[0004] According to an aspect of the present disclosure, there is provided a video processing apparatus, including: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module includes: a downsampling unit, a shared calculation unit; the downsampling unit is configured to perform downsampling processing on a video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed; the shared calculation unit is configured to, when the motion and optical flow fusion processing module is in the optical flow calculation mode, perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain an optical flow vector of the video to be processed; the shared calculation unit is configured to, when the motion and optical flow fusion processing module is in the motion estimation mode, perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain a motion vector of the video to be processed.

[0005] In a possible implementation, the apparatus further includes: a micro control unit (MCU), a control status register (CSR); the MCU is configured to control the motion and optical flow fusion processing module to perform mode switching between the optical flow calculation mode and the motion estimation mode by configuring the CSR.

[0006] In a possible implementation, the downsampling unit includes: a configuration subunit; the configuration subunit is configured to configure image pyramid search parameters, where the image pyramid search parameters include: the number of search layers, the search range, and the search accuracy.

[0007] In a possible implementation, the shared calculation unit is configured to perform the optical flow calculation task and the motion estimation task in a time-division multiplexing manner.

[0008] In a possible implementation, the device further includes: a data reading module and a shared storage module; the data reading module is configured to read the video to be processed from the shared storage module; the downsampling unit is configured to send the image pyramid of each video frame to be processed in the video to be processed to the shared storage module for storage, wherein the image pyramid of each video frame to be processed stored in the shared storage module can be shared by the motion and optical flow fusion processing module in the optical flow calculation mode or the motion estimation mode.

[0009] In a possible implementation, the shared calculation unit is configured to send the optical flow vector of the video to be processed to the shared storage module for storage; the shared calculation unit is configured to read the optical flow vector of the video to be processed from the shared storage module, and use the optical flow vector of the video to be processed to perform a motion estimation task to obtain the motion vector of the video to be processed; the shared calculation unit is configured to send the motion vector of the video to be processed to the shared storage module for storage.

[0010] In a possible implementation, the device further includes: a video encoding module; the video encoding module is configured to perform video encoding on the video to be processed according to the motion vector of the video to be processed to obtain an encoded video, and send the encoded video to the shared storage module for storage.

[0011] In a possible implementation, the device further includes: a frame interpolation module; the frame interpolation module is configured to perform frame interpolation on the video to be processed according to the optical flow vector of the video to be processed to obtain an interpolated video frame, and send the interpolated video frame to the shared storage module for storage.

[0012] According to another aspect of the present disclosure, a video processing method is provided. The method is applied to a video processing device, and the video processing device includes: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module includes: a downsampling unit and a shared calculation unit; the method includes: controlling the downsampling unit to perform downsampling processing on the video to be processed to obtain the image pyramid of each video frame to be processed in the video to be processed; when the motion and optical flow fusion processing module is in the optical flow calculation mode, controlling the shared calculation unit to perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain the optical flow vector of the video to be processed; when the motion and optical flow fusion processing module is in the motion estimation mode, controlling the shared calculation unit to perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain the motion vector of the video to be processed.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory, where the processor executes the computer program to implement the steps of the above method.

[0014] According to another aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0015] According to another aspect of the present disclosure, there is provided a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0016] According to an embodiment of the present disclosure, a video processing device including a motion and optical flow fusion processing module is designed at the hardware level. The motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit; the downsampling unit is configured to perform downsampling processing on a video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed; the shared computing unit is configured to, when the motion and optical flow fusion processing module is in the optical flow calculation mode, perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain an optical flow vector of the video to be processed; the shared computing unit is configured to, when the motion and optical flow fusion processing module is in the motion estimation mode, perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain a motion vector of the video to be processed. The optical flow calculation and the motion estimation share the image pyramid output by the same downsampling unit and the same shared computing unit, which can reduce redundant hardware modules, thereby effectively improving the hardware utilization rate and reducing the area and power consumption of the chip.

[0017] According to the following detailed description of exemplary embodiments with reference to the accompanying drawings, other features and aspects of the present disclosure will become apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure together with the specification and are used to explain the principles of the present disclosure.

[0019] Figure 1 A block diagram showing a video processing device according to an embodiment of the present disclosure.

[0020] Figure 2 A schematic diagram showing a video processing device according to an embodiment of the present disclosure.

[0021] Figure 3A schematic diagram showing an image pyramid obtained based on a downsampling unit according to an embodiment of the present disclosure.

[0022] Figure 4 A flowchart showing a video processing method according to an embodiment of the present disclosure.

[0023] Figure 5 A block diagram showing an electronic device according to an embodiment of the present disclosure. Detailed implementation manners

[0024] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0025] As used herein, the terms "including", "comprising", "having", or variations thereof are open-ended and include one or more stated features, wholes, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, wholes, elements, steps, components, functions, or groups thereof.

[0026] When an element is referred to as being "connected", "coupled", "responsive", or variations thereof to another element, it can be directly connected, coupled, or responsive to the other element, or intervening elements may be present.

[0027] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Thus, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0028] The term "exemplary" as used herein means "serving as an example, embodiment, or illustration". Any embodiment described herein as "exemplary" does not have to be construed as being superior to or better than other embodiments.

[0029] In addition, for a better illustration of the present disclosure, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can be implemented without some of these specific details. In some instances, methods, means, elements, and circuits well-known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0030] Both motion estimation and optical flow calculation essentially involve the analysis of object motion in images. Motion estimation mainly aims to find the motion relationship between the current frame and the reference frame in inter-frame prediction of video coding, and determines the motion vector by calculating the displacement between matching blocks. Optical flow calculation estimates the motion speed and direction of pixel points in the image, which is also based on the displacement change of pixel points between adjacent images.

[0031] In the video coding scenario, motion estimation and optical flow calculation are two completely independent processes. Motion estimation is performed during video compression coding based on minimum residual estimation for motion search, which is exclusive to the video encoder. Optical flow calculation is based on motion search for the best-matching pixels, usually calculated using a general module, with relatively high search accuracy but longer time consumption.

[0032] The motion estimation and optical flow calculation in the prior art have at least the following problems. 1) Duplication and waste of hardware resources: In the video coding scenario, building a complete set of downsampling units (e.g., image pyramid generation units) for the motion estimation module and the optical flow calculation module respectively will occupy a large amount of chip area. In addition, the motion estimation module and the optical flow calculation module may both require some arithmetic logic units (ALUs) of the same type, such as adders, multipliers, etc. for calculating similarity metrics (e.g., sum of absolute differences, mean square error, etc.). The independent design of the two modules will prevent these ALUs from being effectively shared, reducing the utilization rate of hardware resources. 2) Increase in chip area and power consumption: Due to the redundant hardware design, the chip area increases, which also means more transistors are needed, resulting in more electrical energy consumption during operation. 3) Problem of data inconsistency: The independent motion estimation module and optical flow calculation module may adopt different image preprocessing methods when processing data, resulting in differences in the details of the data processed by the two modules. Such differences may cause problems in subsequent video processing. For example, when enhancing video quality based on the results of motion estimation and optical flow calculation, due to data inconsistency, phenomena such as image incoherence or artifacts may occur.

[0033] To solve the above problems, the embodiments of the present disclosure provide a video processing device that combines the optical flow calculation process and the motion estimation process, reduces redundant hardware modules, effectively improves hardware utilization, and reduces the area and power consumption of the chip. The video processing device provided by the embodiments of the present disclosure will be described in detail below.

[0034] Figure 1 The block diagram of a video processing device according to an embodiment of the present disclosure is shown. As Figure 1As shown in the figure, the video processing device includes: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module includes: a downsampling unit, a shared computing unit; the downsampling unit is used to perform downsampling processing on the video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed; the shared computing unit is used to, when the motion and optical flow fusion processing module is in the optical flow calculation mode, perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain an optical flow vector of the video to be processed; the shared computing unit is used to, when the motion and optical flow fusion processing module is in the motion estimation mode, perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain a motion vector of the video to be processed.

[0035] According to an embodiment of the present disclosure, a video processing device including a motion and optical flow fusion processing module is designed at the hardware level. The motion and optical flow fusion processing module includes: a downsampling unit, a shared computing unit; the optical flow calculation and motion estimation share the image pyramid output by the same downsampling unit and the same shared computing unit, which can reduce duplicate hardware modules, thereby effectively improving the hardware utilization rate and reducing the area and power consumption of the chip.

[0036] In a possible implementation manner, the video processing device further includes: a data reading module, a shared storage module; the data reading module is used to read the video to be processed from the shared storage module.

[0037] The video to be processed is the original image data stored in the shared storage module that needs to perform video processing (for example, video encoding, video frame interpolation).

[0038] The data reading module reads the video to be processed from the shared storage module, and then sends the video to be processed to the motion and optical flow fusion processing module to perform subsequent video processing.

[0039] In an example, the shared storage module may be a Double DataRate (DDR) synchronous dynamic random access memory, and may also be set to other forms of storage modules according to the actual application scenario. The present disclosure does not make specific limitations on this.

[0040] In an example, the data reading module reads the video to be processed from the shared storage module and sends the read video to be processed to the motion and optical flow fusion processing module by means of Direct Memory Access (DMA).

[0041] Figure 2 A schematic diagram of a video processing device according to an embodiment of the present disclosure is shown. As Figure 2As shown, the data reading module reads the video to be processed from the DDR through the DMA method and sends the video to be processed to the motion and optical flow fusion processing module.

[0042] The downsampling unit included in the motion and optical flow fusion processing module performs downsampling on the video to be processed to obtain the image pyramid of each video frame to be processed in the video to be processed.

[0043] The image pyramid is a structure that interprets an image at multiple resolutions. By sampling pixels at multiple scales on the original image, multiple images with different resolutions are generated. These images are arranged in a pyramid shape, with the bottom being the image with the highest resolution level (e.g., the original image), and above it are a series of images with gradually decreasing pixel (size).

[0044] In one example, for each video frame to be processed in the video to be processed, the downsampling unit performs multi-scale downsampling on the video frame to be processed to obtain the image pyramid of the processed video frame.

[0045] In one example, the image pyramid of each video frame to be processed includes: N layers of images with different resolutions. Among them, the specific value of N and the resolution values of each layer of images can be flexibly set according to the actual application scenario, and the present disclosure does not make specific limitations on this.

[0046] Figure 3 The schematic diagram showing the image pyramid obtained based on the downsampling unit according to the embodiments of the present disclosure is as follows. As Figure 3 shown in (a) of, the downsampling unit includes processing elements (PEs) PE0 - PE4. The PEs PE0 - PE4 are used to sequentially perform downsampling on the input video frame to be processed to obtain the image pyramid diagram of the video frame to be processed, as Figure 3 shown in (b) of, including: layers 0 - 4. As Figure 3 shown, in the image pyramid, the upper layer image is obtained by downsampling the lower layer image.

[0047] In a possible implementation manner, the downsampling unit is used to send the image pyramid of each video frame to be processed in the video to be processed to the shared storage module for storage. Among them, the image pyramid of each video frame to be processed stored in the shared storage module can be shared by the motion and optical flow fusion processing module in the optical flow calculation mode or the motion estimation mode.

[0048] The downsampling unit obtains the image pyramid of each to-be-processed video frame in the to-be-processed video, and sends the image pyramid of each to-be-processed video frame in the to-be-processed video to the shared storage module for storage, so as to utilize the shared storage module to realize the sharing of the image pyramid of each to-be-processed video frame in the to-be-processed video by the motion and optical flow fusion processing module in the optical flow calculation mode or the motion estimation mode, so that when the subsequent shared calculation unit executes the optical flow calculation task or the motion estimation task, it can be called from the shared storage module.

[0049] During the video processing, both motion estimation and optical flow calculation rely on the multi-resolution image information provided by the image pyramid generated by the downsampling unit. Sharing the downsampling unit for motion estimation and optical flow calculation can ensure that motion estimation and optical flow calculation use exactly the same image pyramid data. Since the data is generated by the same downsampling unit, the data generation method, parameter settings, etc. are all consistent. Such a processing method effectively solves the data consistency problem in the video quality enhancement scenario, and can avoid the problem of image incoherence or artifacts caused by data differences due to the use of data generated by different methods in the motion estimation and optical flow calculation in the video quality enhancement scenario.

[0050] As Figure 2 shown, the downsampling unit sends the image pyramid to the DDR for storage.

[0051] In a possible implementation manner, the downsampling unit includes: a configuration subunit; the configuration subunit is used to configure the image pyramid search parameters, where the image pyramid search parameters include: the number of search layers, the search range, and the search accuracy.

[0052] By configuring the image pyramid search parameters, the search for multiple layers of images in the image pyramid can be completed. When searching, start from the first layer, that is, the top layer. The calculation vector obtained for each layer is used as a reference for the next layer. Based on this calculation vector, the search accuracy is improved for the next layer, and the search accuracy is sequentially improved until the nth layer, that is, the bottom layer. In addition, the search range of the current image block in the current to-be-processed video frame on the previous adjacent video frame or the previous and next adjacent video frames is dynamically configurable.

[0053] In an example, the configurable number of search layers includes: 1-N. For example, it can be configured as 1, 3, 5, 7, etc. The number of search layers for the optical flow calculation task and the motion estimation task can be the same or different, and the present disclosure does not make specific limitations on this.

[0054] In one example, the configurable search ranges include: 1024×1024, 1024×512, 512×512, 256×256, 256×128, 128×128, etc. The configurable search range can also be configured with other specific range values according to the actual application scenario, and the present disclosure does not make specific limitations thereto. The search ranges for the optical flow calculation task and the motion estimation task can be the same or different, and the present disclosure does not make specific limitations thereto.

[0055] In one example, the configurable search precisions include: 1-pixel precision, 1 / 2-pixel precision, 1 / 4-pixel precision, etc. Among them, in the image pyramid, the search precision of the lower layer is greater than or equal to that of the upper layer. The configurable search precision can also be configured with other specific precision values according to the actual application scenario, and the present disclosure does not make specific limitations thereto. The search precisions for the optical flow calculation task and the motion estimation task can be the same or different, and the present disclosure does not make specific limitations thereto.

[0056] The optical flow calculation and the motion estimation share the image pyramid generated by the downsampling unit, and a shared calculation unit that can be shared by the optical flow calculation and the motion estimation is designed by using the commonality between the optical flow calculation and the motion estimation. The downsampling unit usually includes hardware resources such as complex sampling circuits and storage units. The shared calculation unit includes some general ALUs such as adders, multipliers, and shifters. Through sharing, the repeated hardware modules are effectively reduced, the hardware utilization rate is improved, and the area and power consumption of the chip can be reduced.

[0057] In one example, the motion and optical flow fusion processing module can switch between the optical flow calculation mode and the motion estimation mode.

[0058] By switching the motion and optical flow fusion processing module between the optical flow calculation mode and the motion estimation mode, the optical flow calculation task and the motion estimation task can reuse the shared calculation unit in the motion and optical flow fusion processing module.

[0059] In a possible implementation manner, the video processing device further includes: a microcontroller unit (MCU), a control and status register (CSR); the MCU is used to control the motion and optical flow fusion processing module to perform mode switching between the optical flow calculation mode and the motion estimation mode by configuring the CSR.

[0060] By configuring the CSR by the MCU, the motion and optical flow fusion processing module is controlled to perform mode switching between the optical flow calculation mode and the motion estimation mode, and the control of each relevant module during the execution of the optical flow calculation task and the motion estimation task is realized.

[0061] Such as Figure 2As shown, the MCU controls modules such as the data reading module, the motion and optical flow fusion processing module, the video encoding module, and the frame interpolation module by configuring the CSR.

[0062] In a possible implementation, a shared computing unit is used to execute the optical flow calculation task and the motion estimation task in a time-division multiplexing manner.

[0063] Since the shared computing unit can execute both the optical flow calculation task and the motion estimation task, the shared computing unit is set to execute the optical flow calculation task and the motion estimation task in a time-division multiplexing manner, thereby improving the task execution efficiency and resource utilization rate.

[0064] In a possible implementation, a shared computing unit is used to send the optical flow vectors of the video to be processed to the shared storage module for storage.

[0065] During the execution of the optical flow calculation task, the shared computing unit can send the optical flow vectors of the video to be processed to the shared storage module for storage for subsequent use when needed.

[0066] In a possible implementation, a shared computing unit is used to read the optical flow vectors of the video to be processed from the shared storage module, and use the optical flow vectors of the video to be processed to execute the motion estimation task to obtain the motion vectors of the video to be processed; the shared computing unit is used to send the motion vectors of the video to be processed to the shared storage module for storage.

[0067] During the execution of the motion estimation task, the shared computing unit can directly use the optical flow vectors of the video to be processed stored in the shared storage module for motion estimation, effectively improving the motion estimation efficiency and reducing the motion estimation time. In addition, the motion vectors of the video to be processed are sent to the shared storage module for storage for subsequent use when needed.

[0068] In a possible implementation, the video processing device further includes: a video encoding module; the video encoding module is used to perform video encoding on the video to be processed according to the motion vectors of the video to be processed to obtain the encoded video, and send the encoded video to the shared storage module for storage.

[0069] The motion and optical flow fusion processing module can send the motion vectors of the video to be processed obtained by executing the motion estimation task to the video encoding module, or the video encoding module reads the motion vectors of the video to be processed from the shared storage module. Furthermore, as Figure 2 shown, the video encoding module performs video encoding on the video to be processed according to the motion vectors of the video to be processed to obtain the encoded video, and sends the encoded video to the shared storage module for storage for subsequent use when needed.

[0070] In a possible implementation, the video processing device further includes: a frame interpolation module; the frame interpolation module is configured to perform frame interpolation on the video to be processed according to the optical flow vector of the video to be processed, obtain an interpolated video frame, and send the interpolated video frame to the shared storage module for storage.

[0071] The motion and optical flow fusion processing module may send the optical flow vector of the video to be processed obtained by performing the optical flow calculation task to the frame interpolation module, or the frame interpolation module reads the optical flow vector of the video to be processed from the shared storage module. Furthermore, as Figure 2 shown, the frame interpolation module performs frame interpolation on the video to be processed according to the optical flow vector of the video to be processed, obtains an interpolated video frame, and sends the interpolated video frame to the shared storage module for storage for subsequent call when needed.

[0072] According to an embodiment of the present disclosure, a video processing device including a motion and optical flow fusion processing module is designed at the hardware level. The motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit; the downsampling unit is configured to perform downsampling processing on the video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed; the shared computing unit is configured to, when the motion and optical flow fusion processing module is in the optical flow calculation mode, perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain the optical flow vector of the video to be processed; the shared computing unit is configured to, when the motion and optical flow fusion processing module is in the motion estimation mode, perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed to obtain the motion vector of the video to be processed. The optical flow calculation and motion estimation share the image pyramid output by the downsampling unit and share the same shared computing unit, which can reduce redundant hardware modules, thereby effectively improving the hardware utilization rate and reducing the area and power consumption of the chip.

[0073] Figure 4 The flowchart of a video processing method provided according to an embodiment of the present disclosure is shown. This method is applied to Figure 1 or Figure 2 the video processing device shown. The video processing device includes: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit. As Figure 4 shown, this method includes:

[0074] In step S41, control the downsampling unit to perform downsampling processing on the video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed.

[0075] In step S42, when the motion and optical flow fusion processing module is in the optical flow calculation mode, the shared computing unit is controlled to perform an optical flow calculation task based on the image pyramid of each to-be-processed video frame in the to-be-processed video, and an optical flow vector of the to-be-processed video is obtained.

[0076] In step S43, when the motion and optical flow fusion processing module is in the motion estimation mode, the shared computing unit is controlled to perform a motion estimation task based on the image pyramid of each to-be-processed video frame in the to-be-processed video, and a motion vector of the to-be-processed video is obtained.

[0077] Applying the above Figure 1 or Figure 2 The specific process of video processing performed by the video processing device shown can refer to the detailed description of the above Figures 1 to 3 shown related embodiments, which will not be elaborated here.

[0078] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be elaborated here.

[0079] The embodiments of the present disclosure further provide an electronic device, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the above method.

[0080] The embodiments of the present disclosure further provide a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0081] The embodiments of the present disclosure further provide a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0082] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Referring to Figure 5 , the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 5 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0083] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0084] In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, such as the memory 1932 including computer program instructions, and the above computer program instructions can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

[0085] A computer-readable storage medium may be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punched card or raised structures in grooves storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not construed as an instantaneous signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through an optical fiber cable), or electrical signals transmitted through wires.

[0086] The computer programs (or computer-readable program instructions) described herein can be downloaded to various computing / processing devices from a computer-readable storage medium or downloaded to an external computer or an external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0087] The computer programs (or computer program instructions) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0088] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises a manufacture including instructions that implement various aspects of the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0090] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other devices implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0091] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which comprises one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the boxes may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by special-purpose hardware-based systems that perform the specified functions or acts, or by combinations of special-purpose hardware and computer instructions.

[0092] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A video processing device, characterized in that: include: Motion and optical flow fusion processing module; The motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit; The downsampling unit is used to perform downsampling processing on the video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed; The shared computing unit is configured to, when the motion and optical flow fusion processing module is in an optical flow computing mode, perform an optical flow computing task based on the image pyramid of each to-be-processed video frame in the to-be-processed video determined by the downsampling unit to obtain an optical flow vector of the to-be-processed video; The shared computing unit is used to, when the motion and optical flow fusion processing module is in a motion estimation mode, perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed determined by the downsampling unit to obtain a motion vector of the video to be processed.

2. The device according to claim 1, characterized in that The device also includes: a micro control unit MCU and a control status register CSR; The MCU is used to control the motion and optical flow fusion processing module to switch between the optical flow calculation mode and the motion estimation mode by configuring the CSR.

3. The device according to claim 1, characterized in that The downsampling unit comprises: a configuration subunit; The configuration subunit is used to configure image pyramid search parameters, wherein the image pyramid search parameters include: search layers, search range, and search accuracy.

4. The device according to claim 1, characterized in that The shared computing unit is used to execute the optical flow computing task and the motion estimation task in a time-division multiplexing manner.

5. The device according to claim 1, characterized in that The device also includes: a data reading module and a shared storage module; The data reading module is used to read the video to be processed from the shared storage module; The downsampling unit is used to send the image pyramid of each to-be-processed video frame in the to-be-processed video to the shared storage module for storage, wherein the image pyramid of each to-be-processed video frame in the to-be-processed video stored in the shared storage module can be shared by the motion and optical flow fusion processing module in the optical flow calculation mode or the motion estimation mode.

6. The device according to claim 5, characterized in that The shared computing unit is used to send the optical flow vector of the video to be processed to the shared storage module for storage; The shared computing unit is used to read the optical flow vector of the video to be processed from the shared storage module, and use the optical flow vector of the video to be processed to perform a motion estimation task to obtain the motion vector of the video to be processed; The shared computing unit is used to send the motion vector of the video to be processed to the shared storage module for storage.

7. The device according to claim 5, characterized in that The device also includes: a video encoding module; The video encoding module is used to perform video encoding on the video to be processed according to the motion vector of the video to be processed to obtain an encoded video, and send the encoded video to the shared storage module for storage.

8. The device according to claim 5, characterized in that The device further comprises: a frame interpolation module; The frame interpolation module is used to perform frame interpolation on the video to be processed according to the optical flow vector of the video to be processed to obtain an interpolated video frame, and send the interpolated video frame to the shared storage module for storage.

9. A video processing method, characterized in that: The method is applied to a video processing device, the video processing device comprises: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module comprises: a downsampling unit and a shared calculation unit; the method comprises: Controlling the downsampling unit to perform downsampling processing on the video to be processed, and obtaining an image pyramid of each video frame to be processed in the video to be processed; When the motion and optical flow fusion processing module is in the optical flow calculation mode, controlling the shared calculation unit to perform the optical flow calculation task based on the image pyramid of each to-be-processed video frame in the to-be-processed video determined by the downsampling unit, so as to obtain the optical flow vector of the to-be-processed video; When the motion and optical flow fusion processing module is in the motion estimation mode, the shared computing unit is controlled to perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed determined by the downsampling unit to obtain a motion vector of the video to be processed.

10. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 9.

11. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 9 are implemented.

Images