Video processing device and method, electronic equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of computers, in particular to a video processing device and method, electronic equipment, a storage medium and a computer program product. The motion and optical flow fusion processing module comprises a downsampling unit used for performing downsampling processing on a to-be-processed video to obtain an image pyramid of each to-be-processed video frame in the to-be-processed video; the sharing calculation unit is used for executing an optical flow calculation task to obtain an optical flow vector of the to-be-processed video based on an image pyramid of each to-be-processed video frame in the to-be-processed video under the condition that the motion and optical flow fusion processing module is in an optical flow calculation mode; and the sharing calculation unit is used for executing a motion estimation task to obtain a motion vector of the to-be-processed video based on the image pyramid of each to-be-processed video frame in the to-be-processed video under the condition that the motion and optical flow fusion processing module is in the motion estimation mode. According to the embodiment of the invention, the hardware utilization rate can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing device, method, electronic device, storage medium, and computer program product. Background Art

[0002] In the video encoding scenario, motion estimation and optical flow calculation are two completely independent processes. Motion estimation is a motion search based on minimum residual estimation during video compression encoding, which is exclusive to video encoders. Optical flow calculation is a motion search based on the best matching pixel, which is usually calculated using a general module. The search has high accuracy and takes a long time. Summary of the invention

[0003] In view of this, the present disclosure proposes a video processing device, method, electronic device, storage medium and computer program product.

[0004] According to one aspect of the present disclosure, a video processing device is provided, comprising: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module comprises: a downsampling unit and a shared computing unit; the downsampling unit is used to perform downsampling processing on a video to be processed, and obtain an image pyramid of each video frame to be processed in the video to be processed; the shared computing unit is used to perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed when the motion and optical flow fusion processing module is in an optical flow calculation mode, and obtain an optical flow vector of the video to be processed; the shared computing unit is used to perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed when the motion and optical flow fusion processing module is in a motion estimation mode, and obtain a motion vector of the video to be processed.

[0005] In a possible implementation, the device further includes: a microcontroller unit MCU and a control status register CSR; the MCU is used to control the motion and optical flow fusion processing module to switch between the optical flow calculation mode and the motion estimation mode by configuring the CSR.

[0006] In a possible implementation, the downsampling unit includes: a configuration subunit; the configuration subunit is used to configure image pyramid search parameters, wherein the image pyramid search parameters include: search layers, search range, and search accuracy.

[0007] In a possible implementation, the shared computing unit is used to execute the optical flow computing task and the motion estimation task in a time-division multiplexing manner.

[0008] In a possible implementation, the device further includes: a data reading module and a shared storage module; the data reading module is used to read the video to be processed from the shared storage module; the downsampling unit is used to send the image pyramid of each video frame to be processed in the video to be processed to the shared storage module for storage, wherein the image pyramid of each video frame to be processed in the video to be processed stored in the shared storage module can be shared by the motion and optical flow fusion processing module in the optical flow calculation mode or the motion estimation mode.

[0009] In one possible implementation, the shared computing unit is used to send the optical flow vector of the video to be processed to the shared storage module for storage; the shared computing unit is used to read the optical flow vector of the video to be processed from the shared storage module, and use the optical flow vector of the video to be processed to perform a motion estimation task to obtain the motion vector of the video to be processed; the shared computing unit is used to send the motion vector of the video to be processed to the shared storage module for storage.

[0010] In a possible implementation, the device further includes: a video encoding module; the video encoding module is used to perform video encoding on the video to be processed according to the motion vector of the video to be processed to obtain the encoded video, and send the encoded video to the shared storage module for storage.

[0011] In a possible implementation, the device also includes: a frame interpolation module; the frame interpolation module is used to perform frame interpolation on the video to be processed according to the optical flow vector of the video to be processed to obtain an interpolated video frame, and send the interpolated video frame to the shared storage module for storage.

[0012] According to another aspect of the present disclosure, a video processing method is provided, the method is applied to a video processing device, the video processing device includes: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit; the method includes: controlling the downsampling unit to perform downsampling processing on a video to be processed, and obtaining an image pyramid of each video frame to be processed in the video to be processed; when the motion and optical flow fusion processing module is in an optical flow calculation mode, controlling the shared computing unit to perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed, and obtaining an optical flow vector of the video to be processed; when the motion and optical flow fusion processing module is in a motion estimation mode, controlling the shared computing unit to perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed, and obtaining a motion vector of the video to be processed.

[0013] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0014] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0015] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0016] According to the embodiment of the present disclosure, a video processing device including a motion and optical flow fusion processing module is designed at the hardware level. The motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit; the downsampling unit is used to downsample the video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed; the shared computing unit is used to perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed when the motion and optical flow fusion processing module is in an optical flow calculation mode to obtain an optical flow vector of the video to be processed; the shared computing unit is used to perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed when the motion and optical flow fusion processing module is in a motion estimation mode to obtain a motion vector of the video to be processed. The optical flow calculation and motion estimation reuse the image pyramid output by the same downsampling unit, and reuse the same shared computing unit, which can reduce repeated hardware modules, thereby effectively improving hardware utilization and reducing chip area and power consumption.

[0017] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0019] Figure 1 A block diagram of a video processing device according to an embodiment of the present disclosure is shown.

[0020] Figure 2 A schematic diagram of a video processing device according to an embodiment of the present disclosure is shown.

[0021] Figure 3A schematic diagram showing a method of obtaining an image pyramid based on a downsampling unit according to an embodiment of the present disclosure.

[0022] Figure 4 A flowchart of a video processing method according to an embodiment of the present disclosure is shown.

[0023] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0025] As used herein, the terms "comprises," "including," "having," or variations thereof are open ended and include one or more stated features, integers, elements, steps, parts, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, parts, functions, or groups thereof.

[0026] When an element is referred to as being "connected," "coupled," "responsive," or variations thereof, to another element, it may be directly connected, coupled, or responsive to the other element or intervening elements may be present.

[0027] Although the terms first, second, third, etc. may be used to describe various elements / operations in this article, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Therefore, without departing from the teachings of the present invention, the first element / operation in some embodiments may be referred to as the second element / operation in other embodiments.

[0028] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0029] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0030] Motion estimation and optical flow calculation both essentially involve the analysis of the motion of objects in an image. Motion estimation mainly searches for the motion relationship between the current frame and the reference frame in inter-frame prediction of video coding, and determines the motion vector by calculating the displacement between matching blocks. Optical flow calculation estimates the speed and direction of the movement of pixels in an image, which is also based on the displacement change of pixels between adjacent images.

[0031] In the video encoding scenario, motion estimation and optical flow calculation are two completely independent processes. Motion estimation is a motion search based on minimum residual estimation during video compression encoding, which is exclusive to video encoders. Optical flow calculation is a motion search based on the best matching pixel, which is usually calculated using a general module. The search has high accuracy and takes a long time.

[0032] There are at least the following problems with motion estimation and optical flow calculation in the prior art. 1) Duplication and waste of hardware resources: In the video encoding scenario, building a complete set of downsampling units (for example, image pyramid generation units) for the motion estimation module and the optical flow calculation module respectively will occupy a large amount of chip area. In addition, the motion estimation module and the optical flow calculation module may both require some of the same type of arithmetic and logic units (ALUs), such as adders, multipliers, etc., for calculating similarity metrics (for example, the sum of absolute differences, mean square errors, etc.). The independent design of the two modules will prevent these ALUs from being effectively shared, reducing the utilization of hardware resources. 2) Increased chip area and power consumption: Due to the repeated design of hardware, the increase in chip area also means that more transistors are required, which will result in more power consumption during operation. 3) Problems causing data inconsistency: The independent motion estimation module and optical flow calculation module may use different image preprocessing methods when processing data, resulting in differences in details between the data processed by the two modules. This difference may cause problems in the subsequent video processing process. For example, when enhancing the video quality based on the results of motion estimation and optical flow calculation, image incoherence or artifacts may occur due to data inconsistency.

[0033] In order to solve the above problems, the embodiment of the present disclosure provides a video processing device, which combines the optical flow calculation process and the motion estimation process, reduces repeated hardware modules, effectively improves hardware utilization, and reduces chip area and power consumption. The video processing device provided by the embodiment of the present disclosure is described in detail below.

[0034] Figure 1 FIG. 2 is a block diagram of a video processing device according to an embodiment of the present disclosure. Figure 1As shown, the video processing device includes: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit; the downsampling unit is used to downsample the video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed; the shared computing unit is used to perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed when the motion and optical flow fusion processing module is in an optical flow calculation mode to obtain an optical flow vector of the video to be processed; the shared computing unit is used to perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed when the motion and optical flow fusion processing module is in a motion estimation mode to obtain a motion vector of the video to be processed.

[0035] According to an embodiment of the present disclosure, a video processing device including a motion and optical flow fusion processing module is designed at the hardware level. The motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit. Optical flow calculation and motion estimation reuse the image pyramid output by the same downsampling unit, and reuse the same shared computing unit, which can reduce duplicate hardware modules, thereby effectively improving hardware utilization and reducing chip area and power consumption.

[0036] In a possible implementation, the video processing device further includes: a data reading module and a shared storage module; the data reading module is used to read the video to be processed from the shared storage module.

[0037] The video to be processed is raw image data stored in the shared storage module and needs to be processed (eg, video encoding, video frame insertion).

[0038] The data reading module reads the video to be processed from the shared storage module, and then sends the video to be processed to the motion and optical flow fusion processing module to perform subsequent video processing.

[0039] In one example, the shared storage module may be a double data rate synchronous dynamic random access memory (Double Data Rate, DDR), and may also be configured as other forms of storage modules according to actual application scenarios, which is not specifically limited in the present disclosure.

[0040] In one example, the data reading module reads the video to be processed from the shared storage module through direct memory access (DMA), and sends the read video to be processed to the motion and optical flow fusion processing module.

[0041] Figure 2 FIG. 2 is a schematic diagram showing a video processing device according to an embodiment of the present disclosure. Figure 2As shown, the data reading module reads the video to be processed from the DDR through DMA, and sends the video to be processed to the motion and optical flow fusion processing module.

[0042] The downsampling unit included in the motion and optical flow fusion processing module performs downsampling processing on the video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed.

[0043] An image pyramid is a structure that interprets an image at multiple resolutions. It generates multiple images of different resolutions by sampling the original image at multiple scales. These images are arranged in a pyramid shape, with the image with the highest resolution (e.g., the original image) at the bottom and a series of images with gradually decreasing pixels (size) at the top.

[0044] In one example, for each to-be-processed video frame in the to-be-processed video, the downsampling unit performs a multi-scale downsampling process on the to-be-processed video frame to obtain an image pyramid of the processed video frame.

[0045] In one example, the image pyramid of each video frame to be processed includes: N layers of images with different resolutions, wherein the specific value of N and the resolution value of each layer of images can be flexibly set according to the actual application scenario, and the present disclosure does not make specific limitations on this.

[0046] Figure 3 FIG. 2 is a schematic diagram showing a method of obtaining an image pyramid based on a downsampling unit according to an embodiment of the present disclosure. Figure 3 As shown in (a) in FIG. 1 , the downsampling unit includes processing elements (PE) PE0-PE4. PE0-PE4 are used to sequentially perform downsampling processing on the input video frame to be processed to obtain an image pyramid diagram of the video frame to be processed, as shown in FIG. Figure 3 As shown in (b) in the figure, it includes layers 0-4. Figure 3 As shown, in the image pyramid, the upper layer image is obtained by downsampling the lower layer image.

[0047] In a possible implementation, the downsampling unit is used to send the image pyramid of each video frame to be processed in the video to be processed to the shared storage module for storage, wherein the image pyramid of each video frame to be processed in the video to be processed stored in the shared storage module can be shared by the motion and optical flow fusion processing module in the optical flow calculation mode or the motion estimation mode.

[0048] The downsampling unit obtains the image pyramid of each video frame to be processed in the video to be processed, and sends the image pyramid of each video frame to be processed in the video to be processed to the shared storage module for storage, thereby using the shared storage module to achieve the sharing of the image pyramid of each video frame to be processed in the video to be processed by the motion and optical flow fusion processing module in the optical flow calculation mode or the motion estimation mode, so that when the subsequent shared calculation unit performs the optical flow calculation task or the motion estimation task, it can be called from the shared storage module.

[0049] In the video processing process, motion estimation and optical flow calculation both rely on the multi-resolution image information provided by the image pyramid generated by the downsampling unit. Motion estimation and optical flow calculation share the downsampling unit, which ensures that motion estimation and optical flow calculation use exactly the same image pyramid data. Since the data is generated by the same downsampling unit, the data generation method and parameter settings are consistent. This processing method effectively solves the data consistency problem in the video quality enhancement scenario, and can avoid the problem of image incoherence or artifacts caused by data differences between motion estimation and optical flow calculation due to the use of different methods to generate data in the video quality enhancement scenario.

[0050] like Figure 2 As shown, the downsampling unit sends the image pyramid to DDR for storage.

[0051] In a possible implementation, the downsampling unit includes: a configuration subunit; the configuration subunit is used to configure image pyramid search parameters, wherein the image pyramid search parameters include: search layers, search range, and search accuracy.

[0052] By configuring the image pyramid search parameters, you can complete the search for multiple layers of images in the image pyramid. When searching, start from the first layer, which is the top layer. The calculation vector obtained in each layer is used as a reference for the next layer. In the next layer, the search accuracy is improved based on this calculation vector, and the search accuracy is improved in sequence until the nth layer, which is the bottom layer. In addition, the search range of the current image block in the current video frame to be processed on the previous adjacent video frame or the two adjacent video frames before and after can be dynamically configured.

[0053] In one example, the configurable number of search layers includes: 1-N. For example, it can be configured as 1, 3, 5, 7, etc. The number of search layers for the optical flow calculation task and the motion estimation task can be the same or different, and this disclosure does not specifically limit this.

[0054] In one example, the configurable search range includes: 1024×1024, 1024×512, 512×512, 256×256, 256×128, 128×128, etc. The configurable search range can also be configured with other specific range values ​​according to the actual application scenario, and the present disclosure does not make specific limitations on this. The search ranges of the optical flow calculation task and the motion estimation task can be the same or different, and the present disclosure does not make specific limitations on this.

[0055] In one example, the configurable search precision includes: 1 pixel precision, 1 / 2 pixel precision, 1 / 4 pixel precision, etc. Among them, in the image pyramid, the search precision of the lower layer is greater than or equal to the search precision of the upper layer. The configurable search precision can also be configured with other specific precision values ​​according to the actual application scenario, and the present disclosure does not make specific limitations on this. The search precision of the optical flow calculation task and the motion estimation task can be the same or different, and the present disclosure does not make specific limitations on this.

[0056] Optical flow calculation and motion estimation share the image pyramid generated by the downsampling unit, and utilizing the commonalities between optical flow calculation and motion estimation, a shared computing unit that can be shared by optical flow calculation and motion estimation is designed. The downsampling unit usually includes complex sampling circuits, storage units and other hardware resources, and the shared computing unit includes some general ALUs such as adders, multipliers, and shifters. Through sharing, repeated hardware modules are effectively reduced, hardware utilization is improved, and the chip area and power consumption can be reduced.

[0057] In one example, the motion and optical flow fusion processing module can switch between an optical flow calculation mode and a motion estimation mode.

[0058] By switching the motion and optical flow fusion processing module between the optical flow calculation mode and the motion estimation mode, the optical flow calculation task and the motion estimation task can reuse the shared calculation unit in the motion and optical flow fusion processing module.

[0059] In a possible implementation, the video processing device also includes: a microcontroller unit (MCU) and a control status register (CSR); the MCU is used to control the motion and optical flow fusion processing module to switch between the optical flow calculation mode and the motion estimation mode by configuring the CSR.

[0060] By configuring CSR through MCU, the motion and optical flow fusion processing module is controlled to switch between the optical flow calculation mode and the motion estimation mode, and the control of various related modules during the execution of the optical flow calculation task and the motion estimation task is realized.

[0061] like Figure 2As shown, the MCU controls the data reading module, motion and optical flow fusion processing module, video encoding module, frame interpolation module, etc. by configuring the CSR.

[0062] In a possible implementation, the shared computing unit is used to perform the optical flow computing task and the motion estimation task in a time-division multiplexing manner.

[0063] Since the shared computing unit can perform both optical flow calculation tasks and motion estimation tasks, setting the shared computing unit to perform optical flow calculation tasks and motion estimation tasks in a time-sharing multiplexing manner can improve task execution efficiency and resource utilization.

[0064] In a possible implementation, the shared computing unit is used to send the optical flow vector of the video to be processed to the shared storage module for storage.

[0065] When executing the optical flow calculation task, the shared computing unit can send the optical flow vector of the video to be processed to the shared storage module for storage so as to be called later when needed.

[0066] In one possible implementation, the shared computing unit is used to read the optical flow vector of the video to be processed from the shared storage module, and use the optical flow vector of the video to be processed to perform a motion estimation task to obtain the motion vector of the video to be processed; the shared computing unit is used to send the motion vector of the video to be processed to the shared storage module for storage.

[0067] In the process of performing the motion estimation task, the shared computing unit can directly use the optical flow vector of the video to be processed stored in the shared storage module to perform motion estimation, effectively improving the efficiency of motion estimation and reducing the time consumption of motion estimation. In addition, the motion vector of the video to be processed is sent to the shared storage module for storage, so as to be called when needed later.

[0068] In a possible implementation, the video processing device also includes: a video encoding module; a video encoding module, which is used to encode the video to be processed according to the motion vector of the video to be processed, obtain the encoded video, and send the encoded video to the shared storage module for storage.

[0069] The motion and optical flow fusion processing module can send the motion vector of the video to be processed obtained by performing the motion estimation task to the video encoding module, or the video encoding module reads the motion vector of the video to be processed from the shared storage module, and then, Figure 2 As shown, the video encoding module performs video encoding on the video to be processed according to the motion vector of the video to be processed to obtain the encoded video, and sends the encoded video to the shared storage module for storage so as to be called when needed later.

[0070] In a possible implementation, the video processing device also includes: a frame interpolation module; the frame interpolation module is used to perform frame interpolation on the video to be processed according to the optical flow vector of the video to be processed to obtain an interpolated video frame, and send the interpolated video frame to the shared storage module for storage.

[0071] The motion and optical flow fusion processing module can send the optical flow vector of the video to be processed obtained by executing the optical flow calculation task to the frame interpolation module, or the frame interpolation module reads the optical flow vector of the video to be processed from the shared storage module, and then, Figure 2 As shown, the frame interpolation module performs frame interpolation on the video to be processed according to the optical flow vector of the video to be processed to obtain an interpolated video frame, and sends the interpolated video frame to the shared storage module for storage so as to be called later when needed.

[0072] According to the embodiment of the present disclosure, a video processing device including a motion and optical flow fusion processing module is designed at the hardware level. The motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit; the downsampling unit is used to downsample the video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed; the shared computing unit is used to perform an optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed when the motion and optical flow fusion processing module is in an optical flow calculation mode to obtain an optical flow vector of the video to be processed; the shared computing unit is used to perform a motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed when the motion and optical flow fusion processing module is in a motion estimation mode to obtain a motion vector of the video to be processed. The optical flow calculation and motion estimation reuse the image pyramid output by the downsampling unit, and reuse the same shared computing unit, which can reduce repeated hardware modules, thereby effectively improving hardware utilization and reducing chip area and power consumption.

[0073] Figure 4 A flowchart of a video processing method provided according to an embodiment of the present disclosure is shown. The method is applied to Figure 1 or Figure 2 The video processing device shown in the figure includes: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit. Figure 4 As shown, the method includes:

[0074] In step S41, the downsampling unit is controlled to perform downsampling processing on the video to be processed, and an image pyramid of each video frame to be processed in the video to be processed is obtained.

[0075] In step S42, when the motion and optical flow fusion processing module is in the optical flow calculation mode, the shared computing unit is controlled to perform the optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed, so as to obtain the optical flow vector of the video to be processed.

[0076] In step S43, when the motion and optical flow fusion processing module is in the motion estimation mode, the shared computing unit is controlled to perform the motion estimation task based on the image pyramid of each video frame to be processed in the video to be processed, so as to obtain the motion vector of the video to be processed.

[0077] Apply the above Figure 1 or Figure 2 For the specific process of the video processing device shown in FIG. 10 , please refer to the above Figures 1 to 3 The detailed description of the related embodiments shown is not repeated here.

[0078] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0079] An embodiment of the present disclosure further provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0080] The embodiment of the present disclosure further provides a non-volatile computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0081] The embodiments of the present disclosure further provide a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the steps of the above method are implemented when the computer program is executed by a processor.

[0082] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 5 , the electronic device 1900 can be provided as a server or a terminal device. Figure 5 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0083] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0084] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0085] A computer-readable storage medium may be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0086] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0087] The computer program (or computer program instructions) used to perform the operation of the present disclosure may be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-dependent instruction, a microcode, a firmware instruction, a state setting data, or a source code or an object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0088] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0090] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0091] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0092] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A video processing device, characterized in that: include: Motion and optical flow fusion processing module; The motion and optical flow fusion processing module includes: a downsampling unit and a shared computing unit; The downsampling unit is used to perform downsampling processing on the video to be processed to obtain an image pyramid of each video frame to be processed in the video to be processed; The shared computing unit is configured to, when the motion and optical flow fusion processing module is in an optical flow computing mode, perform an optical flow computing task based on an image pyramid of each video frame to be processed in the video to be processed, so as to obtain an optical flow vector of the video to be processed; The shared computing unit is used to perform a motion estimation task based on an image pyramid of each video frame to be processed in the video to be processed, so as to obtain a motion vector of the video to be processed when the motion and optical flow fusion processing module is in a motion estimation mode.

2. The device according to claim 1, characterized in that The device also includes: a micro control unit MCU and a control status register CSR; The MCU is used to control the motion and optical flow fusion processing module to switch between the optical flow calculation mode and the motion estimation mode by configuring the CSR.

3. The device according to claim 1, characterized in that The downsampling unit comprises: a configuration subunit; The configuration subunit is used to configure image pyramid search parameters, wherein the image pyramid search parameters include: search layers, search range, and search accuracy.

4. The device according to claim 1, characterized in that The shared computing unit is used to execute the optical flow computing task and the motion estimation task in a time-division multiplexing manner.

5. The device according to claim 1, characterized in that The device also includes: a data reading module and a shared storage module; The data reading module is used to read the video to be processed from the shared storage module; The downsampling unit is used to send the image pyramid of each to-be-processed video frame in the to-be-processed video to the shared storage module for storage, wherein the image pyramid of each to-be-processed video frame in the to-be-processed video stored in the shared storage module can be shared by the motion and optical flow fusion processing module in the optical flow calculation mode or the motion estimation mode.

6. The device according to claim 5, characterized in that The shared computing unit is used to send the optical flow vector of the video to be processed to the shared storage module for storage; The shared computing unit is used to read the optical flow vector of the video to be processed from the shared storage module, and use the optical flow vector of the video to be processed to perform a motion estimation task to obtain the motion vector of the video to be processed; The shared computing unit is used to send the motion vector of the video to be processed to the shared storage module for storage.

7. The device according to claim 5, characterized in that The device also includes: a video encoding module; The video encoding module is used to perform video encoding on the video to be processed according to the motion vector of the video to be processed to obtain an encoded video, and send the encoded video to the shared storage module for storage.

8. The device according to claim 5, characterized in that The device further comprises: a frame interpolation module; The frame interpolation module is used to perform frame interpolation on the video to be processed according to the optical flow vector of the video to be processed to obtain an interpolated video frame, and send the interpolated video frame to the shared storage module for storage.

9. A video processing method, characterized in that: The method is applied to a video processing device, the video processing device comprising: a motion and optical flow fusion processing module; the motion and optical flow fusion processing module comprises: a downsampling unit and a shared calculation unit; the method comprises: Controlling the downsampling unit to perform downsampling processing on the video to be processed, and obtaining an image pyramid of each video frame to be processed in the video to be processed; When the motion and optical flow fusion processing module is in the optical flow calculation mode, controlling the shared calculation unit to perform the optical flow calculation task based on the image pyramid of each video frame to be processed in the video to be processed, so as to obtain the optical flow vector of the video to be processed; When the motion and optical flow fusion processing module is in the motion estimation mode, the shared computing unit is controlled to perform a motion estimation task based on an image pyramid of each video frame to be processed in the video to be processed, so as to obtain a motion vector of the video to be processed.

10. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 9.

11. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 9 are implemented.

12. A computer program product, comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 9 are implemented.

Images