A method for optimizing motion vector and related equipment

Abstract

The present application provides a method for optimizing motion vectors and related devices thereof, which can improve the accuracy of motion vectors of small objects. The method of the present application includes: obtaining first position information of a small object region in a current video frame, and second position information of the small object region in a reference video frame, the current video frame and the reference video frame are two adjacent video frames, and the small object region contains a target small object; obtaining a motion vector of the small object region of the current video frame according to the first position information and the second position information; obtaining multiple motion vectors of multiple first image blocks in the current video frame according to the current video frame and the reference video frame, wherein each first image block corresponds to a motion vector, and the region formed by the multiple first image blocks includes the small object region; determining a better motion vector between the motion vector of the target first image block and the motion vector of the small object region, and the target first image block is any one of the multiple first image blocks.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of multimedia technology, and in particular to a method for optimizing motion vectors and related devices thereof. Background Art

[0002] Motion estimation and motion compensation (MEMC) technology can insert one or more interpolation frames between two adjacent video frames in the video to be played to increase the video frame rate. Motion estimation is one of the key technologies in MEMC technology. Interpolation frames generated based on accurate motion estimation results can give users a smooth visual experience after the frame rate is increased. Interpolation frames generated based on inaccurate motion estimation results will affect the video viewing effect.

[0003] In the process of motion estimation, the video frame can be divided into multiple non-overlapping image blocks, and then the motion vector of the image block is calculated. It is worth noting that the accuracy of the motion vector is usually related to the scene content of the video image, for example, the scene of small object motion. In this scene, the selection of the image block size usually affects the accuracy of the motion vector. When the size of the image block is too large, the motion vector of the small object cannot fully converge during calculation, resulting in the motion vector of the small object being inaccurate. When the size of the image block is too small, the pixel information around the small object cannot be effectively used, which also results in the motion vector of the small object being inaccurate.

[0004] Therefore, how to improve the accuracy of motion vectors of small objects has become an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a method for optimizing a motion vector and related devices thereof, which can improve the accuracy of the motion vector of a small object.

[0006] A first aspect of an embodiment of the present application provides a method for optimizing a motion vector, the method comprising:

[0007] When the video frame rate needs to be increased, first obtain the first position information of the small object area in the current video frame and the second position information of the small object area in the reference video frame. The current video frame and the reference video frame are two adjacent video frames in the video, and the small object area is an area containing the target small object. For example, the target small object can be a small object such as a football or a basketball. Correspondingly, the small object area can be a small area containing a football, a small area containing a basketball, and so on.

[0008] Since the first position information of the small object area can indicate the position of the small object area in the current video frame, and the second position information of the small object area can indicate the position of the small object area in the reference video frame, the displacement of the small object area from the position in the current video frame to the position in the reference video frame can be determined based on the first position information and the second position information of the small object area, that is, the motion vector of the small object area in the current video frame is obtained based on the first position information and the second position information of the small object area.

[0009] The video frame image is divided into multiple image blocks, and then the motion vector of each image block between the current video frame and the reference video frame is calculated, and multiple first image blocks that can cover the small object area are selected. For example, a video frame image is divided into 64 image blocks, and the small object area is a football. The area occupied by the football includes 9 first image blocks, then 9 first image blocks containing the football are selected from the 64 image blocks of the video frame. The area formed by these 9 first image blocks covers the football, and in general, the area formed by these 9 first image blocks will be larger than the football.

[0010] Finally, the motion vector of each first image block is compared with the motion vector of the small object area to determine a better motion vector of the target small object. Specifically, the better motion vector is selected from the motion vector of the target first image block and the motion vector of the small object area as the final motion vector of the target first image block, and the target first image block is any first image block among the multiple first image blocks. After obtaining the final motion vector of each first image block, it is equivalent to obtaining a motion vector of the target small object with higher accuracy.

[0011] It can be seen from the above method that in the present application, the motion vector of the small object area of ​​the current video frame generated based on the position information of the small object area in the current video frame and the reference video frame is first obtained, and the motion vectors of multiple first image blocks of the current video frame are obtained based on the MEMC technology, wherein the area formed by the multiple first image blocks contains the small object area. Then, the motion vector of each first image block is compared with the motion vector of the small object area, and a better motion vector can be selected as the final motion vector of each first image block. Since the motion vector of the target small object is determined by combining the motion vector obtained by the traditional algorithm and the motion vector obtained based on the position information of the target small object, the accuracy of the estimated motion vector of the target small object can be improved.

[0012] In a possible implementation, obtaining first position information of a small object region in a current video frame and second position information of a small object region in a reference video frame includes: obtaining first position information of the small object region in the current video frame and second position information of the small object region in the reference video frame through a first neural network. As can be seen from the aforementioned implementation, the position information of the small object region in the current video frame and the reference video frame can be accurately obtained through the first neural network.

[0013] In a possible implementation, determining a better motion vector between the motion vector of the target first image block and the motion vector of the small object region includes: determining a better motion vector between the motion vector of the target first image block and the motion vector of the small object region according to the degree of overlap between the target first image block and the small object region. As can be seen from the aforementioned implementation, when comparing the motion vector of the target first image block with the motion vector of the small object region, the degree of overlap between the target first image block and the small object region may be considered, thereby accurately selecting a better motion vector as the motion vector finally determined for the target first image block.

[0014] In a possible implementation, according to the degree of overlap between the target first image block and the small object region, determining a better motion vector among the motion vector of the target first image block and the motion vector of the small object region includes: for the target first image block, according to the motion vector of the small object region and the motion vector of the target first image block, respectively, obtaining the second image block corresponding to the target first image block and the third image block corresponding to the target first image block in the reference frame. Then, based on the degree of overlap between the target first image block and the small object region, the first matching degree between the target first image block and the second image block can be obtained. Then, the second matching degree between the target first image block and the third image block is obtained. Finally, the first matching degree and the second matching degree are compared. If the first matching degree and the second matching degree are not equal, then among the motion vectors of the target first image block and the motion vectors of the small object area, the motion vector corresponding to the larger matching degree is used as the motion vector finally determined by the target first image block. If the first matching degree and the second matching degree are equal, then among the motion vectors of the target first image block and the motion vectors of the small object area, the earlier motion vector is used as the motion vector finally determined by the target first image block or a motion vector is randomly selected as the motion vector finally determined by the target first image block.

[0015] In a possible implementation, the method further includes: obtaining motion vectors of the remaining areas of the current video frame except for the multiple first image blocks according to the current video frame and the reference video frame, wherein the remaining areas include multiple fourth image blocks, and the motion vectors of the remaining areas include multiple motion vectors of the multiple fourth image blocks, and each fourth image block corresponds to a motion vector. It can be seen from the aforementioned implementation that when the motion vector of each first image block in the current video frame is obtained based on the MEMC technology, the motion vectors of the remaining areas of the current video frame except for the multiple first image blocks (i.e., the motion vectors of each fourth image block) can also be obtained. Therefore, after obtaining the finalized motion vector of each first image block in the current video frame, combined with the motion vectors of the remaining areas in the current video frame, it is equivalent to obtaining the finalized motion vector of each image block in the current video frame, making the solution more comprehensive.

[0016] In a possible implementation, after determining a better motion vector between the motion vector of the target first image block and the motion vector of the small object area, the method further includes: generating an interpolation frame for inserting between the current video frame and the reference video frame based on the better motion vector and the motion vectors of the remaining areas to obtain a video with an increased frame rate. From the aforementioned implementation, it can be seen that after obtaining the final determined motion vector of each image block in the current video frame, an interpolation frame can be generated based on the final determined motion vectors of all image blocks in the current video frame to obtain a video with an increased frame rate, thereby completing the conversion of the video frame rate.

[0017] In a possible implementation, the method further includes: first obtaining the third position information of the small object area in the interpolation frame according to the preset interpolation phase, the first position information and the second position information. Then, the third position information is inserted between the first position information and the second position information to obtain the trajectory information of the small object. Next, the trajectory information of the small object is corrected according to the motion vector of the remaining area to obtain the corrected trajectory information of the small object. Finally, the small object in the video with the increased frame rate is highlighted according to the corrected trajectory information of the small object. It can be seen from the aforementioned implementation that the trajectory information of the small object can be estimated based on the first position information and the second position information, and the trajectory information of the small object can be corrected. Based on the corrected trajectory information of the small object, the small object can be highlighted in the video, thereby improving the accuracy of the small object trajectory drawing and optimizing the user viewing effect.

[0018] In a possible implementation, after the trajectory information of the small object is corrected according to the motion vector of the remaining area and the corrected trajectory information of the small object is obtained, the method further includes: first, according to the corrected trajectory information of the small object, cropping the next video frame of the current video frame or the next video frame of the reference video frame to obtain the cropped next video frame, and then inputting the cropped next video frame into the first neural network to obtain the fourth position information of the small object area in the next video frame. From the aforementioned implementation, it can be seen that after obtaining the corrected trajectory information of the small object, the next video frame can be cropped based on the trajectory information to reduce the number of pixels in the next video frame. Therefore, when the first neural network processes the cropped next video frame, the workload of the first neural network can be reduced, thereby reducing the power consumption of the entire solution.

[0019] In one possible implementation, if there are multiple small object regions, before obtaining the third position information of the small object region in the interpolation frame according to the preset interpolation phase, the first position information and the second position information, the method also includes: performing time domain clustering on the first position information of the multiple small object regions in the current video frame and the second position information of the multiple small object regions in the reference video frame to determine the first position information of each small object region in the current video frame and the second position information of each small object region in the reference video frame. From the aforementioned implementation, it can be seen that if there are multiple small object regions, the first position information of the multiple small object regions in the current video frame and the second position information of the multiple small object regions in the reference video frame can be matched one-to-one through time domain clustering, thereby determining the first position information of each small object region in the current video frame and the second position information of each small object region in the reference video frame. Therefore, after obtaining the position information of each small object area, the trajectory information of each small object can be estimated and corrected. Based on the corrected trajectory information of each small object, each small object can be highlighted in the video, thereby improving the accuracy of small object trajectory drawing and optimizing the user viewing effect.

[0020] In one possible implementation, before obtaining the first position information of the small object region in the current video frame and the second position information of the small object region in the reference video frame through the first neural network, the method further includes: if it is determined through the second neural network that the current video frame and the reference video frame include the small object region, then sending the first identifier and the second identifier to the first neural network through the second neural network, the first identifier being used to indicate that the current video frame includes the small object region, and the second identifier being used to indicate that the reference video frame includes the small object region. From the aforementioned implementation, it can be seen that after the second neural network determines that the current video frame and the reference video frame include the small object region, it can input the first identifier and the second identifier to the first neural network, so that after the first neural network receives the first identifier, it obtains the first position information of the small object region in the current video frame, and after the first neural network receives the second identifier, it obtains the second position information of the small object region in the reference video frame.

[0021] In a possible implementation, before obtaining the first position information of the small object region in the current video frame and the second position information of the small object region in the reference video frame through the first neural network, the method further includes: if the second neural network identifies that the scene presented by the current video frame and the scene presented by the reference video frame are both target scenes, then the second neural network sends a first identifier and a second identifier to the first neural network, the first identifier is used to indicate that the scene presented by the current video frame is the target scene, and the second identifier is used to indicate that the scene presented by the reference video frame is the target scene, and the target scene is a scene containing a target small object, such as a scene of a football player kicking a football, etc. From the aforementioned implementation, it can be known that the second neural network can input the first identifier and the second identifier to the first neural network after determining that the scene presented by the current video frame and the scene presented by the reference video frame are both target scenes, so that the first neural network obtains the first position information of the small object region in the current video frame after receiving the first identifier, and obtains the second position information of the small object region in the reference video frame after receiving the second identifier. It should be understood that if the second neural network recognizes that the scene corresponding to the current video frame and the reference video frame is not the target scene, the current video frame and the reference video frame will not be sent to the first neural network.

[0022] In one possible implementation, the first position information of the small object area includes the first center coordinates of the small object area in the current video frame and the first size of the small object area, and the second position information of the small object area includes the second center coordinates of the small object area in the reference video frame and the second size of the small object area.

[0023] In one possible implementation, the first neural network is a multilayer perceptron, a convolutional neural network, a recursive neural network or a recurrent neural network, and the second neural network is a multilayer perceptron, a convolutional neural network, a recursive neural network or a recurrent neural network, which improves the flexibility and selectivity of the solution.

[0024] A second aspect of an embodiment of the present application provides a device for optimizing a motion vector, the device comprising:

[0025] The first acquisition module is used to obtain first position information of the small object area in the current video frame and second position information of the small object area in the reference video frame. The current video frame and the reference video frame are two adjacent video frames, and the small object area contains the target small object.

[0026] The second acquisition module is used to acquire the motion vector of the small object area of ​​the current video frame according to the first position information and the second position information.

[0027] The third acquisition module is used to acquire multiple motion vectors of multiple first image blocks in the current video frame according to the current video frame and the reference video frame, wherein each first image block corresponds to a motion vector, and the area formed by the multiple first image blocks includes a small object area.

[0028] The determination module is used to determine a better motion vector between a motion vector of a target first image block and a motion vector of a small object area, wherein the target first image block is any first image block among a plurality of first image blocks.

[0029] It can be seen from the above device that in the present application, the motion vector of the small object area of ​​the current video frame generated based on the position information of the small object area in the current video frame and the reference video frame is first obtained, and the motion vectors of multiple first image blocks of the current video frame are obtained based on the MEMC technology, wherein the area formed by the multiple first image blocks contains the small object area. Then, the motion vector of each first image block is compared with the motion vector of the small object area, and a better motion vector can be selected as the final motion vector of each first image block. Since the motion vector of the target small object is determined by combining the motion vector obtained by the traditional algorithm and the motion vector obtained based on the position information of the target small object, the accuracy of the estimated motion vector of the target small object can be improved.

[0030] In a possible implementation, the first acquisition module is further used to acquire first position information of the small object region in the current video frame and second position information of the small object region in the reference video frame through the first neural network.

[0031] In a possible implementation, the determination module is further configured to determine a better motion vector among the motion vector of the target first image block and the motion vector of the small object region according to the degree of overlap between the target first image block and the small object region.

[0032] In a possible implementation, the determination module is further used to: determine the second image block corresponding to the target first image block in the reference frame according to the motion vector of the small object area. Determine the third image block corresponding to the target first image block in the reference frame according to the motion vector of the target first image block. Obtain a first matching degree between the target first image block and the second image block according to the degree of overlap between the target first image block and the small object area. Obtain a second matching degree between the target first image block and the third image block. Compare the first matching degree and the second matching degree, and determine a better motion vector between the motion vector of the target first image block and the motion vector of the small object area.

[0033] In a possible implementation, the third acquisition module is also used to obtain the motion vector of the remaining area in the current video frame except the multiple first image blocks based on the current video frame and the reference video frame, the remaining area includes multiple fourth image blocks, and the motion vector of the remaining area includes multiple motion vectors of the multiple fourth image blocks, wherein each fourth image block corresponds to a motion vector.

[0034] In a possible implementation, the device further includes: a generation module, configured to generate an interpolation frame to be inserted between the current video frame and the reference video frame according to the better motion vector and the motion vectors of the remaining regions, so as to obtain a video with an increased frame rate.

[0035] In a possible implementation, the device further includes: a fourth acquisition module, used to acquire third position information of the small object region in the interpolation frame according to a preset interpolation phase, the first position information and the second position information. An insertion module, used to insert the third position information between the first position information and the second position information to obtain trajectory information of the small object. A correction module, used to correct the trajectory information of the small object according to the motion vector of the remaining region to obtain the corrected trajectory information of the small object. A display module, used to highlight the small object in the video with the increased frame rate according to the corrected trajectory information of the small object.

[0036] In a possible implementation, the device further includes: a cropping module, which is used to crop the next video frame of the current video frame or the next video frame of the reference video frame according to the corrected trajectory information of the small object to obtain the cropped next video frame. The cropping module is also used to input the cropped next video frame into the first neural network.

[0037] In one possible implementation, the device also includes: a clustering module, used to perform time-domain clustering on first position information of multiple small object regions in the current video frame and second position information of multiple small object regions in the reference video frame to determine the first position information of each small object region in the current video frame and the second position information in the reference video frame.

[0038] In one possible implementation, the device also includes: an analysis module, which is used to send a first identifier and a second identifier to the first neural network through the second neural network if it is determined through the second neural network that the current video frame and the reference video frame include a small object area, the first identifier being used to indicate that the current video frame includes a small object area, and the second identifier being used to indicate that the reference video frame includes a small object area.

[0039] The first acquisition module is also used to obtain first position information of the small object area in the current video frame through the first neural network after the first neural network receives the first identifier, and to obtain second position information of the small object area in the reference video frame through the second neural network after the first neural network receives the second identifier.

[0040] In one possible implementation, the device also includes: an analysis module, which is used to send a first identifier and a second identifier to the first neural network through the second neural network if the second neural network identifies that the scene presented by the current video frame and the scene presented by the reference video frame are both target scenes, the first identifier being used to indicate that the scene presented by the current video frame is the target scene, and the second identifier being used to indicate that the scene presented by the reference video frame is the target scene, and the target scene is a scene containing a small target object.

[0041] The first acquisition module is also used to obtain first position information of the small object area in the current video frame through the first neural network after the first neural network receives the first identifier, and to obtain second position information of the small object area in the reference video frame through the second neural network after the first neural network receives the second identifier.

[0042] In one possible implementation, the first position information of the small object area includes the first center coordinates of the small object area in the current video frame and the first size of the small object area, and the second position information of the small object area includes the second center coordinates of the small object area in the reference video frame and the second size of the small object area.

[0043] In one possible implementation, the first neural network is a multilayer perceptron, a convolutional neural network, a recursive neural network or a recurrent neural network, and the second neural network is a multilayer perceptron, a convolutional neural network, a recursive neural network or a recurrent neural network, which improves the flexibility and selectivity of the solution.

[0044] A third aspect of an embodiment of the present application provides a device for optimizing motion vectors, the device comprising: a processor and a transmission interface, the processor being configured to call program instructions stored in a memory to execute a method as described in the first aspect or any possible implementation of the first aspect.

[0045] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer or a processor, enable the computer or the processor to execute a method as described in the first aspect or any possible implementation of the first aspect.

[0046] A fifth aspect of an embodiment of the present application provides a computer program product comprising instructions, wherein the computer program product includes program instructions, and when the program instructions are executed on a computer or a processor, the computer or the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0047] A sixth aspect of the embodiments of the present application provides a system for improving a video frame rate, the system comprising an apparatus and a device for optimizing a motion vector, the apparatus being communicatively connected to the device, the apparatus comprising at least one memory and at least one communication interface; the processor, the memory and the communication interface are connected via a communication bus and communicate with each other;

[0048] at least one memory for storing code;

[0049] At least one processor is used to execute the aforementioned code to perform the method as described in any one of the first aspect, so as to obtain a video with an improved frame rate;

[0050] At least one communication interface is used to communicate with a device or a communication network to send the video with a higher frame rate to the device or the communication network.

[0051] In one possible implementation, the communication interface may be a software development kit or an application programming interface.

[0052] In an embodiment of the present application, a motion vector of a small object region of a current video frame generated based on the position information of the small object region in the current video frame and the reference video frame is first obtained, and the motion vectors of a plurality of first image blocks of the current video frame are obtained based on the MEMC technology, wherein the region formed by the plurality of first image blocks includes the small object region. Then, the motion vector of each first image block is compared with the motion vector of the small object region, and a better motion vector can be selected as the final determined motion vector of each first image block. Since the motion vector of the target small object is determined by combining the motion vector obtained by the traditional algorithm and the motion vector obtained based on the position information of the target small object, the accuracy of the estimated motion vector of the target small object can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0054] Figure 2 A hardware architecture diagram of a device for optimizing motion vectors provided in an embodiment of the present application;

[0055] Figure 3 A structural diagram of a system for improving video frame rate provided in an embodiment of the present application;

[0056] Figure 4 A schematic flow chart of a method for optimizing motion vectors provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of a small object area provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of a current video frame and a reference video frame provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of a current video frame provided in an embodiment of the present application;

[0060] Figure 8 Another schematic diagram of a current video frame and a reference video frame provided in an embodiment of the present application;

[0061] Fig. 9 A schematic diagram of the structure of a device for optimizing motion vectors provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0063] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0064] MEMC technology can insert one or more interpolation frames between two adjacent video frames in a video to increase the video frame rate. Motion estimation is one of the key technologies in MEMC technology. The accuracy of motion estimation results will affect the quality of interpolation frames, and thus affect the viewing effect of the video after the frame rate is increased.

[0065] Take the small object motion scene as an example, for example, the scene of football players playing football, the scene of basketball players playing basketball, the scene of trainers training with football, basketball, volleyball and other balls, etc. In such small object motion scenes, small objects such as football, basketball, volleyball, etc. can be regarded as small target objects in such scenes. In the motion estimation process, the video frame is usually divided into multiple non-overlapping image blocks, and then the motion vector of the image block is calculated. However, since the video frame contains small target objects, the size of the image block is often difficult to accurately grasp. When the size of the image block is too large or too small, it is impossible to obtain a more accurate motion vector of the small target object, thereby affecting the quality of the interpolated frame.

[0066] Artificial intelligence (AI) technology is a technical discipline that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence. AI technology achieves the best results by sensing the environment, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Using artificial intelligence for video processing is a common application of artificial intelligence. Therefore, in order to improve the accuracy of motion vectors of small objects, this application combines AI technology and MEMC technology to provide a method for optimizing motion vectors.

[0067] The method for optimizing motion vectors provided in the embodiments of the present application can be applicable to various terminal devices. Correspondingly, the device for optimizing motion vectors provided in the embodiments of the present application can be terminal products in various forms, such as smart phones, tablet computers, smart glasses, wearable devices, cameras and camcorders, etc. Figure 1 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. Figure 1 As shown, the terminal 100 may include an antenna system 110, a radio frequency (RF) circuit 120, a processor 130, a memory 140, a camera 150, an audio circuit 160, a display screen 170, one or more sensors 180 and a wireless transceiver 190, etc.

[0068] The antenna system 110 may be one or more antennas, or an antenna array composed of multiple antennas. The RF circuit 120 may include one or more analog RF transceivers, and the RF circuit 120 may also include one or more digital RF transceivers, and the RF circuit 120 is coupled to the antenna system 110. It should be understood that in various embodiments of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc. The RF circuit 120 can be used for various types of cellular wireless communications.

[0069] The processor 130 may include a communication processor, which may be used to control the RF circuit 120 to receive and send signals through the antenna system 110, and the signals may be voice signals, media signals or control signals. The processor 130 may include various general processing devices, such as a general central processing unit (CPU), a system on chip (SOC), a processor integrated on an SOC, a separate processor chip or controller, etc.; the processor 130 may also include a dedicated processing device, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a digital signal processor (DSP), a dedicated video or graphics processor, a graphics processing unit (GPU) and a neural-network processing unit (NPU). The processor 130 may be a processor group composed of multiple processors, and the multiple processors are coupled to each other through one or more buses. The processor may include an analog-to-digital converter (ADC) and a digital-to-analog converter (DAC) to achieve signal connection between different components of the device. The processor 130 is used to process media signals such as images, audio and video.

[0070] The memory 140 is coupled to the processor 130. Specifically, the memory 140 may be coupled to the processor 130 via one or more memory controllers. The memory 140 may be used to store computer program instructions, including a computer operating system (OS) and various user applications. The memory 140 may also be used to store user data, such as calendar information, contact information, acquired image information, audio information or other media files. The processor 130 may read computer program instructions or user data from the memory 140, or store computer program instructions or user data into the memory 140, to implement related processing functions. The memory 140 may be a non-power-off volatile memory, such as an EMMC (Embedded Multi Media Card), a UFS (Universal Flash Storage) or a read-only memory (ROM), or other types of static storage devices that can store static information and instructions. It may also be a power-off volatile memory (volatile memory), such as a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions. It may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other computer-readable storage medium that can be used to carry or store program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited to these. The memory 140 may exist independently, or the memory 140 may be integrated with the processor 130 .

[0071] The camera 150 is used to collect images or videos, and can be triggered to start by application instructions to realize the functions of taking pictures or videos, such as taking pictures or videos of any scene. The camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The imaging lens is mainly used to converge the light emitted or reflected by all objects in the camera angle (also referred to as the scene to be photographed, the target scene, and can also be understood as the scene image that the user expects to shoot) to form an image; the filter is mainly used to filter out the excess light waves in the light (for example, light waves other than visible light, such as infrared); the image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and input it into the processor 130 for subsequent processing. Among them, the camera can be located in front of the terminal device or on the back of the terminal device. The specific number and arrangement of the camera can be flexibly determined according to the needs of the designer or manufacturer's strategy, and this application does not limit it.

[0072] The audio circuit 160 is coupled to the processor 130. The audio circuit 160 may include a microphone 161 and a speaker 162. The microphone 161 may receive sound input from the outside world, and the speaker 162 may play audio data. It should be understood that the terminal 100 may have one or more microphones and one or more earphones, and the embodiment of the present application does not limit the number of microphones and earphones.

[0073] The display screen 170 is used to display information input by the user and various menus of information provided to the user. These menus are associated with specific internal modules or functions. The display screen 170 can also accept user input, such as accepting control information such as enabling or disabling. Specifically, the display screen 170 may include a display panel 171 and a touch panel 172. Among them, the display panel 171 may be configured by a liquid crystal display (LCD), an organic light-emitting diode (OLED), a light-emitting diode (LED) display device or a cathode ray tube (CRT). The touch panel 172, also known as a touch screen, a touch-sensitive screen, etc., can collect user contact or non-contact operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch panel 172, which may also include somatosensory operations; the operations include single-point control operations, multi-point control operations, and other operation types.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 172 may include two parts: a touch detection device and a touch controller. The touch detection device detects the signal brought by the user's touch operation and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into information that can be processed by the processor 130, and then sends it to the processor 130, and can receive commands sent by the processor 130 and execute them. Furthermore, the touch panel 172 may cover the display panel 171, and the user may operate on or near the touch panel 172 covered on the display panel 171 according to the content displayed on the display panel 171, which includes but is not limited to a soft keyboard, a virtual mouse, virtual buttons, icons, etc. After the touch panel 172 detects the operation on or near it, it transmits it to the processor 130 through the I / O subsystem 10 to determine the user input, and then the processor 130 provides corresponding visual output on the display panel 171 through the I / O subsystem 10 according to the user input. Although in Figure 1 In the embodiment, the touch panel 172 and the display panel 171 are used as two independent components to implement the input and output functions of the terminal 100, but in some embodiments, the touch panel 172 and the display panel 171 can be integrated to implement the input and output functions of the terminal 100.

[0074] The sensor 180 may include an image sensor, a motion sensor, a proximity sensor, an ambient noise sensor, a sound sensor, an accelerometer, a temperature sensor, a gyroscope or other types of sensors, and various combinations thereof. The processor 130 drives the sensor 180 to receive various information such as audio signals, image signals, motion information, etc. through the sensor controller 12 in the I / O subsystem 10, and the sensor 180 transmits the received information to the processor 130 for processing.

[0075] The wireless transceiver 190 can provide wireless connection capabilities to other devices, which can be peripheral devices such as wireless headsets, Bluetooth headsets, wireless mice, wireless keyboards, etc., or wireless networks, such as Wireless Fidelity (WiFi) networks, Wireless Personal Area Networks (WPAN) or other Wireless Local Area Networks (WLAN), etc. The wireless transceiver 190 can be a Bluetooth-compatible transceiver, used to wirelessly couple the processor 130 to peripheral devices such as Bluetooth headsets and wireless mice, and the wireless transceiver 190 can also be a WiFi-compatible transceiver, used to wirelessly couple the processor 130 to wireless networks or other devices.

[0076] The terminal 100 may also include other input devices 14, which are coupled to the processor 130 to receive various user inputs, such as input numbers, names, addresses, and media selections, etc. The other input devices 14 may include a keyboard, physical buttons (press buttons, rocker buttons, etc.), a dial, a slide switch, a joystick, a click wheel, an optical mouse (an optical mouse is a touch-sensitive surface that does not display visual output, or an extension of a touch-sensitive surface formed by a touch screen), etc.

[0077] The terminal 100 may also include the above-mentioned I / O subsystem 10, which may include other input device controllers 11 for receiving signals from other input devices 14 or sending control or drive information of the processor 130 to other input devices 14. The I / O subsystem 10 may also include the above-mentioned sensor controller 12 and display controller 13, which are respectively used to realize the exchange of data and control information between the sensor 180 and the display screen 170 and the processor 130.

[0078] The terminal 100 may also include a power supply 101 to supply power to other components including 110-190 of the terminal 100, and the power supply may be a rechargeable or non-rechargeable lithium-ion battery or a nickel-metal hydride battery. Further, when the power supply 101 is a rechargeable battery, it may be coupled to the processor 130 through a power management system, so as to implement functions such as managing charging, discharging, and power consumption adjustment through the power management system.

[0079] It should be understood that Figure 1 The terminal 100 in the example is only an example, and does not limit the specific form of the terminal 100. The terminal 100 may also include Figure 1 Other components that exist or may be added in the future that are not shown.

[0080] In an optional solution, the RF circuit 120, the processor 130 and the memory 140 may be partially or completely integrated on one chip, or may be three independent chips. The RF circuit 120, the processor 130 and the memory 140 may include one or more integrated circuits arranged on a printed circuit board (PCB).

[0081] Figure 2 A hardware architecture diagram of a device for optimizing motion vectors provided in an embodiment of the present application, wherein the device 200 for optimizing motion vectors may be, for example, a processor chip. For example, Figure 2 The hardware architecture diagram shown in can be Figure 1 0 is an exemplary architecture diagram of the processor 130 in FIG. 1 , and the method for optimizing motion vectors provided in an embodiment of the present application can be applied to the processor chip.

[0082] refer to Figure 2 The device 200 includes: at least one CPU, memory, microcontroller unit (MCU), GPU, NPU, memory bus, receiving interface and sending interface, etc. Figure 2 Not shown, the device 200 may also include an application processor (AP), a decoder, and a dedicated video or image processor.

[0083] The above-mentioned parts of the device 200 are coupled through a connector. Exemplarily, the connector includes various interfaces, transmission lines or buses, etc. These interfaces are usually electrical communication interfaces, but may also be mechanical interfaces or other forms of interfaces, which are not limited in this embodiment.

[0084] Optionally, the CPU may be a single-CPU processor or a multi-CPU processor; Optionally, the CPU may be a processor group consisting of multiple processors, and the multiple processors are coupled to each other through one or more buses. The receiving interface may be an interface for data input of a processor chip. In an optional case, the receiving interface and the sending interface may be a High Definition Multimedia Interface (HDMI), a V-By-One interface, an Embedded Display Port (eDP), a Mobile Industry Processor Interface (MIPI), or a Display Port (DP), etc. The memory may refer to the aforementioned description of the memory 140.

[0085] In an optional case, the above-mentioned parts are integrated on the same chip; in another optional case, the CPU, GPU, decoder, receiving interface and sending interface are integrated on one chip, and the various parts inside the chip access the external memory through the bus. The dedicated video / graphics processor can be integrated on the same chip as the CPU, or it can exist as a separate processor chip. For example, the dedicated video / graphics processor can be a dedicated ISP. In an optional case, the NPU can also be used as an independent processor chip. The NPU is used to implement various neural network or deep learning related operations. Optionally, the image processing method and image processing framework provided in the embodiment of the present application can be implemented by a GPU or an NPU, or by a dedicated graphics processor.

[0086] The chip involved in the embodiments of the present application is a system manufactured on the same semiconductor substrate using an integrated circuit process, also called a semiconductor chip, which can be a collection of integrated circuits formed on a substrate (usually a semiconductor material such as silicon) using an integrated circuit process, and its outer layer is usually encapsulated by a semiconductor packaging material. The integrated circuit may include various functional devices, each of which includes transistors such as logic gate circuits, metal oxide semiconductor (MOS) transistors, bipolar transistors or diodes, and may also include other components such as capacitors, resistors or inductors. Each functional device can work independently or under the action of necessary driver software, and can realize various functions such as communication, calculation, or storage.

[0087] In addition, the method for optimizing motion vectors provided in the embodiment of the present application can be applied to network devices. Correspondingly, the device for optimizing motion vectors provided in the embodiment of the present application can be a network device product, such as a backend server, etc. When the device is a network device, the device can be combined with a terminal device to form a Figure 3 The system for increasing the video frame rate shown ( Figure 3 A structural diagram of a system for improving the video frame rate provided in an embodiment of the present application). In the system, the device for optimizing motion vectors is connected to the terminal device for communication. When a user needs to improve the video frame rate of a video to be processed, the terminal device can send the video to be processed to the device for optimizing motion vectors, and the device can process the video to be processed to obtain a video with an improved frame rate, and feed the video with an improved frame rate back to the terminal device for the user to watch.

[0088] Specifically, the device includes at least one memory and at least one communication interface. The processor, memory and communication interface are connected through a communication bus and communicate with each other. At least one memory is used to store code. At least one processor is used to execute the above code to execute the method for optimizing motion vectors provided in the embodiment of the present application, so as to obtain a video with an increased frame rate. At least one communication interface is used to communicate with a device or a communication network to send the video with an increased frame rate to the device or the communication network.

[0089] In one possible implementation, the communication interface may be a software development kit (SDK) or an application programming interface (API).

[0090] Figure 4 A flow chart of a method for optimizing motion vectors provided in an embodiment of the present application. Figure 4 As shown, the method includes:

[0091] 401. Obtain first position information of a small object region in a current video frame and second position information of the small object region in a reference video frame through a first neural network, wherein the current video frame and the reference video frame are two adjacent video frames, and the small object region includes a target small object.

[0092] When it is necessary to increase the video frame rate of the video to be processed, each video frame of the video to be processed is first obtained frame by frame. After obtaining at least two video frames, the two adjacent video frames can be regarded as a video frame combination, and a video frame combination includes a current video frame and a reference video frame. It should be noted that the timing of the current video frame and the reference video frame in the video to be processed is adjacent. For example, after the first video frame and the second video frame of the video to be processed are obtained in sequence, the first video frame and the second video frame can be used as a video frame combination. In this combination, the first video frame and the second video frame can be the current video frame and the reference video frame of each other, that is, the first video frame is used as the current video frame and the second video frame is used as the reference video frame, or the second video frame is used as the current video frame and the first video frame is used as the reference video frame. It can be seen that this embodiment does not limit the timing sequence between the current video frame and the reference video frame.

[0093] For the video to be processed, not every video frame contains the target small object. Therefore, in order to improve efficiency, the second neural network can be used to filter the video frames of the video to be processed in a variety of ways, which will be introduced below:

[0094] (1) After the second neural network receives the current video frame and the reference video frame successively, it can determine whether the current video frame and the reference video frame include a small object area, and the small object area includes the target small object. If the second neural network determines that the current video frame and the reference video frame include a small object area, it sends a first identifier and a second identifier to the first neural network, the first identifier is used to indicate that the current video frame includes a small object area, and the second identifier is used to indicate that the reference video frame includes a small object area. After receiving the first identifier and the second identifier, the first neural network can detect the position of the target small object on the current video frame and the reference video frame.

[0095] (2) After the second neural network receives the current video frame and the reference video frame successively, it can perform scene recognition on the current video frame and the reference video frame. If the second neural network recognizes that the scene presented by the current video frame and the scene presented by the reference video frame are both target scenes, the first identifier and the second identifier are sent to the first neural network through the second neural network. The first identifier is used to indicate that the scene presented by the current video frame is the target scene, and the second identifier is used to indicate that the scene presented by the reference video frame is the target scene. It should be noted that the target scene is a scene containing a target small object, such as the aforementioned small object motion scene, etc. After receiving the first identifier and the second identifier, the first neural network can detect the position of the target small object on the current video frame and the reference video frame.

[0096] In addition, if the second neural network determines that a certain video frame does not contain a small object area or the scene presented by the video frame is not the target scene, the video frame can be processed in a variety of ways. For example, the second neural network sends a third identifier to the first neural network. After receiving the third identifier, the first neural network will not detect the position of the target small object for the video frame. For another example, the second neural network will not send any information to the first neural network, and the first neural network will not receive the video frame, so the first neural network cannot perform any processing on the video frame, and so on.

[0097] After receiving the first identifier, the first neural network can obtain the first position information of the small object area in the current video frame, and after receiving the second identifier, the first neural network can obtain the second position information of the small object area in the reference video frame. Among them, the first position information of the small object area includes the first center coordinates of the small object area in the current video frame and the first size of the small object area, and the second position information of the small object area includes the second center coordinates of the small object area in the reference video frame and the second size of the small object area. It should be noted that the first center coordinates and the second center coordinates of the small object area are usually different, indicating that when the video is played from the current video frame to the reference video frame, the position of the small object area has changed, that is, the position of the target small object has changed. The first size of the small object area refers to the length and width of the small object area in the current video frame, and the second size of the small object area refers to the length and width of the small object area in the reference video frame. It should be understood that the first size and the second size of the small object area may be the same or different, and there is no limitation here.

[0098] For ease of understanding, the following combination Figure 5 Further introduction to the small object area. Figure 5 A schematic diagram of the small object area provided in an embodiment of the present application. Figure 5 The content shown is the small object area in the current video frame and the small object area in the reference video frame. Figure 5 In the video, the small object area is usually a square area, which contains a small football. Since the small football is a round object, the area in the small object area except the small football is the pixel point of the small football's surrounding environment. Based on the first center coordinates of the small object area, the length and width of the small object area, the first position of the entire small object area in the current video frame can be calculated. Similarly, based on the second center coordinates of the small object area, the length and width of the small object area, the second position of the entire small object area in the reference video frame can be calculated. It can be seen that when the video is played from the current video frame to the reference video frame, the small object area moves from the first position to the second position, that is, the small football moves from the first position to the second position.

[0099] It should be understood that the first neural network can be any one of the models such as multi-layer perceptron (MLP), convolutional neural networks (CNN), recursive neural network (RNN), etc., and the second neural network can also be any one of the models such as MLP, CNN, recursive neural network, RNN, etc., without limitation here.

[0100] It should also be understood that the first neural network and the second neural network in the embodiment of the present application are both trained neural network models. The following is a brief introduction to the training process of the first neural network and the second neural network:

[0101] (1) Before model training, a batch of video frames to be trained is obtained, and the real position information of the small object area in each video frame to be trained is determined in advance. After the training starts, each video frame to be trained can be input into the first model to be trained (before inputting into the training video frame, a corresponding number of identifiers have been input into the first model to be trained, and one identifier corresponds to one video frame to be trained input into the model). Then, the position information of the small object area in each video frame to be trained is obtained through the first model to be trained. Finally, the difference between the position information of the small object area in each video frame to be trained output by the first model to be trained and the real position information is calculated by the target loss function. If the difference between the two position information corresponding to a video frame to be trained is within the qualified range, it is regarded as a qualified video frame to be trained. If it is outside the qualified range, it is regarded as an unqualified video frame to be trained. If there are only a small number of qualified video frames to be trained in the batch of video frames to be trained, the parameters of the first model to be trained are adjusted, and another batch of video frames to be trained are used for retraining until there are a large number of qualified video frames to be trained to obtain the first neural network.

[0102] (2) Before model training, a batch of video frames to be trained is obtained, and the real identifier corresponding to each video frame to be trained is determined in advance. The real identifier corresponding to a video frame to be trained is used to indicate that the video frame to be trained includes a small object area or the scene presented by the video frame to be trained is a target scene (i.e., a scene containing a target small object). After training begins, each video frame to be trained can be input into the second model to be trained. Then, the identifier corresponding to each video frame to be trained is obtained through the second model to be trained. Finally, the difference between the identifier corresponding to each video frame to be trained output by the second model to be trained and the real identifier is calculated by the target loss function. If the difference between the two identifiers corresponding to a video frame to be trained is within the qualified range, it is regarded as a qualified video frame to be trained. If it is outside the qualified range, it is regarded as an unqualified video frame to be trained. If there are only a small number of qualified video frames to be trained in the batch of video frames to be trained, the parameters of the second model to be trained are adjusted, and another batch of video frames to be trained are used for retraining until there are a large number of qualified video frames to be trained to obtain the second neural network.

[0103] 402. Obtain a motion vector of a small object region in a current video frame according to the first position information and the second position information.

[0104] After obtaining the first position information and the second position information of the small object region through the first neural network, the motion vector of the small object region of the current video frame can be obtained according to the first position information and the second position information. Specifically, the first position of the small object region in the current video frame can be determined based on the first position information, and the second position of the small object region in the reference video frame can be determined based on the second position information. Therefore, the displacement generated when the small object region moves from the first position to the second position can be calculated, that is, the motion vector of the small object region of the current video frame (for details, please refer to Figure 5 Motion vectors of the small object area shown in ).

[0105] 403. Acquire multiple motion vectors of multiple first image blocks in the current video frame and motion vectors of remaining areas except the multiple first image blocks in the current video frame according to the current video frame and the reference video frame, wherein each first image block corresponds to a motion vector, the area formed by the multiple first image blocks includes a small object area, the remaining areas include multiple fourth image blocks, the motion vectors of the remaining areas include multiple motion vectors of the multiple fourth image blocks, and each fourth image block corresponds to a motion vector.

[0106] After obtaining the current video frame and the reference video frame, the motion vector of each image block in the current video frame can also be obtained based on the MEMC technology. Figure 6 Further introduction to MEMC technology. Figure 6A schematic diagram of a current video frame and a reference video frame provided in an embodiment of the present application. Figure 6 As shown, the current video frame is divided into a plurality of non-overlapping image blocks (including the first image block and the fourth image block), and each image block has the same size. The current video frame is divided into two regions, one of which contains a small object region (i.e. Figure 6 The area surrounded by the thick line in the video frame, for example, the area containing the small football), which is composed of multiple first image blocks, and the other area does not contain the small object area (i.e. Figure 6 The rest of the area except the area surrounded by the thick line, for example, the environment around the small football in the video frame), this area is composed of multiple fourth image blocks. Similarly, the reference video frame is also divided into multiple non-overlapping image blocks, and the size of each image block is the same (the size of the image block of the current video frame is the same as the size of the image block of the reference video frame, and the size of the image block can be set according to actual needs, which is not limited here). Therefore, with the image block of the reference video frame as a reference, motion estimation can be performed on each image block in the current video frame, so as to calculate the motion vector of each image block in the current video frame, that is, the motion vector of each first image block and the motion vector of each fourth image block in the current video frame. For example, in Figure 6 In the present invention, the motion vector of a first image block in the current video frame represents the displacement between the position of the first image block in the current video frame and the position of the first image block in the reference video frame.

[0107] It should be understood that step 403 and step 401 may be executed synchronously or asynchronously, which is not limited here. For example, step 403 and step 401 are executed simultaneously, or step 403 is executed before step 401, or step 403 is executed after step 401, and so on.

[0108] 404. Determine a better motion vector between a motion vector of a target first image block and a motion vector of a small object region, wherein the target first image block is any first image block among the plurality of first image blocks.

[0109] After obtaining the motion vector of the small object region of the current video frame and the motion vector of each first image block in the current video frame, the motion vector of each first image block can be compared with the motion vector of the small object region, so as to select a better motion vector as the final motion vector of each first image block. For example, after obtaining the motion vector of the small object region of the current video frame and the motion vectors of the first image block A, the first image block B and the first image block C in the current video frame, the motion vector of the first image block A can be compared with the motion vector of the small object region, and the better motion vector can be selected as the final motion vector of the first image block A, and the motion vector of the first image block B can be compared with the motion vector of the small object region, and the better motion vector can be selected as the final motion vector of the first image block B, and the motion vector of the first image block C can be compared with the motion vector of the small object region, and the better motion vector can be selected as the final motion vector of the first image block C.

[0110] For the sake of convenience, the following uses any one of the multiple first image blocks (the first image block is referred to as the target first image block) as an example to further introduce the process of determining the preferred motion vector. Specifically, the preferred motion vector can be determined from the motion vector of the target first image block and the motion vector of the small object area according to the degree of overlap between the target first image block and the small object area. Figure 7 The aforementioned degree of overlap is introduced. Figure 7 A schematic diagram of a current video frame provided in an embodiment of the present application. Figure 7 As shown, after the current video frame is divided into multiple image blocks, the area formed by multiple first image blocks contains a small object area (i.e. Figure 7 The area surrounded by the dotted line in the figure). Generally, the area formed by the multiple first image blocks is larger than or equal to the small object area. When the area formed by the multiple first image blocks is larger than the small object area, the degree of overlap between a part of the first image blocks and the small object area is 100% (i.e., this part of the first image blocks is completely covered by the small object area), and the degree of overlap between another part of the first image blocks and the small object area is less than 100% (i.e., this part of the first image blocks is not completely covered by the small object area). For example, in the area formed by the multiple first image blocks, the degree of overlap between the first image block in the upper left corner and the small object area is 25%, while the degree of overlap between the first image block in the middle and the small object area is 100%.

[0111] Based on the above introduction, the degree of overlap between the target first image block and the small object area can be determined in the current video frame. After determining the degree of overlap between the target first image block and the small object area, a better motion vector can be determined from the motion vector of the target first image block and the motion vector of the small object area. Specifically, the process of determining the better motion vector may include the following steps: (1) Determine the second image block corresponding to the target first image block in the reference frame according to the motion vector of the small object area. (2) Determine the third image block corresponding to the target first image block in the reference frame according to the motion vector of the target first image block. (3) Obtain a first matching degree between the target first image block and the second image block according to the degree of overlap between the target first image block and the small object area. (4) Obtain a second matching degree between the target first image block and the third image block. (5) Compare the first matching degree and the second matching degree, and determine the motion vector corresponding to the larger matching degree as the better motion vector from the motion vector of the target first image block and the motion vector of the small object area.

[0112] For further understanding, the following Figure 8 The above process of determining a better motion vector is further introduced. Figure 8 Another schematic diagram of the current video frame and the reference video frame provided in the embodiment of the present application. Figure 8As shown, let the motion vector of the small object area be b2, and the motion vector of the target first image block be b3. First, according to the direction of the motion vector b2, the second image block corresponding to the target first image block can be determined in the reference video frame (i.e., a certain image block matching the target first image block in the reference frame), and according to the direction of the motion vector b3, the third image block corresponding to the target first image block can be determined in the reference video frame (i.e., another image block matching the target first image block in the reference frame). Then, the third matching degree between the target first image block and the second image block is calculated. It should be noted that the third matching degree is the original matching degree between the target first image block and the second image block. Considering the degree of overlap between the target first image block and the small object area, the original matching degree and the overlapping degree between the two image blocks are calculated to obtain the first matching degree (generally, when the overlapping degree is less than 100%, the first matching degree will be less than the third matching degree, and when the overlapping degree is equal to 100%, the first matching degree will be equal to the third matching degree). It can be understood that the first matching degree is the final matching degree between the target first image block and the second image block. Next, the second matching degree between the target first image block and the third image block is calculated (the second matching degree is calculated in the same manner as the third matching degree). Finally, the first matching degree and the second matching degree are compared. If the first matching degree is greater than the second matching degree, the motion vector b2 is selected as the motion vector finally determined by the target first image block. If the first matching degree is less than the second matching degree, the motion vector b3 is selected as the motion vector finally determined by the target first image block. If the first matching degree is equal to the second matching degree, the selection can be made in a variety of ways. For example, a motion vector is randomly selected from the motion vector b2 and the motion vector b3 as the motion vector finally determined by the target first image block. For example, if the time when the motion vector b2 is obtained is earlier than the time when the motion vector b3 is obtained, the motion vector b2 is used as the motion vector finally determined by the target first image block. If the time when the motion vector b3 is obtained is earlier than the time when the motion vector b2 is obtained, the motion vector b3 is used as the motion vector finally determined by the target first image block, and so on. It should be understood that the remaining image blocks except the target first image block in all the first image blocks in the current video frame also need to perform the above process, which will not be repeated here.

[0113] 405. Generate an interpolation frame for inserting between the current video frame and the reference video frame according to the better motion vector and the motion vectors of the remaining regions, so as to obtain a video with an increased frame rate.

[0114] After obtaining the final motion vector of each first image block, combined with the motion vector of each fourth image block, it is equivalent to obtaining the final motion vector of each image block in the current video frame. After obtaining the final motion vector of each image block in the current video frame, an interpolation frame can be generated based on the final motion vectors of all image blocks in the current video frame, and the interpolation frame can be inserted between the current video frame and the reference video frame of the video to be processed to obtain a video with an increased frame rate, thereby completing the conversion of the video frame rate.

[0115] 406. Obtain third position information of the small object region in the interpolation frame according to the preset interpolation phase, the first position information, and the second position information.

[0116] After obtaining the first position information and the second position information of the small object area through the first neural network, the preset interpolation phase can be further obtained. It should be noted that the interpolation phase can be used to indicate the position of the interpolation frame between the current video frame and the reference video frame. Therefore, based on the interpolation phase, the first position information and the second position information, the third position information of the small object area in the interpolation frame can be estimated. The third position information of the small object area includes the third center coordinates of the small object area in the reference video frame and the third size of the small object area. It can be understood that the third center coordinates of the small object area are different from the first center coordinates and the second center coordinates of the small object area, indicating that when the video is played from the current video frame to the interpolation frame and then from the interpolation frame to the reference video frame, the position of the small object area has changed, that is, the position of the target small object has changed. The third size of the small object area refers to the length and width of the small object area in the interpolation frame. It should be understood that the first size, the second size and the third size of the small object area can be all the same, partially the same, or all different, and there is no limitation here.

[0117] 407. Insert the third position information between the first position information and the second position information to obtain trajectory information of the small object.

[0118] After obtaining the third position information, the third position information can be inserted between the first position information and the second position information, and the three position information can be connected to obtain the trajectory information of the small object, which can be used to indicate the movement trajectory of the target small object in the video.

[0119] 408. Correct the trajectory information of the small object according to the motion vectors of the remaining areas to obtain corrected trajectory information of the small object.

[0120] However, the trajectory information of the small object obtained based on the position information of the small object area may not indicate the real trajectory of the target small object. For example, if the perspective of shooting the current video frame is switched with the perspective of shooting the reference video frame (that is, the target small object does not move, but the camera shooting the video moves), the position of the target small object changes when the video is played from the current video frame to the reference video frame, while the position of the target small object does not actually change. In this case, the trajectory information of the small object needs to be corrected so that the trajectory information can accurately indicate the real trajectory of the target small object.

[0121] Specifically, if the viewing angle of the current video frame is switched to the viewing angle of the reference video frame, the environment around the target small object will change significantly when the video is played from the current video frame to the reference video frame. Therefore, after obtaining the motion vectors of the remaining areas except the first image blocks in the current video frame, the trajectory information of the small object can be corrected based on the motion vectors of the remaining areas to obtain the corrected trajectory information of the small object, and the corrected trajectory information of the small object can be used to indicate the real trajectory of the target small object.

[0122] 409. Highlight the small object in the video with an increased frame rate according to the corrected trajectory information of the small object.

[0123] After obtaining the corrected trajectory information of the small object, the real trajectory of the target small object can be determined based on the trajectory information, so as to track the target small object in the video with an increased frame rate and highlight the target small object.

[0124] Furthermore, if there are multiple small object regions in the video frame, before step 406, the first position information of the multiple small object regions in the current video frame and the second position information of the multiple small object regions in the reference video frame can be matched one by one through time domain clustering (time domain clustering can cluster information that is temporally related), so as to determine the first position information of each small object region in the current video frame and the second position information in the reference video frame. For example, suppose that there are small object regions D, small object regions E, and small object regions F in both the current video frame and the reference video frame. After the current video frame is input into the first neural network, the first position information of the small object region D, the first position information of the small object region E, and the first position information of the small object region F can be obtained. After the reference video frame is input into the first neural network, the second position information of the small object region D, the second position information of the small object region E, and the second position information of the small object region F can be obtained. In order to match the first position information and the second position information of the same small object area, this part of the position information can be clustered in the time domain to distinguish the position information of different small object areas, that is, the first position information of small object area D and the second position information of small object area D are clustered into one category, the first position information of small object area E and the second position information of small object area E are clustered into one category, and the first position information of small object area F and the second position information of small object area F are clustered into one category.

[0125] After determining the first position information and the second position information of each small object area, the trajectory information of each small object can be estimated and corrected. Based on the corrected trajectory information of each small object, each target small object can be highlighted in the video. For details, please refer to steps 406 to 409, which will not be repeated here.

[0126] 410. Crop the next video frame of the current video frame or the next video frame of the reference video frame according to the corrected trajectory information of the small object to obtain a cropped next video frame, and input the cropped next video frame into the first neural network.

[0127] After obtaining the corrected trajectory information of the small object, the estimated position information of the small object area in the next video frame can be obtained based on the trajectory information. Based on the estimated position information, the estimated position of the small object area in the next video frame can be determined. After obtaining the next video frame of the current video frame or the next video frame of the reference video frame, the small object area in the next video frame is retained based on the estimated position of the small object area, and other areas except the small object area in the next video frame are cropped, thereby reducing the number of pixels in the next video frame to reduce the workload of the first neural network. The cropped next video frame is then input into the first neural network to obtain the first position information of the small object area in the next video frame through the first neural network.

[0128] It is worth noting that the next video frame can form a new video frame combination with its corresponding previous video frame, and the new video frame combination is executed as steps 401 to 410 to continue to improve the video frame rate, which will not be repeated here.

[0129] In this embodiment, the motion vector of the small object area of ​​the current video frame generated based on the position information of the small object area in the current video frame and the reference video frame is first obtained, and the motion vectors of multiple first image blocks of the current video frame are obtained based on the MEMC technology, wherein the area formed by the multiple first image blocks includes the small object area. Then, the motion vector of each first image block is compared with the motion vector of the small object area, and a better motion vector can be selected as the final determined motion vector of each first image block. Since the motion vector obtained by the traditional algorithm and the motion vector obtained based on the position information of the target small object are combined when determining the motion vector of the target small object, the accuracy of the estimated motion vector of the target small object can be improved.

[0130] In the embodiment of the present application, the motion vector calculated based on the position information of the small object analyzed by the neural network and the motion vector obtained by the traditional algorithm are combined to improve the accuracy of the motion vector of the target small object; further, for the scene containing multiple small objects, the embodiment of the present application performs time domain filtering and trajectory prediction on multiple target small objects, and corrects the trajectory of the target small object in combination with the motion vector of the background area other than the target small object area, thereby improving the accuracy of the trajectory prediction of the target small object. On the other hand, the video frame input to the neural network is dynamically cropped based on the prediction result to reduce the calculation amount of the neural network, thereby effectively reducing the operation time of the NPU and improving the real-time detection of the target small object. Especially in the case of huge neural network calculation amount, while improving the NPU operation efficiency, the detection rate of the target small object is improved, and the image quality of the high frame rate (greater than or equal to 60Hz) video is improved.

[0131] In an optional situation, the embodiment of the present application may also adopt different display methods according to the movement speed of the small object. For small objects that move at a slow speed or are severely affected by human interference, they may be displayed in the form of continuous flashing dots, and for small objects that move quickly, they may be displayed in the form of continuous tracks.

[0132] The above is a detailed description of the method for optimizing motion vectors provided in the embodiment of the present application. The following is an introduction to the device for optimizing motion vectors provided in the embodiment of the present application. Fig. 9 A schematic diagram of a structure of a device for optimizing motion vectors provided in an embodiment of the present application, such as Fig. 9 As shown, the device comprises:

[0133] The first acquisition module 901 is used to obtain first position information of the small object area in the current video frame and second position information of the small object area in the reference video frame. The current video frame and the reference video frame are two adjacent video frames, and the small object area contains the target small object.

[0134] The second acquisition module 902 is used to acquire the motion vector of the small object area in the current video frame according to the first position information and the second position information.

[0135] The third acquisition module 903 is used to acquire multiple motion vectors of multiple first image blocks in the current video frame according to the current video frame and the reference video frame, wherein each first image block corresponds to a motion vector, and the area formed by the multiple first image blocks includes a small object area.

[0136] It should be understood that, in a possible case, the first acquisition module 901, the second acquisition module 902, and the third acquisition module 903 may be the same module.

[0137] The determination module 904 is used to determine a better motion vector between the motion vector of the target first image block and the motion vector of the small object area, where the target first image block is any first image block among the multiple first image blocks.

[0138] In a possible implementation, the first acquisition module 901 is used to acquire the first position information of the small object region in the current video frame and the second position information of the small object region in the reference video frame through the first neural network. Specifically, the first acquisition module 901 is used to call the program instructions stored in the storage unit to run the first neural network, or the first acquisition unit 901 is used to call the hardware resources solidified with the program instructions to run the first neural network. Alternatively, the first acquisition module 901 can also be referred to as the first neural network.

[0139] In a possible implementation, the determination module 904 is further configured to determine a better motion vector among the motion vector of the target first image block and the motion vector of the small object region according to the degree of overlap between the target first image block and the small object region.

[0140] In a possible implementation, the determination module 904 is further used to: determine the second image block corresponding to the target first image block in the reference frame according to the motion vector of the small object area. Determine the third image block corresponding to the target first image block in the reference frame according to the motion vector of the target first image block. Obtain a first matching degree between the target first image block and the second image block according to the degree of overlap between the target first image block and the small object area. Obtain a second matching degree between the target first image block and the third image block. Compare the first matching degree and the second matching degree, and determine a better motion vector between the motion vector of the target first image block and the motion vector of the small object area.

[0141] In a possible implementation, the third acquisition module 903 is also used to obtain the motion vector of the remaining area in the current video frame except the multiple first image blocks based on the current video frame and the reference video frame, the remaining area includes multiple fourth image blocks, and the motion vector of the remaining area includes multiple motion vectors of the multiple fourth image blocks, wherein each fourth image block corresponds to a motion vector.

[0142] In a possible implementation, the device further includes: a generation module 905, configured to generate an interpolation frame for insertion between the current video frame and the reference video frame according to the better motion vector and the motion vectors of the remaining regions, so as to obtain a video with an increased frame rate.

[0143] In a possible implementation, the device further includes: a fourth acquisition module 907, which is used to obtain the third position information of the small object area in the interpolation frame according to the preset interpolation phase, the first position information and the second position information. An insertion module 908, which is used to insert the third position information between the first position information and the second position information to obtain the trajectory information of the small object. A correction module 909, which is used to correct the trajectory information of the small object according to the motion vector of the remaining area to obtain the corrected trajectory information of the small object. A display module 910, which is used to highlight the small object in the video with the increased frame rate according to the corrected trajectory information of the small object.

[0144] In a possible implementation, the device further includes: a cropping module 611, which is used to crop the next video frame of the current video frame or the next video frame of the reference video frame according to the corrected trajectory information of the small object to obtain the cropped next video frame. The cropping module is also used to input the cropped next video frame into the first neural network.

[0145] In one possible implementation, the device also includes: a clustering module 906, which is used to perform time domain clustering on the first position information of multiple small object regions in the current video frame and the second position information of multiple small object regions in the reference video frame to determine the first position information of each small object region in the current video frame and the second position information in the reference video frame.

[0146] In one possible implementation, the device also includes: an analysis module 900, which is used to send a first identifier and a second identifier to the first neural network through the second neural network if it is determined through the second neural network that the current video frame and the reference video frame include a small object area, the first identifier being used to indicate that the current video frame includes a small object area, and the second identifier being used to indicate that the reference video frame includes a small object area.

[0147] The first acquisition module 901 is also used to obtain first position information of the small object area in the current video frame through the first neural network after the first neural network receives the first identifier, and to obtain second position information of the small object area in the reference video frame through the second neural network after the first neural network receives the second identifier.

[0148] In one possible implementation, the device also includes: an analysis module 900, which is used to send a first identifier and a second identifier to the first neural network through the second neural network if the second neural network identifies that the scene presented by the current video frame and the scene presented by the reference video frame are both target scenes, the first identifier being used to indicate that the scene presented by the current video frame is the target scene, and the second identifier being used to indicate that the scene presented by the reference video frame is the target scene, and the target scene is a scene containing a small target object.

[0149] The first acquisition module 901 is also used to obtain first position information of the small object area in the current video frame through the first neural network after the first neural network receives the first identifier, and to obtain second position information of the small object area in the reference video frame through the second neural network after the first neural network receives the second identifier.

[0150] In one possible implementation, the first position information of the small object area includes the first center coordinates of the small object area in the current video frame and the first size of the small object area, and the second position information of the small object area includes the second center coordinates of the small object area in the reference video frame and the second size of the small object area.

[0151] In one possible implementation, the first neural network is a multilayer perceptron, a convolutional neural network, a recursive neural network or a recurrent neural network, and the second neural network is a multilayer perceptron, a convolutional neural network, a recursive neural network or a recurrent neural network, which improves the flexibility and selectivity of the solution.

[0152] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and will not be repeated here.

[0153] The present application also relates to a computer-readable storage medium, including instructions, which, when executed on a computer, cause the computer to execute the following Figure 4 The method shown.

[0154] The present application also relates to a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute Figure 4 The method shown.

[0155] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0156] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

Claims

1. A method for optimizing a motion vector, characterized in that: The method comprises: Acquire first position information of a small object region in a current video frame and second position information of the small object region in a reference video frame, wherein the current video frame and the reference video frame are two adjacent video frames, and the small object region includes a target small object; Acquire a motion vector of a small object region of the current video frame according to the first position information and the second position information; Acquire multiple motion vectors of multiple first image blocks in the current video frame according to the current video frame and the reference video frame, wherein each first image block corresponds to a motion vector, and an area formed by the multiple first image blocks includes the small object area; A better motion vector is determined between a motion vector of a target first image block and a motion vector of the small object region, wherein the target first image block is any first image block among the plurality of first image blocks.

2. The method according to claim 1, characterized in that The obtaining of first position information of the small object region in the current video frame and second position information of the small object region in the reference video frame includes: The first position information of the small object region in the current video frame and the second position information of the small object region in the reference video frame are obtained through a first neural network.

3. The method according to claim 2, characterized in that The determining of a better motion vector between the motion vector of the target first image block and the motion vector of the small object area comprises: According to the degree of overlap between the target first image block and the small object region, a more optimal motion vector is determined between the motion vector of the target first image block and the motion vector of the small object region.

4. The method according to claim 3, characterized in that The determining, according to the overlap degree between the target first image block and the small object region, a better motion vector between the motion vector of the target first image block and the motion vector of the small object region comprises: Determining, in the reference video frame, a second image block corresponding to the target first image block according to the motion vector of the small object area; Determining, in the reference video frame, a third image block corresponding to the target first image block according to a motion vector of the target first image block; Acquire a first matching degree between the target first image block and the second image block according to the overlapping degree between the target first image block and the small object region; Acquire a second matching degree between the target first image block and the third image block; The first matching degree and the second matching degree are compared, and a better motion vector is determined between the motion vector of the target first image block and the motion vector of the small object area.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: According to the current video frame and the reference video frame, motion vectors of remaining areas in the current video frame except the multiple first image blocks are obtained, the remaining areas include multiple fourth image blocks, and the motion vectors of the remaining areas include multiple motion vectors of the multiple fourth image blocks, wherein each fourth image block corresponds to a motion vector.

6. The method according to claim 5, characterized in that After determining a better motion vector between the motion vector of the target first image block and the motion vector of the small object area, the method further includes: An interpolation frame for inserting between the current video frame and the reference video frame is generated according to the better motion vector and the motion vectors of the remaining areas, so as to obtain a video with an improved frame rate.

7. The method according to claim 6, characterized in that The method further comprises: Acquire third position information of the small object area in the interpolation frame according to a preset interpolation phase, the first position information, and the second position information; Inserting the third position information between the first position information and the second position information to obtain trajectory information of the small object; Correcting the trajectory information of the small object according to the motion vectors of the remaining areas to obtain corrected trajectory information of the small object; The small object in the video with the increased frame rate is highlighted according to the corrected trajectory information of the small object.

8. The method according to claim 7, characterized in that After the trajectory information of the small object is corrected according to the motion vector of the remaining area to obtain the corrected trajectory information of the small object, the method further includes: Cropping the next video frame of the current video frame or the next video frame of the reference video frame according to the corrected trajectory information of the small object to obtain a cropped next video frame; The cropped next video frame is input into the first neural network.

9. The method according to any one of claims 7 or 8, characterized in that: If there are multiple small object regions, before obtaining the third position information of the small object region in the interpolation frame according to the preset interpolation phase, the first position information and the second position information, the method further includes: Time domain clustering is performed on first position information of multiple small object regions in the current video frame and second position information of the multiple small object regions in the reference video frame to determine first position information of each small object region in the current video frame and second position information in the reference video frame.

10. The method according to any one of claims 2 to 4, characterized in that: Before obtaining the first position information of the small object region in the current video frame and the second position information of the small object region in the reference video frame through the first neural network, the method further includes: If it is determined through the second neural network that the current video frame and the reference video frame include the small object region, a first identifier and a second identifier are sent to the first neural network through the second neural network, wherein the first identifier is used to indicate that the current video frame includes the small object region, and the second identifier is used to indicate that the reference video frame includes the small object region; The acquiring, by using a first neural network, the first position information of the small object region in the current video frame and the second position information of the small object region in the reference video frame comprises: After the first neural network receives the first identifier, the first position information of the small object area in the current video frame is obtained through the first neural network, and after the first neural network receives the second identifier, the second position information of the small object area in the reference video frame is obtained through the second neural network.

11. The method according to any one of claims 1 to 4, characterized in that: The first position information of the small object area includes the first center coordinates of the small object area in the current video frame and the first size of the small object area, and the second position information of the small object area includes the second center coordinates of the small object area in the reference video frame and the second size of the small object area.

12. A device for optimizing motion vectors, characterized in that: The device comprises: A first acquisition module is used to acquire first position information of a small object region in a current video frame and second position information of the small object region in a reference video frame, wherein the current video frame and the reference video frame are two adjacent video frames, and the small object region includes a target small object; A second acquisition module, configured to acquire a motion vector of a small object region of the current video frame according to the first position information and the second position information; a third acquisition module, configured to acquire, according to the current video frame and the reference video frame, a plurality of motion vectors of a plurality of first image blocks in the current video frame, wherein each first image block corresponds to a motion vector, and an area formed by the plurality of first image blocks includes the small object area; The determination module is used to determine a better motion vector between a motion vector of a target first image block and a motion vector of the small object area, wherein the target first image block is any first image block among the multiple first image blocks.

13. The device according to claim 12, characterized in that The first acquisition module is also used to acquire the first position information of the small object area in the current video frame and the second position information of the small object area in the reference video frame through a first neural network.

14. The device according to claim 13, characterized in that The determination module is further configured to determine a better motion vector among the motion vector of the target first image block and the motion vector of the small object region according to the degree of overlap between the target first image block and the small object region.

15. The device according to claim 14, characterized in that The determining module is also used for: Determining, in the reference video frame, a second image block corresponding to the target first image block according to the motion vector of the small object area; Determining, in the reference video frame, a third image block corresponding to the target first image block according to a motion vector of the target first image block; Acquire a first matching degree between the target first image block and the second image block according to the overlapping degree between the target first image block and the small object region; Acquire a second matching degree between the target first image block and the third image block; The first matching degree and the second matching degree are compared, and a better motion vector is determined between the motion vector of the target first image block and the motion vector of the small object area.

16. The device according to any one of claims 13 to 15, characterized in that The device also includes: A fourth acquisition module is used to acquire, according to the current video frame and the reference video frame, motion vectors of remaining areas in the current video frame except the multiple first image blocks, wherein the remaining areas include multiple fourth image blocks, and the motion vectors of the remaining areas include multiple motion vectors of the multiple fourth image blocks, wherein each fourth image block corresponds to a motion vector.

17. The device according to claim 16, characterized in that The device also includes: A generation module is used to generate an interpolation frame for inserting between the current video frame and the reference video frame according to the better motion vector and the motion vectors of the remaining areas, so as to obtain a video with an improved frame rate.

18. The device according to claim 17, characterized in that The device also includes: a fifth acquisition module, configured to acquire third position information of the small object region in the interpolation frame according to a preset interpolation phase, the first position information, and the second position information; An insertion module, used for inserting the third position information between the first position information and the second position information to obtain trajectory information of the small object; A correction module, used for correcting the trajectory information of the small object according to the motion vector of the remaining area to obtain the corrected trajectory information of the small object; The display module is used to highlight the small object in the video with the increased frame rate according to the corrected trajectory information of the small object.

19. The device according to claim 18, characterized in that The device also includes: A cropping module, used for cropping the next video frame of the current video frame or the next video frame of the reference video frame according to the corrected trajectory information of the small object, to obtain a cropped next video frame; The cropping module is also used to input the cropped next video frame into the first neural network.

20. The device according to any one of claims 18 or 19, characterized in that The device also includes: The clustering module is used to perform time-domain clustering on the first position information of multiple small object regions in the current video frame and the second position information of the multiple small object regions in the reference video frame to determine the first position information of each small object region in the current video frame and the second position information in the reference video frame.

21. The device according to any one of claims 13 to 15, characterized in that The device also includes: an analysis module, configured to send a first identifier and a second identifier to the first neural network through the second neural network if it is determined through the second neural network that the current video frame and the reference video frame include the small object region, wherein the first identifier is used to indicate that the current video frame includes the small object region, and the second identifier is used to indicate that the reference video frame includes the small object region; The first acquisition module is also used to acquire the first position information of the small object area in the current video frame through the first neural network after the first neural network receives the first identifier, and to acquire the second position information of the small object area in the reference video frame through the second neural network after the first neural network receives the second identifier.

22. The device according to any one of claims 12 to 15, characterized in that The first position information of the small object area includes the first center coordinates of the small object area in the current video frame and the first size of the small object area, and the second position information of the small object area includes the second center coordinates of the small object area in the reference video frame and the second size of the small object area.

23. A device for optimizing motion vectors, characterized in that: include: A processor and a transmission interface, wherein the processor is configured to call program instructions stored in a memory to execute the method according to any one of claims 1 to 11.

24. A computer-readable storage medium comprising instructions, which, when executed on a computer or a processor, enable the computer or the processor to perform the method according to any one of claims 1 to 11.

25. A computer program product comprising instructions, the computer program product comprising program instructions, which, when executed on a computer or a processor, cause the computer or the processor to perform the method according to any one of claims 1 to 11.

Images