Method and apparatus for motion estimation of video images

Abstract

The application provides a motion estimation method and device for video images, to improve the accuracy of motion estimation. The method comprises: determining the backward motion vector of a first image, the forward motion vector and the first SAD of a second image frame, determining the forward intermediate frame according to the forward motion vector and the second SAD, and determining the backward intermediate frame according to the backward motion vector, calculating the first similarity between the i-th sub-image block in the backward intermediate frame and the sub-image block of the first image frame, and the second similarity between the i-th sub-image block in the forward intermediate frame and the sub-image block of the first image frame, determining the first matching error according to the first similarity and the first SAD corresponding to the first matching pair, and determining the second matching error according to the second similarity and the second SAD corresponding to the second matching pair; and taking the motion vector corresponding to the smaller matching error in the first matching error and the second matching error as the motion vector of the i-th sub-image block in the intermediate frame to be inserted.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for motion estimation of video images. Background Technology

[0002] Frame rate enhancement technology increases the video frame rate by estimating the motion state between preceding and following reference frames and generating new interpolated frames using these frames and the motion state. This allows for dynamic frame interpolation, ensuring smoother visuals, improved clarity, and enhanced viewing experience. Current technologies typically use forward or backward motion estimation for video motion estimation. However, when the foreground in the video moves—for example, when the foreground from the previous frame moves to the location of the background in the next frame—the motion vectors obtained for the background region during forward and backward motion estimation become inaccurate, affecting the accuracy of the interpolated frames. Summary of the Invention

[0003] This application provides a motion estimation method and apparatus for video images to improve the accuracy of motion estimation.

[0004] In a first aspect, embodiments of this application provide a motion estimation method for video images, including:

[0005] Acquire a first image frame and a second image frame, wherein the first image frame is the previous image frame adjacent to the second image frame; the first image frame and the second image frame each include N sub-image blocks;

[0006] Determine the backward motion vector and the first absolute error value (SAD) of each sub-image block in the first image frame, and the forward motion vector and the second SAD of each sub-image block in the second image frame;

[0007] The backward intermediate frame is determined based on the N sub-image blocks in the first image frame and the corresponding backward motion vector, and the forward intermediate frame is determined based on the N sub-image blocks in the second image frame and the corresponding forward motion vector.

[0008] Determine the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair;

[0009] Wherein, the first matching pair includes determining the k1th sub-image block in the first image frame and the k2th sub-image block in the second image frame corresponding to the backward motion vector used by the i-th sub-image block in the backward intermediate frame; the second matching pair includes determining the s1th sub-image block in the second image frame and the s2th sub-image block in the first image frame corresponding to the forward motion vector used by the i-th sub-image block in the forward intermediate frame.

[0010] A first matching error is determined based on the first similarity and the first SAD corresponding to the first matching pair, and a second matching error is determined based on the second similarity and the second SAD corresponding to the second matching pair.

[0011] The motion vector corresponding to the smaller of the first and second matching errors is used as the motion vector of the i-th sub-image block in the intermediate frame to be inserted.

[0012] Based on the above scheme, forward and backward intermediate frames are determined by forward motion estimation and backward motion estimation. The similarity between the forward and backward intermediate frames and the source image is compared. The matching error is determined by the similarity and the absolute error value. Then, the optimal motion vector is selected from the forward and backward motion vectors. This method can effectively improve the accuracy of motion estimation.

[0013] In one possible implementation, determining the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair includes: obtaining image edge information and image detail information of the i-th sub-image block of the backward intermediate frame, and obtaining image edge information and image detail information of the k1-th sub-image block in the first image frame of the first matching pair;

[0014] A first edge similarity is determined between the image edge information of the i-th sub-image block and the image edge information of the k1-th sub-image block in the backward intermediate frame, and a first detail similarity is determined between the image detail information of the i-th sub-image block and the image detail information of the k1-th sub-image block. The weighted sum of the first edge similarity and the first detail similarity is the first similarity.

[0015] or,

[0016] Determining the first similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair includes:

[0017] Obtain the image edge information and image detail information of the i-th sub-image block in the forward intermediate frame, and obtain the image edge information and image detail information of the s2-th sub-image block in the first image frame of the second matching pair;

[0018] A second edge similarity is determined between the image edge information of the i-th sub-image block and the image edge information of the s2-th sub-image block in the forward intermediate frame, and a second detail similarity is determined between the image detail information of the i-th sub-image block and the image detail information of the s2-th sub-image block. The weighted sum of the second edge similarity and the second detail similarity is the second similarity.

[0019] In one possible implementation, the method further includes: before determining a first similarity between the i-th sub-image block of the backward intermediate frame and a sub-image block in the first image frame of the first matching pair, and a second similarity between the i-th sub-image block of the forward intermediate frame and a sub-image block in the first image frame of the second matching pair, determining that the grayscale error between the grayscale image of the i-th sub-image block of the backward intermediate frame and the grayscale image of the i-th sub-image block of the forward intermediate frame is greater than an error threshold;

[0020] The step of determining the first matching error based on the first similarity and the first SAD corresponding to the first matching pair includes:

[0021] The grayscale error is adjusted to a first value based on the first similarity, and the weighted sum of the first value and the first SAD corresponding to the first matching pair is taken as the first matching error.

[0022] The second matching error is determined based on the second similarity and the second SAD corresponding to the second matching pair, including:

[0023] The grayscale error value is adjusted to a second value based on the second similarity, and the weighted sum of the second value and the second SAD corresponding to the second matching pair is taken as the second matching error.

[0024] In one possible implementation, when the grayscale difference is less than or equal to a set threshold, the first matching error is the first SAD corresponding to the first matching pair, and the second matching error is the second SAD corresponding to the second matching pair; or,

[0025] The first matching error is the weighted sum of the grayscale error and the first SAD corresponding to the first matching pair, and the second matching error is the weighted sum of the grayscale error and the second SAD corresponding to the second matching pair.

[0026] In one possible implementation, image edge information of a designated sub-image block is obtained. This designated sub-image block is either the i-th sub-image block in the forward intermediate frame, or the i-th sub-image block in the backward intermediate frame, or the k1-th sub-image block in the first image frame, or the s2-th sub-image block in the first image frame. This includes:

[0027] Determine the first pixel contrast in the horizontal direction and the second pixel contrast in the vertical direction in the set sub-image block; the first pixel contrast is determined based on the pixel values ​​of the pixels included in the row where the center pixel of the set sub-image block is located, and the second pixel contrast is determined based on the pixel values ​​of the pixels included in the column where the center pixel is located;

[0028] The sum of the contrast of the first pixel and the contrast of the second pixel is used as the filtering coefficient to filter the set sub-image block, thereby obtaining the edge information of the set sub-image block.

[0029] In one possible implementation, image detail information of a designated sub-image block is obtained. The designated sub-image block is the i-th sub-image block in the forward intermediate frame, or the i-th sub-image block in the backward intermediate frame, or the k1-th sub-image block in the first image frame, or the s2-th sub-image block in the first image frame, including:

[0030] Gaussian filtering is applied to the specified sub-image block to obtain the image low-level information of the specified sub-image block. The image low-level information is then subtracted from the image information of the specified sub-image block to obtain the image detail information of the specified sub-image block.

[0031] In one possible implementation, determining the grayscale error between the grayscale image of the i-th sub-image block of the backward intermediate frame and the grayscale image of the i-th sub-image block of the forward intermediate frame includes:

[0032] A first gray-level histogram of the grayscale image of the i-th sub-image block in the forward intermediate frame and a second gray-level histogram of the grayscale image of the i-th sub-image block in the backward intermediate frame are determined. The first gray-level histogram is used to characterize the number of pixels in the i-th sub-image block in the forward intermediate frame whose grayscale values ​​are distributed across M grayscale levels, and the second gray-level histogram is used to characterize the number of pixels in the i-th sub-image block in the backward intermediate frame whose grayscale values ​​are distributed across M grayscale levels.

[0033] The grayscale error is determined based on the first grayscale level histogram and the second grayscale level histogram, and the grayscale error satisfies the condition shown in the following formula:

[0034]

[0035] Wherein, E represents grayscale error, and a k This represents the weight corresponding to the k-th gray level. This represents the number of pixels in the i-th sub-image block of the forward intermediate frame whose grayscale values ​​are distributed at the k-th grayscale level. M represents the number of pixels in the i-th sub-image block in the backward intermediate frame whose gray values ​​are distributed at the k-th gray level, where M is a positive integer.

[0036] Secondly, embodiments of this application provide a motion estimation device for video images, comprising:

[0037] The acquisition module is used to acquire a first image frame and a second image frame, wherein the first image frame is the previous image frame adjacent to the second image frame; the first image frame and the second image frame each include N sub-image blocks;

[0038] The determination module is used to determine the backward motion vector and the first absolute error value (SAD) of each sub-image block of the first image frame, and the forward motion vector and the second SAD of each sub-image block of the second image frame.

[0039] The backward intermediate frame is determined based on the N sub-image blocks in the first image frame and the corresponding backward motion vector, and the forward intermediate frame is determined based on the N sub-image blocks in the second image frame and the corresponding forward motion vector.

[0040] Determine the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair;

[0041] Wherein, the first matching pair includes determining the k1th sub-image block in the first image frame and the k2th sub-image block in the second image frame corresponding to the backward motion vector used by the i-th sub-image block in the backward intermediate frame; the second matching pair includes determining the s1th sub-image block in the second image frame and the s2th sub-image block in the first image frame corresponding to the forward motion vector used by the i-th sub-image block in the forward intermediate frame.

[0042] A first matching error is determined based on the first similarity and the first SAD corresponding to the first matching pair, and a second matching error is determined based on the second similarity and the second SAD corresponding to the second matching pair.

[0043] The motion vector corresponding to the smaller of the first and second matching errors is used as the motion vector of the i-th sub-image block in the intermediate frame to be inserted.

[0044] In one possible implementation, the determining module, when determining the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, is specifically used to: obtain the image edge information and image detail information of the i-th sub-image block of the backward intermediate frame, and obtain the image edge information and image detail information of the k1-th sub-image block in the first image frame of the first matching pair;

[0045] A first edge similarity is determined between the image edge information of the i-th sub-image block and the image edge information of the k1-th sub-image block in the backward intermediate frame, and a first detail similarity is determined between the image detail information of the i-th sub-image block and the image detail information of the k1-th sub-image block. The weighted sum of the first edge similarity and the first detail similarity is the first similarity.

[0046] or,

[0047] The determining module, when determining the first similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair, is specifically used for:

[0048] Obtain the image edge information and image detail information of the i-th sub-image block in the forward intermediate frame, and obtain the image edge information and image detail information of the s2-th sub-image block in the first image frame of the second matching pair;

[0049] A second edge similarity is determined between the image edge information of the i-th sub-image block and the image edge information of the s2-th sub-image block in the forward intermediate frame, and a second detail similarity is determined between the image detail information of the i-th sub-image block and the image detail information of the s2-th sub-image block. The weighted sum of the second edge similarity and the second detail similarity is the second similarity.

[0050] In one possible implementation, the determining module is further configured to: determine, before determining the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair, that the grayscale error between the grayscale image of the i-th sub-image block of the backward intermediate frame and the grayscale image of the i-th sub-image block of the forward intermediate frame is greater than an error threshold;

[0051] The determining module, when determining the first matching error based on the first similarity and the first SAD corresponding to the first matching pair, is specifically used for:

[0052] The grayscale error is adjusted to a first value based on the first similarity, and the weighted sum of the first value and the first SAD corresponding to the first matching pair is taken as the first matching error.

[0053] The determining module, when determining the second matching error based on the second similarity and the second SAD corresponding to the second matching pair, is specifically used for:

[0054] The grayscale error is adjusted to a second value based on the second similarity, and the weighted sum of the second value and the second SAD corresponding to the second matching pair is taken as the second matching error.

[0055] In one possible implementation, when the grayscale error is less than or equal to a set threshold, the first matching error is the first SAD corresponding to the first matching pair, and the second matching error is the second SAD corresponding to the second matching pair; or,

[0056] The first matching error is the weighted sum of the grayscale error and the first SAD corresponding to the first matching pair, and the second matching error is the weighted sum of the grayscale error and the second SAD corresponding to the second matching pair.

[0057] In one possible implementation, the determining module, when acquiring the image edge information of a designated sub-image block, determines that the designated sub-image block is either the i-th sub-image block in the forward intermediate frame, or the i-th sub-image block in the backward intermediate frame, or the k1-th sub-image block in the first image frame, or the s2-th sub-image block in the first image frame. Specifically, this is used for:

[0058] Determine the first pixel contrast in the horizontal direction and the second pixel contrast in the vertical direction in the set sub-image block; the first pixel contrast is determined based on the pixel values ​​of the pixels included in the row where the center pixel of the set sub-image block is located, and the second pixel contrast is determined based on the pixel values ​​of the pixels included in the column where the center pixel is located;

[0059] The sum of the contrast of the first pixel and the contrast of the second pixel is used as the filtering coefficient to filter the set sub-image block, thereby obtaining the edge information of the set sub-image block.

[0060] In one possible implementation, the determining module, when acquiring image detail information of a designated sub-image block, determines that the designated sub-image block is the i-th sub-image block in the forward intermediate frame, or the i-th sub-image block in the backward intermediate frame, or the k1-th sub-image block in the first image frame, or the s2-th sub-image block in the first image frame, specifically for:

[0061] Gaussian filtering is applied to the specified sub-image block to obtain the image low-level information of the specified sub-image block. The image low-level information is then subtracted from the image information of the specified sub-image block to obtain the image detail information of the specified sub-image block.

[0062] In one possible implementation, the determining module, when determining the grayscale error between the grayscale image of the i-th sub-image block of the backward intermediate frame and the grayscale image of the i-th sub-image block of the forward intermediate frame, is specifically used for:

[0063] A first gray-level histogram of the grayscale image of the i-th sub-image block in the forward intermediate frame and a second gray-level histogram of the grayscale image of the i-th sub-image block in the backward intermediate frame are determined. The first gray-level histogram is used to characterize the number of pixels in the i-th sub-image block in the forward intermediate frame whose grayscale values ​​are distributed across M grayscale levels. The second gray-level histogram is used to characterize the number of pixels in the i-th sub-image block in the backward intermediate frame whose grayscale values ​​are distributed across M grayscale levels.

[0064] The gray level error is determined based on the first gray level histogram and the second gray level histogram;

[0065] The grayscale error satisfies the condition shown in the following formula:

[0066]

[0067] Wherein, E represents grayscale error, and a k This represents the weight corresponding to the k-th gray level. This represents the number of pixels in the i-th sub-image block of the forward intermediate frame whose grayscale values ​​are distributed at the k-th grayscale level. M represents the number of pixels in the i-th sub-image block in the backward intermediate frame whose gray values ​​are distributed at the k-th gray level, where M is a positive integer.

[0068] Thirdly, embodiments of this application provide an execution device, including:

[0069] Memory, used to store program instructions;

[0070] A processor is configured to acquire program instructions stored in the memory and execute the method described in the first aspect and different implementations of the first aspect according to the acquired program instructions.

[0071] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect and different implementations of the first aspect.

[0072] The technical effects of any of the implementation methods in the second to fourth aspects can be found in the first aspect and the technical effects of different implementation methods of the first aspect, which will not be repeated here. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 A schematic diagram of a video image provided in an embodiment of this application;

[0075] Figure 2 A schematic diagram illustrating a usage scenario of a display device provided in an embodiment of this application;

[0076] Figure 3 A configuration block diagram of a control device 100 provided in an embodiment of this application;

[0077] Figure 4 A hardware configuration block diagram of a display device 200 provided in an embodiment of this application;

[0078] Figure 5 This application provides a schematic diagram of the software architecture of a terminal device.

[0079] Figure 6A This is a schematic diagram of the system architecture provided for an embodiment of this application;

[0080] Figure 6B Another system architecture diagram provided for an embodiment of this application;

[0081] Figure 7 This is a schematic diagram of the electronic device structure provided in the embodiments of this application;

[0082] Figure 8 A flowchart illustrating a motion estimation method for video images provided in an embodiment of this application;

[0083] Figure 9 This application provides a schematic diagram of the distribution of a sub-image block in an image frame.

[0084] Figure 10 A schematic diagram illustrating motion estimation provided in an embodiment of this application;

[0085] Figure 11 A schematic diagram of the first region range in the second image frame provided in the embodiments of this application;

[0086] Figure 12 A schematic diagram illustrating the numbering of sub-image blocks within the first region provided in an embodiment of this application;

[0087] Figure 13 A schematic diagram of motion vectors provided for embodiments of this application;

[0088] Figure 14 A schematic diagram illustrating the determination of pixel values ​​in a forward intermediate frame, provided in an embodiment of this application;

[0089] Figure 15 A schematic diagram of a bilinear interpolation method provided in an embodiment of this application;

[0090] Figure 16 A schematic diagram of a grayscale histogram provided for an embodiment of this application;

[0091] Figure 17 A schematic diagram of a motion estimation device for video images provided in an embodiment of this application;

[0092] Figure 18 This is a schematic diagram of an execution device provided in an embodiment of this application. Detailed Implementation

[0093] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0094] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0095] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0096] Frame rate enhancement technology increases the video frame rate by estimating the motion state between preceding and following reference frames, generating new interpolated frames using the preceding and following frames and the motion state, and inserting them into the original video. Frame rate enhancement technology enables dynamic frame interpolation, ensuring smoothness, clarity, and improved visual effects and viewing experience. Ideally, the intermediate frames interpolated by forward motion estimation and backward motion estimation are identical, resulting in minimal image interpolation. Figure 1 As shown, assuming the white area represents the background region and remains stationary, the ball moves horizontally to the right, from the dashed line area to the solid line area. In backward motion estimation, the background corresponding to the ball in the previous frame is covered by the foreground in the current frame. Therefore, the background in the previous frame that contains the ball in the current frame cannot accurately find a matching block in the current frame, resulting in inaccurate motion estimation of the background region covered by the foreground in the previous frame. Similarly, in forward motion estimation, the background region exposed after the ball moves in the previous frame is the foreground region of the previous frame, and therefore, the best matching block cannot be accurately matched in the current frame.

[0097] To address the aforementioned problems, this application provides a motion estimation method and apparatus for video images. This method calculates the forward and backward motion vectors between two adjacent video frames, and inserts a forward intermediate frame based on the forward motion vector and a backward intermediate frame based on the backward motion vector. Further, it determines a first similarity between the i-th sub-image block of the backward intermediate frame and a sub-image block in the first image frame of a first matching pair, and a second similarity between the i-th sub-image block of the forward intermediate frame and a sub-image block in the first image frame of a second matching pair. A first matching error is determined based on the first similarity and the first similarity vector (SAD) corresponding to the first matching pair, and a second matching error is determined based on the second similarity and the second similarity vector (SAD) corresponding to the second matching pair. The motion vector corresponding to the smaller of the first and second matching errors is used as the motion vector of the i-th sub-image block in the intermediate frame to be inserted. Through this scheme, the optimal motion vector is selected from the forward and backward motion vectors using the image information of the image frames, thereby improving the accuracy of motion estimation.

[0098] The motion estimation method for video images provided in this application can be implemented by an execution device. In some embodiments, the execution device can be a terminal device. The terminal device can be a display device with display functionality. Display devices can include: smart TVs, mobile phones, tablet computers, etc.

[0099] The following description uses the display device as an example to illustrate the structure and application scenarios of the execution device. Figure 2 This is a schematic diagram illustrating a usage scenario of the display device in the embodiment. For example... Figure 2 As shown, the display device 200 can also communicate with the server 400, and the user can operate the display device 200 through the smart device 300 or the control device 100. In one possible example, the server 400 can transmit video images to the display device 200, and the display device 200 can perform motion estimation methods for the video images.

[0100] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes at least one of infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via a wired connection. Users can control the display device 200 by inputting user commands through at least one method, such as buttons on the remote control, voice input, or control panel input.

[0101] In some embodiments, the smart device 300 may include any one of a mobile terminal, tablet computer, computer, laptop computer, AR / VR device, etc.

[0102] In some embodiments, a smart device 300 may also be used to control the display device 200. For example, an application running on the smart device may be used to control the display device 200.

[0103] In some embodiments, the smart device 300 and the display device 200 may also be used for data communication.

[0104] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0105] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0106] Figure 3 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 3 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, and a memory. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0107] In some embodiments, the communication interface 130 is used for external communication and includes at least one of a WIFI chip, a Bluetooth module, an NFC module, or an alternative module.

[0108] In some embodiments, the user input / output interface 140 includes at least one of a microphone, touchpad, sensor, button, or alternative module.

[0109] The following description uses display device 200 as an example to illustrate the embodiment. It should be understood that... Figure 4 The display device 200 shown is merely an example, and the display device 200 may have more than Figure 4 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0110] Figure 4 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown.

[0111] In some embodiments, the display device 200 includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280.

[0112] In some embodiments, the controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to an nth interface for input / output.

[0113] In some embodiments, the display 260 includes a display screen component for presenting an image, a driving component for driving image display, a component for receiving image signals output from a controller, and a user control UI interface, etc.

[0114] In some embodiments, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0115] In some embodiments, the tuner 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0116] In some embodiments, the communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220.

[0117] In some embodiments, detector 230 is used to acquire signals from the external environment or interactions with the outside world. For example, detector 230 includes a light receiver (not shown) for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0118] In some embodiments, the external device interface 240 may include, but is not limited to, one or more interfaces such as: High Definition Multimedia Interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0119] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0120] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations via various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0121] In some embodiments, the object can be any of the optional objects, such as a hyperlink, an icon, or other operable controls. Operations related to the selected object include: displaying links to hyperlinked pages, documents, images, etc., or performing operations corresponding to the program associated with the icon.

[0122] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0123] The CPU processor is the control center of the display device 200, including the system-on-a-chip (SoC), such as... Figure 4 As shown, a CPU is used to execute operating system and application instructions stored in memory, as well as various interactive instructions received from external input, to execute various applications, data, and content, ultimately displaying and playing various audio and video content. A CPU processor can include multiple processors, such as a main processor and one or more sub-processors.

[0124] In some embodiments, a graphics processor is used to generate at least one of various graphical objects, such as icons, operation menus, and user-input-based graphics. The graphics processor includes an arithmetic logic unit (ALU) that performs calculations based on various user-input interactive commands and displays various objects according to display attributes; it also includes a renderer that renders the various objects obtained from the ALU, the rendered objects being displayed on a monitor.

[0125] In some embodiments, a video processor is configured to receive an external video signal and perform at least one of the following video processing operations according to a standard encoding and decoding protocol of the input signal: decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc., to obtain a signal that can be directly displayed or played on the display device 200.

[0126] In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image compositing module, a frame rate conversion module, and a display formatting module. The demultiplexing module demultiplexes the input audio and video data streams. The video decoding module processes the demultiplexed video signal, including decoding and scaling. The image compositing module, such as an image synthesizer, overlays and blends a GUI signal generated by a graphics generator based on user input or its own generation with the scaled video image to generate a displayable image signal. The frame rate conversion module converts the input video frame rate. The display formatting module modifies the received frame rate-converted video output signal to conform to a display format, such as outputting RGB data signals.

[0127] In some embodiments, an audio processor is configured to receive external audio signals, perform decompression and decoding according to a standard codec protocol of the input signals, and at least one of the following processes: noise reduction, digital-to-analog conversion, and amplification, to obtain a sound signal that can be played in a speaker.

[0128] In some embodiments, the user can input user commands through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input commands through the GUI. Alternatively, the user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0129] In some embodiments, a "user interface" is the medium through which an application or operating system interacts and exchanges information with a user, enabling the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0130] In some embodiments, the user interface 280 is an interface that can be used to receive control input (e.g., physical buttons on the display device body, or others).

[0131] In some embodiments, the display device's system may include a kernel, a command interpreter (shell), a file system, and applications. The kernel, shell, and file system together form the basic operating system structure, allowing users to manage files, run programs, and use the system. Upon power-up, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, and runs and maintains virtual memory, the scheduler, signals, and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. Applications are compiled into machine code after startup, forming a process.

[0132] See Figure 5 In some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Layer"), and the kernel layer.

[0133] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0134] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications in the application layer. Applications can access system resources and obtain system services during execution through the API interface.

[0135] like Figure 5 As shown, the application framework layer in this embodiment includes managers, content providers, and a view system. The managers include at least one of the following modules: an Activity Manager for interacting with all running activities in the system; a Location Manager for providing system services or applications with access to system location services; a Package Manager for retrieving various information related to application packages currently installed on the device; a Notification Manager for controlling the display and clearing of notification messages; and a Window Manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0136] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling display window changes (e.g., shrinking the display window, shaking the display, distorting the display, etc.).

[0137] In some embodiments, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0138] In some embodiments, the kernel layer is a layer between hardware and software. For example... Figure 5 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power management module, etc.

[0139] In other embodiments, the execution device may be an electronic device, which may be implemented by one or more servers, including local servers and cloud servers. See also Figure 6A As shown, server 500 can be implemented using a physical server or a virtual server. The server can be implemented using a single server or a server cluster consisting of multiple servers. The motion estimation method for video images provided in this application can be implemented using either a single server or a server cluster. Figure 6A Taking the connection between server 500, terminal device 600, and display device 200 as an example, server 500 can execute motion estimation methods for video images. In some scenarios, server 500 can also receive motion estimation tasks for video images sent by terminal device 600, or send the motion estimation results of video images to terminal device 600. In other scenarios, server 500 can also receive motion estimation tasks for video images sent by display device 200 and perform motion estimation, or display the intermediate frame image to be interpolated obtained based on the motion estimation results through display device 200. Figure 6B As shown, taking the connection between server 500 and display device 200 as an example, server 500 can execute motion estimation methods for video images. In some scenarios, server 500 can receive motion estimation tasks for video images sent by display device 200, perform motion estimation based on the motion estimation tasks, and send the intermediate frames to be interpolated obtained based on the motion estimation to display device 200. The electronic device can also be a personal computer, a handheld or laptop device, or a mobile device (such as a mobile phone, tablet computer, personal digital assistant, etc.).

[0140] As an example, see Figure 7 As shown, the electronic device may include a processor 510 and a communication interface 520. The electronic device may also include a memory 530. Of course, the electronic device may also include other components. Figure 7 Not shown in the image.

[0141] The communication interface 520 is used to communicate with the display device to receive motion estimation tasks of video images sent by the display device, or to send motion estimation results of video images or intermediate frames to be inserted based on motion estimation results to the display device.

[0142] In the embodiments of this application, the processor 510 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0143] Processor 510 is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and routes. It executes various functions and processes data by running or executing software programs and / or modules stored in memory 530, and by calling data stored in memory 530. Optionally, processor 510 may include one or more processing units. Processor 510 may be, for example, a processor, microprocessor, controller, or other control component. It may be a general-purpose central processing unit (CPU), a general-purpose processor, a digital signal processing unit (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0144] The memory 530 can be used to store software programs and modules. The processor 510 executes various functional applications and data processing by running the software programs and modules stored in the memory 530. The memory 530 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function, etc.; the data storage area may store data created according to business processing, etc. As a non-volatile computer-readable storage medium, the memory 530 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 530 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, disk, optical disk, etc. The memory 530 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 530 in this embodiment can also be a circuit or any other means capable of implementing storage functions for storing program instructions and / or data.

[0145] It should be noted that the above Figure 2-7 The structure shown is merely an example, and the embodiments of the present invention are not limited thereto.

[0146] This application provides a motion estimation method for video images. Figure 8 An exemplary flow of a motion estimation method for video images is illustrated, which can be executed by an execution device, which may be... Figure 4 The display device 200 shown can specifically perform motion estimation via a controller 250 within the display device 200. Alternatively, the execution device can be... Figure 7 The electronic device shown can specifically perform motion estimation through its processor 510. The specific process is as follows:

[0147] 801, acquire the first image frame and the second image frame.

[0148] In some embodiments, the first image frame and the second image frame are consecutive video frames captured by the acquisition device. The acquisition device may be an electronic police device, electronic monitoring device, surveillance camera, video recorder, or terminal device with video acquisition function (such as a laptop, computer, mobile phone, television, etc.).

[0149] For example, after the acquisition device acquires video frames, the server obtains a first image frame and a second image frame from the acquisition device. The first image frame is the preceding image frame adjacent to the second image frame.

[0150] In some embodiments, the server receives a video file sent by the acquisition device, the video file including the first image frame and the second image frame. The video file can be an encoded video file. Thus, the server can decode the received video file to obtain the first image frame and the second image frame. Encoding the video can effectively reduce the file size, facilitating transmission. This improves the video transmission speed, thereby increasing the efficiency of subsequent confirmation of video events. The method for obtaining the encoded bitstream data can be any applicable method, including but not limited to: Real-Time Streaming Protocol (RTSP), Open Network Video Interface Forum (ONVIF) standard, or proprietary protocols.

[0151] In some embodiments, after acquiring the first image frame and the second image frame, the first image frame and the second image frame can be divided into multiple sub-image blocks. As an example, the first image frame and the second image frame can each include N sub-image blocks. For instance, if the original resolution of the first image frame and the second image frame is 1440×810, then the first image frame and the second image frame can be divided into 480×270 sub-image blocks, each sub-image block having a resolution of 3×3. Figure 9 As shown.

[0152] 802, determine the backward motion vector and the first absolute error value SAD of each sub-image block of the first image frame, and the forward motion vector and the second SAD of each sub-image block of the second image frame.

[0153] In some embodiments, the determination of the backward motion vector can be achieved as follows: Taking a first sub-image block in a first image frame as an example, the first sub-image block is any one of N sub-image blocks. A first region is determined in the second image frame for the first sub-image block. The first region is the area obtained by expanding the second sub-image block in the second image frame by a predetermined factor. The position of the first sub-image block in the first image frame is the same as the position of the second sub-image block in the second image frame. Therefore, a sub-image block matching the first sub-image block is determined within the first region, such as... Figure 10 As shown. For example, the index position of the first sub-image block in the N sub-image blocks included in the first image frame can be represented as (50, 60). Then, the range of the first region in the second image frame can be the sub-image blocks with index positions in the range of (48, 58) to (52, 62), such as... Figure 11 As shown. Further, sub-image blocks matching the first sub-image block can be determined within the first region. Specifically, the absolute error value SAD between the first sub-image block and each sub-image block within the first region is determined. The absolute error value represents the sum of the absolute values ​​of the pixel grayscale differences between two sub-image blocks. As an example, the sub-image blocks within the ranges (48, 58) and (52, 65) are numbered as follows: Figure 12 As shown, the absolute error values ​​between the first sub-image block and sub-image blocks 1-25 are 30, 74, 40, 80, 46, 75, 68, 25, 48, 35, 69, 45, 32, 87, 22, 26, 45, 55, 69, 57, 54, 23, 58, 41, 13, respectively. Further, the sub-image block with the smallest absolute error value between itself and the first sub-image block within the first range is selected as the sub-image block that matches the first sub-image block. The center pixel points of the two matched sub-image blocks are determined to be at the first pixel index position in the first image frame and the second pixel index position in the second image frame, respectively. The backward motion vector of the first sub-image block is determined using the first pixel index position and the second pixel index position. Continuing with the above example, the absolute error value between the first sub-image block and sub-image block 25 is the smallest, so sub-image block 25 is determined to be the sub-image block that matches the first sub-image block. The backward motion vector is determined based on the pixel index positions of the center points of the first sub-image block and the matched sub-image block. For example, each sub-image block is 3×3 in size. The center point of the 25th sub-image block is located at the pixel index (155, 185) in the second image frame. The center point of the first sub-image block is located at the pixel index (149, 179). The backward motion vector corresponding to the first sub-image block can be determined based on the index positions of the two matched image blocks. The backward motion vector of the first sub-image block can be represented by mv, which is (6, 6). The first SAD of the backward motion vector corresponding to the first sub-image block is 13.

[0154] In some embodiments, the forward motion vector can be determined as follows: Taking a third sub-image block in a second image frame as an example, the third sub-image block is any one of the N sub-image blocks included in the second image frame. A second region is determined in the first image frame for the third sub-image block. The second region is the region in the first image frame expanded by a predetermined multiple with the fourth sub-image block as the center. The position of the fourth sub-image block in the first image frame is the same as the position of the third sub-image block in the second image frame. Further, a sub-image block matching the third sub-image block can be determined within the second region. The sub-image block with the smallest absolute error value between the second region and the third sub-image block is taken as the sub-image block matching the third sub-image block, such as... Figure 10 As shown. The center pixel points of the two matched sub-image blocks are determined to be at the third pixel index position in the first image frame and the fourth pixel index position in the second image frame, respectively. The forward motion vector of the third sub-image block is determined by the third pixel index position and the fourth pixel index position, and the absolute error value between the two matched sub-image blocks is used as the second SAD.

[0155] 803. Determine the backward intermediate frame based on the N sub-image blocks in the first image frame and the corresponding backward motion vectors, and determine the forward intermediate frame based on the N sub-image blocks in the second image frame and the corresponding forward motion vectors.

[0156] In some embodiments, when the intermediate frame to be inserted is determined to be the midpoint between the first image frame and the second image frame, for each sub-image block of the first image frame, the pixel index position of each sub-image block in the backward intermediate frame can be determined based on half of the backward motion vector corresponding to each sub-image block. Then, the backward intermediate frame is determined based on each sub-image block in the first image frame and its corresponding pixel index position. Similarly, for each sub-image block of the second image frame, the pixel index position of each sub-image block in the forward intermediate frame can be determined based on half of the forward motion vector corresponding to each sub-image block; the forward intermediate frame is then determined based on each sub-image block in the second image frame and its corresponding pixel index position.

[0157] In some embodiments, to ensure a one-to-one correspondence between motion vectors and pixels in the original image frame for convenient value retrieval, the number of motion vectors corresponding to each sub-image block can be proportionally enlarged to the size of the original image frame. Taking the first image frame as an example, the resolution of the first image frame is 1440×810. When the first image frame is divided into 480×270 sub-image blocks, 480×270 backward motion vectors can be obtained after motion estimation. The motion vector of each sub-image block is the motion vector corresponding to 3×3 pixels in that sub-image block. Therefore, each backward motion vector can be copied 3×3 times to obtain a backward motion vector of the same size as the first image frame. Figure 13 As shown, the left image displays the backward motion vectors corresponding to 3×3 sub-image blocks, with each block corresponding to one motion vector. After copying the number of motion vectors 3×3, the vector image corresponding to the first image frame is obtained, with a size of 6×6. In the vector image corresponding to the first image frame, each pixel corresponds to one motion vector, and the motion vectors corresponding to each pixel within a sub-image block are the same, as shown below. Figure 13 The parts marked with a diagonal line have the same vector.

[0158] In some embodiments, after determining the motion vector corresponding to a pixel, the position and number of intermediate frames to be inserted are determined. For example, the number and position of the intermediate frames to be inserted can be adjusted according to actual conditions, and this application does not impose specific limitations on this. For example, when the number of intermediate frames to be inserted is one frame, the position of the intermediate frame to be inserted can be the middle position between the first image frame and the second image frame. As another example, when the number of intermediate frames to be inserted is two frames, the positions of the intermediate frames to be inserted can be one-third and two-thirds of the distance between the first image frame and the second image frame, respectively. Taking one intermediate frame as an example, the value of each pixel can be determined based on the position of half of the motion vector corresponding to that pixel.

[0159] As an example, the forward motion vector can be represented by mv x If the pixel value of the second image frame can be represented as (i, j), then the position of the pixel corresponding to this pixel in the preceding intermediate frame can be represented as (i′, j′). i′ and j′ satisfy the conditions shown in the following formula:

[0160] Taking the second image frame as an example, such as Figure 14 As shown, the position of the pixel in the second image frame is (1245, 567). If the forward motion vector corresponding to this pixel is (4,4), then according to half of the motion vector, the position of the pixel in the forward intermediate frame corresponding to this pixel in the second image frame is (1247, 569). Therefore, the pixel value of this pixel in the second image frame is assigned to the pixel position with index (1247, 569) in the forward intermediate frame.

[0161] In some embodiments, when the point is between two pixels in the channel, bilinear interpolation can be used, performing interpolation once in the x-direction and once in the y-direction, according to the following formula:

[0162] f(i+u,j+v)=(1-u)(1-v)f(i,j)+u(1-v)f(i+1,j)+v(1-u)f(i,j+1)+uvf(i+1,j+1);

[0163] Where u represents the step size in the y direction, and v represents the step size in the x direction, as shown below. Figure 15 As shown.

[0164] In some embodiments, edge protection may be performed during the determination of motion vectors so that the motion vectors do not extend beyond the original image.

[0165] 804, determine the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair.

[0166] The first matching pair includes the k1th sub-image block in the first image frame and the k2th sub-image block in the second image frame corresponding to the backward motion vector used by the i-th sub-image block in the backward intermediate frame; the second matching pair includes the s1th sub-image block in the second image frame and the s2th sub-image block in the first image frame corresponding to the forward motion vector used by the i-th sub-image block in the forward intermediate frame.

[0167] In some embodiments, determining the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair can be achieved as follows: obtaining the image edge information and image detail information of the i-th sub-image block of the backward intermediate frame, and obtaining the image edge information and image detail information of the k1-th sub-image block in the first image frame of the first matching pair. Further, a first edge similarity can be determined between the image edge information of the i-th sub-image block and the image edge information of the k1-th sub-image block of the backward intermediate frame, and a first detail similarity can be determined between the image detail information of the i-th sub-image block and the image detail information of the k1-th sub-image block. The weighted sum of the first edge similarity and the first detail similarity is used as the first similarity.

[0168] In some embodiments, determining the first similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair can be achieved by: obtaining the image edge information and image detail information of the i-th sub-image block of the forward intermediate frame, and obtaining the image edge information and image detail information of the s2-th sub-image block in the first image frame of the second matching pair. Further, a second edge similarity can be determined between the image edge information of the i-th sub-image block and the image edge information of the s2-th sub-image block of the forward intermediate frame, and a second detail similarity can be determined between the image detail information of the i-th sub-image block and the image detail information of the s2-th sub-image block. The weighted sum of the second edge similarity and the second detail similarity is taken as the second similarity.

[0169] In some embodiments, a sub-image block is defined as the i-th sub-image block in the forward intermediate frame, or the i-th sub-image block in the backward intermediate frame, or the k1-th sub-image block in the first image frame, or the s2-th sub-image block in the first image frame. Obtaining the image edge information of the defined sub-image block can be achieved as follows: determining the image edge information through the difference in grayscale values ​​at the edges. Specifically, the first pixel contrast can be determined based on the pixel values ​​of the pixels in the row containing the center pixel of the defined sub-image block, and the second pixel contrast can be determined based on the pixel values ​​of the pixels in the column containing the center pixel. In some scenarios, multiple pixel value differences can be obtained by determining the difference in pixel values ​​between every two adjacent pixels in the row containing the center pixel of the defined sub-image block, and then summing these multiple pixel value differences to obtain the first pixel contrast. Similarly, multiple pixel value differences can be obtained by determining the difference in pixel values ​​between every two adjacent pixels in the column containing the center pixel of the defined sub-image block, and then summing these multiple pixel value differences to obtain the second pixel contrast. In other scenarios, the first pixel contrast can be obtained by determining the differences in pixel values ​​between any two non-adjacent pixels in the row containing the center pixel of a sub-image block, and then summing these differences. Similarly, the second pixel contrast can be obtained by determining the differences in pixel values ​​between any two non-adjacent pixels in the column containing the center pixel of a sub-image block. In yet another scenario, the first pixel contrast can be obtained by determining the differences in pixel values ​​between any two pixels symmetrical to the center pixel in the row containing the center pixel of a sub-image block, and then summing these differences. Similarly, the second pixel contrast can be obtained by determining the differences in pixel values ​​between any two pixels symmetrical to the center pixel in the column containing the center pixel of a sub-image block. For example, the first pixel contrast and the second pixel contrast satisfy the conditions shown in the following formula:

[0170] HF h =Curve(|∑ q＝(x-i,y) I q -∑ q＝(x+i,y) I q |);

[0171] HF v =Curve(|∑ q＝(x,y-i) I q -∑ q＝(x,y+i) I q |);

[0172] Among them, HF h Indicates the contrast of the first pixel, HF v This represents the contrast of the second pixel, and Curve represents the adaptive adjustment control parameter, i = 1, 2, 3, ...

[0173] Furthermore, the sum of the contrast of the first pixel and the contrast of the second pixel can be used as a filtering coefficient to filter the set sub-image block and obtain the edge information of the set sub-image block.

[0174] In some embodiments, the filter coefficients satisfy the conditions shown in the following formula:

[0175] HF = HF h +HF v ;

[0176] Among them, HF h Indicates the contrast of the first pixel, HF v HF represents the contrast of the second pixel.

[0177] In some embodiments, the sub-image block is designated as the i-th sub-image block in the forward intermediate frame, or the i-th sub-image block in the backward intermediate frame, or the k1-th sub-image block in the first image frame, or the s2-th sub-image block in the first image frame. Obtaining the image detail information of the designated sub-image block can be achieved as follows: Gaussian filtering is applied to the designated sub-image block to obtain its underlying image information. The image detail information is obtained by subtracting the underlying image information from the image information of the designated sub-image block. The image detail information satisfies the condition shown in the following formula:

[0178] D tl =If(I);

[0179] Among them, D tl This represents the image detail information of the set sub-image block, where I represents the image information of the set sub-image block, and f(I) represents the image underlying information of the set sub-image block.

[0180] 805. Determine the first matching error based on the first similarity and the first SAD corresponding to the first matching pair, and determine the second matching error based on the second similarity and the second SAD corresponding to the second matching pair.

[0181] 806, take the motion vector corresponding to the smaller of the first and second matching errors as the motion vector of the i-th sub-image block in the intermediate frame to be inserted.

[0182] Based on the above scheme, forward and backward intermediate frames are determined by forward motion estimation and backward motion estimation. The similarity between the forward and backward intermediate frames and the source image is compared. The matching error is determined by the similarity and the absolute error value. Then, the optimal motion vector is selected from the forward and backward motion vectors. This method can effectively improve the accuracy of motion estimation.

[0183] In some embodiments, before determining the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair, it is determined that the grayscale error between the grayscale image of the i-th sub-image block of the backward intermediate frame and the grayscale image of the i-th sub-image block of the forward intermediate frame is greater than an error threshold.

[0184] In some embodiments, determining the grayscale error between the grayscale image of the i-th sub-image block in the backward intermediate frame and the grayscale image of the i-th sub-image block in the forward intermediate frame can be achieved as follows: determining a first grayscale level histogram of the grayscale image of the i-th sub-image block in the forward intermediate frame, and a second grayscale level histogram of the grayscale image of the i-th sub-image block in the backward intermediate frame. The first grayscale histogram characterizes the number of pixels in the i-th sub-image block of the forward intermediate frame whose grayscale values ​​are distributed across M grayscale levels, and the second grayscale histogram characterizes the number of pixels in the i-th sub-image block of the backward intermediate frame whose grayscale values ​​are distributed across M grayscale levels, where M is a positive integer. As an example, the grayscale value from 0 to 255 can be divided into M levels. The grayscale range of the first grayscale level can be represented as [0, 255 / M], the grayscale range of the second grayscale level can be represented as [255 / M+1, (255 / M)×2], and so on, to determine the grayscale range corresponding to each grayscale level. When M equals 5, the first grayscale histogram and the second grayscale histogram are as follows: Figure 16 As shown. Further, the grayscale error can be determined based on the first grayscale level histogram and the second grayscale level histogram. The grayscale error satisfies the condition shown in the following formula:

[0185]

[0186] Where E represents grayscale error, a k The weight corresponding to the k-th gray level is described in the following description. This represents the number of pixels in the i-th sub-image block in the forward intermediate frame whose grayscale value is distributed at the k-th grayscale level. M represents the number of pixels in the i-th sub-image block of the backward intermediate frame whose grayscale values ​​are distributed across the k-th grayscale level, where M is a positive integer. For example, the number of pixels in the i-th sub-image block of the forward intermediate frame whose grayscale values ​​are distributed across the 5 grayscale levels are 4, 7, 5, 3, 7, respectively, while the number of pixels in the i-th sub-image block of the backward intermediate frame whose grayscale values ​​are distributed across the 5 grayscale levels are 3, 6, 7, 3, 4, respectively. In some scenarios, the grayscale error between the i-th sub-image block of the backward intermediate frame and the i-th sub-image block of the forward intermediate frame is E = 1 + 1 + 2 + 3 = 7.

[0187] In some embodiments, when it is determined that the grayscale error between the grayscale image of the i-th sub-image block in the backward intermediate frame and the grayscale image of the i-th sub-image block in the forward intermediate frame is greater than a set threshold, a first similarity between the i-th sub-image block in the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and a second similarity between the i-th sub-image block in the forward intermediate frame and the sub-image block in the first image frame of the second matching pair can be determined.

[0188] In some embodiments, when it is determined that the grayscale error between the grayscale image of the i-th sub-image block in the backward intermediate frame and the grayscale image of the i-th sub-image block in the forward intermediate frame is greater than a set threshold, and when it is determined that the first similarity between the i-th sub-image block in the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block in the forward intermediate frame and the sub-image block in the first image frame of the second matching pair, the first matching error can be determined based on the first similarity and the first SAD corresponding to the first matching pair, and the second matching error can be determined based on the second similarity and the second SAD corresponding to the second matching pair.

[0189] Taking the determination of the first matching error based on the first similarity and the first SAD corresponding to the first matching pair as an example, it can be achieved as follows: adjust the grayscale error to a first value based on the first similarity, and use the weighted sum of the first value and the first SAD corresponding to the first matching pair as the first matching error. Specifically, the larger the first similarity, the smaller the first value obtained by adjusting the grayscale error; conversely, the smaller the first similarity, the larger the first value obtained by adjusting the grayscale error. Similarly, determining the second matching error based on the second similarity and the second SAD corresponding to the second matching pair can be achieved as follows: adjust the grayscale error to a second value based on the second similarity, and use the weighted sum of the second value and the second SAD corresponding to the second matching pair as the second matching error. Likewise, the larger the second similarity, the smaller the second value obtained by adjusting the grayscale error; conversely, the smaller the second similarity, the larger the second value obtained by adjusting the grayscale error.

[0190] Furthermore, the first matching error can be determined by the weighted sum of the first value and the first SAD, and the second matching error can be determined by the weighted sum of the second value and the second SAD.

[0191] In some embodiments, when the grayscale difference is less than or equal to a set threshold, the first matching error is the first SAD corresponding to the first matching pair, and the second matching error is the second SAD corresponding to the second matching pair. In other embodiments, the first matching error is the weighted sum of the grayscale error value and the first SAD corresponding to the first matching pair, and the second matching error is the weighted sum of the grayscale error value and the second SAD corresponding to the second matching pair.

[0192] Based on the same technical concept, embodiments of this application provide a motion estimation device 1700 for video images, such as... Figure 17 As shown. The device 1700 can perform any step in the motion estimation method for video images described above; to avoid repetition, it will not be described again here. The device 1700 includes an acquisition module 1701 and a determination module 1702.

[0193] The acquisition module 1701 is used to acquire a first image frame and a second image frame, wherein the first image frame is the previous image frame adjacent to the second image frame; the first image frame and the second image frame each include N sub-image blocks;

[0194] The determination module 1702 is used to determine the backward motion vector and the first absolute error value (SAD) of each sub-image block of the first image frame, and the forward motion vector and the second SAD of each sub-image block of the second image frame.

[0195] The backward intermediate frame is determined based on the N sub-image blocks in the first image frame and the corresponding backward motion vector, and the forward intermediate frame is determined based on the N sub-image blocks in the second image frame and the corresponding forward motion vector.

[0196] Determine the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair;

[0197] Wherein, the first matching pair includes determining the k1th sub-image block in the first image frame and the k2th sub-image block in the second image frame corresponding to the backward motion vector used by the i-th sub-image block in the backward intermediate frame; the second matching pair includes determining the s1th sub-image block in the second image frame and the s2th sub-image block in the first image frame corresponding to the forward motion vector used by the i-th sub-image block in the forward intermediate frame.

[0198] A first matching error is determined based on the first similarity and the first SAD corresponding to the first matching pair, and a second matching error is determined based on the second similarity and the second SAD corresponding to the second matching pair.

[0199] The motion vector corresponding to the smaller of the first and second matching errors is used as the motion vector of the i-th sub-image block in the intermediate frame to be inserted.

[0200] In some embodiments, the determining module 1702, when determining the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, is specifically used to: obtain the image edge information and image detail information of the i-th sub-image block of the backward intermediate frame, and obtain the image edge information and image detail information of the k1-th sub-image block in the first image frame of the first matching pair;

[0201] A first edge similarity is determined between the image edge information of the i-th sub-image block and the image edge information of the k1-th sub-image block in the backward intermediate frame, and a first detail similarity is determined between the image detail information of the i-th sub-image block and the image detail information of the k1-th sub-image block. The weighted sum of the first edge similarity and the first detail similarity is the first similarity.

[0202] or,

[0203] The determining module 1702, when determining the first similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair, is specifically used for:

[0204] Obtain the image edge information and image detail information of the i-th sub-image block in the forward intermediate frame, and obtain the image edge information and image detail information of the s2-th sub-image block in the first image frame of the second matching pair;

[0205] A second edge similarity is determined between the image edge information of the i-th sub-image block and the image edge information of the s2-th sub-image block in the forward intermediate frame, and a second detail similarity is determined between the image detail information of the i-th sub-image block and the image detail information of the s2-th sub-image block. The weighted sum of the second edge similarity and the second detail similarity is the second similarity.

[0206] In some embodiments, the determining module 1702 is further configured to: determine, before determining the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair, that the grayscale error between the grayscale image of the i-th sub-image block of the backward intermediate frame and the grayscale image of the i-th sub-image block of the forward intermediate frame is greater than an error threshold;

[0207] The determining module 1702, when determining the first matching error based on the first similarity and the first SAD corresponding to the first matching pair, is specifically used for:

[0208] The grayscale error is adjusted to a first value based on the first similarity, and the weighted sum of the first value and the first SAD corresponding to the first matching pair is taken as the first matching error.

[0209] The determining module 1702, when determining the second matching error based on the second similarity and the second SAD corresponding to the second matching pair, is specifically used for:

[0210] The grayscale error is adjusted to a second value based on the second similarity, and the weighted sum of the second value and the second SAD corresponding to the second matching pair is taken as the second matching error.

[0211] In some embodiments, when the grayscale error is less than or equal to a set threshold, the first matching error is the first SAD corresponding to the first matching pair, and the second matching error is the second SAD corresponding to the second matching pair; or,

[0212] The first matching error is the weighted sum of the grayscale error and the first SAD corresponding to the first matching pair, and the second matching error is the weighted sum of the grayscale error and the second SAD corresponding to the second matching pair.

[0213] In some embodiments, when the determining module 1702 acquires the image edge information of a designated sub-image block, the designated sub-image block is either the i-th sub-image block in the forward intermediate frame, or the i-th sub-image block in the backward intermediate frame, or the k1-th sub-image block in the first image frame, or the s2-th sub-image block in the first image frame. Specifically, it is used for:

[0214] The first pixel contrast in the horizontal direction and the second pixel contrast in the vertical direction in the set sub-image block are determined; the first pixel contrast is determined based on the pixel values ​​of the pixels in the row containing the center pixel of the set sub-image block, and the second pixel contrast is determined based on the pixel values ​​of the pixels in the column containing the center pixel.

[0215] The sum of the contrast of the first pixel and the contrast of the second pixel is used as the filtering coefficient to filter the set sub-image block, thereby obtaining the edge information of the set sub-image block.

[0216] In some embodiments, when the determining module 1702 acquires the image detail information of a designated sub-image block, the designated sub-image block is the i-th sub-image block in the forward intermediate frame, or the i-th sub-image block in the backward intermediate frame, or the k1-th sub-image block in the first image frame, or the s2-th sub-image block in the first image frame, specifically used for:

[0217] Gaussian filtering is applied to the specified sub-image block to obtain the image low-level information of the specified sub-image block. The image detail information of the specified sub-image block is obtained by subtracting the image low-level information from the image information of the specified sub-image block.

[0218] In some embodiments, the determining module 1702, when determining the grayscale error between the grayscale image of the i-th sub-image block of the backward intermediate frame and the grayscale image of the i-th sub-image block of the forward intermediate frame, is specifically used for:

[0219] A first gray-level histogram of the grayscale image of the i-th sub-image block in the forward intermediate frame and a second gray-level histogram of the grayscale image of the i-th sub-image block in the backward intermediate frame are determined. The first gray-level histogram is used to characterize the number of pixels in the i-th sub-image block in the forward intermediate frame whose grayscale values ​​are distributed across M grayscale levels. The second gray-level histogram is used to characterize the number of pixels in the i-th sub-image block in the backward intermediate frame whose grayscale values ​​are distributed across M grayscale levels.

[0220] The gray level error is determined based on the first gray level histogram and the second gray level histogram;

[0221] The grayscale error satisfies the condition shown in the following formula:

[0222]

[0223] Wherein, E represents grayscale error, and a k This represents the weight corresponding to the k-th gray level. This represents the number of pixels in the i-th sub-image block of the forward intermediate frame whose grayscale values ​​are distributed at the k-th grayscale level. M represents the number of pixels in the i-th sub-image block in the backward intermediate frame whose gray values ​​are distributed at the k-th gray level, where M is a positive integer.

[0224] Based on the same technical concept, embodiments of this application provide an execution device 1800, which can implement any step of the motion estimation method for video images discussed above. Please refer to... Figure 18 The device includes a memory 1801 and a processor 1802.

[0225] The memory 1801 is used to store program instructions;

[0226] The processor 1802 is used to call the program instructions stored in the memory and execute the motion estimation method of the video image according to the obtained program.

[0227] In the embodiments of this application, the processor 1802 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0228] Memory 1801, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 1801 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 1801 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. Memory 1801 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0229] Based on the same technical concept, embodiments of this application provide a computer-readable storage medium, including: computer program code, which, when executed on a computer, causes the computer to perform the motion subset method for video images as described above. Since the principle by which the above-described computer-readable storage medium solves the problem is similar to that of the motion estimation method for video images, the implementation of the above-described computer-readable storage medium can be found in the implementation of the method, and repeated details will not be elaborated further.

[0230] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0231] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0232] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0233] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0234] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A motion estimation method for video images, characterized in that, include: Acquire a first image frame and a second image frame, wherein the first image frame is the previous image frame adjacent to the second image frame; The first image frame and the second image frame each include N sub-image blocks; Determine the backward motion vector and the first absolute error value (SAD) of each sub-image block in the first image frame, and the forward motion vector and the second SAD of each sub-image block in the second image frame; The backward intermediate frame is determined based on the N sub-image blocks in the first image frame and the corresponding backward motion vector, and the forward intermediate frame is determined based on the N sub-image blocks in the second image frame and the corresponding forward motion vector. Determine the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair; Wherein, the first matching pair includes determining the k1th sub-image block in the first image frame and the k2th sub-image block in the second image frame corresponding to the backward motion vector used by the i-th sub-image block in the backward intermediate frame; the second matching pair includes determining the s1th sub-image block in the second image frame and the s2th sub-image block in the first image frame corresponding to the forward motion vector used by the i-th sub-image block in the forward intermediate frame. A first matching error is determined based on the first similarity and the first SAD corresponding to the first matching pair, and a second matching error is determined based on the second similarity and the second SAD corresponding to the second matching pair. The motion vector corresponding to the smaller of the first and second matching errors is used as the motion vector of the i-th sub-image block in the intermediate frame to be interpolated.

2. The method as described in claim 1, characterized in that, Determining the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair includes: obtaining the image edge information and image detail information of the i-th sub-image block of the backward intermediate frame, and obtaining the image edge information and image detail information of the k1-th sub-image block in the first image frame of the first matching pair; A first edge similarity is determined between the image edge information of the i-th sub-image block and the image edge information of the k1-th sub-image block in the backward intermediate frame, and a first detail similarity is determined between the image detail information of the i-th sub-image block and the image detail information of the k1-th sub-image block. The weighted sum of the first edge similarity and the first detail similarity is the first similarity. or, Determining the first similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair includes: Obtain the image edge information and image detail information of the i-th sub-image block in the forward intermediate frame, and obtain the image edge information and image detail information of the s2-th sub-image block in the first image frame of the second matching pair; A second edge similarity is determined between the image edge information of the i-th sub-image block and the image edge information of the s2-th sub-image block in the forward intermediate frame, and a second detail similarity is determined between the image detail information of the i-th sub-image block and the image detail information of the s2-th sub-image block. The weighted sum of the second edge similarity and the second detail similarity is the second similarity.

3. The method as described in claim 2, characterized in that, The method further includes: Before determining the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair, it is determined that the grayscale error between the grayscale image of the i-th sub-image block of the backward intermediate frame and the grayscale image of the i-th sub-image block of the forward intermediate frame is greater than the error threshold. The step of determining the first matching error based on the first similarity and the first SAD corresponding to the first matching pair includes: The grayscale error is adjusted to a first value based on the first similarity, and the weighted sum of the first value and the first SAD corresponding to the first matching pair is taken as the first matching error. The second matching error is determined based on the second similarity and the second SAD corresponding to the second matching pair, including: The grayscale error is adjusted to a second value based on the second similarity, and the weighted sum of the second value and the second SAD corresponding to the second matching pair is taken as the second matching error.

4. The method as described in claim 3, characterized in that, When the grayscale error is less than or equal to a set threshold, the first matching error is the first SAD corresponding to the first matching pair, and the second matching error is the second SAD corresponding to the second matching pair; or... The first matching error is the weighted sum of the grayscale error and the first SAD corresponding to the first matching pair, and the second matching error is the weighted sum of the grayscale error and the second SAD corresponding to the second matching pair.

5. The method according to any one of claims 2-4, characterized in that, Obtain image edge information of a designated sub-image block, wherein the designated sub-image block is the i-th sub-image block in the forward intermediate frame, or the i-th sub-image block in the backward intermediate frame, or the k1-th sub-image block in the first image frame, or the s2-th sub-image block in the first image frame, including: Determine the first pixel contrast in the horizontal direction and the second pixel contrast in the vertical direction in the set sub-image block; the first pixel contrast is determined based on the pixel values ​​of the pixels included in the row where the center pixel of the set sub-image block is located, and the second pixel contrast is determined based on the pixel values ​​of the pixels included in the column where the center pixel is located; The sum of the contrast of the first pixel and the contrast of the second pixel is used as the filtering coefficient to filter the set sub-image block, thereby obtaining the edge information of the set sub-image block.

6. The method according to any one of claims 2-4, characterized in that, Obtain image detail information of a defined sub-image block, wherein the defined sub-image block is the i-th sub-image block in the forward intermediate frame, or the i-th sub-image block in the backward intermediate frame, or the k1-th sub-image block in the first image frame, or the s2-th sub-image block in the first image frame, including: Gaussian filtering is applied to the specified sub-image block to obtain the image low-level information of the specified sub-image block. The image low-level information is then subtracted from the image information of the specified sub-image block to obtain the image detail information of the specified sub-image block.

7. The method as described in claim 3 or 4, characterized in that, The determination of the grayscale error between the grayscale image of the i-th sub-image block of the backward intermediate frame and the grayscale image of the i-th sub-image block of the forward intermediate frame includes: A first gray level histogram of the grayscale image of the i-th sub-image block in the forward intermediate frame and a second gray level histogram of the grayscale image of the i-th sub-image block in the backward intermediate frame are determined. The first gray level histogram is used to characterize the number of pixels in the i-th sub-image block in the forward intermediate frame whose grayscale values ​​are distributed across M gray levels. The second gray level histogram is used to characterize the number of pixels in the i-th sub-image block in the backward intermediate frame whose grayscale values ​​are distributed across M gray levels. The gray level error is determined based on the first gray level histogram and the second gray level histogram; The grayscale error satisfies the condition shown in the following formula: Wherein, E represents grayscale error, and a k This represents the weight corresponding to the k-th gray level. This represents the number of pixels in the i-th sub-image block of the forward intermediate frame whose grayscale values ​​are distributed at the k-th grayscale level. M represents the number of pixels in the i-th sub-image block in the backward intermediate frame whose gray values ​​are distributed at the k-th gray level, where M is a positive integer.

8. A motion estimation device for video images, characterized in that, include: The acquisition module is used to acquire a first image frame and a second image frame, wherein the first image frame is the previous image frame adjacent to the second image frame; The first image frame and the second image frame each include N sub-image blocks; The determination module is used to determine the backward motion vector and the first absolute error value (SAD) of each sub-image block of the first image frame, and the forward motion vector and the second SAD of each sub-image block of the second image frame. The backward intermediate frame is determined based on the N sub-image blocks in the first image frame and the corresponding backward motion vector, and the forward intermediate frame is determined based on the N sub-image blocks in the second image frame and the corresponding forward motion vector. Determine the first similarity between the i-th sub-image block of the backward intermediate frame and the sub-image block in the first image frame of the first matching pair, and the second similarity between the i-th sub-image block of the forward intermediate frame and the sub-image block in the first image frame of the second matching pair; Wherein, the first matching pair includes determining the k1th sub-image block in the first image frame and the k2th sub-image block in the second image frame corresponding to the backward motion vector used by the i-th sub-image block in the backward intermediate frame; the second matching pair includes determining the s1th sub-image block in the second image frame and the s2th sub-image block in the first image frame corresponding to the forward motion vector used by the i-th sub-image block in the forward intermediate frame. A first matching error is determined based on the first similarity and the first SAD corresponding to the first matching pair, and a second matching error is determined based on the second similarity and the second SAD corresponding to the second matching pair. The motion vector corresponding to the smaller of the first and second matching errors is used as the motion vector of the i-th sub-image block in the intermediate frame to be interpolated.

9. An execution device, characterized in that, include: Memory, used to store program instructions; A processor is configured to acquire program instructions stored in the memory and execute the method according to any one of claims 1-7 in accordance with the acquired program instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method described in any one of claims 1-7.

Images