Video processing method, device and storage medium

Abstract

Embodiments of the present application provide a video processing method and device and a storage medium, and relate to the technical field of terminals. The method comprises: if the motion amplitude of two first images in video data is greater than or equal to a preset threshold, performing downsampling processing on the two first images respectively to obtain two second images corresponding to the two first images; generating optical flow field data corresponding to the two first images according to the optical flow field data of the two second images; generating a third image according to the optical flow field data corresponding to the two first images and the two first images; and inserting the third image between the two first images to obtain processed video data. In this way, since the electronic device performs optical flow estimation on the two second images with low resolution, the corresponding relationship between image pixels can be better estimated, the difficulty of pixel matching is reduced, a high-quality intermediate frame image is generated, the possibility of image distortion and deformation of the video screen played by the device is effectively reduced, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to video processing methods, devices and storage media. Background Technology

[0002] With the continuous development of terminal technology, electronic devices such as mobile phones, tablets, and wearable devices can provide users with a variety of applications and services. For example, electronic devices support users in installing various game applications.

[0003] However, when users use games on electronic devices, if objects in the game screen move quickly, the game screen may become distorted or warped, thus affecting the user experience. Summary of the Invention

[0004] This application provides a video processing method, device, and storage medium, which are applied in the field of terminal technology and can improve the image quality of video screens in applications such as games.

[0005] In a first aspect, embodiments of this application propose a video processing method. The method includes: acquiring video data, the video data including two consecutive first images; if the motion amplitude of the two first images is greater than or equal to a preset threshold, downsampling the two first images respectively to obtain two second images corresponding to the two first images; generating optical flow field data corresponding to the two first images based on the optical flow field data of the two second images; generating a third image based on the optical flow field data corresponding to the two first images and the two first images; inserting the third image between the two first images to obtain processed video data; and playing the processed video data.

[0006] Thus, if the motion amplitude of two adjacent first images is greater than or equal to a preset threshold, it indicates that the motion amplitude of the scene or object in the two adjacent first images is large (i.e., a large-scale motion scene). In response, the electronic device downsamples the two adjacent first images to obtain two lower-resolution second images. Then, based on the optical flow field data of the two lower-resolution second images, it obtains the optical flow field data of two higher-resolution first images. Subsequently, the electronic device fuses the optical flow field data of the two higher-resolution first images with the original resolution two first images to generate a third image (i.e., an intermediate frame image) inserted between the original resolution two first images. Since the electronic device performs optical flow estimation on the two lower-resolution second images, it can better estimate the correspondence between image pixels, reduce the difficulty of pixel matching, and generate a high-quality intermediate frame image. This effectively reduces the possibility of image distortion in the video played by the device and improves video quality.

[0007] In one possible implementation, a first parameter is determined by comparing the image features of two first images; the first parameter is used to indicate the magnitude of the motion amplitude of the two first images.

[0008] In this way, by comparing the image features of two first images to determine the first parameter (the magnitude of the motion amplitude of the two first images), we can prepare for the subsequent determination of a new image between two first images with a large motion amplitude, or a new image between two first images with a small motion amplitude.

[0009] In one possible implementation, if the motion amplitude of the two first images is greater than or equal to a preset threshold, the two first images are downsampled respectively to obtain two second images corresponding to the two first images, including: if the motion amplitude of the two first images indicated by the first parameter is greater than or equal to the preset threshold, a first scale factor is determined according to the first parameter; the two first images are downsampled respectively according to the first scale factor to obtain two second images corresponding to the two first images.

[0010] In this way, by targeting two first images with large motion amplitudes, the degree of downsampling processing of the two first images can be determined based on the magnitude of the motion amplitude. By appropriately reducing the resolution of the two first images through downsampling processing, the displacement of the two first images can be kept within a small range in subsequent optical flow estimation. This allows the corresponding pixel points in the first images to be found within a relatively small search range, thereby effectively reducing the possibility of image distortion in the video played by electronic devices and improving the user experience.

[0011] In one possible implementation, determining a first parameter by comparing the image features of two first images includes: acquiring pixel information of the two first images; determining a pixel difference value between the two first images by comparing the pixel information of the two first images; the first parameter includes the pixel difference value, which is used to indicate the magnitude of the motion amplitude of the two first images, and the pixel difference value is positively correlated with the motion amplitude.

[0012] In this way, by comparing the pixel information of two first images and determining the pixel difference between them, we can prepare for the subsequent determination of a new image between two first images with large motion amplitudes or between two first images with small motion amplitudes.

[0013] In one possible implementation, determining the pixel difference value of two first images by comparing their pixel information includes: subtracting the grayscale values ​​of the two first images to obtain the difference between them; determining the absolute value of the difference between the two first images; and determining the pixel difference value of the two first images based on the absolute value of the difference between them.

[0014] In this way, by subtracting the gray values ​​of the two first images, the difference between the two first images is obtained; the absolute value of the difference between the two first images is determined; and the pixel difference value between the two first images is determined based on the absolute value of the difference between the two first images. This can prepare for the subsequent determination of the magnitude of the motion amplitude, thereby preparing for the subsequent determination of a new image between two first images with a large motion amplitude, or a new image between two first images with a small motion amplitude.

[0015] In one possible implementation, if the motion amplitude of the two first images is greater than or equal to a preset threshold, the two first images are downsampled respectively to obtain two second images corresponding to the two first images, including: if the pixel difference value of the two first images is greater than or equal to a first threshold, the two first images are downsampled respectively to obtain two second images corresponding to the two first images.

[0016] In this way, by downsampling the two first images when the pixel difference between them is greater than or equal to a first threshold (large motion amplitude), optical flow estimation can be performed on the second image at a smaller resolution, thus preparing for obtaining a new image between the two first images.

[0017] In one possible implementation, if the pixel difference between two first images is greater than or equal to a first threshold, the two first images are downsampled to obtain two second images corresponding to the two first images, including: if the pixel difference between two first images is greater than or equal to the first threshold, a second scale factor is determined based on the pixel difference between the two first images; based on the second scale factor, the two first images are downsampled to obtain two second images corresponding to the two first images.

[0018] In this way, by targeting two first images with large motion amplitudes, a second scale factor is determined based on the pixel difference between the two first images. That is, the downsampling amplitude of the two first images is determined based on the magnitude of the motion amplitude. By appropriately reducing the resolution of the two first images through downsampling, the displacement of the two first images can be kept within a small range in subsequent optical flow estimation. This allows the corresponding pixel points in the first images to be found within a relatively small search range, thereby effectively reducing the possibility of image distortion in the video played by electronic devices and improving the user experience.

[0019] In one possible implementation, determining a first parameter by comparing feature information of two first images includes: extracting feature information of the two first images; the feature information includes at least one of the following features of the image: color, shape, texture, contrast, and brightness; determining the image similarity of the two first images based on the feature information of the two first images; the first parameter includes the image similarity, which is used to indicate the motion amplitude of the two first images, and the image similarity is negatively correlated with the motion amplitude.

[0020] In this way, by determining the motion amplitude of the two first images based on their feature information, we can prepare for the subsequent determination of a new image between two first images with large motion amplitudes, or between two first images with small motion amplitudes.

[0021] In one possible implementation, if the motion amplitude of the two first images is greater than or equal to a preset threshold, the two first images are downsampled to obtain two second images corresponding to the two first images, including: if the image similarity of the two first images is less than or equal to a second threshold, the two first images are downsampled to obtain two second images corresponding to the two first images.

[0022] In this way, by downsampling the two first images when the image similarity between the two first images is less than or equal to the second threshold (large motion amplitude), optical flow estimation can be performed on the second image at a smaller resolution, thus preparing for obtaining a new image between the two first images.

[0023] In one possible implementation, if the image similarity between two first images is less than or equal to a second threshold, the two first images are downsampled to obtain two second images corresponding to the two first images. This includes: if the image similarity between two first images is less than or equal to the second threshold, determining a third scale factor based on the image similarity between the two first images; and downsampling the two first images based on the third scale factor to obtain two second images corresponding to the two first images.

[0024] In this way, by targeting two first images with large motion amplitudes, a third scale factor can be determined based on the image similarity between the two first images. That is, the downsampling amplitude of the two first images is determined based on the magnitude of the motion amplitude. By appropriately reducing the resolution of the two first images through downsampling, the displacement of the two first images can be kept within a small range in subsequent optical flow estimation. This allows the corresponding pixels in the first images to be found within a relatively small search range, thereby effectively reducing the possibility of image distortion in the video played by electronic devices and improving the user experience.

[0025] In one possible implementation, optical flow field data of two second images can be obtained based on a preset flow estimation algorithm; optical flow field data of two first images can be generated based on the optical flow field data of the two second images, including: upsampling the optical flow field data of the two second images to obtain the optical flow field data of the two first images.

[0026] In this way, for two first images with large motion amplitudes, by upsampling the optical flow field data of the two second images, the resolution of the upsampled image is made consistent with the resolution of the first image. This allows a new image between the two first images to be generated based on the optical flow field data corresponding to the two first images and the two first images themselves.

[0027] In one possible implementation, the two first images include a fourth image and a fifth image; generating a third image based on the optical flow field data corresponding to the two first images and the two first images includes: generating a sixth image based on the optical flow field data corresponding to the fourth image and the fourth image; and generating a seventh image based on the optical flow field data corresponding to the fifth image and the fifth image; and performing image fusion on the sixth image and the seventh image to obtain the third image.

[0028] In this way, by first generating the sixth and seventh images, and then fusing the sixth and seventh images to obtain the third image, the possibility of the third image being missing or inaccurate can be reduced, the quality of the third image can be improved, and the possibility of image distortion and deformation in the video played by the device can be effectively reduced, thereby improving the user experience.

[0029] In one possible implementation, if the motion amplitude of the two first images is less than a preset threshold, optical flow field data of the two first images are obtained based on a preset flow estimation algorithm; an eighth image is generated based on the optical flow field data of the two first images and the two first images; the eighth image is inserted between the two first images to obtain processed video data; and the processed video data is played.

[0030] In this way, when the objects in the two first images undergo small-scale motion, optical flow estimation is performed on the two first images respectively to obtain the optical flow field data corresponding to the two first images. Based on the optical flow field data corresponding to the two first images and the two first images, an eighth image is determined. By interpolating the eighth image between the two first images, the frame rate of the video data can be improved, and the smoothness of the video picture can be enhanced.

[0031] In a second aspect, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory is used to store code instructions and the processor is used to execute the code instructions to perform the methods described in the first aspect or any possible implementation thereof.

[0032] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0033] Fourthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0034] Fifthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation thereof. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0035] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0036] It should be understood that the second to fifth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of a scenario for the video processing method provided in an embodiment of this application;

[0038] Figure 2 This is a schematic diagram illustrating the distortion of the game screen provided in an embodiment of this application.

[0039] Figure 3 A flowchart illustrating a video processing method provided in an embodiment of this application;

[0040] Figure 4 This is a schematic diagram of another video processing flow provided in an embodiment of this application;

[0041] Figure 5 This is a schematic diagram of another video processing procedure provided in an embodiment of this application;

[0042] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0043] Figure 7 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0044] Figure 8 This is a schematic block diagram of a video processing apparatus provided in an embodiment of this application. Detailed Implementation

[0045] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0046] 1. Optical Flow Estimation Methods

[0047] Optical flow estimation methods are used to obtain optical flow field data of an image and the corresponding optical flow confidence level.

[0048] Among them, the optical flow field data is a two-dimensional vector field that represents the optical flow of each pixel in the image. The optical flow is the velocity and direction of the pixel caused by the motion of objects in the image on the image plane.

[0049] Optical flow estimation methods assume that the brightness of two adjacent images is constant and that the motion of objects is smooth. By calculating the displacement of corresponding pixels in two adjacent images, optical flow field data of the two images is obtained, thereby describing the motion of objects in the images. For example, in a video of a running person, optical flow estimation methods can calculate the direction and velocity of motion of pixels of various parts of the human body in each image of the video. Secondly, optical flow confidence is used to indicate the reliability of the optical flow estimation result (optical flow field data). The optical flow confidence can be any value in the interval [0,1]. The closer the optical flow confidence is to 1, the more reliable the corresponding optical flow estimation result (optical flow field data) is, that is, the more the calculated optical flow of the pixels matches the actual situation; the closer the optical flow confidence is to 0, the less reliable the corresponding optical flow estimation result (optical flow field data) is, that is, the less the calculated optical flow of the pixels matches the actual situation.

[0050] 2. Interpolation methods

[0051] In the process of video frame interpolation, after obtaining the optical flow field data of two adjacent images in the video, the interpolation method can be used to determine the intermediate frame image between the two images based on the optical flow field data of the two images and the two images. The interpolation method can also be used to insert the intermediate frame image between the two adjacent images.

[0052] 3. Other terms

[0053] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0054] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0056] 4. Electronic equipment

[0057] The electronic devices in this application embodiment may include handheld devices with video processing functions, vehicle-mounted devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, in-vehicle devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0058] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0059] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0060] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0061] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0062] Electronic devices support users installing various game applications, allowing users to play games on their devices. For example, Figure 1 This is a schematic diagram of a scenario for the video processing method provided in an embodiment of this application. Figure 1 Let's take a laptop computer as an example. Figure 1 As shown in Figure 'a', the laptop displays interface 101, which is the desktop interface of the laptop and includes game icons. In response to a click on a game icon in interface 101, the laptop displays interface 102, as shown... Figure 1 As shown in b, interface 102 may include a start game icon. In response to a click on the start game icon in interface 102, the laptop can obtain video data from the game application server. After performing optical flow estimation and interpolation processing on the video data, the laptop plays the processed video data on the display screen, such as... Figure 1 As shown in c, one frame of the processed video data is displayed on interface 103.

[0063] However, when the video resolution is high and objects in the video undergo large-scale motion, traditional optical flow estimation and interpolation methods can lead to image distortion, affecting the user's visual experience and potentially causing dizziness or discomfort, significantly reducing the user experience. For example, ... Figure 2 As shown, when a running figure appears in the video (i.e., a large-scale motion scene), the background of the video may be distorted.

[0064] The reason for the above problems is that optical flow estimation and interpolation are sensitive to resolution and motion amplitude. In related technologies, optical flow estimation usually defines a specific area around the current pixel as the search range to find corresponding pixels in adjacent frames. However, when objects move at large scales in high-resolution images, the positional changes of corresponding pixels between adjacent frames are significant, exceeding the search range set by the optical flow estimation method. This can lead to errors in optical flow estimation when calculating the displacement of corresponding pixels in adjacent frames (resulting in inaccurate optical flow estimation), which in turn can cause distortion and other problems in the new frames (intermediate frame images) inserted through interpolation.

[0065] In view of this, embodiments of this application provide a video processing method, apparatus, and storage medium. The method includes: after an electronic device acquires video data, determining the motion amplitude of two consecutive images by comparing image features of two consecutive images in the video data. For two images with large motion amplitudes, the electronic device can acquire an intermediate frame image by executing Scheme 1 below; for two images with small motion amplitudes, the electronic device can acquire an intermediate frame image by executing Scheme 2 below. The two consecutive images can also be described as two adjacent images.

[0066] Option 1: When the motion amplitude of two adjacent images in the video data is greater than or equal to a preset threshold, it indicates that large-scale motion has occurred in the video frame. The electronic device can downsample the two adjacent images to obtain two lower-resolution images. Optical flow estimation is then performed on these two lower-resolution images to obtain optical flow field data corresponding to them. Subsequently, the optical flow field data corresponding to these two lower-resolution images is upsampled to obtain optical flow field data corresponding to two original-resolution images. Based on the optical flow field data corresponding to these two original-resolution images and the two original-resolution images, two intermediate frame images are determined. Image fusion is then performed on these two intermediate frame images to obtain the final intermediate frame image inserted between the two original-resolution images.

[0067] For scenes with large-scale motion in video footage, electronic devices downsample two adjacent images to obtain two lower-resolution images. Based on these two lower-resolution images, the electronic devices perform optical flow estimation, which can better estimate the correspondence between image pixels, reduce the difficulty of pixel matching, and generate high-quality intermediate frame images. This can effectively reduce the possibility of image distortion in the video footage played by the device.

[0068] Furthermore, by targeting two adjacent images with large motion amplitudes, the degree of downsampling processing for the two images can be determined based on the magnitude of the motion amplitude. By downsampling, the resolution of the two images can be appropriately reduced, so that the displacement of the two images is within a smaller range in subsequent optical flow estimation. This allows the electronic device to find the corresponding pixel in the image within a relatively small search range, thereby improving the accuracy of optical flow estimation and further reducing the possibility of image distortion in the video played by the device.

[0069] For example, when the motion amplitude is large, a larger scale factor is used for large-scale downsampling, allowing pixel displacements to be accurately captured in a lower-resolution image. This improves the accuracy of optical flow estimation, providing a foundation for subsequent operations such as optical flow-based intermediate frame image generation, and reducing image distortion and other problems. The scale factor indicates the degree to which the original image is reduced or enlarged in the spatial dimension. The scale factor can be a positive number; for example, if the scale factor is denoted as α, then the scale factor α can be 3, etc. A larger scale factor indicates a greater degree of reduction or enlargement of the original image in the spatial dimension. During downsampling, a larger scale factor indicates a smaller resolution image after downsampling; a smaller scale factor indicates a smaller degree of scaling or enlarging of the original image in the spatial dimension. During downsampling, a smaller scale factor indicates a smaller resolution image after downsampling.

[0070] Option 2: When the motion amplitude of two adjacent images in the video data is less than a preset threshold, it indicates that small-scale motion has occurred in the video frame. The electronic device can directly perform optical flow estimation on these two images to obtain the corresponding optical flow field data, without needing to downsample these two images and then perform optical flow estimation on the two downsampled lower-resolution images. Then, based on the optical flow field data corresponding to these two images, and these two images themselves, two intermediate frame images are determined. Image fusion of these two intermediate frame images yields the final intermediate frame image inserted between the two images.

[0071] Next, combined Figure 3 This application introduces a video processing method provided by embodiments thereof. For example, Figure 3 This is a flowchart illustrating a video processing method provided in an embodiment of this application. It should be understood that... Figure 3Let's take an electronic device as an example to illustrate this. Figure 3 As shown, the method includes:

[0072] S301. Acquire video data, which includes two consecutive first images.

[0073] In one possible implementation, in response to a user's action of opening the target application, the electronic device launches the target application and retrieves the video data of the target application from the target application server. The target application can be a game application or a video application, etc., and the user's action of opening the target application can be similar to clicking the "Start Game" icon on interface 102.

[0074] S302. By comparing the image features of the two first images, a first parameter is determined. The first parameter is used to indicate the magnitude of the motion amplitude of the two first images.

[0075] In some implementations, image features may include pixel information and / or feature information. Pixel information may include the position and grayscale value of a pixel, while feature information may include color, shape, texture, contrast, and brightness.

[0076] In some implementations, the first parameter can be a value in the range [0,1]. The first parameter can be positively correlated with the motion amplitude of the two first images. That is, the larger the first parameter is, the greater the motion amplitude of the two first images.

[0077] In one possible implementation, the first parameter can be positively correlated with the pixel information of the two first images, or negatively correlated with the feature information of the two first images. The first parameter can be determined by the pixel information and / or feature information of the two first images. Specific determination methods include the following:

[0078] Method 1, which determines the motion amplitude of two first images based on pixel information, can be found in the following implementation method: Figure 4 Examples are not detailed here;

[0079] Method 2: The specific implementation method for determining the motion amplitude of the two first images using feature information can be found in [reference needed]. Figure 5 Examples are not detailed here.

[0080] Method 3: Determine the motion amplitude of the two first images using pixel information and feature information. Specifically, this can be achieved by: obtaining the motion amplitude S1 of the two first images determined by their pixel information; obtaining the motion amplitude S2 of the two first images determined by their feature information; and obtaining the final motion amplitude S of the two first images using a preset algorithm. This application does not limit the preset algorithm; for example, S can be calculated using the following formula:

[0081] S = 0.5S1 + 0.5S2

[0082] S303. Determine whether the motion amplitude of the two first images indicated by the first parameter is greater than or equal to a preset threshold.

[0083] In some implementations, the preset threshold can be a value in the range [0,1]. Taking a preset threshold of 0.6 as an example, if the first parameter is 0.8, which is greater than the preset threshold, it indicates that the motion amplitude of the two first images is large, that is, the objects in the picture are undergoing large-scale motion.

[0084] In some implementations, it is determined whether the motion amplitude of the two first images indicated by the first parameter is greater than or equal to a preset threshold. If yes, S304 is executed; otherwise, S311 is executed.

[0085] S304. Determine the first scale factor based on the first parameter.

[0086] In some implementations, the first scaling factor can be a value greater than 1. In some possible implementations, the first parameter can be positively correlated with the first scaling factor; that is, the larger the first parameter, the larger the first scaling factor.

[0087] In some possible implementations, the first scale factor can be determined based on the first parameter using a preset first mapping table. The preset first mapping table is used to indicate the correspondence between the first parameter and the first scale factor. For example, the preset first mapping table can be as shown in Table 1. Referring to Table 1, if the first parameter is 0.8, then the first scale factor is 1.8.

[0088] It should be understood that the first parameter can be any value other than those listed in Table 1 (such as 0.5). This application embodiment does not limit this, as long as the preset first mapping table satisfies that the first parameter is positively correlated with the first scale factor.

[0089] Table 1

[0090] First parameter First scale factor 0.2 1.2 0.4 1.4 0.6 1.6 0.8 1.8 1.0 2.0

[0091] It should be understood that, in addition to determining the first scale factor based on the first parameter through a preset first mapping table, the first scale factor can also be calculated through a preset function or formula, wherein the independent variable of the function is the first parameter, and the dependent variable of the function is the first scale factor. It should be understood that the embodiments of this application do not limit the specific implementation method for determining the first scale factor based on the first parameter.

[0092] S305. Based on the first scale factor, downsample the two first images respectively to obtain two second images corresponding to the two first images.

[0093] In one possible implementation, two second images corresponding to the two first images can be obtained based on the first scale factor and the two first images. The two second images can be represented by the following formulas:

[0094] I0' = Resize(I0, 1 / α)

[0095] I1' = Resize(I1, 1 / α)

[0096] Where α represents the first scale factor, I0 and I1 represent two first images respectively, the Resize function is used to adjust the size of the first image I0 and the first image I1 respectively, and the Resize function performs magnification or reduction operation on the input first image according to the first scale factor to obtain a new second image with a different scale.

[0097] When α is greater than 1, Resize(I0,1 / α) can shrink the first image I0 (the same applies to I1); when α is less than 1, Resize(I0,1 / α) can enlarge the first image I0 (the same applies to I1); when α is equal to 1, Resize(I0,1 / α) can leave the scale of the first image I0 unchanged (the same applies to I1).

[0098] As the formula shows, the larger the first scaling factor α, the smaller 1 / α, indicating a greater degree of scaling of the first image, which means a smaller scale of the second image. For example, when α = 2, the scale of I0' is 1 / 2 of the scale of I0; when α = 3, the scale of I0' is 1 / 3 of the scale of I0. Therefore, it can be seen that the scale of I0' when α = 3 is smaller than the scale of I0' when α = 2.

[0099] In this embodiment of the application, the greater the motion amplitude, the larger the first parameter, the larger the first scale factor, and the smaller the resolution of the two second images obtained by downsampling the two first images.

[0100] In this embodiment, determining the first scale factor based on the first parameter can be understood as determining the first scale factor based on the magnitude of the motion amplitude of the two first images. That is, determining the degree of downsampling of the two first images based on the magnitude of the motion amplitude. By downsampling (determining the two second images), the resolution of the two first images can be appropriately reduced, so that when performing optical flow estimation on the two second images, the displacement of the two second images is within a small range. This allows the electronic device to find the corresponding pixels in the second images within a relatively small search range, improving the accuracy of optical flow estimation. This further reduces the possibility of image distortion in the video played by the device, thereby improving the user experience.

[0101] S306. Perform optical flow estimation on the two second images respectively to obtain the optical flow field data corresponding to the two second images.

[0102] In one implementation, the positional changes (including motion direction and velocity) of each pixel in the second image 1 corresponding to the pixel in the second image 2 can be determined, thereby determining the optical flow of each pixel in the second image 1, and thus determining one optical flow field data corresponding to the two second images. Conversely, the positional changes (including motion direction and velocity) of each pixel in the second image 2 corresponding to the pixel in the second image 1 can also be determined, thereby determining the optical flow of each pixel in the second image 2, and thus determining another optical flow field data corresponding to the two second images.

[0103] For example, if the two second images are I0' and I1' respectively, then the optical flow field data F0' corresponding to I0' and I1' indicates the position change (including the direction of movement and speed) of each pixel in I0' and the corresponding pixel in I1'. For example, if the pixel with coordinates (x0', y0') in I0' has coordinates (x1', y1') in I1', then the direction of movement of the pixel is from (x0', y0') to (x1', y1'), and the speed of movement of the pixel can be expressed as the distance from (x0', y0') to (x1', y1') divided by the time interval between the two second images I0' and I1'.

[0104] For example, if the two second images are I0' and I1' respectively, then another optical flow field data F1' corresponding to I0' and I1' indicates the position change (including movement direction and speed) of each pixel in I1' and the corresponding pixel in I0'. For example, the pixel with coordinates (x1', y1') in I1' has coordinates (x0', y0') in I0', then the movement direction of the pixel is from (x1', y1') to (x0', y0'), and the movement speed of the pixel can be expressed as the distance from (x1', y1') to (x0', y0') divided by the time interval between the two second images I0' and I1'.

[0105] S307. Upsample the optical flow field data corresponding to the two second images according to the first scale factor to obtain the optical flow field data corresponding to the two first images.

[0106] In one possible implementation, the optical flow field data corresponding to the two first images can be obtained based on the first scale factor and the optical flow field data corresponding to the two second images. The optical flow field data corresponding to the two first images can be expressed by the following formula:

[0107] F0 = Resize(F0', α)

[0108] F1 = Resize(F1', α)

[0109] Where α represents the first scale factor, F0' represents one optical flow field data corresponding to the two second images, and F1' represents another optical flow field data corresponding to the two second images. The Resize function can scale up or down the optical flow field data corresponding to the second image according to the first scale factor to obtain a new optical flow field data corresponding to the first image at a different scale.

[0110] When α is greater than 1, Resize(F0',α) can be used to enlarge the optical flow field data F0' corresponding to the second image (the same applies to F1'); when α is less than 1, Resize(F0',α) can be used to shrink the optical flow field data F0' corresponding to the second image (the same applies to F1'); when α is equal to 1, Resize(F0',α) can leave the scale of the optical flow field data F0' corresponding to the second image unchanged (the same applies to F1').

[0111] S308. Determine the third image based on the optical flow field data corresponding to the two first images and the two first images.

[0112] In one possible implementation, the sixth image can be determined based on the optical flow field data F0 corresponding to the first image and the first image I0; at the same time, the seventh image can be determined based on another optical flow field data F1 corresponding to the first image and the first image I1. The sixth image and the seventh image can be fused to obtain the third image.

[0113] In one example, the sixth and seventh images can be fused using a pre-defined neural network. The pre-defined neural network can fuse the two images based on auxiliary information, which may include optical flow confidence and the similarity between the sixth and seventh images.

[0114] S309. Insert a third image between the two first images to obtain the processed video data.

[0115] In some implementations, a third image can be inserted between two first images using interpolation to obtain processed video data. Inserting a third image between the two first images increases the frame rate of the video data, resulting in smoother playback.

[0116] S310, Play back the processed video data.

[0117] In some implementations, the processed video data is played on the display screen of an electronic device after it has been obtained.

[0118] S311. Perform optical flow estimation on the two first images respectively to obtain the optical flow field data corresponding to the two first images.

[0119] In one implementation, the positional changes (including movement direction and velocity) of each pixel in the first image 1 corresponding to the pixel in the first image 2 can be determined, thereby determining the optical flow of each pixel in the first image 1, and thus determining one optical flow field data corresponding to the two first images. Conversely, the positional changes (including movement direction and velocity) of each pixel in the first image 2 corresponding to the pixel in the first image 1 can also be determined, thereby determining the optical flow of each pixel in the first image 2, and thus determining another optical flow field data corresponding to the two first images.

[0120] For example, if the two first images are I0 and I1 respectively, then the optical flow field data F0 corresponding to I0 and I1 indicates the position change (including the direction of movement and speed) of each pixel in I0 and the corresponding pixel in I1. For example, if the pixel with coordinates (x0, y0) in I0 has coordinates (x1, y1) in I1, then the direction of movement of the pixel is from (x0, y0) to (x1, y1), and the speed of movement of the pixel can be expressed as the distance from (x0, y0) to (x1, y1) divided by the time interval between the two first images I0 and I1.

[0121] For example, if the two first images are I0 and I1 respectively, then another optical flow field data F1 corresponding to I0 and I1 indicates the position change (including movement direction and speed) of each pixel in I1 and the corresponding pixel in I0. For example, if the pixel with coordinates (x1, y1) in I1 has coordinates (x0, y0) in I0, then the movement direction of the pixel is from (x1, y1) to (x0, y0), and the movement speed of the pixel can be expressed as the distance from (x1, y1) to (x0, y0) divided by the time interval between the two first images I0 and I1.

[0122] S312. Determine the eighth image based on the optical flow field data corresponding to the two first images and the two first images.

[0123] In one possible implementation, the ninth image can be determined based on the optical flow field data F0 corresponding to the first image and the first image I0; at the same time, the tenth image can be determined based on another optical flow field data F1 corresponding to the first image and the first image I1. The ninth image and the tenth image can be fused to obtain the eighth image.

[0124] S313. Insert the eighth image between the two first images to obtain the processed video data.

[0125] S314. Play back the processed video data.

[0126] In some implementations, the specific implementation of S313-S314 can refer to S309-S310, which will not be repeated here. In the embodiments of this application, when the objects in the two first images undergo small-scale motion, optical flow estimation is performed on the two first images respectively to obtain optical flow field data corresponding to the two first images. Based on the optical flow field data and the two first images, an eighth image (intermediate frame image) is determined, and the eighth image is inserted between the two first images by interpolation.

[0127] In this embodiment, if the motion amplitude of two adjacent first images is greater than or equal to a preset threshold, it indicates that the motion amplitude of the scene or object in the two adjacent first images is large (i.e., a large-scale motion scene). In response, the electronic device downsamples the two adjacent first images to obtain two lower-resolution second images. Then, based on the optical flow field data of the two lower-resolution second images, it obtains optical flow field data for two higher-resolution first images. Subsequently, the electronic device fuses the optical flow field data of the two higher-resolution first images with the original resolution two first images to generate a third image (i.e., an intermediate frame image) inserted between the original resolution two first images. Because the electronic device uses two lower-resolution second images for optical flow estimation, it can better estimate the correspondence between image pixels, reducing the difficulty of pixel matching and generating a high-quality intermediate frame image. This effectively reduces the possibility of image distortion in the video playback.

[0128] based on Figure 3 In this embodiment, the first parameter can be the pixel difference value between two first images. The pixel difference value can be used to indicate the magnitude of motion between the two first images, and the pixel difference value is positively correlated with the magnitude of motion between the two first images. Next, in conjunction with... Figure 4 This section describes the specific possible implementations of determining the motion amplitude of two first images using pixel information from the two first images.

[0129] For example, Figure 4 This is a schematic diagram of another video processing procedure provided in an embodiment of this application. (Refer to...) Figure 4 The method may include:

[0130] S401. Acquire video data, which includes two consecutive first images.

[0131] In some implementations, the specific implementation of S401 can refer to S301, which will not be repeated here.

[0132] S402. Obtain the pixel information of the two first images.

[0133] In some implementations, if the two first images are grayscale images, each pixel of the two first images can be traversed to obtain the grayscale value of each pixel in the two first images respectively.

[0134] In some possible implementations, if the two first images are color images, they can be converted into grayscale images first by using color space conversion algorithms, and then each pixel of the two first images can be traversed to obtain the grayscale value of each pixel in the two first images respectively.

[0135] S403. By comparing the pixel information of the two first images, the pixel difference value of the two first images is determined. The pixel difference value is used to indicate the magnitude of the motion amplitude of the two first images.

[0136] In some implementations, the pixel difference value can be a value greater than 0, such as 150. The larger the pixel difference value between the two first images, the greater the motion amplitude of the two first images; conversely, the smaller the pixel difference value between the two first images, the smaller the motion amplitude of the two first images.

[0137] In some implementations, the pixel information (grayscale values) of two first images can be subtracted to obtain the difference between the two first images; then the difference between the two first images can be determined; then the absolute values ​​of the differences between the two first images can be added together, and then the sum can be divided by the total number of pixels in the first image to determine the pixel difference value between the two first images.

[0138] For example, if the two first images are I0 and I1 respectively, assuming that both first images are grayscale images, for the (i,j)th pixel in the first image, its grayscale value in the first image I0 is I0(i,j), and its grayscale value in the first image I1 is I1(i,j). The difference between I0 and I1 is I0(i,j)-I1(i,j), and the absolute value of the difference between I0 and I1 is |I0(i,j)-I1(i,j)|. Then, the absolute values ​​of the differences of each pixel are added together to obtain the sum, and then the sum is divided by the total number of pixels in the first image to determine the pixel difference between the two first images.

[0139]

[0140] Where m is the number of pixels in each row of the first image, n is the number of pixels in each column of the first image, i represents the row index of the first image, and j represents the column index of the first image.

[0141] S404. Determine whether the pixel difference between the two first images is greater than or equal to the first threshold.

[0142] In some implementations, the first threshold is a value greater than 0. If the pixel difference between two first images is greater than or equal to the first threshold, it indicates that the motion amplitude of the two first images is large, that is, the objects in the picture are undergoing large-scale motion.

[0143] In some implementations, it is determined whether the pixel difference between the two first images is greater than or equal to a first threshold. If yes, S405 is executed; otherwise, S412 is executed.

[0144] S405. Determine the second scale factor based on the pixel difference between the two first images.

[0145] In some implementations, the second scale factor can be a value greater than 1. In some possible implementations, the pixel difference between the two first images can be positively correlated with the second scale factor; that is, the greater the pixel difference between the two first images, the larger the second scale factor.

[0146] In some possible implementations, a second scale factor can be determined based on the pixel difference values ​​of two first images using a preset second mapping table. The preset second mapping table indicates the correspondence between the pixel difference values ​​of the two first images and the second scale factor. For example, the preset second mapping table can be as shown in Table 2. Referring to Table 2, if the pixel difference value of the two first images is 50, then the second scale factor is 1.2. It should be understood that the pixel difference value of the two first images can be other values ​​besides those listed in Table 2 (e.g., 120). This application embodiment does not limit this; the preset second mapping table only needs to satisfy the positive correlation between the pixel difference values ​​of the two first images and the second scale factor.

[0147] Table 2

[0148] Pixel difference between the two first images Second scale factor 50 1.2 100 1.4 150 1.6 200 1.8

[0149] It should be understood that, in addition to determining the second scale factor based on the pixel difference values ​​of two first images through a preset second mapping table, the second scale factor can also be calculated through a preset function or formula, wherein the independent variable of the function is the pixel difference values ​​of the two first images, and the dependent variable of the function is the second scale factor. It should be understood that the embodiments of this application do not limit the specific implementation method for determining the second scale factor based on the pixel difference values ​​of two first images.

[0150] S406. Based on the second scale factor, downsample the two first images respectively to obtain two second images corresponding to the two first images.

[0151] S407. Perform optical flow estimation on the two second images respectively to obtain the optical flow field data corresponding to the two second images.

[0152] S408. Upsample the optical flow field data corresponding to the two second images according to the second scale factor to obtain the optical flow field data corresponding to the two first images.

[0153] S409. Determine the third image based on the optical flow field data corresponding to the two first images and the two first images.

[0154] S410. Insert a third image between the two first images to obtain the processed video data.

[0155] S411, Play the processed video data.

[0156] S412. Perform optical flow estimation on the two first images respectively to obtain the optical flow field data corresponding to the two first images.

[0157] S413. Determine the eighth image based on the optical flow field data corresponding to the two first images and the two first images.

[0158] S414. Insert the eighth image between the two first images to obtain the processed video data.

[0159] S415, Play back the processed video data.

[0160] In some implementations, the specific implementations of S406-S415 can refer to S305-S314, and will not be repeated here. In the embodiments of this application, if the pixel difference value of two adjacent first images is greater than or equal to a first threshold, it indicates that the scene or object in the two adjacent first images has a large motion amplitude (i.e., a large-scale motion scene). In this regard, the electronic device performs downsampling processing on the two adjacent first images to obtain two second images with lower resolution. Then, it obtains optical flow field data of two first images with higher resolution based on the optical flow field data of the two second images with lower resolution. Subsequently, the electronic device performs fusion processing on the optical flow field data of the two first images with higher resolution and the two first images with original resolution to generate a third image (i.e., an intermediate frame image) inserted between the two first images with original resolution. Since the electronic device uses two second images with lower resolution when performing optical flow estimation, it can better estimate the correspondence between image pixels, reduce the difficulty of pixel matching, and generate a high-quality intermediate frame image, which can effectively reduce the possibility of image distortion in the video played by the device.

[0161] Based on the foregoing Figure 3 In this embodiment, the first parameter can be the image similarity between two first images. The image similarity between two first images can be used to characterize the magnitude of motion between the two first images. The image similarity between two first images is negatively correlated with the magnitude of motion between the two first images; the greater the image similarity between two first images, the smaller the magnitude of motion between the two first images. The following is combined with... Figure 5 This section describes the specific possible implementations of determining the motion amplitude of two first images using feature information from the two first images.

[0162] For example, Figure 5 This is a schematic diagram illustrating another video processing procedure provided in an embodiment of this application. (Refer to...) Figure 5 The method may include:

[0163] S501. Acquire video data, which includes two consecutive first images.

[0164] In some implementations, the specific implementation of S501 can refer to S301, which will not be repeated here.

[0165] S502. Extract feature information from the two first images.

[0166] In some implementations, the structural similarity index measure (SSIM) can be used to divide the two first images into multiple local regions, and the brightness, contrast, and structural features of each region can be calculated. In other implementations, a convolutional neural network (CNN) can be used to extract feature information from the two first images.

[0167] S503. Based on the feature information of the two first images, determine the image similarity between the two first images. The image similarity is used to indicate the motion amplitude of the two first images.

[0168] In some implementations, image similarity can be a value in the interval [0,1], for example, 0.6. The image similarity between two first images is negatively correlated with the motion amplitude of the two first images. That is, the smaller the image similarity between two first images, the greater the motion amplitude of the two first images, and vice versa.

[0169] In some implementations, the motion amplitude of two first images can be determined using the SSIM method. For example, for each local region of the two first images, the similarity score of each local region is calculated according to the SSIM method. Then, the similarity scores of all local regions are averaged to obtain the image similarity between the two first images.

[0170] S504. Determine whether the image similarity between the two first images is less than or equal to the second threshold.

[0171] In some implementations, the second threshold is a value in the interval [0,1]. If the similarity between two first images is less than or equal to the second threshold, it indicates that the motion amplitude of the two first images is large, that is, the objects in the picture are undergoing large-scale motion.

[0172] S505. Determine the third scale factor based on the image similarity between the two first images.

[0173] In some implementations, the third scale factor can be a value greater than 1. In some possible implementations, the image similarity between two first images can be negatively correlated with the third scale factor; that is, the smaller the image similarity between two first images, the larger the third scale factor.

[0174] In some possible implementations, a third scale factor can be determined based on the image similarity of two first images using a preset third mapping table. The preset third mapping table indicates the correspondence between the image similarity of two first images and the third scale factor. For example, the preset third mapping table can be as shown in Table 3. Referring to Table 3, if the image similarity of two first images is 0.8, then the third scale factor is 1.2. It should be understood that the image similarity of two first images can be other values ​​besides those listed in Table 3. This application embodiment does not limit this; the preset third mapping table only needs to satisfy the negative correlation between the image similarity of two first images and the third scale factor.

[0175] Table 3

[0176] Image similarity between the two first images Third scale factor 0.8 1.2 0.6 1.4 0.4 1.6 0.2 1.8

[0177] It should be understood that, in addition to determining the third scale factor based on the image similarity of two first images through a preset third mapping table, the third scale factor can also be calculated through a preset function or formula, where the independent variable of the function is the image similarity of the two first images, and the dependent variable of the function is the third scale factor. It should be understood that the specific implementation method for determining the third scale factor based on the image similarity of two first images in this application is not limited. S506: Downsampling processing is performed on the two first images according to the third scale factor to obtain two second images corresponding to the two first images.

[0178] S507. Perform optical flow estimation on the two second images respectively to obtain the optical flow field data corresponding to the two second images.

[0179] S508. Upsample the optical flow field data corresponding to the two second images according to the third scale factor to obtain the optical flow field data corresponding to the two first images.

[0180] S509. Determine the third image based on the optical flow field data corresponding to the two first images and the two first images.

[0181] S510. Insert a third image between the two first images to obtain the processed video data.

[0182] S511, Play back the processed video data.

[0183] S512. Perform optical flow estimation on the two first images respectively to obtain the optical flow field data corresponding to the two first images.

[0184] S513. Determine the eighth image based on the optical flow field data corresponding to the two first images and the two first images.

[0185] S514. Insert the eighth image between the two first images to obtain the processed video data.

[0186] S515, Play back the processed video data.

[0187] In some implementations, the specific implementation methods of S506-S515 can refer to S305-S314, which will not be repeated here.

[0188] In this embodiment, if the image similarity between two adjacent first images is less than or equal to a second threshold, it indicates that the scene or object in the two adjacent first images has a large motion amplitude (i.e., a large-scale motion scene). To address this, the electronic device downsamples the two adjacent first images to obtain two second images with lower resolution. Then, it upsamples the optical flow field data of the two lower-resolution second images to obtain optical flow field data of two higher-resolution first images. Subsequently, the electronic device fuses the optical flow field data of the two higher-resolution first images with the original resolution two first images to generate a third image (i.e., an intermediate frame image) inserted between the original resolution two first images. Because the electronic device uses two lower-resolution second images for optical flow estimation, it can better estimate the correspondence between image pixels, reducing the difficulty of pixel matching and generating a high-quality intermediate frame image. This effectively reduces the possibility of image distortion in the video playback.

[0189] It should be noted that the video processing method provided in this application is not only applicable to the gaming field, but can also be applied to other scenarios that require improved video quality. For example, it can enable slow-motion replay in sports broadcasts; enhance visual effects and make video smoother in short video creation and film post-production; improve the quality of medical images in telemedicine; provide richer visual information in autonomous driving; improve the quality of video and make video smoother in security monitoring; improve image quality in virtual reality (VR) and augmented reality (AR); and enhance the image in low-bandwidth environments in cloud gaming and streaming services, as well as improve real-time ray tracing rendering effects and large-scale scene rendering effects, etc., which will not be elaborated upon in this application.

[0190] The video processing method provided in this application can be implemented using an electronic device. To facilitate understanding of the video processing method provided in this application, the hardware structure of the electronic device provided in this application is described below.

[0191] For example, Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface, etc.

[0192] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may include hardware, software, or a combination of software and hardware.

[0193] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0194] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0195] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the aforementioned memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0196] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc.

[0197] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0198] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc. The data storage area may store data created during the use of the electronic device, etc. Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in internal memory 121 and / or instructions stored in memory disposed within the processor.

[0199] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of an electronic device.

[0200] Figure 7This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application. The layered architecture divides the software of the electronic device into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, the hardware abstraction layer (HAL), and the kernel layer.

[0201] The application layer can include a series of application packages. Application packages can include applications such as games, photo galleries, short videos, and video clips.

[0202] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0203] In this embodiment, the application framework layer includes a video processing service. The video processing service provides an interface for the application layer's application, enabling the application to call the underlying video processing module through the video processing service.

[0204] In some embodiments, the application framework layer may also include, for example, an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, etc., and this application embodiment does not impose any limitations on this.

[0205] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0206] The Android runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries consist of two parts: one part contains the functionalities that Java needs to call, and the other part contains the core Android libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer into binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0207] The HAL layer is a wrapper around Linux kernel drivers, providing an interface to the upper layers and shielding them from the implementation details of the lower-level hardware. In this embodiment, the HAL layer may include a video processing module. The video processing module can obtain video data from the application layer, execute the steps of the aforementioned embodiments, insert intermediate frame images between every two adjacent images in the video data to obtain processed video data, and then return the processed video data to the application layer, which plays the video frame corresponding to the processed video data.

[0208] It should be noted that the video processing module is not limited to the hardware abstraction layer, but can also be in other layers, such as the system library or the framework layer. This application does not limit this.

[0209] In some embodiments, the HAL layer may also include a Wi-Fi HAL, an audio HAL, a camera HAL, etc.

[0210] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, audio drivers, and sensor drivers.

[0211] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0212] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0213] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the method steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0214] The embodiments of this application can divide the apparatus for implementing the method into functional modules based on the above method examples. Figure 8This is a schematic block diagram of a video processing apparatus provided in an embodiment of this application. (Refer to...) Figure 8 The video processing device 800 includes a processor 801, a communication interface 802, and a memory 803. The processor 801, communication interface 802, and memory 803 communicate with each other via internal connections. The memory 803 stores instructions, and the processor 801 executes the instructions stored in the memory 803. The communication interface 802 can be used to send signals to other devices (such as the processor 801 or a touchscreen of an electronic device) and to receive signals from other devices (such as the memory 803). For example, the communication interface 802 reads instructions stored in the memory 803 and sends those instructions to the processor 801.

[0215] It should be understood that the device 800 may specifically be an electronic device as described in the above embodiments, and may be used to execute the various steps and / or processes corresponding to the electronic device in the above method embodiments. Optionally, the memory 803 may include read-only memory and random access memory, and provide instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information. The processor 801 may be used to execute instructions stored in the memory, and when the processor 801 executes instructions stored in the memory, the processor 801 is used to execute the various steps and / or processes of the above method embodiments.

[0216] It should be understood that, in the embodiments of this application, the processor may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0217] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or as a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor executes the instructions in the memory, combining them with its hardware to complete the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0218] The video processing method provided in this application can be applied to electronic devices with video processing capabilities. The electronic devices include terminal devices, and the specific device form of the terminal devices can be referred to the above-described related descriptions, which will not be repeated here.

[0219] This application provides an electronic device, which includes a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the electronic device to perform the above-described method.

[0220] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0221] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0222] In one possible implementation, a computer-readable medium may include random access memory (RAM), read-only memory (ROM), compact discread-only memory (CD-ROM) or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium intended to carry or store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0223] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0224] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0225] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video processing method, characterized in that, Applied to electronic devices, the method includes: Acquire video data, the video data comprising two consecutive first images; If the motion amplitude of the two first images is greater than or equal to a preset threshold, the two first images are downsampled respectively to obtain two second images corresponding to the two first images; Optical flow field data corresponding to the two first images are generated based on the optical flow field data of the two second images; A third image is generated based on the optical flow field data corresponding to the two first images and the two first images themselves; The third image is inserted between the two first images to obtain processed video data; Play the processed video data.

2. The method according to claim 1, characterized in that, The method further includes: A first parameter is determined by comparing the image features of the two first images; the first parameter is used to indicate the magnitude of the motion amplitude of the two first images.

3. The method according to claim 2, characterized in that, If the motion amplitude of the two first images is greater than or equal to a preset threshold, the two first images are downsampled to obtain two second images corresponding to the two first images, including: If the motion amplitude of the two first images indicated by the first parameter is greater than or equal to a preset threshold, a first scale factor is determined according to the first parameter. The two first images are downsampled according to the first scale factor to obtain two second images corresponding to the two first images.

4. The method according to claim 2 or 3, characterized in that, The step of determining the first parameter by comparing the image features of the two first images includes: Obtain the pixel information of the two first images; The pixel difference value between the two first images is determined by comparing the pixel information of the two first images. The first parameter includes the pixel difference value, which is used to indicate the magnitude of the motion amplitude of the two first images, and the pixel difference value is positively correlated with the motion amplitude.

5. The method according to claim 4, characterized in that, The step of determining the pixel difference value between the two first images by comparing their pixel information includes: The difference between the two first images is obtained by subtracting their grayscale values. Determine the absolute value of the difference between the two first images; The pixel difference value of the two first images is determined based on the absolute value of the difference between them.

6. The method according to any one of claims 1 to 5, characterized in that, If the motion amplitude of the two first images is greater than or equal to a preset threshold, the two first images are downsampled to obtain two second images corresponding to the two first images, including: If the pixel difference between the two first images is greater than or equal to a first threshold, the two first images are downsampled to obtain two second images corresponding to the two first images.

7. The method according to claim 6, characterized in that, If the pixel difference between the two first images is greater than or equal to a first threshold, downsampling is performed on the two first images respectively to obtain two second images corresponding to the two first images, including: If the pixel difference between the two first images is greater than or equal to the first threshold, a second scale factor is determined based on the pixel difference between the two first images. Based on the second scale factor, the two first images are downsampled to obtain two second images corresponding to the two first images.

8. The method according to claim 2 or 3, characterized in that, The step of determining the first parameter by comparing the feature information of the two first images includes: Extract feature information from the two first images; the feature information includes at least one of the following features of the image: color, shape, texture, contrast, and brightness; Based on the feature information of the two first images, determine the image similarity between the two first images; The first parameter includes the image similarity, which is used to indicate the motion amplitude of the two first images, and the image similarity is negatively correlated with the motion amplitude.

9. The method according to any one of claims 1 to 3, or 8, characterized in that, If the motion amplitude of the two first images is greater than or equal to a preset threshold, the two first images are downsampled to obtain two second images corresponding to the two first images, including: If the image similarity between the two first images is less than or equal to the second threshold, the two first images are downsampled to obtain two second images corresponding to the two first images.

10. The method according to claim 9, characterized in that, If the image similarity between the two first images is less than or equal to a second threshold, the two first images are downsampled to obtain two second images corresponding to the two first images, including: If the image similarity between the two first images is less than or equal to the second threshold, a third scale factor is determined based on the image similarity between the two first images; Based on the third scale factor, the two first images are downsampled to obtain two second images corresponding to the two first images.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: acquiring optical flow field data of the two second images respectively based on a preset flow estimation algorithm; The step of generating optical flow field data corresponding to the two first images based on the optical flow field data of the two second images includes: performing upsampling processing on the optical flow field data of the two second images respectively to obtain the optical flow field data corresponding to the two first images.

12. The method according to any one of claims 1 to 11, characterized in that, The two first images include a fourth image and a fifth image; the step of generating a third image based on the optical flow field data corresponding to the two first images and the two first images includes: A sixth image is generated based on the optical flow field data corresponding to the fourth image and the fourth image; and a seventh image is generated based on the optical flow field data corresponding to the fifth image and the fifth image. The sixth image and the seventh image are fused together to obtain the third image.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: If the motion amplitude of the two first images is less than a preset threshold, optical flow field data of the two first images are obtained respectively based on a preset flow estimation algorithm. An eighth image is generated based on the optical flow field data of the two first images and the two first images themselves; The eighth image is inserted between the two first images to obtain processed video data; Play the processed video data.

14. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the method as described in any one of claims 1 to 13.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 13.

16. A chip system, characterized in that, It includes at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being configured to run a computer program or instructions to perform the method as described in any one of claims 1 to 13.

17. A computer program product, characterized in that, Includes a computer program that, when run, causes a computer to perform the method as described in any one of claims 1 to 13.

Images