Video processing method, device, computer equipment and storage medium

By determining the motion mask and optical flow between video frames, selecting pixels with the largest pixel value to update the video frame, the problems of low efficiency and large memory usage in the realization of long-exposure time-lapse photography effects in the existing technology are solved, and high efficiency is achieved video processing and memory saving.

CN116193279BActive Publication Date: 2024-11-15ARASHI VISION INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211705293.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-29
Publication Date
2024-11-15
Estimated Expiration
2042-12-29

AI Technical Summary

Technical Problem

When the existing video processing methods realize the long exposure time-lapse photography effect, the interpolation and frame stacking process consume a lot of time, resulting in low efficiency and large memory space.

Method used

By determining the motion mask and optical flow between video frames, selecting the pixel point with the largest pixel value as the target pixel point, updating the target video frame, realizing the long exposure time-lapse photography effect without additional frame insertion, reducing the computing amount and saving memory.

Benefits of technology

Improves the efficiency and effectiveness of video processing, saves memory space, and avoids additional frame insertion steps.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116193279B_ABST
    Figure CN116193279B_ABST
Patent Text Reader

Abstract

The present application relates to a video processing method, device, computer equipment and storage medium. The method includes: determining a first motion mask between adjacent video frames in a plurality of video frames of a video; determining a first motion region and a second motion region of adjacent video frames according to the first motion mask, and calculating the optical flow between the first motion region and the second motion region; determining the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion region and the optical flow; the candidate point is determined based on the pixel points selected based on the optical flow when the optical flow is greater than a first threshold; among the pixel points corresponding to each candidate point and the pixel points at the corresponding positions in a plurality of video frames, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point. The method has a fast processing speed and can achieve a good long-exposure time-lapse photography effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a video processing method, apparatus, computer equipment and storage medium. Background Art

[0002] As people's living standards improve, they have higher and higher requirements for video quality, and they also want videos to present a variety of cool special effects. For example, for a normally shot video of traffic, after post-processing with the special effect of tailing of car lights, the effect of long-exposure time-lapse photography can be achieved. In order to achieve the effect of long-exposure time-lapse photography, the current method usually adopted is to first insert frames into the original video, and then use frame stacking for tailing, but video insertion will consume a lot of time. At the same time, storing the inserted video frames before frame stacking will also dramatically increase the memory space occupied, resulting in low efficiency and poor effect of video processing. Summary of the invention

[0003] Based on this, it is necessary to provide a video processing method, device, computer equipment and computer-readable storage medium with fast processing speed and capable of achieving good long-exposure time-lapse photography effects in response to the above technical problems.

[0004] The present application provides a video processing method. The method comprises:

[0005] determining a first motion mask between adjacent video frames in a plurality of video frames of the video;

[0006] determining a first motion region and a second motion region of the adjacent video frames according to the first motion mask, and calculating an optical flow between the first motion region and the second motion region;

[0007] Determine, according to the pixel points corresponding to the optical flow in the first motion area and the optical flow, the pixel points corresponding to each candidate point; the candidate point is determined based on the pixel points selected by the optical flow when the optical flow is greater than a first threshold;

[0008] Among the pixel points corresponding to the candidate points and the pixel points at corresponding positions in the multiple video frames, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point.

[0009] In one of the embodiments, before determining a first motion mask between adjacent video frames in a plurality of video frames of the video, the method further comprises:

[0010] Performing a reduction process on adjacent video frames among the plurality of video frames of the video to obtain reduced adjacent video frames;

[0011] Performing grayscale processing on the reduced adjacent video frames to obtain adjacent grayscale images;

[0012] Determining a first motion mask between adjacent video frames in a plurality of video frames of the video comprises:

[0013] In a plurality of video frames of the video, grayscale values ​​corresponding to pixel points in adjacent grayscale images are subjected to difference processing to obtain grayscale difference values; and a difference grayscale image is obtained based on the grayscale difference values;

[0014] In the difference grayscale image, the sliding box is moved according to the target step size, and each time the sliding box is moved, the number of pixels in the sliding box whose grayscale value is greater than the preset grayscale value is counted; when the counted number of pixels is greater than the preset number, the pixels in the sliding box are assigned a first grayscale value; when the counted number of pixels is less than or equal to the preset number, the pixels in the sliding box are assigned a second grayscale value, until the pixel points in each area of ​​the difference grayscale image are assigned values, thereby obtaining a first motion mask.

[0015] In one embodiment, calculating the optical flow between the first motion region and the second motion region comprises:

[0016] determining a target moving object according to the first moving area and the second moving area;

[0017] Determine the pixel points of the target moving object in the first moving area, and determine the pixel points of the target moving object in the second moving area;

[0018] Based on the pixel points of the target moving object in the first moving area and the pixel points of the target moving object in the second moving area, a plurality of optical flows are determined.

[0019] In one of the embodiments, before determining the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow, the method further includes:

[0020] When the optical flow is greater than a first threshold, the first threshold is used as the number of interpolated frames;

[0021] Dividing the optical flow into equal parts according to the number of interpolated frames to obtain equal-divided values;

[0022] Determine candidate points based on the equal-division value and the pixel points corresponding to the optical flow in the first motion area.

[0023] In one embodiment, the method further comprises:

[0024] When the optical flow is greater than or equal to a second threshold value and is not greater than the first threshold value, determining a candidate point according to the optical flow and a pixel point corresponding to the optical flow in the first motion area;

[0025] Determine a pixel point corresponding to the candidate point in the second motion area;

[0026] Among the pixel points corresponding to the candidate points and the pixel points at corresponding positions in the multiple video frames, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point.

[0027] In one embodiment, after obtaining the target video frame according to the position and pixel value corresponding to the target pixel point, the method further includes:

[0028] Get the tailing quantity;

[0029] Adjacent video frames of the tailing number are selected from the plurality of video frames, and pixel points at the target position in the target video frame are updated. When the updating is completed, the updated target video frame is obtained.

[0030] In one embodiment, after obtaining the target video frame according to the position and pixel value corresponding to the target pixel point, the method further includes:

[0031] When the tailing quantity is greater than the acceleration multiple, determining a frame number difference according to the tailing quantity and the acceleration multiple;

[0032] Determine a first pixel point of a target video frame among the plurality of video frames according to the frame number difference;

[0033] Acquire a plurality of new video frames whose number of frames is equal to the acceleration multiple, and determine a second pixel point in the plurality of new video frames;

[0034] Selecting a pixel point with the largest pixel value from the determined first pixel point and the second pixel point as a replacement pixel point;

[0035] The pixel point at the target position in the next target video frame is updated according to the replacement pixel point.

[0036] The present application provides a video processing device. The device comprises:

[0037] A first determination module, configured to determine a first motion mask between adjacent video frames among a plurality of video frames of the video;

[0038] a determination and calculation module, configured to determine a first motion region and a second motion region of the adjacent video frames according to the first motion mask, and calculate an optical flow between the first motion region and the second motion region;

[0039] A second determination module, configured to determine the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate point is determined based on the pixel points selected by the optical flow when the optical flow is greater than a first threshold;

[0040] The selection and updating module is used to select the pixel with the largest pixel value as the target pixel point among the pixel points corresponding to the candidate points and the pixel points at the corresponding positions in the multiple video frames, and obtain the target video frame according to the position and pixel value corresponding to the target pixel point.

[0041] In one of the embodiments, the first determination module is further used to perform downscaling processing on adjacent video frames among the multiple video frames of the video to obtain downsized adjacent video frames; perform grayscale processing on the downsized adjacent video frames to obtain adjacent grayscale images; perform difference processing on the grayscale values ​​corresponding to the pixels in the adjacent grayscale images among the multiple video frames of the video to obtain each grayscale difference; obtain a difference grayscale image based on each grayscale difference; move the sliding box according to the target step size in the difference grayscale image, and count the number of pixels in the sliding box whose grayscale values ​​are greater than a preset grayscale value each time the sliding box is moved, when the counted number of pixels is greater than the preset number, assign the pixels in the sliding box to the first grayscale value, and when the counted number of pixels is less than or equal to the preset number, assign the pixels in the sliding box to the second grayscale value, until the pixel points in each area of ​​the difference grayscale image are assigned values, thereby obtaining a first motion mask.

[0042] In one embodiment, the determination and calculation module is also used to determine the target motion object based on the first motion area and the second motion area; determine the pixel points of the target motion object in the first motion area, and determine the pixel points of the target motion object in the second motion area; determine multiple optical flows based on the pixel points of the target motion object in the first motion area and the pixel points in the second motion area.

[0043] In one of the embodiments, the second determination module is further used to, when the optical flow is greater than a first threshold, use the first threshold as the number of interpolation frames; divide the optical flow into equal parts according to the number of interpolation frames to obtain an equal-division value; and determine candidate points based on the equal-division value and the pixel points corresponding to the optical flow in the first motion area.

[0044] In one of the embodiments, the second determination module is also used to determine a candidate point based on the optical flow and the pixel points corresponding to the optical flow in the first motion area when the optical flow is greater than or equal to a second threshold and not greater than the first threshold; determine the pixel points corresponding to the candidate point in the second motion area; select the pixel point with the largest pixel value as the target pixel point among the pixel points corresponding to the candidate point and the pixel points at the corresponding positions in the multiple video frames, and obtain the target video frame according to the position and pixel value corresponding to the target pixel point.

[0045] In one of the embodiments, the selection and update module is also used to obtain the number of tailings; adjacent video frames with the number of tailings are selected from multiple video frames, and the pixel points at the target position in the target video frame are updated. When the update is completed, the updated target video frame is obtained.

[0046] In one of the embodiments, the selection and update module is also used to determine the frame number difference according to the tail number and the acceleration multiple when the tail number is greater than the acceleration multiple; determine the first pixel point of the target video frame in the multiple video frames according to the frame number difference; obtain multiple new video frames whose frame number is equal to the acceleration multiple, and determine the second pixel point in the multiple new video frames; select the pixel point with the largest pixel value from the determined first pixel point and the second pixel point as the replacement pixel point; and update the pixel point at the target position in the next target video frame according to the replacement pixel point.

[0047] The present application also provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0048] The present application also provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0049] The present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0050] The video processing method, device, computer equipment and storage medium are as follows: determining a first motion mask between adjacent video frames in a plurality of video frames of a video; determining a first motion region and a second motion region of adjacent video frames according to the first motion mask, and calculating the optical flow between the first motion region and the second motion region; determining the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion region and the optical flow; the candidate point is determined based on the pixel points selected based on the optical flow when the optical flow is greater than a first threshold; selecting the pixel point with the largest pixel value as the target pixel point among the pixel points corresponding to each candidate point and the pixel points at the corresponding positions in a plurality of video frames, and obtaining the target video frame according to the position and pixel value corresponding to the target pixel point. By judging whether the optical flow is greater than the first threshold, the pixel points participating in the video frame processing in the motion region are screened, and the amount of calculation is further reduced. In addition, by comparing the pixel points, the target pixel point with the largest pixel value is selected to update the target video frame, so as to realize a long exposure time-lapse photography effect without additional video interpolation, which not only greatly saves memory space, but also effectively improves the efficiency of video processing and optimizes the effect of video processing.

[0051] The present application also provides a video processing method, the method comprising:

[0052] Grouping multiple video frames of the video to obtain at least two video frame groups;

[0053] Acquire a first video frame and a second video frame from each of the video frame groups;

[0054] sequentially determining a second motion mask between the first video frame and the second video frame of each group;

[0055] determining an optical flow between the first video frame and the second video frame based on the second motion mask;

[0056] Using the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each of the video frame groups;

[0057] Determine the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the third motion region of the adjacent video frames in each of the video frame groups and the optical flow; the candidate point is determined based on the pixel value selected based on the optical flow when the optical flow is greater than the first threshold; the third motion region and the fourth motion region are motion regions of different video frames in the adjacent video frames;

[0058] Among the determined pixel points, a pixel point with the largest pixel value is selected as a target pixel point, and a target video frame is obtained according to a position and a pixel value corresponding to the target pixel point.

[0059] The present application also provides a video processing device. The device comprises:

[0060] A grouping module, used for grouping a plurality of video frames of the video to obtain at least two video frame groups;

[0061] An acquisition module, used for acquiring a first video frame and a second video frame from each of the video frame groups;

[0062] A first determining module, configured to sequentially determine a second motion mask between the first video frame and the second video frame of each group;

[0063] a second determining module, configured to determine an optical flow between the first video frame and the second video frame based on the second motion mask;

[0064] A processing module, configured to use the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each of the video frame groups;

[0065] A third determination module is used to determine the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the third motion area of ​​the adjacent video frames in each of the video frame groups and the optical flow in turn; the candidate point is determined based on the pixel value selected based on the optical flow when the optical flow is greater than the first threshold; the third motion area and the fourth motion area are motion areas of different video frames in the adjacent video frames;

[0066] The updating module is used to select a pixel point with the largest pixel value from the determined pixel points as a target pixel point, and obtain a target video frame according to a position and a pixel value corresponding to the target pixel point.

[0067] The present application also provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0068] The present application also provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0069] The present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0070] The above-mentioned video processing method, device, computer equipment and storage medium obtain at least two video frame groups by grouping multiple video frames of a video; obtain the first video frame and the second video frame from each video frame group; determine the second motion mask between the first video frame and the second video frame of each group in turn; determine the optical flow between the first video frame and the second video frame based on the second motion mask; use the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each video frame group; realize that multiple video frames share one optical flow, effectively saving the calculation amount of the optical flow, and not only that, determine the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the third motion area of ​​adjacent video frames in each video frame group and the optical flow in turn; the candidate point is determined based on the pixel value selected based on the optical flow when the optical flow is greater than the first threshold; the third motion area and the fourth motion area are motion areas of different video frames in adjacent video frames; among the determined pixel points, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point. A method for achieving long-exposure time-lapse photography effect without additional video frame insertion is implemented, which not only greatly saves memory space, but also further improves the efficiency of video processing and optimizes the effect of video processing by sharing optical flow through multiple video frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A diagram showing an application environment of a video processing method in an embodiment;

[0072] Figure 2 is a schematic flow chart of a video processing method in one embodiment;

[0073] Figure 3a is a corresponding grayscale image in adjacent video frames in one embodiment;

[0074] Figure 3b is another corresponding grayscale image in an adjacent video frame in one embodiment;

[0075] Figure 3c is a schematic diagram of a motion mask in one embodiment;

[0076] Figure 4 A schematic diagram of a flow chart of a step of determining an optical flow in another embodiment;

[0077] Figure 5a is a schematic diagram of a target video frame in one embodiment;

[0078] Figure 5b is a schematic diagram of an updated target video frame in one embodiment;

[0079] Figure 6a A schematic diagram of the effect of a maximum value stacking algorithm in one embodiment;

[0080] Figure 6b A schematic diagram of the effect of the video processing method of the present application in one embodiment;

[0081] Figure 7a A schematic diagram of the effect of a maximum value stacking algorithm in another embodiment;

[0082] Figure 7b A schematic diagram of the effect of the video processing method of the present application in another embodiment;

[0083] Figure 8 is a flow chart of a video processing method in another embodiment;

[0084] Fig. 9 is a structural block diagram of a video processing device in one embodiment;

[0085] Fig.10 is a structural block diagram of a video processing device in another embodiment;

[0086] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0088] The video processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. This application can be executed by the terminal 102 or the server 104, and this embodiment is described by taking the execution of the terminal 102 as an example.

[0089] The terminal 102 determines a first motion mask between adjacent video frames in a plurality of video frames of a video; the terminal 102 determines a first motion region and a second motion region of adjacent video frames according to the first motion mask, and calculates an optical flow between the first motion region and the second motion region; the terminal 102 determines the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion region and the optical flow; the candidate point is determined based on the pixel points selected based on the optical flow when the optical flow is greater than a first threshold; the terminal 102 selects the pixel point with the largest pixel value as the target pixel point from the pixel points corresponding to each candidate point and the pixel points at the corresponding positions in the plurality of video frames, and obtains the target video frame according to the position and pixel value corresponding to the target pixel point.

[0090] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0091] In one embodiment, Figure 2 As shown, a video processing method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0092] S202: Determine a first motion mask between adjacent video frames among a plurality of video frames of a video.

[0093] Wherein, adjacent video frames may refer to two adjacent video frames, multiple video frames may refer to video frames arranged in a certain order in the same video, and adjacent video frames may be adjacent video frames in multiple video frames arranged in a certain order. For example, multiple video frames may be arranged in the order in which the video frames are played, and the adjacent video frames may be the previously played video frame and the next played video frame. The first motion mask may be used to reflect an image of motion between adjacent video frames.

[0094] Specifically, the terminal can perform downscaling processing on adjacent video frames in multiple video frames of the video to obtain downsized adjacent video frames; perform grayscale processing on the downsized adjacent video frames to obtain adjacent grayscale images, and in the multiple video frames of the video, perform difference processing on the grayscale values ​​corresponding to the pixels in the adjacent grayscale images to obtain each grayscale difference; obtain a difference grayscale image based on each grayscale difference; move the sliding box in the difference grayscale image according to the target step size, and count the number of pixels in the sliding box whose grayscale values ​​are greater than the preset grayscale value during each movement of the sliding box, when the counted number of pixels is greater than the preset number, assign the first grayscale value to the pixels in the sliding box, and when the counted number of pixels is less than or equal to the preset number, assign the second grayscale value to the pixels in the sliding box, until the pixel points in each area of ​​the difference grayscale image are all assigned, and the first motion mask is obtained. Figure 3a is a corresponding grayscale image in adjacent video frames in one embodiment; Figure 3b is another corresponding grayscale image in an adjacent video frame in an embodiment; as shown in the figure, the adjacent grayscale image may be Figure 3a and Figure 3b As shown, Figure 3c is a schematic diagram of a motion mask in an embodiment; according to the adjacent grayscale image, the following can be obtained: Figure 3c The first motion mask is shown.

[0095] Among them, the grayscale difference may refer to the difference after subtracting the grayscale values ​​of corresponding pixels. The difference grayscale map may refer to a grayscale map generated based on the grayscale difference. The target step size may refer to the step size of each movement of the sliding box in the difference grayscale map. The sliding box can be used to limit the range of the currently judged pixel in the difference grayscale map. The preset grayscale value may refer to a preset grayscale value, which can be used to judge the grayscale value of the pixel. The preset number may refer to a preset number, which can be used to judge the number of pixels. The first grayscale value and the second grayscale value may refer to the grayscale values ​​after the grayscale values ​​of the pixels in the difference grayscale map are binarized. For example, the first grayscale value may be 1 and the second grayscale value may be 0.

[0096] In one embodiment, the terminal may further resize adjacent video frames among the multiple video frames according to a preset ratio to obtain resized adjacent video frames, and perform grayscale processing on the resized adjacent video frames to obtain adjacent grayscale images.

[0097] The preset ratio may refer to a preset ratio, and the preset ratio may be used to reduce the size of the video frame. For example, the preset ratio may be 0.75, which means that the video frame is reduced to 0.75 times of the original video frame.

[0098] In one embodiment, the grayscale values ​​corresponding to the pixel points in adjacent grayscale images determined in multiple grayscale images are subjected to difference processing to obtain each grayscale difference value. The terminal may perform difference processing on the grayscale values ​​corresponding to the pixel points in adjacent grayscale images determined in multiple grayscale images to obtain the grayscale value after difference, and take the absolute value of each grayscale value after difference to obtain each grayscale difference value.

[0099] For example, adjacent grayscale images are grayscale image A and grayscale image B. The grayscale values ​​of the pixels in grayscale image A are {33, 76, 89, 188, 125}, and the grayscale values ​​of the pixels in grayscale image B are {12, 54, 66, 198, 215}. The grayscale values ​​of the pixels in grayscale image A and grayscale image B are subtracted correspondingly {33-12, 76-54, 89-66, 188-198, 125-215} to obtain the grayscale values ​​after difference: {21, 22, 23, -10, -90}. The absolute value of each grayscale value after difference is taken to obtain the grayscale difference values ​​{21, 22, 23, 10, 90}.

[0100] S204, determining a first motion region and a second motion region of adjacent video frames according to the first motion mask, and calculating an optical flow between the first motion region and the second motion region.

[0101] The first motion region may refer to a region in the previous video frame of the adjacent video frames that corresponds to the first grayscale value in the first motion mask. The second motion region may refer to a region in the next video frame of the adjacent video frames that corresponds to the first grayscale value in the first motion mask. Optical flow can be used to reflect the motion of a moving object between adjacent video frames. For example, the moving object is A, and the coordinate positions corresponding to the center position of a certain window in A on the adjacent video frames i and j are (1, 3) and (5, 6), respectively. Then the optical flow of the center position of the certain window is (5-1, 6-3) = (4, 3) or (1-5, 3-6) = (-4, -3).

[0102] Specifically, the terminal may first determine the first grayscale area in the first motion mask, and determine the corresponding first motion area and second motion area in adjacent video frames according to the first grayscale area. And calculate multiple optical flows corresponding to the moving objects in the first motion area and the second motion area. It should be noted that there may be one or more moving objects in the first motion area and the second motion area, and one moving object in the first motion area and the second motion area may correspond to multiple pixels, and each pair of corresponding pixels in the first motion area and the second motion area corresponds to an optical flow.

[0103] The first grayscale region may refer to a region where the grayscale value of a pixel is the first grayscale value. A moving object may refer to an object whose corresponding position changes between the first moving region and the second moving region. For example, in the first moving region and the second moving region, there is a vehicle C, and the center of the vehicle C is located at (3, 4) in the first moving region and at (4, 6) in the second moving region. Since (3, 4) is different from (4, 6), that is, the corresponding position of the vehicle C changes, the vehicle C is a moving object.

[0104] S206, determining the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate point is determined based on the pixel points selected based on the optical flow when the optical flow is greater than the first threshold.

[0105] Among them, the candidate point may refer to a point determined based on the pixel point corresponding to the optical flow and the optical flow. For example, the candidate point may be a point determined based on the pixel point corresponding to the optical flow and the equal-division value of the optical flow, and the equal-division value may refer to the equal-division value. It should be pointed out that here, the present application is not limited to the equal-division value of the optical flow, and the proportion of the optical flow in the calculation of the candidate point may be adjusted accordingly according to the specific situation. The first threshold may refer to a threshold for judging the optical flow. The pixel point corresponding to the candidate point may refer to the pixel point corresponding to the candidate point in the second motion area.

[0106] For example, the position of the pixel corresponding to the optical flow in the first motion area is (a, b), the optical flow is (m, n), the number of interpolated frames is d, d is an integer, and the candidate point H i The calculation formula can be:

[0107] Where i = d. Assuming the optical flow is (6, 9), the number of interpolated frames is 2, and the position of the pixel corresponding to the optical flow in the first motion area is (1, 2), it is determined that there are two candidate points. Substituting into the above formula, we can get:

[0108] The number of interpolated frames can be used to determine the candidate points corresponding to the optical flow and the number of candidate points. For example, when the optical flow is (6, 9), when the number of interpolated frames is 2, the number of candidate points can be 2. The equal-division values ​​of the optical flow can be (2, 3) and (4, 6).

[0109] In one embodiment, when the optical flow is greater than a first threshold, the first threshold is used as the number of interpolated frames; the optical flow is equally divided according to the number of interpolated frames to obtain a processing result; and the candidate point is determined based on the processing result and the pixel point corresponding to the optical flow in the first motion area. The processing result may refer to the equally divided value of the optical flow, that is, the result of equally divided processing of the optical flow.

[0110] For example, the optical flow is (m, n), the number of interpolated frames is d, d is an integer, and the calculation formula of the processing result can be: Where d = i. For example, if the number of interpolated frames is 2 and the optical flow is (6, 9), it means that two candidate points need to be equally divided in (6, 9). Substituting the above processing result into the calculation formula, the processing results are (2, 3) and (4, 6).

[0111] In one embodiment, determining the candidate points includes determining the candidate points based on the pixel points corresponding to the optical flow and the equal-division value of the optical flow when the optical flow is greater than a first threshold, or determining the candidate points based on the pixel points corresponding to the optical flow and the equal-division value of the optical flow when the maximum value of the optical flow is greater than the first threshold, or first determining the optical flow product value between the optical flow and the optical flow coefficient, and when the maximum value of the optical flow product value is greater than the first threshold, determining the candidate points based on the pixel points corresponding to the optical flow and the equal-division value of the optical flow.

[0112] The optical flow coefficient may refer to a coefficient used to amplify or reduce the optical flow. For example, the optical flow coefficient may correspond to a preset ratio. When the preset ratio is 0.25, the optical flow coefficient may be 4. The maximum value of the optical flow may refer to a larger value in the optical flow. For example, if the optical flow is (3, 4), the maximum value of the optical flow is 4. The optical flow product value may refer to the product of the optical flow and the optical flow coefficient.

[0113] In one embodiment, when the optical flow is greater than or equal to the second threshold and not greater than the first threshold, a candidate point is determined based on the optical flow and the pixel points corresponding to the optical flow in the first motion area; the pixel points corresponding to the candidate points are determined in the second motion area; among the pixel points corresponding to the candidate points and the pixel points at corresponding positions in multiple video frames, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained based on the position and pixel value corresponding to the target pixel point.

[0114] Among them, the second threshold can be used to determine the threshold of optical flow, the second threshold is less than or equal to the first threshold, and the first threshold and the second threshold are different thresholds. The pixel value may refer to the value corresponding to the pixel point, and the pixel value may refer to the grayscale value or the sum of the values ​​of the three channels of RGB (red, green and blue). For example, the RGB of the pixel point is (80, 90, 180), then the pixel value of the pixel point is: 80+90+180=350. The target pixel point may refer to the point used to replace the pixel point at the target position in the target video frame. The target video frame may refer to the video frame that needs to update the pixel points, and the target video frame can be any video frame among multiple video frames.

[0115] In one embodiment, when the optical flow is greater than or equal to the second threshold value and is not greater than the first threshold value, a pixel corresponding to a candidate point is determined in the second motion region based on the pixel corresponding to the optical flow and the optical flow in the first motion region, and the pixel with the largest pixel value is selected as the target pixel among the pixel corresponding to the candidate point and the pixel at the corresponding position in multiple video frames, and the target video frame is obtained based on the position and pixel value corresponding to the target pixel. For example, the coordinates of the pixel corresponding to the optical flow in the first motion region are (1, 2), and the optical flow is (3, 4), then the candidate point is (1+3, 2+4)=(4, 6), that is, the coordinates of the pixel corresponding to the candidate point in the second motion region are (4, 6).

[0116] In one embodiment, when the optical flow is less than the second threshold, the current round of updating the pixel points at the target position in the target video frame according to the optical flow is skipped, and a new optical flow is selected to update the pixel points at the next target in the target video frame.

[0117] Among them, the target position can refer to the position of the pixel point to be updated in the target video frame. The pixel points updated each time are different, and the target position is different. For example, the target position of the pixel point updated corresponding to the current optical flow can be (1, 2), and the target position of the pixel point updated corresponding to the next optical flow can be (1, 3).

[0118] S208, selecting a pixel with the largest pixel value as a target pixel from among the pixel points corresponding to each candidate point and the pixel points at corresponding positions in a plurality of video frames, and obtaining a target video frame according to the position and pixel value corresponding to the target pixel point.

[0119] Specifically, the terminal can select the pixel point with the largest pixel value as the target pixel point among the pixel points corresponding to each candidate point and the pixel points at the positions corresponding to the candidate points in multiple video frames. After determining the target video frame among the multiple video frames, the target position is determined in the target video frame according to the position of the target pixel point, and then the pixel value in the target position is updated according to the pixel value of the target pixel point to obtain an updated target video frame.

[0120] In one embodiment, when the target video frame is the first video frame (the first video frame) among multiple video frames, and the adjacent video frames are the first video frame and the second video frame, the pixel with the largest pixel value is selected as the target pixel among the pixel points corresponding to the optical flow in the first motion area and the pixel points corresponding to each candidate point in the second motion area, and the pixel point at the target position in the first video frame is updated based on the target pixel point.

[0121] In one embodiment, when the target video frame is not the first video frame among the multiple video frames, and is not the last video frame among the multiple video frames, the pixel with the largest pixel value is selected as the target pixel among the determined pixels and the pixel at the target position in the target video frame of the multiple video frames, and the pixel at the target position in the target video frame is updated according to the target pixel. For example, Figure 5a is a schematic diagram of a target video frame in one embodiment; Figure 5b FIG. 1 is a schematic diagram of a target video frame after updating in one embodiment; as shown in the figure, when Figure 5a After replacing all the pixels in the motion area of ​​the target video frame, the following can be obtained: Figure 5b A schematic diagram showing a good tailing effect is shown.

[0122] In one embodiment, when the target video frame is the last video frame (the last video frame) among multiple video frames, and the adjacent video frames are the last video frame and the penultimate video frame, the pixel with the largest pixel value is selected as the target pixel among the pixel points corresponding to the optical flow in the first motion area and the pixel points corresponding to each candidate point in the second motion area, and the pixel point at the target position in the last video frame is updated according to the target pixel point.

[0123] In one embodiment, after the pixel points at the target position in the target video frame are updated based on all optical flows between the first motion area and the second motion area in adjacent video frames, the next adjacent video frame is selected from multiple video frames to update the pixel points at the target position in the target video frame.

[0124] In one embodiment, after S208, the terminal can obtain the tailing number; select adjacent video frames of the tailing number from multiple video frames, update the pixel points at the target position in the target video frame, and when the update is completed, obtain the updated target video frame.

[0125] Among them, the tail number can refer to the number of video frames used to obtain the target video frame. For example, the tail number is 4 frames, indicating that the adjacent video frames of 4 video frames are to be used: the first frame and the second frame, the second frame and the third frame, the third frame and the fourth frame. After the pixel point at the target position in the target video frame (the first frame or the fourth frame) is updated 4 times, the target video frame is played.

[0126] In one embodiment, after S208, the terminal can also determine the frame number difference according to the tailing number and the acceleration multiple when the tailing number is greater than the acceleration multiple; determine the first pixel point of the target video frame in multiple video frames according to the frame number difference; obtain multiple new video frames whose frame number is equal to the acceleration multiple, and determine the second pixel point in the multiple new video frames; select the pixel point with the largest pixel value from the determined first pixel point and the second pixel point as the replacement pixel point; and update the pixel point at the target position in the next target video frame according to the replacement pixel point.

[0127] Among them, the acceleration multiple may refer to the playback speed when playing the target video frame. The frame number difference may refer to the difference between the number of frames. The first pixel point may refer to the pixel point in the target video frame. The number of frames may refer to the number of video frames. The second pixel point may refer to the pixel point in the new video. The replacement pixel point may refer to the point used for the pixel point at the target position in the target video frame.

[0128] For example, the tail length (tail number) is 32 frames, and the acceleration multiple is 16 times. First, the 1st to 32nd frames are synthesized into one frame (target video frame) and then played out; because the acceleration multiple is half of the tail length and the frame difference is 16, the next frame of the output video (the next target video frame) needs to synthesize the 17th to 48th frames into one frame and play it out. So we can first save the target pixel with the largest pixel value in "frames 17 to 32", that is, save the target pixel of the target video frame in "frames 17 to 32", and then only need to determine the pixel with the largest pixel value in "frames 33 to 48" to obtain the target pixel, compare it with the pixel value of the target pixel saved previously, select the target pixel with the larger pixel value as the replacement pixel, and update the pixel at the target position in the next target video frame according to the replacement pixel, so there is no need to recalculate the target pixel in "frames 17 to 32".

[0129] The above-mentioned video processing method determines the first motion mask between adjacent video frames in multiple video frames of the video; determines the first motion region and the second motion region of the adjacent video frames according to the first motion mask, and calculates the optical flow between the first motion region and the second motion region; determines the motion region where the moving object exists in the adjacent video frames by the motion mask to determine the optical flow between the motion regions, which effectively reduces the amount of calculation compared to the traditional method of directly calculating the optical flow between complete video frames. In addition, the pixel points corresponding to each candidate point are determined according to the pixel points corresponding to the optical flow in the first motion region and the optical flow; the candidate point is determined based on the pixel points selected based on the optical flow when the optical flow is greater than the first threshold; among the pixel points corresponding to each candidate point and the pixel points at the corresponding positions in multiple video frames, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point. By judging whether the optical flow is greater than the first threshold, the pixels participating in the video frame processing in the moving area are screened to further reduce the amount of calculation. In addition, by comparing the pixels, the target pixel with the largest pixel value is selected to update the target video frame, thereby realizing a long-exposure time-lapse photography effect without the need for additional video insertion. This not only greatly saves memory space, but also effectively improves the efficiency of video processing and optimizes the effect of video processing.

[0130] In one embodiment, Figure 4 As shown, the steps of determining the optical flow include:

[0131] S402: Determine a target moving object according to the first moving area and the second moving area.

[0132] The target moving object may refer to the same object that appears in both the first moving area and the second moving area, and the position of the object in the first moving area is different from the position in the second moving area, that is, the target moving object.

[0133] Specifically, the terminal can determine the first motion area in the previous video frame of the adjacent video frames based on the first motion mask and identify the moving object in the first motion area, and then determine the second motion area in the next video frame of the adjacent video frames based on the first motion mask and identify the moving object in the second motion area, and determine the target motion object from the moving objects in the first motion area and the moving objects in the second motion area.

[0134] In one embodiment, determining the first motion region in a previous video frame in adjacent video frames based on the first motion mask includes the terminal determining a first grayscale region based on the first motion mask, and then determining a first motion region in a previous video frame in adjacent video frames corresponding to the first grayscale region.

[0135] In one embodiment, determining the second motion region in a subsequent video frame among adjacent video frames according to the first motion mask includes the terminal determining a first grayscale region according to the first motion mask, and then determining a second motion region in a subsequent video frame among adjacent video frames corresponding to the first grayscale region.

[0136] S404: Determine pixel points of the target moving object in the first moving area, and determine pixel points of the target moving object in the second moving area.

[0137] Specifically, the terminal can determine the first target area where the target moving object is located in the first moving area, and use the pixel points in the first target area as the pixel points of the target moving object; determine the second target area where the target moving object is located in the second moving area, and use the pixel points in the second target area as the pixel points of the target moving object.

[0138] The first target area may refer to an area where the target moving object is located in the first moving area, and the second target area may refer to an area where the target moving object is located in the second moving area.

[0139] S406: Determine a plurality of optical flows based on the pixel points of the target moving object in the first moving area and the pixel points of the target moving object in the second moving area.

[0140] Specifically, the terminal can first determine the correspondence between the pixel points of the target moving object in the first moving area and the pixel points of the target moving object in the second moving area, determine multiple corresponding pixel points of the target moving object between the first moving area and the second moving area based on the correspondence, and determine multiple optical flows based on the multiple corresponding pixel points.

[0141] Among them, the corresponding relationship can refer to the corresponding relationship between the pixel points of the target moving object in the first moving area and the pixel points in the second moving area. For example, the target moving object is pedestrian D, and the center position of the left shoulder of pedestrian D, the corresponding pixel point in the first moving area is (2, 3), and the corresponding pixel point in the second moving area is (3, 4), wherein (2, 3) and (3, 4) are the position coordinates of the pixel points, and the corresponding relationship is that (2, 3) in the first moving area corresponds to (3, 4) in the second moving area. Then the optical flow of the center position of the left shoulder of pedestrian D can be (2-3, 3-4) = (-1, -1), or (3-2, 4-3) = (1, 1).

[0142] In this embodiment, by determining the target motion object based on the first motion area and the second motion area, determining the pixel points of the target motion object in the first motion area, and determining the pixel points of the target motion object in the second motion area, multiple optical flows are determined based on the pixel points of the target motion object in the first motion area and the pixel points in the second motion area, so that all optical flows between the first motion area and the second motion area can be obtained, so as to subsequently update the pixel points at the target position in the target video frame.

[0143] As an example, the following embodiments are given:

[0144] Step 1. Input the original video n frames (tail length, i.e. the number of tails);

[0145] Step 2. Starting from the nth frame until the 2nd frame, take the nth frame and the n-1th frame, pre-process and reduce them to 0.25 times the grayscale image (only one pre-processing operation is required for continuous interpolation), calculate the motion value (motion) between the two reduced images, that is, count the grayscale difference in a certain area, and obtain a motion mask (motion_mask).

[0146] Step 3. Use an optical flow algorithm based on the motion mask (with parameters set to favor the calculation of small displacements) to calculate the optical flow within the reduced image mask; (the optical flow algorithm here can be any optical flow algorithm, and only the optical flow of the area with a grayscale value of 1 in the motion mask needs to be calculated).

[0147] Step 4. Directly on the nth frame original image, for the pixels mapped to the inside of the motion mask, update the pixel values ​​of the nth frame original image according to the corresponding optical flow:

[0148] Step 4.1. If the optical flow is multiplied by the corresponding coefficient (optical flow coefficient) and is greater than x pixels, the optical flow is equally divided into y candidate points (equivalent to inserting y frames); if it is less than z pixels, it is skipped without any processing; it is recommended to take x=y, z=1, which can be selected arbitrarily according to the needs of performance and effect;

[0149] Step 4.2. Compare the current pixel value (the pixel value of the pixel at the target position in the target video frame) with the pixel value of the candidate point, and take the largest one to replace the current pixel point (the pixel at the target position in the target video frame).

[0150] In the white area (grayscale value is 1) of the motion mask (motion_mask), according to the optical flow calculation, for example: according to the optical flow, in the 32nd frame, the object at position a appears at position a+u in the 31st frame (this u is the optical flow, a and u are two-dimensional positions such as a=(1,2)). Therefore, in the real world, during the time from the 31st frame to the 32nd frame, all points on the line "a~a+u" of the 31st frame will pass through position a of the 32nd frame.

[0151] If you set to insert 3 frames between each frame, then you should take the points at the three positions "a+0.25u, a+0.5u, a+0.75u" in the 31st frame, compare them with the point at position a in the 32nd frame, and take the point with the largest pixel value and assign it to position a in the 32nd frame. Then do the above operation for all the points in "motion_mask".

[0152] The next round should take the 31st and 30th frames. After completing the previous operations (assuming that the optical flow this time is v), the last step is to compare the five points: the points at the three positions "a+0.25v, a+0.5v, a+0.75v" in the 30th frame, the point at position a in the 31st frame, and the point at position a in the 32nd frame (the point at position a in the 32nd frame after the previous round of assignment), and take the largest one and assign it to position a in the 32nd frame.

[0153] Step 5. After all the interpolation frames from the nth frame to the first frame are replaced, n interpolation frames are obtained, and a synthesized 1-frame result is obtained; Figure 6a A schematic diagram of the effect of a maximum value stacking algorithm in one embodiment; Figure 6b A schematic diagram of the effect of the video processing method of the present application in one embodiment; Figure 7a A schematic diagram of the effect of a maximum value stacking algorithm in another embodiment; Figure 7b FIG. 1 is a schematic diagram showing the effect of the video processing method of the present application in another embodiment. Figure 6a and Figure 6b Contrast, and Figure 7a and Figure 7b It can be clearly seen that the coherence effect of the synthesized frame of the present application is better than that of the general maximum value stacking algorithm.

[0154] Step 6. Output the synthesized frame. When the tail length is greater than the acceleration multiple, n interpolated video frames can be saved for reuse.

[0155] In one embodiment, Figure 8 As shown, another video processing method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0156] S802, grouping multiple video frames of the video to obtain at least two video frame groups;

[0157] Wherein, adjacent video frames may refer to two adjacent video frames, multiple video frames may refer to video frames arranged in a certain order in the same video, and adjacent video frames may be adjacent video frames in multiple video frames arranged in a certain order. For example, multiple video frames may be arranged in the order in which the video frames are played, and the adjacent video frames may be the previously played video frame and the next played video frame. A video frame group may refer to a group including at least two video frames.

[0158] Specifically, the terminal may group the multiple video frames according to the number of groups to obtain at least two video frame groups. The number of groups may refer to the number of video frame groups, for example, the number of groups may be 4.

[0159] In one embodiment, the terminal may first determine the total number of multiple video frames, determine the number of video frames in each video frame group based on the total number and the number of groups, and then select video frames of the same number from the multiple video frames in turn to obtain at least two video frame groups.

[0160] The total number may refer to the number of multiple video frames, and the number of video frames may refer to the number of video frames.

[0161] For example, there are 32 video frames in total and the number of groups is 4. Based on the total number 32 and the number of groups 4, it can be determined that the number of video frames in each video frame group is 8. Then, video frames 1 to 8, 9 to 16, 17 to 24, and 25 to 32 are selected from the 32 video frames in turn, thereby obtaining 4 video frame groups.

[0162] S804, acquiring a first video frame and a second video frame from each video frame group;

[0163] The first video frame may refer to a video frame in the video frame group used to determine the optical flow between adjacent video frames. The second video frame may refer to another video frame in the video frame group used to determine the optical flow between adjacent video frames.

[0164] Specifically, the terminal may first obtain a target frame sequence number, and obtain a first video frame and a second video frame from each video frame group according to the target frame sequence number.

[0165] Among them, the target frame sequence number can refer to the serial numbers of the first video frame and the second video frame in the video frame group respectively. For example, each video frame group contains 8 video frames, and the serial numbers in each video frame group are 1 to 8. The target frame sequence numbers can be 4 and 6, which means that the first video frame and the second video frame are the fourth frame and the sixth frame in the video frame group respectively.

[0166] S806 , sequentially determine a second motion mask between the first video frame and the second video frame of each group.

[0167] The second motion mask may be used to reflect an image of motion between the first video frame and the second video frame.

[0168] Specifically, the terminal can perform grayscale processing on the first video frame and the second video frame, or first perform downscaling processing on the first video frame and the second video frame to obtain downsized first video frame and second video frame, and then perform grayscale processing on the downsized first video frame and second video frame to obtain two corresponding grayscale images, and perform difference processing on the grayscale values ​​corresponding to the pixels in the two determined grayscale images to obtain each grayscale difference; obtain a difference grayscale image based on each grayscale difference; move the sliding box in the difference grayscale image according to the target step size, and count the number of pixels in the sliding box whose grayscale value is greater than the preset grayscale value during each movement of the sliding box; when the counted number of pixels is greater than the preset number, the pixels in the sliding box are assigned the first grayscale value; when the counted number of pixels is less than or equal to the preset number, the pixels in the sliding box are assigned the second grayscale value, until the pixel points in each area of ​​the difference grayscale image are assigned values, and a second motion mask is obtained.

[0169] Among them, the grayscale difference may refer to the difference after subtracting the grayscale values ​​of corresponding pixels. The difference grayscale map may refer to a grayscale map generated based on the grayscale difference. The target step size may refer to the step size of each movement of the sliding box in the difference grayscale map. The sliding box can be used to limit the range of the currently judged pixel in the difference grayscale map. The preset grayscale value may refer to a preset grayscale value, which can be used to judge the grayscale value of the pixel. The preset number may refer to a preset number, which can be used to judge the number of pixels. The first grayscale value and the second grayscale value may refer to the grayscale values ​​after the grayscale values ​​of the pixels in the difference grayscale map are binarized. For example, the first grayscale value may be 1 and the second grayscale value may be 0.

[0170] In one embodiment, grayscale processing is performed on the first video frame and the second video frame to obtain two corresponding grayscale images, including that the terminal can resize the first video frame and the second video frame according to a preset ratio to obtain two corresponding resized video frames, and grayscale processing is performed on the corresponding two resized video frames to obtain two corresponding grayscale images.

[0171] The preset ratio may refer to a preset ratio, and the preset ratio may be used to reduce the size of the video frame. For example, the preset ratio may be 0.75, which means that the video frame is reduced to 0.75 times of the original video frame.

[0172] In one embodiment, the grayscale values ​​corresponding to the pixel points in the two determined grayscale images are subjected to difference processing to obtain each grayscale difference value. The terminal may perform difference processing on the grayscale values ​​corresponding to the pixel points in the two determined grayscale images to obtain the grayscale value after difference, and take the absolute value of each grayscale value after difference to obtain each grayscale difference value.

[0173] For example, the first video frame and the second video frame correspond to grayscale image A and grayscale image B respectively, the grayscale values ​​of the pixels in grayscale image A are {33, 76, 89, 188, 125}, and the grayscale values ​​of the pixels in grayscale image B are {12, 54, 66, 198, 215}, and the grayscale values ​​of the pixels in grayscale image A and grayscale image B are subtracted correspondingly by {33-12, 76-54, 89-66, 188-198, 125-215} to obtain the grayscale values ​​after difference: {21, 22, 23, -10, -90}, and the absolute value of each grayscale value after difference is taken to obtain the grayscale difference values ​​{21, 22, 23, 10, 90}.

[0174] S808: Determine an optical flow between the first video frame and the second video frame based on the second motion mask.

[0175] The optical flow can be used to reflect the motion of the moving object between the first video frame and the second video frame. For example, the moving object is A, and the coordinates of the center position of a certain window in A in the adjacent video frames i and j are (1, 3) and (5, 6), respectively. Then the optical flow of the center position of the certain window is (5-1, 6-3) = (4, 3) or (1-5, 3-6) = (-4, -3).

[0176] Specifically, the terminal may first determine the first grayscale area in the second motion mask, and determine the first target motion area and the second target motion area corresponding to the first video frame and the second video frame respectively according to the first grayscale area. And calculate multiple optical flows corresponding to the moving objects in the first target motion area and the second target motion area. It should be pointed out that the first target motion area and the second target motion area may have one or more moving objects, and one moving object in the first target motion area and the second target motion area may correspond to multiple pixels, and each pair of corresponding pixels in the first target motion area and the second target motion area corresponds to an optical flow.

[0177] Among them, the first grayscale area may refer to an area where the grayscale value of the pixel point is the first grayscale value. The first target motion area may refer to an area in the first video frame corresponding to the first grayscale value in the second motion mask. The second target motion area may refer to an area in the second video frame corresponding to the first grayscale value in the second motion mask. A moving object may refer to an object whose corresponding position in the first target motion area and the second target motion area changes. For example, in the first target motion area and the second target motion area, there is a vehicle C, and the center of the vehicle C is at (3, 4) in the first target motion area, and the center of the vehicle C is at (4, 6) in the second target motion area. Because (3, 4) is not the same as (4, 6), that is, the corresponding position of the vehicle C changes, then the vehicle C is a moving object.

[0178] In one embodiment, the terminal can first determine the first target motion area corresponding to the first video frame and the second target motion area corresponding to the second video frame based on the second motion mask, and determine the target motion object based on the first target motion area and the second target motion area; determine the pixel points of the target motion object in the first target motion area, and determine the pixel points of the target motion object in the second target motion area; determine multiple optical flows based on the pixel points of the target motion object in the first target motion area and the pixel points in the second target motion area.

[0179] The target motion object may refer to the same object that appears in both the first target motion region and the second target motion region, and the position of the object in the first target motion region is different from that in the second target motion region, that is, the target motion object.

[0180] Specifically, determining the target motion object based on the first target motion area and the second target motion area includes that the terminal can determine the first target motion area in the first video frame based on the second motion mask, and identify the moving object in the first target motion area, and then determine the second target motion area in the second video frame based on the second motion mask, and identify the moving object in the second target motion area, and determine the target motion object from the moving objects in the first target motion area and the moving objects in the second target motion area.

[0181] In one embodiment, determining the first target motion region corresponding to the first video frame according to the second motion mask includes the terminal determining the first grayscale region according to the second motion mask, and then determining the first target motion region corresponding to the first video frame and the first grayscale region.

[0182] In one embodiment, determining the second target motion region in the second video frame according to the second motion mask includes the terminal determining a first grayscale region according to the second motion mask, and then determining a second motion region of the second video frame corresponding to the first grayscale region.

[0183] In one embodiment, determining the pixel points of the target motion object in the first target motion area and determining the pixel points of the target motion object in the second target motion area includes that the terminal can determine the third target area where the target motion object is located in the first target motion area, and use the pixel points in the third target area as the pixel points of the target motion object; determine the fourth target area where the target motion object is located in the second target motion area, and use the pixel points in the fourth target area as the pixel points of the target motion object.

[0184] The third target region may refer to a region where the target moving object is located in the first target moving region, and the fourth target region may refer to a region where the target moving object is located in the second target moving region.

[0185] In one embodiment, based on the pixel points of the target motion object in the first target motion area and the pixel points in the second target motion area, determining multiple optical flows includes that the terminal can first determine the correspondence between the pixel points of the target motion object in the first target motion area and the pixel points of the target motion object in the second target motion area, determine multiple corresponding pixel points of the target motion object between the first target motion area and the second target motion area based on the corresponding relationship, and determine multiple optical flows based on the multiple corresponding pixel points.

[0186] Among them, the corresponding relationship can refer to the corresponding relationship between the pixel points of the target motion object in the first target motion area and the pixel points in the second target motion area. For example, the target motion object is a pedestrian D, and the center position of the left shoulder of pedestrian D, the corresponding pixel point in the first target motion area is (2, 3), the corresponding pixel point in the second target motion area is (3, 4), and the corresponding pixel point in the second target motion area is (3, 4), wherein (2, 3) and (3, 4) are the position coordinates of the pixel points, and the corresponding relationship is that (2, 3) in the first target motion area corresponds to (3, 4) in the second target motion area. Then the optical flow of the center position of the left shoulder of pedestrian D can be (2-3, 3-4) = (-1, -1), or (3-2, 4-3) = (1, 1).

[0187] S810: Taking the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each video frame group.

[0188] S812, determine the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the third motion area of ​​adjacent video frames in each video frame group and the optical flow; the candidate point is determined based on the pixel value selected based on the optical flow when the optical flow is greater than the first threshold; the third motion area and the fourth motion area are motion areas of different video frames in adjacent video frames.

[0189] Among them, the third motion area may refer to the area in the previous video frame in the adjacent video frames corresponding to the first grayscale value in the second motion mask. The fourth motion area may refer to the area in the next video frame in the adjacent video frames corresponding to the first grayscale value in the second motion mask. The candidate point may refer to the pixel point corresponding to the optical flow and the point determined by the optical flow. For example, the candidate point may be a point determined based on the pixel point corresponding to the optical flow and the equal-division value of the optical flow, and the equal-division value may refer to the equal-division value. It should be pointed out that here, the present application is not limited to the equal-division value of the optical flow, and the proportion of the optical flow in the calculation of the candidate point can be adjusted accordingly according to the specific situation. The first threshold value may refer to the threshold value used to judge the optical flow. The pixel point corresponding to the candidate point may refer to the pixel point corresponding to the candidate point in the second motion area.

[0190] For example, the position of the pixel corresponding to the optical flow in the third motion area is (a, b), the optical flow is (m, n), the number of interpolated frames is d, d is an integer, and the candidate point H i The calculation formula can be:

[0191] Where i = d. Assuming the optical flow is (6, 9), the number of interpolated frames is 2, and the position of the pixel corresponding to the optical flow in the third motion area is (1, 2), it is determined that there are two candidate points. Substituting into the above formula, we can get:

[0192] The number of interpolated frames can be used to determine the candidate points corresponding to the optical flow and the number of candidate points. For example, when the optical flow is (6, 9), when the number of interpolated frames is 2, the number of candidate points can be 2. The equal-division values ​​of the optical flow can be (2, 3) and (4, 6).

[0193] In one embodiment, when the optical flow is greater than a first threshold, the first threshold is used as the number of interpolated frames; the optical flow is equally divided according to the number of interpolated frames to obtain a processing result; and the candidate point is determined based on the processing result and the pixel point corresponding to the optical flow in the third motion area. The processing result may refer to the equally divided value of the optical flow, that is, the result of equally divided processing of the optical flow.

[0194] For example, the optical flow is (m, n), the number of interpolated frames is d, d is an integer, and the calculation formula of the processing result can be: Where d = i. For example, if the number of interpolated frames is 2 and the optical flow is (6, 9), it means that two candidate points need to be equally divided in (6, 9). Substituting the above processing result into the calculation formula, the processing results are (2, 3) and (4, 6).

[0195] In one embodiment, determining the candidate points includes determining the candidate points based on the pixel points corresponding to the optical flow and the equal-division value of the optical flow when the optical flow is greater than a first threshold, or determining the candidate points based on the pixel points corresponding to the optical flow and the equal-division value of the optical flow when the maximum value of the optical flow is greater than the first threshold, or first determining the optical flow product value between the optical flow and the optical flow coefficient, and when the maximum value of the optical flow product value is greater than the first threshold, determining the candidate points based on the pixel points corresponding to the optical flow and the equal-division value of the optical flow.

[0196] The optical flow coefficient may refer to a coefficient used to amplify or reduce the optical flow. For example, the optical flow coefficient may correspond to a preset ratio. When the preset ratio is 0.25, the optical flow coefficient may be 4. The maximum value of the optical flow may refer to a larger value in the optical flow. For example, if the optical flow is (3, 4), the maximum value of the optical flow is 4. The optical flow product value may refer to the product of the optical flow and the optical flow coefficient.

[0197] In one embodiment, when the optical flow is greater than or equal to the second threshold and not greater than the first threshold, a candidate point is determined based on the optical flow and the pixel points corresponding to the optical flow in the third motion area; the pixel points corresponding to the candidate points are determined in the fourth motion area; among the pixel points corresponding to the candidate points and the pixel points at corresponding positions in multiple video frames, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained based on the position and pixel value corresponding to the target pixel point.

[0198] Among them, the second threshold can be used to determine the threshold of optical flow, the second threshold is less than or equal to the first threshold, and the first threshold and the second threshold are different thresholds. The pixel value may refer to the value corresponding to the pixel point, and the pixel value may refer to the grayscale value or the sum of the values ​​of the three channels of RGB (red, green and blue). For example, the RGB of the pixel point is (80, 90, 180), then the pixel value of the pixel point is: 80+90+180=350. The target pixel point may refer to the point used to replace the pixel point at the target position in the target video frame. The target video frame may refer to the video frame that needs to update the pixel points, and the target video frame can be any video frame among multiple video frames.

[0199] In one embodiment, when the optical flow is less than the second threshold, the current round of updating the pixel points at the target position in the target video frame according to the optical flow is skipped, and a new optical flow is selected to update the pixel points at the next target in the target video frame.

[0200] Among them, the target position can refer to the position of the pixel point to be updated in the target video frame. The pixel points updated each time are different, and the target position is different. For example, the target position of the pixel point updated corresponding to the current optical flow can be (1, 2), and the target position of the pixel point updated corresponding to the next optical flow can be (1, 3).

[0201] S814, among the determined pixel points, select the pixel point with the largest pixel value as the target pixel point, and obtain the target video frame according to the position and pixel value corresponding to the target pixel point.

[0202] Specifically, each time in a video frame group, among the determined pixel points, the pixel point with the largest pixel value is selected as the target pixel point, and after the pixel points at the target position in the target video frame of multiple video frames are updated according to the target pixel point, the pixel point with the largest pixel value is saved, and then in the next video frame group, among the determined pixel points, the pixel point with the largest pixel value is selected as the target pixel point, and the pixel points at the target position in the target video frame of multiple video frames are updated according to the target pixel point, and the pixel points with the largest pixel values ​​saved in the two video frame groups are compared accordingly to obtain the pixel point with a larger pixel value as the target pixel point, and the pixel points at the target position in the target video frame of multiple video frames are updated according to the target pixel point, and finally, the pixel point with the largest pixel value is obtained as the target pixel point, and the pixel points at the target position in the target video frame of multiple video frames are updated according to the target pixel point.

[0203] In one embodiment, when the target video frame is the first video frame (the first video frame) among multiple video frames, and the adjacent video frames are the first video frame and the second video frame, the pixel with the largest pixel value is selected as the target pixel among the pixel points corresponding to the optical flow in the third motion area and the pixel points corresponding to each candidate point in the fourth motion area, and the pixel point at the target position in the first video frame is updated based on the target pixel point.

[0204] In one embodiment, when the target video frame is not the first video frame among the multiple video frames, and is not the last video frame among the multiple video frames, the pixel with the largest pixel value is selected as the target pixel among the determined pixels and the pixel at the target position in the target video frame of the multiple video frames, and the pixel at the target position in the target video frame is updated according to the target pixel. For example, Figure 5a is a schematic diagram of a target video frame in one embodiment; Figure 5b FIG. 1 is a schematic diagram of a target video frame after updating in one embodiment; as shown in the figure, when Figure 5a After replacing all the pixels in the motion area of ​​the target video frame, the following can be obtained: Figure 5b A schematic diagram showing a good tailing effect is shown.

[0205] In one embodiment, when the target video frame is the last video frame (the last video frame) among multiple video frames, and the adjacent video frames are the last video frame and the penultimate video frame, the pixel with the largest pixel value is selected as the target pixel among the pixel points corresponding to the optical flow in the third motion area and the pixel points corresponding to each candidate point in the fourth motion area, and the pixel point at the target position in the last video frame is updated according to the target pixel point.

[0206] In one embodiment, after the pixel points at the target position in the target video frame are updated based on all optical flows between the third motion area and the fourth motion area in adjacent video frames, the next adjacent video frame is selected from multiple video frames to update the pixel points at the target position in the target video frame.

[0207] The above-mentioned video processing method obtains at least two video frame groups by grouping multiple video frames of a video; obtains a first video frame and a second video frame from each video frame group; determines the second motion mask between the first video frame and the second video frame of each group in turn; determines the optical flow between the first video frame and the second video frame based on the second motion mask; uses the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each video frame group; realizes that multiple video frames share one optical flow, effectively saving the calculation amount of the optical flow, and not only that, determines the pixel points corresponding to each candidate point in turn according to the pixel points corresponding to the optical flow in the third motion area of ​​adjacent video frames in each video frame group and the optical flow; the candidate point is determined based on the pixel value selected based on the optical flow when the optical flow is greater than the first threshold; the third motion area and the fourth motion area are motion areas of different video frames in adjacent video frames; among the determined pixel points, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point. A method for achieving long-exposure time-lapse photography effect without additional video frame insertion is implemented, which not only greatly saves memory space, but also further improves the efficiency of video processing and optimizes the effect of video processing by sharing optical flow through multiple video frames.

[0208] As an example, this instance is as follows:

[0209] (1) Acceleration idea: Multiple frame sequences (multiple video frames) can share the same optical flow for interpolation.

[0210] The shared optical flow can be calculated by the first and last frames of the optional sequence, but the illumination of the first and last frames varies greatly, which is not conducive to the calculation of optical flow and motion value, and large displacement is not conducive to the calculation of optical flow.

[0211] You can also select two frames in the middle of the sequence to calculate the optical flow. The smaller the interval between the two selected frames, the higher the accuracy of the calculation of the optical flow and motion value. However, the sharing ability of the first and last frames is weakened, which may cause tailing incoherence.

[0212] (2) Acceleration plan and possible loss of effect:

[0213] Interpolation is performed for every 8 frames, and there is a small probability of tailing with a small number of gaps. When the movement is faster, more gaps will appear. Interpolation is performed for every 4 frames, and there are small gaps when the movement is faster.

[0214] (3) Specific acceleration algorithm

[0215] Step 1. Change from inputting all N frames to inputting N' frames each time, where N is an integer multiple of N'. Now, N=32, N'=8;

[0216] Step 2. Taking each group of 8 input frames as an example, first take the i-th frame and the j-th frame (for example, i=3, j=6) from the 1 to 8 frames, and calculate the motion value and optical flow according to the above core algorithm steps;

[0217] Step 3. For each temporarily stored pixel, take the pixel with the largest pixel value between the original pixel and the current 8 frames, and traverse the pixels in the motion area for each frame according to the core algorithm steps of steps S202 to S208 according to the optical flow calculated previously. Interpolate the optical flow to obtain y points, take the pixel with the largest pixel value, and assign it to the result;

[0218] Step 4. After completing 4 sets of input, retrieve the results and clear the temporary storage.

[0219] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0220] Based on the same inventive concept, the embodiment of the present application also provides a video processing device for implementing the above-mentioned video processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more video processing device embodiments provided below can refer to the limitations on the video processing method above, and will not be repeated here.

[0221] In one embodiment, Fig. 9As shown, a video processing device is provided, including: a first determination module 902, a determination and calculation module 904, a second determination module 906, and a selection and update module 908, wherein:

[0222] A first determination module 902, configured to determine a first motion mask between adjacent video frames in a plurality of video frames of a video;

[0223] A determination and calculation module 904, configured to determine a first motion region and a second motion region of adjacent video frames according to a first motion mask, and calculate an optical flow between the first motion region and the second motion region;

[0224] A second determination module 906 is used to determine the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate point is determined based on the pixel points selected based on the optical flow when the optical flow is greater than the first threshold;

[0225] The selection and update module 908 is used to select the pixel with the largest pixel value as the target pixel point among the pixel points corresponding to each candidate point and the pixel points at corresponding positions in multiple video frames, and obtain the target video frame according to the position and pixel value corresponding to the target pixel point.

[0226] In one embodiment, the first determination module 902 is also used to perform downscaling processing on adjacent video frames in multiple video frames of the video to obtain downsized adjacent video frames; perform grayscale processing on the downsized adjacent video frames to obtain adjacent grayscale images; in multiple video frames of the video, perform difference processing on the grayscale values ​​corresponding to the pixels in the adjacent grayscale images to obtain each grayscale difference; obtain a difference grayscale image based on each grayscale difference; move the sliding box in the difference grayscale image according to the target step size, and count the number of pixels in the sliding box whose grayscale value is greater than the preset grayscale value during each movement of the sliding box, when the counted number of pixels is greater than the preset number, assign the first grayscale value to the pixels in the sliding box, and when the counted number of pixels is less than or equal to the preset number, assign the second grayscale value to the pixels in the sliding box, until the pixel points in each area of ​​the difference grayscale image are assigned values, and the first motion mask is obtained.

[0227] In one embodiment, the determination and calculation module 904 is also used to determine the target motion object based on the first motion area and the second motion area; determine the pixel points of the target motion object in the first motion area, and determine the pixel points of the target motion object in the second motion area; determine multiple optical flows based on the pixel points of the target motion object in the first motion area and the pixel points in the second motion area.

[0228] In one of the embodiments, the second determination module 906 is further used to use the first threshold as the number of interpolation frames when the optical flow is greater than the first threshold; divide the optical flow into equal parts according to the number of interpolation frames to obtain an equal-division value; and determine the candidate point based on the equal-division value and the pixel points corresponding to the optical flow in the first motion area.

[0229] In one of the embodiments, the second determination module 906 is also used to determine a candidate point based on the optical flow and the pixel points corresponding to the optical flow in the first motion area when the optical flow is greater than or equal to the second threshold and not greater than the first threshold; determine the pixel points corresponding to the candidate point in the second motion area; select the pixel point with the largest pixel value as the target pixel point among the pixel points corresponding to the candidate point and the pixel points at corresponding positions in multiple video frames, and obtain the target video frame based on the position and pixel value corresponding to the target pixel point.

[0230] In one embodiment, the selection and update module 908 is also used to obtain the tailing number; each adjacent video frame with the tailing number is selected from multiple video frames, and the pixel points at the target position in the target video frame are updated. When the update is completed, the updated target video frame is obtained.

[0231] In one embodiment, the selection and update module 908 is also used to determine the frame number difference according to the tail number and the acceleration multiple when the tail number is greater than the acceleration multiple; determine the first pixel point of the target video frame in multiple video frames according to the frame number difference; obtain multiple new video frames whose frame number is equal to the acceleration multiple, and determine the second pixel point in the multiple new video frames; select the pixel point with the largest pixel value from the determined first pixel point and the second pixel point as the replacement pixel point; and update the pixel point at the target position in the next target video frame according to the replacement pixel point.

[0232] The above embodiment determines a first motion mask between adjacent video frames in multiple video frames of a video; determines a first motion region and a second motion region of adjacent video frames according to the first motion mask, and calculates the optical flow between the first motion region and the second motion region; determines a motion region in which a moving object exists in adjacent video frames through the motion mask to determine the optical flow between the motion regions, which effectively reduces the amount of calculation compared to the traditional method of directly calculating the optical flow between complete video frames. Moreover, the pixel points corresponding to each candidate point are determined according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate point is determined based on the pixel points selected based on the optical flow when the optical flow is greater than the first threshold; among the pixel points corresponding to each candidate point and the pixel points at the corresponding positions in multiple video frames, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point, and by judging whether the optical flow is greater than the first threshold, the pixel points participating in the video frame processing in the motion area are screened to further reduce the amount of calculation. In addition, by comparing the pixel points corresponding to the optical flow, the target pixel point with the largest pixel value is selected to update the target video frame, thereby realizing a long exposure time-lapse photography effect without additional video interpolation, which not only greatly saves memory space, but also effectively improves the efficiency of video processing and optimizes the effect of video processing.

[0233] In one embodiment, Fig.10 As shown, a video processing device includes: a grouping module 1002, an acquisition module 1004, a first determination module 1006, a second determination module 1008, a processing module 1010, a third determination module 1012 and an updating module 1014, wherein:

[0234] A grouping module 1002 is used to group multiple video frames of a video to obtain at least two video frame groups;

[0235] An acquisition module 1004 is used to acquire a first video frame and a second video frame from each video frame group;

[0236] A first determination module 1006, configured to sequentially determine a second motion mask between the first video frame and the second video frame of each group;

[0237] A second determination module 1008, configured to determine an optical flow between the first video frame and the second video frame based on a second motion mask;

[0238] A processing module 1010 is used to use the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each video frame group;

[0239] The third determination module 1012 is used to determine the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the third motion region of the adjacent video frames in each video frame group and the optical flow in turn; the candidate point is determined based on the pixel value selected based on the optical flow when the optical flow is greater than the first threshold; the third motion region and the fourth motion region are motion regions of different video frames in the adjacent video frames;

[0240] The updating module 1014 is used to select a pixel point with the largest pixel value from the determined pixel points as a target pixel point, and obtain a target video frame according to the position and pixel value corresponding to the target pixel point.

[0241] In the above embodiment, at least two video frame groups are obtained by grouping multiple video frames of a video; a first video frame and a second video frame are obtained from each video frame group; a second motion mask between the first video frame and the second video frame of each group is determined in turn; based on the second motion mask, an optical flow between the first video frame and the second video frame is determined; the optical flow between the first video frame and the second video frame of each group is used as the optical flow corresponding to each video frame group; multiple video frames share one optical flow, which effectively saves the calculation amount of the optical flow. In addition, the pixel points corresponding to each candidate point are determined in turn according to the pixel points corresponding to the optical flow in the third motion area of ​​adjacent video frames in each video frame group and the optical flow; the candidate point is determined based on the pixel value selected based on the optical flow when the optical flow is greater than the first threshold; the third motion area and the fourth motion area are motion areas of different video frames in adjacent video frames; among the determined pixel points, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point. A method for achieving long-exposure time-lapse photography effect without additional video frame insertion is implemented, which not only greatly saves memory space, but also further improves the efficiency of video processing and optimizes the effect of video processing by sharing optical flow through multiple video frames.

[0242] Each module in the above video processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0243] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. The computer device is described as a terminal. The internal structure diagram thereof may be as follows: Fig.11As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video processing method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0244] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0245] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and the above embodiments are implemented when the processor executes the computer program.

[0246] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above embodiments are implemented.

[0247] In one embodiment, a computer program product is provided, including a computer program, and the computer program implements the above embodiments when executed by a processor.

[0248] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0249] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0250] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0251] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video processing method, characterized in that: The method comprises: determining a first motion mask between adjacent video frames in a plurality of video frames of the video; determining a first motion region and a second motion region of the adjacent video frames according to the first motion mask, and calculating an optical flow between the first motion region and the second motion region; Determine, according to the pixel points corresponding to the optical flow in the first motion area and the optical flow, the pixel points corresponding to each candidate point; the candidate point is determined based on the pixel points selected by the optical flow when the optical flow is greater than a first threshold; Among the pixel points corresponding to the candidate points and the pixel points at corresponding positions in the multiple video frames, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point; the target video frame is a video frame within the multiple video frames and has a trailing effect.

2. The method according to claim 1, characterized in that Before determining a first motion mask between adjacent video frames in a plurality of video frames of the video, the method further comprises: Performing a reduction process on adjacent video frames among the plurality of video frames of the video to obtain reduced adjacent video frames; Performing grayscale processing on the reduced adjacent video frames to obtain adjacent grayscale images; Determining a first motion mask between adjacent video frames in a plurality of video frames of the video comprises: In a plurality of video frames of the video, grayscale values ​​corresponding to pixel points in adjacent grayscale images are subjected to difference processing to obtain grayscale difference values; and a difference grayscale image is obtained based on the grayscale difference values; In the difference grayscale image, the sliding box is moved according to the target step size, and each time the sliding box is moved, the number of pixels in the sliding box whose grayscale value is greater than the preset grayscale value is counted; when the counted number of pixels is greater than the preset number, the pixels in the sliding box are assigned a first grayscale value; when the counted number of pixels is less than or equal to the preset number, the pixels in the sliding box are assigned a second grayscale value, until the pixel points in each area of ​​the difference grayscale image are assigned values, thereby obtaining a first motion mask.

3. The method according to claim 1, characterized in that: The calculating the optical flow between the first motion region and the second motion region comprises: determining a target moving object according to the first moving area and the second moving area; Determine the pixel points of the target moving object in the first moving area, and determine the pixel points of the target moving object in the second moving area; Based on the pixel points of the target moving object in the first moving area and the pixel points of the target moving object in the second moving area, a plurality of optical flows are determined.

4. The method according to claim 3, characterized in that Before determining the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow, the method further includes: When the optical flow is greater than a first threshold, the first threshold is used as the number of interpolated frames; Dividing the optical flow into equal parts according to the number of interpolated frames to obtain equal-divided values; Determine candidate points based on the equal-division value and the pixel points corresponding to the optical flow in the first motion area.

5. The method according to claim 1, characterized in that The method further comprises: When the optical flow is greater than or equal to a second threshold value and is not greater than the first threshold value, determining a candidate point according to the optical flow and a pixel point corresponding to the optical flow in the first motion area; Determine a pixel point corresponding to the candidate point in the second motion area; Among the pixel points corresponding to the candidate points and the pixel points at corresponding positions in the multiple video frames, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point.

6. The method according to claim 1, characterized in that After obtaining the target video frame according to the position and pixel value corresponding to the target pixel point, the method further includes: Get the tailing quantity; Adjacent video frames of the tailing number are selected from the plurality of video frames, and pixel points at the target position in the target video frame are updated. When the updating is completed, the updated target video frame is obtained.

7. The method according to claim 6, characterized in that After obtaining the target video frame according to the position and pixel value corresponding to the target pixel point, the method further includes: When the tailing quantity is greater than the acceleration multiple, determining a frame number difference according to the tailing quantity and the acceleration multiple; Determine a first pixel point of a target video frame among the plurality of video frames according to the frame number difference; Acquire a plurality of new video frames whose number of frames is equal to the acceleration multiple, and determine a second pixel point in the plurality of new video frames; Selecting a pixel point with the largest pixel value from the determined first pixel point and the second pixel point as a replacement pixel point; The pixel point at the target position in the next target video frame is updated according to the replacement pixel point.

8. A video processing method, characterized in that: The method comprises: Grouping multiple video frames of the video to obtain at least two video frame groups; Acquire a first video frame and a second video frame from each of the video frame groups; sequentially determining a second motion mask between the first video frame and the second video frame of each group; determining an optical flow between the first video frame and the second video frame based on the second motion mask; Using the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each of the video frame groups; Determine the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the third motion region of the adjacent video frames in each of the video frame groups and the optical flow; the candidate point is determined based on the pixel value selected based on the optical flow when the optical flow is greater than the first threshold; the third motion region and the fourth motion region are motion regions of different video frames in the adjacent video frames; Among the determined pixel points, the pixel point with the largest pixel value is selected as the target pixel point, and the target video frame is obtained according to the position and pixel value corresponding to the target pixel point; the target video frame is a video frame within the multiple video frames and has a trailing effect.

9. A video processing device, characterized in that: The device comprises: A first determination module, configured to determine a first motion mask between adjacent video frames among a plurality of video frames of the video; a determination and calculation module, configured to determine a first motion region and a second motion region of the adjacent video frames according to the first motion mask, and calculate an optical flow between the first motion region and the second motion region; A second determination module, configured to determine the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the first motion area and the optical flow; the candidate point is determined based on the pixel points selected by the optical flow when the optical flow is greater than a first threshold; A selection and update module is used to select the pixel point with the largest pixel value as the target pixel point among the pixel points corresponding to each candidate point and the pixel points at the corresponding positions in the multiple video frames, and obtain the target video frame according to the position and pixel value corresponding to the target pixel point; the target video frame is a video frame within the multiple video frames and has a trailing effect.

10. A video processing device, characterized in that: The device comprises: A grouping module, used for grouping a plurality of video frames of the video to obtain at least two video frame groups; An acquisition module, used for acquiring a first video frame and a second video frame from each of the video frame groups; A first determining module, configured to sequentially determine a second motion mask between the first video frame and the second video frame of each group; a second determining module, configured to determine an optical flow between the first video frame and the second video frame based on the second motion mask; A processing module, configured to use the optical flow between the first video frame and the second video frame of each group as the optical flow corresponding to each of the video frame groups; A third determination module is used to determine the pixel points corresponding to each candidate point according to the pixel points corresponding to the optical flow in the third motion area of ​​the adjacent video frames in each of the video frame groups and the optical flow in turn; the candidate point is determined based on the pixel value selected based on the optical flow when the optical flow is greater than the first threshold; the third motion area and the fourth motion area are motion areas of different video frames in the adjacent video frames; An updating module is used to select a pixel point with the largest pixel value as a target pixel point among the determined pixel points, and obtain a target video frame according to the position and pixel value corresponding to the target pixel point; the target video frame is a video frame within the multiple video frames and has a trailing effect.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.