Video processing method, device, computer equipment and storage medium

Abstract

The present application relates to a video processing method, device, computer equipment and storage medium. The method comprises: obtaining a target video frame; obtaining at least one front adjacent video frame that is a predetermined number of frames adjacent to the target video frame; obtaining at least one rear adjacent video frame that is a predetermined number of frames adjacent to the target video frame; processing the target video frame based on at least one front adjacent video frame and at least one rear adjacent video frame to obtain a processed target video frame processing result. The method of the embodiment of the present application is adopted to process the target video frame by symmetrically obtaining the front adjacent video frame and the rear adjacent video frame of the target video frame. While considering the influence of the front adjacent video frame on the target video frame, the influence of the rear adjacent video frame is also fully considered, thereby effectively resisting the influence caused by the delay of the moving process during the video processing process, thereby improving the processing effect of the target video frame.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of image processing technology, video portrait segmentation technology has been widely used in scenarios such as video conferencing software, live video and video shooting. For example, the virtual background function of video conferencing software can separate the portrait and background in the video. Users can choose any image to replace the background to effectively protect the user's environmental privacy.

[0003] Due to the conditional restrictions and real-time requirements of the video scene, after separating the portrait and the background in the video, the portrait in the video will have a "white edge" problem. The traditional technology uses a mean filter to smooth the video frame image, but this method does not have a good processing effect. Summary of the invention

[0004] Based on this, it is necessary to provide a video processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the processing effect of video frame images in response to the above technical problems.

[0005] A video processing method, the method comprising:

[0006] Get the target video frame;

[0007] Acquire at least one preceding adjacent video frame that is a predetermined number of frames adjacent to the target video frame;

[0008] Acquire at least one subsequent adjacent video frame that is adjacent to the target video frame by the predetermined number of frames;

[0009] The target video frame is processed based on the at least one previous adjacent video frame and the at least one subsequent adjacent video frame to obtain a processed target video frame processing result.

[0010] In one embodiment, the method for obtaining the processing result of the previous adjacent video frame includes:

[0011] Perform foreground image extraction processing on the previous adjacent video frame to obtain the previous adjacent video frame processing result.

[0012] In one embodiment, the method of obtaining the previous adjacent frame feature vector corresponding to the previous adjacent video frame processing result includes:

[0013] Scaling the processing result of the previous adjacent video frame to a predetermined size;

[0014] Vectorization processing is performed on the processing result of the previous adjacent video frame of the predetermined size to obtain a previous adjacent frame feature vector.

[0015] A video processing device, comprising:

[0016] A first acquisition module, used to acquire a target video frame;

[0017] A second acquisition module is used to acquire at least one previous adjacent video frame that is a predetermined number of frames adjacent to the target video frame;

[0018] A third acquisition module is used to acquire at least one subsequent adjacent video frame that is adjacent to the target video frame by the predetermined number of frames;

[0019] The frame processing module is used to process the target video frame based on the at least one previous adjacent video frame and the at least one subsequent adjacent video frame to obtain a processed target video frame processing result.

[0020] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned video processing method when executing the computer program.

[0021] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned video processing method when executed by a processor.

[0022] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned video processing method are implemented.

[0023] The above-mentioned video processing method, apparatus, computer equipment, computer-readable storage medium and computer program product obtain a target video frame; obtain at least one front adjacent video frame that is a predetermined number of frames before the target video frame; obtain at least one rear adjacent video frame that is a predetermined number of frames after the target video frame; and process the target video frame based on at least one front adjacent video frame and at least one rear adjacent video frame to obtain a processed target video frame processing result. The method of the above-mentioned embodiment is adopted to symmetrically obtain the front adjacent video frame and the rear adjacent video frame of the target video frame, and correct the target video frame. While considering the influence of the front adjacent video frame on the target video frame, the influence of the rear adjacent video frame is also fully considered, thereby effectively resisting the influence caused by the delay of the moving process during the video processing process, thereby improving the processing effect of the target video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A diagram showing an application environment of a video processing method in an embodiment;

[0025] Figure 2 is a schematic flow chart of a video processing method in one embodiment;

[0026] Figure 3 A schematic diagram of selecting a virtual background for a video in a specific embodiment;

[0027] Figure 4 A schematic diagram of the overall flow of a video processing method in a specific embodiment;

[0028] Figure 5 A schematic diagram of the effect of a video processing method in a specific embodiment;

[0029] Figure 6 is a structural block diagram of a video processing device in one embodiment;

[0030] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] In one embodiment, the video processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the application environment may involve both the terminal 102 and the server 104. In some embodiments, the application environment may also involve the terminal 106. The terminal 102 and the terminal 106 may communicate with the server 104 via a network or the like. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104, or may be placed on a cloud or other network server.

[0033] Specifically, the user can shoot a video through the terminal 102, and the video obtained by the terminal 102 can be sent to the server 104 through the communication network. After the server 104 obtains the video transmitted by the terminal 102 and processes the video, it returns the processed video to the terminal 102 for playback. In some embodiments, for example, in a live broadcast scene or a video conference scene, the server 104 also sends the processed video to the terminal 106 for playback. Specifically, when processing the video, after the server 104 obtains the video from the terminal 102, it obtains the target video frame from the video, and obtains at least one front adjacent video frame adjacent to the target video frame by a predetermined number of frames, and obtains at least one rear adjacent video frame adjacent to the target video frame by a predetermined number of frames, and processes the target video frame based on the at least one front adjacent video frame and the at least one rear adjacent video frame to obtain the processed target video frame processing result. The server 104 can delay the target video frame by a predetermined number of frames, and replace the initial video frame processing result in the target video frame with the processed target video frame processing result, that is, obtain the processed video, and send the processed video to the terminal 102 and the terminal 106 for playback.

[0034] In one embodiment, when the data processing capability of the terminal 102 meets the requirements, the application environment of the video processing method provided in the embodiment of the present application may only involve the terminal 102. Specifically, the user can shoot through the terminal 102 to obtain the video, and the terminal 102 obtains the target video frame, and obtains at least one front adjacent video frame that is adjacent to the target video frame by a predetermined number of frames, and obtains at least one rear adjacent video frame that is adjacent to the target video frame by a predetermined number of frames, and processes the target video frame based on the at least one front adjacent video frame and the at least one rear adjacent video frame to obtain the processed target video frame processing result. The terminal 102 delays the target video frame by a predetermined number, and replaces the initial video frame processing result in the target video frame with the processed target video frame processing result before playing it.

[0035] The terminals 102 and 106 may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart TVs, smart car devices, etc. Portable wearable devices may be smart watches, smart bracelets, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0036] Video portrait segmentation technology is widely used in video conferencing software, live video, video shooting and other scenarios. Taking video conferencing as an example, the virtual background function of video conferencing software can separate the portrait and background in the video. Users can choose any image to replace the background to effectively protect the user's environmental privacy.

[0037] Due to the conditional restrictions and real-time requirements of the video scene, after the portrait and background in the video are separated, the interference of conditions such as noise and lighting may cause video delays. In traditional technology, the target video frame is smoothed and mean filtered in combination with the first N video frames of the target video frame. The commonly used method is mean filtering. However, this method only relies on the first N video frames to smooth the target video frame, which is prone to delays, and in turn, when the selected virtual background is synthesized with the separated portrait, it is easy to bring obvious "white edges" to the edges of the portrait.

[0038] In view of the above problem, in one embodiment, as Figure 2 As shown, a video processing method is provided, which is applied to Figure 1 The terminal 102 and / or server 104 in the example are used for illustration, including:

[0039] Step S202, obtaining a target video frame.

[0040] The target video frame is a video frame in the video that currently needs to be processed. In one embodiment, the video can be a video shot historically or a video shot in real time.

[0041] If the video is a historically shot video, the target video frame can be a video frame at any moment in the video that needs to be processed. If the video is a real-time shot, the target video frame can be a preset number of video frames before the most recently obtained video frame in the video stream. The specific value of the preset number can be set in combination with the accuracy or demand of delayed playback. Taking the preset number K as an example, assuming that the currently obtained video frame is the Nth frame, the target video frame can be the NKth frame in the video stream. Specifically, the number of target video frames is one, that is, each target video frame in the video is processed.

[0042] Step S204: acquiring at least one preceding adjacent video frame that is a predetermined number of frames adjacent to the target video frame.

[0043] In one embodiment, a video frame that is before a target video frame and adjacent to the target video frame is referred to as a previous adjacent video frame. Specifically, at least one previous adjacent video frame that is adjacent to the target video frame by a predetermined number of frames is obtained. The specific value of the predetermined number can be set according to actual technical needs. For example, the predetermined number can be set in combination with the accuracy or requirements of delayed video playback, and is greater than or equal to 1.

[0044] In one embodiment, at least one of the obtained previous adjacent video frames may be a continuous video frame. Taking the video frames contained in the video as 1, 2, ..., K-2, K-1, K, K+1, K+2 as an example, if the target video frame is the Kth frame and the predetermined number of frames is 2 frames, then at least one of the obtained previous adjacent video frames adjacent to the target video frame by 2 frames may be the K-2th frame, the K-1th frame, etc., wherein the K-2th frame, the K-1th frame and the Kth frame are continuous video frames. If the target video is the Kth frame and the predetermined number of frames is 1 frame, then at least one of the obtained previous adjacent video frames adjacent to the target video frame by 1 frame may be the K-1th frame, etc., wherein the K-1th frame and the Kth frame are continuous video frames.

[0045] In another embodiment, at least one of the previous adjacent video frames obtained may be a continuous video frame, and the nearest previous adjacent video frame and the target video frame may also be separated by a first preset interval, and the first preset interval may be set according to actual technical needs. Taking the video frames contained in the video as 1, 2, ..., K-2, K-1, K, K+1, K+2 as an example, if the target video frame is the Kth frame, the predetermined number of frames is 2 frames, and the first preset interval is 1 frame, then at least one of the previous adjacent video frames obtained that is adjacent to the target video frame by 2 frames may be the K-3th frame and the K-2th frame. If the target video is the Kth frame, the predetermined number of frames is 1 frame, and the first preset interval is 1 frame, then at least one of the previous adjacent video frames obtained that is adjacent to the target video frame by 1 frame may be the K-2th frame.

[0046] In another embodiment, at least one of the acquired preceding adjacent video frames may also be video frames spaced apart by a second preset interval, and the second preset interval may be set according to actual technical needs. Taking the video frames contained in the video as 1, 2, ..., K-2, K-1, K, K+1, K+2 as an example, if the target video frame is the Kth frame, and the predetermined number of frames is 2 frames, and the second preset interval is 1 frame, then at least one of the acquired preceding adjacent video frames adjacent to the target video frame by 2 frames may be the K-2th frame and the K-4th frame, wherein the K-3th frame is spaced apart from the K-2th frame and the K-4th frame. If the target video is the Kth frame, the predetermined number of frames is 1 frame, and the second preset interval is 1 frame, then at least one of the acquired preceding adjacent video frames adjacent to the target video frame by 1 frame is the K-2th frame, wherein the K-1th frame is spaced apart from the K-2th frame and the K-4th frame.

[0047] In another embodiment, at least one of the acquired preceding adjacent video frames may also be video frames spaced apart by a second preset interval, and the nearest preceding adjacent video frame is spaced apart from the target video frame by a first preset interval, and the second preset interval and the first preset interval may be set according to actual technical needs. Taking the video frames contained in the video as 1, 2, ..., K-2, K-1, K, K+1, K+2 as an example, if the target video frame is the Kth frame, the predetermined number of frames is 2 frames, the second preset interval is 1 frame, and the first preset interval is 2 frames, then at least one preceding adjacent video frame obtained that is adjacent to the target video frame by 2 frames may be the K-3th frame and the K-5th frame, wherein the K-3th frame is spaced apart from the K-5th frame by the K-4th frame, and the K-3th frame is spaced apart from the K-2th frame and the K-1th frame.

[0048] Step S206, obtaining at least one subsequent adjacent video frame that is a predetermined number of frames adjacent to the target video frame.

[0049] In one embodiment, a video frame that is after a target video frame and adjacent to the target video frame is referred to as a subsequent adjacent video frame. Specifically, at least one subsequent adjacent video frame that is a predetermined number of frames adjacent to the target video frame is obtained. The specific value of the predetermined number can be set according to actual technical needs. For example, the predetermined number can be set in combination with the accuracy or requirements of delayed video playback, and is greater than or equal to 1.

[0050] In one embodiment, at least one subsequent adjacent video frame obtained may be a continuous video frame. Taking the video frames contained in the video as 1, 2, ... K-2, K-1, K, K+1, K+2 as an example, if the target video frame is the Kth frame and the predetermined number of frames is 2 frames, then at least one subsequent adjacent video frame obtained that is adjacent to the target video frame by 2 frames may be the K+1th frame and the K+2th frame, wherein the Kth frame, the K+1th frame and the K+2th frame are continuous video frames. If the target video frame is the Kth frame and the predetermined number of frames is 1 frame, then at least one subsequent adjacent video frame obtained that is adjacent to the target video frame by 1 frame may be the K+1th frame, wherein the Kth frame and the K+1th frame are continuous video frames.

[0051] In another embodiment, at least one subsequent adjacent video frame obtained may be a continuous video frame, and the nearest subsequent adjacent video frame and the target video frame may be separated by a first preset interval, and the first preset interval may be set according to actual technical needs. Taking the video frames contained in the video as 1, 2, ..., K-2, K-1, K, K+1, K+2 as an example, if the target video frame is the Kth frame, the predetermined number of frames is 2 frames, and the first preset interval is 1 frame, then at least one subsequent adjacent video frame obtained that is adjacent to the target video frame by 2 frames may be the K+2th frame or the K+3th frame. If the target video is the Kth frame, the predetermined number of frames is 1 frame, and the first preset interval is 1 frame, then at least one subsequent adjacent video frame obtained that is adjacent to the target video frame by 1 frame may be the K+2th frame.

[0052] In another embodiment, at least one subsequent adjacent video frame obtained may also be a video frame spaced apart by a second preset interval, and the second preset interval may be set according to actual technical needs. Taking the video frames contained in the video as 1, 2, ..., K-2, K-1, K, K+1, K+2, K+3, K+4 as an example, if the target video frame is the Kth frame, and the predetermined number of frames is 2 frames, and the second preset interval is 1 frame, then at least one subsequent adjacent video frame obtained that is adjacent to the target video frame by 2 frames may be the K+2th frame, the K+4th frame, etc., wherein the K+2th frame is spaced apart from the K+4th frame by the K+3th frame. If the target video is the Kth frame, and the predetermined number of frames is 1 frame, and the second preset interval is 1 frame, then at least one subsequent adjacent video frame obtained that is adjacent to the target video frame by 1 frame may be the K+2th frame, etc., wherein the K+2th frame is spaced apart from the K+1th frame.

[0053] In another embodiment, at least one of the acquired subsequent adjacent video frames may also be video frames spaced apart by a second preset interval, and the nearest preceding adjacent video frame may be spaced apart by a first preset interval from the target video frame, and the second preset interval and the first preset interval may be set according to actual technical needs. Taking the video frames contained in the video as 1, 2, ..., K-2, K-1, K, K+1, K+2, K+3, K+4, K+5 as an example, if the target video frame is the Kth frame, the predetermined number of frames is 2 frames, the second preset interval is 1 frame, and the first preset interval is 2 frames, then at least one preceding adjacent video frame of the two adjacent frames after the target video frame may be the K+3th frame and the K+5th frame, wherein the K+3th frame is spaced apart from the K+5th frame by the K+4th frame, and the K+3th frame is spaced apart from the Kth frame by the K+1th frame and the K+2th frame.

[0054] Among them, the number of the front adjacent video frames and the rear adjacent video frames obtained are the same number, that is, the front and rear adjacent video frames of the target video frame are obtained symmetrically. Therefore, by obtaining a symmetrical number of front adjacent video frames and rear adjacent video frames, the problem of obvious white edges on the edges of the portraits caused by the delay in the moving process when the separated portraits are synthesized with the virtual Beijing can be effectively solved. The specific numerical values ​​of the number of the front adjacent video frames and the rear adjacent video frames obtained can be set according to actual technical needs. For example, it can be set in combination with the accuracy or demand of the delayed video playback, and the number can be greater than or equal to one.

[0055] It should be noted that, in the embodiment of the present application, after the target video frame is processed according to the previous adjacent video frame and the subsequent adjacent video frame, it is necessary to subsequently obtain the number of previous adjacent video frames before playing the video after a certain delay. Therefore, in order to effectively improve the user experience, the duration corresponding to the number of delayed video frames does not exceed the duration that can be recognized by the human eye, that is, the setting of this number can be determined in combination with the number of video frames corresponding to the duration that does not exceed the duration that can be recognized by the human eye.

[0056] Step S208: Process the target video frame based on at least one previous adjacent video frame and at least one subsequent adjacent video frame to obtain a processed target video frame processing result.

[0057] By using the front adjacent video frame and the rear adjacent video frame to process the target video frame, that is, by processing the target video frame with a symmetrical number of front adjacent video frames and rear adjacent video frames, the problem caused by delay in the moving process can be effectively solved, and the quality of the target video frame can be effectively improved.

[0058] In one embodiment, processing the target video frame based on at least one preceding adjacent video frame and at least one succeeding adjacent video frame to obtain a processed target video frame processing result may include the following steps S2061 and S2062.

[0059] Step S2061, determining correction coefficients, the correction coefficients including: a front-adjacent frame correction coefficient of a front-adjacent video frame, a rear-adjacent frame correction coefficient of a rear-adjacent video frame, and a target frame correction coefficient of a target video frame.

[0060] The correction coefficient corresponds to each video frame, and the correction coefficient can also be called a weight. In the embodiment of the present application, the correction coefficient corresponding to the previous adjacent video frame is called the previous adjacent frame correction coefficient, which is expressed as a i , i represents the number of preceding adjacent video frames. The correction coefficient corresponding to the target video frame is called the target frame correction coefficient, denoted as b. The correction coefficient corresponding to the following adjacent video frame is called the following adjacent frame correction coefficient, denoted as c j, j represents the number of subsequent adjacent video frames. Since the number of preceding adjacent video frames is the same as the number of subsequent adjacent video frames, i=j exists.

[0061] In the following embodiments, for the convenience of description, the number of the preceding adjacent video frame and the following adjacent video frame obtained is 1, and the number of adjacent predetermined frames is also 1. If the target video frame is represented as the Kth frame, the preceding adjacent video frame and the following adjacent video frame obtained are respectively represented as the K-1th frame and the K+1th frame, that is, a preceding adjacent video frame and a following adjacent video frame of the target video frame are obtained for the purpose of explanation. At this time, the correction coefficients include: a preceding adjacent frame correction coefficient a, a target frame correction coefficient b, and a following adjacent frame correction coefficient c.

[0062] Step S2062, processing the preceding adjacent video frame, the succeeding adjacent video frame and the target video frame according to the correction coefficient to obtain a processed target video frame processing result.

[0063] The preceding adjacent video frame, the following adjacent video frame and the target video frame are processed according to the correction coefficient, and a fusion process may be performed to obtain a processing result of the target video frame after one frame of processing.

[0064] In the specific fusion processing, it can be a weighted fusion processing. Taking a front adjacent video frame and a rear adjacent video of a target video frame as an example, the correction coefficients include: the front adjacent frame correction coefficient a, the target frame correction coefficient b and the rear adjacent frame correction coefficient c, then the sum of the product of the front adjacent video frame and the front adjacent frame correction coefficient a, the product of the rear adjacent video frame and the rear adjacent frame correction coefficient c, and the product of the target video frame and the target frame correction coefficient b can be determined as the processed target video frame processing result corresponding to the target video frame.

[0065] It should be noted that, in the embodiment of the present application, when the target video frame is corrected using the previous adjacent video frame and the next adjacent video frame, only each target video frame is processed, and the number of video frames in the entire video is not changed.

[0066] When determining the correction coefficient in the above step S2061, one embodiment may include step S2161 and step S2261.

[0067] Step S2161, obtain the processing result of the front adjacent video frame, the processing result of the rear adjacent video frame, and the processing result of the initial video frame, the front adjacent video frame processing result is the processing result obtained by performing video processing on the front adjacent video frame, the rear adjacent video frame processing result is the processing result obtained by performing video processing on the rear adjacent video frame, and the initial video frame processing result is the processing result obtained by performing video processing on the target video frame.

[0068] Video processing refers to the process of performing video processing on video frames. Taking the extraction of a foreground image in a video as an example, the foreground image extraction process may be performed on the video frame. Taking the video processing of the previous adjacent video frame as an example, the method for obtaining the processing result of the previous adjacent video frame includes: performing the foreground image extraction process on the previous adjacent video frame to obtain the processing result of the previous adjacent video frame.

[0069] Among them, the foreground image can be any possible foreground image. In the relevant embodiments of the present application, the foreground image is taken as an example for explanation. At this time, the process of extracting the foreground image is the process of extracting the portrait in the video frame and separating the portrait from the background. That is, taking the previous adjacent video frame as an example, the processing result of the previous adjacent video frame can be obtained by extracting the portrait in the previous adjacent video frame. The video processing method of the target video frame and the subsequent adjacent video frame is the same, that is, the processing result of the subsequent adjacent video frame is obtained by extracting the portrait in the subsequent adjacent video frame, and the processing result of the target video frame is obtained by extracting the portrait in the target video frame.

[0070] Specifically, any feasible method can be used to extract and process the foreground image. In one embodiment, a portrait segmentation model can be used. The portrait segmentation model can select any lightweight segmentation model according to actual technical needs, such as a UNet model. Compared with other network models, the video frame feature map processed by the UNet model incorporates more detail information (Low-level feature), including edges, corners, colors, pixels, and gradients. The model can also perform multiple upsampling, so that the segmentation map restores edge information more accurately, thereby effectively improving the processing speed and efficiency. At the same time, there is no need to annotate data for model training, which can save a lot of manpower and time.

[0071] Among them, after the video frame is processed, the obtained video frame processing result can be a feature vector consistent with the size of the video frame. Different numerical values ​​can be used in the feature vector to represent the portrait and the background. The numerical value can be set according to actual technical needs. For example, after extracting the foreground image, 0 is used to indicate that the pixel corresponding to the position belongs to the background, and 1 is used to indicate that the pixel corresponding to the position belongs to the foreground image, thereby obtaining the video frame processing result after extracting the foreground image, that is, the obtained is a two-dimensional feature vector of the video frame.

[0072] Among them, the processing result of the previous adjacent video frame corresponding to the previous adjacent video frame is represented as Mask t-i, i represents the number of previous adjacent video frames, t represents the video frame identifier corresponding to the target video frame, and the video frame identifier is used to uniquely identify the target video frame and can be set according to actual technical needs. In one embodiment, it can be the timestamp corresponding to the target video frame. The initial video frame processing result corresponding to the target video frame is represented as Mask t The processing result of the subsequent adjacent video frame corresponding to the subsequent adjacent video frame is represented as Mask t+j , j represents the number of the next adjacent video frames. The number of the previous adjacent video frames is the same as the number of the next adjacent video frames, that is, i=j.

[0073] It should be noted that, taking the extraction of foreground image as an example, the Mask of the processing result of the previous adjacent video frame t-i , Initial video frame processing result Mask t And the processing result Mask of the adjacent video frames t+j It refers to the processing result of the entire video frame, regardless of whether there is one or more portraits in the video frame. That is, no matter whether there is one or more portraits in the video frame, the subsequent processing method is the same.

[0074] Step S2261, analyzing the processing result of the previous adjacent video frame, the processing result of the subsequent adjacent video frame, and the processing result of the initial video frame to obtain a correction coefficient.

[0075] When analyzing the processing result of the preceding adjacent video frame, the processing result of the following adjacent video frame, and the processing result of the initial video frame to obtain the correction coefficient, the characteristic matrix composed of the characteristic vector of the processing result of the following adjacent video frame can be combined. Specifically, analyzing the processing result of the preceding adjacent video frame, the processing result of the following adjacent video frame, and the processing result of the initial video frame to obtain the correction coefficient can include the following steps S22611 and S22612.

[0076] Step S22611, determine a feature matrix, wherein the feature matrix includes: a preceding adjacent frame feature vector corresponding to the preceding adjacent video frame processing result, a succeeding adjacent frame feature vector corresponding to the succeeding adjacent video frame processing result, and a target frame feature vector corresponding to the initial video frame processing result.

[0077] Among them, the feature matrix can be combined with the previous adjacent video frame processing result Mask t-i The corresponding feature vector M of the previous adjacent frame t-i , Initial video frame processing result Mask t The corresponding target frame feature vector M t , and the processing result Mask of the adjacent video frames t+j The corresponding feature vector M of the next adjacent frame t+j Determine, in one example, it can be Mask of the processing result of the previous adjacent video framet-i The corresponding feature vector M of the previous adjacent frame t-i , Initial video frame processing result Mask t The corresponding target frame feature vector M t , and the processing result Mask of the adjacent video frames t+j The corresponding feature vector M of the next adjacent frame t+j composition.

[0078] Among them, when determining the feature matrix by combining the feature vector of the previous adjacent frame, the feature vector of the target frame and the feature vector of the subsequent adjacent frame, the row and column positions of the feature vector of the previous adjacent frame, the feature vector of the target frame and the feature vector of the subsequent adjacent frame in the feature matrix can be set according to predetermined positions or can be not distinguished.

[0079] For example, in some embodiments, the positions of the feature vectors of the preceding adjacent frame, the feature vectors of the target frame, and the feature vectors of the following adjacent frames in the feature matrix may be determined according to the sequence of the video frames. Taking a preceding adjacent video frame and a following adjacent video frame of the target video frame as an example, and each feature vector occupies a column of the feature matrix, the determined feature matrix may be expressed as:

[0080] (previous adjacent frame feature vector, target frame feature vector, next adjacent frame feature vector)

[0081] Taking the case where a previous adjacent video frame and a subsequent adjacent video frame of the target video frame are obtained, and each feature vector occupies one row of the feature matrix, the determined feature matrix can be expressed as:

[0082]

[0083] For example, in some other embodiments, it is not necessary to limit the positions of the preceding adjacent frame feature vector, the target frame feature vector, and the following adjacent frame feature vector in the feature matrix. Similarly, taking a preceding adjacent video frame and a following adjacent video frame of the target video frame as an example, and each feature vector occupies a column of the feature matrix, the determined feature matrix can be any one of the following various methods:

[0084] (previous adjacent frame feature vector, target frame feature vector, next adjacent frame feature vector)

[0085] (previous adjacent frame feature vector, next adjacent frame feature vector, target frame feature vector)

[0086] (Feature vector of the adjacent frame before the target frame feature vector and the feature vector of the adjacent frame after the target frame feature vector)

[0087] (The feature vector of the adjacent frame after the target frame feature vector and the feature vector of the adjacent frame before it)

[0088] (feature vector of the next adjacent frame, feature vector of the previous adjacent frame, feature vector of the target frame)

[0089] (Feature vector of the next adjacent frame, feature vector of the target frame, feature vector of the previous adjacent frame)

[0090] If a preceding adjacent video frame and a following adjacent video frame of a target video frame are obtained, and each feature vector occupies one row of the feature matrix, the feature matrix can be determined in any of the following ways:

[0091]

[0092]

[0093] Wherein, when obtaining the feature vector of the preceding adjacent frame, the feature vector of the target frame and the feature vector of the following adjacent frame, the corresponding feature vectors of the video frame can be obtained in the same manner respectively. Taking the preceding adjacent frame feature vector corresponding to the processing result of the preceding adjacent video frame as an example, the manner of obtaining the preceding adjacent frame feature vector corresponding to the processing result of the preceding adjacent video frame includes: scaling the processing result of the preceding adjacent video frame to a predetermined size; vectorizing the processing result of the preceding adjacent video frame of the predetermined size to obtain the feature vector of the preceding adjacent frame. Correspondingly, when obtaining the feature vector of the target frame and the feature vector of the following adjacent frame, the processing result of the following adjacent video frame can be scaled to a predetermined size, vectorized to the processing result of the following adjacent video frame of the predetermined size to obtain the feature vector of the following adjacent frame; and the processing result of the target video frame can be scaled to a predetermined size, vectorized to the processing result of the target video frame of the predetermined size to obtain the feature vector of the target frame.

[0094] Therefore, by first preprocessing the video frame processing result, and then vectorizing the preprocessed video frame processing result, that is, converting it into a corresponding video frame feature vector, the real-time processing and accuracy requirements can be taken into account at the same time. The preprocessing method can be selected according to actual technical needs, and in one embodiment, it can be a scaling process, and the vectorization process can be performed in any feasible way.

[0095] Specifically, the predetermined size can be set according to actual technical needs. In theory, the larger the size, the better the effect of the target video frame obtained after the final processing, but the corresponding real-time performance of the processing is worse. Within the acceptable real-time performance range, in one embodiment, the preset size can be set to 16*16, and the 16*16 video frame processing result is vectorized to obtain a corresponding 1*256 video frame feature vector, that is, the previous adjacent frame feature vector, the next adjacent frame feature vector and the target frame feature vector are all 1*256 video frame feature vectors, that is, the determined feature matrix can be a 3*256 feature matrix.

[0096] Step S22612, estimate the target frame feature vector and feature matrix to obtain the correction coefficient.

[0097] The target frame feature vector and feature matrix are estimated to obtain the correction coefficient by any feasible calculation method according to actual technical needs.

[0098] In one embodiment of the present application, a ridge regression algorithm can be used to estimate the correction coefficient, wherein the ridge regression algorithm is a biased estimation regression method specifically used for collinear data analysis, and is an improved least squares estimation method. The regression coefficient obtained by the ridge regression algorithm is more practical and more reliable. The regression coefficient in the ridge regression algorithm is the correction coefficient in the embodiment of the present application.

[0099] Accordingly, in one embodiment of the present application, the target frame feature vector and feature matrix are estimated to obtain the correction coefficient, specifically including:

[0100] Determine a coefficient matrix to be confirmed corresponding to the feature matrix and a deviation coefficient of the coefficient matrix to be confirmed, wherein the coefficient matrix to be confirmed includes: a coefficient to be confirmed of a preceding adjacent frame corresponding to a feature vector of a preceding adjacent frame, a coefficient to be confirmed of a succeeding adjacent frame corresponding to a feature vector of a succeeding adjacent frame, and a coefficient to be confirmed of a target frame corresponding to a feature vector of a target frame;

[0101] The coefficient to be confirmed in the coefficient matrix to be confirmed when the fitting relationship amount is the smallest is determined as the correction coefficient. The fitting relationship amount is the sum of the difference norm and the estimated deviation. The difference norm is the norm corresponding to the difference between the predicted variable and the target frame feature vector. The predicted variable is the product of the feature matrix and the coefficient matrix to be confirmed. The estimated deviation is the product of the deviation coefficient and the norm corresponding to the coefficient matrix to be confirmed.

[0102] The matrix of coefficients to be confirmed can be denoted as α, and the matrix elements in the matrix of coefficients to be confirmed are the coefficients to be confirmed. The coefficients to be confirmed refer to the correction coefficients to be determined corresponding to the video frame. In the embodiment of the present application, the coefficients to be confirmed corresponding to the preceding adjacent video frame are referred to as the preceding adjacent frame coefficients to be confirmed, the coefficients to be confirmed corresponding to the succeeding adjacent video frame are referred to as the succeeding adjacent frame coefficients to be confirmed, and the coefficients to be confirmed corresponding to the target video frame are referred to as the target frame coefficients to be confirmed.

[0103] It should be noted that the coefficient matrix to be confirmed corresponds to the feature matrix, that is, the element position of the coefficient to be confirmed in the coefficient matrix to be confirmed is consistent with the element position of the video frame feature vector in the feature matrix, to ensure that the video frame feature vector and the correction coefficient correspond.

[0104] For example, taking a previous adjacent video frame and a subsequent adjacent video frame of a target video frame as an example, if the feature matrix is ​​expressed as:

[0105] (previous adjacent frame feature vector, target frame feature vector, next adjacent frame feature vector)

[0106] Then the coefficient matrix to be confirmed corresponding to the feature matrix is ​​expressed as:

[0107] (Previous adjacent frame coefficients to be confirmed, target frame coefficients to be confirmed, and subsequent adjacent frame coefficients to be confirmed)

[0108] The deviation coefficient of the coefficient matrix to be confirmed is recorded as λ, and the deviation coefficient is a constant and a non-negative number. In theory, the larger the deviation coefficient, the smaller the regression coefficient. However, if the deviation coefficient is too large, the estimated value of the regression coefficient will deviate further from the true value, resulting in poor effect of the processed target video frame. Therefore, the deviation coefficient should be reasonably set according to actual technical needs. In one embodiment, the deviation coefficient can be set to 1.

[0109] The product of the feature matrix and the coefficient matrix to be confirmed is used as the prediction variable, so that the prediction variable can reflect the effect of the processed target video frame. The norm corresponding to the difference between the prediction variable and the target frame feature vector is used as the difference norm. Therefore, the smaller the value of the fitting relationship, the smaller the difference norm, that is, the closer the prediction variable is to the target frame feature vector. Since the prediction variable is determined according to the feature matrix, and the feature matrix includes the feature vector of the previous adjacent frame, the feature vector of the target frame, and the feature vector of the next adjacent frame, the closer the prediction variable is to the feature vector of the target frame, the better the effect of processing the target video frame.

[0110] The product of the deviation coefficient and the norm corresponding to the coefficient matrix to be confirmed is taken as the estimated deviation, so that the estimated deviation can characterize the penalty term for the fitting relationship quantity, which can be expressed as λ‖α‖, which is used to balance the variance and deviation in the fitting process. The addition of the penalty term makes the estimation of the regression coefficient no longer an unbiased estimate, thereby obtaining a higher calculation accuracy.

[0111] Specifically, in the embodiment of the present application, when the value of the fitting relationship amount is the smallest, the corresponding coefficient to be confirmed in the coefficient matrix to be confirmed is determined as the correction coefficient, so that a coefficient that effectively resists the influence caused by the delay of the moving process can be obtained.

[0112] Taking the acquisition of a preceding adjacent video frame and a following adjacent video frame of the target video frame as an example, the processing result of the preceding adjacent video frame is represented as Mask t-1 , the feature vector of the previous adjacent frame is represented as M t-1 , the initial video frame processing result is represented as Mask t , the target frame feature vector is represented as M t , and the processing result of the adjacent video frames is represented as Mask t+1 , the feature vector of the next adjacent frame is represented as M t+1 .

[0113] The determined eigenvector is expressed as (M t-1 M t M t+1 ) as an example, the coefficient matrix α to be confirmed is expressed as: (a bc), and the predicted variable is expressed as: (M t-1 M t M t+1 )*α, the estimated deviation is expressed as: λ‖α‖, then the fitting relationship y is expressed as:

[0114]

[0115] Determine the coefficient matrix α corresponding to the minimum value of the fitting relationship quantity y, and determine the coefficients to be confirmed in the coefficient matrix α as the correction coefficients, that is, the processing result Mask of the previous adjacent video frame t-1 The corresponding correction coefficient of the previous adjacent frame is a, and the initial video frame processing result Mask t The corresponding target frame correction coefficient is b, and the adjacent video frame processing result Mask t+1 The corresponding correction coefficient of the subsequent adjacent frame is c, where the value range of the correction coefficient is between 0 and 1.

[0116] It should be noted that when the correction coefficient is determined by the ridge regression method, when extracting the foreground image in the application embodiment, when 0 is used to indicate that the pixel corresponding to the position belongs to the background, and 1 is used to indicate that the pixel corresponding to the position belongs to the foreground image, what is obtained is a two-dimensional feature vector of the video frame, which is equivalent to completing the decentralization and standardization of the data in the process of extracting the foreground image, thereby further improving the data processing efficiency during the ridge regression processing.

[0117] After determining the correction coefficients corresponding to the video frame processing results, the corresponding video frames can be processed according to the correction coefficients to obtain the processed target video frame processing results.

[0118] Specifically, in the above step S2062, the preceding adjacent video frame, the following adjacent video frame and the target video frame are processed according to the correction coefficient to obtain the processed target video frame processing result, which may specifically include:

[0119] According to the correction coefficient of the previous adjacent frame, the correction coefficient of the next adjacent frame, and the correction coefficient of the target frame, the processing result of the previous adjacent video frame, the processing result of the next adjacent video frame, and the processing result of the initial video frame are fused to obtain the processed target video frame processing result.

[0120] In a specific example, the weighted sum of the processing result of the front adjacent video frame, the processing result of the rear adjacent video frame, and the processing result of the initial video frame can be performed according to the correction coefficient of the front adjacent frame, the correction coefficient of the rear adjacent frame, and the correction coefficient of the target frame. That is, the sum of the product of the front adjacent frame correction coefficient and the processing result of the front adjacent video frame, the product of the rear adjacent frame correction coefficient and the processing result of the rear adjacent video frame, and the product of the target frame correction coefficient and the processing result of the initial video frame can be used as the processed target video frame processing result, which is expressed as Mask t ′.

[0121] Taking the above-mentioned acquisition of a front adjacent video frame and a rear adjacent video of the target video frame as an example, the obtained processed target video frame processing result can be expressed by the formula:

[0122] Mask′ t =a*Mask t-1 +b*Mask t +c*Mask t+1

[0123] In one embodiment, after obtaining the processed target video frame processing result, it also includes: delaying the target video frame by a predetermined number, and replacing the initial video frame processing result in the target video frame with the processed target video frame processing result before playing.

[0124] Among them, taking a front adjacent video frame and a rear adjacent video of a target video frame as an example, after obtaining the target video frame, the target video frame is stored, and the rear adjacent video frame after the target video frame is obtained, and the target video frame is corrected in combination with 1 frame of front adjacent video frame and 1 frame of rear adjacent video frame, after obtaining the processed target video frame processing result, the processed target video frame processing result replaces the initial video frame processing result in the target video frame, and then the target video frame is played in combination with the processed target video frame processing result.

[0125] In the scenario of real-time video streaming, such as video conferencing, live broadcasting, etc., the target video frame is not played in real time after it is obtained in real time, but needs to be played after the adjacent video frame is obtained. Therefore, the target video frame is actually played after a delay of a predetermined number of frames. By setting the predetermined number reasonably, as long as the value of the predetermined number is set reasonably, the human eye cannot perceive the existence of this delay, so that a better playback effect of the video frame processing result can be obtained while playing in real time.

[0126] In the above video processing method, by acquiring a target video frame; acquiring at least one front adjacent video frame of a predetermined number of frames before the target video frame; acquiring at least one rear adjacent video frame of a predetermined number of frames after the target video frame; processing the target video frame based on at least one front adjacent video frame and at least one rear adjacent video frame, a processing result of the processed target video frame is obtained. The method of the above embodiment is adopted, by symmetrically acquiring the front adjacent video frame and the rear adjacent video frame of the target video frame, the target video frame is corrected, the characteristics of the front adjacent video frame and the rear adjacent video frame are fully utilized, and the influence of the front adjacent video frame and the rear adjacent video frame on the target video frame are considered, that is, while considering the influence of the front adjacent video frame on the target video frame, the influence of the rear adjacent video frame is also fully considered, which can offset the delay problem caused by processing the target video frame only according to the first N video frames of the target video frame, thereby effectively resisting the influence caused by the delay of the moving process in the video processing process, thereby improving the processing effect of the target video frame.

[0127] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and a specific embodiment. It should be understood that the specific embodiment described here is only used to explain the present application and is not used to limit the present application.

[0128] In a specific embodiment, a user conducts a real-time video conference through a video conferencing software. The video conferencing software is provided with a virtual background function, such as Figure 3 The figure shows a schematic diagram of selecting a virtual background for a video. The user can select any image to replace the background of the video playback area. The selected image can be an image preset in the video conferencing software or an image uploaded by the user through the image upload interface. The user can select a virtual background at any time, that is, before the video conference starts or during the video conference, and there is no restriction here.

[0129] When the user selects a virtual background for a video, the virtual background function can separate the portrait and background in the video. The virtual background function mainly includes portrait segmentation, custom portrait segmentation, portrait segmentation SDK, and video portrait segmentation. Among them, portrait segmentation can identify the complete outline of the human body in the picture, segment it from the background, and return a grayscale image and a foreground portrait image. Custom portrait segmentation can identify the outline of the facial features and the background segmentation, and supports the segmentation of hair, facial features, clothing, etc. of a single portrait image, which can be used for fun expression production and other gameplay. The portrait segmentation SDK can call the SDK offline in real-time video streaming scenarios to achieve portrait segmentation, which is suitable for background replacement in short videos, live broadcasts, online meetings and other scenarios. Video portrait segmentation can process user-uploaded videos in the cloud and output the segmented videos for video editing and production, entertainment film and television background replacement, etc.

[0130] Combination Figure 4 As shown, in a specific example, taking a front adjacent video frame and a rear adjacent video frame of a target video frame as an example, the overall process of the video processing method in the embodiment of the present application can be described as follows.

[0131] First, a target video frame is obtained, where the target video frame is a video frame to be processed.

[0132] Then, the model is extracted by the foreground image (e.g. Figure 4 The foreground image extraction model can be a portrait segmentation model. The portrait segmentation model can select any lightweight segmentation model according to actual technical needs. In one embodiment, the UNet model is selected to separate the portrait and background in the target video frame, and the initial video frame processing result Mask is obtained accordingly. t .

[0133] Get the processing result Mask of the previous adjacent video frame of the previous video frame (previous adjacent video frame) adjacent to the target video frame t-1 , the processing result Mask of the next adjacent video frame (next adjacent video frame) t+1 The processing result of the preceding adjacent video frame and the processing result of the succeeding adjacent video frame may be obtained by pre-processing or by real-time processing.

[0134] Taking a data stream played in non-real time as an example, each video frame may be processed to obtain a corresponding video frame processing result (i.e., a foreground image), and then each video frame processing result may be stored. When the embodiment of the present application is tried, each video frame processing result may be directly obtained from a corresponding storage location. Alternatively, each video frame may be processed in real time to obtain a corresponding video frame processing result during the video stream processing.

[0135] Taking the real-time playback data stream as an example, since the front adjacent video frame is obtained before the target video frame is obtained, the corresponding front adjacent video frame processing result can be obtained when the front adjacent video frame is obtained, and processed and stored, and the rear adjacent video frame is obtained after the target video frame is obtained. Therefore, the rear adjacent video frame processing result corresponding to the rear adjacent video frame can be obtained in real time after the rear adjacent video frame is obtained.

[0136] Then, the front adjacent video frame processing result, the initial video frame processing result and the rear adjacent video frame processing result are scaled to a predetermined size, and the front adjacent video frame processing result, the initial video frame processing result and the rear adjacent video frame processing result of the predetermined size are vectorized to obtain a front adjacent frame feature vector, a target frame feature vector and a rear adjacent frame feature vector; wherein the predetermined size can be set according to actual technical needs. In one embodiment, the preset size can be set to 16*16, that is, a 1*256 front adjacent frame feature vector M corresponding to the subsequent vectorization process t-1 , target frame feature vector M t and the feature vector M of the next adjacent frame t+1 .

[0137] Determine a feature matrix, the feature matrix includes: a feature vector of a preceding adjacent frame, a feature vector of a target frame, and a feature vector of a succeeding adjacent frame. The feature matrix of an embodiment can be expressed as (M t-1 M t M t+1 ).

[0138] Determine the coefficient matrix to be confirmed corresponding to the feature matrix and the deviation coefficient of the coefficient matrix to be confirmed, wherein the coefficient matrix to be confirmed includes: the coefficients to be confirmed of the preceding adjacent frame corresponding to the feature vector of the preceding adjacent frame, the coefficients to be confirmed of the succeeding adjacent frame corresponding to the feature vector of the succeeding adjacent frame, and the coefficients to be confirmed of the target frame corresponding to the feature vector of the target frame.

[0139] Then, the coefficient to be confirmed in the coefficient matrix to be confirmed when the fitting relationship amount is the smallest is determined as the correction coefficient. The fitting relationship amount is the sum of the difference norm and the estimated deviation. The difference norm is the norm corresponding to the difference between the predicted variable and the target frame feature vector. The predicted variable is the product of the feature matrix and the coefficient matrix to be confirmed. The estimated deviation is the product of the deviation coefficient and the norm corresponding to the coefficient matrix to be confirmed.

[0140] Among them, the feature vector M of the previous adjacent frame t-1 The corresponding coefficient of the previous adjacent frame to be confirmed is recorded as a, and the feature vector of the next adjacent frame is M t+1 The corresponding coefficient of the next adjacent frame to be confirmed is recorded as c, and the target frame feature vector M tThe corresponding target frame coefficient to be confirmed is recorded as b, then the coefficient matrix α to be confirmed is expressed as: (abc), and the predicted variable is expressed as: (M t-1 M t M t+1 )*α, the estimated deviation is expressed as: λ‖α‖, then the fitting relationship can be expressed as:

[0141]

[0142] Determine the coefficient matrix α corresponding to the minimum value of the fitting relationship, and determine the coefficients to be confirmed in the coefficient matrix α as the correction coefficients, that is, the processing result Mask of the previous adjacent video frame t-1 The corresponding correction coefficient of the previous adjacent frame is a, and the initial video frame processing result Mask t The corresponding target frame correction coefficient is b, and the adjacent video frame processing result Mask t+1 The corresponding correction coefficient of the subsequent adjacent frame is c.

[0143] Then, according to the correction coefficient of the previous adjacent frame, the correction coefficient of the next adjacent frame, and the correction coefficient of the target frame, the processing result of the previous adjacent video frame, the processing result of the next adjacent video frame, and the processing result of the initial video frame are fused to obtain the processed processing result of the target video frame; wherein the processed processing result of the target video frame is represented as Mask t ′, the calculation formula of the processed target video frame processing result is expressed as:

[0144] Mask′ t =a*Mask t-1 +b*Mask t +c*Mask t+1

[0145] When obtaining the processing result of the processed target video frame, since it is necessary to obtain at least one subsequent adjacent video frame of a predetermined number of adjacent frames after the target video frame, it is actually equivalent to delaying the target video frame by a predetermined number. Therefore, after obtaining the target video frame, the initial video frame processing result in the target video frame is replaced by the processed target video frame processing result for playback.

[0146] like Figure 5 FIG. 1 is a schematic diagram showing the effect of the video processing method, wherein: Figure 5-1 Schematic diagram of a target video frame after smoothing filtering or mean filtering is performed on the target video frame in combination with the first N video frames of the target video frame in the conventional technology, Figure 5-2This is a schematic diagram of a processed target video frame after processing the target video frame in an embodiment of the present application by combining at least one preceding adjacent video frame of a predetermined number of frames preceding the target video frame and at least one succeeding adjacent video frame of a predetermined number of frames succeeding the target video frame. Figure 5-1 It can be clearly seen from Figure 5-2 that the traditional technology method brings obvious "white edges" at the edge of the portrait in the target video frame. Figure 5-1 It is indicated by a dotted line in the figure, and the method of the embodiment of the present application has a better processing effect on the target video frame.

[0147] It should be understood that, although the steps in each flow chart involved in the above-described embodiment are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in each flow chart involved in the above-described embodiment may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0148] Based on the same inventive concept, the present application also provides a video processing device for implementing the above-mentioned video processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations of one or more video processing device embodiments provided below can refer to the limitations of the video processing method above, and will not be repeated here.

[0149] In one embodiment, if Figure 6 As shown, a video processing device 600 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: a first acquisition module 610, a second acquisition module 620, a third acquisition module 630 and a frame processing module 640, wherein:

[0150] The first acquisition module 610 is used to acquire a target video frame.

[0151] The second acquisition module 620 is configured to acquire at least one preceding adjacent video frame that is a predetermined number of frames adjacent to the target video frame.

[0152] The third acquisition module 630 is configured to acquire at least one subsequent adjacent video frame that is adjacent to the target video frame by the predetermined number of frames.

[0153] The frame processing module 640 is used to process the target video frame based on the at least one previous adjacent video frame and the at least one subsequent adjacent video frame to obtain a processed target video frame processing result.

[0154] In one embodiment, the frame processing module 640 includes:

[0155] The correction coefficient determination unit is used to determine the correction coefficient, wherein the correction coefficient includes: a previous adjacent frame correction coefficient of the previous adjacent video frame, a subsequent adjacent frame correction coefficient of the subsequent adjacent video frame, and a target frame correction coefficient of the target video frame.

[0156] The frame processing unit is used to process the preceding adjacent video frame, the succeeding adjacent video frame and the target video frame according to the correction coefficient to obtain a processed target video frame processing result.

[0157] In one embodiment, the correction coefficient determination unit is used to obtain a processing result of a preceding adjacent video frame, a processing result of a succeeding adjacent video frame, and a processing result of an initial video frame, wherein the processing result of the preceding adjacent video frame is a processing result obtained by performing video processing on the preceding adjacent video frame, the processing result of the succeeding adjacent video frame is a processing result obtained by performing the video processing on the succeeding adjacent video frame, and the initial video frame processing result is a processing result obtained by performing the video processing on the target video frame; the processing result of the preceding adjacent video frame, the processing result of the succeeding adjacent video frame, and the processing result of the initial video frame are analyzed to obtain the correction coefficient.

[0158] In one embodiment, the correction coefficient determination unit is used to determine a feature matrix, which includes: a preceding adjacent frame feature vector corresponding to the preceding adjacent video frame processing result, a succeeding adjacent frame feature vector corresponding to the succeeding adjacent video frame processing result, and a target frame feature vector corresponding to the initial video frame processing result; the target frame feature vector and the feature matrix are estimated to obtain the correction coefficient.

[0159] In one embodiment, the correction coefficient determination unit is used to determine the coefficient matrix to be confirmed corresponding to the feature matrix and the deviation coefficient of the coefficient matrix to be confirmed, the coefficient matrix to be confirmed includes: the coefficient to be confirmed of the previous adjacent frame corresponding to the feature vector of the previous adjacent frame, the coefficient to be confirmed of the subsequent adjacent frame corresponding to the feature vector of the subsequent adjacent frame, and the coefficient to be confirmed of the target frame corresponding to the feature vector of the target frame; and determine the coefficient to be confirmed in the coefficient matrix to be confirmed when the fitting relationship amount is the smallest as the correction coefficient, the fitting relationship amount is the sum of the difference norm and the estimated deviation, the difference norm is the norm corresponding to the difference between the predicted variable and the feature vector of the target frame, the predicted variable is the product of the feature matrix and the coefficient matrix to be confirmed, and the estimated deviation is the product of the deviation coefficient and the norm corresponding to the coefficient matrix to be confirmed.

[0160] In one of the embodiments, the frame processing unit is used to fuse the previous adjacent frame processing result, the subsequent adjacent frame processing result, and the initial video frame processing result according to the previous adjacent frame correction coefficient, the subsequent adjacent frame correction coefficient, and the target frame correction coefficient to obtain a processed target video frame processing result.

[0161] In one embodiment, the video processing device 600 further includes:

[0162] The video playing module is used to delay the target video frame by the predetermined number, and replace the initial video frame processing result in the target video frame with the processed target video frame processing result before playing.

[0163] For the specific definition of the video processing device, please refer to the definition of the video processing method above, which will not be repeated here. Each module in the above video processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0164] In one embodiment, a computer device is provided, whose internal structure diagram can be as follows: Figure 7As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0165] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0166] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0167] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0168] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0169] Those of ordinary skill in the art can understand that all or part of the processes in the methods for implementing the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0171] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A video processing method, characterized in that: The method comprises: Get the target video frame; Acquire at least one preceding adjacent video frame that is a predetermined number of frames adjacent to the target video frame; Acquire at least one subsequent adjacent video frame that is adjacent to the target video frame by the predetermined number of frames; Obtaining a front adjacent video frame processing result, a rear adjacent video frame processing result, and an initial video frame processing result corresponding to the front adjacent video frame, the rear adjacent video frame, and the target video frame; Determine a feature matrix, the feature matrix comprising: a front adjacent frame feature vector corresponding to the front adjacent video frame processing result, a rear adjacent frame feature vector corresponding to the rear adjacent video frame processing result, and a target frame feature vector corresponding to the initial video frame processing result; Determine a coefficient matrix to be confirmed corresponding to the feature matrix and a deviation coefficient of the coefficient matrix to be confirmed, wherein the coefficient matrix to be confirmed includes: a coefficient to be confirmed of a preceding adjacent frame corresponding to the feature vector of the preceding adjacent frame, a coefficient to be confirmed of a succeeding adjacent frame corresponding to the feature vector of the succeeding adjacent frame, and a coefficient to be confirmed of a target frame corresponding to the feature vector of the target frame; Determine the coefficient to be confirmed in the coefficient matrix to be confirmed when the fitting relationship amount is the smallest as the correction coefficient, wherein the fitting relationship amount is the sum of the difference norm and the estimated deviation, the difference norm is the norm corresponding to the difference between the predicted variable and the target frame feature vector, the predicted variable is the product of the feature matrix and the coefficient matrix to be confirmed, and the estimated deviation is the product of the deviation coefficient and the norm corresponding to the coefficient matrix to be confirmed; The preceding adjacent video frame, the following adjacent video frame and the target video frame are processed according to the correction coefficient to obtain a processed target video frame processing result.

2. The video processing method according to claim 1, characterized in that: The correction coefficients include: a previous adjacent frame correction coefficient of the previous adjacent video frame, a subsequent adjacent frame correction coefficient of the subsequent adjacent video frame, and a target frame correction coefficient of the target video frame.

3. The video processing method according to claim 2, characterized in that: The preceding adjacent video frame processing result is a processing result obtained by performing video processing on the preceding adjacent video frame, the following adjacent video frame processing result is a processing result obtained by performing the video processing on the following adjacent video frame, and the initial video frame processing result is a processing result obtained by performing the video processing on the target video frame.

4. The video processing method according to claim 3, characterized in that: The processing of the preceding adjacent video frame, the following adjacent video frame and the target video frame according to the correction coefficient to obtain a processed target video frame processing result includes: According to the previous adjacent frame correction coefficient, the subsequent adjacent frame correction coefficient, and the target frame correction coefficient, the previous adjacent video frame processing result, the subsequent adjacent video frame processing result, and the initial video frame processing result are fused to obtain a processed target video frame processing result.

5. The video processing method according to claim 3, characterized in that: The method further comprises: The target video frame is delayed by the predetermined number, and the initial video frame processing result in the target video frame is replaced by the processed target video frame processing result before playing.

6. A video processing device, characterized in that: The device comprises: A first acquisition module, used to acquire a target video frame; A second acquisition module is used to acquire at least one previous adjacent video frame that is a predetermined number of frames adjacent to the target video frame; A third acquisition module is used to acquire at least one subsequent adjacent video frame that is adjacent to the target video frame by the predetermined number of frames; A frame processing module, comprising a correction coefficient determination unit and a frame processing unit; The correction coefficient determination unit is used to obtain the front adjacent video frame processing results, the rear adjacent video frame processing results and the initial video frame processing results corresponding to the front adjacent video frame, the rear adjacent video frame and the target video frame; determine a feature matrix, the feature matrix including: a front adjacent frame feature vector corresponding to the front adjacent video frame processing result, a rear adjacent frame feature vector corresponding to the rear adjacent video frame processing result, and a target frame feature vector corresponding to the initial video frame processing result; determine a coefficient matrix to be confirmed corresponding to the feature matrix, and a deviation coefficient of the coefficient matrix to be confirmed, the coefficient matrix to be confirmed including : the coefficient of the preceding adjacent frame to be confirmed corresponding to the feature vector of the preceding adjacent frame, the coefficient of the following adjacent frame to be confirmed corresponding to the feature vector of the following adjacent frame, and the coefficient of the target frame to be confirmed corresponding to the feature vector of the target frame; the coefficient of the to-be-confirmed coefficient in the to-be-confirmed coefficient matrix when the fitting relationship amount is the smallest is determined as the correction coefficient, the fitting relationship amount is the sum of the difference norm and the estimated deviation, the difference norm is the norm corresponding to the difference between the predicted variable and the feature vector of the target frame, the predicted variable is the product of the feature matrix and the coefficient matrix to be confirmed, and the estimated deviation is the product of the deviation coefficient and the norm corresponding to the coefficient matrix to be confirmed; The frame processing unit is used to process the preceding adjacent video frame, the succeeding adjacent video frame and the target video frame according to the correction coefficient to obtain a processed target video frame processing result.

7. The video processing device according to claim 6, characterized in that: The correction coefficients include: a previous adjacent frame correction coefficient of the previous adjacent video frame, a subsequent adjacent frame correction coefficient of the subsequent adjacent video frame, and a target frame correction coefficient of the target video frame.

8. The video processing device according to claim 7, characterized in that: The preceding adjacent video frame processing result is a processing result obtained by performing video processing on the preceding adjacent video frame, the following adjacent video frame processing result is a processing result obtained by performing the video processing on the following adjacent video frame, and the initial video frame processing result is a processing result obtained by performing the video processing on the target video frame.

9. The video processing device according to claim 8, characterized in that: The processing of the preceding adjacent video frame, the following adjacent video frame and the target video frame according to the correction coefficient to obtain a processed target video frame processing result includes: According to the previous adjacent frame correction coefficient, the subsequent adjacent frame correction coefficient, and the target frame correction coefficient, the previous adjacent video frame processing result, the subsequent adjacent video frame processing result, and the initial video frame processing result are fused to obtain a processed target video frame processing result.

10. The video processing device according to claim 8, characterized in that: The device also includes: The target video frame is delayed by the predetermined number, and the initial video frame processing result in the target video frame is replaced by the processed target video frame processing result before playing.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the video processing method according to any one of claims 1 to 5 are implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 5 are implemented.

13. A computer program product, comprising computer instructions, wherein when the computer instructions are executed by a processor, the steps of the video processing method according to any one of claims 1 to 5 are implemented.

Images