Video processing method, video processing device, storage medium and electronic equipment

Abstract

The invention relates to the technical field of image processing, in particular to a video processing method and device, a computer readable storage medium and electronic equipment. The method comprises the following steps: determining a motion matrix between a current frame of a video sequence and a reference frame of the current frame; determining a second pixel point matched with the first pixel point of the current frame in the reference frame according to the motion matrix so as to obtain a pixel point matching pair; based on the first segmented image of the current frame and the second segmented image of the reference frame, determining a regional relationship between pixel points in the pixel point matching pair; and according to the regional relationship, fusing the first depth map of the current frame and the second depth map of the reference frame to obtain a target depth map of the current frame. The video sequence depth value prediction efficiency and the inter-frame stability can be improved.

Description

Background Art

[0002] The depth value prediction of a video sequence includes the depth value prediction of a single frame image and the smoothing of inter-frame prediction results.

[0003] In the related art, the corresponding pixel points of the previous and next frames are matched using optical flow or deep learning methods, which requires a lot of calculations and may cause incorrect fusion of the previous and next frames due to motion blur in the edge area, resulting in low accuracy of depth value prediction.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a video processing method, a video processing device, a computer-readable medium and an electronic device, thereby improving the efficiency and accuracy of video depth prediction at least to a certain extent.

[0006] According to a first aspect of the present disclosure, a video processing method is provided, including: determining a motion matrix between a current frame of a video sequence and a reference frame of the current frame; determining, based on the motion matrix, a second pixel point that matches a first pixel point of the current frame in the reference frame to obtain a pixel point matching pair; determining, based on a first segmented image of the current frame and a second segmented image of the reference frame, a regional relationship between pixels in the pixel point matching pair; and, based on the regional relationship, fusing a first depth map of the current frame and a second depth map of the reference frame to obtain a target depth map of the current frame.

[0007] According to a second aspect of the present disclosure, a video processing device is provided, comprising: a motion matrix determination module, configured to determine a motion matrix between a current frame of a video sequence and a reference frame of the current frame; a pixel matching pair determination module, configured to determine, based on the motion matrix, a second pixel matching a first pixel of the current frame in the reference frame to obtain a pixel matching pair; a region relationship determination module, configured to determine, based on a first segmented image of the current frame and a second segmented image of the reference frame, a region relationship between pixels in the pixel matching pair; and a target depth map generation module, configured to fuse a first depth map of the current frame and a second depth map of the reference frame based on the region relationship to obtain a target depth map of the current frame.

[0008] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video processing method described in the first aspect is implemented.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the video processing method described in the first aspect above.

[0010] According to a fifth aspect of the present disclosure, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the steps of the video processing method as described in the first aspect.

[0011] The technical solution disclosed in this disclosure has the following beneficial effects:

[0012] On the one hand, the present invention determines the matching pixels between the current frame and the reference frame through a motion matrix, which can reduce the amount of calculation for pixel matching and improve the efficiency of video depth value prediction. On the other hand, the present invention determines the segmentation area relationship of the pixels in the pixel matching pair based on the segmented image, thereby fusing the depth values ​​based on the segmentation area relationship, reducing the problem of erroneous fusion caused by motion blur in the edge area and improving the accuracy of video depth value prediction.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0015] Figure 1 A schematic diagram showing a system architecture to which exemplary embodiments of the present disclosure may be applied;

[0016] Figure 2 A schematic flow chart showing a video processing method in an exemplary embodiment of the present disclosure;

[0017] Figure 3 A schematic flow chart showing a method for determining a motion matrix in an exemplary embodiment of the present disclosure;

[0018] Figure 4 A schematic flow chart showing a method for determining regional relationships in an exemplary embodiment of the present disclosure;

[0019] Figure 5A schematic diagram showing a flow chart of a method for fusing a first depth map and a second depth map in an exemplary embodiment of the present disclosure;

[0020] Figure 6 A schematic diagram of spatial filtering in an exemplary embodiment of the present disclosure is shown;

[0021] Figure 7 A schematic flow chart showing another method for fusing a first depth map and a second depth map in an exemplary embodiment of the present disclosure;

[0022] Figure 8 A schematic flow chart showing another video processing method in an exemplary embodiment of the present disclosure;

[0023] Fig. 9 A schematic diagram showing the composition of a video processing device in an exemplary embodiment of the present disclosure;

[0024] Fig.10 A schematic diagram showing an electronic device to which the exemplary embodiments of the present disclosure can be applied. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0027] The depth prediction of an image refers to calculating the distance from a point in space to the camera plane. The depth prediction task of a video sequence mainly includes two parts: predicting the depth value of a single frame image and smoothing the inter-frame prediction results.

[0028] The input images of the previous and next frames often have different depth prediction results for similar scenes at different times due to external factors such as lighting changes and camera motion, resulting in flickering depth values ​​and low stability of depth prediction in the video sequence.

[0029] To address the above issues, the Alpha Blending algorithm is applied to the fusion of depth maps between frames. The Alpha channel was originally a method for processing transparent colors, and later it generally refers to superimposing two images at a certain coefficient ratio to obtain the final fusion result.

[0030] On this basis, a related technology targets pixel point p of frame f, takes the pixel point corresponding to pixel point p in each frame in the frame interval [fa,f+a], and uses a median filter to update the depth prediction value of pixel point p in frame f. At the same time, for motion scenes, it is proposed to use the optical flow method to calculate the motion vector of each pixel point to obtain a more accurate prediction result. It takes 1 second to process an image with a resolution of 720x576, and the prediction efficiency is relatively low, which is not conducive to deployment on mobile devices.

[0031] In another related technology, depth map prediction of video sequences can be performed based on deep learning, and timing information can be introduced based on RNN (recurrent neural network), LSTM (long short-term memory network), etc., so as to improve the stability of the prediction result sequence while improving the prediction accuracy.

[0032] For the above-mentioned related technologies, they often fuse by finding corresponding pixels in the previous and next frames. However, such methods ignore the consistency of pixel neighborhood information while considering the relationship between frames. Although it solves the problem of depth value flickering between frames to a certain extent, it also brings about the problem of inconsistent pixel neighborhood representation.

[0033] There are a lot of object motions in the video sequence. The pixel coordinate point (x, y) in frame f is located inside an object (foreground), but in frame f+1, the coordinate point (x, y) may become a background point. In order to find more accurate corresponding points in the previous and next frames for fusion, that is, to better align the previous and next frames, the related technology uses the optical flow method to match the pixels of the previous and next frames. The calculation of optical flow requires a lot of computing power, resulting in low efficiency in predicting the depth value of the video sequence.

[0034] Moreover, for deep learning solutions, the complexity of their network structure and data dependence make it difficult to directly deploy them in video sequence depth value prediction tasks that require high processing efficiency.

[0035] In view of the above problems, an exemplary embodiment of the present disclosure provides a video processing method.

[0036] Next, combine Figure 1 The system architecture of the operating environment of this exemplary embodiment is exemplarily described.

[0037] Figure 11 shows a schematic diagram of a system architecture, which may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, an intelligent wearable electronic device (such as a smart watch), and the server 120 generally refers to a background system that provides related services of the video processing method in this exemplary embodiment. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to perform data exchange.

[0038] In an exemplary implementation, the video processing method of the present disclosure may be performed by the terminal 110. For example, the terminal 110 may determine a motion matrix between a current frame of a video sequence and a reference frame of the current frame, and then determine pixel point matching pairs between the current frame and the reference frame based on the motion matrix; based on the segmented region relationship of the pixel point matching pairs, the depth maps of the current frame and the reference frame are fused to obtain a target depth map of the current frame.

[0039] In another exemplary implementation, the video processing method of the present disclosure may also be performed by the server 120. For example, the server 120 may determine a motion matrix between a current frame of a video sequence and a reference frame of the current frame, and then determine pixel point matching pairs between the current frame and the reference frame based on the motion matrix; based on the segmented region relationship of the pixel point matching pairs, the depth maps of the current frame and the reference frame are fused to obtain a target depth map of the current frame.

[0040] In another exemplary implementation, the video processing method of the present disclosure may also be jointly performed by the server 120 and the terminal 110. For example, the terminal 110 uploads a local video sequence to the server 120. After the server 120 obtains the video sequence, it may determine a motion matrix between a current frame of the video sequence and a reference frame of the current frame, and then determine pixel point matching pairs between the current frame and the reference frame based on the motion matrix; based on the segmented region relationship of the pixel point matching pairs, the depth maps of the current frame and the reference frame are fused to obtain a target depth map of the current frame.

[0041] As can be seen from the above, the execution subject of the video processing method in this exemplary embodiment may be the terminal 110 and / or the server 120, and the present disclosure does not limit this.

[0042] Combine the following Figure 2 A video processing method in this exemplary embodiment is described. Figure 2 An exemplary process of the video processing method is shown, which may include:

[0043] Step S210, determining a motion matrix between a current frame of a video sequence and a reference frame of the current frame;

[0044] Step S220, determining, according to the motion matrix, a second pixel point that matches the first pixel point of the current frame in the reference frame to obtain a pixel point matching pair;

[0045] Step S230, determining a region relationship between pixels in the pixel matching pair based on the first segmented image of the current frame and the second segmented image of the reference frame;

[0046] Step S240: fusing the first depth map of the current frame and the second depth map of the reference frame according to the region relationship to obtain a target depth map of the current frame.

[0047] Based on the above method, on the one hand, the matching pixels between the current frame and the reference frame are determined through the motion matrix, which can reduce the calculation amount of pixel matching and improve the efficiency of video depth value prediction; on the other hand, the segmentation area relationship of the pixels in the pixel matching pair is determined according to the segmented image, so that the depth values ​​are fused according to the segmentation area relationship, reducing the error fusion problem caused by motion blur in the edge area and improving the accuracy of video depth value prediction.

[0048] Below Figure 2 The steps shown are explained in detail.

[0049] In step S210, a motion matrix between a current frame of a video sequence and a reference frame of the current frame is determined.

[0050] In an exemplary implementation, the reference frame of the current frame may include any frame before the current frame, such as a previous frame adjacent to the current frame or a frame that is one frame apart from the current frame and before the current frame.

[0051] For example, Figure 3 A schematic flow chart of a method for determining a motion matrix in an exemplary embodiment of the present disclosure is shown. Figure 3 , the method may include steps S310 to S330.

[0052] in:

[0053] In step S310, the distance between the feature points of the current frame and the feature points of the reference frame is calculated, and matching pairs of feature points of the current frame and the reference frame are determined according to the distance.

[0054] In an exemplary implementation, SIFT (Scale-invariant feature transform) features may be used as feature points.

[0055] For example, when an object moves, in order to take into account the matching relationship and computational complexity of the corresponding points in the previous and next frames, the SIFT features of the previous and next frame images can be calculated. The SIFT features are invariant to rotation, scaling, and brightness changes, and are also robust to changes in viewing angle and noise. Each feature point has a 128-dimensional feature expression, which includes the size, direction, and scale information of the feature point.

[0056] After obtaining the SIFT feature point representations of the previous and next frame images respectively, for each SIFT feature of the current frame image, the Euclidean distance between the SIFT feature and each SIFT feature in the reference frame can be calculated. Based on the SIFT feature points in the reference frame with the smallest Euclidean distance, the corresponding SIFT feature points in the reference frame of the SIFT feature points in the current frame can be determined, thereby obtaining a feature point matching pair of the current frame and the reference frame.

[0057] In step S320, singular value decomposition is performed on the feature point matching pairs to obtain a homography matrix between the current frame and the reference frame.

[0058] After obtaining the feature point matching pairs, the SVD (Singular value decomposition) algorithm can be used for each feature point matching pair to obtain the homography matrix between the current frame and the reference frame.

[0059] The homography matrix is ​​a projection matrix from one plane to another plane, which can be used to describe the position mapping relationship between the world coordinate system and the pixel coordinate system, that is, it contains both the intrinsic and extrinsic parameters of the camera. Let the homography matrix be denoted as H, t is the current frame, t-1 is the reference frame of the current frame, and z is the scale factor. The coordinate relationship between the current frame and the reference frame can be expressed by the following formula (1):

[0060]

[0061] In formula (1), (x(t), y(t)) is the pixel coordinate point in the current frame, and (x(t-1), y(t-1)) is the pixel coordinate point in the reference frame. The 1 in formula (1) is to convert the entire pixel coordinate into a homogeneous coordinate. Homogeneous coordinates can make some image operations (such as scaling, rotation, and translation) can be calculated in the form of matrix multiplication.

[0062] In step S330, a motion matrix is ​​determined according to the homography matrix.

[0063] Exemplarily, a specific implementation of step S330 may include: the homography matrix may be directly determined as the motion matrix.

[0064] Exemplarily, another specific implementation method of step S330 may include: processing the homography matrix based on a random sampling consistency algorithm to determine the motion matrix.

[0065] For example, the RANSAC (Random Sample Consensus) algorithm can be used to process the homography matrix and filter outliers to determine the motion matrix so that the motion matrix can better fit the projection model between the current frame and the reference frame, thereby improving the accuracy of pixel matching between the current frame and the reference frame.

[0066] Through the above steps S310 to S330, the homography matrix of the previous and next frames can be estimated by the feature matching method, so that the corresponding points of the current frame and the reference frame can be matched according to the homography matrix in the subsequent steps. In this way, the computational complexity of the corresponding point matching can be reduced and the efficiency can be improved.

[0067] Exemplarily, another specific implementation of step S210 may include: determining a motion matrix corresponding to each segmented region based on the first segmented image of the current frame and the second segmented image of the reference frame.

[0068] For example, for a video frame, the scene can be divided into a foreground object and a background, the background is relatively static, the motion mostly occurs in the foreground object, and the pixels of the foreground have motion consistency. On this basis, the calculated SIFT feature points can be divided into two parts, the foreground and the background, according to the first subject segmentation image of the current frame and the second subject segmentation image of the reference frame, and the homography matrix of the foreground part is calculated according to the feature points of the foreground part, thereby obtaining the motion matrix of the foreground part, and the homography matrix of the background part is calculated according to the feature points of the background part, thereby obtaining the motion matrix of the background part.

[0069] Of course, the current frame and the reference frame can also be divided into multiple regions according to the same segmentation method, and the homography matrix can be calculated for each corresponding region separately, so as to obtain the motion matrix of each segmented region separately. In this way, the redundancy of the computational complexity of the pixel-level optical flow is avoided, and at the same time, the algorithm is given more flexibility. Different homography matrices can be calculated according to the number and size of the segmented regions, so as to better align the pixels of the current frame and the reference frame.

[0070] By using the regional motion matrix, the pixel alignment effect between frames can be improved and the accuracy of pixel alignment matching can be improved.

[0071] Exemplarily, another specific implementation of step S210 may include: determining a motion matrix between a current frame of the video sequence and a reference frame of the current frame based on a dense optical flow algorithm.

[0072] For example, for some video sequence depth prediction scenarios that do not require high prediction efficiency but high prediction accuracy, the motion matrix between the reference frame and the current frame can also be obtained based on the dense optical flow algorithm. For example, a deep learning network (such as the PWCNet optical flow model) or a differential-based LK (Lucas–Kanade) algorithm can be used to solve the motion matrix between the reference frame and the current frame.

[0073] Through the homography matrix of each region, a separate motion matrix for each segmented region can be obtained. The dense optical flow method can be used to obtain the motion vector of each pixel in the entire image. Compared with the homography matrix, the motion vector has a more accurate correspondence, which makes the pixel matching of the reference frame and the current frame more accurate, thereby improving the accuracy of video depth value prediction.

[0074] In step S220, a second pixel point that matches the first pixel point of the current frame in the reference frame is determined according to the motion matrix to obtain a pixel point matching pair.

[0075] After obtaining the motion matrix between the current frame and the reference frame, when estimating the depth value of each pixel point in the current frame, a relationship mapping can be performed based on the motion matrix, so that each pixel point coordinate in the current frame can find its corresponding pixel point coordinate in the reference frame, thereby determining the pixel point matching pair between the current frame and the reference frame based on the pixel point coordinates of the current frame and its corresponding pixel point coordinates in the reference frame.

[0076] In step S230, based on the first segmented image of the current frame and the second segmented image of the reference frame, the regional relationship between the pixels in the pixel matching pair is determined.

[0077] For example, Figure 4 A flowchart of a method for determining a regional relationship in an exemplary embodiment of the present disclosure is shown. Figure 4 , the method may include steps S410 to S450.

[0078] In step S410, determining a first segmented region to which a first pixel point in the pixel matching pair belongs in a first segmented image;

[0079] In step S420, determining a second segmented area to which a second pixel point in the pixel matching pair belongs in the second segmented image;

[0080] In step S430, it is determined whether the first segmented area and the second segmented area are the same, if they are the same, the process goes to step S440, otherwise, the process goes to step S450;

[0081] In step S440, it is determined that the region relationship between the first pixel point and the second pixel point in the pixel matching pair belongs to the same segmented region;

[0082] In step S450, it is determined that the region relationship between the first pixel point and the second pixel point in the pixel matching pair belongs to different segmented regions.

[0083] Among them, judging whether the first segmentation area and the second segmentation area are the same can be understood as judging whether the first segmentation area and the second segmentation area belong to the same segmentation scene, such as whether they belong to pixels in the same object, whether they are both background pixels, etc.

[0084] For example, the current frame and the reference frame may be segmented according to the same segmentation method to obtain a first segmentation map and a second segmentation map. For example, the current frame and the reference frame are segmented into foreground and background regions to obtain a first segmentation image and a second segmentation image respectively. When the first pixel belongs to the foreground region in the first segmentation image and the second pixel also belongs to the foreground region in the second segmentation image, or when the first pixel belongs to the background region in the first segmentation image and the second pixel also belongs to the background region in the second segmentation image, their region relationship is determined to belong to the same segmentation region; when the first pixel belongs to the foreground region in the first segmentation image and the second pixel belongs to the background region in the second segmentation image, or when the first pixel belongs to the background region in the first segmentation image but the second pixel also belongs to the foreground region in the second segmentation image, their region relationship is determined to belong to different segmentation regions.

[0085] Continue to refer Figure 2 In step S240, the first depth map of the current frame and the second depth map of the reference frame are fused according to the region relationship to obtain a target depth map of the current frame.

[0086] Through the above steps S410 to S450, the regional relationship of each pixel matching pair of the reference frame and the current frame can be determined. According to the regional relationship of each pixel matching pair, the depth value in the first depth map of the current frame and the depth value in the second depth map of the reference frame can be fused to obtain a target depth map of the current frame.

[0087] For example, Figure 5 A schematic diagram of a method for fusing a first depth map and a second depth map in an exemplary embodiment of the present disclosure is shown. Figure 5 , the method may include steps S510 to S530.

[0088] In step S510, for each pixel matching pair, a first neighborhood pixel adjacent to the first pixel is determined in the current frame according to a preset window, and a second neighborhood pixel adjacent to the second pixel is determined in the reference frame.

[0089] When fusing the current frame with the reference frame, if the corresponding pixels are simply weighted averaged, the consistency of the representation of the pixel neighborhood will be ignored, resulting in flickering in the spatial domain of the current frame, especially in the edge area of ​​the object, which will produce more spatial non-smoothness. Based on this, the pixels can be filtered in both the spatial domain and the temporal domain at the same time.

[0090] For example, for each pixel matching pair, a neighborhood area of ​​the first pixel can be determined in the current frame with the first pixel as the center according to a preset spatial filtering window, and the pixels in the neighborhood area of ​​the first pixel are the first neighborhood pixels. At the same time, in the reference frame, a neighborhood area of ​​the second pixel can be determined with the second pixel matching the first pixel as the center according to a preset spatial filtering window, and the pixels in the neighborhood area of ​​the second pixel are the second neighborhood pixels.

[0091] The preset window size is 5, the sampling interval of each point in the window is 2, and the first pixel is Figure 6 Taking the pixel 61 in FIG. 1 as an example, the neighborhood area of ​​the first pixel may include the following: Figure 6 The 25 pixels shown.

[0092] In step S520, a first temporal weight of the first neighborhood pixel and a second temporal weight of the second neighborhood pixel are determined based on an area relationship between the first neighborhood pixel and a second neighborhood pixel matched by the first neighborhood pixel in the reference frame.

[0093] Exemplarily, a specific implementation of step S520 may include: when the regional relationship is that they belong to the same segmented area, determining that the first time domain weight and the second time domain weight are both positive numbers; when the regional relationship is that they belong to different segmented areas, determining that the first time domain weight is a positive number and determining that the second time domain weight is 0.

[0094] Among them, when the segmentation area to which the first neighborhood pixel point belongs in the first segmentation image and the segmentation area to which the second neighborhood pixel point belongs in the second segmentation image are the same, the area relationship is determined to belong to the same segmentation area; otherwise, the area relationship is determined to belong to different segmentation areas.

[0095] For example, the regional relationship between the first neighborhood pixel point and the second neighborhood pixel point that matches the first neighborhood pixel point in the reference frame can be determined according to the above-mentioned steps S410 to S450, and the first temporal weight of the first neighborhood pixel point and the second temporal weight of the second neighborhood pixel point can be determined based on the regional relationship.

[0096] Exemplarily, the first time domain weight can be determined by the following formula (2), and correspondingly, the second time domain weight can be determined by the following formula (3).

[0097]

[0098] In formula (2), alpha∈(0,1). In other words, when the area relationship between the first neighborhood pixel point and the corresponding second neighborhood pixel point is that they belong to the same segmented area, the first time domain weight can be any value between 0 and 1, which can be customized according to user needs. The corresponding second time domain weight can be determined by formula (3), that is, the sum of the first time domain weight and the second time domain weight is 1. When the area relationship between the first neighborhood pixel point and the corresponding second neighborhood pixel point is that they belong to different segmented areas, the first time domain weight is determined to be 1 by formula (2), and the second domain weight can be determined to be 0 by formula (3).

[0099] w temporal ( x t-1 ,y t-1 ) =1-w temporal ( x t ,y t ) (3)

[0100] Of course, the sum of the first time domain weight and the second time domain weight may not be 1, and may be customized in advance, as long as the requirements corresponding to different regional relationships are met. This exemplary embodiment does not specifically limit this.

[0101] In step S530, the first depth value of the first neighborhood pixel and the second depth value of the second neighborhood pixel are fused according to the first temporal weight and the second temporal weight to obtain a target depth map of the current frame.

[0102] The first depth value is determined according to the first depth map, and the second depth value is determined according to the second depth map.

[0103] In an exemplary embodiment, the first depth map of the current frame may include a single-frame depth map of the current frame, and the second depth map of the reference frame may include a depth map obtained by processing the single-frame depth map of the reference frame using the video processing method of the present disclosure.

[0104] Taking the reference frame as the previous frame of the current frame as an example, the current frame is the t-th frame, then the reference frame is the t-1 frame, the first depth map of the current frame is the single-frame depth map of the t-th frame, and the second depth map of the reference frame can be understood as the target depth map of the reference frame, that is, it includes the depth map obtained after the single-frame depth map of the t-1th frame and the target depth map of the t-2th frame are fused using the video processing method in the present disclosure when the reference frame is the current frame.

[0105] Of course, the second depth map of the reference frame may also be a single-frame depth map of the reference frame, which is not particularly limited in this exemplary embodiment.

[0106] Exemplarily, a specific implementation of step S530 may include: for each first neighborhood pixel point, determining a first spatial domain weight of the first neighborhood pixel point according to the pixel position distance between the first pixel point and the first neighborhood pixel point, determining a first value range weight of the first neighborhood pixel point according to the pixel value difference between the first pixel point and the first neighborhood pixel in each channel of the RGB image of the current frame and the pixel value difference in the first segmented image of the current frame; determining a first spatial domain weight of the first neighborhood pixel point according to the first spatial domain weight and the first value range weight; for each second neighborhood pixel point, determining a first spatial domain weight of the first neighborhood pixel point according to the pixel value difference between the second pixel point and the second neighborhood pixel point Position distance, determine the second spatial domain weight of the second neighborhood pixel point, and determine the second value domain weight of the second neighborhood pixel point according to the pixel value difference between the second pixel point and the second neighborhood pixel point in each channel of the RGB image of the reference frame and the pixel value difference in the second segmented image of the reference frame; determine the second spatial domain weight of the second neighborhood pixel point according to the second spatial domain weight and the second value domain weight; and fuse the first depth value of the first neighborhood pixel point and the second depth value of the second neighborhood pixel point according to the first spatial domain weight, the first temporal domain weight, the second spatial domain weight, and the second temporal domain weight to obtain the target depth map of the current frame.

[0107] As mentioned above, the present disclosure can perform spatial domain filtering and temporal domain filtering on pixel points at the same time. For spatial domain filtering, bilateral filtering can be used. Bilateral filtering can smooth and denoise the image while retaining high-frequency information (such as edge details). It also considers the neighborhood spatial coordinates and pixel value similarity of the image, which are respectively denoted as the spatial domain kernel and the value domain kernel. The spatial domain weight (denoted as w d ) measures the coordinate distance between two pixels. The closer the distance, the higher the weight. The range weight (denoted as w r ) measures the similarity between two pixels. The more similar the pixel values ​​of the two points are, the higher the weight is.

[0108] Among them, the spatial weight w d and range weight w r They can be determined by the following formulas (4) and (5) respectively.

[0109]

[0110] wr(x,y,i,j)=l R *(R(x,y)-R(i,j))+l G *(G(x,y)-G(i,j))+l B *(B(x,y)-B(i,j))+l S *(S(x,y)-S(i,j)) (5)

[0111] In formulas (4) and (5), (x, y) represents the pixel point that needs to be spatially filtered (i.e., the center coordinate pixel point of the preset window), such as the first pixel point or the second pixel point mentioned above, which can be Figure 6 The pixel point 61 in (i, j) is a neighborhood pixel point of the pixel point that needs to be spatially filtered (i.e., other coordinate pixel points in the preset window except the central coordinate point), such as the first neighborhood pixel point or the second neighborhood pixel point.

[0112] In formula (4), σ is the standard deviation of the Gaussian kernel function in the spatial domain, which is used to control the weight of the pixel position, w d It is determined by the Euclidean distance calculation of the pixel position coordinates. In formula (5), R, G, and B represent the R channel, G channel, and B channel of the RGB image, respectively, S represents the segmented image, and l R ,l G ,l B ,l S are the weight coefficients corresponding to the R channel, G channel, B channel and the segmented image. r It is determined by the difference in pixel values, so R(x,y) in formula (5) represents the pixel value of the first pixel on the R channel, and R(i,j) represents the pixel value of the neighborhood pixel of the first pixel on the R channel. The other channels and segmented images are similar, and they all represent corresponding pixel values.

[0113] In the flat area of ​​the image, the difference between the pixel values ​​of adjacent pixels is small, and the range weight w r It plays a leading role and smoothes the area. In the edge area, the difference between the pixel values ​​of adjacent pixels is large, and the value range weight w r Close to 0, so as to preserve the edge details of the original image.

[0114] by Figure 6For example, the black curved edge represents a boundary in the image, that is, the boundary of the segmented area, 61 is the center point of the filter window, and the value range of the window pixel point on the left side is large compared with the right side. Therefore, for the filtering effect of 61, the point on the right side of the filter window dominates.

[0115] For example, the first pixel and the second pixel can be taken as the central pixel respectively, and the spatial domain weight and the range weight of each neighborhood pixel can be calculated according to the above formula (4) and formula (5), and then the product of the spatial domain weight and the temporal domain weight can be used as the spatial domain weight of the neighborhood pixel. For example, the product of the first spatial domain weight and the first range weight is the first spatial domain weight, and the product of the second spatial domain weight and the second range weight is the second spatial domain weight.

[0116] Exemplarily, after determining the first spatial domain weight and the first temporal domain weight of each first neighborhood pixel point, as well as the second spatial domain weight and the second temporal domain weight of the second neighborhood pixel point corresponding to each first neighborhood pixel point in the reference frame, the first depth value of the first neighborhood pixel point and the second depth value of the second neighborhood pixel point can be fused by the following formula (6) to obtain the target depth value of the first pixel point in the current frame, and then the target depth map of the current frame is obtained according to the target depth value of the first pixel point in each pixel matching pair.

[0117]

[0118] In formula (6), w d 、w r 、w temporal They represent the spatial weight, value weight, and time weight respectively; K is the radius of the preset window. For example, if the preset window size is 5, the window radius is 3. (x t ,y t ) represents the pixel in the current frame (i.e., the first pixel), (x t-1 ,y t-1 ) represents (x t ,y t ) is calculated by the motion matrix to obtain the matching point (i.e., the second pixel matching the first pixel) in the reference frame (such as the frame before the current frame). d (x t ,y t ,i,j) represents the coordinate point (x t ,y t ) in the surrounding window (x t +i,y t +j), D t (x t ,y t ,i,j) represents (x t ,yt ) in the surrounding window (x t +i,y t +j) coordinate point depth value. t (x t-1 ,y t-1 ,i,j) represents (x t-1 ,y t-1 ) in the surrounding window (x t-1 +i,y t-1 +j) coordinate point’s depth value.

[0119] In other words, for each first pixel point, the first depth value of each first neighborhood pixel point can be weighted and summed according to the first spatial weight, the first value weight and the first temporal weight of each first neighborhood pixel point to obtain the first spatiotemporal filtering result, and the second depth value of each second neighborhood pixel point can be weighted and summed according to the second spatial weight, the second value weight and the second temporal weight of each second neighborhood pixel point to obtain the second spatiotemporal filtering result. After summing the first spatiotemporal filtering result and the second spatiotemporal filtering result, normalizing them according to the weight (i.e., the denominator in formula (6)) can obtain the target pixel value D of the first pixel point. t (x t ,y t ).

[0120] Taking the final target depth value determined by formula (6) and the segmented area including the foreground area and the background area as an example, in the present disclosure, the first temporal weight may include three situations:

[0121] (1) The current frame coordinate point (x t ,y t ) belongs to the foreground, and its corresponding point in the previous frame (x t-1 ,y t-1 ) also belongs to the foreground, then w temporal (x t ,y t ) is alpha, alpha∈(0,1); (2) the coordinate point of the current frame (x t ,y t ) belongs to the background, and its corresponding point in the previous frame (x t-1 ,y t-1 ) also belongs to the background, then w temporal (x t ,y t ) is alpha, alpha∈(0,1); (3) The coordinate point of the current frame (x t ,y t ), and its corresponding point in the previous frame (x t-1 ,y t-1 ) belong to different segmentation areas, then wtemporal (x t ,y t ) is 1.

[0122] In this way, for the points in the edge area, they belong to different segmented areas in the reference frame and the current frame. In this case, by setting the time domain weight of the corresponding points in the reference frame to 0, it can be expressed as not performing time domain filtering on the points in the edge area, but only performing spatial domain filtering on them. Therefore, for moving objects, the problem of incorrect fusion caused by motion blur in the edge area can be further reduced.

[0123] For example, Figure 7 FIG. 2 is a flow chart showing another method for fusing a first depth map and a second depth map in an exemplary embodiment of the present disclosure. Figure 7 The method may include steps S710 to S720. Wherein:

[0124] In step S710, for each pixel matching pair, a third temporal weight of the first pixel and a fourth temporal weight of the second pixel are determined according to the region relationship.

[0125] Exemplarily, a specific implementation of step S710 may include: when the regional relationship is that they belong to the same segmented area, determining that the third time domain weight and the fourth time domain weight are both positive numbers; when the regional relationship is that they belong to different segmented areas, determining that the third time domain weight is a positive number and determining that the fourth time domain weight is 0; wherein, when the segmented area to which the first pixel point belongs in the first segmented image and the segmented area to which the second pixel point belongs in the second segmented image are the same, determining that the regional relationship is that they belong to the same segmented area; otherwise, determining that the regional relationship is that they belong to different segmented areas.

[0126] In other words, the regional relationship between the first pixel and the second pixel in each pixel matching pair can be determined according to the above-mentioned steps S410 to S450, and the third time domain weight of the first pixel and the fourth time domain weight of the second pixel can be determined according to the regional relationship.

[0127] Different from the first time domain weight and the second time domain weight, the first time domain weight is the weight of the first neighborhood pixel of the first pixel, the second time domain weight is the weight of the second neighborhood pixel corresponding to the first neighborhood pixel, and the third time domain weight is the weight of the first pixel itself, and the fourth time domain weight is the weight of the second pixel corresponding to the first pixel itself.

[0128] Similarly, the third time domain weight and the fourth time domain weight can be determined according to the above formula (2) and formula (3), respectively, which will not be described in detail here.

[0129] Continue to refer Figure 7 In step S720, based on the third time domain weight and the fourth time domain weight, the third depth value of the first pixel and the fourth depth value of the second pixel are fused to obtain the target depth map of the current frame.

[0130] Among them, the third depth value is determined according to the first depth map, and the fourth depth value is determined according to the second depth map. That is, the third depth value is the depth value of the first pixel in the first depth map, and the fourth depth value is the depth value of the second pixel in the second depth map. For detailed descriptions of the first depth map and the second depth map, please refer to the relevant content mentioned above, which will not be repeated here.

[0131] Exemplarily, a specific implementation of step S720 may include: spatially filtering the third depth value and the fourth depth value, respectively, to obtain a first spatial filtering result of the third depth value and a second spatial filtering result of the fourth depth value; fusing the first spatial filtering result and the second spatial filtering result, respectively, according to the third time domain weight and the fourth time domain weight, to obtain a target depth value of the first pixel; and obtaining a target depth map of the current frame based on the target depth value of the first pixel in each pixel matching pair.

[0132] Among them, the third depth value and the fourth depth value are spatially filtered respectively to obtain a first spatial filtering result of the third depth value and a second spatial filtering result of the fourth depth value, including: according to a preset window, determining a first neighborhood pixel point adjacent to the first pixel point in the current frame, and determining a second neighborhood pixel point adjacent to the second pixel point in the reference frame; for each first neighborhood pixel point, determining a first spatial domain weight of the first neighborhood pixel point according to a pixel position distance between the first pixel point and the first neighborhood pixel point, and determining a first value domain weight of the first neighborhood pixel point according to a pixel value difference between the first pixel point and the first neighborhood pixel in each channel of the RGB image of the current frame and a pixel value difference in the first segmented image of the current frame; according to each The method comprises the following steps of: spatially filtering the third depth value of the first pixel point according to the first spatial weight and the first range weight of each of the first neighborhood pixel points to obtain a first spatial filtering result of the third depth value; for each second neighborhood pixel point, determining the second spatial weight of the second neighborhood pixel point according to the pixel position distance between the second pixel point and the second neighborhood pixel point, and determining the second range weight of the second neighborhood pixel point according to the pixel value difference between the second pixel point and the second neighborhood pixel point in each channel of the RGB image of the reference frame and the pixel value difference in the second segmented image of the reference frame; spatially filtering the fourth depth value of the second pixel point according to the second spatial weight and the second range weight of each of the second neighborhood pixel points to obtain a second spatial filtering result of the fourth depth value.

[0133] For example, for the first pixel, the first spatial domain weight and the first value range weight of each first neighborhood pixel of the first pixel can be calculated according to the above formulas (4) and (5), respectively, and then according to the following formula (7), based on the first spatial domain weight and the first value range weight of each first neighborhood pixel, the depth value of each first neighborhood pixel is weighted and summed to obtain the first spatial filtering result of the third depth value of the first pixel.

[0134]

[0135] In formula (7), Din t (i, j) is the depth value of the neighboring pixel of the center point, w d (x,y,i,j) is (x t ,y t ) in the surrounding window (x t +i,y t +j) Spatial weight of coordinate point, w r (x,y,i,j) is (x t ,y t ) in the surrounding window (xt +i,y t +j) The value range weight of the coordinate point.

[0136] Similarly, for the second pixel point that matches the first pixel point, the second spatial domain weight and the second value domain weight of each second neighborhood pixel point of the second pixel point can be calculated according to the above formulas (4) and (5), respectively, and then according to the following formula (7), based on the second spatial domain weight and the second value domain weight of each second neighborhood pixel point, the depth value of each second neighborhood pixel point is weighted and summed to obtain the second spatial filtering result of the fourth depth value of the second pixel point.

[0137] Exemplarily, the target depth value of the first pixel point can be obtained by performing the first spatial filtering result and the second spatial filtering result according to the third time domain weight and the fourth time domain weight using the following formula (8).

[0138] Dout t (x t ,y t )=w temporal (x t ,y t )×Din t (x t ,y t )+(1-w temporal (x t ,y t ))×Din t-1 (x t-1 ,y t-1 ) (8)

[0139] In formula (8), w temporal (x t ,y t ) is the third time domain weight, Din t (x t ,y t ) is the first spatial filtering result, (1-w temporal (x t ,y t )) is the fourth time domain weight, Din t-1 (x t-1 ,y t-1 ) is the second spatial filtering result.

[0140] Of course, the first spatial filtering result and the second spatial filtering result may also be obtained by other spatial filtering methods, such as median filtering.

[0141] In other words, another video processing method disclosed in the present invention may include: determining a motion matrix between a current frame of a video sequence and a reference frame of the current frame; determining, based on the motion matrix, a second pixel point that matches a first pixel point of the current frame in the reference frame to obtain a pixel matching pair; spatially filtering a depth value of a first pixel point in the pixel matching pair and a depth value of a second pixel point in the pixel matching pair, respectively, to obtain a first spatial filtering depth value and a second spatial filtering depth value correspondingly; and fusing the first spatial filtering depth value and the second spatial filtering depth value to obtain a target depth map of the current frame.

[0142] Among them, fusing the first spatial filtering depth value and the second spatial filtering depth value to obtain the target depth map of the current frame can include: determining the regional relationship between the pixels in the pixel matching pair based on the first segmented image of the current frame and the second segmented image of the reference frame; determining the fusion weight of the first spatial filtering depth value and the second spatial filtering depth value according to the regional relationship, and fusing the first spatial filtering depth value and the second spatial filtering depth value according to the fusion weight to obtain the target depth map of the current frame.

[0143] In another exemplary embodiment, the regional relationship may not be considered when determining the fusion weight of the first spatial filtering depth value and the second spatial filtering depth value. For example, after obtaining the motion matrix between the current frame and the reference frame, when estimating the depth value of the current first pixel of the current frame, the current second pixel corresponding to the current first pixel of the current frame in the reference frame is directly obtained according to the motion matrix to determine a matching pair of the current first pixel corresponding to the current pixel; for the current pixel matching pair, the depth value of the current first pixel can be spatially filtered in the current frame, and the depth value of the current second pixel can be spatially filtered in the reference frame to obtain the first spatial filtering depth value of the current first pixel and the second spatial filtering depth value of the current second pixel, respectively; according to the preset weight of the current frame and the preset weight of the reference frame, the first spatial filtering depth value and the second spatial filtering depth value are fused to obtain the target depth value of the current first pixel, and then the target depth map of the current frame is obtained according to the target depth value of each current first pixel.

[0144] That is to say, in the present disclosure, while spatial filtering is performed on the current frame, each neighboring pixel point of the current pixel point of the current frame can be temporally filtered separately, that is, the temporal weight of each neighboring pixel point is determined according to the regional relationship of the pixel point matching pair determined for each neighboring pixel point, so as to fuse the reference frame and the current frame based on formula (6); it is also possible to perform temporal filtering on only the spatial filtering depth value obtained after spatial filtering of the current pixel point of the current frame, instead of performing temporal filtering on each neighboring pixel point used in the spatial filtering, so as to fuse the current frame and the reference frame based on formula (8), and this exemplary embodiment does not make any special limitation on this.

[0145] For example, Figure 8 FIG. 2 is a flow chart showing another video processing method in an exemplary embodiment of the present disclosure. Figure 8 , the method may include steps S810 to S850. Wherein: in step S810, the input data of the previous and next frames are obtained; in step S820, the motion matrix of the previous and next frames is estimated; in step S830, the spatial domain weights and temporal domain weights of the corresponding pixels of the t-th frame and the t-1-th frame are calculated; in step S840, the previous and next frames are fused according to the weights; in step S850, the target depth map of the t-th frame is output.

[0146] For example, the input data of the previous and next frames may include the RGB image, segmentation image and depth image of the t-1 frame, and the RGB image, segmentation image and depth image of the t frame. The motion matrix of the previous and next frames can be estimated based on the segmentation images of the t-1 frame and the t frame, such as by respectively estimating the homography matrix of each segmentation area through the feature matching method, and obtaining the motion matrix of the segmentation area according to the homography matrix; then, for the current pixel to be processed in the t frame, the pixel corresponding to the current pixel to be processed in the t-1 frame is determined through the motion matrix, and the spatial domain weight and range weight of each neighboring pixel of the current pixel to be processed can be respectively calculated through the above formula (4) and formula (5), and then the two are multiplied to obtain the spatial domain weight of each neighboring pixel of the current pixel to be processed. Similarly, the spatial domain weight and range weight of each neighboring pixel of the corresponding pixel of the current pixel to be processed in the t-1 frame can also be respectively calculated through the above formula (4) and formula (5), and then the two are multiplied to obtain the spatial domain weight of each neighboring pixel of the corresponding pixel. The temporal weight of each neighboring pixel of the current pixel to be processed in the t-th frame and the temporal weight of the neighboring pixel corresponding to the neighboring pixel in the t-1-th frame can be obtained respectively through the above formulas (2) and (3); then, according to the calculated spatial domain weights and value domain weights of the neighboring pixels of the previous and next frames, the corresponding pixels in the depth maps of the previous and next frames are fused based on the above formula (6) to obtain the target depth value of the current pixel to be processed in the t-th frame; finally, based on the target depth value of each pixel to be processed in the t-th frame, the target depth map of the t-th frame can be output, and the target depth map of the t-th frame is used as the final estimated depth map of the t-th frame in the video sequence.

[0147] In the present disclosure, through the motion matrix, each pixel point of the current frame can quickly find the corresponding pixel point on the reference frame image, which can improve the efficiency of pixel matching and further improve the overall efficiency of video sequence depth prediction.

[0148] At the same time, by performing spatial filtering on the pixels of the current frame and the corresponding pixels of the reference frame, and then performing temporal filtering, the surrounding neighborhood information of the pixels is taken into account while fusing the previous and next frames, ensuring the consistency of the representation of the pixel neighborhood and avoiding flickering in the spatial domain of the current frame.

[0149] Furthermore, by segmenting the regions and determining the temporal weights when fusing the previous and next frames based on the regional relationships, the problem of incorrect fusion in the edge areas due to motion blur can be reduced, further improving the accuracy and stability of the depth map prediction of the video sequence.

[0150] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0151] For further reference, Fig. 9 As shown, in the implementation of this example, a video processing device 900 is also provided, including a motion matrix determination module 910, a pixel matching pair determination module 920, a region relationship determination module 930, and a target depth map generation module 940. Among them: the motion matrix determination module 910 is configured to determine the motion matrix between the current frame of the video sequence and the reference frame of the current frame; the pixel matching pair determination module 920 is configured to determine the second pixel matching the first pixel of the current frame in the reference frame according to the motion matrix to obtain a pixel matching pair; the region relationship determination module 930 is configured to determine the region relationship between the pixels in the pixel matching pair based on the first segmented image of the current frame and the second segmented image of the reference frame; the target depth map generation module 940 is configured to fuse the first depth map of the current frame and the second depth map of the reference frame according to the region relationship to obtain a target depth map of the current frame.

[0152] In an exemplary embodiment, based on the aforementioned embodiment, the target depth map generation module 940 can be specifically configured as follows: for each pixel matching pair, according to a preset window, determine a first neighborhood pixel adjacent to the first pixel in the current frame, and determine a second neighborhood pixel adjacent to the second pixel in the reference frame; determine a first temporal weight of the first neighborhood pixel and a second temporal weight of the second neighborhood pixel according to an area relationship between the first neighborhood pixel and the second neighborhood pixel matched by the first neighborhood pixel in the reference frame; and fuse the first depth value of the first neighborhood pixel and the second depth value of the second neighborhood pixel according to the first temporal weight and the second temporal weight to obtain a target depth map of the current frame; wherein the first depth value is determined according to the first depth map, and the second depth value is determined according to the second depth map.

[0153] In an exemplary embodiment, based on the aforementioned embodiment, the first temporal weight of the first neighborhood pixel and the second temporal weight of the second neighborhood pixel are determined according to the regional relationship between the first neighborhood pixel and the second neighborhood pixel matched by the first neighborhood pixel in the reference frame, including: when the regional relationship is that they belong to the same segmented area, determining that the first temporal weight and the second temporal weight are both positive numbers; when the regional relationship is that they belong to different segmented areas, determining that the first temporal weight is a positive number and determining that the second temporal weight is 0; wherein, when the segmented area to which the first neighborhood pixel belongs in the first segmented image and the segmented area to which the second neighborhood pixel belongs in the second segmented image are the same, determining that the regional relationship is that they belong to the same segmented area; otherwise, determining that the regional relationship is that they belong to different segmented areas.

[0154] In an exemplary embodiment, based on the aforementioned embodiment, the first depth value of the first neighborhood pixel and the second depth value of the second neighborhood pixel are fused according to the first temporal weight and the second temporal weight to obtain the target depth map of the current frame, including: for each first neighborhood pixel, according to the pixel position distance between the first pixel and the first neighborhood pixel, determining the first spatial domain weight of the first neighborhood pixel, according to the pixel value difference between the first pixel and the first neighborhood pixel in each channel of the RGB image of the current frame and the pixel value difference in the first segmented image of the current frame, determining the first value domain weight of the first neighborhood pixel; determining the first spatial domain weight of the first neighborhood pixel according to the first spatial domain weight and the first value domain weight. Weight; for each second neighborhood pixel point, determine the second spatial domain weight of the second neighborhood pixel point according to the pixel position distance between the second pixel point and the second neighborhood pixel point, and determine the second value domain weight of the second neighborhood pixel point according to the pixel value difference between the second pixel point and the second neighborhood pixel point in each channel of the RGB image of the reference frame and the pixel value difference in the second segmented image of the reference frame; determine the second spatial domain weight of the second neighborhood pixel point according to the second spatial domain weight and the second value domain weight; fuse the first depth value of the first neighborhood pixel point and the second depth value of the second neighborhood pixel point according to the first spatial domain weight, the first temporal domain weight, the second spatial domain weight, and the second temporal domain weight to obtain the target depth map of the current frame.

[0155] In an exemplary embodiment, based on the foregoing embodiment, the target depth map generation module 940 can also be specifically configured as: for each pixel matching pair, according to the regional relationship, determine the third time domain weight of the first pixel and the fourth time domain weight of the second pixel; based on the third time domain weight and the fourth time domain weight, fuse the third depth value of the first pixel and the fourth depth value of the second pixel to obtain the target depth map of the current frame; wherein the third depth value is determined according to the first depth map, and the fourth depth value is determined according to the second depth map.

[0156] In an exemplary embodiment, based on the aforementioned embodiment, determining the third time domain weight of the first pixel and the fourth time domain weight of the second pixel according to the regional relationship includes: when the regional relationship is that they belong to the same segmented area, determining that the third time domain weight and the fourth time domain weight are both positive numbers; when the regional relationship is that they belong to different segmented areas, determining that the third time domain weight is a positive number and determining that the fourth time domain weight is 0; wherein, when the segmented area to which the first pixel belongs in the first segmented image and the segmented area to which the second pixel belongs in the second segmented image are the same, determining that the regional relationship is that they belong to the same segmented area; otherwise, determining that the regional relationship is that they belong to different segmented areas.

[0157] In an exemplary embodiment, based on the aforementioned embodiment, the third depth value of the first pixel and the fourth depth value of the second pixel are fused based on the third time domain weight and the fourth time domain weight to obtain the target depth map of the current frame, including: spatially filtering the third depth value and the fourth depth value respectively to obtain a first spatial filtering result of the third depth value and a second spatial filtering result of the fourth depth value; fusing the first spatial filtering result and the second spatial filtering result respectively according to the third time domain weight and the fourth time domain weight to obtain the target depth value of the first pixel; and obtaining the target depth map of the current frame based on the target depth value of the first pixel in each pixel matching pair.

[0158] In an exemplary embodiment, based on the foregoing embodiment, the third depth value and the fourth depth value are spatially filtered respectively to obtain a first spatial filtering result of the third depth value and a second spatial filtering result of the fourth depth value, including: according to a preset window, determining a first neighborhood pixel adjacent to the first pixel in the current frame, and determining a second neighborhood pixel adjacent to the second pixel in the reference frame; for each first neighborhood pixel, determining a first spatial weight of the first neighborhood pixel according to a pixel position distance between the first pixel and the first neighborhood pixel, and determining a first spatial weight of the first neighborhood pixel according to a pixel value difference between the first pixel and the first neighborhood pixel in each channel of the RGB image of the current frame and a pixel value difference in the first segmented image of the current frame. first range weight; spatially filter the third depth value of the first pixel according to the first spatial weight and the first range weight of each of the first neighborhood pixels to obtain a first spatial filtering result of the third depth value; for each second neighborhood pixel, determine the second spatial weight of the second neighborhood pixel according to the pixel position distance between the second pixel and the second neighborhood pixel, and determine the second range weight of the second neighborhood pixel according to the pixel value difference between the second pixel and the second neighborhood pixel in each channel of the RGB image of the reference frame and the pixel value difference in the second segmented image of the reference frame; spatially filter the fourth depth value of the second pixel according to the second spatial weight and the second range weight of each of the second neighborhood pixels to obtain a second spatial filtering result of the fourth depth value.

[0159] In an exemplary embodiment, based on the aforementioned embodiment, the motion matrix determination module 910 can be specifically configured to: calculate the distance between the feature points of the current frame and the feature points of the reference frame, and determine the feature point matching pairs of the current frame and the reference frame according to the distance; perform singular value decomposition on the feature point matching pairs to obtain the homography matrix between the current frame and the reference frame; and determine the motion matrix according to the homography matrix.

[0160] In an exemplary implementation, based on the foregoing embodiment, determining the motion matrix according to the homography matrix includes: processing the homography matrix based on a random sampling consistency algorithm to determine the motion matrix.

[0161] In an exemplary implementation, based on the aforementioned embodiment, the motion matrix determination module 910 may also be specifically configured to: determine the motion matrix corresponding to each segmented area based on the first segmented image of the current frame and the second segmented image of the reference frame.

[0162] In an exemplary implementation, based on the foregoing embodiment, the motion matrix determination module 910 may also be specifically configured to: determine the motion matrix between the current frame of the video sequence and the reference frame of the current frame based on a dense optical flow algorithm.

[0163] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0164] The exemplary embodiments of the present disclosure also provide an electronic device for executing the above-mentioned video processing method, and the electronic device may be the above-mentioned terminal 110. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned video processing method by executing the executable instructions.

[0165] Below Fig.10 The mobile terminal 1000 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Fig.10 The construction in can also be applied to fixed type equipment.

[0166] like Fig.10 As shown, the mobile terminal 1000 may specifically include: a processor 1001, a memory 1002, a bus 1003, a mobile communication module 1004, an antenna 1, a wireless communication module 1005, an antenna 2, a display screen 1006, a camera module 1007, an audio module 1008, a power module 1009 and a sensor module 1010.

[0167] Processor 1001 may include one or more processing units. For example, processor 1010 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0168] For example, the input video may be processed by a graphics processor to obtain a target depth map of the current frame.

[0169] The processor 1001 may be connected to the memory 1002 or other components via a bus 1003 .

[0170] The memory 1002 may be used to store computer executable program codes, which include instructions. The processor 1001 executes various functional applications and data processing of the mobile terminal 1000 by running the instructions stored in the memory 1002. The memory 1002 may also store application data, such as images, videos and other files.

[0171] The communication function of the mobile terminal 1000 can be implemented by the mobile communication module 1004, antenna 1, wireless communication module 1005, antenna 2, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 204 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1000. The wireless communication module 1005 can provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. applied to the mobile terminal 1000.

[0172] The display screen 1006 is used to implement display functions, such as displaying a user interface, an image, or a video processed according to the video processing method of the present disclosure. The camera module 1007 is used to implement shooting functions, such as shooting images, videos, etc. The audio module 1008 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 1009 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 1010 may include a depth sensor 10101, a speed sensor 10102, a gyroscope sensor 10103, an air pressure sensor 10104, etc., to implement corresponding sensing detection functions.

[0173] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0174] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification, for example, it can execute Figures 2 to 7 Any one or more steps in .

[0175] The computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0176] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0177] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0178] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0179] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: include: determining a motion matrix between a current frame of a video sequence and a reference frame of the current frame; Determine, according to the motion matrix, a second pixel point that matches the first pixel point of the current frame in the reference frame to obtain a pixel point matching pair; Determining a region relationship between pixels in the pixel matching pair based on the first segmented image of the current frame and the second segmented image of the reference frame; According to the region relationship, the first depth map of the current frame and the second depth map of the reference frame are fused to obtain a target depth map of the current frame.

2. The video processing method according to claim 1, characterized in that: The fusing the first depth map of the current frame and the second depth map of the reference frame according to the region relationship to obtain a target depth map of the current frame includes: For each pixel matching pair, according to a preset window, determine a first neighborhood pixel adjacent to the first pixel in the current frame, and determine a second neighborhood pixel adjacent to the second pixel in the reference frame; Determining a first temporal weight of the first neighborhood pixel and a second temporal weight of the second neighborhood pixel according to an area relationship between the first neighborhood pixel and a second neighborhood pixel matched by the first neighborhood pixel in the reference frame; According to the first temporal weight and the second temporal weight, a first depth value of the first neighborhood pixel and a second depth value of the second neighborhood pixel are merged to obtain a target depth map of the current frame; The first depth value is determined according to the first depth map, and the second depth value is determined according to the second depth map.

3. The video processing method according to claim 2, characterized in that: The determining, according to the area relationship between the first neighborhood pixel point and the second neighborhood pixel point matched by the first neighborhood pixel point in the reference frame, a first temporal weight of the first neighborhood pixel point and a second temporal weight of the second neighborhood pixel point comprises: In a case where the area relationship is that the areas belong to the same segmented area, determining that the first time domain weight and the second time domain weight are both positive numbers; In a case where the area relationship is that the areas belong to different segmented areas, determining the first time domain weight to be a positive number and determining the second time domain weight to be 0; Among them, when the segmentation area to which the first neighborhood pixel point belongs in the first segmentation image and the segmentation area to which the second neighborhood pixel point belongs in the second segmentation image are the same, the area relationship is determined to belong to the same segmentation area; otherwise, the area relationship is determined to belong to different segmentation areas.

4. The video processing method according to claim 2, characterized in that: The fusing, according to the first temporal weight and the second temporal weight, the first depth value of the first neighborhood pixel and the second depth value of the second neighborhood pixel to obtain the target depth map of the current frame, includes: For each first neighborhood pixel point, a first spatial domain weight of the first neighborhood pixel point is determined according to a pixel position distance between the first pixel point and the first neighborhood pixel point, and a first value range weight of the first neighborhood pixel point is determined according to a pixel value difference between the first pixel point and the first neighborhood pixel in each channel of the RGB image of the current frame and a pixel value difference in the first segmented image of the current frame; Determining a first spatial domain weight of the first neighborhood pixel point according to the first spatial domain weight and the first range weight; For each second neighborhood pixel point, determine a second spatial domain weight of the second neighborhood pixel point according to the pixel position distance between the second pixel point and the second neighborhood pixel point, and determine a second value range weight of the second neighborhood pixel point according to the pixel value difference between the second pixel point and the second neighborhood pixel point in each channel of the RGB image of the reference frame and the pixel value difference in the second segmented image of the reference frame; Determining a second spatial domain weight of the second neighborhood pixel according to the second spatial domain weight and the second range weight; According to the first spatial domain weight, the first temporal domain weight, the second spatial domain weight, and the second temporal domain weight, the first depth value of the first neighborhood pixel and the second depth value of the second neighborhood pixel are fused to obtain a target depth map of the current frame.

5. The video processing method according to claim 1, characterized in that: The fusing the first depth map of the current frame and the second depth map of the reference frame according to the region relationship to obtain a target depth map of the current frame includes: For each pixel matching pair, determining a third time domain weight of the first pixel and a fourth time domain weight of the second pixel according to the region relationship; Based on the third temporal weight and the fourth temporal weight, fusing the third depth value of the first pixel and the fourth depth value of the second pixel to obtain a target depth map of the current frame; The third depth value is determined according to the first depth map, and the fourth depth value is determined according to the second depth map.

6. The video processing method according to claim 5, characterized in that: The determining, according to the area relationship, a third time domain weight of the first pixel and a fourth time domain weight of the second pixel includes: In a case where the area relationship is that the areas belong to the same segmented area, determining that the third time domain weight and the fourth time domain weight are both positive numbers; In the case where the area relationship is that the areas belong to different segmented areas, determining the third time domain weight to be a positive number and determining the fourth time domain weight to be 0; Among them, when the segmentation area to which the first pixel point belongs in the first segmentation image and the segmentation area to which the second pixel point belongs in the second segmentation image are the same, the area relationship is determined to belong to the same segmentation area; otherwise, the area relationship is determined to belong to different segmentation areas.

7. The video processing method according to claim 5, characterized in that: Based on the third temporal weight and the fourth temporal weight, fusing the third depth value of the first pixel and the fourth depth value of the second pixel to obtain a target depth map of the current frame, including: Performing spatial filtering on the third depth value and the fourth depth value respectively to obtain a first spatial filtering result of the third depth value and a second spatial filtering result of the fourth depth value; According to the third temporal weight and the fourth temporal weight, respectively fusing the first spatial filtering result and the second spatial filtering result to obtain a target depth value of the first pixel; A target depth map of the current frame is obtained based on the target depth value of the first pixel in each pixel matching pair.

8. The video processing method according to claim 7, characterized in that: The third depth value and the fourth depth value are spatially filtered respectively to obtain a first spatial filtering result of the third depth value and a second spatial filtering result of the fourth depth value, including: According to a preset window, determining a first neighborhood pixel point adjacent to the first pixel point in the current frame, and determining a second neighborhood pixel point adjacent to the second pixel point in the reference frame; For each first neighborhood pixel point, a first spatial domain weight of the first neighborhood pixel point is determined according to a pixel position distance between the first pixel point and the first neighborhood pixel point, and a first value range weight of the first neighborhood pixel point is determined according to a pixel value difference between the first pixel point and the first neighborhood pixel in each channel of the RGB image of the current frame and a pixel value difference in the first segmented image of the current frame; Performing spatial filtering on the third depth value of the first pixel point according to the first spatial domain weight and the first value range weight of each of the first neighborhood pixels to obtain a first spatial filtering result of the third depth value; For each second neighborhood pixel point, determine a second spatial domain weight of the second neighborhood pixel point according to the pixel position distance between the second pixel point and the second neighborhood pixel point, and determine a second value range weight of the second neighborhood pixel point according to the pixel value difference between the second pixel point and the second neighborhood pixel point in each channel of the RGB image of the reference frame and the pixel value difference in the second segmented image of the reference frame; According to the second spatial domain weight and the second value domain weight of each of the second neighborhood pixels, spatial filtering is performed on the fourth depth value of the second pixel to obtain a second spatial filtering result of the fourth depth value.

9. The video processing method according to claim 1, characterized in that: The determining of a motion matrix between a current frame of a video sequence and a reference frame of the current frame comprises: Calculating the distance between the feature points of the current frame and the feature points of the reference frame, and determining a matching pair of feature points of the current frame and the reference frame according to the distance; Performing singular value decomposition on the feature point matching pairs to obtain a homography matrix between the current frame and the reference frame; The motion matrix is ​​determined according to the homography matrix.

10. The video processing method according to claim 9, characterized in that: Determining the motion matrix according to the homography matrix includes: The homography matrix is ​​processed based on a random sampling consistency algorithm to determine the motion matrix.

11. The video processing method according to claim 1, characterized in that: The determining of a motion matrix between a current frame of a video sequence and a reference frame of the current frame comprises: Based on the first segmented image of the current frame and the second segmented image of the reference frame, a motion matrix corresponding to each segmented area is determined.

12. The video processing method according to claim 1, characterized in that: The determining of a motion matrix between a current frame of a video sequence and a reference frame of the current frame comprises: Based on a dense optical flow algorithm, a motion matrix between a current frame of a video sequence and a reference frame of the current frame is determined.

13. A video processing device, characterized in that: include: a motion matrix determination module configured to determine a motion matrix between a current frame of a video sequence and a reference frame of the current frame; a pixel point matching pair determination module, configured to determine, according to the motion matrix, a second pixel point that matches the first pixel point of the current frame in the reference frame, so as to obtain a pixel point matching pair; A region relationship determination module is configured to determine a region relationship between pixels in the pixel matching pair based on the first segmented image of the current frame and the second segmented image of the reference frame; The target depth map generation module is configured to fuse the first depth map of the current frame and the second depth map of the reference frame according to the region relationship to obtain a target depth map of the current frame.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. An electronic device, characterized in that: include: one or more processors; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 12.

Images