Method, apparatus and storage medium for generating depth video

Abstract

The application provides a depth video generation method, device, equipment and storage medium. The depth video generation method comprises: acquiring a depth map corresponding to each image in an input video to form a depth map sequence; identifying a depth map subsequence corresponding to the same shot in the depth map sequence; for each depth map subsequence, taking a first frame depth map in the depth map subsequence as a reference frame, and performing the following calibration operation on each frame after the reference frame to obtain a calibrated depth map subsequence: acquiring a weight corresponding to each pixel matching pair of a current frame depth map and a previous frame calibrated depth map; based on the weight of each pixel matching pair, minimizing the difference between the current frame depth map and the previous frame calibrated depth map to obtain a current frame calibrated depth map for the next frame calibration; and performing time sequence splicing based on each calibrated depth map subsequence to generate a depth video. The application can improve the quality of the depth video as a naked-eye three-dimensional display material.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device and storage medium for generating depth video. Background Technology

[0002] Glasses-free 3D display technology, as an important development direction of next-generation display technology, has received widespread attention from academia and industry in recent years. Unlike traditional stereoscopic display technology, glasses-free 3D display technology does not require users to wear any form of auxiliary glasses or head-mounted devices, allowing them to directly view images with depth and stereoscopic effect with the naked eye, greatly improving the comfort and convenience of the user experience.

[0003] In the content production process for glasses-free 3D displays, depth video is a key material. It can generate virtual perspective images through depth image rendering technology, and then synthesize them into interlaced images or multi-view content required for glasses-free 3D displays.

[0004] However, acquiring high-quality depth video still faces significant challenges. High-precision depth map acquisition often relies on specialized depth cameras or LiDAR equipment, which is costly and has limited applicability. Monocular video-based depth estimation algorithms offer a low-cost content acquisition path by independently estimating the depth of each frame of a 2D image in the video to generate a corresponding depth map. However, there is a possibility of temporal jitter in the depth values ​​of the same spatial point in adjacent frames. When such depth video is directly used for naked-eye 3D display, the jitter in depth values ​​between consecutive frames manifests as noticeable flickering of objects in the image, severely impacting the viewing experience and even causing visual fatigue and dizziness. Summary of the Invention

[0005] In view of the above, embodiments of this application provide a method, apparatus, device, and storage medium for generating depth video to solve at least one problem existing in the background art.

[0006] Firstly, a method for generating depth video is provided, the method comprising: Obtain the depth map corresponding to each frame of the input video to form a depth map sequence; Identify depth map subsequences corresponding to the same shot within the depth map sequence; For each of the depth map subsequences, using the first frame depth map in the depth map subsequence as the reference frame, the following calibration operation is performed on each frame following the reference frame to obtain the calibrated depth map subsequence: Obtain the weight corresponding to each pixel matching pair in the current frame depth map and the previous frame calibrated depth map, wherein the weight is negatively correlated with the depth value of the corresponding pixel matching pair in the current frame depth map and negatively correlated with the difference in depth value between the pixel matching pair in the previous frame calibrated depth map and the current frame depth map; Based on the weight of each pixel matching pair, the difference between the current frame depth map and the previous frame calibrated depth map is minimized to obtain the current frame calibrated depth map for calibration of the next frame. A depth video is generated by temporally stitching together the calibrated depth map subsequences.

[0007] In conjunction with the first aspect, in an optional implementation, obtaining the weights corresponding to each pixel matching pair between the current frame depth map and the calibrated depth map of the previous frame includes: Obtain the first weight of each pixel matching pair in the current frame depth map and the calibrated depth map of the previous frame. The first weight is inversely proportional to the depth value of the pixel matching pair in the current frame depth map. The second weight of each pixel matching pair in the current frame depth map and the previous frame calibrated depth map is obtained. The second weight is calculated based on a preset depth range and the depth value difference between the pixel matching pair in the previous frame calibrated depth map and the current frame depth map. The larger the depth value difference, the smaller the second weight. For each pixel matching pair, the first weight and the second weight of the pixel matching pair are multiplied to obtain the weight of the pixel matching pair.

[0008] In conjunction with the first aspect, in an optional implementation, minimizing the difference between the current frame depth map and the previous frame calibrated depth map based on the weights of each pixel matching pair to obtain the current frame calibrated depth map for calibration of the next frame includes: The depth value of any pixel in the calibrated depth map of the current frame is set to satisfy a linear relationship with the depth value of the corresponding pixel in the depth map of the current frame. The linear relationship includes the affine transformation parameters to be solved. Using the weights of each pixel matching pair as coefficients, the squares of the depth deviations of each pixel matching pair are weighted and summed to construct the objective function, wherein the depth deviation is: the difference between the depth value of the pixel matching pair in the current frame depth map after transformation by the linear relationship and the depth value in the calibrated depth map of the previous frame. Find the affine transformation parameters that minimize the objective function; Using the solved affine transformation parameters, the depth value of each pixel in the current frame depth map is transformed to obtain the current frame calibrated depth map.

[0009] In conjunction with the first aspect, in an optional implementation, the process of solving for the affine transformation parameters that minimize the objective function includes: Based on the weights of each pixel matching pair, a weighted average is calculated on the depth values ​​in the calibrated depth map of the previous frame to obtain a first weighted average, and a weighted average is calculated on the depth values ​​in the depth map of the current frame to obtain a second weighted average. Based on the weights of each pixel matching pair, the first weighted mean, the second weighted mean, the depth values ​​of each pixel matching pair in the calibrated depth map of the previous frame, and the depth values ​​in the depth map of the current frame, the weighted variance and the weighted covariance are calculated. The ratio of the weighted covariance to the sum of the weighted variance and a preset constant is used as the scaling factor in the affine transformation parameters. The first weighted mean minus the product of the scale factor and the second weighted mean is used as the offset in the affine transformation parameters.

[0010] In conjunction with the first aspect, in an alternative implementation, prior to the step of performing the calibration operation, the method further includes: Set the depth values ​​in each frame's depth map that are greater than a preset threshold to the preset threshold.

[0011] In conjunction with the first aspect, in an optional implementation, before the step of obtaining the weights corresponding to each pixel matching pair in the current frame depth map and the calibrated depth map of the previous frame, the method further includes: For each frame of the depth map in the depth map subsequence, a non-linear mapping is performed on the depth value of each pixel in the depth map, wherein the non-linear mapping is such that: for any two depth intervals of equal length, the interval length after mapping of the interval composed of values ​​that are closer to the camera is greater than the interval length after mapping of the interval composed of values ​​that are farther from the camera. Determine the maximum and minimum depth values ​​of the mapped depth map, and based on the maximum and minimum depth values ​​of the depth map, linearly scale the depth value of each pixel in the depth map to a preset depth range.

[0012] In conjunction with the first aspect, in an optional implementation, the depth map corresponding to each frame of the input video is obtained by performing depth estimation on the input video based on a trained depth estimation model; The training process of the depth estimation model includes: Multiple training image groups are obtained, and a label depth map corresponding to all training images in each training image group is obtained. The training image group includes an original two-dimensional image and an enhanced image obtained by modifying the texture of the original two-dimensional image. The enhanced image has the same image style and geometric structure as the original two-dimensional image. Based on each training image in the training image group and the corresponding label depth map, the initial depth estimation model is trained to obtain the trained depth estimation model.

[0013] In conjunction with the first aspect, in an optional implementation, the initial depth estimation model is a pre-trained depth estimation model; the step of training the initial depth estimation model based on each training image in the training image group and the corresponding label depth map to obtain a trained depth estimation model includes: The predicted depth map obtained by performing depth estimation on the original two-dimensional image using the pre-trained depth estimation model is used as the label depth map. Each training image in the training image group is used as input to retrain the pre-trained depth estimation model to obtain the trained depth estimation model.

[0014] In conjunction with the first aspect, in an optional implementation, the input resolution of the initial depth estimation model is set to be smaller than the resolution of the label depth map; the method further includes: For each pixel position in the target depth map that meets the input resolution, determine the corresponding region of the pixel position in the label depth map, and use the minimum value among all depth values ​​in the corresponding region as the depth value of the pixel position to shrink the label depth map to the target depth map. The step of training an initial depth estimation model based on each training image in the training image group and the corresponding label depth map to obtain a trained depth estimation model includes: The initial depth estimation model is trained based on the training image scaled down to the input resolution and the target depth map to obtain the trained depth estimation model.

[0015] In conjunction with the first aspect, in an alternative implementation, the method is executed in the video memory of a graphics processing unit (GPU), and the GPU does not transfer the data processed during the generation of the depth video between the GPU and the central processing unit.

[0016] Secondly, a depth video generation apparatus is provided, the apparatus comprising: The acquisition module is configured to acquire the depth map corresponding to each frame of the input video to form a depth map sequence; The identification module is configured to identify depth map subsequences corresponding to the same shot in the depth map sequence; The calibration module is configured to, for each of the depth map subsequences, use the first frame depth map in the depth map subsequence as the reference frame, and perform the following calibration operation on each frame following the reference frame to obtain a calibrated depth map subsequence: Obtain the weight corresponding to each pixel matching pair in the current frame depth map and the previous frame calibrated depth map, wherein the weight is negatively correlated with the depth value of the corresponding pixel matching pair in the current frame depth map and negatively correlated with the difference in depth value between the pixel matching pair in the previous frame calibrated depth map and the current frame depth map; Based on the weight of each pixel matching pair, the difference between the current frame depth map and the previous frame calibrated depth map is minimized to obtain the current frame calibrated depth map for calibration of the next frame. The generation module is configured to perform temporal stitching based on each of the calibrated depth map subsequences to generate a depth video.

[0017] Thirdly, an electronic device is provided, including a processor, a memory, and an executable program stored in the memory and executable by the processor, wherein the processor, when executing the executable program, performs steps of a depth video generation method as provided in any alternative implementation of the first aspect.

[0018] Fourthly, a storage medium is provided having an executable program stored thereon, which, when executed by a processor, implements the steps of the depth video generation method provided by any alternative implementation of the first aspect.

[0019] This application provides a depth video generation method, apparatus, device, and storage medium. It acquires depth maps corresponding to each frame of an input video to form a depth map sequence, identifies depth map sub-sequences corresponding to the same shot, and then performs frame-by-frame recursive calibration on each depth map sub-sequence based on its first frame depth map. Because each frame processing assigns higher weights to pixels with smaller depth values ​​and lower weights to pixels with larger depth values ​​based on the current frame's depth value, and higher weights to pixels with smaller depth differences and lower weights to pixels with larger depth differences based on inter-frame depth differences, the difference between the current frame's depth map and the previously calibrated depth map is minimized under weight constraints. This makes the calibration operation of the current frame's depth map more inclined to preserve the depth details of nearby objects or scenes. Furthermore, while suppressing depth value jitter between frames, it minimizes the impact of sudden depth value changes caused by real motion. This effectively suppresses inter-frame flicker in the depth video while preserving real motion information in the scene, improving the quality of the depth video as naked-eye 3D display material, and thus enhancing the viewing comfort of subsequent naked-eye 3D displays. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a depth video generation method according to one embodiment; Figure 2 yes Figure 1 The flowchart of step S131 is shown below; Figure 3 yes Figure 1 The flowchart of step S132 is shown below; Figure 4 yes Figure 3 The flowchart of step S1323 is shown below; Figure 5 This is a flowchart illustrating a training method for a depth estimation model according to one embodiment. Figure 1 ; Figure 6 A diagram showing an example of augmented training data and the corresponding ground truth depth values ​​is provided. Figure 7 A schematic diagram of a stylized image exhibiting geometric deviations is shown; Figure 8 A comparative diagram showing the rendering artifacts caused by blurring depth boundaries and the effect of suppressing artifacts is presented; Figure 9 This is a flowchart illustrating a training method for a depth estimation model according to one embodiment. Figure 2 ; Figure 10 A comparative diagram showing the impact of different depth map encoding quality on naked-eye 3D rendering effects is presented; Figure 11 This is a flowchart illustrating a method for generating naked-eye 3D content according to an embodiment; Figure 12 This is a schematic diagram of the structure of a depth video generation device according to an embodiment. Detailed Implementation

[0021] To make the technical solution and beneficial effects of this application more apparent and understandable, a detailed description is provided below by listing specific embodiments. The accompanying drawings are not necessarily drawn to scale, and local features may be enlarged or reduced to more clearly show the details of the local features; unless otherwise defined, the technical and scientific terms used herein have the same meanings as those in the technical field to which this application pertains.

[0022] The embodiments in this application are not exhaustive, but merely illustrative of some embodiments, and are not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0023] In each embodiment of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0024] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0025] In related technologies, the generation of content for naked-eye 3D displays mainly follows these technical paths: First, relying on professional 3D modeling software for manual modeling and rendering, and generating multi-view images through a virtual camera array. This method offers controllable image quality but has a long production cycle, high cost, and difficulty in handling complex real-world scenes. Second, using a multi-view camera array for real-world shooting, and synthesizing naked-eye 3D content by synchronously acquiring multi-view images. This method suffers from expensive equipment, complex systems, and cumbersome post-processing. Third, using 2D-to-3D algorithms to convert planar videos, but traditional algorithms struggle to accurately restore the scene's depth structure, easily resulting in depth distortion and visual artifacts.

[0026] To address the aforementioned issues, content generation based on depth images (RGBD) is gradually emerging as a promising solution. Depth video data contains color image information and its corresponding pixel depth information. Virtual perspective images can be generated through depth image rendering technology, thereby synthesizing multi-view content and effectively alleviating the high cost problem of traditional modeling and shooting.

[0027] However, acquiring depth video itself remains challenging. High-precision depth map acquisition often relies on specialized depth cameras or LiDAR equipment, which is costly and has limited applicability. Monocular video-based depth estimation algorithms offer a low-cost content acquisition path, generating visually high-quality depth maps by independently estimating the depth of each frame in the video. However, for video depth estimation tasks, accurate video depth data is difficult to obtain, and fitting temporal relationships through networks is challenging, resulting in depth estimation results prone to significant temporal flicker noise. The depth estimates for the same spatial point in adjacent frames may exhibit temporal jitter. When such depth videos are directly used for naked-eye 3D display, the jitter in depth values ​​between consecutive frames manifests as noticeable flickering of objects in the image, severely impacting the viewing experience and even causing visual fatigue and dizziness.

[0028] For ease of understanding, a brief explanation of some of the terms used in this application is provided.

[0029] Depth estimation: Inferring the distance from each pixel to the camera from a single 2D image or video, providing spatial depth information for applications such as 3D reconstruction and augmented reality. It predicts the 3D structure of a scene by analyzing visual features (such as texture, occlusion, and lighting).

[0030] Depth of field: refers to the range of object distances within which an optical system (such as a camera lens) can form a sharp image. Its size is affected by aperture, focal length, and shooting distance. In photography, depth of field is often controlled to achieve the artistic effect of a sharp subject and a blurred background.

[0031] Glasses-free 3D: A technology that allows users to view 3D stereoscopic displays directly without wearing special glasses. It separates and projects images for the left and right eyes through optical structures. The main implementation methods include parallax barriers, lenticular lenses, and directional light sources.

[0032] Cylinder-lens naked-eye 3D devices: These devices use a lenticular lens array as the beam-splitting element to alternately distribute pixels to the left and right eyes, creating a stereoscopic vision. They have advantages such as low brightness loss, continuous viewing angle, and the ability for multiple people to watch simultaneously, and are widely used in advertising displays, televisions, and mobile terminals.

[0033] CUDA-OpenGL interoperability is a technology that allows the CUDA (Compute Unified Device Architecture) computing kernel and OpenGL (Open Graphics Library) graphics resources to directly share video memory data, avoiding data copying between the graphics processing unit (GPU) and the central processing unit (CPU). It enables deep integration of real-time rendering and parallel computing, improving the processing efficiency of graphics applications.

[0034] Artifacts: Unnatural visual traces in images, videos, or 3D reconstruction results caused by algorithm defects, data compression, or hardware limitations, such as jagged edges, holes, spatial misalignment, etc., which seriously affect visual quality.

[0035] This application provides a depth video generation method that improves the temporal stability of monocular image depth estimation without relying on temporal network training, thereby improving the quality of depth video and reducing visual flicker caused by random jitter of depth values ​​in naked-eye 3D display.

[0036] like Figure 1 As shown, the depth video generation method includes the following steps: S110: Obtain the depth map corresponding to each frame of the input video to form a depth map sequence; S120: Identify depth map subsequences corresponding to the same shot within a depth map sequence; S130: For each depth map subsequence, take the first frame depth map in the depth map subsequence as the reference frame, and perform a calibration operation on each frame after the reference frame to obtain a calibrated depth map subsequence. The calibration operation includes steps S131 to S132. S131: Obtain the weights corresponding to each pixel matching pair in the current frame depth map and the previous frame calibrated depth map. The weights are negatively correlated with the depth values ​​of the corresponding pixel matching pairs in the current frame depth map and negatively correlated with the difference in depth values ​​between the pixel matching pairs in the previous frame calibrated depth map and the current frame depth map. S132: Based on the weight of each pixel matching pair, minimize the difference between the current frame depth map and the previous frame calibrated depth map to obtain the current frame calibrated depth map for calibration in the next frame. S140: Based on the time-series stitching of each calibrated depth map subsequence, a depth video is generated.

[0037] The depth video generation method provided in this application can be applied to any electronic device with image processing capabilities, such as a server or a terminal with image processing software installed. The server can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center; the terminal can be a PC (Personal Computer), smartphone, PDA (Personal Digital Assistant), tablet computer, or glasses-free 3D display device. This application does not impose any limitations on the specific type of electronic device.

[0038] In step S110 above, for each frame of the input video, a trained depth estimation model (e.g., a monocular image depth estimation model) can be used to estimate the depth of that frame to obtain the corresponding depth map. The depth maps corresponding to each frame of the input video are arranged in order according to the timestamp of each frame in the input video to form a depth map sequence.

[0039] The input video can be an RGB video stream captured in real time by a monocular camera, or it can be any pre-acquired two-dimensional video, such as movie video.

[0040] The depth estimation model can employ an encoder-decoder structure, such as one built on a convolutional neural network or Transformer architecture. The model input is a two-dimensional image, and the output is a depth map of the same size as the input image. This embodiment does not impose specific limitations on the model structure. The training process of the depth estimation model can be found in the relevant descriptions in the embodiments below, and will not be elaborated here.

[0041] In addition, depth maps corresponding to each frame of the input video can be obtained by methods such as binocular stereo matching.

[0042] Each pixel in the depth map corresponds to a depth information value, which can be either a depth value or an inverse depth value, depending on the model task. The depth value represents the actual distance between the pixel's spatial point and the camera; a smaller depth value indicates a closer distance to the camera. The inverse depth value is negatively correlated with the depth value and can be the reciprocal of the depth value or a linear transformation thereof; a larger inverse depth value indicates a closer spatial point to the camera.

[0043] For ease of description, the embodiments of this application mainly use depth values ​​as an example for illustration. It is understood that when depth video generation is performed using inverse depth values, the relevant content description can be adjusted adaptively. For example, "the weight corresponding to the pixel matching pair is negatively correlated with the depth value" can be adjusted to "the weight corresponding to the pixel matching pair is positively correlated with the inverse depth value".

[0044] In step S120 above, a shot refers to a segment of continuous shooting during video recording. When a shot changes, the content of the image often changes significantly, and the depth distribution between two consecutive frames lacks temporal continuity. Therefore, to improve the temporal stability of monocular image depth estimation, this step performs shot detection on the depth map sequence to identify depth map subsequences corresponding to the same shot within the depth map sequence. This identification process can use dense feature registration methods (such as HomoMatcher (a homography-based matching method), RoMa (a robust matching algorithm), or advanced dense optical flow methods) to perform dense registration between adjacent video frames. If registration fails, it can be determined that a shot has changed. Alternatively, shot detection can be performed using inter-frame image differences (such as color histogram differences, sum of pixel differences, etc.). When the difference exceeds a preset threshold, it is determined that a shot has changed.

[0045] Based on the shot detection results, the depth map sequence is divided into multiple depth map subsequences, each subsequence corresponding to a consecutive shot segment. If the entire video contains only one shot, then the corresponding depth map subsequence is the depth map sequence itself.

[0046] In step S130 above, the depth map subsequence corresponding to each shot is subjected to temporal calibration. This temporal calibration is used to suppress random jitter of depth values ​​between adjacent frames within the same shot, while reducing the impact of depth value changes caused by real motion, so as to preserve real motion information in the scene.

[0047] For each depth map subsequence, the first frame depth map in the subsequence is used as the reference frame, which does not require calibration. For each frame after the reference frame, steps S131 to S132 are executed. Let the current frame depth map be the nth frame depth map to be calibrated in the depth map subsequence, and the previous calibrated depth map be the (n-1)th calibrated frame depth map, where n is an integer greater than 1.

[0048] In step S131, the weights of each pixel matching pair in the current frame depth map and the previous frame calibrated depth map are determined based on the current frame depth map and the previous frame calibrated depth map. It can be understood that if the current frame depth map is the second frame depth map in the depth map subsequence (i.e., n=2), then the first frame depth map (the reference frame) can be used as the previous frame calibrated depth map of the current frame depth map.

[0049] For ease of description, the term "adjacent frames" will be used below to refer to "the depth map of the current frame and the calibrated depth map of the previous frame".

[0050] Pixels corresponding to the same spatial point in adjacent frames constitute a pixel matching pair. For each pixel matching pair, the weight of the pixel matching pair satisfies the following conditions: it is negatively correlated with its depth value in the depth map of the current frame, and negatively correlated with the difference in depth values ​​between adjacent frames (the calibrated depth map of the previous frame and the depth map of the current frame).

[0051] Through the aforementioned negative correlation, pixels with smaller depth values ​​in the current frame depth map are assigned higher weights, while pixels with larger depth values ​​are assigned lower weights. This makes the calibration operation of the current frame depth map more inclined to preserve the depth details of nearby objects or scenes. At the same time, pixels with larger depth value differences between frames (i.e., more drastic depth changes) are assigned lower weights, while pixels with smaller depth value differences (i.e., more gradual depth changes) are assigned higher weights. This allows the calibration operation of the current frame depth map to suppress depth value jitter between frames while minimizing the impact of abrupt changes in depth values ​​caused by real motion.

[0052] The weights corresponding to pixel matching pairs are used to characterize the importance of the pixel matching pair in subsequent calibration. A first weight function can be constructed to calculate the first weight, and a second weight function can be constructed to calculate the second weight. The weights corresponding to the pixel matching pairs are calculated based on the first and second weights. For example, the first weight can be inversely proportional to the depth value, and the second weight can be inversely proportional to the difference in depth values ​​between frames or exhibit an exponentially decreasing relationship. The embodiments of this application do not specifically limit the function form used, as long as the aforementioned negative correlation is satisfied.

[0053] In step S132, after obtaining the weights of each pixel matching pair, the current frame depth map is adjusted based on the weights to minimize the difference between the current frame depth map and the calibrated depth map of the previous frame, and the adjusted depth map is used as the calibrated depth map of the current frame. This operation of minimizing the difference can be achieved by constructing an objective function with weights as coefficients and solving for the adjustment amount of the current frame depth map using an optimization algorithm. The optimization algorithm can be weighted least squares, Gauss-Newton, or similar algorithms.

[0054] After the calibration of the depth map of the nth frame is completed, the calibrated depth map is used as the calibrated depth map of the previous frame when processing the (n+1)th frame, thereby realizing frame-by-frame recursive calibration.

[0055] By sequentially performing sub-steps S131 and S132 on all frames following the reference frame in each depth map subsequence, the corresponding calibrated depth map subsequence can be obtained. In this calibrated depth map subsequence, the reference frame remains unchanged, and the depth value of each pixel in the depth map of each subsequent frame is adjusted to be temporally continuous with the depth value of the corresponding pixel in the depth map of the previous frame.

[0056] In step S140, after obtaining the calibrated depth map subsequence, the calibrated depth map subsequences are spliced ​​together in the time order of the depth map sequence to form a depth video corresponding to the depth map sequence.

[0057] For example, in step S120 above, when identifying depth map subsequences corresponding to the same shot in the depth map sequence, the start and end frame positions of each depth map subsequence in the depth map sequence can be recorded. Based on this position information, the calibrated depth map subsequences are arranged sequentially according to their time order, i.e., the subsequence corresponding to the first shot is placed first, the subsequence corresponding to the second shot follows immediately, and so on, until finally a calibrated depth map sequence that is completely consistent with the original depth map sequence in frame order and number of frames is formed, thus obtaining the depth video. Each frame depth map in this depth video corresponds to one frame image in the original input video.

[0058] In the depth video generation method provided in this embodiment, depth maps corresponding to each frame of the input video are obtained to form a depth map sequence, and depth map sub-sequences corresponding to the same shot are identified. Then, for each depth map sub-sequence, frame-by-frame recursive calibration is performed based on the depth map of its first frame. Because in each frame processing, pixels with smaller depth values ​​are assigned higher weights and pixels with larger depth values ​​are assigned lower weights according to the depth value of the current frame, and pixels with smaller differences are assigned higher weights and pixels with larger differences are assigned lower weights according to the depth differences between frames, the difference between the current frame depth map and the calibrated depth map of the previous frame is minimized under weight constraints. This makes the calibration operation of the current frame depth map more inclined to preserve the depth details of nearby objects or scenes. Furthermore, while suppressing depth value jitter between frames, the impact of depth value abrupt changes caused by real motion is minimized. This effectively suppresses inter-frame flicker in the depth video while preserving real motion information in the scene, improving the quality of the depth video as naked-eye 3D display material, and thus improving the viewing comfort of subsequent naked-eye 3D displays.

[0059] In some embodiments, before performing step S130, the method may further include: setting depth values ​​greater than a preset threshold in the depth map of each frame as the preset threshold.

[0060] When multiple viewers are simultaneously viewing a naked-eye 3D display device based on a lenticular lens grating, the effective depth range is often limited to avoid ghosting caused by excessive parallax. In this embodiment, before performing calibration on the depth map subsequence, a depth map cropping process is first performed, that is, the depth values ​​of pixels in the depth map with depth values ​​higher than a preset threshold are assigned the preset threshold value. This expands the relative variation range of near-field depth values ​​in the overall depth distribution, highlighting the stereoscopic effect of near-field objects, while suppressing ghosting caused by excessively large far-field depth values ​​leading to parallax exceeding the fusion range.

[0061] The value of the preset threshold can be determined based on the output scale of the depth estimation model. When the depth value is in meters, a preset threshold of 30 meters is more appropriate.

[0062] In some embodiments, in order to effectively highlight the stereoscopic effect of near objects in naked-eye 3D display, the method may further include the following before performing step S131: For each frame of the depth map in the depth map subsequence, the depth value of each pixel in the depth map is non-linearly mapped; the maximum and minimum depth values ​​of the mapped depth map are determined, and based on the maximum and minimum depth values ​​of the depth map, the depth value of each pixel in the depth map is linearly scaled to a preset depth range.

[0063] The nonlinear mapping is such that, for any two depth intervals of equal length, the interval length after mapping of the interval composed of values ​​closer to the camera is greater than the interval length after mapping of the interval composed of values ​​farther from the camera.

[0064] In this embodiment, nonlinear mapping is used to transform the depth value through a nonlinear function, so that the change in the foreground region in the input space is non-uniformly stretched in the output space, and the change in the background region is non-uniformly compressed in the output space.

[0065] In depth space, the foreground region is composed of pixels with smaller depth values, while the background region is composed of pixels with larger depth values.

[0066] In some examples, the nonlinear mapping employs a logarithmic function, which may take the following form: ,in, This is the original depth value. The depth value is the result of nonlinear mapping, and β is a preset parameter. By adjusting the value of β, the stretching and compression ratio of the foreground and background can be flexibly controlled to adapt to the depth distribution characteristics of different scenes.

[0067] For example, the preset parameter β can be 0.3, making the nonlinear mapping function affect the near-field depth value ( Smaller depth values ​​have a larger slope, which can effectively stretch the depth value distribution in the foreground area; for the depth values ​​in the background... The slope gradually decreases (at a relatively large rate) to compress the depth value of the distant view.

[0068] Maximum depth value based on the mapped depth map and minimum depth value It can process all depth values ​​of the mapped depth map. Linear scaling to the preset depth of field range of naked-eye 3D Inside, to obtain the target depth value The linear scaling formula used is as follows: .

[0069] The preset depth of field range can be adjusted according to actual needs to meet the different users' requirements for stereoscopic effect, offering high flexibility.

[0070] This embodiment performs non-linear mapping processing on each depth value in the depth map and linearly scales it to a preset depth range, so that the near depth values ​​obtain a wider numerical distribution range in the scaled depth map. This allows for more precise differentiation of near objects at different distances when calculating weights based on depth values, thereby enhancing the temporal calibration stability of the near area.

[0071] In some embodiments, such as Figure 2 As shown, in step S131 above, obtaining the weights corresponding to each pixel matching pair between the current frame depth map and the previous frame calibrated depth map may include: S1311: Obtain the first weight of each pixel matching pair in the current frame depth map and the calibrated depth map of the previous frame. The first weight is inversely proportional to the depth value of the pixel matching pair in the current frame depth map. S1312: Obtain the second weight of each pixel matching pair in the current frame depth map and the previous frame calibrated depth map. The second weight is calculated based on the preset depth range and the depth value difference between the pixel matching pair in the previous frame calibrated depth map and the current frame depth map. The greater the depth value difference, the smaller the second weight. S1313: For each pixel matching pair, multiply the first weight and the second weight of the pixel matching pair to obtain the weight of the pixel matching pair.

[0072] The above steps S1311 and S1312 can be executed simultaneously to improve the efficiency of weight acquisition for each pixel matching pair.

[0073] In this embodiment, for pixel matching pairs at the same coordinate position in the depth map of the current frame (the nth frame) and the calibrated depth map of the previous frame (the (n-1th frame)). Let its depth value in the calibrated depth map of the previous frame be . Its depth value in the current frame depth map is The pixel matching pair weight By first weight With the second weight Multiplying them together yields the result, i.e. .

[0074] In some examples, the first weight It can be represented as: ,in, For a very small positive number (e.g., 10) -6 ),set up It is used to prevent numerical calculation anomalies that may occur when the depth value is a minimum or 0.

[0075] The first weight is inversely proportional to the depth value of the pixel matching pair in the current frame depth map, so that pixels closer to the camera (with smaller depth values) receive higher weights, thereby imposing stronger temporal constraints on the near-field area during the calibration operation.

[0076] In some examples, the second weight Based on pixel matching pairs The difference in depth values ​​between the calibrated depth map of the previous frame and the depth map of the current frame. The calculation shows that the greater the difference in depth, the smaller the second weight.

[0077] Second weight It can be represented as: ,in, This is a control factor, and for example, it can be 2. Based on the preset depth of field range Settings, such as taking the maximum depth value With minimum depth value The product of the difference and a preset percentage (such as 10% or 20%).

[0078] The second weight is calculated based on a preset depth range and the difference in depth values ​​between the pixel matching pair and the depth map of the previous frame and the current frame. Thus, when the depth value of a pixel changes slightly between adjacent frames (jitter), the second weight takes a larger value, so that the current frame calibration process will force the depth value of the pixel in the current frame to approach the depth value of the previous frame. When the depth value changes drastically (abruptly), the second weight takes a smaller value, so that the current frame calibration process allows the pixel to retain a larger depth change, thereby reducing the impact of abrupt changes in depth value caused by real motion.

[0079] In some embodiments, such as Figure 3As shown, in step S132 above, minimizing the difference between the current frame depth map and the previous frame calibrated depth map based on the weight of each pixel matching pair to obtain the current frame calibrated depth map for the next frame calibration may include the following steps: S1321: Set the depth value of any pixel in the calibrated depth map of the current frame to satisfy a linear relationship with the depth value of the corresponding pixel in the depth map of the current frame. The linear relationship includes the affine transformation parameters to be solved.

[0080] For example, suppose the current frame depth map is the nth frame depth map in the current depth map subsequence to be calibrated, the linear relationship can be expressed as: ;in, For the pixel matching pairs in the calibrated depth map of the nth frame The depth values ​​of the relevant pixels, For the pixel matching pair in the depth map of the nth frame The depth values ​​of the relevant pixels, and These are the scale factor and offset, respectively, in the affine transformation parameters to be solved for the depth map of the nth frame.

[0081] S1322: Using the weights of each pixel matching pair as coefficients, the squares of the depth deviations of each pixel matching pair are weighted and summed to construct the objective function, where the depth deviation is the difference between the depth value of the pixel matching pair in the current frame depth map after linear transformation and the depth value in the calibrated depth map of the previous frame.

[0082] For example, the solution is obtained by weighted least squares method. and Minimize the objective function, objective function Represented as: .

[0083] in, For pixel matching pairs between the depth map of frame n and the calibrated depth map of frame n-1 The weight, For pixel matching pairs Depth value in the depth map of the nth frame The depth value obtained after linear transformation. For pixel matching pairs The depth value in the calibrated depth map in frame n-1.

[0084] S1323: Solve for the affine transformation parameters that minimize the objective function.

[0085] Find the affine transformation parameters that minimize the objective function. and The weighted least squares problem has a closed-form analytical solution. For example, the weights of each pixel matching pair can be used to perform weighted statistics on the depth values ​​in the calibrated depth map of the previous frame and the depth map of the current frame. Based on the results of the weighted statistics, the scale factor and offset in the affine transformation parameters can be determined.

[0086] S1324: Using the solved affine transformation parameters, transform the depth value of each pixel in the current frame depth map to obtain the current frame calibrated depth map.

[0087] For example, it can be done according to the above linear relationship. Using the parameters obtained from the solution and A linear transformation is performed on the depth value of each pixel in the depth map of the nth frame. The resulting depth map is the calibrated depth map of the nth frame.

[0088] After calibrating the depth map of frame n (the current frame), the calibrated depth map of frame n is used as the calibrated depth map of the previous frame of the (n+1)th frame. Together with the depth map of frame n+1, the weights corresponding to each pixel matching pair in the depth map of frame n+1 and the calibrated depth map of frame n, as well as the affine transformation parameters used for the depth map of frame n+1, are calculated to calibrate the depth map of frame n+1. This process is repeated frame by frame until the calibration operation of all frames after the reference frame within the depth map subsequence is completed.

[0089] In this embodiment, a linear mapping relationship is established between the current frame depth value and the calibrated depth value of the previous frame within a weighted least squares framework. The weights of each pixel matching pair are used as constraint coefficients to achieve adaptive calibration of each pixel in the current frame depth map. The depth values ​​of pixels with higher weights (close-up shots, small inter-frame differences) are forced to approximate the calibrated depth value of the previous frame, effectively suppressing random temporal jitter. Conversely, the depth values ​​of pixels with lower weights (distant shots, large inter-frame differences) are allowed to differ significantly from the calibrated depth map of the previous frame, reducing the possibility of false smoothing of real motion information. This frame-by-frame recursive calibration method effectively improves the temporal stability of depth changes within the entire shot subsequence.

[0090] In some embodiments, such as Figure 4 As shown, in step S1323 above, solving for the affine transformation parameters that minimize the objective function may include the following steps: S13231: Based on the weight of each pixel matching pair, the depth values ​​in the calibrated depth map of the previous frame are weighted and averaged to obtain the first weighted average, and the depth values ​​in the depth map of the current frame are weighted and averaged to obtain the second weighted average.

[0091] For example, the first weighted mean of all depth values ​​in the calibrated depth map of the (n-1)th frame is calculated. The calculation formula is: ;in, For pixel matching pairs The weight, For pixel matching pairs The depth value in the calibrated depth map in frame n-1.

[0092] Calculate the second weighted mean of all depth values ​​in the depth map of frame n. The calculation formula is: ;in, For pixel matching pairs The weight, For pixel matching pairs The depth value in the depth map of the nth frame.

[0093] S13232: Calculate the weighted variance and weighted covariance based on the weights of each pixel matching pair, the first weighted mean, the second weighted mean, the depth value of each pixel matching pair in the calibrated depth map of the previous frame, and the depth value in the depth map of the current frame.

[0094] For example, the weighted variance of the current frame depth map is calculated. The calculation formula is as follows: ;in, For pixel matching pairs The weight, For pixel matching pairs The depth value in the current frame's depth map. It is the second weighted average of all depth values ​​in the current frame's depth map.

[0095] For example, the weighted covariance between the depth map of frame n and the depth map of the calibrated frame (frame n-1) is calculated. The calculation formula is as follows: ;in, For pixel matching pairs The weight, For pixel matching pairs The depth value in the calibrated depth map of the previous frame. For pixel matching pairs The depth value in the depth map of the nth frame. The first weighted average of all depth values ​​in the calibrated depth map of frame n-1. It is the second weighted average of all depth values ​​in the depth map of the nth frame.

[0096] S13233: The ratio of the weighted covariance to the sum of the weighted variance and a preset constant is used as the scaling factor in the affine transformation parameters.

[0097] S13234: Subtract the product of the scale factor and the second weighted mean from the first weighted mean, and use it as the offset in the affine transformation parameters.

[0098] Here, scale factor The calculation formula is expressed as: ;in, The weighted variance of the current frame depth map. The weighted covariance between the current frame depth map and the previous calibrated frame depth map. For a very small positive number (e.g., 10) -6 (), used to prevent division by zero and increase numerical stability.

[0099] offset The calculation formula is expressed as: ;in, It is the first weighted average of all depth values ​​in the depth map of the (n-1)th calibrated frame. It is the second weighted average of all depth values ​​in the depth map of the nth frame.

[0100] This embodiment transforms the weighted least squares problem between the current frame depth value and the calibrated depth value of the previous frame into a closed-form analytical solution through the above steps. It can quickly obtain the optimal affine transformation parameters without iterative optimization, thus achieving high computational efficiency when performing temporal calibration on each frame depth map, which is beneficial to meeting the needs of real-time depth video processing.

[0101] In some embodiments, the depth video generation method is executed in the video memory of a graphics processing unit (GPU), and the data processed during the generation of the depth video is not transferred between the GPU and the central processing unit (CPU).

[0102] In the process of generating the depth video, the steps of acquiring the depth map sequence, identifying the depth map subsequence, performing calibration operations on the depth map subsequence, and stitching are all performed in the GPU memory. The color image data of each frame in the input video, the depth data of the depth map sequence, and the intermediate data generated in each step are continuously stored in the GPU memory during the execution of the depth video generation method, and are not transmitted through the data bus between the GPU and the CPU.

[0103] This can alleviate the transmission latency and bandwidth consumption caused by data copying between the CPU and GPU, enabling the acquisition, calibration, and stitching of depth map sequences to be completed efficiently within the GPU, thereby improving the overall processing efficiency of depth video generation.

[0104] Furthermore, the depth video generated in the aforementioned embodiments is typically used for interlacing graph synthesis in glasses-free 3D displays. The generation process of the interlacing graph involves computational operations such as viewpoint synthesis, pixel remapping, and optical parameter matching. Especially on high-resolution and large-size glasses-free 3D display devices, the computational load required for real-time processing is substantial. In traditional processing workflows, the generation and subsequent rendering of depth video are often performed in stages, requiring multiple exchanges of intermediate data between the CPU and GPU, making it difficult to simultaneously meet the real-time playback requirements of high resolution and high frame rate. If all intermediate data is pre-rendered and stored to avoid transmission delays, it will consume a large amount of storage space and reduce the flexibility and real-time interaction capabilities during playback.

[0105] Therefore, in some embodiments, the method may further include: placing all steps of the aforementioned video depth generation process and the subsequent interlacing map synthesis and rendering steps within the graphics processor's video memory for unified execution. In the complete processing pipeline, the input video's color image data, depth map data for each frame, intermediate data generated during calibration, and the finally generated interlacing map data are all stored in the GPU's video memory and are not transmitted through the data bus between the GPU and CPU. This further alleviates the latency and bandwidth consumption caused by data copying between the CPU and GPU, and minimizes GPU video memory copying except for computation, effectively improving performance. It supports 8K high-resolution real-time rendering, and users can adjust the depth of field in real time for the most comfortable viewing experience.

[0106] In related technologies, some video depth estimation schemes focus on data acquisition, using depth cameras to obtain scene scale information and then performing scale restoration on the monocular depth estimation results. However, this method requires additional data acquisition steps and can only acquire the depth of natural images. For 3D animated films, the depth data is licensed from film companies but is usually not publicly available, and the original depth information is easily damaged during manual restoration or post-processing. For AI-generated videos, the depth is even more unknown. Although monocular depth estimation technology is very effective in monocular image depth estimation, capable of generating visually high-quality depth maps, accurate video depth data is difficult to obtain for video frame depth estimation. Furthermore, fitting temporal relationships using networks is challenging, and easily obtained depth estimation results often exhibit significant temporal flicker noise.

[0107] Therefore, in some embodiments, a method for training a depth estimation model is provided. The depth estimation model trained by the method can be used in step S110 of the aforementioned embodiments, that is, to perform depth estimation on each frame of the input video to obtain a depth map corresponding to each frame, thereby providing a more stable single-frame depth estimation result for subsequent time-series processing.

[0108] like Figure 5 As shown, the training method for the depth estimation model may include steps S220 to S240.

[0109] S220: Obtain multiple training image groups, and a label depth map corresponding to all training images in each training image group. The training image group includes the original two-dimensional image and the enhanced image obtained by texture modification of the original two-dimensional image. The enhanced image has the same image style and geometric structure as the original two-dimensional image.

[0110] In this embodiment, each training image group may include an original two-dimensional image and multiple enhanced images generated from the original two-dimensional image after texture modification.

[0111] Specifically, a portion of the original 2D image can be selected, and its texture modified to generate multiple enhanced images of each original 2D image. Texture modification can be performed using deep learning-based image editing tools. Modifications include, but are not limited to, changing the color of objects in the image (such as clothing color), adjusting the brightness and color temperature of ambient lighting, and replacing background textures. During texture modification, explicit geometric constraints are applied to ensure that the contours and spatial positions of key elements such as faces and limbs in the original 2D image remain unchanged; that is, the enhanced image and the original 2D image are as geometrically consistent as possible. Figure 6 As shown, Figure 6 An example of augmented training data and the corresponding ground truth depth values ​​are shown. To protect privacy, the eye area of ​​the faces in the image has been pixelated.

[0112] However, despite the aforementioned geometric constraints, image editing tools may still output images with other styles or images where some geometric structures are shifted or distorted during the generation process, such as... Figure 7 As shown, Figure 7 The leftmost image is the original image, while the middle and rightmost images are two other stylized images mistakenly generated during image editing. If these stylized images with geometric deviations are directly used for model training, they will negatively impact the network's learning of geometric invariance. Therefore, after generating the enhanced images, manual screening (e.g., comparing the enhanced and original images with a semi-transparent overlay) can be used to remove samples with inconsistent geometric structures. The retained enhanced images, along with the original 2D images, constitute a training image set, where all images share the same labeled depth map as a supervision signal.

[0113] In one implementation, the original two-dimensional image and the labeled depth map can be a pair of RGB-D images synchronously acquired by a depth camera for the same real scene.

[0114] In another implementation, a pre-trained depth estimation model can be used to estimate the depth of the original 2D image, and the estimated predicted depth map can be used as the label depth map for the training image group. This method can complete model fine-tuning without real depth labels and is suitable for scenarios where it is not easy to obtain real depth labels, such as 3D animation, computer-generated images, or AI-generated videos.

[0115] S240: Based on each training image in the training image group and the corresponding label depth map, train the initial depth estimation model to obtain the trained depth estimation model.

[0116] Each training image within the training image set (including the original image and its augmented image) is used as the model input, and the shared label depth map of the training image set is used as the supervision signal to train the initial depth estimation model. During training, the model is forced to learn the characteristic that the output depth does not change with appearance changes such as texture or lighting, thereby enhancing the model's robustness to texture changes.

[0117] In some embodiments, the initial depth estimation model is a pre-trained depth estimation model; step S240 may include: using the predicted depth map obtained by performing depth estimation on the original two-dimensional image using the pre-trained depth estimation model as the label depth map, and using each training image in the training image group as input to retrain the pre-trained depth estimation model to obtain the trained depth estimation model.

[0118] In this embodiment, the pre-trained depth estimation model can be a monocular depth estimation model pre-trained on large-scale data. For each frame of a 2D image in the input video, the pre-trained model is first used to perform depth estimation to obtain the corresponding depth map. If the output of the pre-trained model is a disparity map (i.e., the reciprocal of the depth value), it needs to be converted into a depth map before further processing. The pre-trained model is then fine-tuned and retrained using the aforementioned training image set. The fine-tuning process can use a small learning rate, updating only some network layer parameters, to improve the geometric consistency under texture variation conditions while retaining the existing generalization ability of the pre-trained model.

[0119] Through the above training, the obtained depth estimation model exhibits stronger stability for slight texture fluctuations and lighting changes during single-frame inference. When this model is applied to frame-by-frame depth estimation of video, the geometric consistency of the depth maps output independently in each frame in static or quasi-static regions is effectively improved. This provides a more reliable single-frame depth map for the timing calibration in step S130 of the aforementioned embodiment, helping to further alleviate timing flicker.

[0120] Depth estimation models based on architectures such as the Vision Transformer exhibit a space complexity that increases quadratically with increasing image resolution. Training depth estimates often requires downscaling RGB images to a resolution acceptable to the video memory, resulting in a downscaled depth image of the same resolution. This is typically much smaller than the original image and the depth map captured by the device (including depth maps generated by rendering techniques), usually below 1500×1500. Related techniques use differentiable linear interpolation to bring the depth map back to the original resolution before calculating the loss. However, this introduces significant uncertainty into depth estimation. For example, at the foreground-background boundary, depth estimation tends to learn intermediate values ​​between the foreground and background, failing to capture a clear boundary. Linear interpolation easily smooths the depth values ​​at the boundary to an intermediate value between the foreground and background depths, resulting in blurred depth boundaries (as shown in the red box area in the figure). When such depth maps are used for depth-image-based rendering, the blurred depth values ​​at the foreground-background boundary cause color interpolation, producing noticeable visual artifacts, such as… Figure 8 As shown in the left figure.

[0121] To mitigate the aforementioned artifacts, this embodiment employs a neighborhood minimum sampling strategy to reduce the resolution of the label depth map. With this strategy, the foreground boundary is clearly preserved in the reduced-resolution depth map. When rendering based on this depth map, color intrusion artifacts at the foreground-background boundary are effectively suppressed, as shown in the following effect: Figure 8 As shown in the right figure.

[0122] In some embodiments, the input resolution of the initial depth estimation model is set to be smaller than the resolution of the label depth map; based on Figure 5 ,like Figure 9 As shown, the method may further include the following steps: S230: For each pixel position in the target depth map that matches the input resolution, determine the corresponding region of the pixel position in the label depth map, and use the minimum value among all depth values ​​in the corresponding region as the depth value of the pixel position to shrink the label depth map into the target depth map. The above step S240 may further include: S241: Based on the training image scaled down to the input resolution and the target depth map, train the initial depth estimation model to obtain the trained depth estimation model.

[0123] In this embodiment, by executing step S230, for each pixel position in the target resolution depth map, its corresponding region in the original resolution label depth map is determined (for example, when the scaling ratio is r, the corresponding region is a pixel neighborhood of size r×r), and the minimum value among all depth values ​​in this region is taken as the depth value of the target pixel. Since the neighborhood minimum operation tends to retain the smaller foreground depth values, the depth map after resolution scaling can maintain a clear foreground edge, rather than a transition value between the foreground and background. This minimum depth value is the estimation target that the network needs to learn during training.

[0124] After the above processing, the down-resolution RGB image is used only as input to the depth estimation network; furthermore, the RGB images or input videos mentioned in other embodiments of this application refer to the original resolution data. The depth map output by the depth estimation network has the same resolution as the network input resolution, and is saved at this resolution without being upscaled to the original resolution through linear interpolation.

[0125] In some embodiments, when encoding and storing the depth map output by the depth estimation model, the H.265 encoding format and a preset constant rate factor (CRF), such as CRF=13 or a higher quality parameter, can be used to reduce the impact of encoding loss on depth accuracy. The encoding loss of the depth map has a significant impact on the quality of naked-eye 3D rendering. For a clear comparison, Figure 10 The image demonstrates the difference in rendering quality when the viewpoint is expanded from the main viewpoint to the extreme viewpoint. The left image is encoded in H.265 and saved with CRF=18, resulting in lower quality; the right image is encoded in H.265 and saved with CRF=8, resulting in higher quality.

[0126] In some embodiments, the method may further include: copying the original resolution color image frames of the input video to the texture memory of the GPU; accessing the depth maps of each frame in the depth video in a GPU parallel computing thread; for each pixel of the target display screen, determining its corresponding coordinates in the depth map, and searching for the maximum depth value within a preset neighborhood centered on the coordinates; and updating the mesh vertex coordinates for naked-eye 3D rendering based on the searched maximum depth value.

[0127] In this embodiment, the original resolution RGB video frames are directly copied to the OpenGL texture memory via the CUDA-OpenGL interoperability mechanism for subsequent rendering; the depth maps of each frame in the depth video are accessed in CUDA parallel computing threads. Each CUDA thread corresponds to a pixel on the target display screen, whose resolution is the final naked-eye 3D display resolution and is independent of the resolution of the original RGB video and depth video. For each screen pixel, the CUDA thread first calculates its normalized pixel coordinates in screen space, and then maps these coordinates to the corresponding coordinate position in the depth frame. In the depth frame, the maximum depth value is searched within a predetermined neighborhood centered on this coordinate position, and the coordinate values ​​of the corresponding mesh vertex are updated with this maximum value, thereby generating 3D mesh vertices for subsequent multi-view rendering.

[0128] In some embodiments, the search radius of the preset neighborhood range is determined based on the resolution of the input video, wherein a first search radius is used when the resolution of the input video is lower than a preset threshold, and a second search radius is used when the resolution of the input video is higher than the preset threshold, wherein the first search radius is greater than the second search radius.

[0129] The radius of the aforementioned neighborhood search can be adaptively adjusted based on the sharpness of the input video. Specifically, when the video is blurry and lacks detail, a larger search radius can be used to reduce color artifacts caused by depth value blurring at the foreground-background boundary. Conversely, when the video is sharp and has clear edges, the search radius should be reduced to avoid incorrectly increasing the depth values ​​of too many background areas to the foreground, resulting in obvious outward expansion artifacts around foreground objects. The correspondence between the search radius and video sharpness can be determined by a preset threshold. For example, when the average gradient or Laplacian variance of the video is below a certain value, it is considered a blurry video, and a larger search radius is used.

[0130] In some embodiments, the search radius can be set to a uniform value for the same video footage, independent of variations across frames within the video. In practical applications, unless video clips of different resolutions are intentionally spliced ​​together into the same footage, the image clarity within the same video footage typically remains relatively stable. Therefore, a uniform neighborhood search radius parameter can be set for each video, eliminating the need to adjust the search radius frame by frame within the video. For example, a uniform search radius parameter can be set for each movie, without needing to set different neighborhood search radii for each frame within the video, thus simplifying parameter configuration and computational overhead while ensuring rendering quality.

[0131] The various embodiments or implementation methods described in this specification are presented in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other.

[0132] It is worth noting that, without contradiction, different embodiments can be arbitrarily combined. For example, some or all of the steps of different embodiments can be arbitrarily combined, and one embodiment can be arbitrarily combined with the optional implementations of other embodiments.

[0133] In some embodiments, a method for generating glasses-free 3D content is provided, such as... Figure 11 As shown, the method includes the following steps: S310: Input 2D video.

[0134] S320: Acquire RGBD video. For example, this step may include steps S110 to S140 in the foregoing embodiments.

[0135] S330: Based on RGBD video, it performs multi-view rendering and generates rendered images. For example, it can perform multi-view rendering based on depth image based rendering (DIBR).

[0136] S340: Generates an interlaced image from the rendered image using a trans-interlacing algorithm. This interlaced image can be played on glasses-free 3D devices.

[0137] This embodiment can support the production of videos from 2D to RGBD and then to naked-eye 3D videos, generating interlaced images for lenticular naked-eye 3D devices. This type of naked-eye 3D device can support multiple viewers simultaneously.

[0138] In summary, the embodiments of this application have at least the following beneficial effects: 1. A novel depth video production technique is proposed, which not only improves the stability of depth estimation for single-frame images, but also enhances the stability of depth estimation for consecutive frames. It can be used to produce high-quality depth videos with stable temporal sequences as materials for naked-eye 3D displays, effectively alleviating the problem of visual discomfort caused by inter-frame temporal jumps.

[0139] 2. Based on CUDA-OpenGL interoperability technology, a real-time naked-eye 3D video player compatible with multiple data types has been implemented. It also realizes full GPU pipeline processing from compressed RGB-D video input to naked-eye 3D interlaced image output, which reduces CPU-GPU data transfer overhead and avoids GPU memory copying as much as possible except for computation, thereby improving performance. It supports 8K high-resolution real-time rendering, and users can adjust the depth of field in real time to obtain the most comfortable viewing experience.

[0140] 3. A 3D video artifact suppression algorithm is proposed, which can adapt to low-resolution depth estimation results. It only requires storing one more low-resolution depth video than the original input video, resulting in a small storage volume. At the same time, it greatly weakens artifacts.

[0141] Figure 12 This is a schematic diagram of the structure of a depth video generation device according to one embodiment. Figure 12 As shown, the depth video generation device 100 includes: The acquisition module 101 is configured to acquire the depth map corresponding to each frame of the input video to form a depth map sequence; The identification module 102 is configured to identify depth map subsequences corresponding to the same shot in the depth map sequence; The calibration module 103 is configured to, for each of the depth map subsequences, take the first frame depth map in the depth map subsequence as the reference frame, and perform the following calibration operation on each frame following the reference frame to obtain a calibrated depth map subsequence: Obtain the weight corresponding to each pixel matching pair in the current frame depth map and the previous frame calibrated depth map, wherein the weight is negatively correlated with the depth value of the corresponding pixel matching pair in the current frame depth map and negatively correlated with the difference in depth value between the pixel matching pair in the previous frame calibrated depth map and the current frame depth map; Based on the weight of each pixel matching pair, the difference between the current frame depth map and the previous frame calibrated depth map is minimized to obtain the current frame calibrated depth map for calibration of the next frame. The generation module 104 is configured to perform temporal stitching based on each of the calibrated depth map subsequences to generate a depth video.

[0142] In some embodiments, the acquisition module 101 is configured to: Obtain the first weight of each pixel matching pair in the current frame depth map and the calibrated depth map of the previous frame. The first weight is inversely proportional to the depth value of the pixel matching pair in the current frame depth map. The second weight of each pixel matching pair in the current frame depth map and the previous frame calibrated depth map is obtained. The second weight is calculated based on a preset depth range and the depth value difference between the pixel matching pair in the previous frame calibrated depth map and the current frame depth map. The larger the depth value difference, the smaller the second weight. For each pixel matching pair, the first weight and the second weight of the pixel matching pair are multiplied to obtain the weight of the pixel matching pair.

[0143] In some embodiments, the calibration module 103 is configured to: The depth value of any pixel in the calibrated depth map of the current frame is set to satisfy a linear relationship with the depth value of the corresponding pixel in the depth map of the current frame. The linear relationship includes the affine transformation parameters to be solved. Using the weights of each pixel matching pair as coefficients, the squares of the depth deviations of each pixel matching pair are weighted and summed to construct the objective function, wherein the depth deviation is: the difference between the depth value of the pixel matching pair in the current frame depth map after transformation by the linear relationship and the depth value in the calibrated depth map of the previous frame. Find the affine transformation parameters that minimize the objective function; Using the solved affine transformation parameters, the depth value of each pixel in the current frame depth map is transformed to obtain the current frame calibrated depth map.

[0144] In some embodiments, the calibration module 103 is configured to: Based on the weights of each pixel matching pair, a weighted average is calculated on the depth values ​​in the calibrated depth map of the previous frame to obtain a first weighted average, and a weighted average is calculated on the depth values ​​in the depth map of the current frame to obtain a second weighted average. Based on the weights of each pixel matching pair, the first weighted mean, the second weighted mean, the depth values ​​of each pixel matching pair in the calibrated depth map of the previous frame, and the depth values ​​in the depth map of the current frame, the weighted variance and the weighted covariance are calculated. The ratio of the weighted covariance to the sum of the weighted variance and a preset constant is used as the scaling factor in the affine transformation parameters. The first weighted mean minus the product of the scale factor and the second weighted mean is used as the offset in the affine transformation parameters.

[0145] In some embodiments, the apparatus further includes: The preprocessing module is configured to set the depth values ​​in each frame depth map that are greater than a preset threshold as the preset threshold.

[0146] In some embodiments, the calibration module 103 is further configured to: For each frame of the depth map in the depth map subsequence, the depth value of each pixel in the depth map is non-linearly mapped, and the maximum and minimum depth values ​​of the mapped depth map are determined. The non-linear mapping is such that for any two depth intervals of equal length, the interval length after mapping of the interval composed of values ​​closer to the camera is greater than the interval length after mapping of the interval composed of values ​​farther from the camera. Based on the maximum and minimum depth values ​​of the depth map, the depth value of each pixel in the depth map is linearly scaled to a preset depth range.

[0147] In some embodiments, the depth map corresponding to each frame of the input video is obtained by performing depth estimation on the input video based on a trained depth estimation model. The device further includes: a training module; the training module is configured to: Multiple training image groups are obtained, and a label depth map corresponding to all training images in each training image group is obtained. The training image group includes an original two-dimensional image and an enhanced image obtained by modifying the texture of the original two-dimensional image. The enhanced image has the same image style and geometric structure as the original two-dimensional image. Based on each training image in the training image group and the corresponding label depth map, the initial depth estimation model is trained to obtain the trained depth estimation model.

[0148] In some embodiments, the initial depth estimation model is a pre-trained depth estimation model; the training module is configured to: The predicted depth map obtained by performing depth estimation on the original two-dimensional image using the pre-trained depth estimation model is used as the label depth map. Each training image in the training image group is used as input to retrain the pre-trained depth estimation model to obtain the trained depth estimation model.

[0149] In some embodiments, the input resolution of the initial depth estimation model is set to be smaller than the resolution of the labeled depth map; the training module is configured to: For each pixel position in the target depth map that meets the input resolution, determine the corresponding region of the pixel position in the label depth map, and use the minimum value among all depth values ​​in the corresponding region as the depth value of the pixel position to shrink the label depth map to the target depth map. The initial depth estimation model is trained based on the training image scaled down to the input resolution and the target depth map to obtain the trained depth estimation model.

[0150] In some embodiments, the device operates in the video memory of a graphics processing unit (GPU), and the GPU does not transfer data processed during the generation of the depth video between the GPU and the central processing unit.

[0151] The depth video generation apparatus provided in this application belongs to the same concept as the depth video generation method provided in the above-mentioned embodiments of this application. It can execute the depth video generation method provided in any of the above-mentioned embodiments of this application and has the corresponding functional modules and beneficial effects for executing the depth video generation method. Technical details not described in detail in this embodiment can be found in the specific processing content of the depth video generation method provided in the above-mentioned embodiments of this application, and will not be repeated here.

[0152] It should be understood that the various modules in the depth video generation device can be implemented in the form of processor calling software, or the various modules can be implemented in the form of hardware circuits. The functions of some or all modules can be realized through the design of the hardware circuits, which can be understood as one or more processors.

[0153] This application also provides an electronic device, including a processor, a memory, and an executable program stored in the memory and executable by the processor. When the processor runs the executable program, it performs the steps of the depth video generation method provided in any of the foregoing embodiments.

[0154] This application also provides a computer-readable storage medium storing an executable program thereon, which, when executed by a processor, implements the steps of the depth video generation method provided in any of the foregoing embodiments.

[0155] For ease of understanding, the following focuses on explaining the terminology used in this embodiment: In this application embodiment, the processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a Central Processing Unit (CPU), a microprocessor, a Graphics Processing Unit (GPU) (which can be understood as a type of microprocessor), or a Digital Signal Processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. The logical relationships of the aforementioned hardware circuits are fixed or reconfigurable. For example, the processor is a hardware circuit implemented using an Application-Specific Integrated Circuit (ASIC) or a Programmable Logic Device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units or modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a Neural Network Processing Unit (NPU), a Tensor Processing Unit (TPU), a Deep Learning Processing Unit (DPU), etc.

[0156] The computer-readable storage medium provided in this embodiment can execute the depth video generation method of the above embodiments. Its implementation principle and technical effects are similar to those of the above embodiments, and will not be repeated here.

[0157] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0158] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0159] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0160] The various embodiments or implementation methods described in this specification are presented in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other.

[0161] In the description of this specification, references to "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating depth video, characterized in that, The method includes: Obtain the depth map corresponding to each frame of the input video to form a depth map sequence; Identify depth map subsequences corresponding to the same shot within the depth map sequence; For each of the depth map subsequences, using the first frame depth map in the depth map subsequence as the reference frame, the following calibration operation is performed on each frame following the reference frame to obtain the calibrated depth map subsequence: Obtain the weight corresponding to each pixel matching pair in the current frame depth map and the previous frame calibrated depth map, wherein the weight is negatively correlated with the depth value of the corresponding pixel matching pair in the current frame depth map and negatively correlated with the difference in depth value between the pixel matching pair in the previous frame calibrated depth map and the current frame depth map; Based on the weight of each pixel matching pair, the difference between the current frame depth map and the previous frame calibrated depth map is minimized to obtain the current frame calibrated depth map for calibration of the next frame. A depth video is generated by temporally stitching together the calibrated depth map subsequences.

2. The depth video generation method according to claim 1, characterized in that, The step of obtaining the weights corresponding to each pixel matching pair in the current frame depth map and the previous frame calibrated depth map includes: Obtain the first weight of each pixel matching pair in the current frame depth map and the calibrated depth map of the previous frame. The first weight is inversely proportional to the depth value of the pixel matching pair in the current frame depth map. The second weight of each pixel matching pair in the current frame depth map and the previous frame calibrated depth map is obtained. The second weight is calculated based on a preset depth range and the depth value difference between the pixel matching pair in the previous frame calibrated depth map and the current frame depth map. The larger the depth value difference, the smaller the second weight. For each pixel matching pair, the first weight and the second weight of the pixel matching pair are multiplied to obtain the weight of the pixel matching pair.

3. The depth video generation method according to claim 1, characterized in that, The step of minimizing the difference between the current frame depth map and the previous frame calibrated depth map based on the weight of each pixel matching pair to obtain the current frame calibrated depth map for the next frame calibration includes: The depth value of any pixel in the calibrated depth map of the current frame is set to satisfy a linear relationship with the depth value of the corresponding pixel in the depth map of the current frame. The linear relationship includes the affine transformation parameters to be solved. Using the weights of each pixel matching pair as coefficients, the squares of the depth deviations of each pixel matching pair are weighted and summed to construct the objective function, wherein the depth deviation is: the difference between the depth value of the pixel matching pair in the current frame depth map after transformation by the linear relationship and the depth value in the calibrated depth map of the previous frame. Find the affine transformation parameters that minimize the objective function; Using the solved affine transformation parameters, the depth value of each pixel in the current frame depth map is transformed to obtain the current frame calibrated depth map.

4. The depth video generation method according to claim 3, characterized in that, The affine transformation parameters that minimize the objective function are obtained by solving for: Based on the weights of each pixel matching pair, a weighted average is calculated on the depth values ​​in the calibrated depth map of the previous frame to obtain a first weighted average, and a weighted average is calculated on the depth values ​​in the depth map of the current frame to obtain a second weighted average. Based on the weights of each pixel matching pair, the first weighted mean, the second weighted mean, the depth values ​​of each pixel matching pair in the calibrated depth map of the previous frame, and the depth values ​​in the depth map of the current frame, the weighted variance and the weighted covariance are calculated. The ratio of the weighted covariance to the sum of the weighted variance and a preset constant is used as the scaling factor in the affine transformation parameters. The first weighted mean minus the product of the scale factor and the second weighted mean is used as the offset in the affine transformation parameters.

5. The depth video generation method according to claim 1, characterized in that, Prior to performing the calibration operation, the method further includes: Set the depth values ​​in each frame's depth map that are greater than a preset threshold to the preset threshold.

6. The depth video generation method according to claim 1, characterized in that, Before the step of obtaining the weights corresponding to each pixel matching pair in the current frame depth map and the calibrated depth map of the previous frame, the method further includes: For each frame of the depth map in the depth map subsequence, a non-linear mapping is performed on the depth value of each pixel in the depth map, wherein the non-linear mapping is such that: for any two depth intervals of equal length, the interval length after mapping of the interval composed of values ​​that are closer to the camera is greater than the interval length after mapping of the interval composed of values ​​that are farther from the camera. Determine the maximum and minimum depth values ​​of the mapped depth map, and based on the maximum and minimum depth values ​​of the depth map, linearly scale the depth value of each pixel in the depth map to a preset depth range.

7. The depth video generation method according to any one of claims 1 to 6, characterized in that, The depth map corresponding to each frame of the input video is obtained by performing depth estimation on the input video based on the trained depth estimation model. The training process of the depth estimation model includes: Multiple training image groups are obtained, and a label depth map corresponding to all training images in each training image group is obtained. The training image group includes an original two-dimensional image and an enhanced image obtained by modifying the texture of the original two-dimensional image. The enhanced image has the same image style and geometric structure as the original two-dimensional image. Based on each training image in the training image group and the corresponding label depth map, the initial depth estimation model is trained to obtain the trained depth estimation model.

8. The depth video generation method according to claim 7, characterized in that, The initial depth estimation model is a pre-trained depth estimation model; the step of training the initial depth estimation model based on each training image in the training image group and the corresponding label depth map to obtain the trained depth estimation model includes: The predicted depth map obtained by performing depth estimation on the original two-dimensional image using the pre-trained depth estimation model is used as the label depth map. Each training image in the training image group is used as input to retrain the pre-trained depth estimation model to obtain the trained depth estimation model.

9. The depth video generation method according to claim 7, characterized in that, The initial depth estimation model has an input resolution that is smaller than the resolution of the label depth map; the method further includes: For each pixel position in the target depth map that meets the input resolution, determine the corresponding region of the pixel position in the label depth map, and use the minimum value among all depth values ​​in the corresponding region as the depth value of the pixel position to shrink the label depth map to the target depth map. The step of training an initial depth estimation model based on each training image in the training image group and the corresponding label depth map to obtain a trained depth estimation model includes: The initial depth estimation model is trained based on the training image scaled down to the input resolution and the target depth map to obtain the trained depth estimation model.

10. The depth video generation method according to any one of claims 1 to 6, characterized in that, The method is executed in the video memory of the graphics processing unit (GPU), and the data processed during the generation of the depth video is not transferred between the GPU and the central processing unit.

11. A depth video generation device, characterized in that, The device includes: The acquisition module is configured to acquire the depth map corresponding to each frame of the input video to form a depth map sequence; The identification module is configured to identify depth map subsequences corresponding to the same shot in the depth map sequence; The calibration module is configured to, for each of the depth map subsequences, use the first frame depth map in the depth map subsequence as the reference frame, and perform the following calibration operation on each frame following the reference frame to obtain a calibrated depth map subsequence: Obtain the weight corresponding to each pixel matching pair in the current frame depth map and the previous frame calibrated depth map, wherein the weight is negatively correlated with the depth value of the corresponding pixel matching pair in the current frame depth map and negatively correlated with the difference in depth value between the pixel matching pair in the previous frame calibrated depth map and the current frame depth map; Based on the weight of each pixel matching pair, the difference between the current frame depth map and the previous frame calibrated depth map is minimized to obtain the current frame calibrated depth map for calibration of the next frame. The generation module is configured to perform temporal stitching based on each of the calibrated depth map subsequences to generate a depth video.

12. An electronic device, characterized in that, It includes a processor, a memory, and an executable program stored in the memory and capable of being run by the processor, wherein the processor, when running the executable program, performs the steps of the depth video generation method as described in any one of claims 1 to 10.

13. A storage medium, characterized in that, It stores an executable program that, when executed by a processor, implements the steps of the depth video generation method as described in any one of claims 1 to 10.

Images