A method for decoding multiple dynamic video streams

Abstract

This invention relates to the field of computer technology and discloses a method for decoding multiple dynamic video streams. The method includes: receiving multiple compressed video streams with camera pose parameters; determining a minimum necessary viewpoint subset based on the user's current viewpoint; decoding only this minimum necessary viewpoint subset and simultaneously parsing the embedded depth map; constructing a locally dense point cloud or depth-guided light field based on the decoded frames; and directly rendering the output image according to the target viewpoint. This invention reduces terminal load and bandwidth requirements through selective decoding and local light field synthesis, achieving efficient and smooth free-viewpoint interaction while ensuring synthesis quality.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a method for decoding multi-channel dynamic video streams. Background Technology

[0002] With the rapid development of virtual reality, free-viewpoint video, and immersive interactive applications, efficient processing of multiple dynamic video streams has become a key technological foundation for supporting high-quality 3D visual experiences. Traditional video systems typically treat multi-view acquired video as independent two-dimensional image sequences, encoding, transmitting, and decoding them separately, and then generating the desired viewpoint image on the client side using view composition or interpolation algorithms. While this paradigm is structurally simple and intuitive, its essence remains limited to planar video processing logic, failing to fully utilize the geometric and photometric consistency between multi-view data. This results in a large amount of redundant information being repeatedly transmitted, increasing bandwidth overhead and the computational burden on the terminal.

[0003] Multi-stream video processing for free-viewpoint interaction heavily relies on high-density viewpoint coverage and low-latency response capabilities. Existing solutions generally employ a full-viewpoint decoding strategy, meaning that regardless of the user's current viewing direction, complete video frames from all camera perspectives must be decoded simultaneously before being handed over to the rendering engine for 3D reconstruction or viewpoint switching. This two-stage "decode first, then synthesize" process not only results in significant resource waste but also struggles to meet the stringent frame rate and energy efficiency requirements of real-time interactive scenarios. Especially in mobile devices or edge computing environments, continuously decoding multiple high-definition video streams can easily trigger memory bottlenecks and power consumption spikes, hindering the large-scale deployment of free-viewpoint technology.

[0004] Existing technologies have the following problems in integrating neural rendering and video encoding and decoding: traditional video coding standards focus on pixel-domain compression efficiency and lack the ability to explicitly model the structure of three-dimensional scenes; although the emerging Neural Radiation Field (NeRF) can achieve high-quality synthesis of new perspectives, its training process usually depends on the original image input and has high computational cost, making it difficult to directly embed into real-time video decoding pipelines.

[0005] NeRF reconstruction in dynamic scenes needs to handle complex factors such as temporal consistency, motion modeling, and lighting changes. However, existing decoder architectures do not have dedicated data paths and parameter update mechanisms designed for these requirements. Therefore, while ensuring smooth interaction and visual fidelity, there is an urgent need for a novel "decoding-as-reconstruction" decoding method that can directly map multiple compressed video streams into renderable 3D representations to overcome the fundamental limitations of traditional paradigms in terms of data efficiency, computational load, and functional integration. Summary of the Invention

[0006] This invention provides a method for decoding multiple dynamic video streams, aiming to solve the technical problems of excessive data transmission load, excessive computing resource consumption, and difficulty in guaranteeing real-time performance caused by the need to simultaneously decode all independent video streams from different viewpoints in free-viewpoint interactive scenarios. Existing technologies typically involve fully decoding the video streams from all camera viewpoints, and then using 3D reconstruction or image interpolation algorithms on the client side to synthesize the image from the user-specified viewpoint. This process not only requires extremely high network bandwidth to transmit redundant data but also incurs huge decoding and rendering overhead on terminal devices, making it difficult to achieve a smooth interactive experience, especially on mobile devices or low-power embedded platforms.

[0007] This invention provides a method for decoding multiple dynamic video streams, comprising: Receive a set of compressed video streams generated from multiple synchronously acquired viewpoints. Each video stream in the set of compressed video streams contains intra-frame coded frames and inter-frame predicted frames, and is supplemented with camera pose parameters aligned with a three-dimensional spatial coordinate system. Based on the user's current interactive viewpoint direction and field of view range, a minimum necessary viewpoint subset is determined. This minimum necessary viewpoint subset consists of several basic viewpoints that can cover the user's current visible area and meet the texture synthesis quality threshold. Decoding is performed only on the video streams contained in the minimum necessary viewpoint subset; the remaining unselected video streams remain compressed and are not decoded. Based on the image frames in the decoded minimum necessary view subset and their corresponding camera pose parameters, construct a light field representation guided by a local dense point cloud or depth map; Based on the target viewpoint parameters specified by the user, the target viewpoint image is directly rendered and output from the local light field representation.

[0008] Preferably, based on the user's current interaction viewpoint direction and field of view range, a minimum necessary viewpoint subset is determined, including: Obtain the center line-of-sight vector, horizontal field of view, and vertical field of view of the user's current interactive viewpoint; Projecting the optical axis direction of each camera from each acquisition angle onto a unit sphere forms the viewpoint distribution topology; Calculate the neighboring region of the user's gaze vector in the topology of the view distribution, and filter out candidate viewpoints that are located in the neighboring region and whose view frustum overlaps with the user's view frustum to a preset overlap threshold. The candidate viewpoints are sorted in ascending order of their angle with the user's line of sight, and then added to the viewpoint subset in turn until the pixel integrity of the composite area covered by the viewpoint subset under the target viewpoint reaches a preset integrity threshold. At this point, the filtering stops, and the resulting set is the minimum necessary viewpoint subset.

[0009] Preferably, each video stream in the compressed video stream set has embedded depth auxiliary information during the encoding stage; The depth auxiliary information is compressed together with the texture image through joint encoding. Its form is a single-channel depth map corresponding to each frame. The resolution of the depth map is consistent with that of the texture image. The depth value is stored in 16-bit unsigned integer format, representing the Euclidean distance from the camera optical center to the scene surface point. During the decoding process, when decoding a video stream, its corresponding depth map data is parsed synchronously, and the depth map and texture image are aligned at the pixel level to form a texture-depth frame pair with geometric information.

[0010] Preferably, based on image frames in the decoded minimum necessary view subset and their corresponding camera pose parameters, a light field representation guided by a local dense point cloud or depth map is constructed, including: For each decoded texture-depth frame pair, backproject it from its respective camera coordinate system to the global world coordinate system to generate a 3D spatial point cloud. The color attribute of each point is assigned by the red, green and blue components of the corresponding pixel, and the spatial coordinates are calculated by the depth value together with the camera intrinsic and extrinsic parameter matrices. Voxelization fusion is performed on point clouds generated from multiple perspectives. The voxel resolution is set to 5 cm, and the point with the highest consistency between the color mean and the normal vector is retained within each voxel. If the point cloud density is less than 0.5 points per cubic centimeter, then the edge-preserving interpolation algorithm based on the depth map gradient is enabled to fill holes between adjacent viewpoints.

[0011] Preferably, voxelization fusion is performed on point clouds generated from multiple viewpoints, including: The three-dimensional space is divided into cubic grids with a side length of 5 cm as voxel units; Within each voxel cell, calculate the average color of all points falling into that voxel, and obtain the normal vector of each point by fitting a plane through neighborhood points; The point with the most stable color mean among the points whose normal vector is greater than 0.9 and whose angle cosine between the normal vector and the average normal vector of the neighborhood is retained is taken as the representative point of the voxel.

[0012] Preferably, an edge-preserving interpolation algorithm based on depth map gradients is used for hole filling, including: Calculate the gradient magnitude of the depth map at the corresponding pixel position in adjacent viewpoints; Using the inverse of the gradient magnitude as the interpolation weight, linear interpolation is preferentially performed along the direction of gentle depth change; During interpolation, color propagation is constrained to be less than the object boundary to avoid texture aliasing.

[0013] Preferably, the target viewpoint image is directly rendered and output from the local light field representation based on the target viewpoint parameters specified by the user, including: A virtual imaging plane is established based on the camera pose parameters from the target's perspective; Perform forward mapping on the local dense point cloud, projecting each 3D point onto the virtual imaging plane to generate a preliminary rendered image; For pixel areas in the initial rendered image that are not covered by projection, a multi-view texture blending strategy based on view weight is used to fill them. The view weight is determined by the cosine of the angle between the source view and the target view and the depth continuity. Outputs a complete target view image with anti-aliasing and color consistency correction.

[0014] Preferably, the depth continuity is measured by calculating the variance of the depth gradient in the neighborhood of the target pixel. The smaller the variance, the better the depth continuity, and the higher the corresponding viewpoint weight.

[0015] Preferably, the anti-aliasing process employs multi-sampling technology, and the color consistency correction is achieved through global white balance and gamma correction.

[0016] Preferably, a dynamic update mechanism is also included: During the continuous change of the user's viewpoint, the rate and direction of viewpoint change are monitored in real time; if the rate of viewpoint change exceeds a preset threshold, key frames in the adjacent potential viewpoint subset are pre-decoded and cached in memory. When the user's view enters a new area, it immediately switches to pre-decoded data to avoid screen stuttering caused by decoding delay; the key frame of the pre-decoded data is an intra-frame encoded frame, whose decoding does not depend on other frames and can be used independently.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention introduces a minimum necessary view subset selection mechanism based on user interaction perspective, which fundamentally avoids full decoding of all multi-channel video streams, thereby reducing the amount of data decoding and memory usage.

[0018] 2. By combining depth-assisted information with local light field construction technology, any intermediate viewpoint can be synthesized with high quality using only a small number of decoding viewpoints, effectively solving the computational redundancy problem caused by full-view decoding in traditional solutions.

[0019] 3. The dynamic pre-decoding mechanism further improves the smoothness of interaction and can maintain low latency response even in high-speed perspective switching scenarios. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention; Figure 2This is a schematic diagram of the core principle framework of the minimum necessary perspective subset selection mechanism based on user interaction perspective in this invention. Figure 3 This is a flowchart illustrating the logical process of receiving and selectively decoding multi-channel compressed video streams in this invention. Figure 4 This is a flowchart illustrating the logical flow framework for constructing a locally dense point cloud and generating a depth map-guided light field representation in this invention. Figure 5 This is a flowchart illustrating the logical flow of target view image rendering and hole filling in this invention. Figure 6 This is a schematic diagram of the multi-level interaction relationship and data flow between the terminal and the multi-view video source in this invention. Detailed Implementation

[0021] Example 1: Reference Figures 1 to 6 This invention provides a method for decoding multiple dynamic video streams. Its core lies in dynamically selecting a minimum necessary viewpoint subset based on the user's current interactive viewpoint, and performing decoding operations only on the video streams within this minimum necessary viewpoint subset. This significantly reduces the computational load and network bandwidth requirements of the terminal device while ensuring the quality of free-viewpoint synthesis. The following will describe in detail the specific implementation process of the dynamic video stream decoding method.

[0022] The method first performs step S1: receiving a set of compressed video streams generated from multiple synchronously acquired viewpoints. Each video stream in the set contains intra-coded frames and inter-predicted frames, and is appended with camera pose parameters aligned to a three-dimensional spatial coordinate system. All video streams have undergone time synchronization processing at the encoding end to ensure strict temporal consistency of the images captured by each viewpoint at the same physical moment.

[0023] Camera pose parameters are represented by rotational components in quaternion form and translational components in 3D vector form. Together, they form a complete extrinsic parameter matrix used for subsequent coordinate system transformation. The video stream is compressed using efficient video coding standards such as H.265 or AV1 to balance compression efficiency and decoding compatibility.

[0024] Depth auxiliary information is embedded in each video stream. This depth auxiliary information is compressed together with the texture image through joint coding. It is in the form of a single-channel depth map corresponding to each frame. The resolution of the depth map is consistent with that of the texture image. The depth value is stored in 16-bit unsigned integer format, representing the Euclidean distance from the camera optical center to a point on the scene surface.

[0025] The depth map and texture image are kept pixel-level aligned during the encoding process to ensure that texture-depth frame pairs with geometric information can be directly formed after decoding.

[0026] Then, step S2 is executed: Based on the user's current interactive viewpoint direction and field of view range, a minimum necessary viewpoint subset is determined. This minimum necessary viewpoint subset consists of several basic viewpoints that can cover the user's current visible area and meet the texture synthesis quality threshold.

[0027] Specifically, the system first obtains the center gaze vector, horizontal field of view, and vertical field of view of the user's current interactive perspective. The center gaze vector is calculated in real time from the interaction signals fed back by the user's input device, such as a mouse, gyroscope, or touchscreen. Its starting point is located at the virtual observation point, and its ending point points in the user's current gaze direction. The horizontal and vertical field of view are jointly determined by the physical resolution of the display device and the scaling ratio set by the user, with typical values ​​ranging from 60 to 120 degrees horizontally and 40 to 90 degrees vertically.

[0028] Next, the optical axis directions of the cameras at each acquisition angle are projected onto a unit sphere, forming a viewpoint distribution topology. This viewpoint distribution topology is based on a spherical coordinate system, with each acquisition angle corresponding to a point on the sphere, whose latitude and longitude are uniquely determined by the rotation component in the camera extrinsic parameter matrix.

[0029] Subsequently, the neighboring region of the user's gaze vector in the viewpoint distribution topology is calculated. This neighboring region is defined as a spherical circular area centered on the endpoint of the user's gaze vector with a radius equal to the maximum permissible viewpoint offset angle. The maximum permissible viewpoint offset angle is dynamically adjusted according to a preset synthesis quality threshold, typically 30 degrees. Within this neighboring region, candidate viewpoints whose overlap between the user's viewpoint and the viewpoint frustum is greater than a preset overlap threshold are selected.

[0030] The overlap of the view frustums is obtained by calculating the ratio of the intersection volume to the union volume of the two view frustums in the world coordinate system, with a preset overlap threshold of 40%. The candidate viewpoints are sorted in ascending order of their angle with the user's line of sight, and are added to the viewpoint subsets in turn, until the pixel integrity of the composite region covered by the viewpoint subset reaches the preset integrity threshold under the target viewpoint.

[0031] Pixel integrity is defined as the proportion of the total number of pixels on the target view imaging plane that are effectively covered by at least one source view. The preset integrity threshold is set to 95%. At this point, the filtering stops, and the resulting set is the minimum necessary view subset.

[0032] Next, step S3 is executed: decoding is performed only on the video streams contained in the minimum necessary viewpoint subset; the remaining unselected video streams remain compressed and are not decoded. The decoding operation is performed by a hardware-accelerated decoder or a software decoding library, supporting multi-threaded parallel processing to improve efficiency. During decoding, the corresponding depth map data is parsed synchronously, and the depth map and texture image are aligned at the pixel level to form texture-depth frame pairs with geometric information.

[0033] Depth map parsing is achieved by reading the reserved auxiliary information channels in the video stream. These channels have already embedded the depth map as an additional plane into the main video stream during the encoding stage. Alignment between the texture image and the depth map is accomplished by verifying the consistency of their timestamps and frame numbers. If there is a slight deviation, linear interpolation is used for compensation to ensure strict synchronization. The decoded texture-depth frame pairs are temporarily stored in a cache for subsequent use by the light field construction module.

[0034] Then, step S4 is performed: Based on the image frames in the decoded minimum necessary view subset and their corresponding camera pose parameters, a locally dense point cloud or depth map-guided light field representation is constructed. This step specifically includes: for each decoded texture-depth frame pair, backprojecting it from its respective camera coordinate system to the global world coordinate system to generate a 3D spatial point cloud. The backprojection process is based on the following formula: ; Represents the coordinates of a three-dimensional point in the world coordinate system. These are the pixel coordinates (homogeneous form) of the image plane. For the corresponding depth value, For the camera intrinsic parameter matrix, and These are the rotation matrix and translation vector in the camera's extrinsic parameter matrix, respectively. The color attribute of each point is assigned by the red, green, and blue components of the corresponding pixel, and the spatial coordinates are calculated using the formulas described above.

[0035] Point clouds generated from multiple perspectives are fused using voxelization, with a voxel resolution of 5 cm, meaning the 3D space is divided into cubic meshes with sides of 5 cm. Within each voxel, the point with the highest consistency between the mean color and the normal vector is retained. The normal vector is calculated by fitting a plane to the neighborhood points, and consistency is measured by the cosine of the angle between the normal vector of that point and the average normal vector of the neighborhood; points with a cosine value greater than 0.9 are considered to have high consistency.

[0036] If the point cloud density is less than 0.5 points per cubic centimeter, an edge-preserving interpolation algorithm based on depth map gradients is used to fill holes between adjacent viewpoints. This edge-preserving interpolation algorithm uses the depth map gradient magnitude as weight, prioritizing interpolation along directions with gentle depth changes to avoid texture aliasing caused by crossing object boundaries. The resulting point cloud constitutes a locally dense point cloud, serving as the basis for the depth map-guided light field representation.

[0037] Finally, step S5 is executed: based on the user-specified target viewpoint parameters, the target viewpoint image is directly rendered and output from the local light field representation, completing the free-viewpoint interactive display. This step specifically includes: establishing a virtual imaging plane based on the camera pose parameters of the target viewpoint. The resolution of the virtual imaging plane is consistent with the display device, and its position and orientation are uniquely determined by the extrinsic parameter matrix of the target viewpoint. Forward mapping is performed on the local dense point cloud, projecting each 3D point onto the virtual imaging plane to generate a preliminary rendered image. The projection process uses a standard perspective transformation, with the following formula: ; The pixel coordinates are from the target's perspective. The intrinsic and extrinsic parameters are the target viewpoint's intrinsic and extrinsic parameters. Pixel regions in the initial rendered image not covered by projection are called "holes" and need to be filled. The filling adopts a multi-view texture blending strategy based on viewpoint weights, whereby the viewpoint weights are determined by the cosine of the angle between the source and target viewpoints and the depth continuity.

[0038] A larger cosine value of the angle indicates a closer viewing angle, and thus a higher weight. Depth continuity is measured by calculating the variance of the depth gradient within the neighborhood of the target pixel; a smaller variance indicates better continuity, and thus a higher weight. The final output is a complete target view image after anti-aliasing and color consistency correction. Anti-aliasing employs multi-sampling technology, and color consistency correction is achieved through global white balance and gamma correction to ensure a unified color tone in the composite image from different viewing angles.

[0039] Furthermore, the method also includes a dynamic update mechanism, corresponding to step S6: During the continuous change of the user's viewpoint, the rate and direction of viewpoint change are monitored in real time. The rate of viewpoint change is obtained by calculating the angle between the user's gaze vectors of two consecutive frames and dividing by the time interval, in degrees per second. If the rate of viewpoint change exceeds a preset threshold, typically 60 degrees per second, keyframes in adjacent potential viewpoint subsets are pre-decoded and cached in memory.

[0040] The adjacent potential view subset is defined as the contiguous region of the current minimum necessary view subset on the view distribution topology, and its range is predicted jointly by the view change direction and rate. The pre-decoded keyframes are intra-coded frames, whose decoding does not depend on other frames and can be used independently. When the user's view enters a new region, the system immediately switches to pre-decoded data to avoid screen stuttering caused by decoding delays. This dynamic update mechanism effectively improves the smoothness of response in high-speed interactive scenarios.

[0041] At the system level, this invention can also be implemented using dedicated hardware or a general-purpose computing platform. The multi-stream video receiving module is responsible for loading a set of compressed video streams from a network interface or local storage and verifying their time synchronization and pose parameter integrity. The minimum necessary viewpoint subset determination module executes all the logic of the aforementioned S2 step; its implementation can be based on a spherical geometry computation library and a spatial indexing structure such as an octree to accelerate the retrieval of neighboring viewpoints.

[0042] The selective decoding module calls the underlying decoding engine to decode only the selected video stream and simultaneously extracts the depth map. The local lighting field construction module performs point cloud generation, voxel fusion, and hole filling; its computationally intensive parts can be accelerated by the graphics processor. The target viewpoint rendering module completes forward mapping and texture blending, outputting the final image to the display buffer. The dynamic pre-decoding module monitors user interaction signals, triggers pre-decoding tasks, and manages the buffer queue to ensure low-latency switching.

[0043] The entire methodology is highly feasible on resource-constrained devices. By decoding only the minimum necessary subset of viewpoints, the number of decoding paths is reduced from a full 12 paths to 3-4 paths in typical scenarios, reducing decoding computation by over 60%. Network transmission only requires fetching selected video streams on demand, reducing bandwidth usage by 50%. Local light field representation avoids full-scene 3D reconstruction, keeping memory usage within a reasonable range. The dynamic pre-decoding mechanism further optimizes the user experience, maintaining a rendering frame rate of over 30 frames per second even during rapid viewpoint scanning.

[0044] In summary, this invention constructs a complete multi-channel dynamic video stream decoding solution through precise viewpoint selection, efficient decoding strategy, robust light field construction, and intelligent dynamic preloading mechanism. It solves the core problems of computational redundancy and bandwidth waste in traditional free-viewpoint video systems, and provides a practical technical path for immersive interactive applications on mobile terminals and embedded platforms.

[0045] Example 2: This example is applied to a digital twin video fusion system for detention centers. It requires the fusion of more than 80 high-definition surveillance video streams into a three-dimensional digital twin model of the detention center to achieve a panoramic and interactive monitoring perspective.

[0046] S1: Receives a collection of compressed video streams generated by 80 or more simultaneously deployed high-definition cameras within the detention center. Each video stream contains intra-coded frames and inter-predicted frames, and the additional camera pose parameters must be strictly aligned with the global coordinate system of the detention center's digital twin model. The pose parameters use quaternions to represent rotational components and three-dimensional vectors to represent translational components, while also labeling the areas to which the cameras belong, such as entrances / exits, cell passageways, and exercise yards. Each video stream embeds a 16-bit single-channel depth map with the same resolution as the texture image; the depth value represents the Euclidean distance from the camera to the surface of the cell scene.

[0047] S2: Determine the minimum necessary viewpoint subset. Obtain the center line-of-sight vector and field of view range of the current interactive viewpoint of the monitoring personnel. The center line-of-sight vector is preferentially associated with key monitoring areas in the digital twin model. Project the optical axis directions of each camera onto a unit sphere to form a viewpoint distribution topology, and filter out candidate viewpoints where the overlap between the view frustum and the user's view frustum is greater than 45%. Sort the candidate viewpoints according to the angle between them and the user's line of sight, and add them to the subset in turn until the pixel integrity reaches 96%. The resulting subset is the minimum necessary viewpoint subset, and the typical number of channels can be reduced from 80 to 5-8.

[0048] S3: Decoding is performed only on video streams within the minimum necessary viewpoint subset, while simultaneously parsing depth map data to form pixel-aligned texture-depth frame pairs. Unselected video streams remain compressed, reducing the decoding load on the terminal.

[0049] S4: Construct a locally dense point cloud and a depth map-guided light field representation. The texture-depth frame pairs are back-projected from the camera coordinate system to the global coordinate system of the detention center's digital twin model to generate a point cloud, which is then fused using voxel units with 5 cm sides. Points with a cosine similarity of normal vector within the voxel are retained if the point cloud density is less than 0.5 points / cm³. If the point cloud density is less than 0.5 points / cm³, a depth map gradient edge-preserving interpolation algorithm is used to fill holes, ensuring the integrity of critical structures such as prison fences, doors, and windows.

[0050] S5: Render the target viewpoint image. A virtual imaging plane is established based on the virtual observation pose of the digital twin model, and the local point cloud is projected to generate a preliminary rendered image. Holes are filled using a multi-view texture blending strategy based on viewpoint weights, where the weights are determined by the cosine of the viewpoint angle and depth continuity. Anti-aliasing and color consistency correction are performed on the rendered image, outputting a monitoring screen that precisely matches the digital twin model.

[0051] S6: Enable dynamic pre-decoding mechanism. Real-time monitoring of the viewing angle switching rate of the monitoring personnel. If the rate exceeds 60° / s, pre-decode and cache intra-coded frames of adjacent viewing angle subsets of key areas. When the user switches to a new monitoring area, the cached data is directly invoked to avoid screen stuttering.

[0052] Example 3: This example is applied to a multi-channel video fusion system in a prison cell. It addresses the characteristics of small space, multiple obstructions, and dense viewing angles in prison cells, and achieves high-performance decoding and synthesis of monitoring images without blind spots.

[0053] S1: Receives a collection of synchronized compressed video streams generated by 6-10 high-definition cameras deployed within the cell. The cameras are installed in locations such as corners and ceilings within the cell, and the pose parameters attached to the video streams indicate the specific installation location and orientation. Each video stream embeds a single-channel depth map, with depth values ​​adapted to the ranging needs of personnel, beds, tables, chairs, and other objects within the cell.

[0054] S2: Determine the minimum necessary viewpoint subset. Obtain the center line-of-sight vector of the current monitoring viewpoint and focus on the area of ​​personnel activity within the cell. Project the camera's optical axis onto a unit sphere to construct the viewpoint topology. Due to the limited space in the cell, increase the frustum overlap threshold to 50%. Add the subsets according to the angle between the viewpoint and the user's line of sight until pixel integrity reaches 98%. The typical number of channels can be reduced from 10 to 3-4.

[0055] S3: Decodes only the video stream within the minimum necessary viewpoint subset, simultaneously parsing the depth map to form texture-depth frame pairs. For localized depth map gaps caused by occlusion within the cell, preliminary completion is performed during the decoding stage.

[0056] S4: Construct a local light field representation. Backproject the texture-depth frame pair to generate a point cloud, and fuse it using voxel units with a side length of 3 cm to improve the detail reproduction of small objects within the cell. If the point cloud density is insufficient, enable an edge-preserving interpolation algorithm, prioritizing interpolation along the edges of walls, beds, and other gently sloping depth directions to avoid texture aliasing.

[0057] S5: Render the target viewpoint image. Establish a virtual imaging plane adapted to the prison space and project the point cloud to generate a preliminary image. Fill the voids caused by occlusion of personnel bodies using multi-view texture blending. Enhance the clarity of personnel movement details in the output image to meet the needs of prison behavior analysis.

[0058] S6: Enable dynamic pre-decoding mechanism. Monitor the perspective changes caused by personnel movement within the cell. If the perspective change rate is greater than 50° / s, pre-decode keyframes from adjacent perspectives. When personnel move to a new area within the cell, quickly switch to pre-decoded data to ensure the continuity of the monitoring footage.

[0059] Example 4: Scenarios of multi-channel video surveillance system access in rail transit, schools, and water conservancy projects. This embodiment is applied to multi-channel video surveillance systems covering a wide range of scenarios, such as rail transit, schools, and water conservancy, and supports mixed access of different types of cameras and cross-scenario monitoring perspective synthesis.

[0060] S1: Receives a collection of synchronized compressed video streams generated by various types of cameras, including fixed platform cameras and moving track cameras in rail transit, panoramic cameras for school playgrounds and fixed-point cameras in corridors, and monitoring cameras for dams and sluice gates in water conservancy projects. It performs unified conversion on video streams with different encoding formats, and adds pose parameters adapted to the global coordinate system of each scene. Depth map embedding needs to balance long-distance ranging in large-scale scenes with detailed ranging requirements in small areas.

[0061] S2: Determine the minimum necessary view subset. Differentiated filtering rules are set for different sub-scenes. For rail transit scenes, views covering track sections and platform edges are prioritized, with a view frustum overlap threshold of 40%. For school scenes, views in densely populated areas such as playgrounds and school gates are prioritized, with a view frustum overlap threshold of 42%. For water conservancy scenes, views at dam crack monitoring points and sluice gate opening / closing areas are prioritized, with a view frustum overlap threshold of 38%. For all scenes, the pixel integrity threshold is set to 95%, and the number of decoding paths is reduced to 5%-10% of the total number of paths based on the scene size.

[0062] S3: Decodes the video stream within the minimum necessary viewpoint subset and simultaneously parses the depth map. Supports multi-threaded parallel decoding, adapting to the computing power of different terminals, such as high-performance servers in monitoring centers and lightweight processors in mobile inspection terminals.

[0063] S4: Construct a layered local light field representation. For large-scale scenes, a layered voxel fusion strategy is adopted, dividing the scene into a distant, mid-range, and near-range layer, with different voxel resolutions for each layer. The voxel resolution for the distant layer is set to 10 cm, for the mid-range layer to 5 cm, and for the near-range layer to 3 cm. A hole-filling algorithm is enabled as needed to ensure the integrity of the scene at different levels.

[0064] S5: Renders target view images adapted to multiple terminals. It adjusts virtual imaging plane parameters according to the resolution of different devices such as monitoring center screens and mobile inspection terminals. For rail transit scenes, it enhances the clarity of train tracks; for school scenes, it enhances the visibility of areas with large crowds; and for water conservancy scenes, it enhances the contrast of dam deformation areas. Output images support simultaneous display on multiple terminals.

[0065] S6: Enables a dynamic pre-decoding mechanism. For the trajectories of mobile cameras in rail transit, the movement routes of people in schools, and key inspection paths in water conservancy monitoring, potential viewpoint subsets are predicted. If the viewpoint change rate is greater than 65° / s, keyframes of the potential viewpoint are pre-decoded and cached. When the monitoring viewpoint switches along the predicted path, the cached data is quickly retrieved, achieving low-latency cross-scene monitoring.

[0066] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0067] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for decoding multiple dynamic video streams, characterized in that, include: Receive a set of compressed video streams generated from multiple synchronously acquired viewpoints. Each video stream in the set of compressed video streams contains intra-frame coded frames and inter-frame predicted frames, and is supplemented with camera pose parameters aligned with a three-dimensional spatial coordinate system. Based on the user's current interactive viewpoint direction and field of view range, a minimum necessary viewpoint subset is determined. This minimum necessary viewpoint subset consists of several basic viewpoints that can cover the user's current visible area and meet the texture synthesis quality threshold. Decoding is performed only on the video streams contained in the minimum necessary viewpoint subset; the remaining unselected video streams remain compressed and are not decoded. Based on the image frames in the decoded minimum necessary view subset and their corresponding camera pose parameters, construct a light field representation guided by a local dense point cloud or depth map; Based on the target viewpoint parameters specified by the user, the target viewpoint image is directly rendered and output from the local light field representation.

2. The method for decoding multiple dynamic video streams according to claim 1, characterized in that, Based on the user's current interactive viewpoint direction and field of view range, determine a minimum necessary subset of viewpoints, including: Obtain the center line-of-sight vector, horizontal field of view, and vertical field of view of the user's current interactive viewpoint; Projecting the optical axis direction of each camera from each acquisition angle onto a unit sphere forms the viewpoint distribution topology; Calculate the neighboring region of the user's gaze vector in the topology of the view distribution, and filter out candidate viewpoints that are located in the neighboring region and whose view frustum overlaps with the user's view frustum to a preset overlap threshold. The candidate viewpoints are sorted in ascending order of their angle with the user's line of sight, and then added to the viewpoint subset in turn until the pixel integrity of the composite area covered by the viewpoint subset under the target viewpoint reaches a preset integrity threshold. At this point, the filtering stops, and the resulting set is the minimum necessary viewpoint subset.

3. The method for decoding multiple dynamic video streams according to claim 2, characterized in that, Each video stream in the compressed video stream set has had depth auxiliary information embedded during the encoding stage; The depth auxiliary information is compressed together with the texture image through joint encoding. Its form is a single-channel depth map corresponding to each frame. The resolution of the depth map is consistent with that of the texture image. The depth value is stored in 16-bit unsigned integer format, representing the Euclidean distance from the camera optical center to the scene surface point. During the decoding process, when decoding a video stream, its corresponding depth map data is parsed synchronously, and the depth map and texture image are aligned at the pixel level to form a texture-depth frame pair with geometric information.

4. The method for decoding multiple dynamic video streams according to claim 3, characterized in that, Based on image frames in the decoded minimum necessary view subset and their corresponding camera pose parameters, a light field representation guided by a local dense point cloud or depth map is constructed, including: For each decoded texture-depth frame pair, backproject it from its respective camera coordinate system to the global world coordinate system to generate a 3D spatial point cloud. The color attribute of each point is assigned by the red, green and blue components of the corresponding pixel, and the spatial coordinates are calculated by the depth value together with the camera intrinsic and extrinsic parameter matrices. Voxelization fusion is performed on point clouds generated from multiple perspectives. The voxel resolution is set to 5 cm, and the point with the highest consistency between the color mean and the normal vector is retained within each voxel. If the point cloud density is less than 0.5 points per cubic centimeter, then the edge-preserving interpolation algorithm based on the depth map gradient is enabled to fill holes between adjacent viewpoints.

5. The method for decoding multiple dynamic video streams according to claim 4, characterized in that, Voxelization fusion of point clouds generated from multiple perspectives, including: The three-dimensional space is divided into cubic grids with a side length of 5 cm as voxel units; Within each voxel cell, calculate the average color of all points falling into that voxel, and obtain the normal vector of each point by fitting a plane through neighborhood points; The point with the most stable color mean among the points whose normal vector is greater than 0.9 and whose angle cosine between the normal vector and the average normal vector of the neighborhood is retained is taken as the representative point of the voxel.

6. The method for decoding multiple dynamic video streams according to claim 5, characterized in that, Enable edge-preserving interpolation based on depth map gradients for hole filling, including: Calculate the gradient magnitude of the depth map at the corresponding pixel position in adjacent viewpoints; Using the inverse of the gradient magnitude as the interpolation weight, linear interpolation is preferentially performed along the direction of gentle depth change; During interpolation, color propagation is constrained to be less than the object boundary to avoid texture aliasing.

7. The method for decoding multiple dynamic video streams according to claim 6, characterized in that, Based on the user-specified target viewpoint parameters, the target viewpoint image is directly rendered and output from the local light field representation, including: A virtual imaging plane is established based on the camera pose parameters from the target's perspective; Perform forward mapping on the local dense point cloud, projecting each 3D point onto the virtual imaging plane to generate a preliminary rendered image; For pixel areas in the initial rendered image that are not covered by projection, a multi-view texture blending strategy based on view weight is used to fill them. The view weight is determined by the cosine of the angle between the source view and the target view and the depth continuity. Outputs a complete target view image with anti-aliasing and color consistency correction.

8. The method for decoding multiple dynamic video streams according to claim 7, characterized in that, The depth continuity is measured by calculating the variance of the depth gradient in the neighborhood of the target pixel. The smaller the variance, the better the depth continuity, and the higher the corresponding viewpoint weight.

9. The method for decoding multiple dynamic video streams according to claim 8, characterized in that, The anti-aliasing process employs multi-sampling technology, and the color consistency correction is achieved through global white balance and gamma correction.

10. The method for decoding multiple dynamic video streams according to claim 9, characterized in that, It also includes a dynamic update mechanism: During the continuous change of the user's viewpoint, the rate and direction of viewpoint change are monitored in real time; if the rate of viewpoint change exceeds a preset threshold, key frames in the adjacent potential viewpoint subset are pre-decoded and cached in memory. When the user's view enters a new area, it immediately switches to pre-decoded data to avoid screen stuttering caused by decoding delay; the key frame of the pre-decoded data is an intra-frame encoded frame, whose decoding does not depend on other frames and can be used independently.

Images