Railway station three-dimensional video generation method and system based on adaptive visual angle

Abstract

The invention discloses a railway station three-dimensional video generation method and system based on a self-adaptive visual angle, and relates to the technical field of passenger station video surveillance, and the method comprises the steps: arranging a plurality of cameras and depth sensors in a railway station, and employing a network clock synchronization mechanism to synchronously collect video streams and point cloud data, generating a real-time sensing data set with a unified timestamp; constructing a static geometric model based on the building information model of the passenger station, and fusing the static geometric model with the real-time data set to generate a dynamic scene point cloud; for a plurality of candidate virtual view angles, counting the coverage degree, the shielding rate, the view angle conversion cost and the key area weight, and calculating view angle scores through a comprehensive scoring function; and selecting the optimal virtual view angle with the highest score, and performing three-dimensional rendering on the dynamic scene point cloud based on the view angle to generate a corresponding three-dimensional video frame. Through a comprehensive scoring mechanism of a virtual view angle, adaptive monitoring coverage is realized, and the monitoring precision and the view angle flexibility are improved.

Description

Technical Field

[0001] The present application relates to the technical field of passenger station video surveillance, and in particular to a method and system for generating three-dimensional video of a railway passenger station based on an adaptive viewing angle. Background Art

[0002] Railway passenger stations are hubs in modern urban public transportation systems. With waiting, boarding, alighting, shopping, and leisure activities concentrated in a confined space, high passenger density and high mobility create significant pressure on security management and operational scheduling. As a primary technical means of ensuring security, video surveillance has been widely deployed in platforms, waiting rooms, entrances and exits, and passageways. CCTV systems provide real-time monitoring of passenger behavior, equipment operation, and emergencies.

[0003] Traditional railway station surveillance systems primarily rely on two-dimensional cameras, with security personnel monitoring the feeds from each camera on multiple screens in the monitoring room. This model features fixed camera placement, a single viewing angle, and a lack of depth information, making it difficult to accurately analyze complex scenes such as occlusion, crowding, and the presence of people. Human monitoring also presents challenges such as fatigue and missed detections. In real-world environments, evacuation guidance, abnormal behavior identification, and emergency response require real-time and comprehensive information. However, fixed-viewpoint two-dimensional video cannot effectively represent the spatial relationships of events, especially in multi-story stations and three-dimensional transportation spaces. While multi-camera systems expand coverage, they suffer from frequent image switching and a lack of a global three-dimensional perspective, making them unable to meet the goal of "full coverage, no blind spots."

[0004] In recent years, with the advancement of artificial intelligence and computer vision technologies, some research has begun exploring the deployment of intelligent cameras in railway stations to automatically identify anomalies using behavioral analysis algorithms. However, these technologies are mostly based on monocular or binocular perspectives, emphasizing the detection of pedestrians, objects, or unusual events within a single image. Their performance depends on target resolution and occlusion. In describing their multi-camera intelligent surveillance system, Ozer et al. pointed out that traditional target detection and tracking algorithms are generally suitable for low-occlusion, high-resolution scenes. Complex backgrounds and lighting variations can cause foreground extraction and tracking to fail. Their system requires hardware to process large amounts of data and cope with variations such as light flicker, shadows, and rapid exposure adjustments. This demonstrates that traditional two-dimensional surveillance faces technical bottlenecks in areas such as lighting, occlusion, and multi-view calibration.

[0005] On the other hand, multi-view 3D reconstruction technology has been widely studied in the field of computer vision. Using multiple cameras or mobile cameras to reconstruct a scene can obtain 3D point clouds and depth information. Typical multi-baseline stereo algorithms require the selection of a reference perspective and solve the depth by calculating the pixel correspondence between different perspectives. Princeton University's multi-view reconstruction tutorial points out that multi-baseline stereo is prone to occlusion problems when the number of cameras increases or the baseline is large, and occlusion will contribute error values ​​to the cost function. The use of volumetric methods such as voxel coloring can explicitly consider occlusion to a certain extent, but how to determine from which perspectives each voxel is visible and how to arrange the scanning order remain difficulties. Traditional 3D reconstruction methods are usually oriented towards static scenes and cannot meet the real-time and integrity requirements of dynamic crowd scenes in railway stations. Summary of the Invention

[0006] The present application provides a method, system, storage medium, computer program product and electronic device for generating three-dimensional videos of railway passenger stations based on adaptive perspective, which are used to at least solve the problems of fixed perspective, limited coverage, occlusion and lack of real-time performance of two-dimensional monitoring systems in current related technologies.

[0007] In the first aspect, an embodiment of the present application provides a method for generating three-dimensional videos of railway passenger stations based on adaptive perspectives, the method comprising: arranging multiple cameras and multiple depth sensors in the railway passenger station, synchronously collecting time-series video streams and point cloud data of each sensor through a network clock synchronization mechanism, and generating a real-time sensing data set with a unified timestamp; constructing a static geometric model for reflecting the layout of railway passenger station facilities based on the building information model of the railway passenger station, and fusing the static geometric model with the real-time sensing data set to generate a corresponding dynamic scene point cloud; for each candidate virtual perspective in the candidate virtual perspective set, counting the coverage, occlusion rate, and perspective transformation cost corresponding to the candidate virtual perspective and key area weights, and calling a comprehensive scoring function to calculate the corresponding perspective score; the virtual perspective is the perspective of the virtual camera; the best virtual observation perspective with the highest corresponding perspective score is selected, and the dynamic scene point cloud is three-dimensionally rendered based on the best virtual observation perspective to generate a corresponding three-dimensional video frame; wherein, the coverage represents the proportion of key targets visible under the current candidate virtual perspective, the occlusion rate represents the proportion of the target area occluded under the current candidate virtual perspective to the total area, the perspective transformation cost represents the conversion cost required to move from the perspective of the previous three-dimensional video frame to the current candidate virtual perspective, and the key area weight represents the importance of the key area covered by the current candidate frame perspective.

[0008] In a second aspect, an embodiment of the present application provides a three-dimensional video generation system for a railway passenger station based on an adaptive perspective, the system comprising: a heterogeneous sensing acquisition unit, for arranging multiple cameras and multiple depth sensors in the railway passenger station, synchronously acquiring time-series video streams and point cloud data of each sensor through a network clock synchronization mechanism, and generating a real-time sensing data set with a unified timestamp; a dynamic point cloud generation unit, for constructing a static geometric model reflecting the layout of the railway passenger station facilities based on the building information model of the railway passenger station, and fusing the static geometric model with the real-time sensing data set to generate a corresponding dynamic scene point cloud; a virtual perspective scoring unit, for counting the coverage corresponding to each candidate virtual perspective in the candidate virtual perspective set , occlusion rate, perspective transformation cost and key area weight, and call the comprehensive scoring function to calculate the corresponding perspective score; the virtual perspective is the perspective of the virtual camera; a three-dimensional video rendering unit is used to select the best virtual observation perspective with the highest corresponding perspective score, and perform three-dimensional rendering on the dynamic scene point cloud based on the best virtual observation perspective to generate a corresponding three-dimensional video frame; wherein, the coverage represents the proportion of key targets visible under the current candidate virtual perspective, the occlusion rate represents the proportion of the target area obscured under the current candidate virtual perspective to the total area, the perspective transformation cost represents the conversion cost required to move from the perspective of the previous three-dimensional video frame to the current candidate virtual perspective, and the key area weight represents the importance of the key area covered by the perspective of the current candidate frame.

[0009] In a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the method for generating three-dimensional video of a railway passenger station based on adaptive perspective of any embodiment of the present application.

[0010] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the method for generating three-dimensional video of a railway passenger station based on adaptive perspective of any embodiment of the present application are implemented.

[0011] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for generating three-dimensional video of a railway passenger station based on adaptive perspective of any embodiment of the present application.

[0012] The method and system for generating three-dimensional videos of railway passenger stations based on adaptive viewing angles provided by this application can produce at least the following technical effects: (1) By deploying multiple cameras and depth sensors at railway stations and synchronously collecting video streams and point cloud data through a network clock synchronization mechanism, the information collected by different sensors can be combined to generate a real-time sensor data set with a unified timestamp, providing a more comprehensive and three-dimensional scene perception capability than traditional two-dimensional monitoring systems. In addition, the generation of dynamic scene point clouds and the fusion of static geometric models enable the monitoring system to capture the spatial information of key targets in real time.

[0013] (2) By evaluating the candidate virtual viewpoint set, including comprehensive analysis of dimensions such as viewpoint coverage, occlusion rate, viewpoint change cost, and key area weight, the optimal virtual observation viewpoint with the highest viewpoint score is selected. In this way, multi-dimensional optimization is used to improve the accuracy of viewpoint selection, reduce occlusion to a greater extent, and more intelligently adapt to complex changes in the current scene, thereby ensuring the quality and comprehensiveness of video surveillance.

[0014] (3) By performing 3D rendering of the dynamic scene point cloud based on the optimal virtual observation perspective, a clearer and more realistic 3D surveillance video frame can be generated. Thus, the optimal perspective is selected and the transition is smooth, avoiding the image freeze and discontinuity caused by frequent camera perspective switching in traditional methods. At the same time, the consideration of perspective conversion costs makes 3D video rendering more efficient, avoiding excessive waste of computing resources and improving rendering efficiency.

[0015] This technical solution, through the introduction of a comprehensive scoring mechanism for virtual perspectives, intelligently selects the optimal perspective for virtual 3D video rendering, achieving adaptive surveillance coverage and avoiding the occlusion and blind spot issues inherent in traditional surveillance. Furthermore, the spatial understanding and real-time sensor data synchronization mechanism based on the Building Information Model further ensures the system's efficiency and real-time performance, providing more accurate and comprehensive technical support for railway station security monitoring and operational scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 A flowchart illustrating an example of a method for generating a three-dimensional video of a railway passenger station based on an adaptive viewing angle according to an embodiment of the present application is shown; Figure 2 An operational flow chart of an example of generating a dynamic scene point cloud according to an embodiment of the present application is shown; Figure 3An operational flowchart of an example of adaptive occlusion-compensated depth estimation according to an embodiment of the present application is shown; Figure 4 A schematic diagram of a simulation effect showing an example of comparison of detection accuracy between the baseline algorithm and the adaptive view algorithm within 10 time frames; Figure 5 A schematic diagram showing the simulation effect of an example comparing the occlusion rate between the baseline algorithm and the adaptive view angle algorithm within 10 time frames; Figure 6 A structural block diagram of an example of a railway passenger station three-dimensional video generation system based on adaptive viewing angle according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] It should be noted that some experts and scholars have proposed some novel directions for railway passenger video surveillance.

[0020] Specifically, in "A Train Station Surveillance System: Challenges and Solutions," Ozer et al. introduced an intelligent camera system for train stations. The system utilizes a multi-layered "intelligent camera-server" architecture, using smart cameras to detect passenger gestures in real time and send alerts to a monitoring center. The paper notes that the basic principle of a multi-camera surveillance system is for dedicated security personnel to monitor multiple images in a monitoring room, but due to limited attention spans, system effectiveness is difficult to guarantee. To improve efficiency, intelligent cameras need to automatically identify anomalies and push relevant images to operators. However, traditional object detection and tracking algorithms are generally suitable for scenes with low occlusion and high resolution, and are not suitable for station environments with complex backgrounds and unstable lighting. The paper also summarizes the main challenges encountered in deploying the intelligent camera system: 1) Hardware: processing massive amounts of video data for real-time computation; 2) Environmental: station lighting suffers from flickering, shadows, and reflections, resulting in frequent changes in camera parameters and exposure; 3) Calibration: geometric calibration of cameras is difficult due to the complex structure of platforms, stairs, and ramps; 4) Tracking: occlusion by multiple people and changes in posture can easily lead to target tracking loss, requiring specialized graph-theoretic algorithms. Therefore, it is difficult for a two-dimensional multi-camera system to fully describe the complex three-dimensional space.

[0021] In traditional 3D reconstruction technology, multi-view Figure 3 3D reconstruction reconstructs scene depth by combining image information from multiple cameras. Multi-baseline stereo algorithms calculate depth by selecting a reference view and matching pixels across views. A Princeton University textbook on multi-view reconstruction notes that occlusion issues increase significantly with increasing the number of cameras or baseline distance. Occlusions can introduce mismatches into the error function, affecting reconstruction accuracy. Volumetric methods such as voxel coloring discretize a scene into a voxel grid and determine whether each voxel belongs to an object by determining whether its color is consistent across different images. However, this method requires a predetermined voxel visibility order: all voxels occluding a voxel must be processed before calculating its color consistency, otherwise incorrect determinations can occur. Furthermore, voxel coloring relies on good lighting and texture. If the scene lacks texture or experiences drastic lighting changes, it is prone to "bumping" or loss of detail. These traditional 3D reconstruction methods are primarily designed for static or slowly changing scenes and lack support for real-time dynamic scenes. This makes them difficult to directly apply to crowded and frequently occluded environments like railway stations.

[0022] The paper "Recent Advances in Video Analytics for Rail Network Surveillance for Security, Trespass and Suicide Prevention - A Survey" points out that with the development of machine learning, many CCTV surveillance systems have integrated multiple sensors to automatically detect static objects and unusual events for public space management. These methods involve object detection, behavior recognition, and anomaly detection. However, the review points out that railway surveillance still faces significant challenges. Existing algorithms rely on large amounts of training data, and their performance fluctuates with lighting and environmental changes. Furthermore, the paper emphasizes that future railway surveillance will require the integration of diverse sensor types (electro-optical, thermal infrared, acoustic, etc.) and computing architectures to form an integrated monitoring and decision-making system. This also means that single 2D vision algorithms cannot meet security management needs, requiring cross-domain information fusion and 3D scene understanding.

[0023] It should be understood that the purpose of the above description of the current related art is only to facilitate the public to better understand the inventive spirit and motivation of this application, and is not to be construed as limiting this application. In addition, the technical solutions described in the above-mentioned current related art are not prior art and may also be undisclosed technical solutions, such as solutions under research or in the laboratory stage.

[0024] In the technical solutions of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0025] Digital twins are a newly emerging concept that uses sensor data and models to construct digital replicas of physical systems. For railway stations, digital twins must encompass a wide range of information, including station structure, equipment status, and passenger flow. However, existing digital twins often use Building Information Modeling (BIM) as a foundation to construct static scenes, unable to update the dynamic status of personnel and equipment in real time. Furthermore, the modeling process requires extensive manual data collection and point cloud post-processing, lacking the ability to rapidly generate dynamic 3D video. Furthermore, 3D scanning equipment such as LiDAR is expensive and has limited deployment in railway scenarios, making comprehensive coverage difficult to achieve. Therefore, integrating digital twins with multi-view dynamic video to generate real-time 3D surveillance footage remains a research gap.

[0026] Figure 1 A flowchart of an example of a method for generating a three-dimensional video of a railway passenger station based on an adaptive viewing angle according to an embodiment of the present application is shown.

[0027] The execution entity of the method of the present application embodiment can be any controller or processor with computing or processing capabilities, specifically a railway passenger intelligent monitoring platform. By integrating multi-sensor data, building a digital scene model, and employing a dynamic view optimization algorithm, it generates real-time 3D video of the station space, providing intuitive and effective support for security command, passenger flow analysis, and emergency response.

[0028] In some examples, it can be integrated into an electronic device or terminal through software, hardware, or a combination of software and hardware, and the type of terminal or electronic device can be diverse, such as a mobile phone, tablet computer, or desktop computer, etc.

[0029] like Figure 1 As shown, in step S110, multiple cameras and multiple depth sensors are arranged at the railway passenger station, and the time-series video streams and point cloud data of each sensor are synchronously collected through the network clock synchronization mechanism, and a real-time sensing data set with a unified timestamp is generated.

[0030] In some implementations, multiple heterogeneous sensing nodes are deployed in key areas of railway stations. Each heterogeneous sensing node contains multiple high-resolution cameras and depth sensors. Camera installation adheres to the principle of "full coverage, no blind spots," with viewing angles designed based on building structure and passenger flow paths. Depth sensors are primarily deployed in areas with dense passenger flow or severe obstruction to provide accurate depth information. During the acquisition process, each camera generates a continuous image sequence, and the depth sensor outputs a point cloud sequence. To ensure data fusion quality, the sensors require calibration and synchronization.

[0031] Sensor calibration and spatiotemporal synchronization first require calibrating the camera's internal and external parameters. This internal calibration involves calibrating the focal length, principal point, and distortion coefficient of each camera to obtain an internal parameter matrix. External calibration involves selecting significant feature points (such as corners and pillars) based on the station's static geometric model. This extrinsic calibration determines the camera's position and pose, thereby establishing the transformation between the world coordinate system and the camera coordinate system.

[0032] Furthermore, to achieve time synchronization, the system uses a unified network clock and aligns all image and point cloud data using timestamps. During the acquisition process, if the timestamp deviation of a sensor exceeds a threshold, its data can be adjusted through interpolation. This unified timestamp across time series datasets enables precise alignment of data from different sensors, avoiding visual information misalignment or spatiotemporal inconsistencies caused by data deviations.

[0033] In step S120, a static geometric model reflecting the layout of railway station facilities is constructed based on the building information model of the railway station, and the static geometric model is fused with the real-time sensor data set to generate a corresponding dynamic scene point cloud.

[0034] It should be noted that the station's Building Information Modeling (BIM) provides geometric information of each structural component. The model is exported through computer-aided design software and converted into a mesh that can be used for 3D rendering. Depth sensors are used to scan the station environment when there are no passengers, obtaining point clouds. The scanned point cloud and the BIM model are registered by the Iterative Closest Point (ICP) algorithm to optimize the model coordinate system. The resulting static geometric model is recorded as .

[0035] During the implementation process, the BIM model first provides an accurate geometric framework, which includes the three-dimensional coordinates, dimensions, relative positions and other information of each facility. On this basis, the real-time sensor data (such as video streams and point cloud data) collected synchronously are fused. Point cloud data is usually obtained by depth sensors and contains spatial position and depth information about each point in the scene. By docking with the static geometric model, the real-time collected data can be combined with the existing spatial structure to generate a dynamic scene point cloud that reflects the facility layout and personnel activities in the passenger station at the current moment. With static geometry model Fusion, building a dynamic scene point cloud of the complete scene In addition, a spatial octree is used to manage scenes in layers, significantly improving the efficiency of subsequent rendering and perspective evaluation.

[0036] In step S130, for each candidate virtual perspective in the candidate virtual perspective set, the coverage, occlusion rate, perspective conversion cost and key area weight corresponding to the candidate virtual perspective are counted, and a comprehensive scoring function is called to calculate the corresponding perspective score.

[0037] Here, the virtual perspective refers to the perspective of a virtual camera. This refers to the viewing angle and field of view defined by the virtual camera. It simulates the camera's position and orientation in three-dimensional space to select the optimal viewing angle. When generating images or videos, the virtual camera's perspective determines the rendered content.

[0038] Specifically, the coverage represents the proportion of key targets visible under the current candidate virtual perspective, the occlusion rate represents the proportion of the target area that is occluded under the current candidate virtual perspective to the total area, the perspective transformation cost represents the conversion cost required to move from the perspective of the previous three-dimensional video frame to the current candidate virtual perspective, and the key area weight represents the importance of the key area covered by the current candidate frame perspective.

[0039] Based on the evaluation of these four factors, the system calculates the overall score for each candidate virtual perspective using a comprehensive scoring function. For example, the scores of each factor can be weighted and combined to arrive at a final perspective score. This multi-dimensional virtual perspective evaluation mechanism accurately selects the optimal viewing angle, ensuring coverage of key areas and maximizing perspective effectiveness and stability. This ensures that each frame clearly captures the scene's key dynamics, avoiding the issues associated with frequent perspective switching in traditional surveillance systems.

[0040] In step S140 , the best virtual observation perspective with the highest corresponding perspective score is selected, and the dynamic scene point cloud is three-dimensionally rendered based on the best virtual observation perspective to generate a corresponding three-dimensional video frame.

[0041] Here, the system selects the optimal virtual perspective with the highest score based on the aforementioned virtual perspective evaluation. It then performs 3D rendering on the dynamic scene point cloud based on this perspective to generate the final 3D video frame. This rendering process combines the aforementioned static geometric model with real-time sensor data, taking into account factors such as scene lighting, texture, and occlusion, to produce highly realistic 3D video.

[0042] The three-dimensional rendering process can be achieved by using various depth map rendering technologies in computer graphics. For example, first, based on the selected optimal virtual perspective, the corresponding perspective matrix is ​​determined, the perspective transformation is performed, and the point cloud data is mapped to the image plane; then, the depth information in the scene is processed by ray tracing or rasterization algorithms to generate clear three-dimensional video frames. Since the perspective transformation cost is comprehensively considered in the optimal perspective calculation process, the rendering process will be combined with the rendering result of the previous frame for a smooth transition, avoiding the visual freeze problem caused by frequent perspective switching, and can quickly respond and generate efficient video frames in dynamic scenes.

[0043] Figure 2 An operational flowchart of an example of generating a dynamic scene point cloud according to an embodiment of the present application is shown.

[0044] like Figure 2 As shown, in step S210, the real-time collected depth data and the dynamic targets in the video stream are fused according to the real-time sensing data set to generate a dynamic scene depth map.

[0045] Here, information captured by different types of sensors (such as depth sensors and cameras) is processed synchronously to generate a depth map of a dynamic scene, thereby reflecting the depth information of objects, people, and spatial structures in the scene.

[0046] Specifically, a depth sensor provides the distance from each pixel to the sensor (i.e., depth information). Depth data is typically stored as a point cloud, where each point contains spatial coordinates and a corresponding depth value. Furthermore, dynamic targets in video streams are typically identified using computer vision algorithms (such as deep learning-based object detection models). These dynamic targets are extracted from the video stream and annotated or segmented to form a two-dimensional target region corresponding to the depth data.

[0047] In data fusion, depth data and dynamic target detection results in the video stream need to be fused through spatial alignment and temporal synchronization. Specifically, the data collected by the depth sensor will be mapped to each frame of the video stream, and the depth information will be matched with the image data of the dynamic target. For example, clock synchronization is used to ensure that the time of the depth data and the video frame is consistent through clock synchronization technology, and camera calibration technology is used to perform geometric calibration between cameras so that the depth image and the video image are aligned in the same space.

[0048] In some embodiments, the depth value of an image pixel may be calculated by feature matching and disparity calculation.

[0049] Specifically, select the reference camera , in its image Detection feature points On other cameras Image In the example, corresponding points are searched by depth-guided block matching. , calculate the disparity . Using the baseline length obtained by calibration and focal length , calculate the feature points according to the disparity formula Depth :

[0050] , formula (1) in, A constant to prevent division by zero.

[0051] Thus, a dynamic scene depth map containing scene depth information is generated. The depth values ​​of the background and dynamic targets are accurately captured, and the depth information of the dynamic targets can be effectively updated to reflect the movement and changes of the targets.

[0052] In step S220 , an occluded pixel region in the dynamic scene depth map is identified, and the dynamic scene depth map is compensated based on the depth data of the occluded pixel region, thereby generating a dynamic scene point cloud.

[0053] It should be noted that in actual scenarios, targets and obstacles may block part of the line of sight or affect the collection of depth data. Therefore, the processing of occluded areas is crucial to ensure the accuracy of three-dimensional reconstruction.

[0054] Specifically, occlusion often occurs when a dynamic target or object blocks other objects in the field of view, preventing the acquisition of complete depth information. The system analyzes pixel data in the depth map of the dynamic scene to identify occluded areas, for example, by separating foreground and background and detecting depth outliers.

[0055] Once the occluded area is identified, the missing depth data can be filled through compensation methods. The compensation strategies can be diverse, such as compensation based on surrounding pixel interpolation, depth data inference, or physical model-based compensation. For example, in depth data inference, the depth information before and after the target is used to fill the missing area through inference.

[0056] By compensating for occluded areas and eliminating depth data loss caused by dynamic object occlusion or scene complexity, the system can recover and improve missing data in the dynamic scene depth map. Ultimately, the compensated depth map is converted into a 3D point cloud, accurately recording the position, depth, and color information of each point in 3D space. The resulting dynamic scene point cloud not only reflects the motion trajectory of dynamic objects, but also accurately reproduces the physical structure and spatial relationships of the scene.

[0057] Regarding the implementation details of step S220, in some examples of the embodiments of the present application, in view of the dynamic and occluded characteristics of the passenger station environment, an adaptive occlusion-compensated depth estimation algorithm (AOCDE) is proposed in this article, which jointly estimates depth using multi-view images and depth sensor data.

[0058] Figure 3 An operational flowchart of an example of adaptive occlusion-compensated depth estimation according to an embodiment of the present application is shown.

[0059] like Figure 3 As shown, in step S310, image gradient consistency and bidirectional matching are used to detect whether a pixel is occluded. When at least one first pixel does not satisfy bidirectional matching or color consistency in multiple perspectives, each first pixel is marked as an occluded pixel area.

[0060] Specifically, image gradient consistency is achieved by comparing the reference image ( ) and the source image ( ) to detect occlusion.

[0061] , formula (2) Where, is the reference image Medium pixels The gradient at is the source image Medium pixels The gradient at Represents the gradient consistency threshold. When the image gradient is inconsistent, it means that the pixel may be occluded.

[0062] Bidirectional matching detects occlusion by comparing the matching of the same pixel in the reference image and the source image.

[0063] , formula (3) , formula (4) Where, is the reference image Pixels in With the source image Corresponding pixels difference metrics (e.g., color difference, pixel value difference); is the source image Pixels in With reference image The corresponding pixels in The difference measure.

[0064] Through bidirectional matching, it is detected whether there is a mismatch between the reference image and the source image. If the matching error in a certain direction exceeds a certain threshold, such as , then the pixel is marked as an occluded area.

[0065] In step S320 , it is detected whether the blocked pixel area is located in the coverage area of ​​the depth sensor.

[0066] The depth sensor's coverage area refers to the spatial range within which it can directly perceive and provide valid depth data. In practical applications, depth sensors (such as LiDAR or infrared depth cameras) typically have a specific field of view and measurement distance. Areas outside this range cannot obtain valid depth information. Therefore, for pixels marked as occluded, it is necessary to confirm whether they are within the depth sensor's effective coverage area to determine whether depth sensor data can be used for compensation.

[0067] In step S331, if the occluded pixel area is located in the coverage area of ​​the depth sensor, the point cloud data provided by the depth sensor is used to calculate the depth value corresponding to the occluded pixel area through the nearest neighbor interpolation algorithm.

[0068] In some embodiments, based on the pixel region marked as occluded, the image coordinates of the region (e.g., the pixel's position in the image) are used to convert them into three-dimensional coordinates. Next, assuming that the depth point cloud data is sparsely distributed in space, a nearest neighbor interpolation algorithm is used based on the neighborhood position of the occluded pixel in the point cloud to find the point closest to the pixel position in the depth point cloud data, thereby obtaining the corresponding depth value, effectively supplementing the depth information of the occluded region.

[0069] In step S333 , if the occluded pixel region is not located in the coverage area of ​​the depth sensor, a depth estimation algorithm based on Kalman filtering is used to predict the depth value of the occluded pixel region.

[0070] The Kalman filter is an effective recursive filtering method widely used for state estimation in dynamic systems. It can estimate missing depth information based on available depth data and the motion trajectory of dynamic targets. Specifically, the target's motion trajectory is first estimated using dynamic target detection and tracking algorithms in the video stream (such as object detection based on optical flow or deep learning). The motion trajectory provides the target's position changes over time, which helps the Kalman filter estimate the depth value. The Kalman filter then predicts the missing depth value by considering the target's dynamic model (such as uniform linear motion or acceleration model) as well as measurement noise (from sensor and image processing errors).

[0071] , formula (5) Where, For the moment Depth estimation, For the moment The depth value of is the depth mean of the spatial neighborhood based on the occluded pixel area, is the Kalman filter gain.

[0072] Here, the Kalman gain is used to combine the depth estimate of the previous moment with the depth mean of the surrounding spatial area. The two are weighted and fused to obtain a more accurate depth estimation value. Kalman gain The role of is to balance the influence between historical depth values ​​and spatial depth mean, ensuring that changes in dynamic scenes are taken into account during prediction.

[0073] By combining the depth information of the previous moment and the depth data of the local spatial area, the weighted mechanism of Kalman filtering is used to improve the accuracy and robustness of depth estimation. The depth value of the occluded area can be accurately estimated in the blind spot of the depth sensor, ensuring the integrity of the depth data in dynamic scenes.

[0074] In step S340, the predicted depth map of the occluded pixel area is fused with the dynamic scene depth map, and the fused depth map is converted into a three-dimensional point cloud to obtain a corresponding dynamic scene point cloud.

[0075] Here, the depth information of occluded pixel areas predicted by the depth sensor or Kalman filter algorithm is fused with the dynamic scene depth map. The estimated depth map is converted into a 3D point cloud and mapped to the world coordinate system using the camera extrinsic parameter matrix. The point clouds from different cameras and depth sensors are fused, and noise is removed using weighted averaging and voxel filtering. Motion compensation is applied to consecutive time frames to ensure the spatiotemporal continuity of the point cloud.

[0076] Through this fusion, depth information from occluded areas is effectively added to the original depth map, achieving a more complete and accurate scene representation. The fused depth map is then converted into a 3D point cloud, where each point represents a spatial location and a corresponding depth value in the scene. The resulting dynamic scene point cloud is not only more complete and accurate, but also better reflects the spatial structure and dynamic changes in the actual scene, providing a high-quality 3D model.

[0077] Regarding the implementation details of the perspective selection in step S130, in some examples of the embodiments of the present application, an Adaptive Viewpoint Optimization (AVO) algorithm may be used to analyze scene distribution and event attention in real time to dynamically plan an optimal perspective sequence.

[0078] It's important to note that in multi-view 3D reconstruction, choosing the right viewing angle is crucial to the final video quality. Traditional methods typically use fixed reference angles or preset paths, but in dynamic environments, this can easily lead to long-term occlusion of certain areas or discontinuous images.

[0079] The specific implementation details of the AVO algorithm mainly include the construction of candidate perspective sets, the design of perspective evaluation indicators, and the calculation of perspective scoring functions.

[0080] Regarding the candidate perspective set construction operation, set the perspective position parameters of the virtual camera in three-dimensional space , Represents the position information of the virtual camera, Represents the coordinates of the virtual camera in three-dimensional space; and sets the viewing angle by Euler angles definition, are the yaw angle, pitch angle, and roll angle respectively.

[0081] Specifically, the spatial position and Euler angles of the virtual camera can be laid out according to the space of the static set model to ensure that the virtual camera can cover all key areas in the station (such as platforms, waiting rooms, passages, etc.), while taking into account the maximum coverage of the viewing angle and the passage layout within the station.

[0082] Then, the monitoring area is extracted from the static geometric model and discretized to generate a preset number of candidate virtual perspectives covering the monitoring area; The candidate virtual perspectives are represented as .

[0083] To ensure comprehensive monitoring of key station areas, candidate viewpoints are generated by discretizing the station's architectural layout. Through a rational spatial partitioning process, multiple candidate virtual viewpoints are generated, each evaluated, and the optimal viewpoint is ultimately selected for dynamic 3D surveillance video frame rendering.

[0084] Here, the monitoring area mainly refers to the feasible areas in the railway passenger station, such as waiting rooms and platforms. According to the spatial geometry model of the station, the passable area of ​​the station is discretized and sampled, multiple viewing angles are selected, and candidate viewing angles are generated in combination with the Euler angle range to ensure that the core areas of the station can be effectively covered, especially the crowded areas and key equipment areas.

[0085] To quantify the quality of each candidate perspective, we define four perspective evaluation metrics: coverage, occlusion rate, perspective transformation cost, and key area weight. Furthermore, we use a perspective scoring function to quantitatively calculate the perspective score of each candidate perspective.

[0086] Specifically, the comprehensive scoring function is: , formula (6) Where, Indicates the current candidate virtual perspective The viewing angle rating, Indicates the current candidate virtual perspective The coverage, Indicates the current candidate virtual perspective The occlusion rate, Indicates the current candidate virtual perspective The perspective change cost, Indicates the current candidate virtual perspective The weight of key areas; is the weight coefficient, which adjusts the impact of different scoring items on the final perspective score.

[0087] In formula (6), is an adjustable parameter, satisfying In each time frame, the system calculates the scoring function for all candidate perspectives and selects the perspective with the highest score. The virtual camera position for the current frame.

[0088] , formula (7) Where, Indicates the total number of targets in the scene; Target The visible area indicates that the target is within the viewing angle The area under observation; Target total area.

[0089] In formula (7), coverage represents the proportion of the target area that can be observed from the current viewing angle. It reflects the effectiveness of the viewing angle. The higher the coverage, the more comprehensive the coverage of the scene from the viewing angle, ensuring the monitoring coverage of important areas.

[0090] , formula (8) Where, Indicates the target Indicates the viewing angle The obscured area below.

[0091] In Equation (8), the occlusion rate is used to evaluate the occlusion of the target under the current viewing angle. This scoring item is used to determine the effectiveness of the viewing angle. The lower the occlusion rate, the clearer the target can be observed from that viewing angle. By calculating the occlusion rate, information loss caused by occlusion is reduced, providing clearer scene monitoring data.

[0092] , formula (9) Where, represents the viewing angle from the previous 3D video frame, Representation perspective The corresponding virtual camera position, Representation perspective The corresponding virtual camera position, Represents the weight coefficient used to adjust the pitch angle change during the viewing angle transformation process. Representation perspective The Euler angles of Representation perspective The Euler angles of Indicates the maximum range of change of the virtual camera. Indicates the maximum range of Euler angle variation.

[0093] In Equation (9), the perspective change cost is used to evaluate the cost of switching from the previous perspective to the current perspective, primarily considering changes in position and Euler angles (pitch, yaw, and roll). The smaller the perspective change cost, the smoother the perspective switch and the faster the system's response. By calculating the perspective change cost, the system can assess the smoothness and cost of perspective switching, improve the system's real-time responsiveness, and ensure smooth monitoring.

[0094] , formula (10) Where, Representation perspective The key area weights under Representation perspective Field of view, Represents each key area in the scene ,The key areas are preset based on the mission requirements; is the indicator function, when the key area When it is in the field of view of the current perspective, the value is 1, otherwise it is 0; For key areas The regional weight indicates the importance of the region; Represents the sum of the area weights of all key areas in the scene.

[0095] In formula (10), the key area weight is used to evaluate the importance of key areas from a certain perspective, for example, giving priority to areas with dense passenger flow or potential safety hazards in the station. Specifically, according to the task requirements, key areas in the station are preset, such as entrances and exits, ticket gates, escalators, etc. The weight value of each key area is Set it according to the importance of the area.

[0096] In some examples of the embodiments of the present application, the regional weights of key areas may also be dynamically adjusted according to real-time monitoring results.

[0097] Specifically, based on the dynamic scene point cloud, the passenger flow density and abnormal behavior events in each key area are identified. Abnormal behavior events include any one of the following: a fall event, a reverse traffic event, or crossing a warning zone.

[0098] Here, the sub-point cloud analysis of each corresponding key area in the dynamic scene point cloud is performed to calculate the passenger flow density index.

[0099] , formula (11) Where, For key areas Passenger flow density index, For key areas The number of passenger flow point clouds in For key areas The total amount of point cloud covering the area.

[0100] Abnormal behavior detection can use a variety of target detection algorithms (such as YOLO) to identify dynamic objects in the video and use depth information to determine whether abnormal behavior (such as falling, walking against traffic, crossing the warning zone, etc.) has occurred. These events will be marked as abnormal behavior and trigger further response.

[0101] Furthermore, when it is monitored that the passenger flow density of the first key area exceeds a preset passenger flow density threshold and / or an abnormal behavior event occurs, the area weight of the first key area is increased.

[0102] By dynamically increasing or decreasing the monitoring weight of various station areas based on real-time passenger density and abnormal behavior event detection results, the system can optimize monitoring strategies based on the real-time situation within the station. For example, if the passenger density in a certain area exceeds a threshold or an abnormal behavior event occurs, the monitoring weight of that area is automatically increased, ensuring that critical areas receive priority attention within the entire monitoring system. This improves response speed to abnormal events and ensures that dynamic high-risk areas are effectively monitored.

[0103] Regarding the details of rendering the three-dimensional video frame in step S140, in some examples of the embodiments of the present application, the dynamic scene point cloud is cropped based on the optimal virtual observation perspective, the geometry outside the virtual camera field of view is eliminated, and the Z-buffer (depth buffer) is used for depth testing to ensure pixel visibility.

[0104] When generating 3D video frames, the dynamic scene point cloud must first be cropped based on the optimal virtual viewing angle. The purpose of cropping is to remove geometry outside the virtual camera's field of view to reduce the rendering burden and ensure that only the portion visible to the camera is rendered, improving rendering efficiency. To ensure that the depth information of each pixel accurately reflects the position of objects in the scene, a depth test is performed using Z-buffer technology to determine the visibility of each pixel. Depth values ​​are then compared to determine whether the pixel should be rendered.

[0105] The video frames captured by multiple cameras are projected onto the surface of the dynamic scene point cloud to form a texture map. The infrared image of the depth sensor is used to estimate the ambient light distribution. Combined with the local illumination calculation based on the Phong (Lambert) model of the railway station, the brightness fitting of the three-dimensional video frame is achieved.

[0106] Texture mapping is used here to achieve more realistic 3D video rendering. Specifically, by projecting video frames captured by multiple cameras onto the surface of a dynamic scene point cloud, a corresponding texture map is generated. This allows detailed information about the station scene (such as texture and color) to be mapped onto the 3D model. Next, the ambient lighting distribution is estimated using infrared images from the depth sensor, and the Phong illumination model is used to calculate the illumination for each pixel, ensuring that the generated 3D video frames have realistic brightness and lighting effects.

[0107] Specifically, the ambient light intensity is estimated using the infrared image of the depth sensor , which provides background information for lighting calculation. The Phong model is used to calculate local lighting for each pixel. The Phong lighting model calculates the brightness of each pixel by considering the combination of ambient light, diffuse light, and specular light.

[0108] , formula (12) Where, is the light source direction, is the surface normal vector, is the reflection vector, is the optimal virtual observation angle vector, Indicates the ambient light intensity, represents the ambient light reflection coefficient, represents the diffuse reflectance, represents the specular reflection coefficient, Indicates the light intensity of the light source, The specular exponent controls the degree of scattering of specular reflections.

[0109] Based on texture mapping and lighting calculations, the generated 3D video frames not only have high-quality details, but also can realistically reproduce lighting and environmental changes, greatly enhancing the realism of the visual effects. By combining data from the depth sensor and the Phong lighting model, it can accurately reflect the brightness and lighting changes in the scene, improving the visual quality of the video frames.

[0110] In the embodiments of the present application, during the rendering process, a Z-buffer depth test is used to ensure accurate display of the scene. The infrared image of the depth sensor is used to estimate the ambient light distribution, and the local lighting effect is calculated using the Phong lighting model to ensure that the rendering results have realistic lighting and reflection effects. Furthermore, the video sequence obtained by continuous rendering is compressed by a video encoder and transmitted to the monitoring center. The encoder can dynamically adjust the bit rate and resolution based on the network bandwidth. To ensure the real-time performance of the system, GPU-based parallel rendering and encoding can be used.

[0111] To validate the effectiveness of the proposed adaptive perspective 3D video generation method, this study designed a comparative experiment. Because real-world station environments are complex and difficult to replicate, this study employed virtual scene simulation combined with real-world data to construct a 3D station simulation consisting of platforms, waiting rooms, and entrances and exits. Pedestrian models were randomly generated within the scene to simulate passenger entry, exit, waiting, and boarding. Some areas were also obstructed by objects (such as pillars and billboards) and subjected to varying lighting conditions.

[0112] 1. Experimental Setup Baseline algorithm: Traditional multi-view 3D reconstruction and fixed-view rendering. A fixed reference viewpoint and a preset camera path are used, and the position and pose of the virtual camera are not adjusted throughout the experiment. Depth estimation uses a standard multi-baseline stereo method without occlusion compensation.

[0113] Adaptive viewpoint algorithm: The adaptive viewpoint optimization algorithm proposed in this paper is used. The candidate viewpoint set covers multiple high-angle and low-angle viewpoints above the platform and waiting room. Depth estimation uses the AOCDE algorithm, while using the depth sensor to provide supplementary data. The parameters of the viewpoint scoring function are set to , and dynamically adjust the weights of key areas according to passenger flow density.

[0114] Evaluation indicators: In order to comprehensively evaluate the effect, three indicators are set: Detection accuracy: The accuracy of pedestrian detection based on 3D video, used to evaluate the algorithm's ability to recognize targets.

[0115] Occlusion rate: In the rendering result, the ratio of the obscured target area to the total target area, which is used to measure the improvement effect of perspective selection on occlusion.

[0116] Rendering delay: The delay from data acquisition to 3D image display, used to verify the real-time performance of the system.

[0117] 2. Experimental Results Figure 4 A schematic diagram of a simulation effect showing an example of comparison of detection accuracy between the baseline algorithm and the adaptive view algorithm within 10 time frames.

[0118] like Figure 4 As shown in the figure, the detection accuracy of the adaptive view algorithm is always higher than that of the baseline algorithm and keeps increasing steadily over time, which shows that the adaptive view can avoid occlusion in dynamic scenes and improve the integrity of pedestrian detection.

[0119] Figure 5 The figure shows a simulation effect diagram of an example comparing the occlusion rate between the baseline algorithm and the adaptive view algorithm within 10 time frames.

[0120] like Figure 5 As shown in the figure, the occlusion rate of the baseline algorithm slowly decreases over time, but remains between 0.23 and 0.30. The occlusion rate of the adaptive viewpoint algorithm is significantly lower and stabilizes after the seventh frame. This shows that dynamic viewpoint selection can significantly reduce occlusion and improve observation integrity.

[0121] Furthermore, experimental measurements showed that the adaptive view algorithm's rendering latency averaged 150 ms, while the baseline algorithm's latency was 120 ms. While this slight increase in latency still met real-time monitoring requirements (<200 ms), the adaptive view algorithm significantly outperformed the baseline algorithm in detection accuracy and occlusion rate, while maintaining acceptable real-time performance.

[0122] 3. Discussion Experimental results show that the adaptive perspective-based 3D video generation method proposed in this paper can effectively improve monitoring quality by adjusting the position and posture of the virtual camera used for 3D video rendering in real time. The main advantages include:

[0123] Strong occlusion handling capabilities: The viewing angle scoring function comprehensively considers coverage and occlusion rate, automatically adjusting the viewing angle in obstructed areas to significantly reduce the occlusion rate. Compared to traditional fixed viewing angles, the occlusion rate was reduced by approximately 40% in experiments.

[0124] High target detection accuracy: Without changing the hardware configuration, the algorithm significantly improves target detection accuracy by optimizing the observation angle and depth estimation, especially for detecting obscured pedestrians in crowded crowds.

[0125] Flexibility and Scalability: The weights in the viewpoint scoring function can be adjusted based on business needs. When emergencies occur, the system can quickly focus on key areas and provide timely attention to abnormal scenarios. The candidate viewpoint set can also be expanded or reduced in real time based on station structure and network bandwidth, adapting to station deployments of varying sizes.

[0126] Acceptable real-time performance: Although the adaptive perspective algorithm requires additional perspective calculation and depth compensation, resulting in a slight increase in rendering latency, it is still within an acceptable range for real-time monitoring. Future hardware acceleration or algorithm optimization can further reduce latency.

[0127] This paper addresses the challenges of traditional railway station video surveillance, such as a single viewing angle, severe occlusion, and a lack of 3D information. By implementing multi-sensor acquisition, dynamic depth estimation, viewpoint optimization, and real-time rendering, the system generates continuous and complete 3D video footage within complex station environments. Compared to traditional multi-camera fixed-viewpoint approaches, this method significantly improves object detection accuracy and occlusion handling. Experimental results demonstrate an average improvement of approximately 15% in detection accuracy and a nearly 40% reduction in occlusion, all while maintaining real-time performance.

[0128] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of combined actions, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0129] Figure 6 A structural block diagram of an example of a railway passenger station three-dimensional video generation system based on adaptive viewing angle according to an embodiment of the present application is shown.

[0130] like Figure 6 As shown, the railway passenger station three-dimensional video generation system 600 based on adaptive perspective includes a heterogeneous sensor acquisition unit 610, a dynamic point cloud generation unit 620, a virtual perspective scoring unit 630 and a three-dimensional video rendering unit 640.

[0131] The heterogeneous sensing acquisition unit 610 is used to deploy multiple cameras and multiple depth sensors at the railway passenger station, synchronously collect the time-series video streams and point cloud data of each sensor through the network clock synchronization mechanism, and generate a real-time sensing data set with a unified timestamp.

[0132] The dynamic point cloud generation unit 620 is used to construct a static geometric model reflecting the layout of the railway passenger station facilities based on the building information model of the railway passenger station, and fuse the static geometric model with the real-time sensor data set to generate a corresponding dynamic scene point cloud.

[0133] The virtual perspective scoring unit 630 is used to count the coverage, occlusion rate, perspective transformation cost and key area weight corresponding to each candidate virtual perspective in the candidate virtual perspective set, and call the comprehensive scoring function to calculate the corresponding perspective score; the virtual perspective is the perspective of the virtual camera.

[0134] The 3D video rendering unit 640 is configured to select the best virtual observation perspective with the highest corresponding perspective score, and perform 3D rendering on the dynamic scene point cloud based on the best virtual observation perspective to generate a corresponding 3D video frame.

[0135] Among them, the coverage represents the proportion of key targets visible under the current candidate virtual perspective, the occlusion rate represents the proportion of the target area occluded under the current candidate virtual perspective to the total area, the perspective transformation cost represents the conversion cost required to move from the perspective of the previous three-dimensional video frame to the current candidate virtual perspective, and the key area weight represents the importance of the key area covered by the current candidate frame perspective.

[0136] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the steps of any of the above-mentioned methods for generating three-dimensional videos of railway passenger stations based on adaptive perspectives in the present application.

[0137] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs any step of the above-mentioned method for generating a three-dimensional video of a railway passenger station based on an adaptive perspective.

[0138] In some embodiments, an embodiment of the present application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the steps of a method for generating three-dimensional video of a railway passenger station based on an adaptive perspective.

[0139] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0140] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to: mobile communication devices, ultra-mobile personal computer devices, portable entertainment devices or other onboard electronic devices with data interaction functions.

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0142] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a general hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating three-dimensional video of a railway passenger station based on adaptive viewing angle, characterized in that: The method comprises: Multiple cameras and depth sensors are deployed at railway stations. Network clock synchronization is used to synchronously collect time-series video streams and point cloud data from each sensor, generating a real-time sensor dataset with a unified timestamp. constructing a static geometric model reflecting the layout of railway passenger station facilities based on the building information model of the railway passenger station, and fusing the static geometric model with the real-time sensor data set to generate a corresponding dynamic scene point cloud; For each candidate virtual perspective in the candidate virtual perspective set, the coverage, occlusion rate, perspective transformation cost and key area weight corresponding to the candidate virtual perspective are counted, and a comprehensive scoring function is called to calculate the corresponding perspective score; the virtual perspective is the perspective of the virtual camera; Selecting the best virtual observation perspective with the highest corresponding perspective score, and performing three-dimensional rendering on the dynamic scene point cloud based on the best virtual observation perspective to generate a corresponding three-dimensional video frame; Among them, the coverage represents the proportion of key targets visible under the current candidate virtual perspective, the occlusion rate represents the proportion of the target area occluded under the current candidate virtual perspective to the total area, the perspective transformation cost represents the conversion cost required to move from the perspective of the previous three-dimensional video frame to the current candidate virtual perspective, and the key area weight represents the importance of the key area covered by the current candidate frame perspective.

2. The method according to claim 1, characterized in that The fusing of the static geometric model with the real-time sensing data set to generate a corresponding dynamic scene point cloud includes: According to the real-time sensing data set, the real-time collected depth data and the dynamic targets in the video stream are fused to generate a dynamic scene depth map; An occluded pixel region in the dynamic scene depth map is identified, and the dynamic scene depth map is compensated based on depth data of the occluded pixel region, thereby generating a dynamic scene point cloud.

3. The method according to claim 2, characterized in that The identifying of an occluded pixel region in the dynamic scene depth map and compensating the depth map of the dynamic scene based on depth data of the occluded pixel region, thereby generating a dynamic scene point cloud, includes: Detecting whether a pixel is occluded by using image gradient consistency and bidirectional matching, and marking each first pixel as an occluded pixel area when at least one first pixel does not satisfy bidirectional matching or color consistency in multiple perspectives; If the blocked pixel area is located in the coverage area of ​​the depth sensor, the depth value corresponding to the blocked pixel area is calculated using the point cloud data provided by the depth sensor through a nearest neighbor interpolation algorithm; If the occluded pixel area is not located in the coverage area of ​​the depth sensor, a depth estimation algorithm based on Kalman filtering is used to predict the depth value of the occluded pixel area: ， Where, For the moment Depth estimation, For the moment The depth value of is the depth mean of the spatial neighborhood based on the occluded pixel area, is the Kalman filter gain; The predicted depth map of the occluded pixel area is fused with the dynamic scene depth map, and the fused depth map is converted into a three-dimensional point cloud to obtain a corresponding dynamic scene point cloud.

4. The method according to claim 1, wherein The operation of determining the candidate virtual perspective set includes: Set the viewing angle position parameters of the virtual camera in three-dimensional space , Represents the position information of the virtual camera, Represents the coordinates of the virtual camera in three-dimensional space; and sets the viewing angle by Euler angles definition, are the yaw angle, pitch angle and roll angle respectively; Extracting a monitoring area from the static geometric model and discretizing the monitoring area to generate a preset number of candidate virtual perspectives covering the monitoring area; wherein the first The candidate virtual perspectives are represented as .

5. The method according to claim 1, wherein The comprehensive scoring function is: ， Where, Indicates the current candidate virtual perspective The viewing angle rating, Indicates the current candidate virtual perspective The coverage, Indicates the current candidate virtual perspective The occlusion rate, Indicates the current candidate virtual perspective The perspective change cost, Indicates the current candidate virtual perspective The weight of key areas; is the weight coefficient, which adjusts the impact of different scoring items on the final perspective score; ， Where, Indicates the total number of targets in the scene; Target The visible area indicates that the target is within the viewing angle The area under observation; Target total area; ， Where, Indicates the target Indicates the viewing angle The obscured area below; ， Where, represents the viewing angle from the previous 3D video frame, Representation perspective The corresponding virtual camera position, Representation perspective The corresponding virtual camera position, Represents the weight coefficient used to adjust the pitch angle change during the viewing angle transformation process. Representation perspective The Euler angles of Representation perspective The Euler angles of Indicates the maximum variation range of the virtual camera, Indicates the maximum range of variation of Euler angles; ， Where, Representation perspective The key area weights under Representation perspective Field of view, Represents each key area in the scene ,The key areas are preset based on the mission requirements; is the indicator function, when the key area When it is in the field of view of the current perspective, the value is 1, otherwise it is 0; For key areas The regional weight indicates the importance of the region; Represents the sum of the area weights of all key areas in the scene.

6. The method according to claim 5, characterized in that Also includes: Identify passenger flow density and abnormal behavior events in each of the key areas based on the dynamic scene point cloud; the abnormal behavior events include any one of the following: a fall event, a retrograde event, or crossing a warning zone; When it is monitored that the passenger flow density of the first key area exceeds a preset passenger flow density threshold and / or an abnormal behavior event occurs, the area weight of the first key area is increased.

7. The method according to claim 1, characterized in that The performing three-dimensional rendering on the dynamic scene point cloud based on the optimal virtual observation angle to generate a corresponding three-dimensional video frame includes: The dynamic scene point cloud is cropped based on the optimal virtual observation angle, geometric objects outside the field of view of the virtual camera are eliminated, and a depth test is performed using a Z-buffer to ensure pixel visibility; The video frames captured by multiple cameras are projected onto the surface of the dynamic scene point cloud to form a texture map. The infrared image of the depth sensor is used to estimate the ambient light distribution. Combined with the local illumination calculation based on the Phong model of the railway passenger station, the brightness fitting of the 3D video frame is achieved: ， Where, is the light source direction, is the surface normal vector, is the reflection vector, is the optimal virtual observation angle vector, Indicates the ambient light intensity, represents the ambient light reflection coefficient, represents the diffuse reflectance, represents the specular reflection coefficient, Indicates the light intensity of the light source, The specular exponent controls the degree of scattering of specular reflections.

8. A railway passenger station 3D video generation system based on adaptive viewing angle, characterized in that: The system comprises: Heterogeneous sensor acquisition unit, used to deploy multiple cameras and depth sensors at railway stations, synchronously collect time-series video streams and point cloud data from each sensor through a network clock synchronization mechanism, and generate real-time sensor data sets with unified timestamps; a dynamic point cloud generation unit, configured to construct a static geometric model reflecting the layout of the railway passenger station facilities based on the building information model of the railway passenger station, and fuse the static geometric model with the real-time sensor data set to generate a corresponding dynamic scene point cloud; A virtual perspective scoring unit is configured to calculate the coverage, occlusion rate, perspective conversion cost, and key area weight corresponding to each candidate virtual perspective in the candidate virtual perspective set, and to call a comprehensive scoring function to calculate a corresponding perspective score; the virtual perspective is the perspective of the virtual camera; a 3D video rendering unit, configured to select an optimal virtual observation perspective having the highest corresponding perspective score, and perform 3D rendering on the dynamic scene point cloud based on the optimal virtual observation perspective to generate a corresponding 3D video frame; Among them, the coverage represents the proportion of key targets visible under the current candidate virtual perspective, the occlusion rate represents the proportion of the target area occluded under the current candidate virtual perspective to the total area, the perspective transformation cost represents the conversion cost required to move from the perspective of the previous three-dimensional video frame to the current candidate virtual perspective, and the key area weight represents the importance of the key area covered by the current candidate frame perspective.

Images