Studio virtual and real picture synchronous tracking and fusion system

Abstract

The invention relates to the technical field of broadcast television and virtual-real fusion, and particularly discloses a studio virtual-real picture synchronous tracking and fusion system which comprises a multi-sensor fusion tracking module, a full-IP synchronous control module, a virtual-real fusion rendering module and a multi-program sharing scheduling platform. The multi-sensor fusion tracking module fuses various sensor data and outputs high-precision six-degree-of-freedom pose information; the full-IP synchronous control module controls a synchronous error within 10ms based on PTP (Precision Time Protocol) synchronization and dynamic delay compensation; the virtual-real fusion rendering module combines AR / MR rendering, digital twinning and artificial intelligence to realize seamless fusion of virtual and real pictures; the multi-program sharing scheduling platform realizes resource virtualization and flexible reuse. The method solves the tracking defect of a single sensor, improves the virtual-real synchronization precision and visual expressive force, reduces the operation and maintenance cost, improves the intelligent level of news production, and is suitable for studio live broadcast and other scenes.

Description

Technical Field

[0001] This invention relates to the field of broadcasting and television technology and virtual-real fusion technology, specifically to a studio virtual-real image synchronization tracking and fusion system. Background Technology

[0002] With the development of converged media technologies, news studios are placing higher demands on the flexibility, real-time nature, and visual expressiveness of program production. Virtual-real fusion technologies, especially augmented reality (AR) and mixed reality (MR) technologies, have become important means to enhance the visual presentation of news programs.

[0003] In existing technologies, traditional SDI-based virtual studio systems mostly rely on a single sensor for camera and presenter position tracking. This approach presents the following technical problems in practical applications:

[0004] Insufficient tracking accuracy: A single sensor is easily interfered with by factors such as changes in ambient lighting and obstruction by people, which can lead to loss of tracking signals or decreased accuracy, affecting the alignment of virtual and real images.

[0005] Large synchronization error: The lack of a unified time reference between virtual scene rendering and real camera shooting causes misalignment, shaking or delay of virtual and real images during dynamic shooting, especially in multi-camera switching or fast-moving scenes.

[0006] Poor system scalability: The point-to-point connection method based on SDI leads to rigid system resource scheduling, making it difficult to support multiple programs sharing the same studio system, resulting in low equipment utilization and high operation and maintenance costs.

[0007] Insufficient depth of virtual-real integration: Existing systems are mostly limited to simple virtual background replacement, failing to achieve deep integration of the real scene inside the studio with the external digital space, and lacking an immersive visual experience.

[0008] Therefore, how to build a studio system with high precision, low latency, high scalability and deep virtual-real fusion capability has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide a studio virtual and real image synchronous tracking and fusion system, which realizes high-precision pose tracking of the camera and the target object, accurate synchronous fusion of virtual and real images, and supports multiple programs to share system resources, thereby improving the efficiency and flexibility of studio production.

[0010] To solve the above-mentioned technical problems, the technical solution provided by the present invention is: a studio virtual and real image synchronous tracking and fusion system, including: a multi-sensor fusion tracking module, a fully IP-based synchronous control module, a virtual and real fusion rendering module, and a multi-program shared scheduling platform.

[0011] The multi-sensor fusion tracking module includes:

[0012] Photoelectric sensors, installed on the ceiling of the studio, are used to capture infrared reflective markers on the camera and the presenter;

[0013] Infrared sensors are used to provide auxiliary position information in low-light conditions;

[0014] Image recognition sensors are deployed around the camera and the presenter to extract the outline and feature points of target objects in real time through visual recognition algorithms.

[0015] Point cloud sensors, deployed in key locations within the studio, are used to construct a 3D point cloud model of the studio's internal space, providing spatial geometric constraints;

[0016] The multi-sensor fusion tracking module uses a Kalman filter algorithm to fuse data from photoelectric sensors, infrared sensors, image recognition sensors, and point cloud sensors, outputting six-degree-of-freedom pose information for the camera and the presenter. The point cloud sensor data is used to establish a spatial geometric constraint model. When photoelectric or infrared signals are blocked, the system automatically switches to a tracking mode based on the fusion of point cloud and image recognition sensors, ensuring tracking continuity and positioning accuracy. The positioning accuracy between the camera and the target object reaches ±1mm.

[0017] The all-IP-based synchronization control module includes:

[0018] The PTP high-precision time synchronization unit, built on the IEEE 1588 protocol, provides a unified time reference for cameras, rendering servers, tracking modules and broadcast servers in the system.

[0019] The delay compensation unit, based on the "unified timestamp synchronization + dynamic delay compensation" mechanism, measures and tracks the link delay of each link in data acquisition, transmission, rendering and video acquisition in real time, builds a closed-loop compensation model, and dynamically adjusts the timing of virtual scene rendering so that the synchronization error between virtual and real images is controlled within 10ms.

[0020] The all-IP signal scheduling unit adopts the SMPTE ST 2110 standard, which encapsulates video, audio, metadata and tracking data into a unified IP stream. It enables flexible signal routing and resource pooling scheduling through Ethernet switches, and supports multiple programs sharing the same studio system.

[0021] The virtual-real fusion rendering module includes:

[0022] The AR / MR real-time rendering engine is used to generate a virtual scene that matches the camera's viewpoint in real time based on the pose information output by the multi-sensor fusion tracking module.

[0023] The digital twin space extension unit constructs a digital twin model of the studio's interior space and external landmark spaces. Utilizing a self-developed twin matching algorithm, it achieves seamless integration between live camera footage and the virtual external space, creating an immersive visual experience of "internal reality + external extension." Specifically, this unit dynamically calculates the projection mapping relationship between the real-world footage and the virtual twin space based on the camera's real-time pose and lens parameters. Through the overlap control of naked-eye vision and the ultra-high-definition camera's visual focus, it achieves precise matching between the real-world scene and the digital twin space.

[0024] The AI ​​fusion unit locally deploys multiple large language models, including but not limited to DeepSeek, Doubao, Kimi, and Gemini Flash 2.0, and integrates NVIDIA Broadcast software API interfaces to form an integrated AI workflow, enabling voice enhancement during live streaming, improved video conferencing quality, intelligent review, and intelligent output of content to small screens.

[0025] The multi-program shared scheduling platform includes:

[0026] The resource virtualization unit abstracts hardware resources such as cameras, rendering servers, and tracking equipment in the studio into a virtual resource pool;

[0027] The program scheduling unit supports multiple news programs to dynamically apply for studio resources based on time slots or event-driven methods, enabling rapid switching and reuse of the same studio system across different programs;

[0028] The operation and maintenance unit ensures system stability and reliability in high-frequency multi-program usage scenarios through system-level redundancy design and real-time monitoring.

[0029] The advantages of this invention compared to the prior art are:

[0030] This invention solves the problem of single sensors being susceptible to occlusion or lighting effects by combining the complementary characteristics of photoelectric, infrared, image recognition and point cloud sensors through a multi-sensor fusion tracking module, and significantly enhances tracking continuity and robustness.

[0031] This invention is based on the PTP high-precision time synchronization and dynamic delay compensation mechanism, which controls the synchronization error between virtual and real images to within 10ms, effectively avoiding problems such as image misalignment and jitter, and meeting the strict requirements of live broadcast programs.

[0032] This invention adopts an all-IP architecture and the SMPTE ST 2110 standard to achieve flexible scheduling of signals and resources, support multiple programs to share the same studio system, and significantly reduce construction and operation and maintenance costs.

[0033] This invention achieves deep integration of the interior real scene of the studio with the external landmark space through digital twin space extension unit, thereby expanding the visual expressiveness of the virtual studio.

[0034] The artificial intelligence fusion unit of this invention deeply integrates large models with the live broadcast workflow, realizing functions such as voice enhancement, intelligent review, and multi-platform distribution, thereby improving the level of intelligence in news production and broadcasting. Attached Figure Description

[0035] Figure 1 This is a system block diagram of a studio virtual and real image synchronous tracking and fusion system according to the present invention. Detailed Implementation

[0036] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0037] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0038] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0039] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0040] The following is a detailed description of a studio virtual and real image synchronization tracking and fusion system according to the present invention, with reference to the accompanying drawings.

[0041] Combined with appendix Figure 1 This invention will be described in detail below.

[0042] A studio virtual and real image synchronous tracking and fusion system includes a multi-sensor fusion tracking module, a fully IP-based synchronous control module, a virtual and real fusion rendering module, and a multi-program shared scheduling platform. The modules work together to achieve synchronous tracking and fusion of virtual and real images in the studio, while also supporting the reuse of multiple program resources.

[0043] The multi-sensor fusion tracking module is used to collect various sensor data from the studio camera and the target object, and outputs the six-degree-of-freedom pose information of the camera and the target object through a multi-source data fusion algorithm, providing a precise position and attitude reference for virtual-real fusion.

[0044] Specifically, this module includes a photoelectric sensor, an infrared sensor, an image recognition sensor, and a point cloud sensor. The photoelectric sensor is used to capture infrared reflective markers on the camera and the target object to obtain high-precision position information. The infrared sensor is used to provide auxiliary position information under low-light conditions to compensate for the performance limitations of the photoelectric sensor in low-light environments. The image recognition sensor is used to extract the contour and feature points of the target object through visual recognition algorithms to assist in the posture recognition of the target object. The point cloud sensor is used to construct a three-dimensional point cloud model of the studio's internal space, providing spatial geometric constraints to ensure the spatial consistency of the tracking data.

[0045] To improve tracking accuracy and anti-interference capabilities, this module employs a Kalman filter algorithm to fuse data from the four sensors. The algorithm reduces noise, calibrates, and complements the multi-source sensor data, effectively suppressing measurement errors from a single sensor and improving the stability and accuracy of pose information output. Simultaneously, when the photoelectric sensor signal is obstructed, the system automatically switches to a tracking mode based on the fusion of point cloud and image recognition sensors, preventing tracking interruptions and ensuring the continuity of the tracking process. The application of the Kalman filter algorithm effectively solves problems such as signal loss, measurement drift, and environmental interference that are difficult to address with single-sensor solutions, significantly improving the reliability of the tracking system.

[0046] The all-IP-based synchronization control module provides a unified time reference and achieves synchronized control of tracking data and video signals based on a dynamic delay compensation mechanism, solving the core pain point of asynchronous virtual and real images in existing systems.

[0047] This module includes a PTP high-precision time synchronization unit, a delay compensation unit, and an all-IP signal scheduling unit. The PTP high-precision time synchronization unit, built on the IEEE 1588 protocol, provides a unified high-precision time reference for the entire system, ensuring time synchronization of tracking data, video signals, and audio signals. It employs a hierarchical master-slave mode for clock synchronization, automatically selecting the optimal master clock within the system through an optimal master clock algorithm, and achieving sub-microsecond synchronization accuracy combined with hardware timestamps, laying the foundation for the synchronized operation of all system components. The delay compensation unit measures the link delays in real-time for tracking data acquisition, transmission, rendering, and video acquisition, and dynamically adjusts the timing of virtual scene rendering to achieve synchronization between virtual and real images. The all-IP signal scheduling unit adopts the SMPTE ST 2110 standard, encapsulating video, audio, and tracking data into a unified IP stream, enabling flexible signal routing and resource pooling scheduling. Unlike traditional SDI transmission methods, it requires no encoding or decoding, enabling low-cost redundant links and seamless switching, improving signal transmission efficiency and security.

[0048] Furthermore, the delay compensation unit adopts a unified timestamp synchronization + dynamic delay compensation mechanism to build a closed-loop compensation model, collects delay data of each link in real time, and dynamically adjusts the rendering time of the virtual scene through algorithms to ensure that the synchronization error between the virtual image and the real image is controlled within 10ms, meeting the synchronization requirements of high-definition live streaming, real-time interaction and other scenarios.

[0049] The virtual-real fusion rendering module is used to generate virtual scenes in real time based on the pose information output by the multi-sensor fusion tracking module, and then fuse the virtual scenes with the real-world footage captured by the camera to achieve a seamless connection between virtual and real images.

[0050] This module includes an AR / MR real-time rendering engine, a digital twin space expansion unit, and an artificial intelligence fusion unit. The AR / MR real-time rendering engine generates virtual scenes that match the camera's perspective in real time based on pose information. It employs real-time rendering technology similar to Unreal Engine, calculating 3D scene data in real time and outputting it as a 2D image. It can dynamically adjust the presentation angle and details of the virtual scene according to changes in the camera's perspective, ensuring consistency between the virtual scene and the actual shooting perspective. Its rendering process includes CPU geometric information processing, GPU rendering pipeline processing, and frame buffer output, balancing rendering speed and image quality. The digital twin space expansion unit constructs digital twin models of the studio's internal space and external landmark spaces, and uses a twin matching algorithm to achieve projection mapping and visual fusion of the real-world scene and the virtual twin space. The artificial intelligence fusion unit locally deploys a large language model and integrates voice and video enhancement interfaces, forming an integrated artificial intelligence workflow.

[0051] Specifically, the digital twin space extension unit dynamically calculates the projection mapping relationship between the real scene and the virtual twin space based on the real-time pose of the camera and lens parameters. By constructing a virtual camera with parameters consistent with the actual camera, the two-dimensional video pixel coordinates are converted into three-dimensional model texture coordinates. Combined with the overlap control of naked-eye vision and the visual focus of the ultra-high-definition camera, the accurate matching between the real scene and the digital twin space is achieved. The projection error can be controlled at an extremely low level, solving the problems of misalignment and distortion between virtual and real images.

[0052] The large language models deployed locally by the AI ​​fusion unit include one or more of DeepSeek, Doubao, Kimi, and GeminiFlash 2.0. The deployment process can be achieved efficiently with the help of tools such as vllm and multi-card collaboration. At the same time, it integrates the NVIDIA Broadcast software API interface to realize voice enhancement, video call quality improvement, intelligent review and intelligent output of small screen content during live broadcasts, enrich the intelligent functions of studio production and reduce manual operation and maintenance costs.

[0053] The multi-program shared scheduling platform is used to virtualize and dynamically schedule studio hardware resources, supporting multiple programs to share the same system resources, improving resource utilization, reducing equipment investment and maintenance costs, and solving the problems of existing studio resources not being reusable and cumbersome switching.

[0054] The platform comprises a resource virtualization unit, a program scheduling unit, and an operation and maintenance unit. The resource virtualization unit abstracts hardware resources in the studio, such as cameras, rendering servers, and tracking equipment, into a virtual resource pool, enabling unified management and scheduling of hardware resources. Similar to the concept of "physical equipment virtualization and resource pooling" in studio virtualization transformation, it allows for flexible resource allocation to meet the needs of different programs. The program scheduling unit supports multiple news programs dynamically requesting studio resources based on time slices or event-driven methods, enabling rapid switching and reuse of the same studio system across different programs without cumbersome equipment re-tuning, thus improving program production efficiency. The operation and maintenance unit ensures system stability in high-frequency multi-program usage scenarios through system-level redundancy design and real-time monitoring, preventing program production interruptions due to resource scheduling or equipment failures. Similar to stability assurance solutions in high-concurrency live streaming scenarios, it ensures reliable 24 / 7 system operation.

[0055] The specific implementation process of the studio virtual and real image synchronous tracking and fusion system of the present invention is as follows:

[0056] This embodiment uses the XXXX Group's news IP studio as an application scenario to perform engineering deployment and verification of the system.

[0057] (I) Deployment and operation of multi-sensor fusion tracking module

[0058] Six photoelectric sensors were deployed on the ceiling of the studio, covering the entire shooting area. Infrared reflective markers were placed on the presenter and the cameras. Four infrared sensors and four image recognition sensors were deployed around the studio, and two point cloud sensors were deployed in key locations, such as the anchor desk and in front of the large screen backdrop.

[0059] After system startup, the photoelectric sensor serves as the primary tracking source, providing high-frequency pose data. When the host moves to an obstructed area (such as in front of a large screen), and the photoelectric sensor signal attenuates, the system automatically switches to a point cloud sensor and image recognition sensor fusion mode. Utilizing the spatial constraints of the point cloud model and matching image recognition feature points, it continues to output stable pose information. The Kalman filter algorithm performs weighted fusion of the multi-source data to output the final pose. The measured positioning accuracy reaches ±0.8mm, exceeding the design specifications.

[0060] (II) Deployment and Operation of the All-IP Synchronization Control Module

[0061] The system employs a primary and backup PTP clock server to provide IEEE 1588 synchronization clocks for all devices. In the signal link, the delay compensation unit monitors and tracks the end-to-end delay of data acquisition from acquisition to rendering completion, as well as the delay of the camera video signal from acquisition to output in real time. The system dynamically adjusts the output timing of the virtual rendering engine based on the difference between these two values, ensuring that the virtual image and the live-action image are output synchronously. The measured synchronization error is consistently within 8ms, meeting the stringent requirements of news live streaming for image consistency.

[0062] All signals are encapsulated as SMPTE ST 2110 IP streams and scheduled via a 10 Gigabit Ethernet switch. The system uses a resource virtualization unit to abstract four rendering servers, eight cameras, tracking modules, etc., into a resource pool, supporting multiple programs sharing the same studio system within a time slot, improving resource utilization by approximately 60%.

[0063] (III) Operation of the Virtual-Real Fusion Rendering Module

[0064] The AR / MR real-time rendering engine generates virtual scenes in real time based on the pose data output by the tracking module. The digital twin space extension unit pre-constructs high-precision 3D models of external landmarks such as the XX New Development Zone and XX Beach, and spatially aligns them with the point cloud model inside the studio. During program production, when the camera lens is pointed at a designated area within the studio, the system automatically maps the external twin space to that area, achieving a seamless visual extension of the "studio reality - external virtual space".

[0065] The AI ​​fusion unit deploys DeepSeek and Gemini Flash 2.0 models for real-time subtitle generation and content review during live broadcasts. It also integrates the NVIDIA Broadcast API to achieve real-time noise reduction of the host's voice and virtual background enhancement of the live broadcast, significantly improving the production quality and efficiency of live programs.

[0066] (iv) Operational Results

[0067] The system has been running stably in the XXXX Group's news studio for over two years, supporting the daily live and recorded broadcasts of multiple news programs. In application scenarios involving multiple programs, high-frequency use, and rapid switching, the system has operated stably with excellent virtual-real integration, and no broadcast incidents have occurred due to tracking loss or synchronization errors. The project has reached an advanced level in China in terms of system architecture, synchronization accuracy, and intelligence, and has high potential for widespread adoption.

[0068] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A studio virtual and real image synchronization tracking and fusion system, characterized in that, include: The multi-sensor fusion tracking module is used to collect various sensor data from the studio camera and the target object, and output the six-degree-of-freedom pose information of the camera and the target object through a multi-source data fusion algorithm. The all-IP synchronous control module provides a unified time reference and achieves synchronous control of tracking data and video signals based on a dynamic delay compensation mechanism. The virtual-real fusion rendering module is used to generate virtual scenes in real time based on pose information and to fuse the virtual scenes with the live footage captured by the camera for output. A multi-program shared scheduling platform is used to virtualize and dynamically schedule studio hardware resources, supporting multiple programs to share the same system resources.

2. The studio virtual and real image synchronization tracking and fusion system according to claim 1, characterized in that: The multi-sensor fusion tracking module includes: Photoelectric sensors are used to capture infrared reflective markers on cameras and target objects; Infrared sensors are used to provide auxiliary position information in low-light conditions; Image recognition sensors are used to extract the contours and feature points of target objects through visual recognition algorithms; Point cloud sensors are used to construct 3D point cloud models of the studio's interior space, providing spatial geometric constraints.

3. A studio virtual and real image synchronization tracking and fusion system according to claim 2, characterized in that: The multi-sensor fusion tracking module uses a Kalman filter algorithm to fuse data from photoelectric sensors, infrared sensors, image recognition sensors, and point cloud sensors. When the photoelectric sensor signal is blocked, it automatically switches to a tracking mode based on the fusion of point cloud sensors and image recognition sensors.

4. A studio virtual and real image synchronization tracking and fusion system according to claim 3, characterized in that: The all-IP-based synchronization control module includes: The PTP high-precision time synchronization unit, built on the IEEE 1588 protocol, is used to provide a unified time reference. The delay compensation unit is used to measure and track the link delay of each link in data acquisition, transmission, rendering and video acquisition in real time, and dynamically adjust the timing of virtual scene rendering to achieve synchronization of virtual and real images. The all-IP signal scheduling unit adopts the SMPTE ST 2110 standard, which encapsulates video, audio, and tracking data into a unified IP stream, enabling flexible signal routing and resource pooling scheduling.

5. A studio virtual and real image synchronization tracking and fusion system according to claim 4, characterized in that: The delay compensation unit adopts a unified timestamp synchronization + dynamic delay compensation mechanism to construct a closed-loop compensation model, so that the synchronization error between the virtual image and the real image is controlled within 10ms.

6. A studio virtual and real image synchronization tracking and fusion system according to claim 5, characterized in that: The virtual-real fusion rendering module includes: An AR / MR real-time rendering engine is used to generate virtual scenes that match the camera's viewpoint in real time based on pose information. The digital twin space extension unit is used to construct a digital twin model of the studio's interior space and the external landmark space, and to realize the projection mapping and visual fusion of the real scene and the virtual twin space through the twin matching algorithm; The AI ​​fusion unit is used to deploy large language models locally and integrate speech and video enhancement interfaces to form an integrated AI workflow.

7. A studio virtual and real image synchronization tracking and fusion system according to claim 6, characterized in that: The digital twin space extension unit dynamically calculates the projection mapping relationship between the real scene and the virtual twin space based on the real-time pose of the camera and the lens parameters. By controlling the overlap of naked-eye vision and the visual focus of the ultra-high-definition camera, it achieves precise matching between the real scene and the digital twin space.

8. A studio virtual and real image synchronization tracking and fusion system according to claim 7, characterized in that: The large language models deployed locally by the artificial intelligence fusion unit include one or more of DeepSeek, Doubao, Kimi, and Gemini Flash 2.0, and integrate NVIDIA Broadcast software API interfaces to achieve voice enhancement, video call quality improvement, intelligent review, and intelligent output of small-screen content during live streaming.

9. A studio virtual and real image synchronization tracking and fusion system according to claim 8, characterized in that: The multi-program shared scheduling platform includes: The resource virtualization unit is used to abstract cameras, rendering servers, and tracking devices in the studio into a virtual resource pool; The program scheduling unit is used to support multiple news programs to dynamically apply for studio resources in a time-slice or event-driven manner, enabling the same studio system to be quickly switched and reused between different programs; The operation and maintenance unit is used to ensure system stability in high-frequency multi-program usage scenarios through system-level redundancy design and real-time monitoring.

Images