A synchronization method for extended reality

By measuring link delay and adjusting the delay time, and using the tracking device to synchronize phase locking, the problem of synchronizing the camera picture and rendered picture in extended real-life technology is solved, and the picture synchronization is achieved when the camera moves quickly, improving the shooting effect.

CN115733934BActive Publication Date: 2024-11-22BEIJING ENLIGHTV TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202211437454.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-16
Publication Date
2024-11-22
Estimated Expiration
2042-11-16

AI Technical Summary

Technical Problem

During film and television shooting, it is difficult to ensure that the camera's shot is synchronized with the real-time rendering, especially when the camera moves quickly, resulting in limited shooting effects.

Method used

By measuring the link delay time of the image output to the display device and then collecting it to the synthesis server through the camera, and adjusting the delay using the data collected by the tracking device to synchronize the camera, the synthesis server and the tracking device genlock to ensure the screen synchronization.

Benefits of technology

The synchronization between the LED picture and the extended picture is achieved, ensuring that the picture does not tear when the camera moves quickly, improving the shooting effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115733934B_ABST
    Figure CN115733934B_ABST
Patent Text Reader

Abstract

The present invention discloses a synchronization method applied to extended reality, comprising the following steps: S1, measuring the link delay time of the image output to the display device and then collected by the camera to the synthesis server: by successively outputting N frames of black images and M frames of white images; S2, in the actual shooting process, the tracking data collected by the tracking device is transmitted to the virtual engine, the virtual engine renders the image according to the tracking data, and the rendered image is divided into two purposes, and the link delay time is obtained according to S1 so that the pictures taken by the camera transmitted by the two purposes are synchronized with the background image. The synchronization method adopted by the present invention synchronizes the LED background with the virtual extension and maintains synchronization when the camera moves quickly, the LED background is consistent with the extended picture, and when the camera moves quickly, the LED content is synchronized with the extended content without tearing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of extended reality, and in particular to a synchronization method applied to extended reality. Background Art

[0002] XR: Extended Reality refers to a virtual environment created by a variety of hardware devices such as LED displays, cameras, tracking devices, virtual engine servers, etc. At present, extended reality technology is widely used in film and television shooting, and the main application scenarios are as follows:

[0003] The 3D digital scene produced by the virtual engine is rendered in real time and transmitted to the synthesis server. The synthesis server projects the image onto the LED display screen. The camera shoots the LED image in real time. The synthesis server receives the camera content through the acquisition card and expands the LED content. The expanded image content is unified with the LED content to synthesize the final virtual image.

[0004] In the above scenario, the difficult problem to solve is that it is impossible to ensure that the images captured by the camera are synchronized with the real-time rendered images. When the camera moves quickly, it is difficult to synchronize. The cameraman can only limit the speed at which he moves the camera, affecting the shooting effect.

[0005] To this end, we propose a synchronization method applied to extended reality to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide a synchronization method applied to extended reality to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A synchronization method applied to extended reality comprises the following steps:

[0009] S1. Measure the link delay time of the image output to the display device and then collected by the camera to the synthesis server: the synthesis server outputs N frames of black images to the screen of the display device, and then outputs M frames of white images, where N and M are both integers greater than 1. The timing starts when the white image starts to be output. The camera shoots the display device and collects the shot image and transmits it back to the synthesis server. The timing ends when the synthesis server receives the first frame of white image. This period of time is the link delay time of the image output to the display device and then collected by the camera to the synthesis server;

[0010] S2. During the actual shooting process, a tracking device is fixed on the camera. The tracking data collected by the tracking device is transmitted to the virtual engine. The virtual engine renders an image based on the tracking data. The rendered image is transmitted to the synthesis server. The synthesis server divides the image into two uses. One use is to project it onto the screen of the display device. The picture on the display device screen is then photographed by the camera and transmitted back to the synthesis server. The other use is to transmit it through an extended path for extended synthesis. According to the link delay time measured by S1, the delay of the extended path is adjusted to synchronize the picture taken by the camera with the extended synthesized picture. The camera, synthesis server and tracking device are all genlock synchronized, that is, synchronously phase-locked, to ensure that the delay of the entire system is fixed.

[0011] In a further embodiment, the tracking device is a redspy infrared camera tracker that is fixed to the top of the camera and moves with the camera.

[0012] In a further embodiment, the tracking data includes the camera's position (x, y, z), posture (pan, tilt, roll) and viewing angle (fov) information, and the process of the virtual engine rendering the image includes generating an image consistent with the camera's position, posture, and viewing angle based on the tracking data.

[0013] In a further embodiment, the display device includes an LED screen or a curtain.

[0014] In a further embodiment, the virtual engine is Unreal Engine or Notch.

[0015] Compared with the prior art, the present invention has the beneficial effects that the LED picture is consistent with the extended picture and when the camera moves quickly, the LED content and the extended content are still synchronized without tearing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of the actual shooting process of this patent;

[0017] Figure 2 This is a schematic diagram of the delay measurement process of this patent. DETAILED DESCRIPTION

[0018] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] according to Figure 1-2 As shown, the present invention provides a synchronization method applied to extended reality, comprising the following steps:

[0021] S1. Measure the link delay time of the image output to the display device and then collected to the synthesis server through the camera: the synthesis server outputs N frames of black images to the screen of the display device, and then outputs M frames of white images, where N and M are both integers greater than 1. The purpose of N frames of black images is to clear the cached images in the entire link device, so as to accurately identify the delay of the white image in the link. For example, if there is a cached white image in the original device, the measurement of the white image will be inaccurate. The timing starts when the white image starts to be output. The camera shoots the display device and collects the captured image and transmits it back to the synthesis server. The timing ends when the synthesis server receives the first frame of white image. This period of time is the link delay time of the image output to the display device and then collected to the synthesis server through the camera.

[0022] S2. During the actual shooting process, a tracking device is fixed on the camera. The tracking data collected by the tracking device is transmitted to the virtual engine. The virtual engine renders an image based on the tracking data. The rendered image is transmitted to the synthesis server. The synthesis server divides the image into two uses. One use is to project it onto the screen of the display device. The picture on the display device screen is then photographed by the camera and transmitted back to the synthesis server. The other use is to transmit it through an extended path for extended synthesis. According to the link delay time measured by S1, the delay of the extended path is adjusted to synchronize the picture taken by the camera with the extended synthesized picture. The camera, synthesis server and tracking device are all genlock synchronized, that is, synchronously phase-locked, to ensure that the delay of the entire system is fixed.

[0023] In this embodiment, the tracking device is a redspy infrared camera tracker, which is fixed on the top of the camera and moves with the camera. The tracking data includes the camera's position (x, y, z), posture (pan, tilt, roll) and viewing angle (fov) information. The process of rendering the image by the virtual engine includes generating an image consistent with the camera's position, posture and viewing angle according to the tracking data.

[0024] In this embodiment, the display device includes an LED screen or a curtain. The virtual engine is Unreal Engine or Notch.

[0025] The technical solution of the present invention has the beneficial effects that the LED background is synchronized with the extension, and the camera movement ensures no tearing.

[0026] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0027] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A synchronization method applied to extended reality, characterized in that: The steps include: S1. Measure the link delay time of the image output to the display device and then collected by the camera to the synthesis server: the synthesis server outputs N frames of black images to the screen of the display device, and then outputs M frames of white images, where N and M are both integers greater than 1. The timing starts when the white image starts to be output. The camera shoots the display device and collects the shot image and transmits it back to the synthesis server. The timing ends when the synthesis server receives the first frame of white image. This period of time is the link delay time of the image output to the display device and then collected by the camera to the synthesis server; S2. During the actual shooting process, a tracking device is fixed on the camera. The tracking data collected by the tracking device is transmitted to the virtual engine. The virtual engine renders an image based on the tracking data. The rendered image is transmitted to the synthesis server. The synthesis server divides the image into two uses. One use is to project it onto the screen of the display device. The picture on the display device screen is then photographed by the camera and transmitted back to the synthesis server. The other use is to transmit it through an extended path for extended synthesis. According to the link delay time measured by S1, the delay of the extended path is adjusted to synchronize the picture taken by the camera with the extended synthesized picture. The camera, synthesis server and tracking device are all genlock synchronized, that is, synchronously phase-locked, to ensure that the delay of the entire system is fixed.

2. The synchronization method for extended reality according to claim 1, characterized in that: The tracking device is a redspy infrared camera tracker, which is fixed on the top of the camera and moves with the camera.

3. The synchronization method for extended reality according to claim 1, characterized in that: The tracking data includes the position (x, y, z), posture (pan, tilt, roll) and viewing angle (fov) information of the camera, and the process of rendering the image by the virtual engine includes generating an image consistent with the position, posture and viewing angle of the camera according to the tracking data.

4. The synchronization method for extended reality according to claim 1, characterized in that: The display device includes an LED screen or a curtain.

5. The synchronization method for extended reality according to claim 1, characterized in that: The virtual engine is Unreal Engine or Notch.

Citation Information

Patent Citations

  • Information processing apparatus and method thereof, and computer readable storage medium

    CN107766788A

  • Data processing method and system, server and storage medium

    CN112738534A

  • High definition surveillance video

    CN206100307U

  • Two-viewpoint stereoscopic image synthesizing method and system

    WO2016045425A1