A method and system for shooting extended reality

Abstract

The present invention provides an extended reality shooting method and system, the method comprising the following steps: using the PTP method to synchronize the reference clock of the virtual screen display end and the image acquisition end; using the virtual screen display end to alternately display the virtual screens inputted from multiple channels according to a preset frequency and sequence, the refresh frequency of the displayed image is 1 / n times the acquisition frame rate of the image acquisition end, n is a positive integer, n=1,2,3...same value or multiple, and each round of alternately displayed images contains at least one pure color image; using the image acquisition end to acquire the image displayed by the virtual screen display end, and separating the acquired images. Using the scheme provided by the present invention, the image acquisition end can obtain a mixed image of multiple virtual images in one acquisition, and then the acquired images can be separated by the image acquisition end to obtain the separated rendering images and pure color images, so that the shooting images of the extended reality shooting system can be separated for post-processing.

Description

Technical Field

[0001] The present invention relates to the technical field of film and television work processing, and in particular to an extended reality shooting method and system. Background Art

[0002] XR (Extended Reality), also known as "extended reality technology", refers to a combination of real and virtual, human-computer interactive environment generated by computer technology and wearable devices. It is a general term for various forms such as AR, VR, and MR. At present, XR technology has been widely used in the field of virtual broadcasting. Its application principle is: during the shooting process, through camera tracking and real-time image rendering technology, the display medium (such as LED screen) and the virtual scene outside the display medium are integrated and shot in real time, and the camera's real-time portrait is extended according to the camera's perspective and spatial parameters, thereby creating an infinite sense of space. However, when the current shooting system based on the XR principle is shooting in real time, the picture captured by the camera is already a composite picture of the person + the real picture. The person in this composite picture cannot be separated for post-processing. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects of the prior art and provide an extended reality shooting method and system. By adopting the solution provided by the present invention, the shooting picture of the extended reality shooting system can be separated for post-processing.

[0004] A first aspect of the present disclosure relates to an extended reality shooting method, the method comprising the following steps:

[0005] S1. Use PTP to synchronize the reference clocks of the virtual screen display terminal and the image acquisition terminal.

[0006] S2. The virtual images input from multiple channels are alternately displayed according to a preset frequency and sequence through the virtual image display terminal. The refresh frequency of the displayed image is 1 / n times the acquisition frame rate of the image acquisition terminal, where n is a positive integer, n=1, 2, 3...same value or multiple, and each round of alternately displayed images contains at least one pure color image;

[0007] S3. The image displayed by the virtual picture display end is collected by the image collection end, and the collected pictures are separated.

[0008] In some embodiments according to the first aspect of the present disclosure, the virtual image display end includes an LED splicer, an LED display screen and a frequency controller, wherein the frequency controller is used to generate a control frequency; the LED splicer is used to adjust the display order and display frequency of the multiple virtual images according to the control frequency, so that each virtual image is alternately output to the LED display screen according to a preset order and frequency, and the image output frame rate of the LED splicer is 1 / n times the acquisition frame rate of the image acquisition end, where n is a positive integer, n=1,2,3...same value or multiple; the LED display screen is used to display the received image, and the refresh frequency of the LED display screen is equal to the output frame rate of the LED splicer.

[0009] In some embodiments according to the first aspect of the present disclosure, the image acquisition end includes a camera and an image processor, wherein the camera is used to acquire the image displayed by the LED display screen; and the image processor is used to separate the image acquired by the camera based on the control frequency.

[0010] A second aspect of the present disclosure relates to an extended reality shooting system, the system comprising: a rendering server, a virtual picture display terminal, an image acquisition terminal and a PTP server;

[0011] The rendering server is used to generate a rendered virtual picture;

[0012] The PTP server uses the GPS clock as an external synchronization signal and adopts the PTP method to synchronize the reference clocks of the virtual screen display end and the image acquisition end system;

[0013] The virtual image display terminal is used to alternately display the virtual images input from multiple channels according to a preset frequency and sequence, and the refresh frequency of the displayed image is 1 / n times the acquisition frame rate of the image acquisition terminal, where n is a positive integer, n=1, 2, 3...same value or multiple, and each round of alternately displayed images contains at least one pure color image;

[0014] The image acquisition end is used to acquire the image displayed by the virtual image display end and separate the acquired images.

[0015] In some embodiments according to the second aspect of the present disclosure, the virtual image display end includes an LED splicer, an LED display screen and a frequency controller; wherein the LED splicer is used to adjust the display order and display frequency of the multiple virtual images according to the control frequency, so that each virtual image is alternately output to the LED display screen in a preset order and frequency, and the image output frame rate of the LED splicer is 1 / n times the acquisition frame rate of the image acquisition end, n is a positive integer, n=1,2,3...same value or multiple; the LED display screen is used to display the received image, and the refresh frequency of the LED display screen is equal to the output frame rate of the LED splicer.

[0016] In some embodiments according to the second aspect of the present disclosure, the image acquisition end includes a camera and an image processor; wherein the camera is used to acquire images displayed by the LED display screen; and the image processor is used to separate the images acquired by the camera based on the output frame rate of the LED splicing device.

[0017] In order to be able to post-process the captured images, the director needs to capture at least two synchronously displayed images of different contents, of which at least one image is a pure color image, and the rest are rendered virtual images. Existing XR systems are unable to separate the captured images for post-processing during real-time shooting, but the present invention enables the image acquisition end to obtain a mixed image of multiple virtual images in one acquisition by alternately displaying virtual images from different input paths at a very high frequency on the virtual image display end, and setting the refresh frequency of the virtual image display end to be the same value or a multiple of the acquisition frame rate of the image acquisition end. The captured images are then separated by the image acquisition end, and the separated rendered images and pure color images can be obtained for post-processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0019] Figure 1 The invention schematically shows an extended reality shooting system according to an embodiment of the present disclosure;

[0020] Figure 2 Schematically shows Figure 1 A specific implementation of an extended reality shooting system is shown. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and

[0022] It is not intended to limit the invention.

[0023] Those skilled in the art will appreciate that, unless otherwise stated, the terms "first", "second", etc. used herein are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the present invention, "plurality" means more than two, unless otherwise clearly and specifically defined.

[0024] Those skilled in the art will appreciate that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0025] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0026] Those skilled in the art can understand that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0027] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless specifically defined as in the embodiments of this application.

[0028] The present disclosure provides an extended reality shooting method, the method comprising the following steps:

[0029] S1. Use PTP to synchronize the reference clocks of the virtual screen display terminal and the image acquisition terminal.

[0030] S2. The virtual images input from multiple channels are alternately displayed according to a preset frequency and sequence through the virtual image display terminal. The refresh frequency of the displayed image is 1 / n times the acquisition frame rate of the image acquisition terminal, where n is a positive integer, n=1, 2, 3...same value or multiple, and each round of alternately displayed images contains at least one pure color image;

[0031] S3. The image displayed by the virtual screen display end is collected by the image collection end, and the collected images are separated based on the refresh frequency of the virtual screen display end.

[0032] By adopting the shooting method proposed in the present disclosure, the image acquisition end can obtain a mixed image of multiple virtual images in one acquisition. Then, the image acquisition end separates the acquired images to obtain separated rendering images and pure color images for post-processing.

[0033] Figure 1 An exemplary shooting system for implementing the above method is given, which mainly includes: a PTP server, a frequency controller, multiple rendering servers, an LED screen, an LED splicer, a high-speed camera and an image processor.

[0034] The PTP server uses the GPS clock as an external synchronization signal and the PTP method to synchronize the system's reference clock. It distributes the reference time T0 to each device in the system (including LED screens, LED splicers, high-speed cameras, and image processors), so that the entire system is synchronized based on the PTP time, and T0 is the system's reference time.

[0035] The frequency controller is used to control the output frame rate of the LED splicing device and the frame rate synchronization of the image processor to ensure the accurate synthesis and separation of multiple images.

[0036] The rendering server is used to generate a rendered virtual image, and the LED splicing device is used to receive the virtual image, and then output the corresponding image to the LED display screen for display according to the preset output frame rate. The output frame rate of the LED splicing device is equal to the display image refresh rate of the LED screen, and is equal to 1 / n times the camera acquisition frame rate, where n is a positive integer, n times, n=1,2,3….

[0037] The camera collects the mixed images displayed on the LED display and then outputs them to the image processor. The image processor extracts the corresponding images from the acquired image cache according to the preset time frequency to separate the mixed images. The separation frequency is equal to the output frame rate of the LED splicing device. The separated output content is an independent and complete picture content.

[0038] Through the embodiments of the present disclosure, we can achieve the technical purpose of obtaining multiple image display contents in real time when shooting an LED screen with a camera, and can separate the images for later use.

[0039] Reference Figure 1 A video recorder for recording the captured content may also be added to the shooting system described in the embodiment of the present disclosure. The output signal of the camera may be directly output to the image processor or may be output to the video recorder, recorded first and then output to the image processor for image separation processing.

[0040] Those skilled in the art will appreciate that the rendering server in the embodiments of the present disclosure is not limited to Figure 1 The two shown can be used as long as the LED splicing device can support multi-channel signals.

[0041] Those skilled in the art can understand that the rendered image input to the LED splicing device in the disclosed embodiment can be an image of any frame rate, as long as the refresh rate is supported by the camera, and the frame rate of the input rendered image is only used as a fixed frequency switching display when the LED display screen displays the image.

[0042] Reference Figure 2 ,yes Figure 1 First, various technologies involved in this implementation are introduced to fully understand the technical solutions described in the embodiments of the present disclosure.

[0043] 1. SDI: The SDI interface is a "digital component serial interface", while the HD-SDI interface is a broadcast-grade high-definition digital input and output port, where HD stands for high-definition signal. Since the SDI interface cannot directly transmit compressed digital signals, the compressed signals recorded by digital video recorders, hard disks and other devices must be decompressed and output through the SDI interface after playback before entering the SDI system. Repeated decompression and compression will inevitably cause image quality to deteriorate and delay to increase. For this reason, digital video recorders and non-linear editing systems of various formats have their own interfaces for directly transmitting compressed digital signals.

[0044] Hz: Hertz is the unit of frequency in the International System of Units. It is a measure of the number of repetitions of a periodic change per second. Hertz is abbreviated as Hz. One vibration (or oscillation, fluctuation) per second is 1 Hz, or it can be written as times / second or cycles / second. It is named after the German scientist Hertz.

[0045] HDMI: High Definition Multimedia Interface (HDMI) is a fully digital video and sound transmission interface that can send uncompressed audio and video signals. HDMI can be used in set-top boxes, DVD players, personal computers, TVs, game consoles, integrated amplifiers, digital audio and TVs, etc. HDMI can send audio and video signals at the same time. Since audio and video signals use the same wire, it greatly simplifies the installation of system lines.

[0046] LED: LED display is a flat panel display composed of small LED module panels, which is used to display various information such as text, images, and videos.

[0047] LED electronic display screens integrate microelectronics technology, computer technology, and information processing, and have the advantages of bright colors, wide dynamic range, high brightness, long life, stable and reliable operation, etc. LED display screens are widely used in commercial media, cultural performance markets, stadiums, information dissemination, news releases, securities trading, etc., and can meet the needs of different environments.

[0048] 5. PTP: PTP (Precision Time Protocol) is a protocol for synchronizing clocks over a network. When supported by hardware, PTP can achieve sub-microsecond accuracy, which is higher than NTP (Network Time Protocol).

[0049] Reference Figure 2 The system uses two Huawei laptops to connect the LED large-screen splicing device, outputs two 60Hz different display video contents with a resolution of 1920*1080, and the LED large-screen splicing device uses the Shicheng H15. The LED screen supports a maximum refresh rate of 120Hz. The signals of the two computers are connected to the LED splicing device through the HDMI interface. The LED display screen uses Xiuhu P2.6, and the screen refresh rate can be as high as 2840Hz. The LED screen and the LED splicing device are connected through a network cable. The shooting camera uses Sony's FS7, the shutter is adjusted to 120 / 1 second, and the recording format is adjusted to 120 frames per second. The image processor and the camera are connected using a BNC coaxial cable, and the signal is transmitted using the 12G-SDI mode. The image processor has four HDMI ports on the back end, which can output four independent image signals; the GPS clock used in the system is Beidou Video XBD611-XO CPCI-6U timing board, PTP server is HPZ4, network switch is Huawei S5735S, image processor is the company's self-developed XRCG-2U, frequency controller is installed on PTP server HPZ4 for use, and video recorder uses BMD12G.

[0050] Figure 2 The workflow of the system shown is:

[0051] Open two laptops and the LED screen splicer. Use network cables to connect the devices. Use PTP to initialize and synchronize the system clock time. Open the frequency controller setting interface and set the total output frequency to 120Hz, the output patch to 2, and the image output frequency to 60Hz. After the setting is completed, the LED screen displays the image ghosting of the two laptop output images A / B. The rendering server and the LED screen splicer have the same time base. The relative timing delay value of the two output ports is 8.33 milliseconds. After the setting is completed, the LED will display image A at 8.33 milliseconds and image B at the next 8.33 milliseconds. The images are displayed alternately. Since the frequency of alternating images is very high, the LED screen displays a mixed image. Use a camera to shoot the LED screen to display mixed images. Set the camera shutter to 120 / 1 second and the recording format to 120 frames per second. The camera signal can be directly output to the image processor or to the video recorder. Record first and then do post-processing. At this time, by watching the two signals output by the image processor, we can find that they are two independent image signals with an image refresh rate of 60Hz. Using this technology, different display contents can be captured simultaneously on a single LED screen, realizing the requirement of separate mixed shooting of different display contents on the same screen by a single machine.

[0052] In the description of the present invention, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. An extended reality shooting method, characterized in that: include: S1, using PTP to synchronize the reference clocks of the virtual image display end and the image acquisition end; the virtual image display end includes an LED splicer, an LED display screen and a frequency controller; the image acquisition end includes a camera and an image processor; S2. The LED splicer adjusts the display order and display frequency of the received multiple virtual images according to the control frequency generated by the frequency controller, so that each virtual image is alternately output to the LED display screen according to a preset order and frequency, and each round of alternately displayed images contains at least one pure color image; the output frame rate of the LED splicer is equal to the display image refresh rate of the LED display screen, and the output frame rate of the LED splicer is equal to 1 / n times the acquisition frame rate of the camera, where n is a positive integer, n=1, 2, 3…; S3. Capturing the images displayed on the LED display screen through the camera, and separating the images captured by the camera based on the control frequency through the image processor.

2. An extended reality shooting system, characterized in that: include: Rendering server, virtual screen display terminal, image acquisition terminal and PTP server; The rendering server is used to generate a rendered virtual picture; The PTP server uses the GPS clock as an external synchronization signal and adopts the PTP method to synchronize the reference clocks of the virtual image display end and the image acquisition end system; the virtual image display end includes an LED splicer, an LED display screen and a frequency controller; the image acquisition end includes a camera and an image processor; The LED splicing device adjusts the display order and display frequency of the received multiple virtual images according to the control frequency generated by the frequency controller, so that each virtual image is alternately output to the LED display screen according to the preset order and frequency, and each round of alternately displayed images contains at least one pure color image; the output frame rate of the LED splicing device is equal to the display image refresh rate of the LED display screen, and the output frame rate of the LED splicing device is equal to 1 / n times the camera acquisition frame rate, n is a positive integer, n=1,2,3...; the camera acquires the image displayed on the LED display screen, and separates the image acquired by the camera through the image processor based on the control frequency.

Images