A 5D-based virtual film and television interaction and motion capture system

Abstract

The application relates to the technical field of virtual reality, and relates to a 5D-based virtual film and television interaction and motion capture system. The 5D-based virtual film and television interaction and motion capture system specifically comprises a human body recognition device, a display device, a data processing device and an environment rendering device. The 5D virtual film and television interaction and motion capture system based on double Kinect sensor fusion is established, the motion information capture functions of the human hand and the human leg with prominent human body movement characteristics are respectively established, the detection precision is improved, a passive infrared sensor is additionally arranged, the human body is recognized and positioned in a relatively dark environment, a light control technology is established, the light brightness is automatically adjusted according to the dynamic projection environment scene change, the light changes according to the change of the content of the animation, and the self-defined light control system comprises touch screen control and handheld wireless remote control, so that the experience of participants is improved.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, with IPC classification number G09B5 / 06, and specifically to a 5D-based virtual film and television interaction and motion capture system. Background Technology

[0002] With the rapid development of computer technology, traditional 3D technology can no longer meet people's higher demands for film and television products. Based on this, 5G projection technology, utilizing projection technology and visual recognition, can better break down the boundaries between art, science, and technology, blurring the boundaries between real space and projection, immersing viewers in the themed space, and allowing each viewer to experience a unique visual experience. However, at present, 5G technology still has problems such as insufficient motion capture sensitivity, inability to react instantly to human movement in a scene, and inability to change images in a timely manner.

[0003] Patent CN201911244297 provides a motion learning system and method based on room-based interactive projection. It combines an RGB camera with multiple projection devices to create an immersive virtual environment for motion learning. Users observe their own body movements through mirrors, and the RGB camera captures these movements to project images onto a wall. However, RGB cameras are highly sensitive to distance; the closer the physical distance, the higher the accuracy of the depth data, and the farther away, the lower the accuracy. Therefore, in large spaces with added mirrors, the positioning accuracy of RGB cameras is low, and the sensitivity of projection changes is not high, affecting the user experience.

[0004] Patent CN201410553976 provides an interactive holographic illusion teaching system and method. This patent projects 3D onto a black screen placed in a space, and uses a motion-sensing controller to capture human movements within the space and interact with the virtual illusion. However, the Kinect motion-sensing controller described in this patent has certain requirements for lighting. In a completely enclosed environment, relying solely on the light projected by the projector, it will produce image information with a lot of noise, making it impossible to accurately locate the human body's position coordinates, and still causing poor interactivity due to inaccurate recognition.

[0005] Therefore, in response to some of the problems existing in the current 5D virtual reality technology, there is an urgent need to launch a 5D-based virtual film and television interaction and motion capture system. By building 5D-based film and television scenes and improving the accuracy of sensor recognition, the viewing experience of the audience can be improved. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a 5D-based virtual film and television interaction and motion capture system, specifically comprising: a human body recognition device, including a passive infrared sensor and a Kinect sensor, which collects human motion information and transmits it to a data processing device; an imaging device, including a projector, which constructs a dynamic projection environment and transmits the projection data to the data processing device; a data processing device, which acquires human motion information and projection data and performs interactive processing; and an environment rendering device, which performs dynamic environment rendering by building auxiliary hardware facilities.

[0007] Preferably, the passive infrared sensor detects the infrared wavelength emitted by the human body to determine whether the human body has entered the dynamic projection environment, transmits the data of the human body entering the dynamic projection environment to the data processing device, and activates the Kinect sensor.

[0008] Preferably, the Kinect sensor first establishes an image sample library based on human hands and legs, adds sample training labels to the image sample library, and trains on images of human hands and legs in the projection environment; extracts features of human hands and legs through training and performs coordinate mapping, and transmits the position coordinates obtained by coordinate mapping to the mouse control event module.

[0009] Preferably, the mouse control event module is configured such that when the position signal of a human body at a specified point in the dynamic projection environment changes, the changed position signal will serve as a mouse trigger signal to trigger the mouse control event module. At this time, the data processing device generates a return value, and the return value interacts with and changes with the position of the image information in the dynamic projection environment.

[0010] Preferably, there are two Kinect sensors, one mounted on the human hand activity area and the other on the human leg activity area; the Kinect sensor mounted on the human hand activity area can work independently to identify human hand activities; the Kinect sensor mounted on the human leg activity area can work independently to identify human leg activities; at the same time, the two Kinect sensors can work simultaneously to synchronously identify human hand and human leg activity areas.

[0011] Preferably, the Kinect sensor specifically includes an infrared transmitter, an infrared receiver, and an RGB camera; the infrared transmitter and receiver acquire depth data through infrared positioning, and the RGB camera acquires image information within the scene; the depth data and image information are combined to form three-dimensional coordinate data.

[0012] Preferably, the projector includes a ground projector, a wall projector, and an object projector; the display device also includes an LED display screen.

[0013] Preferably, the environment rendering device includes a speaker and a touch screen; the speaker sets corresponding sound effects according to the dynamic projection environment; the touch screen selects the environment and lighting mode according to the dynamic projection environment.

[0014] Preferably, the virtual film and television interaction includes lighting control technology, which automatically adjusts the brightness of the lights according to changes in the dynamic projection environment scene, and is also assisted by a custom lighting control system.

[0015] Preferably, the customized lighting control system includes touch screen control and handheld wireless remote control control.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] (1) This invention establishes a 5D virtual movie and motion capture system based on dual Kinect sensor fusion, and establishes motion information capture functions for human hands and legs with more prominent human motion characteristics, shortening the detection distance of Kinect sensors in space, thereby improving the measurement accuracy of Kinect sensors for depth information; at the same time, the motion information of human hands and legs can be operated separately to realize virtual movie and motion interaction with multiple functions.

[0018] (2) This invention adds a passive infrared sensor to identify the position information of the human body in the dynamic projection environment by detecting the human body temperature wavelength, thereby assisting the Kinect sensor to identify and locate the human body when the environment is relatively dark before the Kinect sensor is activated.

[0019] (3) This invention establishes a lighting control technology that automatically adjusts the brightness of the lights according to changes in the dynamic projection environment, making the lights change according to the content of the animation, thereby creating a more realistic and reasonable virtual interactive environment. At the same time, the custom lighting control system described in this patent includes touch screen control and handheld wireless remote control control, which are used to manually adjust the mode of the dynamic projection environment and the changes in scene lighting according to the participants' senses, thereby enhancing the participants' experience. Attached Figure Description

[0020] Figure 1 This is a diagram showing the components of a 5D-based virtual film and television interaction and motion capture system. Detailed Implementation

[0021] A 5D-based virtual film and television interaction and motion capture system specifically includes: a human body recognition device, comprising a passive infrared sensor and a Kinect sensor, which collects human motion information and transmits it to a data processing device; an imaging device, comprising a projector, which constructs a dynamic projection environment and transmits the projection data to the data processing device; a data processing device, which acquires human motion information and projection data and performs interactive processing; and an environment rendering device, which performs dynamic environment rendering by building auxiliary hardware facilities.

[0022] In a preferred embodiment, the 5D virtual movie interaction, by establishing a video processing technology based on multi-channel and multi-sensor fusion, transforms the movie screen from a single two-dimensional planar display to a three-dimensional screen or combined display screen, and, together with environmental effects that match the real scene, gives the participants an immersive feeling.

[0023] In one embodiment, the passive infrared sensor detects the infrared wavelength emitted by the human body to determine whether the human body has entered the dynamic projection environment, transmits the data of the human body entering the dynamic projection environment to the data processing device, and activates the Kinect sensor.

[0024] In a preferred embodiment, the virtual film and television interaction and motion capture system further includes three-dimensional virtual scene modeling technology, wherein a three-dimensional scene model can be built using a Unity3D virtual engine, and the three-dimensional scene model can be converted into a projection mode by establishing a camera-based position coordinate transformation. Compared with traditional projection video, this method is more realistic and three-dimensional through physical modeling.

[0025] In one implementation, the Kinect sensor first establishes an image sample library based on human hands and legs, adds sample training labels to the image sample library, and trains on images of human hands and legs in the projection environment; extracts features of human hands and legs through training and performs coordinate mapping, and transmits the position coordinates obtained by coordinate mapping to the mouse control event module.

[0026] In a preferred embodiment, the training is carried out by establishing a regularized convolutional neural network model to avoid inaccurate training due to overfitting during sample training. Specifically, the regularized convolutional neural network model inputs the original image into the input layer of the model, extracts the image features in the convolutional layer of the model, uses convolution kernels to convolve the data to obtain recombined data features, reduces the dimensionality of the recombined data features in the pooling layer, and extracts new data features.

[0027] In one embodiment, the mouse control event module is configured such that when the position signal of a human body at a specified point in the dynamic projection environment changes, the changed position signal will serve as a mouse trigger signal to trigger the mouse control event module. At this time, the data processing device generates a return value, and the return value interacts with and changes with the position of the image information in the dynamic projection environment.

[0028] In one embodiment, there are two Kinect sensors, one mounted on the human hand activity area and the other on the human leg activity area. The Kinect sensor mounted on the human hand activity area can work independently to identify human hand activities, and the Kinect sensor mounted on the human leg activity area can work independently to identify human leg activities. At the same time, the two Kinect sensors can work simultaneously to synchronously identify human hand and human leg activity areas.

[0029] In one embodiment, the Kinect sensor specifically includes an infrared transmitter, an infrared receiver, and an RGB camera; the infrared transmitter and receiver acquire depth data through infrared positioning, and the RGB camera acquires image information within the scene; the depth data and image information are combined to form three-dimensional coordinate data.

[0030] In one embodiment, the projector includes a ground projector, a wall projector, and an object projector; the display device also includes an LED display screen.

[0031] In a preferred embodiment, the display device is suitable for a full-screen holographic projection mode based on a projector, and also for a screen-based control mode based on an LED display screen. The screen-based control mode involves reasonably setting up the LED display screen and transmitting the data information collected by the passive infrared sensor and the Kinect sensor to the data processing device, through which the display mode of the LED display screen is interactively controlled.

[0032] In one embodiment, the environment rendering device includes a speaker and a touch screen; the speaker sets corresponding sound effects according to the dynamic projection environment; the touch screen selects the environment and lighting mode according to the dynamic projection environment.

[0033] In a preferred embodiment, the environmental rendering device further includes a fan, a fogging system, and a physical sensing control mode.

[0034] In one embodiment, the virtual film and television interaction includes lighting control technology, which automatically adjusts the brightness of the lights according to changes in the dynamic projection environment scene, and is also assisted by a custom lighting control system.

[0035] In one implementation, the custom lighting control system includes touchscreen control and handheld wireless remote control control.

Claims

1. A 5D-based virtual film and television interaction and motion capture system, characterized in that, Specifically, it includes: a human body recognition device, which includes a passive infrared sensor and a Kinect sensor to collect human motion information and transmit it to a data processing device; and an imaging device, which includes a projector to construct a dynamic projection environment and transmit the projection data to the data processing device. Data processing equipment acquires human motion information and projection data and performs interactive processing; Environment rendering equipment performs dynamic environment rendering by building auxiliary hardware facilities; The Kinect sensor first establishes an image sample library based on human hands and legs, adds sample training labels to the image sample library, and trains on images of human hands and legs in the projection environment; extracts human hand and leg features through training and performs coordinate mapping, and transmits the position coordinates obtained by coordinate mapping to the mouse control event module. The mouse control event module is configured such that when the position signal of a human body at a specified point in the dynamic projection environment changes, the changed position signal will be used as a mouse trigger signal to trigger the mouse control event module. At this time, the data processing device generates a return value, and the return value interacts with and changes with the position of the image information in the dynamic projection environment. The Kinect sensor consists of two parts, one mounted on the human hand's activity area and the other on the human leg's activity area. The Kinect sensor mounted on the human hand's activity area can work independently to identify human hand movements, and the Kinect sensor mounted on the human leg's activity area can work independently to identify human leg movements. At the same time, both Kinect sensors can work simultaneously to synchronously identify human hand and leg activity areas.

2. The 5D-based virtual film and television interaction and motion capture system according to claim 1, characterized in that, The passive infrared sensor detects the infrared wavelength emitted by the human body to determine whether the human body has entered the dynamic projection environment. It then transmits the data of the human body entering the dynamic projection environment to the data processing device and activates the Kinect sensor.

3. The 5D-based virtual film and television interaction and motion capture system according to claim 1, characterized in that, The Kinect sensor specifically includes an infrared transmitter, an infrared receiver, and an RGB camera. The infrared transmitter and receiver acquire depth data through infrared positioning, while the RGB camera acquires image information within the scene. The depth data and image information are combined to form three-dimensional coordinate data.

4. The 5D-based virtual film and television interaction and motion capture system according to claim 1, characterized in that, The projector includes ground projection, wall projection, and object projection; the display device also includes an LED display screen.

5. A 5D-based virtual film and television interaction and motion capture system according to claim 1, characterized in that, The environmental rendering device includes a speaker and a touch screen; the speaker sets corresponding sound effects according to the dynamic projection environment; the touch screen selects the environment and lighting mode according to the dynamic projection environment.

6. A 5D-based virtual film and television interaction and motion capture system according to claim 1, characterized in that, The virtual film and television interaction includes lighting control technology, which automatically adjusts the brightness of the lights according to changes in the dynamic projection environment, and is also assisted by a custom lighting control system.

7. A 5D-based virtual film and television interaction and motion capture system according to claim 6, characterized in that, The custom lighting control system includes touchscreen control and handheld wireless remote control control.

Images