A floating true 3D display multi-person gesture interaction system

Abstract

The application provides a suspended true 3D display multi-person gesture interaction system, which is composed of a light field 3D display screen, a suspended device, a gesture data acquisition device and a computer, and the system comprises three modules, namely a display module, an interaction module and a processing module. The system has the beneficial effect that whether each object can participate in interaction can be checked according to an interaction channel mechanism, and after the check, the coordinate systems of the suspended 3D image and the gesture are unified according to specific rules, so that accurate gesture interaction in multiple modes under the suspended true 3D display is realized. The system can realize independent interaction of multiple persons on multiple 3D objects, and can also realize local overall interaction of a scene or global overall interaction of a scene, and has a natural and real interaction experience.

Description

Technical Field

[0001] This invention relates to a human-computer interaction system, specifically a floating true 3D display multi-person gesture interaction system. Background Technology

[0002] Integrated imaging is a glasses-free true 3D display technology that, combined with levitation devices, can project 3D images into the air, achieving a suspended true 3D display. This display technology provides depth information not available in 2D displays, enhancing realism, without causing convergence-accommodation conflict in the human eye, effectively reducing visual fatigue. Human-computer interaction is the process of information exchange between humans and computers through specific languages. Natural interaction methods, especially gestures, applied to suspended true 3D displays can further leverage the natural and intuitive advantages of 3D displays, allowing for free interaction between people and suspended 3D images in space. Non-contact gesture interaction avoids unnecessary contact, particularly beneficial in reducing the risk of disease transmission, and has found various applications in the context of the COVID-19 pandemic since 2019.

[0003] However, most current gesture interactions for levitation true 3D displays can only achieve single-person, single-object gesture interaction or single-handed interaction of the entire scene, limiting the application scenarios of gesture interactions for levitation true 3D displays. To achieve simultaneous interaction by multiple people and independent interaction by multiple objects, the crosstalk problem urgently needs to be solved. Summary of the Invention

[0004] This invention proposes a floating true 3D display multi-person gesture interaction system, aiming to solve or partially solve the crosstalk problem existing in current floating true 3D display multi-person gesture interaction systems. The system consists of a light field 3D display screen, a levitation device, a gesture data acquisition device, and a computer, and includes three modules: a display module, an interaction module, and a processing module.

[0005] The display module is used to realize a true 3D floating display, including a light field 3D display screen and a levitation device.

[0006] The light field 3D display screen is used to achieve a true 3D display effect without glasses;

[0007] The levitation device is used to suspend the 3D image reproduced by the light field 3D display screen in the air, so as to realize a true levitation 3D display;

[0008] The interaction module consists of a gesture data acquisition device, which is used to collect gesture data from multiple users.

[0009] The processing module is a computer, used to receive gesture data from the interaction module and perform calculations required for interaction, render 3D source material, and drive the light field 3D display screen.

[0010] The spatial relationship between the modules of the system is as follows: the spatial area in which the interaction module effectively collects multiple user gesture data overlaps with the spatial area in which the display module displays 3D images, and the processing module is located outside the overlapping spatial area.

[0011] The system's gesture interaction process is as follows: multiple users make interactive gestures to the floating 3D image. The processing module reads the gesture data collected by the interaction module and checks whether each object can interact with the current hand according to the interaction channel mechanism. After the check is passed, the coordinate system of the floating 3D image and the gesture is unified according to a specific rule. Then, the gesture interaction method is further determined, and the response of each 3D object to the interactive gesture is specifically calculated. Finally, a new frame of 3D source material is rendered and sent to the display module. After passing through the display module, the floating 3D image undergoes changes such as translation, rotation, scaling, and pop-up windows compared to the previous frame, realizing the interaction closed loop.

[0012] Furthermore, the light field 3D display screen can be an integrated imaging 3D display screen, a multi-viewpoint 3D display screen, a holographic 3D display screen, a volumetric 3D display screen, etc.

[0013] Furthermore, the 3D source material corresponding to the integrated imaging 3D display screen, multi-viewpoint 3D display screen, etc., can be directly rendered by building a virtual camera array on development platforms such as OpenGL, Unity3D, and Unreal Engine, or it can be further obtained by processing the image array pixels through OpenCV and CUDA.

[0014] Furthermore, the levitation device can be planar or curved, single-layer or multi-layer, reflective or refractive, and more specifically, it can be a dihedral mirror array, a retroreflector array, an optical mirror, etc.

[0015] Furthermore, the gesture data acquisition device of the interaction module can be a commercially available motion controller such as Microsoft Kinect or Leap Motion, or a binocular or multi-view camera that integrates a human hand joint recognition module.

[0016] Furthermore, the gesture interaction methods are divided according to the interaction object, including independent interaction between multiple users and multiple objects, and overall interaction between multiple users and multiple objects.

[0017] Furthermore, the overall interaction between multiple users and multiple objects can be divided into scene-local overall interaction and scene-global overall interaction.

[0018] Furthermore, the interaction channel mechanism involves creating a container for each 3D object as an interaction channel to store the interaction status of the 3D object. Before each calculation of the 3D object's response to a gesture, the interaction channel of each 3D object is traversed and checked. Judgment conditions are added to determine which hand's gesture controls each object, avoiding interaction crosstalk and flexibly realizing independent interaction of multiple objects and overall interaction.

[0019] Furthermore, the added judgment condition is:

[0020] (1) If the interaction channel of an object is occupied by the ID of a hand other than the current hand, the object will not respond to the gesture of the current hand.

[0021] (2) The interaction channel of the object is occupied by the ID of the current hand, and the object will continue to be controlled by the current hand;

[0022] (3) If the interaction channel of the object is not occupied by any hand ID, then the current hand attempts to establish a new control and controlled relationship with the object and proceeds to the next judgment condition.

[0023] (4) If the hand ID is not in the interaction channel of other objects, put the hand ID into the interaction channel of the object and establish the corresponding control and controlled relationship.

[0024] (5) When multiple hands enter the interaction range of an object at the same time, they are assigned priority from high to low according to their IDs from small to large.

[0025] In particular, in both local and global scene interactions, a single hand can occupy multiple interaction channels, and multiple objects on the same interaction channel will produce the same response.

[0026] Furthermore, the coordinate system rule is implemented through six-point calibration. In calibration mode, the user uses a small ball as a calibration point, and can manually calibrate the ball at six standard points in 3D space to establish a coordinate mapping between the display space of the suspended 3D image and the interaction space of the user's hand, thereby achieving accurate gesture interaction.

[0027] Furthermore, the gesture interaction methods are categorized by the gestures themselves, including five types: grab and translate, swipe and rotate, two-finger pinch and zoom, tap to pop up a window, and global or partial translation / rotation of buttons within a scene. The responses of objects to these gestures include, but are not limited to, translation, rotation, scaling, brightness changes, and color changes.

[0028] The present invention proposes a floating true 3D display multi-person gesture interaction system, which has the advantage of enabling accurate gesture interaction in floating true 3D display. The gesture interaction can be independent interaction between multiple people and multiple 3D objects, or it can be a local-to-whole interaction or a global-to-whole interaction of the scene, providing a natural and realistic interactive experience. Attached Figure Description

[0029] The foregoing aspects and advantages of the present invention will become more apparent and readily understood from the following detailed description taken in conjunction with the accompanying drawings and embodiments, wherein:

[0030] Figure 1 This is a framework diagram of a floating true 3D display multi-person gesture interaction system according to this application.

[0031] Figure 2 A system structure diagram according to an embodiment of this application

[0032] Figure 3 This is a flowchart illustrating the gesture interaction process of a multi-person gesture interaction system for a suspended true 3D display according to this application.

[0033] Figure 4 Diagram of the interactive channel mechanism

[0034] Figure 5 This is a schematic diagram of a six-point coordinate unification method according to an embodiment of this application.

[0035] Figure 6 This application describes five interaction methods according to an embodiment of the present application, including gestures for selecting an object, performing an action, and releasing the object.

[0036] The figure labels in the above figures are:

[0037] 1 Display Module, 110 Integrated Imaging 3D Display Screen, 111 Dihedral Reflector Array, 112 Suspended 3D Image, 2 Interaction Module, 210 Leap Motion Controller, 3 Processing Module, 310 Computer, 410 Hand 1, 411 Hand 2, 412 Hand 3, 420 Suspended 3D Object 1, 421 Suspended 3D Object 2, 422 Suspended 3D Object 3, 430 Object 1 Interaction Channel, 431 Object 2 Interaction Channel, 432 Object 3 Interaction Channel, 510 OpenGL World Coordinate System, 520 Leap Motion Coordinate System, 530 Viewing Direction.

[0038] It should be understood that the above figures are only schematic and are not drawn to scale. Detailed Implementation

[0039] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings. Optional embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of this application.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention pertains. The terminology used herein in the specification of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0041] The following will describe in detail a floating true 3D display multi-person gesture interaction system proposed in this application, with reference to the embodiments and accompanying drawings disclosed in this application.

[0042] Figure 1 The diagram illustrates a framework of a floating true 3D display multi-user gesture interaction system according to this application. The system includes a display module 1, an interaction module 2, and a processing module 3, and forms an interactive closed loop through user viewing and gesture interaction. Figure 2 A system structure diagram according to an embodiment of this application is shown, the system comprising an integrated imaging 3D display screen 110, a dihedral mirror array 111, a Leap Motion motion controller 210, and a computer 310.

[0043] In one embodiment, the display module comprises an integrated imaging 3D display screen and a dihedral reflector array. The integrated imaging 3D display screen is placed horizontally, with the reproduced 3D image located directly above it. The dihedral reflector array, acting as a levitation device, is placed at a 45° angle, causing the reproduced 3D image to suspend in front of the system.

[0044] The 3D source material corresponding to the integrated imaging 3D display screen is rendered in real time using OpenGL. In OpenGL, a virtual camera array is built using the multi-viewport provided by glViewport(). Each viewport displays the image captured by a virtual camera. By setting the position of the viewport in the OpenGL rendering window, the images are stitched together to obtain the integrated imaging 3D source material.

[0045] The interaction module consists of a Leap Motion motion controller. The processing module is a portable computer with an Intel Core i5 Gen7 CPU and an Nvidia GeForce 1050 GPU. The parameters used in this embodiment illustrate that the system proposed in this application does not have stringent computer configuration requirements; in other possible embodiments of this application, higher computer hardware configurations can be used to achieve higher system performance.

[0046] Figure 3 This application presents a flowchart of a gesture interaction system for a floating true 3D display multi-person gesture interaction, which includes the following steps:

[0047] The first step is to read the gesture data collected by the interaction module and send it to the processing module when the user makes an interactive gesture on the floating 3D image.

[0048] In one embodiment, the system's interactive gestures include five types: grab and pan, swipe and rotate, two-finger pinch and zoom, tap pop-up window, and global or partial pan / rotation in button scenarios.

[0049] The second step is to check whether each object interacts with the current gesture based on the interaction channel mechanism.

[0050] In one embodiment, when a user performs independent interaction, the schematic diagram of the interaction channel mechanism is as follows: Figure 4 As shown.

[0051] The states of the hovering 3D objects 1 (420), 2 (421), and 3 (422) participating in the gesture interaction are stored in interaction channels 1 (430), 2 (431), and 3 (432), respectively. The hand identifier ID is a positive integer, which is automatically assigned when each hand enters the Leap Motion interaction range. In the figure, hovering 3D objects 1 (420) and 2 (421) are participating in the interaction, so interaction channels 1 (430) and 2 (431) store the IDs of hands 1 (410) and 3 (412) participating in the interaction, respectively. Since object 3 (422) is not participating in the interaction, the value stored in interaction channel 3 is -1, indicating that it is not occupied.

[0052] At this moment, hand 2 (411) attempts to interact with objects in the scene. Since interaction channels 1 (430) and 2 (431) are already occupied, hand 2 (411)'s interaction with suspended 3D objects 1 (420) and 2 (421) will be rejected, and it can only establish an interaction relationship with object 3 (422). On the other hand, since hand 3 (412) is interacting with suspended 3D object 2 (421), any gesture it makes will not have any effect on object 1 (420) and object 3 (422).

[0053] Furthermore, in situations where the user performs local-to-overall interaction or global-to-overall interaction within a scene, it is possible to use one hand to occupy multiple interaction channels, and multiple objects whose interaction channels are occupied by that hand will produce the same response, thus achieving overall interaction.

[0054] The third step is to unify the coordinates of the floating 3D image and the gesture.

[0055] The coordinate unification rule is established through coordinate mapping. Specifically, by calculating the coordinate mapping matrix between the display space of the floating 3D image and the interaction space of the user's gesture, the coordinates are unified under the same coordinate system through linear transformation. The display space of the floating 3D image is consistent with the OpenGL world coordinate system 510 of the rendering process. The interaction space of the user's hand is the spatial coordinate system defined by Leap Motion.

[0056] Figure 5 This diagram illustrates a six-point calibration coordinate unification method according to an embodiment of this application, unifying the coordinates of gestures captured by LeapMotion and the model coordinates rendered by OpenGL into the OpenGL world coordinate system 510. A small sphere is rendered, and during the six calibration processes, the sphere is placed at positions P0 to P5 of the OpenGL world coordinate system 510. After integrated imaging 3D source rendering and display by the display module, the six spheres appear to float sequentially in the air, perceived by the human eye. The center of the virtual camera array is located at point (0,0,Z0) in the OpenGL world coordinate system 510. T At the location, where Z0 is a real number.

[0057] The user pinches the six small balls in sequence with their hands. When Leap Motion detects the pinching motion, a weighted average of the distal phalanges of the thumb and index finger is recorded as P'. i (X i ,Y i Z i ) T (i = 0, 1, 2, ..., 5). According to P' i Calculate the scaling ratio K between the OpenGL world coordinate system 510 and the Leap Motion coordinate system 520.s for:

[0058]

[0059] Where d X ,d Y ,d Z These represent the distances from the calibration point to the origin O of the OpenGL world coordinate system 510 along the X, Y, and Z axes, respectively; P i (X i ,Y i Z i ) T (i = 0, 1, 2, 3, 4, 5) represents P' i The coordinates of the point in the OpenGL world coordinate system 510. The direction vectors of each axis of the OpenGL world coordinate system 510 in the Leap Motion coordinate space are:

[0060]

[0061] Based on the scaling ratio K calculated above s Given the three-axis direction vectors, calculate the free vector transformation matrix C:

[0062]

[0063] During the mapping process between the origin of the OpenGL world coordinate system 510 and the Leap Motion coordinate system 520, the average coordinates of the five points from P0 to P4 are used as the coordinates of the origin O of the Leap Motion coordinate system 520, denoted as (X... origin ,Y origin Z origin ) T The transformation matrix T mapping point O to the OpenGL world coordinate system 510 is calculated as follows:

[0064]

[0065] The coordinates of Leap Motion coordinate system 520 are represented as (X', Y', Z'). T In the OpenGL world coordinate system 510, coordinates are represented as (X, Y, Z). T Both conditions are met:

[0066]

[0067] Based on the above linear transformation formula, the Leap Motion coordinates are mapped to the OpenGL world coordinate system 510.

[0068] The fourth step is to determine the gesture interaction method.

[0069] Each gesture interaction can be divided into three steps: selecting an object, performing an action, and releasing the object. The object selection stage determines the interaction method; different interaction methods require different gestures, thus abstracting these into different judgment conditions. A gesture that meets the judgment conditions for a particular interaction method occupies the object's interaction channel, then proceeds to the action stage, where the object generates a response corresponding to that gesture. Finally, in the object release stage, if the hand action no longer meets the aforementioned judgment conditions, the object's interaction channel is released, it no longer responds to gestures, and enters a standby state until the next interaction gesture that meets the judgment conditions appears.

[0070] The judgment conditions are mainly divided into two categories: First, the spatial position of the hand satisfies the spatial position judgment condition, for example, the fingers touch the surface of the object; second, the hand movement satisfies the main characteristics of one of the preset interaction methods, for example, the index finger is extended while the other four fingers are curled. The decision is completed in the object selection stage. Only objects that meet the judgment conditions establish an interaction relationship with the hand and enter the next stage, whereby the response of the object to the interactive gesture is calculated.

[0071] Figure 6 The illustration shows gestures corresponding to selecting an object, performing an action, and releasing an object in five interaction modes according to an embodiment of this application.

[0072] The fifth step is to calculate the response of each 3D object to the interactive gesture.

[0073] Once an object is selected by a hand, its interaction channel is occupied by that hand, and a response corresponding to the interaction method expressed by that hand's gesture information occurs. Translation, rotation, and scaling are achieved by multiplying the object's model point-by-point by the same translation, rotation, and scaling transformation matrix. The translation distance, rotation angle, and scaling factor are determined by the distance the hand translates, the distance the index finger translates, and the distance the two fingers cover in a single opening and closing motion during the action phase, respectively.

[0074] The pop-up and closing of windows are implemented by whether or not the window model is rendered using OpenGL. The pop-up window is switched on and off by a tap gesture, and the action phase is completed in an instant. If the direction and speed of the tap meet the conditions, the window will be rendered in the next frame and the window will remain in that state if the window was not rendered in the previous frame, and vice versa.

[0075] Button interaction consists of two steps. The first step is finger interaction with the button. When a finger touches the button, it is considered selected. After a "press" action is performed, the button will descend in the same direction until it touches the bottom after falling below a certain height, which is considered to trigger the button function. The second step is indirect interaction with objects through the button. That is, after the button function is triggered, multiple objects in the scene, either globally or locally, will respond in the same way, such as mechanical movement or changes in brightness. When the finger moves away from the space where the button is located or the button gesture is no longer performed, the button is released, and the objects do not respond to the hand.

[0076] Step 6: Render the new frame of the integrated imaging 3D source material.

[0077] The results of the transformations of the object are sent to the OpenGL shader for processing, rendering a new frame of integrated imaging 3D source material. After floating true 3D display, the 3D image undergoes changes such as translation, rotation, scaling, and pop-up windows compared to the previous frame, realizing an interactive closed loop.

Claims

1. A floating true 3D display multi-person gesture interaction system, characterized in that, The system comprises a display module, an interaction module and a processing module; the spatial relationship of the three modules is that the spatial region in which the interaction module effectively collects multiple user gesture data coincides with the spatial region in which the display module displays a 3D image, and the processing module is located outside the coinciding spatial region; The display module is used for realizing floating true 3D display and comprises a light field 3D display screen and a floating device, wherein the light field 3D display screen is used for realizing naked-eye true 3D display effect, and the floating device is used for floating the 3D image reproduced by the light field 3D display screen in the air to realize floating true 3D display; the interaction module is composed of a gesture data collection device and is used for collecting gesture data of multiple users; the processing module is used for receiving gesture data transmitted by the interaction module, performing calculation required for interaction, rendering a 3D slice source and driving the light field 3D display screen; the system interaction process is that multiple users make interactive gestures on a floating 3D image, the processing module reads gesture data collected by the interaction module, checks whether each object can interact with the current hand according to an interaction channel mechanism, judges a gesture interaction mode when an object can interact with the current hand, the gesture interaction mode comprises an interaction mode divided according to an interactive object and an interaction mode divided according to a gesture, calculates responses of each 3D object to the interactive gesture, the responses comprise translation, rotation, scaling, brightness change and color change, finally renders a new frame of 3D slice source and transmits the 3D slice source into the display module to realize an interactive closed loop; The interaction channel mechanism is that a container is opened for each 3D object as an interaction channel to store a situation in which the 3D object participates in interaction, and before calculating the response of each 3D object to the gesture each time, the interaction channel of each 3D object is checked to increase a judgment condition to determine which hand gesture controls each object; The judgment condition for determining which hand gesture controls each object is as follows: (1) if the interaction channel of the object is occupied by the ID of a hand other than the current hand, the object will not respond to the gesture of the current hand; (2) if the interaction channel of the object is occupied by the ID of the current hand, the object will keep accepting control of the current hand; (3) if the interaction channel of the object is not occupied by the ID of any hand, the current hand tries to establish a new control and controlled relationship with the object, and enters the next judgment condition; (4) if the ID of the current hand is not in the interaction channel of other objects, the ID of the hand is put into the interaction channel of the object to establish a corresponding control and controlled relationship; (5) when multiple hands enter the interaction range of the object at the same time, the hands are assigned from high to low priority according to the ID from small to large; In the case of local overall interaction of a scene and global overall interaction of the scene, a single hand is allowed to occupy multiple interaction channels, and multiple objects whose interaction channels are occupied by the hand produce the same response.

2. The system of claim 1, wherein the system is configured to: In the display module, the light field 3D display screen is an integrated imaging 3D display screen, a super multi-view 3D display screen, a holographic 3D display screen or a volume 3D display screen; and the floating device is planar or curved, single-layer or multi-layer, reflective or refractive.

Images