A metaverse exhibition and display experience system based on air gesture operation

Abstract

The present invention discloses a metaverse exhibition display experience system based on air gesture operation, which belongs to the field of online virtual exhibition technology, and includes the following steps: the Kinect hardware intervenes in advance through the Kinect hardware to perceive the movements of the exhibitors and their arms, and the Kinect hardware includes a camera that can capture a panoramic view; the somatosensory controller intervenes for the second time, and LeapMotion emits detection rays through an infrared transmitter on its device, and collects the returned signals to generate a three-dimensional solid surface. The online exhibition hall in this embodiment improves the authenticity of the online exhibition hall, uses gesture and bone recognition sensing equipment to enhance the interactivity of the online exhibition hall, and adds a guided mode function for visiting the online exhibition hall, so as to realize the complementary advantages of the current online exhibition hall and offline exhibition hall.

Description

Technical Field

[0001] The present invention relates to the technical field of online virtual exhibitions, and in particular to a metaverse exhibition display experience system based on air gesture operation. Background Art

[0002] Since its development, exhibition halls have been deeply loved by people and accepted by more and more people. Different exhibition halls around the world have expanded rapidly. With the development of artificial intelligence and the rise of the metaverse, compared with the traditional offline exhibition halls with shortcomings such as "passive publicity and introduction", "limited by time and space", and "pure picture display", the online virtual exhibition hall that is open 24 hours a day shows greater advantages. It has interconnected data, intelligent operation, content-centric user immersive experience, and a visualization mode that is not restricted by time and space.

[0003] More importantly, it can solve the problem of passive marketing. At present, online exhibitions mainly use various app entrances to enter virtual exhibition halls, and visit online exhibition halls through keyboard, mouse or game controller devices. However, the key operation method adopted by the above devices makes visitors lose the natural interaction of human hands, and lacks the sense of substitution and immersion. The present invention provides a virtual exhibition hall system with a non-contact gesture operation method to realize the exhibition hall's guide function. Summary of the invention

[0004] The purpose of the present invention is to provide a metaverse exhibition display experience system based on air gesture operation. The online virtual exhibition hall has the natural interaction of human hands and has the effects of sense of substitution and immersion, so as to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The metaverse exhibition and display experience system based on air gesture operation includes Kinect hardware, a somatosensory controller, a computing unit and an imaging unit, and includes the following steps:

[0007] The Kinect hardware intervention is to intervene in advance through the Kinect hardware to perceive the movements of the exhibitors and their arms. The Kinect hardware includes a camera that can capture a panoramic view.

[0008] The somatosensory controller is involved for the second time. Leap Motion emits detection rays through an infrared transmitter on its device, collects the returned signals, and generates a three-dimensional solid surface.

[0009] The computing unit takes a same joint coordinate point according to the result of gesture recognition;

[0010] The imaging unit is used for providing attributes of a hand object to reflect the physical characteristics of a detected hand.

[0011] As a further solution of the present invention: wherein, the Kinect hardware analyzes the audience's head direction according to the audience's head position and facial integrity, and binds the head movement to the focused part of the display, and binds the hands to the focused view size, and the view size changes as the audience opens the telescope.

[0012] As a further solution of the present invention: wherein, the somatosensory controller will obtain the real coordinate space coordinates (x, y, z) of the entire hand in real time. If part of the hand is moved out of the effective interactive space, the KinectAzure device will immediately detect the invalid action and prompt the interactor to return to the effective area on the large screen.

[0013] As a further solution of the present invention: wherein, the somatosensory controller captures the minute differences between the hand gestures, uses AI recognition and analysis to separate the hand into the joints and bones of the human skeletal structure.

[0014] As a further solution of the present invention: wherein, the somatosensory controller intervenes for the second time, performs matching analysis on the pre-established motion model, obtains the similarity percentage of each pre-made model comparison, and determines whether the gesture trigger is successful based on the size of the percentage value.

[0015] As a further solution of the present invention: wherein, in the computing unit, the data of the first frame is the initial point coordinates, the data of the second frame is the end point coordinates, and then the left or right movement is determined according to the change of the same coordinate point in the gesture.

[0016] As a further solution of the present invention: wherein, the somatosensory controller assigns an "ID" indicator to the gesture data in the computing unit, and the indicator remains unchanged because the gesture exists within the visible range of the device. If the frame data is displaced, rotated, or scaled, Leap Motion will give a frame motion factor.

[0017] As a further solution of the present invention: wherein, the imaging unit, direction and palm normal direction are vectors describing the direction of the hand in the Leap Motion coordinate system.

[0018] A method for making a gesture for a metaverse exhibition experience based on air gesture operation comprises the following steps:

[0019] 1) Gesture interaction for exhibits. Scene interaction is based on gesture recognition. Visitors move their hands to the scanning button and wait for 3 seconds to trigger the robotic arm to start equipment maintenance.

[0020] After the scan is completed, the scene will prompt the part of the device that needs repair, and you can select and perform the repair operation through gestures;

[0021] 2) Gesture drawing of guided tour routes. Before the visit, the exhibition hall manager uses special gesture recognition to enter the exhibition hall area to draw the tour route, customize the tour route, and provide visitors with a reference for the order of touring the exhibition hall;

[0022] The special gesture uses two gestures, which are alternately held for 5 seconds to start drawing the visitor route, reducing the false trigger rate of entering the drawing mode;

[0023] After activating the virtual tour route setting function through the preset special gesture, the large screen will prompt you to move your palm in the effective detection area to control the tour route indicator cursor;

[0024] If you accidentally operate the tour route or change your mind and want to change the tour point during the process of setting the tour route, you can immediately make another cancellation gesture; the cancellation gesture is a model preset in the program. When the program detects the cancellation gesture, the cursor of the recorded tour route will become an erase cursor;

[0025] 3) Gesture virtual tour preview real scene, first when recording the tour route, move the control indicator cursor to the key tour point, and then use our preset trigger gesture to activate the real-time camera component of the key tour point currently approaching;

[0026] Then the large screen will directly display the real-time camera images of the currently selected key tourist spots, allowing the interactors to observe the actual situation in the museum in real time without arriving at the site, and allowing the interactors to experience the relevant exhibits in an immersive way.

[0027] As a further solution of the present invention: wherein, the erasing cursor only requires moving the hand back, and the cursor will erase the unnecessary route, and there is no need to record from the beginning.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The online exhibition hall in this embodiment improves the authenticity of the online exhibition hall, adopts gesture and bone recognition sensing equipment to enhance the interactivity of the online exhibition hall, and adds a guided mode function for visiting the online exhibition hall, so as to realize the complementary advantages of the current online and offline exhibition halls.

[0030] 2. The online exhibition hall in this embodiment uses some special operation means - gesture recognition and skeleton behavior recognition algorithm in the online virtual exhibition hall system to replace the conventional mouse and keyboard operation mode to visit the virtual exhibition hall.

[0031] 3. The online exhibition hall in this embodiment defines a special gesture to implement the exhibition hall tour route in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a block diagram of the B / S architecture connection in the present invention;

[0033] Figure 2 It is a human-computer interaction connection block diagram in the present invention;

[0034] Figure 3 A schematic diagram of the camera capturing viewing angle in the present invention;

[0035] Figure 4 This is a connection block diagram of the somatosensory controller in the present invention;

[0036] Figure 5 This is a schematic diagram of hand gesture joint markings in the present invention;

[0037] Figure 6 This is a schematic diagram of finger data information in the present invention;

[0038] Figure 7 This is a schematic diagram of the gesture recognition structure in the present invention;

[0039] Figure 8 It is a connection block diagram of the navigation interaction system in the present invention. DETAILED DESCRIPTION

[0040] See also Figure 1 As shown in the figure, online virtual exhibition halls generally adopt B / S architecture, and the mainstream clients currently include Chrome, Edge, and WeChat Webkit browsers. The server is placed on a cloud virtual host, and resource request services are provided through reverse proxy programs such as nginx or apache. After the visitor renders the multimedia resources in the client browser kernel, he or she uses the keyboard, mouse, or touch screen to perform interactive operations. The interactive interface generally clearly identifies several major function buttons, especially framework-level operations such as returning to the upper menu and entering the current selection scene, which will affect the aesthetics of the interface and the consistency of the overall style, and completely break the visual sense of immersive navigation.

[0041] When an important group visits, a guide is often sent to guide the tour offline. However, there are very few well-trained, responsible and professional guides. A few guides often give a few words and pass by the explanation of the scenery, which often leaves tourists with a perfunctory or dry impression. Due to the limited number of guides, especially foreign language explanations, it is difficult to provide every tourist with a good explanation.

[0042] like Figure 2As shown in the figure, when using tourist self-guided tour guides, in order to better serve the tourists, some exhibition halls will be equipped with electronic self-guided tour guides. These intelligent tour guides were popular in museums and tourist attractions in developed countries in the early years. In recent years, scenic spots and cultural and museum exhibition halls in my country have begun to become popular. With the increasing number of self-guided tours and individual tourists in domestic scenic spots and museums, voice guide services as a necessary new service facility have become a highlight of scenic spots and museum service methods. The latest national tourist scenic spot quality classification and assessment standards have included the ability to provide portable electronic voice explanations as a bonus item for upgrading to A, which is a bonus item for national 4A scenic spots and a necessary explanation service item for 5A scenic spots. However, this kind of tour guide method only relies on passively receiving the electronically synthesized or recorded sounds preset in advance by the tour guide machine, which gives participants a limited experience. Everyone experiences the same thing, lacks pertinence, and loses a lot of fun, so the real tour guide efficiency is not high.

[0043] In summary, the above-mentioned exhibition methods have the following disadvantages: traditional offline exhibition halls have the disadvantages of passive publicity and introduction, time and space constraints, and simple picture display. The interactive method of online exhibitions is single and based on the recognition technology of two-dimensional color images. The so-called two-dimensional color image refers to the two-dimensional static image obtained after the scene is photographed by an ordinary camera, and then the content in the image is recognized by computer graphics algorithms. Two-dimensional hand shape recognition can only recognize a few static gestures, and these gestures must be preset in advance, so the workload is large.

[0044] like Figure 3 As shown, based on the shortcomings of the above-mentioned prior art, a metaverse exhibition display experience system based on air gesture operation is now proposed, including the following steps: hardware intervention, through early intervention of Kinect hardware to perceive the movements of exhibitors and arms, the hardware device includes a camera that can capture a panoramic perspective, and after Kinect intervention, the perceived movements are three-dimensionally modeled.

[0045] In this embodiment, Kinect has a 1200w raw camera and depth sensor, which can achieve 4K display. Using Kinect's 3D reconstruction function, when the tourist walks in front of the screen, Kinect will intervene in advance, build a 3D model from the input data, accurately capture the human skeleton of the audience, transmit the signal to the screen, and wake up the whole system. Kinect will also capture some arm movements of the audience. For example, when the audience looks at the whole picture, he only needs to make a telescope gesture with his hand. In this process, the audience's head direction is analyzed according to the audience's head position and the integrity of the face. At the same time, the head movement is bound to the focus part of the display, and the hands are bound to the size of the focus image. The view size changes as the audience's telescope opens.

[0046] like Figure 4 As shown, the somatosensory controller intervenes for the second time, Leap Motion senses the specific hand gestures, and after Kinect wakes up the entire system, it captures the human body walking in front of the platform, and Leap Motion intervenes as the second level. Leap Motion sends out detection rays through an infrared transmitter on its device, and generates a three-dimensional solid surface after collecting the returned signals. At this time, after putting the hand into this part of the effective detection space, the Leap Motion sensor will obtain the real spatial coordinates (x, y, z) of the whole hand in real time. If part of the hand is moved out of the effective interactive space, the KinectAzure device will immediately detect the invalid action and prompt the interactor to return to the effective area on the large screen, realizing the seamless integration of somatosensory and precise gesture recognition.

[0047] like Figure 5 As shown in the figure, the detection and tracking of complete gestures can be achieved by completely tracking each joint of the hands and converting them into digital signals to be sent back to the background program for judgment. In order to distinguish as many gestures as possible and capture the subtle differences between each gesture, we use the AI ​​recognition and analysis function to split the hand into joints and bones at the human skeletal structure level, starting with the palm part, which is directly recognized as a sphere in the system.

[0048] in Figure 5 In the table, each number represents:

[0049]

[0050] like Figure 6 As shown in the figure, it is convenient to track the position of each hand in space. Then, different data objects are created for each finger according to the name of the finger. These objects contain information about the joint angle, phalange length, rotation angle, and spatial coordinates of the corresponding finger. This information will be stored in their respective buffers and compared with other time data on the timeline to obtain motion information such as acceleration and relative displacement.

[0051] Based on this information, we perform a matching analysis with the pre-established motion model to obtain the similarity percentage of each preset model comparison. We only need to judge the value of the percentage to determine whether the gesture trigger is successful.

[0052] Using 3D gesture recognition technology, 3D gesture recognition adds a Z-axis information, which can recognize various hand shapes, gestures and movements. 3D gesture recognition is also the main direction of gesture recognition development. However, this kind of gesture recognition containing certain depth information requires special hardware to implement, which is accomplished through customized industrial sensors and professional optical cameras to statistically analyze sample features and deep learning neural network technology.

[0053] like Figure 7 As shown, the operation unit takes the same joint coordinate point according to the result of gesture recognition. In this embodiment, the data of the first frame to the left of the "0" point in the figure (i.e., the center point of the wrist) is taken as the initial point coordinate, and the data of the second frame is taken as the end point target. Then, the left or right movement is determined according to the change of the same coordinate in the gesture.

[0054] The Leap Motion software assigns it a unique ID indicator. As long as the entity remains within the device's field of view, the ID indicator remains unchanged. The software analyzes the overall movement and, as long as the previous frame data has been shifted, rotated, scaled, etc., the Leap Motion program will give a frame motion factor based on the movement of that hand.

[0055] in:

[0056] 1) Rotation Axis, a direction vector to describe the rotation of the coordinate.

[0057] 2) Rotation Angle: The clockwise rotation angle relative to the rotation coordinate (Cartesian coordinate system).

[0058] 3) Rotation Matrix, a rotation matrix transformation.

[0059] 4) Scale Factor: A factor to describe expansion and contraction.

[0060] 5) Translation, a vector to describe linear motion.

[0061] The imaging unit, the hand object provides some attributes to reflect the physical characteristics of a detected hand

[0062] in:

[0063] The Hand object provides properties that reflect the physical characteristics of a detected hand.

[0064] 1. Palm Position: In the Leap Motion coordinate system, the coordinates of the center of the palm are measured in millimeters.

[0065] 2. Palm Velocity: the speed of the palm movement in millimeters per second.

[0066] 3. Palm Normal: The perpendicular vector to the plane formed by the palm, with the direction of the vector pointing to the inside of the palm.

[0067] 4. Direction: the vector from the center of the palm to the finger.

[0068] 5. Sphere Center, a sphere center that fits the inner arc of the palm. (Assuming you are holding a ball)

[0069] 6. Sphere Radius, same as above, this is the sphere radius. When the hand shape changes, the radius changes accordingly.

[0070] The direction and palm normal direction are vectors that describe the direction of the hand in the Leap Motion coordinate system.

[0071] Content production that matches gestures,

[0072] Step 1: Exhibit item gesture interaction. This embodiment takes the content scenario of virtual machine maintenance as an example. The scene interaction is based on gesture recognition. The visitor moves his hand to the scan button and waits for 3 seconds to trigger the robotic arm to start equipment maintenance. After the scan is completed, the scene prompts the part of the equipment that needs maintenance, and the maintenance operation is performed through gesture selection.

[0073] Step 2: Draw the guided tour route with gestures. Before the visit, the exhibition hall manager uses special gesture recognition to enter the exhibition hall area tour route drawing mode, customize the tour route, and provide visitors with a reference for the order of touring the exhibition hall.

[0074] The special gestures use the two gestures shown in the figure below. The two gestures are alternately maintained for 5 seconds to start drawing the visitor route, reducing the false trigger rate of entering the drawing mode.

[0075] After activating the air gesture function to set the virtual guided tour route through the preset special gestures, the large screen will prompt you to move your palm in the effective detection area to control the guided tour route indicator cursor. In the process of moving your hand, the sensor will continuously convert the real-space movement path of the hand into a guided tour route in the virtual guided tour three-dimensional program. During this process, the program will automatically adsorb and fit the distance between the hand indicator and each key guided tour point, and then generate regular straight lines to present on the large screen.

[0076] If you accidentally change the route or change your mind when setting the route, you can immediately make another gesture to cancel it. This gesture is also a model we preset in the program. When the program detects the cancel gesture, the cursor of the recorded route will become an erase cursor.

[0077] We just need to move our hand back and the cursor will erase the unwanted route without having to record again from the beginning.

[0078] After completing this set of processes, the program will record our tour route and generate relevant tour suggestions and precautions.

[0079] Step 3: Gesture virtual tour previews the real scene. When we set up the tour route, some visitors may have a strong interest in some exhibits in the museum, or be hesitant about which tour route to use. They want to experience the real appearance of the museum in advance and then make a choice that suits their heart.

[0080] At this time, the interactive air gesture guide software is connected to the real-time monitoring of key points in the museum to meet the need to preview relevant points when setting the tour route.

[0081] like Figure 8 As shown, first, when recording the tour route, move the control indicator cursor by hand to approach the key tour point, and then use our preset trigger gesture to activate the real-time camera component of the key tour point currently approaching.

[0082] Then the large screen will directly display the real-time camera images of the currently selected key tourist spots, allowing the interactors to observe the actual situation in the museum in real time without arriving at the site, and allowing the interactors to experience the relevant exhibits in an immersive way.

[0083] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A gesture making method for a metaverse exhibition experience system based on air gesture operation, comprising the following steps: (1) Gesture interaction for exhibits. Scene interaction is based on gesture recognition. Visitors move their hands to the scanning button and wait for 3 seconds to trigger the robotic arm to start equipment maintenance. After the scan is completed, the scene will prompt the part of the device that needs repair, and you can select and perform the repair operation through gestures; (2) Gesture-based tour guide drawing. Before the visit, the exhibition hall manager uses special gesture recognition to enter the exhibition hall area to draw the tour route, customize the tour route, and provide visitors with a reference for the order of touring the exhibition hall; The special gesture uses two gestures, which are alternately held for 5 seconds to start drawing the visitor route, reducing the false trigger rate of entering the drawing mode; After activating the virtual tour route setting function through the preset special gesture, the large screen will prompt you to move your palm in the effective detection area to control the tour route indicator cursor; If you accidentally operate the tour route or change your mind and want to change the tour point during the process of setting the tour route, you can immediately make another gesture to cancel it; the gesture of canceling is a model preset in the program. When the program detects the gesture of canceling, the cursor of recording the tour route will become an erase cursor; (3) Gesture virtual tour previews the real scene. First, when recording the tour route, move the control cursor by hand to approach the key tour point, and then use the preset trigger gesture to activate the real-time camera component of the key tour point currently approaching; Then the big screen will directly show the real-time camera images of the currently selected key tourist spots, allowing the participants to observe the actual situation in the museum in real time without actually arriving there, and to experience the relevant exhibits in person; The metaverse exhibition and display experience system based on air gesture operation includes Kinect hardware, a somatosensory controller, a computing unit and an imaging unit. It is characterized in that The steps include: The Kinect hardware intervention is to intervene in advance through the Kinect hardware to perceive the movements of the exhibitors and their arms. The Kinect hardware includes a camera that can capture a panoramic view. The somatosensory controller is involved for the second time. Leap Motion emits detection rays through an infrared transmitter on its device, collects the returned signals, and generates a three-dimensional solid surface. The computing unit takes a same joint coordinate point according to the result of gesture recognition; The imaging unit is used for providing attributes of a hand object to reflect the physical characteristics of a detected hand.

2. The gesture making method of the metaverse exhibition and display experience system based on air gesture operation according to claim 1, It is characterized in that The Kinect hardware analyzes the viewer's head direction according to the viewer's head position and facial integrity, binds the head movement to the focused part of the display, and binds the hands to the focused view size, and the view size changes as the viewer opens the telescope.

3. The gesture making method of the metaverse exhibition and display experience system based on air gesture operation according to claim 1, It is characterized in that The motion controller will obtain the real coordinate space coordinates (x, y, z) of the entire hand in real time. If part of the hand is moved out of the effective interactive space, the KinectAzure device will immediately detect the invalid action and prompt the interactor to return to the effective area on the large screen.

4. The gesture making method of the metaverse exhibition and display experience system based on air gesture operation according to claim 1 or 3, It is characterized in that The somatosensory controller captures the subtle differences between hand gestures, uses AI recognition and analysis, and separates the hand into the joints and bones of the human skeletal structure.

5. The gesture making method of the metaverse exhibition and display experience system based on air gesture operation according to claim 4, It is characterized in that The somatosensory controller intervenes for the second time to perform matching analysis on the pre-established motion model, obtain the similarity percentage of each pre-made model comparison, and determine whether the gesture trigger is successful based on the size of the percentage value.

6. The gesture making method of the metaverse exhibition and display experience system based on air gesture operation according to claim 1, It is characterized in that The computing unit uses the data of the first frame as the initial point coordinates and the data of the second frame as the end point coordinates, and then determines whether to move left or right based on the changes of the same coordinate point in the gesture.

7. The gesture making method of the metaverse exhibition and display experience system based on air gesture operation according to claim 1 or 6, It is characterized in that The somatosensory controller will assign an "ID" indicator to the gesture data in the computing unit. The indicator will remain unchanged because the gesture is within the visible range of the device. If the frame data is displaced, rotated, or scaled, LeapMotion will give a frame motion factor.

8. The gesture making method of the metaverse exhibition and display experience system based on air gesture operation according to claim 1, It is characterized in that The imaging unit, direction and palm normal direction are vectors describing the direction of the hand in the Leap Motion coordinate system.

9. The gesture making method of the metaverse exhibition and display experience system based on air gesture operation according to claim 1, It is characterized in that The erase cursor only requires moving the hand back, and the cursor will erase the unwanted route without the need to record from the beginning.

Images