Multi-mode interactive digital exhibition hall system

Abstract

The invention discloses a multi-modal interactive digital exhibition hall system, and the system comprises a data collection module which collects the multi-modal data of the operation of a user on an exhibit in a virtual space through a multi-modal perception interface of MR equipment; the data processing module is used for preprocessing the acquired multi-modal data to standardize a data format; the instruction identification module is used for performing instruction identification on the preprocessed multi-modal data to obtain a corresponding operation instruction; and the execution module is used for executing corresponding operation according to the obtained operation instruction to update the object state and behavior in the virtual scene. According to the method, the user operation is perceived through the multi-mode perception interface, the exhibit content is quickly obtained and understood, external interference is reduced, the immersion and interactivity are improved, and the exhibition effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-modal interactive digital exhibition hall system. Background Art

[0002] With the rapid development of information technology, digital exhibition halls have gradually become an important form of modern display and exhibition. Digital exhibition halls provide visitors with a new and immersive visiting experience by integrating a variety of advanced technologies, including mixed reality (MR), gesture recognition, voice control and handle interaction. The application of multimodal interactive technology not only enriches the presentation of exhibition content, but also can update exhibition content in real time, thereby enhancing the audience's participation and interactivity.

[0003] Although some technologies have been applied in digital exhibition halls, the existing solutions still have the following shortcomings:

[0004] Single interaction method: Most current digital exhibition halls only use a single interaction method, such as interaction only through touch screen or simple gesture control. This single interaction method limits the user's operational freedom and experience richness, and cannot fully meet the user's diverse interaction needs.

[0005] Insufficient immersion: Although mixed reality technology has enhanced the user's sense of immersion to a certain extent, the integration and interactivity of virtual content and real environment in the existing system still need to be improved. Users often feel a sense of separation between virtual and reality during the experience, which affects the overall immersive experience.

[0006] Imperfect technology integration: In the current digital exhibition hall system, the integration and application of technologies such as mixed reality, gesture recognition, voice control and handle interaction are not yet sufficient. The synergy between various technologies is not good, which limits the overall performance of the system and user experience. For example, the response speed and accuracy of gesture recognition and voice control are not high, which affects the smoothness of user operation.

[0007] Delayed content updates: Exhibition content in digital exhibition halls usually requires professional technicians to update and maintain, with a long content update cycle and poor flexibility. This delayed content update method is difficult to meet the rapidly changing exhibition needs and the audience's desire for fresh content. Summary of the invention

[0008] In view of the above technical problems, the present invention provides a multimodal interactive digital exhibition hall system, which provides a comprehensive and immersive visiting experience and realizes real-time updating of content and an effective user feedback mechanism.

[0009] An embodiment of the present invention provides a multimodal interactive digital exhibition hall system, which includes: a data acquisition module, which is used to collect multimodal data of a user's operation on an exhibit in a virtual space through a multimodal perception interface of an MR device; a data processing module, which is used to preprocess the collected multimodal data to standardize the data format; an instruction recognition module, which is used to perform instruction recognition on the preprocessed multimodal data and obtain corresponding operation instructions; and an execution module, which is used to execute corresponding operations according to the obtained operation instructions to update the state and behavior of objects in the virtual scene.

[0010] Optionally, the data acquisition module includes: a gesture acquisition module, including a sensor or camera built into the MR device, for capturing the user's hand image and joint position data in real time; a voice acquisition module, including a microphone, for collecting the user's voice signal; and a handle interaction module, for collecting the user's operating movements and posture data in real time.

[0011] Optionally, the data processing module includes: a first processing module, used to pre-process the hand images and joint point position data collected by the gesture acquisition module and convert them into gesture data; a second processing module, used to convert the collected voice signals into voice text data; and a third processing module, used to pre-process the collected operation action and posture data and convert them into operation data.

[0012] Optionally, the instruction recognition module includes: a gesture recognition module, used to match the hand data with a preset gesture template through a deep learning algorithm to determine specific gesture instructions; a voice recognition module, used to recognize the voice text data through a voice recognition model or a voice recognition algorithm to determine specific voice instructions; a handle recognition module, used to convert the operation data processed by the third processing module into predefined operation instructions through mapping.

[0013] Optionally, the system further comprises: a model disassembly and interaction module for disassembling, reassembling and observing preset exhibits in the virtual exhibition hall.

[0014] Optionally, the system further comprises: an exhibit interaction module, which is used for close interaction with the virtual exhibits to enlarge, reduce or rotate the exhibits.

[0015] Optionally, the first processing module is also used to extract key point data of the hand from the collected hand image, and convert the key point data into a skeleton model of the hand; the gesture recognition module is also used to extract the main features of the gesture according to the key features of the skeleton model, compare the extracted main features of the gesture with the samples in the predefined gesture library and perform gesture classification to identify the specific gesture.

[0016] Optionally, the key features include joint angles, finger lengths and relative position features, and the gesture classification method includes a pattern matching algorithm or a deep learning model method.

[0017] Optionally, the gesture recognition module is also used to extract key point data of the hand from the hand image, and convert the extracted key point data into a feature vector, which represents the spatial structure and dynamic information of the gesture; train a learning model, and use the labeled gesture data to train the learning model, and use the trained model to perform gesture recognition on the gesture data to identify specific gestures.

[0018] Optionally, the second processing module is also used to annotate the speech text data, indicate the intention and entity of each sentence, and use the annotated data set to train a language processing model, and obtain a trained speech recognition model for recognizing the speech text data.

[0019] In the technical solution provided by the embodiment of the present invention, multimodal data is collected by a data acquisition module, and preprocessed by a data processing module, and instruction recognition is performed on the preprocessed data to obtain corresponding operation instructions, and the state and behavior of objects in the virtual scene are updated through the operation instructions. Compared with the prior art, the present invention perceives user operations through a multimodal perception interface, quickly obtains and understands exhibit content, reduces external interference, and enhances immersion and interactivity, thereby improving exhibition effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a structural schematic diagram of the multimodal interactive digital exhibition hall system of the present invention.

[0021] Figure 2 It is a structural schematic diagram of creating and adding a virtual exhibition hall in the multimodal interactive digital exhibition hall system of the present invention.

[0022] Figure 3 It is a structural schematic diagram of the exhibition progress stage in the multimodal interactive digital exhibition hall system of the present invention.

[0023] Figure 4 It is a structural schematic diagram of the immersive exhibition experience in the multimodal interactive digital exhibition hall system of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0025] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0026] Please refer to Figure 1 As shown, the present invention provides a multimodal interactive digital exhibition hall system, which includes a data acquisition module 100, a data processing module 200, an instruction recognition module 300 and an execution module 400. The acquisition module 100 is used to collect multimodal data of user operations on exhibits in a virtual space through a multimodal perception interface of an MR device, the data processing module 200 is used to pre-process the collected multimodal data to standardize the data format; the instruction recognition module 300 is used to perform instruction recognition on the pre-processed multimodal data to obtain corresponding operation instructions; the execution module 400 is used to execute corresponding operations according to the obtained operation instructions to update the state and behavior of objects in the virtual scene.

[0027] The multimodal interactive digital exhibition hall system described in the present invention is a highly interactive virtual exhibition hall in which users can visit, operate and process various exhibits. The virtual exhibition hall environment simulates various elements of a real exhibition hall, making the visiting process more immersive. The present invention significantly improves the user's viewing interest and concentration through multimodal interaction and a highly immersive virtual exhibition hall environment, and the highly interactive viewing experience can stimulate the user's enthusiasm for active visits. The virtual exhibition hall and multiple interactive methods make the viewing content more vivid and specific, and users can understand and appreciate the exhibits more intuitively through actual operation and interaction.

[0028] In one embodiment of the present invention, the system further includes a model disassembly and interaction module for disassembling, reassembling and observing the preset exhibits in the virtual exhibition hall. This module enables users to understand complex concepts and structures more deeply through practical operations.

[0029] In one embodiment of the present invention, the system further includes an exhibit interaction module for interacting with virtual exhibits at close range and zooming in, out or rotating the exhibits. This module makes the exhibit display more flexible and helps users pay more attention to details and deeply understand the exhibits. The present invention provides physical collision and interactive simulation functions, allowing users to observe the real performance of exhibits in the virtual exhibition hall, such as collision reaction and force transmission. This immersive experience can enhance the user's memory of the exhibits.

[0030] The present invention makes the visiting process more vivid and interesting through the functions of exhibit disassembly, close-range exhibit interaction and physical collision simulation, and users can understand and remember the exhibits more deeply through interaction and practice.

[0031] The present invention significantly improves the shortcomings of the existing technology through modules such as multimodal interaction, virtual exhibition hall environment, model disassembly and physical collision, and provides an efficient, interactive and personalized digital exhibition hall system. Users can not only enjoy an immersive visiting experience, but also significantly improve the overall exhibition effect.

[0032] During the virtual exhibition hall creation stage, users log in directly to the system through mixed reality (MR) head-mounted devices to create virtual exhibition halls. Users enter the exhibition hall code through the head-mounted device to enter the virtual space. After the exhibition hall is created, the system will generate an exhibition hall code, and users can invite others to join the exhibition hall through the exhibition hall code. During the virtual exhibition hall stage, users communicate and interact in the virtual space through MR devices. The initial settings include basic elements such as the exhibition area, exhibit display area, and interactive screens. Visitors can express actions and voices through MR devices, and upload exhibit models, pictures, and videos to the cloud on the computer. In the virtual space, visitors can manipulate these exhibits and achieve real-time interaction.

[0033] In the virtual exhibition hall, all exhibits can be interacted with at close range. When users share exhibits in the virtual space, other visitors can view and operate these exhibits synchronously, realizing real-time collaboration. For example, when a user manipulates an exhibit model in an MR device, other visitors can also see and interact with it synchronously.

[0034] In one embodiment of the present invention, the data processing module is used to perform noise reduction, smoothing and normalization processing on the collected multimodal data.

[0035] In one embodiment of the present invention, the data acquisition module includes a gesture acquisition module, including a sensor or camera built into the MR device, for capturing the user's hand image and joint position data in real time; the data processing module includes a first processing module, for pre-processing the hand image and joint position data into gesture data; the instruction recognition module includes a gesture recognition module, for matching the hand data with a preset gesture template through a deep learning algorithm to determine a specific gesture instruction. The execution module is used to pass the recognized gesture instruction to the MR device through an API, and the MR device performs corresponding operations according to the gesture instruction, such as selecting or moving a virtual object.

[0036] In one embodiment of the present invention, the data acquisition module includes a voice acquisition module, the voice acquisition module includes a microphone for collecting the user's voice signal, the microphone may be a microphone of the MR device, the data processing module includes a second processing module for converting the collected voice signal into voice text data; the instruction recognition module includes a voice recognition module for identifying the voice text data through a voice recognition model or a voice recognition algorithm to determine a specific voice instruction; an execution module for passing the recognized voice instruction to the MR device through an API, and the MR device performs corresponding operations according to the voice instruction, such as starting functions and controlling virtual objects.

[0037] In one embodiment of the present invention, the data acquisition module includes a handle interaction module for real-time acquisition of user operation actions and posture data; the data processing module includes a third processing module for pre-processing the acquired operation actions and posture data into operation data; the instruction recognition module includes a handle recognition module for converting the operation data processed by the third processing module into predefined operation instructions through mapping, such as instructions for moving or rotating virtual objects. The execution module is used to pass the recognized operation instructions to the MR device through the API, and the MR device performs corresponding operations according to the operation instructions, such as selecting or moving virtual objects.

[0038] In one embodiment of the present invention, the gesture acquisition module includes a depth camera for real-time acquisition of three-dimensional depth data and hand images of the user's hand, including each key point of the hand (such as the position of the finger joints) and its coordinates in three-dimensional space. The first processing module is used to extract the key point data of the hand (such as fingertips, joints, etc.) from the three-dimensional depth data and the hand image. The three-dimensional depth data provided by the depth camera can be used to construct a skeleton model of the hand, mark the specific position and posture of the fingers, and convert the key point data into a skeleton model of the hand for further analysis of the gesture. The coordinates of each key point and the angle information between the fingers are extracted and combined into the overall structure of the hand. The gesture recognition module is also used to extract the main features of the gesture based on the features of the skeleton model such as the joint angle, finger length and relative position. The main features may include the degree of opening, curvature, rotation angle of the palm, etc. of the fingers. The extracted main features are compared with the samples in the predefined gesture library, and the gestures are classified using a pattern matching algorithm (such as dynamic time warping) or a deep learning model (such as a convolutional neural network) to identify the specific gestures made by the user. The recognized gesture instructions are mapped to specific operation commands (such as "grab", "zoom", "rotate", etc.). These instructions are passed to the virtual exhibition system or application through the device's API. The state of objects in the virtual scene is updated according to the operation commands, enabling interaction between users and virtual exhibits.

[0039] In one embodiment of the present invention, the gesture collection module is also used to collect a large number of gesture data sets, including images or video clips of various gestures. The data should cover various gesture samples, such as "grab", "zoom in", "rotate", etc. The data is labeled as the corresponding gesture category for training. The first processing module preprocesses the collected gesture image or video data, including image cropping, scaling, normalization, etc., to standardize the data format, or performs data enhancement on the data, such as rotation, translation, flipping, etc., to increase the robustness of the model. Use a deep learning model (such as a convolutional neural network CNN) or traditional computer vision technology to extract the key point data of the hand from the image. These key points include the joint positions of the fingers and the shape of the palm, etc. The extracted key point data is converted into a feature vector to represent the spatial structure and dynamic information of the gesture. Features include the length, angle, relative position, etc. of the fingers. Train the learning model, and use the labeled gesture data to train the learning model. During the training process, the model optimizes the parameters by minimizing the loss function (such as the cross entropy loss) to improve the recognition accuracy of various gestures. Use the trained model to perform gesture recognition on the gesture data to identify specific gestures.

[0040] During the training process, the validation dataset is used to evaluate the performance of the model and monitor metrics such as accuracy, recall, and F1 score. The test dataset is used to finally evaluate the generalization ability of the model on unseen data. The hyperparameters of the model (such as learning rate, batch size, regularization term, etc.) are adjusted to optimize the performance of the model and the stability of training. Based on the test results, the model may need to be fine-tuned or retrained to ensure the reliability of the model in practical applications. The trained model is integrated into the actual application for real-time gesture recognition. The model receives real-time gesture input from the user and outputs the recognition result through the inference process.

[0041] In one embodiment of the present invention, the second processing module is also used to annotate the speech text data, indicate the intention and entity of each sentence, and use the annotated data set to train the language processing model, and obtain the trained speech recognition model for recognizing the speech text data. The speech recognition module includes BERT, GPT of the Transformer architecture, or a special speech recognition model. The training process includes a speech recognition model (converting speech to text) and a natural language understanding model (parsing the intention of text instructions). The speech recognition module of the present invention can also recognize the intention in the text (such as "play music", "adjust the volume"). The data trained by the model is used to train the intention classifier to identify the user's needs. The recognized voice instructions are mapped to the actual operating system. For example, the "play music" instruction may correspond to starting a music player and playing a specified track. And the execution results are fed back to the user, for example, confirming the volume adjustment or the music played to ensure that the user is satisfied with the system response.

[0042] In one embodiment of the present invention, the hand interaction module includes a glove with built-in sensors for the user to wear, and these sensors can capture the three-dimensional position and movement of the hand, including the bending, extension and movement of the fingers and the palm. The sensor transmits the hand motion data to the third processing module wirelessly or wired. The third processing module processes the motion data captured by the sensor and identifies the specific movements and gestures of the hand through the handle recognition module. For example, by sensing the movement of the glove, the system can recognize gestures such as "grabbing", "rotating" and "dragging". According to the recognized hand movements, the system applies the corresponding operations to the virtual exhibits. For example, the user selects a virtual object through a "grabbing" gesture, adjusts the angle of the object through a "rotating" gesture, or moves the position of the object through a "dragging" gesture.

[0043] The present invention can also realize real-time interaction and collaboration. In the virtual space, the system realizes real-time collaboration between users through low-latency network communication technology and distributed computing architecture. The specific technical scheme is as follows: The MR device is connected to the cloud server via a Wi-Fi network to ensure high-bandwidth and low-latency signal transmission. The user's operations on the exhibits in the virtual space (such as rotation, scaling, and movement) will be captured in real time and transmitted to the cloud server through the network. The cloud server merges and synchronizes the operations of all users, and immediately feeds back the updated exhibit status to other visitors to ensure that they can view and operate the exhibits synchronously. The system uses a collaborative locking mechanism to ensure that when one user operates an exhibit, the operations of other users do not conflict, thereby realizing real-time collaboration.

[0044] The present invention also supports multiple users to interact and communicate synchronously in the virtual exhibition hall. Specifically, the present invention uses a multi-user synchronization framework based on WebRTC (real-time communication) to allow multiple users to connect to the virtual exhibition hall at the same time for real-time audio and video communication and data sharing. To ensure real-time performance, the system uses differential data transmission and local update mechanisms to only transmit changed data to reduce network bandwidth usage.

[0045] The present invention also aims to realize dynamic scene management. All interactive scenes in the virtual exhibition hall are managed in real time by the server. The user's operations and position changes will be instantly synchronized to the MR devices of all participants to maintain a consistent exhibition hall status.

[0046] The present invention can also realize real-time uploading and updating of exhibition content. Specifically, the system adopts a distributed cloud storage service, and users can upload exhibit resources (such as models, pictures, and videos) to cloud storage through a computer or MR device. The system provides a standardized API interface, allowing applications to call exhibit resources stored in the cloud in real time and automatically update exhibition content; users can upload files to the cloud and use the following technical means to ensure that they can be called and shared at any time during the exhibition. The uploaded files are encrypted and access is controlled through fine-grained permission management. Only authorized users can access and operate these files to ensure data security and confidentiality. The system integrates globally distributed CDN services to ensure that users from all regions can quickly and reliably access and download exhibition content. After the file is uploaded, the system will immediately synchronize and update to all users' MR devices to ensure the real-time and consistency of the content during the exhibition.

[0047] The multimodal interaction technology of the present invention is combined with the immersive exhibition experience, so that users can quickly obtain and understand the content of the exhibits, reduce external interference, and improve the exhibition effect. Users can choose appropriate interaction methods and exhibition modes according to their habits and needs, provide personalized exhibition experience, and conduct synchronous interaction and communication through MR devices, promote cooperation and communication between users, and enhance the team's viewing and collaboration capabilities.

[0048] Content preparation stage: Users upload exhibit resources, including models, pictures, texts and videos, through their computers. The content can be updated in real time to ensure the flexibility and real-time nature of the exhibition.

[0049] Creating and joining a virtual exhibition hall: Users log in to the system through the MR headset to create a virtual exhibition hall, enter the exhibition hall code to enter the virtual space, and generate an exhibition hall code to invite others to join. For details, please refer to Figure 2 shown.

[0050] Exhibition stage: please refer to Figure 3 As shown, the initial setting of the virtual exhibition hall displays the basic elements in the virtual exhibition hall (exhibition area, exhibit display area, interactive screen, etc.), users interact through MR devices, and scenes of users performing actions and voice expressions in the virtual space are displayed. Visitors manipulate exhibits in the virtual space and display the process of close-range exhibit interaction, such as dragging, rotating, and enlarging exhibits.

[0051] Close-range exhibit interaction: Users share exhibits in a virtual space, and other visitors can view and operate them simultaneously, achieving real-time collaboration.

[0052] Multimodal interaction technology: integrates gesture recognition, voice control and handle interaction technology to achieve a natural and convenient interactive experience.

[0053] Immersive exhibition experience: please refer to Figure 4As shown, the immersive exhibition environment shows the scene where users enter the virtual three-dimensional space to experience the exhibition. Users can enter the virtual three-dimensional space through MR devices, experience the exhibition process immersively, and perform interactive operations.

[0054] The present invention provides a new digital exhibition hall system through innovative multimodal interaction technology and mixed reality technology, which can effectively improve exhibition efficiency, enhance exhibition interest, provide personalized exhibition experience, support real-time collaboration, and enhance immersion and interactivity, bringing new changes and developments to the exhibition field.

[0055] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal interactive digital exhibition hall system, characterized in that: The system comprises: A data acquisition module, used to collect multimodal data of the user's operation on the exhibits in the virtual space through the multimodal perception interface of the MR device; A data processing module, used to pre-process the collected multi-modal data to standardize the data format; An instruction recognition module is used to perform instruction recognition on the preprocessed multimodal data and obtain corresponding operation instructions; The execution module is used to execute corresponding operations according to the acquired operation instructions to update the state and behavior of objects in the virtual scene.

2. The multimodal interactive digital exhibition hall system according to claim 1, characterized in that: The data acquisition module comprises: A gesture acquisition module, including a sensor or camera built into the MR device, is used to capture the user's hand image and joint position data in real time; A voice collection module, including a microphone, is used to collect the user's voice signal; The handle interaction module is used to collect the user's operation action and posture data in real time.

3. The multimodal interactive digital exhibition hall system according to claim 2, characterized in that: The data processing module comprises: A first processing module, used for pre-processing the hand image and joint point position data collected by the gesture collection module and converting them into gesture data; A second processing module, used for converting the collected voice signal into voice text data; The third processing module is used to pre-process the collected operation action and posture data and convert them into operation data.

4. The multimodal interactive digital exhibition hall system according to claim 3, characterized in that: The instruction recognition module comprises: A gesture recognition module, used to match the hand data with a preset gesture template through a deep learning algorithm to determine a specific gesture instruction; A speech recognition module, used to recognize the speech text data through a speech recognition model or a speech recognition algorithm to determine a specific speech instruction; The handle recognition module is used to convert the operation data processed by the third processing module into predefined operation instructions through mapping.

5. The multimodal interactive digital exhibition hall system according to claim 1, characterized in that: The system further comprises: The model disassembly and interaction module is used to disassemble, reassemble and observe the preset exhibits in the virtual exhibition hall.

6. The multimodal interactive digital exhibition hall system according to claim 1, characterized in that: The system further comprises: The exhibit interaction module is used to interact with virtual exhibits at close range and to enlarge, reduce or rotate the exhibits.

7. The multimodal interactive digital exhibition hall system according to claim 3, characterized in that: The first processing module is also used to extract key point data of the hand from the collected hand image and convert the key point data into a skeleton model of the hand; the gesture recognition module is also used to extract the main features of the gesture based on the key features of the skeleton model, compare the extracted main features of the gesture with the samples in the predefined gesture library and perform gesture classification to identify the specific gesture.

8. The multimodal interactive digital exhibition hall system according to claim 7, characterized in that: The key features include joint angles, finger lengths and relative position features, and the gesture classification method includes a pattern matching algorithm or a deep learning model method.

9. The multimodal interactive digital exhibition hall system according to claim 7, characterized in that: The gesture recognition module is also used to extract key point data of the hand from the hand image, and convert the extracted key point data into a feature vector, wherein the feature vector represents the spatial structure and dynamic information of the gesture; The learning model is trained and the labeled gesture data is used to train the learning model. The trained model is used to perform gesture recognition on the gesture data to identify specific gestures.

10. The multimodal interactive digital exhibition hall system according to claim 4, characterized in that: The second processing module is also used to annotate the speech text data, mark the intention and entity of each sentence, and use the annotated data set to train the language processing model, and obtain the trained speech recognition model for recognizing the speech text data.

Images