Multi-mode interactive digital content display desktop and air gesture recognition system

Through the multimodal interactive digital content display desktop and air gesture recognition system, using technologies such as deep learning and biometric recognition, the problems of insufficient accuracy of gesture recognition, single interaction mode and poor device compatibility are solved, and efficient and personalized The user interaction experience and security have promoted the development of human-computer interaction field.

CN118897622APending Publication Date: 2024-11-05GUANGZHOU WANYU SHENGHE INTELLIGENT TECH CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410952012.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

The existing multimodal interactive digital content display desktop and air gesture recognition systems have insufficient accuracy and robustness in gesture recognition, single interaction methods, lack of personalization, and poor device compatibility.

Method used

Adopt multi-modal interactive digital content display desktop and air gesture recognition system, including data acquisition module, data processing module, interactive control module and display output module, using deep learning algorithms, gradient descent method, user preference learning module, and noise suppression technology , biometric recognition and a variety of display devices to improve the accuracy of gesture recognition, personalized adjustment of interaction methods and device compatibility.

Benefits of technology

Improve the naturalness and efficiency of the user experience, provide personalized interaction methods, enhance the robustness and security of the system, support multi-tasking, ensure device compatibility and security, and promote the field of human-computer interaction develop.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897622A_ABST
    Figure CN118897622A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a multi-mode interactive digital content display desktop and air gesture recognition system which comprises a data acquisition module, a data processing module, an interaction control module and a display output module. By providing a visual, efficient and personalized interaction mode, the user experience is greatly improved, huge potential and value are shown in multiple fields, the multi-mode interaction system fuses gestures, touch control and voice, an intuitive and barrier-free user interface is created, particularly immersive experience is provided in VR / AR, and the user experience is improved by learning user preferences. Meanwhile, biological recognition is adopted to enhance safety, various display technologies are compatible, efficient and sanitary non-contact operation is supported, the system is suitable for multiple scenes such as commercial display, education, medical treatment and office work, and man-machine interaction is promoted to develop in the more natural and intelligent direction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-modal interactive digital content display desktop and an aerial gesture recognition system. Background Art

[0002] With the rapid development of science and technology, human-computer interaction technology is gradually evolving from traditional physical devices such as keyboards and mice to more natural and intuitive methods. As important products under this trend, digital content display desktop and mid-air gesture recognition aim to provide users with a richer and more convenient interactive experience by integrating multiple interactive methods such as mid-air gestures, touch, and voice. This not only improves user operating efficiency, but also greatly enhances the naturalness and immersion of human-computer interaction, providing strong support for the application of cutting-edge technologies such as virtual reality (VR) and augmented reality (AR).

[0003] The existing multimodal interactive digital content display desktop and mid-air gesture recognition system have made significant progress in improving user experience, but there are still some technical bottlenecks and background defects that need to be solved:

[0004] 1. Insufficient accuracy and robustness of gesture recognition: Although current gesture recognition technology can improve the accuracy of recognition to a certain extent through algorithms such as deep learning, its accuracy and robustness still face challenges in complex environments (such as lighting changes, background interference, etc.). This may cause the system to misrecognize the user's gesture, thereby affecting the user experience.

[0005] 2. Single interaction mode and lack of personalization: Although the system supports multiple interaction modes, in actual applications, there is often a lack of a mechanism to automatically adjust the interaction mode according to user habits and needs. This results in different users facing inconvenience or low efficiency when using the same system.

[0006] 3. Display device and technology limitations: Although the system supports a variety of display devices and technologies, there are still differences in compatibility and display effects between different devices. In addition, the existing display technology may be difficult to meet the demand for high-quality display content such as high definition and high frame rate. Summary of the invention

[0007] The purpose of the present invention is to provide a multimodal interactive digital content display desktop and an aerial gesture recognition system to solve the problems of insufficient gesture recognition accuracy and robustness, single interaction mode and lack of personalization in the existing digital content display desktop and aerial gesture recognition mentioned in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] Multimodal interactive digital content display desktop and mid-air gesture recognition system, including:

[0010] A data collection module, used to collect the user's mid-air gestures, touch and voice signals;

[0011] A data processing module is used to identify and analyze the collected signals and convert them into control instructions; the data processing module further includes a deep learning algorithm, wherein the algorithm uses a gradient descent method Optimize model parameters (w);

[0012] An interactive control module, which performs corresponding operations according to the control instructions and can automatically adjust the interactive mode according to the user's preferences;

[0013] Display output module, which provides visual feedback for user interaction and supports virtual reality and augmented reality display modes.

[0014] Preferably, the deep learning algorithm optimizes the gesture recognition model by learning a large amount of training data D, that is, improves the accuracy and robustness of gesture recognition by minimizing the loss function L, wherein the loss function can be expressed as:

[0015]

[0016] Where l is the loss function of a single sample, y i is the true label of the i-th sample, f(x i ; w) is the model's predicted value for the i-th sample, and w is the model parameter.

[0017] Preferably, the interactive control module also includes a user preference learning module for analyzing user usage habits and historical data to automatically adjust the layout of the interactive interface, the mapping method of control instructions or the response speed to better meet the personalized needs of different users.

[0018] Preferably, the display output module supports a variety of display devices and display technologies, including head-mounted displays, holographic projections, large screens and touch screens.

[0019] Preferably, the data acquisition module also includes noise suppression and signal enhancement technology, which improves the signal-to-noise ratio of the collected air gesture, touch and voice signals by using a filter F and a gain factor G, that is:

[0020] S enhanced =F(S raw )*G

[0021] Where S enhanced is the enhanced signal, S raw is the original signal.

[0022] Preferably, the system further comprises a security verification module for performing identity authentication through the user's biometrics or password, PIN code, so as to ensure the security of the system and the protection of the user's privacy. The user's biometrics include fingerprint Fp and facial recognition Fr. The identity authentication can be expressed as:

[0023] Verification=(F p >T p )∧(F r >T r )

[0024] Where Tp and Tr are the preset fingerprint and face recognition thresholds respectively;

[0025] The security verification module uses a face recognition algorithm based on deep learning, where the face feature vector F is generated by a feedforward neural network and its similarity S is calculated by cosine similarity:

[0026]

[0027] Among them, F1 and F2 are two face feature vectors to be compared.

[0028] Preferably, the data processing module uses a convolutional neural network (CNN) as a basic architecture, and the output o of each layer in the CNN can be calculated by the following formula: o=σ(W*I+b);

[0029] Among them, σ is the nonlinear activation function, W is the weight matrix, * represents the convolution operation, I is the input image, and b is the bias vector.

[0030] Preferably, the data processing module further comprises a recursive neural network RNN ​​structure for processing time series data, including speech signals, and the hidden state ht of the RNN is updated at each time step t as follows:

[0031] h t =f(W h |h t-1 , x t ]+b h )

[0032] Among them, x t is the input of the current time step, W h is the weight matrix, b h is the bias vector and f is the nonlinear function.

[0033] Preferably, the interaction control module optimizes the interaction strategy between the user and the system by using a reinforcement learning algorithm, and adjusts the system response by maximizing the cumulative reward R, where the cumulative reward is obtained by accumulating the instantaneous reward rt:

[0034]

[0035] Among them, γ is a discount factor that is used to reflect the importance of future rewards.

[0036] Preferably, the display output module uses a three-dimensional reconstruction algorithm to convert a two-dimensional image into a virtual object in a three-dimensional space. The reconstruction error E of the algorithm can be minimized by a least squares method, that is:

[0037]

[0038] Among them, X is the three-dimensional point cloud, P is the projection matrix, and Y is the coordinates of the feature points in the two-dimensional image.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) Enhanced user experience:

[0041] Users can interact with the system through natural gestures and voice, which is more intuitive and convenient than traditional keyboard and mouse operations; providing an immersive experience, especially in games, virtual reality (VR) and augmented reality (AR) applications, enhancing the user's sense of reality, and making the user feel that they are part of the content through immersive and interactive experience.

[0042] (2) Accessibility:

[0043] For people with limited mobility or special needs, mid-air gestures and voice recognition provide ways to operate devices without physical contact, improving accessibility.

[0044] (3) Health and safety:

[0045] During public health events, contactless gesture controls reduce the need to touch common surfaces, helping to reduce the spread of germs.

[0046] (4) Personalization and adaptability:

[0047] The system can learn and adapt to user preferences and habits to provide a customized interactive experience; it can adjust the interface and content based on the user's actions and voice, such as providing personalized learning materials in educational scenarios.

[0048] (5) Multi-tasking capability:

[0049] Allows users to perform multiple operations at the same time, such as controlling devices without putting down the items in their hands, improving work efficiency.

[0050] (6) High precision and response speed:

[0051] Advanced data processing modules and machine learning algorithms can accurately recognize complex gestures and voice commands and quickly respond to user intentions.

[0052] (7) Safety:

[0053] Biometric authentication (such as facial recognition, fingerprint recognition) enhances system security and prevents unauthorized access.

[0054] (8) Scalability and compatibility:

[0055] The system can be integrated with multiple devices and platforms, including smartphones, tablets, computers and smart home devices, forming a unified interactive network.

[0056] (9) Innovation and future orientation:

[0057] It has promoted the development of human-computer interaction and laid the foundation for the future smart living and working environment.

[0058] (10) Innovation in education and training:

[0059] Introducing multimodal interaction in education and training can make the learning process more vivid and interesting, and increase participation and attractiveness.

[0060] In summary, the multimodal interactive digital content display desktop and air gesture recognition system greatly improves the user experience by providing an intuitive, efficient and personalized interaction method, and shows great potential and value in multiple fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they provide further detailed explanations but do not constitute a limitation of the present invention.

[0062] Figure 1 It is a block diagram of the system composition of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the embodiments of the present invention and the drawings of the specification to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The present invention designs three different embodiments to specifically demonstrate the application of the multimodal interactive digital content display desktop and air gesture recognition system. Each embodiment will focus on a specific aspect of the system to demonstrate its diversity and practicality.

[0065] Example 1: Home Entertainment Center Integration

[0066] In this embodiment, the system is integrated into a home entertainment center to provide users with an immersive viewing and gaming experience.

[0067] Data acquisition module: The camera captures the user's gestures, the microphone receives voice commands, and the touch panel allows direct input.

[0068] Data processing module: Deep learning algorithms analyze gestures and voice in real time and convert them into control commands such as playing / pausing movies, adjusting volume, or switching game menus.

[0069] Interaction Control Module: Automatically learns and remembers user preferences, such as volume level, screen brightness, and frequently used commands.

[0070] Display output module: Provides high-quality video and game images through a head-mounted VR display or a large screen in the room, and supports the interaction of AR elements such as virtual characters.

[0071] Security Authentication Module: Facial recognition ensures that only authorized users can access and control the entertainment center.

[0072] Example 2: Education and training platform

[0073] This embodiment applies the system to online education and corporate training scenarios to enhance the interactivity and participation of distance learning.

[0074] Data collection module: High-precision cameras track the gestures of lecturers and students, and voice recognition captures questions and answers.

[0075] Data processing module: Deep learning models distinguish the actions of lecturers and students and intelligently generate interactive feedback, such as chemical reaction simulations in virtual laboratories.

[0076] Interactive control module: automatically adjusts the course difficulty and content according to learning progress and participation, and provides a personalized learning path.

[0077] Display Output Module: Use AR technology to display 3D models, such as human anatomy or mechanical structures, on students’ desktops to enhance understanding.

[0078] Security Verification Module: Ensure the security of personal learning records through biometric authentication of student accounts.

[0079] Example 3: Medical health monitoring system

[0080] This embodiment applies the system to hospital and home environments for patient rehabilitation and daily health monitoring.

[0081] Data collection module: Non-contact sensors collect the patient's limb movements, and voice recognition receives the patient's reported symptoms.

[0082] Data Processing Module: Deep learning algorithms analyze movement patterns to assess rehabilitation progress or detect early signs of abnormal health conditions.

[0083] Interactive control module: Dynamically adjust the rehabilitation plan according to the patient's specific situation, such as movement guidance for physical therapy.

[0084] Display output module: Uses AR technology to project guidance animations into the patient's surrounding environment to help perform rehabilitation exercises correctly.

[0085] Security Verification Module: Biometric verification ensures that only the patient or authorized medical staff can view health data and adjust recovery plans.

[0086] The above three groups of embodiments demonstrate the wide applicability and flexibility of the multimodal interactive digital content display desktop and air gesture recognition system, which can provide enhanced user experience and efficiency in fields ranging from entertainment, education to medical care.

[0087] Advantages of this multimodal interactive digital content display desktop and air gesture recognition system:

[0088] (1) Enhanced user experience:

[0089] Users can interact with the system through natural gestures and voice, which is more intuitive and convenient than traditional keyboard and mouse operations; providing an immersive experience, especially in games, virtual reality (VR) and augmented reality (AR) applications, enhancing the user's sense of reality, and making the user feel that they are part of the content through immersive and interactive experience.

[0090] (2) Accessibility:

[0091] For people with limited mobility or special needs, mid-air gestures and voice recognition provide ways to operate devices without physical contact, improving accessibility.

[0092] (3) Health and safety:

[0093] During public health events, contactless gesture controls reduce the need to touch common surfaces, helping to reduce the spread of germs.

[0094] (4) Personalization and adaptability:

[0095] The system can learn and adapt to user preferences and habits to provide a customized interactive experience; it can adjust the interface and content based on the user's actions and voice, such as providing personalized learning materials in educational scenarios.

[0096] (5) Multi-tasking capability:

[0097] Allows users to perform multiple operations at the same time, such as controlling devices without putting down the items in their hands, improving work efficiency.

[0098] (6) High precision and response speed:

[0099] Advanced data processing modules and machine learning algorithms can accurately recognize complex gestures and voice commands and quickly respond to user intentions.

[0100] (7) Safety:

[0101] Biometric authentication (such as facial recognition, fingerprint recognition) enhances system security and prevents unauthorized access.

[0102] (8) Scalability and compatibility:

[0103] The system can be integrated with multiple devices and platforms, including smartphones, tablets, computers and smart home devices, forming a unified interactive network.

[0104] (9) Innovation and future orientation:

[0105] It has promoted the development of human-computer interaction and laid the foundation for the future smart living and working environment.

[0106] (10) Innovation in education and training:

[0107] Introducing multimodal interaction in education and training can make the learning process more vivid and interesting, and improve participation and memory.

[0108] In summary, the multimodal interactive digital content display desktop and air gesture recognition system greatly improves the user experience by providing an intuitive, efficient and personalized interaction method, and shows great potential and value in multiple fields.

[0109] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. Multimodal interactive digital content display desktop and mid-air gesture recognition system, Features: include: A data collection module, used to collect the user's mid-air gestures, touch and voice signals; A data processing module is used to identify and analyze the collected signals and convert them into control instructions; the data processing module further includes a deep learning algorithm, wherein the algorithm uses a gradient descent method Optimize model parameters (w) to improve the accuracy of gesture recognition; An interactive control module, which performs corresponding operations according to the control instructions and can automatically adjust the interactive mode according to the user's preferences; Display output module, which provides visual feedback for user interaction and supports virtual reality and augmented reality display modes.

2. The multimodal interactive digital content display desktop and air gesture recognition system according to claim 1, Features: The deep learning algorithm optimizes the gesture recognition model by learning a large amount of training data D, that is, improves the accuracy and robustness of gesture recognition by minimizing the loss function L, where the loss function can be expressed as: Where l is the loss function of a single sample, y i is the true label of the i-th sample, f(x i ; w) is the model's predicted value for the i-th sample, and w is the model parameter.

3. The multimodal interactive digital content display desktop and air gesture recognition system according to claim 1, Features: The interactive control module also includes a user preference learning module for analyzing the user's usage habits and historical data to automatically adjust the layout of the interactive interface, the mapping method of the control instructions or the response speed to better meet the personalized needs of different users.

4. The multimodal interactive digital content display desktop and air gesture recognition system according to claim 1, Features: The display output module supports a variety of display devices and display technologies, including head-mounted displays, holographic projections, large screens and touch screens.

5. The multimodal interactive digital content display desktop and air gesture recognition system according to claim 1, Features: The data acquisition module also includes noise suppression and signal enhancement technology, which uses a filter F and a gain factor G to improve the signal-to-noise ratio of the collected air gesture, touch and voice signals, namely: S enhanced =F(S raw )*G Where S enhanced is the enhanced signal, S raw is the original signal.

6. The multimodal interactive digital content display desktop and air gesture recognition system according to claim 1, Features: The system also includes a security verification module for performing identity authentication through the user's biometrics or password, PIN code, to ensure the security of system use and the protection of user privacy. The user's biometrics include fingerprint Fp and facial recognition Fr, wherein the identity authentication can be expressed as: Verification=(F p >T p )∧(F r >T r ) Where Tp and Tr are the preset fingerprint and face recognition thresholds respectively; The security verification module uses a face recognition algorithm based on deep learning, where the face feature vector F is generated by a feedforward neural network and its similarity S is calculated by cosine similarity: Among them, F1 and F2 are two face feature vectors to be compared.

7. The multimodal interactive digital content display desktop and air gesture recognition system according to claim 1, Features: The data processing module uses a convolutional neural network (CNN) as the basic architecture, and the output o of each layer in CNN is calculated by the following formula: o = σ (W*I + b); Among them, σ is the nonlinear activation function, W is the weight matrix, * represents the convolution operation, I is the input image, and b is the bias vector.

8. The multimodal interactive digital content display desktop and air gesture recognition system according to claim 1, Features: The data processing module further includes a recursive neural network RNN ​​structure for processing time series data, including speech signals, and the hidden state ht of the RNN is updated at each time step t as follows: h t =f(W h [h t-1 ,x t ]+b h ) Among them, x t is the input of the current time step, W h is the weight matrix, b h is the bias vector and f is the nonlinear function.

9. The multimodal interactive digital content display desktop and air gesture recognition system according to claim 1, Features: The interaction control module uses a reinforcement learning algorithm to optimize the interaction strategy between the user and the system, and adjusts the system response by maximizing the cumulative reward R, where the cumulative reward is obtained by accumulating the instantaneous reward rt: Among them, γ is a discount factor that is used to reflect the importance of future rewards.

10. The multimodal interactive digital content display desktop and air gesture recognition system according to claim 1, Features: The display output module uses a three-dimensional reconstruction algorithm to convert a two-dimensional image into a virtual object in a three-dimensional space. The reconstruction error E of the algorithm is minimized by the least squares method, that is: Among them, X is the three-dimensional point cloud, P is the projection matrix, and Y is the coordinates of the feature points in the two-dimensional image.

Citation Information

Cited By

  • Animation demonstration device based on digital media

    CN119620860A

  • Payment information identification and cashier system based on augmented reality technology

    CN120471617A

  • Dynamic interface interaction system based on multi-modal gesture recognition

    CN120762574A