Intelligent interactive digital somatosensory system and method

Abstract

The invention provides an intelligent interactive digital somatosensory system and method, and relates to the field of somatosensory interaction. The system comprises an acquisition module, a posture capture module, a behavior optimization module, a sensor optimization module and a virtual object interaction module. Wherein the acquisition module acquires physiological, behavior and environmental data in real time; the posture capturing module captures the three-dimensional space position and posture change of the user through a deep learning algorithm; the behavior optimization module dynamically adjusts a feedback interaction mode based on user historical data; the sensor optimization module dynamically adjusts the layout and configuration of the sensor; the virtual object interaction module is combined with a virtual reality technology to simulate dynamic physical feedback of a virtual object in real time; the system improves the naturalness and immersion of interaction through multi-sensor cooperation and a deep learning algorithm.

Description

Technical Field

[0001] The present invention relates to the field of somatosensory interaction, and in particular to an intelligent interactive digital somatosensory system and method. Background Art

[0002] Intelligent interactive digital somatosensory systems are one of the technological fields that have emerged in recent years with the rapid development of digital technology and intelligent hardware. With the continuous maturity of wearable devices, virtual reality (VR) and augmented reality (AR) technologies, users' experience requirements are getting higher and higher. Traditional single data collection and feedback methods can no longer meet the needs of personalized and multi-dimensional interaction.

[0003] Existing technical solutions usually use sensor modules and feedback modules to achieve basic user data collection and feedback. For example, biosensors are used to monitor the user's physiological state, or accelerometers are used to collect user movement information and provide feedback through feedback devices such as vibration and sound. Although these solutions can achieve basic interactive functions, most of them have limitations such as a single feedback method, simple interaction form, and inability to dynamically optimize according to the user's personalized needs. It is difficult to provide an in-depth and multi-dimensional user experience. Existing technologies also have major problems in personalized interaction design, especially in the ability to respond to user physiological feedback and behavioral changes in real time, which makes it difficult for the system to automatically adjust the feedback form according to the user's actual needs; and existing technologies can often only handle a single interaction mode, such as touch or simple gesture recognition, and cannot achieve complex and diversified interactive experiences, such as combining voice, gestures, tactile feedback, etc., thereby affecting the user's immersion and interaction depth. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides an intelligent interactive digital somatosensory system and method, which aims to solve the problems in the existing technology of single user interaction mode, inflexible feedback mechanism, and inability to dynamically adjust the interaction content and feedback mode according to the user's real-time status or needs.

[0006] (2) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: an intelligent interactive digital somatosensory system and method, including an acquisition module, a posture capture module, a behavior optimization module, a sensor optimization module and a virtual object interaction module, wherein: The acquisition module includes multiple sensors, including: a physiological data sensor for collecting physiological data such as the user's heart rate, skin galvanic response, and body temperature in real time; a behavioral data sensor for collecting the user's gestures, movements, voice, and facial expression data in real time; and an environmental data sensor for collecting environmental change data of the user's environment, including light intensity, temperature, humidity, and noise. The acquisition module synchronizes the collected data through a data synchronization protocol and transmits the processed data to the data fusion module. The gesture capture module includes a combination of multiple sensors, including a 3D depth camera, an inertial sensor, an accelerometer, and a gyroscope. The gesture capture module uses a deep learning algorithm to capture the user's three-dimensional spatial position, motion trajectory, and posture changes in real time, and provides multimodal feedback based on the user's behavior in real time, including visual feedback, tactile feedback, and audio feedback. The gesture capture module also includes a multi-sensor collaborative tracking algorithm to simultaneously track the user's movements and behavior data to ensure accurate feedback in multi-user interaction scenarios. The behavior optimization module collaborates with the learning unit through a reinforcement learning algorithm based on the user's historical behavior data, and can dynamically adjust the feedback interaction mode and feedback form according to the environmental change data; The sensor optimization module dynamically adjusts the layout and configuration of sensors based on real-time collected user behavior and environmental data, and introduces a dynamic configuration mechanism based on a neuroevolutionary algorithm to achieve real-time optimization of sensor angles, layouts, and sampling frequencies to cope with dynamic changes in users and the environment. The virtual object interaction module combines virtual reality technology with a dynamic physics engine, and can simulate the dynamic physical feedback of virtual objects in real time according to the user's behavior and action changes. The response of the virtual object is adjusted synchronously with the user's interactive action, thereby enhancing the naturalness and immersion of virtual interaction.

[0008] Preferably, the acquisition module is the first link of the system, responsible for acquiring the user's physiological, behavioral and environmental data in real time, including multiple sensors, which are used to collect physiological data, behavioral data and environmental data respectively; wherein the physiological data sensor monitors the user's heart rate, skin electrical response and body temperature, and collects and generates data streams in real time through the sensor, the behavioral data sensor includes gesture, movement, voice and facial expression recognition equipment, which can capture various interactive behaviors of the user, and the behavioral data sensor includes a 3D depth camera, an inertial measurement unit and a microphone; the environmental data sensor is used to monitor changes in the user's environment, such as light intensity, temperature, humidity and noise; all sensors are timestamped through a unified synchronization protocol to ensure that data from different sources remain consistent in time; through the data synchronization protocol MQTT, the collected data is transmitted to the data fusion module in chronological order; this synchronization process ensures the accuracy and consistency of various types of data, so that subsequent data analysis can be seamlessly integrated.

[0009] Preferably, the gesture capture module uses a sensor combination composed of hardware such as a 3D depth camera, an inertial sensor, an accelerometer and a gyroscope to accurately capture the user's position and motion trajectory in three-dimensional space; the 3D depth camera and inertial sensor can capture the user's spatial coordinates in real time and accurately judge the user's movement changes; in the specific implementation process, the 3D depth camera is responsible for obtaining the user's three-dimensional spatial information and recording every subtle movement of the user; the accelerometer and gyroscope are used to detect the user's acceleration and angular velocity to further optimize the accuracy of the motion trajectory; at the same time, the system uses a deep learning algorithm for real-time processing and analysis These data are then used to obtain the user's specific movements, positions, and postures. The system fuses these captured data with behavioral data through deep learning algorithms and provides multimodal feedback based on real-time user data, including visual, tactile, and audio feedback. For example, when a user makes a specific gesture, the system will decide whether to provide tactile feedback, audio feedback, or visual feedback based on the type of action, thereby ensuring the system's real-time response and high-precision interaction. The posture capture module also introduces a multi-sensor collaborative tracking algorithm that can accurately track postures in an environment where multiple users are interacting simultaneously, avoiding feedback lags caused by sensor resource conflicts or interference. In addition, to ensure that each user can receive real-time and accurate feedback in large-scale user interaction scenarios, the posture capture module uses an adaptive scheduling algorithm that can allocate resources based on each user's interaction needs and real-time actions, thereby avoiding response delays caused by insufficient sensor resources.

[0010] Preferably, the data fusion module is responsible for comprehensively analyzing the multi-source data from the physiological data sensor, the behavioral data sensor and the environmental data sensor; the module adopts the visual transformer (ViT) technology and multimodal adaptive graph neural network (GNN) in the deep learning algorithm to fuse the data of multiple sensors to generate a comprehensive user state model. The comprehensive model will combine the user's actions, location and environmental data to provide a comprehensive user behavior view; specifically, the ViT algorithm processes different types of data in parallel and converts them into complementary information streams, thereby improving the accuracy and real-time performance of data analysis. When processing structured data, the GNN algorithm can effectively fuse data from different sensors to generate a comprehensive model with spatial and temporal characteristics. The output result of the data fusion module will be passed to the posture capture module to provide accurate data support for the posture capture. The fused data not only improves the accuracy of user action recognition, but also reflects the user's health status and environmental changes, ensuring the personalization and accuracy of the feedback method.

[0011] Preferably, the behavior optimization module adopts a reinforcement learning algorithm to optimize the interactive feedback mode in real time according to the user's historical interaction data and the environmental data collected in real time; the reinforcement learning is to automatically adjust the form and intensity of feedback by continuously analyzing the user's behavior pattern and its interaction effect with the environment; the actual operation is that the system first analyzes the historical interaction data, evaluates the user's behavior habits and interaction frequency, and then combines the real-time environmental data, such as light intensity, temperature and humidity changes, to determine the specific mode of feedback; in a strong light environment, the system will automatically increase the brightness of the visual feedback, and in a high temperature environment, it may adjust the intensity or frequency of the tactile feedback; in addition, the behavior optimization module also combines the user's multi-level behavior prediction model to comprehensively model the user's behavior pattern. The operation mechanism of the multi-level behavior prediction model includes the following core steps: first, a multi-layer neural network structure is established based on the deep learning algorithm, and each layer corresponds to the behavior pattern analysis of different time scales. For example, the short-term behavior prediction layer focuses on the user's immediate operation and real-time The medium-term behavior prediction layer combines the behavioral feature trends over a period of time, and the long-term behavior prediction layer focuses on the user's habitual operations and long-term interaction patterns; then, the system inputs behavioral data from multiple sources into the prediction model through data fusion technology, and uses time series analysis algorithms to process the data, so that behavioral features at different time scales can be fully extracted and associated; during the model operation, the data output of the short-term layer can provide immediate support for real-time feedback, while the data output of the medium-term and long-term layers provides the system with a higher level of behavioral trend prediction, thereby optimizing future feedback strategies; in addition, the model also introduces an attention mechanism to dynamically adjust the prediction weights of different levels, and correct the model's prediction results in real time according to the user's actual behavior patterns to improve prediction accuracy; combined with this model, the system can quickly respond to users' immediate behavioral needs, and continuously optimize long-term interactive experience, continuously optimize feedback strategies, enhance the system's response speed and feedback accuracy, make the interaction method more intelligent, and improve user experience.

[0012] Preferably, the sensor optimization module is responsible for dynamically adjusting the layout and configuration of the sensor according to the user behavior and environmental data collected in real time. The sensor optimization module introduces a dynamic configuration mechanism based on the neuroevolutionary algorithm. The core idea of ​​this mechanism is to optimize the configuration of the sensor through the neuroevolutionary algorithm (NEA), including the sensor's angle, acquisition range and sampling frequency; the neuroevolutionary algorithm continuously evaluates and improves the layout configuration of the sensor by simulating the evolutionary process, thereby generating the optimal configuration strategy; during the operation of the system, the neural network will adjust the sensor's angle and acquisition range according to the user's behavior pattern, interaction needs and real-time changes in the environment to ensure the accuracy and efficiency of data collection; when the user performs fast movements, the system will automatically predict and dynamically adjust the sensor's sampling frequency according to the motion trajectory to ensure that sufficiently accurate data is captured to avoid inaccurate sampling frequency. Insufficient sensor data can cause information loss or delay. The neuroevolutionary algorithm optimizes the sensor sampling frequency, continuously selecting the most adaptable frequency configuration through selection, crossover, and mutation operations in the genetic algorithm to ensure high-quality data collection. When the user is stationary, the sampling frequency is appropriately reduced to save system resources and reduce unnecessary interference. This optimization process is automatically performed by the neuroevolutionary algorithm, which predicts the stationary state and adjusts the configuration by calculating the user's movement intensity and frequency pattern, thereby improving the efficiency of system resource utilization. The sensor angle and acquisition range are also dynamically adjusted as the user's movement changes. The neuroevolutionary algorithm optimizes the sensor layout in real time based on the sensor position, the user's movement direction, and the relative position between the user and the sensor, ensuring that the sensor can always capture the user's behavior data at the optimal angle. This real-time adjustment mechanism greatly improves data collection accuracy, ensuring that the system always obtains data at the optimal position and angle, and avoiding the loss of feedback accuracy caused by improper angles or insufficient acquisition range. Overall, by introducing the dynamic configuration mechanism of the neuroevolutionary algorithm, the sensor optimization module can intelligently adapt to various user behaviors and environmental changes, adjust the system configuration in real time, ensure the efficiency and accuracy of data collection, and thus achieve the best response to user interaction needs.

[0013] Preferably, the system can simultaneously track the posture, movement and behavior data of multiple users in a multi-user interaction scenario; the multi-sensor collaborative tracking algorithm uses a weighted fusion method to dynamically adjust according to the position and movement data of each user, ensuring that the system can handle the interaction needs of multiple users in a complex environment and provide accurate feedback in real time; in the algorithm, the posture data of each user is first weightedly fused to ensure that the interaction needs of each user can be identified and processed in a timely manner; the multi-sensor collaborative tracking algorithm will dynamically schedule computing resources according to the interaction priority and real-time needs of each user; the interaction priority is evaluated based on multiple factors, including the user's movement complexity, interaction frequency and the system's response time limit requirement. The higher the movement complexity, the higher the interaction frequency and the lower the system's response time limit requirement, the higher the interaction priority, and vice versa; for users with higher interaction priority, more computing resources and processing time will be allocated to ensure their interaction Interoperability can obtain real-time and accurate feedback; this mechanism ensures that even in a multi-user environment, the behavior of each user can be accurately captured and responded to. The higher the complexity of the action, the more complex the user's interactive action involves, the more complex the posture, trajectory or dynamic changes, such as rapid and continuous switching of actions in three-dimensional space, and actions involving the coordination of multiple parts at the same time. The complexity will be significantly higher than a single static action; the higher the interaction frequency, the more times the user interacts per unit time, such as continuous waving, frequent clicking or multiple triggering of voice commands. These behaviors put higher demands on the system's real-time processing capabilities; the lower the system response time limit requirement, the higher the feedback speed requirement of the interaction scenario. For example, when a user performs a fast action, he needs immediate tactile or visual feedback. The delay will affect the interactive experience. These factors jointly affect the evaluation of interaction priority. High-priority users will be given priority in obtaining computing resource support to ensure that their complex, high-frequency, low-latency operations can be responded to quickly and obtain accurate feedback.

[0014] Preferably, the learning unit continuously optimizes the system's feedback strategy and sensor configuration by analyzing historical user behavior and interaction data. Based on deep learning technology, the learning unit can identify user behavior patterns, habits, and changes in needs, thereby optimizing the system's adaptability and personalization. For example, the system might analyze a user's movement trajectory to predict their next action and provide feedback adjustments in advance.

[0015] Preferably, the virtual object interaction module combines virtual reality technology with a dynamic physics engine, and uses the Bullet physics engine to accurately simulate the dynamic physical feedback of virtual objects, thereby achieving a high degree of immersion and naturalness in the process of interaction with users; in this module, the physical properties of virtual objects, including weight, texture, elasticity and hardness, change in real time according to the user's behavior and actions, ensuring that the reaction of the virtual object is highly synchronized with the user's interactive actions; when the user touches the virtual object with his finger, the Bullet physics engine calculates the object's deformation, rebound force, surface friction and other physical properties in real time according to the touch force, direction and contact point, and then adjusts the performance of the virtual object to make it visually and tactilely feedback. can keep pace with the user's movements; if the user touches the object with a finger, the shape and elasticity of the virtual object will show a small rebound or deformation; when the user applies a larger force, the object will deform or rebound more according to the applied force, simulating real physical feedback; the system dynamically adjusts the physical properties of virtual objects, not only enhancing the naturalness of the interaction, but also greatly improving the user's immersion in the virtual objects, making the user's experience in the virtual reality environment more realistic and interactive; this real-time physical feedback not only improves the authenticity of the virtual object's response, but also ensures that every action of the user can get instant and accurate physical feedback, making the virtual interaction process smoother and more immersive.

[0016] Preferably, the application of the deep learning algorithm in the entire system is crucial, especially in sensor data fusion and behavior prediction; the system uses deep learning models, especially convolutional neural networks (CNN) and long short-term memory networks (LSTM), to achieve efficient fusion of sensor data and real-time prediction of user behavior; convolutional neural networks are used to process data from visual sensors and identify user posture and position; while long short-term memory networks are used to process time series data, such as changes in user movements and environmental changes, to help the system predict future interaction needs and provide real-time prediction data support for the behavior optimization module.

[0017] Preferably, when the system is started, the device is first initialized and the connection status of all sensors is checked. The initialization process includes a self-test of each sensor to ensure its accuracy and functionality. After initialization, the system will start data collection, and collect data in real time through the physiological data sensor, behavioral data sensor and environmental data sensor. After time synchronization through the data synchronization protocol, the data is transmitted to the data fusion module for integration and analysis to generate a comprehensive user state model, and the model is passed to the posture capture module; the posture capture module processes the captured user action, position and posture data in real time through a deep learning algorithm, and provides the system with accurate interactive feedback in combination with the user's historical behavioral data and environmental changes; the behavior optimization module optimizes the feedback strategy based on the reinforcement learning algorithm, the sensor optimization module dynamically adjusts the sensor configuration according to the real-time collected data, and the virtual object interaction module provides dynamic physical feedback of the interaction between the virtual object and the user; the entire system works together through multiple sensors to ensure that each user can get timely and accurate feedback in a multi-user interaction scenario.

[0018] (3) Beneficial effects

[0019] The present invention provides an intelligent interactive digital somatosensory system and method. It has the following beneficial effects: 1. By combining virtual reality technology with a dynamic physics engine, the present invention enables the system to accurately simulate the physical feedback of virtual objects in real time based on user behavior and movement changes. The physical properties of virtual objects adjust in real time based on user interaction, ensuring that the object's response is highly synchronized with the user's movements. When a user touches a virtual object, the object's shape and rebound force change based on the touch force and contact point, simulating real physical feedback. This mechanism enhances the naturalness and immersion of virtual interactions, allowing users to experience more realistic interactions in a virtual reality environment and enhancing their immersive experience.

[0020] 2. The present invention uses a sensor optimization module and a behavior optimization module, combined with real-time collected user behavior, environmental data and historical data, to dynamically adjust the sensor layout and configuration, and optimize the interactive feedback method; the sensor optimization module uses a neuroevolution algorithm to optimize the sensor angle, sampling frequency and layout in real time according to the user's behavior pattern and environmental changes, ensuring the accuracy and efficiency of data collection; and the behavior optimization module is based on a reinforcement learning algorithm, which can dynamically adjust the feedback method according to the real-time environment and the user's interaction needs, thereby optimizing the user experience in different environments and improving the adaptability and personalization of the system; through these optimization mechanisms, the system can achieve accurate feedback response in a multi-user interactive environment and meet individualized interaction needs. DETAILED DESCRIPTION

[0021] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0022] The embodiments of the present invention provide an intelligent interactive digital somatosensory system and method, which uses multiple sensors to collect the user's physiological, behavioral, and environmental data in real time, uses deep learning algorithms to fuse and analyze the data, and dynamically optimizes feedback strategies and sensor configurations to achieve high-precision, real-time, and personalized user experience.

[0023] In actual operation, the system is first initialized and configured by the acquisition module; this module includes multiple sensors for acquiring the user's physiological, behavioral and environmental data; at startup, the system confirms the functional status of all sensors through a series of self-test processes, including calibration of physiological data sensors, behavioral data sensors and environmental data sensors; and the acquisition module adopts a unified data synchronization protocol to ensure that data from different sensors can be consistent in time. After the user's physiological data such as heart rate, skin electrical response and body temperature are collected by the sensor, a continuous timestamp data stream is generated; at the same time, the behavioral data sensor captures the user's gestures, movements, voice and facial expressions; the environmental data sensor records parameters such as light, temperature, humidity and noise in the user's environment; then the system uses the MQTT protocol to transmit the data in chronological order to the data fusion module, realizing efficient data integration and real-time transmission.

[0024] After the data is transmitted to the data fusion module, the data fusion module uses the visual transformer (ViT) technology and multimodal adaptive graph neural network (GNN) in the deep learning algorithm to comprehensively analyze the physiological, behavioral and environmental data; first, the data fusion module extracts features from various types of data, among which the ViT algorithm is responsible for processing high-dimensional behavioral data, such as user movement trajectories and facial expressions; then the GNN algorithm processes structured environmental data and associates information such as light intensity, temperature and humidity changes with user behavior characteristics by constructing a time series graph model; the fused data generates a comprehensive user status model, which reflects the user's real-time movements, health status and environmental changes, providing basic data support for subsequent modules.

[0025] The gesture capture module uses a sensor combination consisting of a 3D depth camera, inertial sensor, accelerometer and gyroscope to capture the user's position, motion trajectory and posture changes in three-dimensional space in real time; the system uses deep learning algorithms to analyze sensor data in real time, such as convolutional neural networks (CNN) to process three-dimensional spatial information, and long short-term memory networks (LSTM) to process time series data of user movements; during the capture process, the system provides multimodal feedback based on each movement and position change of the user, including real-time image updates for visual feedback, vibration or pressure for tactile feedback, and voice prompts for sound feedback; for example, when the user makes a "grabbing" gesture, the system determines the position and movement type of the hand based on the data from the 3D camera, and simultaneously generates visual animation and tactile feedback of the virtual object grabbing, thereby ensuring a high degree of synchronization between user perception and system response.

[0026] The behavior optimization module further enhances the intelligence and personalization of interactions. It uses a reinforcement learning algorithm to dynamically optimize interaction strategies based on historical user behavior data and real-time interaction data. The system first analyzes user behavior habits under different environmental conditions. For example, in a high-noise environment, users may prefer visual feedback. Based on this habit, the system optimizes the interaction process by adjusting the brightness of visual feedback, tactile intensity, and other methods. The behavior optimization module introduces a multi-level behavior prediction model, which uses deep neural networks to achieve short-term, medium-term, and long-term predictions of user behavior. Short-term predictions are used for immediate interaction decisions, such as predicting the next action based on the current gesture. Medium-term and long-term predictions are used to optimize interaction strategies, such as adjusting the intensity of tactile feedback based on the user's long-term habits to make it more in line with user needs.

[0027] The sensor optimization module dynamically adjusts the sensor configuration, including the sensor's sampling frequency, angle, and acquisition range, through a neuroevolutionary algorithm. When the user performs high-speed movements, the system increases the sensor's sampling frequency to avoid data loss. Through a simulated evolutionary process, the sensor's angle and acquisition range are dynamically adjusted according to the user's movement direction. For example, when the user swings their arm, the sensor optimization module predicts the next position of the arm based on the user's motion trajectory, and adjusts the sensor's viewing angle in advance to ensure that complete movement details are captured. When the user is stationary, the sensor reduces the sampling frequency to save system resources. This dynamic configuration mechanism ensures that the system can maintain accurate and efficient data collection in different interaction scenarios.

[0028] The virtual object interaction module is the core component of the system; it combines virtual reality technology with the Bullet physics engine to achieve accurate simulation of the dynamic physical feedback of virtual objects; during the process of user interaction with virtual objects, the module will adjust the physical properties of the virtual objects, such as weight, elasticity and texture, in real time according to the user's actions; when the user touches the virtual object with his finger, the system uses the physics engine to calculate the force of the touch point in real time and adjust the deformation and rebound effect of the virtual object; for example, when the user lightly touches the object, the virtual object will produce a slight elastic deformation; when the user applies greater force, the object will undergo greater deformation or displacement according to the magnitude of the force; these adjustments are transmitted to the user in real time through visual and tactile feedback, thereby enhancing the realism and immersion of the virtual interaction.

[0029] The system performs particularly well in multi-user interaction scenarios; through a multi-sensor collaborative tracking algorithm, the system can track the movements and behaviors of multiple users simultaneously; the data fusion module will evaluate the interaction priority of each user, and high-priority users will receive more computing resources to ensure timely response to their interactive operations; for example, in a virtual meeting, the speaker's gestures and voice may be given higher priority, while other users are prioritized according to their interaction frequency; the system dynamically schedules computing resources to ensure that each user's interactive experience can reach the optimal state.

[0030] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent interactive digital somatosensory system, comprising an acquisition module, a posture capture module, a behavior optimization module, a sensor optimization module, and a virtual object interaction module, wherein: The acquisition module includes multiple sensors, including: a physiological data sensor for collecting physiological data such as the user's heart rate, skin galvanic response, and body temperature in real time; a behavioral data sensor for collecting the user's gestures, movements, voice, and facial expression data in real time; and an environmental data sensor for collecting environmental change data of the user's environment, including light intensity, temperature, humidity, and noise. The acquisition module synchronizes the collected data through a data synchronization protocol and transmits the processed data to the data fusion module. The gesture capture module includes a combination of multiple sensors, including a 3D depth camera, an inertial sensor, an accelerometer, and a gyroscope. The gesture capture module uses a deep learning algorithm to capture the user's three-dimensional spatial position, motion trajectory, and posture changes in real time, and provides multimodal feedback based on the user's behavior in real time, including visual feedback, tactile feedback, and audio feedback. The gesture capture module also includes a multi-sensor collaborative tracking algorithm to simultaneously track the user's movements and behavior data to ensure accurate feedback in multi-user interaction scenarios. The behavior optimization module collaborates with the learning unit through a reinforcement learning algorithm based on the user's historical behavior data, and can dynamically adjust the feedback interaction mode and feedback form according to the environmental change data; The sensor optimization module dynamically adjusts the layout and configuration of sensors based on real-time collected user behavior and environmental data, and introduces a dynamic configuration mechanism based on a neuroevolutionary algorithm to achieve real-time optimization of sensor angles, layouts, and sampling frequencies to cope with dynamic changes in users and the environment. The virtual object interaction module combines virtual reality technology with a dynamic physics engine, and can simulate the dynamic physical feedback of virtual objects in real time according to the user's behavior and action changes. The response of the virtual object is adjusted synchronously with the user's interactive action, thereby enhancing the naturalness and immersion of virtual interaction.

2. The intelligent interactive digital somatosensory system according to claim 1, characterized in that: The acquisition module synchronizes the real-time data from the physiological data sensor, behavioral data sensor and environmental data sensor through timestamp marking, and uniformly processes and transmits all data through the data synchronization protocol to ensure the timeliness, consistency and accuracy of the physiological data, behavioral data and environmental data. The data synchronization protocol sends the processed data to the data fusion module for real-time integration; wherein the behavioral data sensor includes a 3D depth camera, an inertial measurement unit and a microphone; the environmental data sensor includes a light sensor, a temperature and humidity sensor, and a noise sensor.

3. The intelligent interactive digital somatosensory system according to claim 1, characterized in that: The data fusion module uses the visual transformer (ViT) and multimodal adaptive graph neural network (GNN) in the deep learning algorithm to fuse data from multiple sensors, generate a comprehensive user state model and perform real-time analysis. The comprehensive user state model includes comprehensive information about the user's movements, location and environment. The data fusion module provides data support for the gesture capture module based on the comprehensive user state model to achieve accurate feedback on user interaction.

4. The intelligent interactive digital somatosensory system according to claim 1, characterized in that: The posture capture module captures the user's three-dimensional spatial position, motion trajectory and posture changes in real time through the 3D depth camera, inertial sensor and motion capture device. The deep learning algorithm fuses the captured posture data with the user's behavioral data and provides multimodal feedback of vision, touch and sound to ensure that each user's action and interaction needs are responded to in a timely and accurate manner. The posture capture module also includes a multi-sensor collaborative optimization and adaptive scheduling algorithm, which can intelligently schedule and allocate sensor resources in large-scale multi-user scenarios to ensure that each user can obtain timely and accurate feedback, especially in crowded and complex environments, to avoid resource conflicts and interference.

5. The intelligent interactive digital somatosensory system according to claim 1, characterized in that: The behavior optimization module collaborates with the learning unit through a reinforcement learning algorithm based on the user's historical behavior data to analyze changes in user behavior in real time, and automatically adjusts the feedback method and optimizes the feedback form according to the collected environmental change data. The environmental adaptability optimization includes automatically adjusting the screen brightness, volume, and tactile feedback intensity. The learning unit continuously learns the feedback strategy during the behavior optimization process and provides predictive data support for the behavior optimization module. At the same time, the behavior optimization module also combines the user's multi-level behavior prediction model to comprehensively model the user's behavior pattern, improve the system's prediction accuracy and real-time performance of user behavior, and further optimize the feedback strategy and interaction method.

6. The intelligent interactive digital somatosensory system according to claim 1, characterized in that: The sensor optimization module monitors user behavior and environmental changes in real time, and in combination with the learning results of the learning unit, dynamically adjusts the layout and configuration of sensors based on environmental change data and user interaction needs. The sensor adjustment includes sensor angle adjustment, acquisition range adjustment, and sampling frequency optimization to ensure data acquisition accuracy and reduce signal interference. The learning unit provides optimization suggestions to improve sensor response efficiency by analyzing the results of multiple sensor adjustments.

7. The intelligent interactive digital somatosensory system according to claim 1, characterized in that: The multi-sensor collaborative tracking algorithm can simultaneously track the posture, movement and behavior data of multiple users in a multi-user interaction scenario. The multi-sensor collaborative tracking algorithm performs resource scheduling and task priority allocation based on each user's interaction priority and real-time needs, ensuring that each user's interaction receives accurate and immediate feedback. The multi-sensor collaborative tracking algorithm uses a weighted fusion method to dynamically adjust based on the user's position and movement data.

8. The intelligent interactive digital somatosensory system according to claim 1, characterized in that: The virtual object interaction module combines virtual reality technology with a dynamic physics engine to simulate the dynamic physical feedback of virtual objects in real time based on user behavior and movement changes. The virtual object's response is synchronized with the user's interactive actions, enhancing the naturalness and immersion of virtual interaction. The physical feedback of the virtual object interaction module is calculated and simulated by the Bullet dynamic physics engine, and the response of the virtual object is adjusted in real time according to the user's posture, movement and behavior data. The response of the virtual object includes weight, texture and elasticity.

9. The intelligent interactive digital body sensing method according to claim 1, characterized in that: The system starts and initializes all sensors, detects the connection status of all devices, and ensures that all sensors are working properly; The physiological data sensor, behavioral data sensor and environmental data sensor collect user and environmental data in real time. The data collection is synchronized through the data synchronization protocol to ensure the timeliness, consistency and accuracy of the physiological data, behavioral data and environmental data; The gesture capture module captures the user's behavior and gestures, and uses deep learning algorithms and sensor fusion technology to combine the user's behavior data to generate and provide multimodal feedback including visual feedback, tactile feedback, and sound feedback; The behavior optimization module uses a reinforcement learning algorithm to optimize the feedback strategy based on the user's historical behavior data and real-time collected environmental data, and dynamically adjusts the feedback interaction method according to environmental change data; The sensor optimization module dynamically adjusts the position, angle, and sampling frequency of the sensor according to user behavior and environmental changes, and the learning unit learns user behavior data and interaction data, continuously adjusts and optimizes feedback strategies and sensor layout, and continuously improves personalization and adaptability; The virtual object interaction module combines virtual reality technology with a dynamic physics engine to simulate the dynamic response of virtual objects in real time and adjust the physical properties of the objects in sync with the user's interactive actions, thereby enhancing the naturalness and immersion of virtual interactions.