Animation game interaction method fusing multi-sensory feedback

Abstract

The invention relates to the technical field of cartoon game interaction, in particular to a cartoon game interaction method fusing multi-sensory feedback, which comprises construction of a distributed sensing network, a timestamp synchronization mechanism, an adaptive algorithm and dynamic feedback signal output. Through cooperative work of a visual module, an auditory module and a tactile module, a machine learning model is combined to predict user intention, a dynamic feedback signal is generated, and multi-sensory cooperative feedback is realized by using high-refresh-rate display, spatial audio and multi-band vibration technologies. According to the method, the defects of the prior art in the aspects of dynamics, diversity and real-time performance can be overcome, the requirements of modern cartoon games for efficient and intelligent interaction are met, and meanwhile good compatibility and expansibility are achieved.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and multimedia technology, specifically a method for interactive animation and games that integrates multi-sensory feedback. Background: With the rapid development of the animation and game industry, users' demands for interactive experiences are increasing, and interactive methods integrating multi-sensory feedback have gradually become a research hotspot. By combining visual, auditory, and tactile feedback, user immersion and participation can be significantly enhanced, thereby increasing the realism and fun of the game experience. However, existing multi-sensory feedback technologies still have shortcomings in the application of animation and game interaction, and there are certain limitations in achieving dynamic feedback requirements in complex scenarios.

[0002] A search revealed a remote control system and method for an indoor multi-sensory training device, published on March 11, 2022. This technology enables remote control of the multi-sensory device through serial communication between a host computer and a slave computer, and can obtain and provide feedback on device status parameters in real time. However, this solution primarily targets the remote control of multi-sensory training devices, limiting its application scenarios and lacking support for the dynamic and diverse nature of multi-sensory feedback in animation and game interactions. Furthermore, this solution relies on specific hardware and communication protocols, resulting in limitations in universality and scalability when adapting to different types of animation and game platforms.

[0003] A search revealed a sound control device based on multi-sensory data and a control apparatus using the same, published on July 26, 2024. This technology achieves precise adjustment of sound signals by collecting users' multi-sensory data and combining data analysis with automatic control technology. However, this solution primarily focuses on optimizing auditory feedback and fails to fully integrate the collaborative feedback mechanisms of other senses (such as touch and vision), resulting in insufficient multi-sensory integration capabilities in animation and game interactions. Furthermore, the processing of multi-sensory data relies on complex computing units, which may increase system latency and affect the real-time interactive experience.

[0004] The aforementioned problems indicate that existing technologies still have significant shortcomings in the application of multi-sensory feedback in animation and game interaction, particularly in terms of the dynamism, diversity, and real-time nature of the feedback. Therefore, this invention provides an animation and game interaction method that integrates multi-sensory feedback, aiming to enhance the immersion and response speed of the interaction by integrating visual, auditory, and tactile feedback, while optimizing the system's compatibility and scalability to meet the demands of modern animation and games for efficient and intelligent interaction methods. Summary of the Invention

[0005] One of the objectives of this invention is to overcome the shortcomings of existing technologies and provide an interactive method for animation and games that integrates multi-sensory feedback.

[0006] The second objective of this invention is to provide a multi-sensory feedback system architecture that can dynamically adapt to multiple hardware platforms.

[0007] The third objective of this invention is to provide a specific implementation step for implementing the above method.

[0008] To achieve the above objectives, the technical mechanism adopted in this invention is as follows: by synchronously processing visual, auditory, and tactile signals on a unified time axis, and combining real-time data analysis and feedback mechanisms, dynamic output signals are generated; at the same time, a distributed computing unit is introduced to perform hierarchical processing of multi-sensory data, so as to reduce system latency and improve response speed.

[0009] Based on a distributed sensor network, a data communication link is constructed between the vision, hearing, and tactile modules. A timestamp synchronization mechanism ensures the consistency of data transmission between the modules. On this basis, an adaptive algorithm is used to analyze user behavior data in real time and generate multi-sensory feedback signals that meet the needs of the scenario.

[0010] A method for interactive animation and game that integrates multi-sensory feedback, characterized by the following steps:

[0011] 1. Construct a distributed sensor network, connecting the visual signal acquisition module, auditory signal acquisition module, and tactile signal acquisition module to the central processing unit respectively;

[0012] 2. A timestamp synchronization mechanism ensures that the data collected by each module remains consistent across the timeline;

[0013] 3. Analyze the collected multi-sensory data using adaptive algorithms to generate dynamic feedback signals;

[0014] 4. The generated feedback signals are transmitted to the visual output device, auditory output device, and tactile output device respectively to form multi-sensory collaborative feedback.

[0015] A multi-sensory feedback system architecture that dynamically adapts to multiple hardware platforms, characterized in that the architecture includes:

[0016] 1. Distributed sensor networks used to collect users' visual, auditory, and tactile data;

[0017] 2. Central processing unit, which includes a timestamp synchronization module, a data hierarchical processing module, and an adaptive algorithm module;

[0018] 3. Output device interface module, used to transmit dynamic feedback signals to different types of output devices.

[0019] A specific implementation step for the above method, characterized in that the specific steps of the method are as follows: a. Construction of a distributed sensor network, specifically a-1. Configuring a visual signal acquisition module, which consists of a high-definition camera and an image processing unit, for capturing the user's actions and expressions in the game scene; a-2. Configuring an auditory signal acquisition module, which consists of a microphone array and an audio processing unit, for capturing the user's sound signals; a-3. Configuring a tactile signal acquisition module, which consists of a pressure sensor and a vibration feedback unit, for capturing the user's tactile input signals; a-4. Connecting the above modules to the central processing unit via a high-speed data bus to ensure that the data transmission rate is not less than 100Mbps.

[0020] b. The implementation of the timestamp synchronization mechanism involves the following steps: b-1. An independent timestamp generator is embedded in each acquisition module to mark the acquisition time of each frame of data; b-2. The central processing unit receives the timestamp data from each module and adjusts the timestamps uniformly using a time calibration algorithm; b-3. The adjusted timestamp data is stored in shared memory for subsequent data processing modules to access.

[0021] c. The implementation of the adaptive algorithm consists of the following steps: c-1. Preprocessing the collected multi-sensory data to remove noise signals and extract key features; c-2. Predicting user behavior intentions using a machine learning model based on the extracted feature data; c-3. Generating dynamic feedback signals by combining the prediction results and determining the output order of each feedback signal through a priority scheduling mechanism.

[0022] d. Output of dynamic feedback signals, the specific steps of which are as follows: d-1. Transmit the generated visual feedback signal to the display or projection device to present dynamic images through high refresh rate display technology; d-2. Transmit the generated auditory feedback signal to the speaker or headphones to enhance the spatial sense of sound through spatial audio technology; d-3. Transmit the generated tactile feedback signal to the vibration feedback device to simulate different tactile experiences through multi-band vibration.

[0023] The specific implementation of step a-1 above is as follows: a-1-1. A high-definition camera is installed directly in front of the user, between 50cm and 100cm from the user's face, with a field of view covering 90 degrees; a-1-2. The image processing unit is connected to the central processing unit via a USB 3.0 interface, with the image resolution set to 1920×1080 and the frame rate at 60fps; a-1-3. The image processing unit has a built-in edge detection algorithm for extracting key point information of the user's actions in real time.

[0024] The specific implementation of step b-1 above is as follows: b-1-1. The timestamp generator uses a high-precision crystal oscillator chip with a frequency stability error of less than ±10ppm; b-1-2. The timestamp data of each acquisition module is transmitted to the central processing unit through the SPI interface with a transmission rate of 1Mbps; b-1-3. The timestamp data is stored in a dual-port RAM with a capacity of 1MB, supporting concurrent access by multiple tasks.

[0025] The specific implementation of step c-2 above is as follows: c-2-1. The machine learning model adopts a convolutional neural network structure, and the input layer receives preprocessed multi-sensory data; c-2-2. The convolutional layer extracts feature data, the pooling layer performs dimensionality reduction, and the fully connected layer outputs the prediction results; c-2-3. The model training dataset contains no less than 100,000 labeled samples, and the cross-validation method is used to optimize the model parameters during training.

[0026] The specific implementation of step d-3 above is as follows: d-3-1. The vibration feedback device uses a piezoelectric ceramic driver with a working frequency range of 20Hz to 500Hz; d-3-2. The multi-band vibration signal is generated through PWM modulation technology with a duty cycle range of 10% to 90%; d-3-3. The vibration feedback device is connected to the central processing unit through an I2C interface, and the control signal update cycle is 1ms.

[0027] The distributed sensor network of this invention connects multiple acquisition modules via a high-speed data bus, ensuring the real-time performance and stability of data transmission. A timestamp synchronization mechanism, using a high-precision crystal oscillator chip and a time calibration algorithm, solves the problem of data transmission asynchrony between multiple modules. An adaptive algorithm combined with a machine learning model can dynamically generate feedback signals based on user behavior, improving the system's intelligence level. The output of the dynamic feedback signal utilizes high refresh rate display technology, spatial audio technology, and multi-band vibration simulation technology to achieve coordinated visual, auditory, and tactile feedback.

[0028] This invention addresses the problem of insufficient dynamic feedback in complex scenarios in existing technologies by combining distributed sensor networks and adaptive algorithms. It reduces system latency and improves response speed through timestamp synchronization and priority scheduling mechanisms. Furthermore, it enhances system compatibility and scalability through modular design and standardized interfaces. These technical features and methods overcome the shortcomings of existing technologies mentioned in the background regarding dynamism, diversity, and real-time performance, meeting the demands of modern animation and games for efficient and intelligent interactive methods. (See attached figures.)

[0029] Figure 1 This is a schematic diagram of the system architecture of the present invention, illustrating the connection relationship between the distributed sensor network, the central processing unit, and the output device interface module.

[0030] Figure 2 The flowchart of the timestamp synchronization mechanism of the present invention describes in detail the process of timestamp generation, calibration and storage.

[0031] Figure 3 The flowchart for the dynamic feedback signal output of the present invention illustrates the specific steps for generating and outputting visual, auditory, and tactile feedback signals.

[0032] The attached figures are labeled as follows: 1. Distributed sensor network; 2. Central processing unit; 3. Timestamp synchronization module; 4. Data hierarchical processing module; 5. Adaptive algorithm module; 6. Output device interface module; 7. Visual signal acquisition module; 8. Auditory signal acquisition module; 9. Tactile signal acquisition module; 10. Vibration feedback device. Detailed implementation methods are described below.

[0033] This invention provides an interactive method for animation games that integrates multi-sensory feedback. Its core lies in the collaborative work among a distributed sensor network 1, a central processing unit 2, and an output device interface module 6 to achieve the dynamic generation and output of multi-sensory feedback. The following is in conjunction with the appendix... Figures 1 to 3 The specific embodiments of the present invention will be described in detail with reference to the component numbers marked in the accompanying drawings.

[0034] In this embodiment, the distributed sensor network 1 consists of a visual signal acquisition module 7, an auditory signal acquisition module 8, and a tactile signal acquisition module 9. These modules are connected to the central processing unit 2 via a high-speed data bus to ensure a data transmission rate of no less than 100 Mbps. The visual signal acquisition module 7 includes a high-definition camera and an image processing unit. The high-definition camera is installed directly in front of the user, between 50 cm and 100 cm from the user's face, with a 90-degree field of view. The image processing unit is connected to the central processing unit 2 via a USB 3.0 interface, with an image resolution of 1920×1080 and a frame rate of 60 fps. The image processing unit incorporates an edge detection algorithm for real-time extraction of key point information of user actions. The auditory signal acquisition module 8 consists of a microphone array and an audio processing unit. After the microphone array captures the user's voice signal, the audio processing unit performs preliminary filtering and enhancement processing on the signal, and then transmits the processed signal to the central processing unit 2 via the high-speed data bus. The tactile signal acquisition module 9 consists of a pressure sensor and a vibration feedback device 10. The pressure sensor captures the user's tactile input signal and connects to the central processing unit 2 through an I2C interface. The control signal update cycle is 1ms.

[0035] The central processing unit 2 internally includes a timestamp synchronization module 3, a data hierarchical processing module 4, and an adaptive algorithm module 5. The timestamp synchronization module 3 is responsible for receiving timestamp data from each acquisition module and calibrating it. Each acquisition module embeds an independent timestamp generator, which uses a high-precision crystal oscillator chip with a frequency stability error of less than ±10ppm. The timestamp data is transmitted to the central processing unit 2 via the SPI interface at a rate of 1Mbps, and then stored in a dual-port RAM with a capacity of 1MB, supporting concurrent access by multiple tasks. The specific process of the timestamp synchronization mechanism is as follows: Figure 2 As shown, each module first generates timestamp data. Then, the central processing unit 2 adjusts the timestamps uniformly using a time calibration algorithm. Finally, the adjusted timestamp data is stored in shared memory for subsequent data processing modules to access. The data layering processing module 4 performs layered processing on the received multi-sensory data. First, it preprocesses the data to remove noise and extract key features. Then, it uses a machine learning model with a convolutional neural network structure to analyze the feature data. The model's input layer receives the preprocessed multi-sensory data, the convolutional layer extracts feature data, the pooling layer performs dimensionality reduction, and the fully connected layer outputs the prediction results. The training dataset contains no fewer than 100,000 labeled samples, and cross-validation is used to optimize the model parameters during training. The adaptive algorithm module 5 generates dynamic feedback signals based on the extracted feature data and determines the output order of each feedback signal through a priority scheduling mechanism.

[0036] Output device interface module 6 is responsible for transmitting dynamic feedback signals to visual, auditory, and tactile output devices, respectively. Visual feedback signals are transmitted to a display or projection device, where dynamic images are presented using high refresh rate display technology. Auditory feedback signals are transmitted to speakers or headphones, where spatial audio technology enhances the spatial feel of the sound. Tactile feedback signals are transmitted to vibration feedback device 10, which simulates different tactile experiences through multi-band vibration. Vibration feedback device 10 uses a piezoelectric ceramic driver with an operating frequency range of 20Hz to 500Hz. The multi-band vibration signals are generated using PWM modulation technology, with a duty cycle range of 10% to 90%. The output flow of the dynamic feedback signal is as follows: Figure 3 As shown, the generated feedback signals are first sorted by a priority scheduling mechanism, and then transmitted to the corresponding output devices to complete the final multi-sensory collaborative feedback.

[0037] The entire system architecture is as follows Figure 1As shown, each acquisition module in the distributed sensor network 1 is connected to the central processing unit 2 via a high-speed data bus. The timestamp synchronization module 3 within the central processing unit 2 ensures data transmission consistency between modules. The data layering processing module 4 and the adaptive algorithm module 5 jointly complete the analysis of multi-sensory data and the generation of feedback signals. Finally, the output device interface module 6 transmits the feedback signals to different types of output devices. Through the design of the above structure and process, this invention achieves synchronous processing of visual, auditory, and tactile signals on a unified time axis, and generates dynamic output signals by combining real-time data analysis and feedback mechanisms, thereby solving the problem of insufficient dynamic feedback in complex scenarios in existing technologies. To better enable those skilled in the art to fully understand and implement this invention, the specific implementation principle of this invention is further explained below with reference to a specific application scenario.

[0038] In interactive scenarios involving anime and games, users engage in multi-sensory interaction with the system by wearing haptic feedback gloves, facing a high-definition camera, and using a microphone array. Taking a virtual reality fighting game as an example, the user plays a warrior who needs to complete combat missions through body movements, voice commands, and haptic input. The entire interaction process consists of the following steps:

[0039] First, in the visual signal acquisition phase, a high-definition camera captures the user's body movement information. The camera is mounted directly in front of the user, approximately 80cm from their face, with a 90-degree field of view, ensuring complete recording of the user's upper body movements. The image processing unit transmits the acquired video data to the central processing unit 2 via a USB 3.0 interface. The image resolution is 1920×1080, and the frame rate is 60fps. The edge detection algorithm built into the image processing unit extracts key user motion points in real time, such as arm movements and footsteps. A high-precision crystal oscillator chip in the timestamp synchronization module 3 generates timestamp data, marking the acquisition time of each frame, and transmits it to the central processing unit 2 via the SPI interface at a rate of 1Mbps. Subsequently, a time calibration algorithm adjusts the timestamps to ensure consistency with the timeline of other sensory data.

[0040] Secondly, during the auditory signal acquisition phase, the user issues attack or defense commands via voice, such as shouting "attack" or "defense." After the microphone array captures the sound signal, the audio processing unit performs preliminary filtering and enhancement processing to remove background noise and improve speech clarity. The processed audio data is also transmitted to the central processing unit 2 via a high-speed data bus, with timestamp data embedded to ensure time synchronization with the visual signal. At this time, the data layering processing module 4 preprocesses the received audio data, using a convolutional neural network model to extract speech features, such as intonation, speech rate, and keyword information. The trained machine learning model predicts the user's intent based on these features, such as determining whether the user has issued an attack command.

[0041] Secondly, during the haptic signal acquisition phase, the user interacts with virtual objects by wearing haptic feedback gloves. For example, when a user grasps a weapon in a virtual scene, pressure sensors on the gloves capture the pressure signals applied by the fingers. The pressure sensors transmit the signals to the central processing unit 2 via an I2C interface, with a control signal update cycle of 1ms. The adaptive algorithm module 5 analyzes the user's grip strength based on the pressure signals and combines visual and auditory data to generate dynamic feedback signals. For example, when the user grips the weapon forcefully, the system generates a stronger vibration feedback signal to simulate the weight of the weapon.

[0042] Finally, in the multi-sensory collaborative feedback stage, the central processing unit 2 determines the output order of each feedback signal through a priority scheduling mechanism. Visual feedback signals are transmitted to the display, where high refresh rate display technology presents dynamic images, such as the trajectory of a weapon when a user swings their arm. Auditory feedback signals are transmitted to headphones, where spatial audio technology enhances the spatial sense of sound, such as the sound of a weapon impact coming from different directions. Tactile feedback signals are transmitted to the vibration feedback device 10, which uses PWM modulation technology to generate multi-band vibration signals with a duty cycle ranging from 10% to 90%, simulating different tactile experiences, such as the vibration or impact of a weapon. The vibration feedback device 10 uses a piezoelectric ceramic driver with an operating frequency range of 20Hz to 500Hz, ensuring accurate simulation of complex tactile effects.

[0043] Throughout the interaction, each acquisition module in the distributed sensor network 1 maintains real-time communication with the central processing unit 2 via a high-speed data bus, ensuring a data transmission rate of no less than 100Mbps. The timestamp synchronization module 3 solves the problem of asynchronous data transmission between multiple modules through a high-precision crystal oscillator chip and a time calibration algorithm, thereby achieving synchronized processing of visual, auditory, and tactile signals on a unified timeline. The data layering processing module 4 and the adaptive algorithm module 5 jointly complete the analysis of multi-sensory data and the generation of feedback signals. Finally, the output device interface module 6 transmits the feedback signals to different types of output devices. Through the design of the above structure and process, this invention achieves dynamic generation and output of multi-sensory signals, significantly improving the user's immersion and response speed.

[0044] All content not described in detail in this specification is prior art known to those skilled in the art, and the model parameters of each electrical appliance are not specifically limited; conventional equipment can be used. Electrical control components not mentioned in this technical solution are not shown in the figures because they are prior art, and will not be described further here.

[0045] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for interactive animation and game development that integrates multi-sensory feedback, characterized in that... This method is implemented through the following steps: a. Constructing a distributed sensor network, connecting the visual signal acquisition module, the auditory signal acquisition module, and the tactile signal acquisition module to the central processing unit respectively; b. Ensure that the data collected by each module remains consistent on the timeline through a timestamp synchronization mechanism; c. Analyze the collected multi-sensory data using an adaptive algorithm to generate dynamic feedback signals; d. Transmit the generated feedback signals to the visual output device, auditory output device, and tactile output device respectively to form multi-sensory collaborative feedback.

2. The method for interactive animation and game integration with multi-sensory feedback according to claim 1, characterized in that... The construction of the distributed sensor network in step a includes the following steps: a-1. Configure a visual signal acquisition module, which consists of a high-definition camera and an image processing unit, to capture the user's actions and expressions in the game scene; a-2. Configure an auditory signal acquisition module, which consists of a microphone array and an audio processing unit, to capture the user's voice signals; a-3. Configure a tactile signal acquisition module, which consists of a pressure sensor and a vibration feedback unit, to capture the user's tactile input signals; a-4. Connect the above modules to the central processing unit through a high-speed data bus to ensure that the data transmission rate is not less than 100Mbps.

3. The animation game interaction method integrating multi-sensory feedback according to claim 1, characterized in that... The implementation of the timestamp synchronization mechanism in step b includes the following steps: b-1. An independent timestamp generator is embedded in each acquisition module to mark the acquisition time of each frame of data; b-2. The central processing unit receives the timestamp data from each module and adjusts the timestamps uniformly through a time calibration algorithm; b-3. The adjusted timestamp data is stored in shared memory for subsequent data processing modules to call.

Images