Deaf-mute barrier-free concert interaction system and method based on multi-modal dynamic mapping

Abstract

The invention relates to a deaf-mute barrier-free concert interaction system and method based on multi-modal dynamic mapping. The system comprises a wearable tactile execution unit, an AI processing unit, a multi-mode synchronous controller and a VR interaction module, and the tactile execution unit outputs multi-dimensional vibration modes to different body areas through differentially arranged linear resonance actuators. The AI processing unit maps real-time audio features into tactile parameters based on a contrast learning model and balances experience and energy consumption through the optimization module, the multi-mode synchronous controller achieves millisecond alignment of audio, vision and tactile, and the VR interaction module maps sign language input into a virtual scene and a tactile feedback adjustment instruction. Through the innovative tactile layout and the cross-modal mapping model, the immersion, rhythm perception and safety of the deaf-mute user are remarkably improved, and the method is suitable for various scenes such as concerts, education and rehabilitation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of tactile interaction and cross-modal perception, and in particular to a deaf-mute barrier-free concert interaction system and method based on multi-modal dynamic mapping. BACKGROUND

[0002] With the rapid development of multi-modal interaction and virtual reality technology, immersive experience of concerts has become a hot research and application. However, existing barrier-free designs are mostly concentrated on visual aspects such as subtitle translation or sign language broadcasting, and for deaf-mutes, the most core rhythm, melody and emotion in music still cannot be fully perceived. Some research attempts to use vibration motors to directly convert audio energy into mechanical vibration output, but this way is usually limited to a single frequency band or equidistant distribution, resulting in monotonous feedback and low recognition, and unable to present the levels and delicate changes of music.

[0003] In terms of synchronization, existing technologies often use simple buffer alignment or fixed delay compensation, which is difficult to ensure real-time consistency of tactile, audio and visual. Once the delay exceeds 30 milliseconds, users will obviously perceive asynchrony, thereby destroying the immersive experience. Especially in the scene of concerts with strong real-time performance, traditional solutions are difficult to meet the strict synchronization requirements.

[0004] In terms of interaction, most existing barrier-free systems are one-way output, and users can only passively receive tactile feedback, lacking channels for active participation and control. Even if a small number of solutions support interaction, they mostly rely on keys or mobile applications, which are cumbersome and do not conform to the natural habits of deaf-mute users. Sign language, as the mother tongue of the deaf-mute community, has not been fully utilized, which further reduces the immersive feeling and personalization of experience.

[0005] In terms of safety, existing tactile feedback systems often ignore physiological differences of users, lacking real-time monitoring of epilepsy or nervous system abnormalities. When the intensity of tactile stimulation is too high or the frequency is too concentrated, it may induce discomfort or even potential danger. However, current systems generally lack physiological monitoring mechanisms such as EEG or heart rate, and cannot intervene in time before the risk occurs.

[0006] In addition, in terms of energy consumption and portability, existing wearable tactile devices generally have short battery life, heavy weight and complex cables. Long-term use not only easily causes fatigue, but also limits the application in dynamic scenes such as concerts.

[0007] Therefore, the existing technology still needs to be improved. SUMMARY

[0008] In view of the deficiencies of the prior art described above, there is an urgent need for an accessible concert interaction system that can achieve high-fidelity dynamic mapping between audio features and haptic feedback, has millisecond-level multi-modal synchronization, supports natural sign language interaction, and has a security protection mechanism, to truly enhance the immersive music experience of the deaf and mute population.

[0009] The technical solutions of the present application are as follows: The present application provides a deaf and mute accessible concert interaction system based on multi-modal dynamic mapping, which comprises: A haptic execution unit configured to output multi-dimensional vibration patterns to different areas of the user's body; An AI processing unit configured to extract features from the input music audio signal and convert the audio features into haptic parameters based on a cross-modal mapping model; A multi-modal synchronization controller configured to time-align the audio signal, haptic parameter and visual signal, and realize synchronized output of multi-modal information; A VR interaction module configured to receive the user's sign language input and map it to an interaction instruction for adjusting the haptic feedback or triggering virtual scene special effects.

[0010] In one embodiment, the haptic execution unit includes a plurality of linear resonant actuators arranged on the user's chest, shoulders and waist area, and the spacing between adjacent actuators in the chest area is smaller than that in the shoulder and waist area, to enhance the haptic perception effect in the chest area.

[0011] In one embodiment, the working frequency of the haptic execution unit is configured to cover the low, medium and high frequency bands related to music perception, and one or more resonance intervals are preferably set to enhance the synesthesia experience.

[0012] In one embodiment, the AI processing unit uses a cross-modal mapping model based on contrastive learning to map music audio features and haptic pattern embeddings into the same space, to obtain haptic parameters consistent with the mood of the music.

[0013] In one embodiment, the haptic parameters include at least one dimension selected from amplitude, frequency, vibration envelope shape and spatial distribution pattern.

[0014] In one embodiment, the multi-modal synchronization controller realizes synchronization of audio, visual and haptic signals through three stages of timestamp comparison, delay estimation and compensation adjustment, and controls the end-to-end delay to within 20 milliseconds.

[0015] In one embodiment, the VR interaction module includes a sign language recognition component configured to map predefined sign language actions to interaction instructions to implement haptic feedback adjustment or virtual scene triggering.

[0016] Another aspect of the present application also provides a deaf-mute barrier-free concert interaction method based on multi-modal dynamic mapping, which comprises the following steps: S1, receiving real-time audio signals and performing feature extraction; S2, inputting the features into a cross-modal mapping model to generate haptic parameters; S3, aligning the audio, haptic and visual signals through a synchronous control mechanism; S4, driving a haptic execution unit to output vibration patterns according to the haptic parameters; S5, receiving user sign language input and modifying haptic feedback or triggering virtual scenes based on the sign language input.

[0017] In one embodiment, the method further comprises: S6, collecting physiological signals of the user in real time during the haptic execution process, and when an abnormal state is detected, automatically reducing the haptic intensity or stopping output to achieve safety protection.

[0018] Another aspect of the present application also provides a computer program product comprising program instructions stored on a non-transitory computer readable medium, which when executed by a processor, causes the processor to perform the method of any one of the above.

[0019] The deaf-mute barrier-free concert interaction system based on multi-modal dynamic mapping has the following beneficial effects in view of the deficiencies in the prior art: 1. At the level of haptic feedback, by differentiating the arrangement of the LRA in the chest region with a smaller interval than the shoulder and waist regions, the sensitive areas of the human body are optimized, and the uniformity of the haptic and the rhythm capture ability are effectively improved. Experimental results show that this design improves the rhythm perception accuracy of the user in the chest area by more than 40%, which is difficult to achieve by the existing uniform arrangement scheme.

[0020] 2. At the level of multi-modal synchronization, by using nanosecond timestamps and a three-stage synchronization verification mechanism, the end-to-end delay of audio, haptic and visual is controlled within 20 milliseconds, and the synchronization error is preferably less than 1 millisecond, which is significantly better than the traditional buffer alignment method. In user subjective tests, the "asynchronous perception rate" is reduced from 72% to less than 10%, resulting in unexpected immersion improvement.

[0021] 3. At the level of interaction, the invention first takes sign language as the core input method, and combines it with virtual scene special effects and tactile parameter adjustment, realizing the active participation and real-time control of users. Compared with traditional buttons or mobile terminal operations, sign language interaction not only has faster operation speed, but also improves the accuracy by more than 20%, greatly improving the interaction experience of deaf and mute users.

[0022] 4. In terms of safety, the invention introduces EEG electroencephalogram monitoring and heart rate detection, which can trigger a safety mode within 300 milliseconds when detecting abnormal signals, reducing tactile intensity or turning off output, thereby effectively avoiding potential risks. This physiological signal-driven protection mechanism has not been reported in existing barrier-free concert systems, and has obvious technological progress.

[0023] 5. At the algorithm level, the invention realizes high-precision mapping of audio features and tactile semantics through a cross-modal contrast learning model, with an emotion recognition accuracy of 92.3%, much higher than the level of less than 60% of traditional methods. This model not only improves the emotional matching degree, but also brings the "synesthesia" effect of tactile and musical emotion linkage, which is an unexpected technical advantage for those skilled in the art.

[0024] 6. In terms of energy consumption and portability, the invention uses a dynamic optimization algorithm to adjust the vibration intensity and frequency, reducing the average power consumption by 37%, and controlling the overall weight to be less than 1.5 kilograms, suitable for long-term wear and mobile scene applications. This improvement solves the problem of short battery life and inconvenience of existing devices.

[0025] In summary, the invention not only solves the defects of existing technology in terms of single tactile, synchronization delay, lack of interaction, insufficient safety, and high energy consumption, but also achieves unexpected technical effects through differentiated arrangement, cross-modal dynamic mapping, millisecond-level synchronization control, sign language interaction, and physiological safety protection. It provides a new immersive concert experience for the deaf and mute population. BRIEF DESCRIPTION OF DRAWINGS

[0026] The invention will be further described below in conjunction with the drawings and examples, in which: Figure 1 The deaf and mute barrier-free concert interactive system architecture based on multi-modal dynamic mapping provided by the invention; Figure 2 The deaf and mute barrier-free concert interactive system block diagram based on multi-modal dynamic mapping provided by the invention; Figure 3 The deaf and mute barrier-free concert interactive system tactile vest motor layout based on multi-modal dynamic mapping provided by the invention; Figure 4The CLIP model training flowchart of the deaf-mute barrier-free concert interaction system based on multi-modal dynamic mapping provided by the present application is provided. Figure 5 The deaf-mute barrier-free concert interaction system based on multi-modal dynamic mapping provided by the present application is compared with the prior art example. Figure 6 The deaf-mute barrier-free concert interaction method based on multi-modal dynamic mapping provided by the present application is provided. DETAILED DESCRIPTION

[0027] To make the purpose, technical scheme and effect of the present application clearer and more explicit, the present application is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. The embodiments of the present application are introduced below with reference to the drawings.

[0028] The deaf-mute barrier-free concert interaction system based on multi-modal dynamic mapping provided by the present application is compared with the prior art example. Figures 1-3 , the system comprises: The haptic execution unit 100 is configured to output multi-dimensional vibration patterns to different regions of the user's body. The deaf-mute barrier-free concert interaction system based on multi-modal dynamic mapping provided by the present application is compared with the prior art example.

[0029] In a further embodiment, the haptic execution unit 100 includes a plurality of linear resonant actuators (LRA) arranged on the user's chest, shoulders and waist area, and the adjacent actuators in the chest area have a smaller spacing than the adjacent actuators in the shoulder and waist area, to enhance the haptic perception effect in the chest area.

[0030] The haptic execution unit 100 is a key component of the system, and in a preferred embodiment includes 30 LRA actuators distributed on the user's chest, shoulders and waist area. The actuators in the chest area are arranged most densely, and the adjacent spacing is preferably smaller than that of the shoulder and waist area, so as to achieve higher haptic resolution at the chest, facilitating the user to capture the rhythm change and drum impact force.

[0031] Specifically, the deaf-mute barrier-free concert interactive system based on multi-modal dynamic mapping provided in the present application preferably takes a wearable vest as the main body, integrates multiple sets of linear resonant actuators (LRA) inside the vest, and differentially arranges the LRA according to different sensitivities of the chest, shoulders and waist, to ensure that the spacing between adjacent actuators in the chest region is smaller than that in the shoulder and waist regions.

[0032] Preferably, the LRA is arranged at the shoulder, chest and waist regions according to the golden section ratio (φ≈1.618), wherein the spacing in the chest region is 8 mm, and the spacing in the shoulder and waist regions is 13 mm, so as to form more intensive tactile feedback in the key positions. This design avoids the problem of sparse tactile feedback caused by uniform arrangement, and enables the user to capture the subtle changes in rhythm and emotion at the heart position. Experimental data show that this differential arrangement scheme improves the tactile uniformity score by 41% and the rhythm sensitivity score by 35% compared with the traditional scheme.

[0033] Further, in another embodiment, the spacing between adjacent motors of the LRA array not only differs in different body regions, but also further satisfies the Fibonacci sequence relationship. Specifically, the spacing between adjacent actuators is arranged in the ratio of 8 mm, 13 mm and 21 mm in sequence, forming a non-equidistant but mathematically regular distribution pattern. This design is inspired by the golden section and spiral growth law in nature, such as pinecone scales, sunflower disks, etc., whose distribution characteristics often naturally match the comfort of human vision and touch.

[0034] In the experiment, two arrangement schemes, equidistant spacing (10 mm) and Fibonacci spacing (8-13-21 mm cycle), were compared, and 20 users were invited to conduct blind tests under the playback of music segments with different rhythm intensities. The results show that the Fibonacci spacing scheme scores higher in tactile clarity, rhythm resolution and immersion than the equidistant scheme, with a 27% improvement in rhythm resolution and an average of 1.5 points (out of 5) in immersion score. This indicates that the arrangement based on the Fibonacci sequence can achieve more rich tactile levels under the same hardware conditions, avoiding monotony and causing tactile fatigue.

[0035] In addition, non-equidistant distribution also has the advantage of reducing interference. When multiple actuators work simultaneously, equidistant arrangement is prone to produce superposition effect at a certain frequency, causing local excessive vibration or resonance phenomenon. The Fibonacci spacing can break this regular overlap, making the tactile distribution more natural and avoiding discomfort caused by excessive concentration. Experimental records show that after using the Fibonacci arrangement, the users' subjective complaints of "tactile stinging" or "excessive vibration" are reduced by 38%, and the overall comfort of the system is significantly improved.

[0036] In scalable applications, the Fibonacci interval can not only be applied to the vertical arrangement in the vest, but also be extended to the horizontal or ring-shaped arrangement. For example, in a waist ring array, the angular distribution of adjacent actuators can also be designed according to the golden angle (about 137.5°) corresponding to the Fibonacci sequence, so as to realize a human tactile coverage closer to the natural law. In this way, the user can obtain balanced and natural tactile experience in any posture.

[0037] In further embodiments, the working frequency of the haptic execution unit is configured to cover the low, medium and high frequency bands related to music perception, and one or more resonance intervals are preferably set to enhance the synesthesia experience.

[0038] Specifically, the working frequency of each linear resonant actuator is preferably set to 70-100 Hz, with 87-92 Hz being the optimal resonance interval, which can significantly enhance the immersion, rhythm and body resonance effects. In actual tests at a concert site, the tactile resolution of the same piece of music in the 87-92 Hz interval increased by 27%, and users generally reflected that the tactile experience was "more in line with the music". In terms of technical implementation, each actuator is driven by an independent PWM channel, which can control the amplitude, frequency and envelope shape; the system supports simultaneous driving of multiple actuators to output composite haptic waveforms, such as outputting low-frequency steady-state vibration at the waist and superimposing short pulses at the shoulders to enhance the sense of hierarchy of drum points and bass lines.

[0039] In further embodiments, the system further comprises an AI processing unit 200 configured to extract features from the input music audio signal and convert the audio features into haptic parameters based on a cross-modal mapping model; At the audio processing and mapping level, the AI processing unit 200 receives real-time audio streams from the site, frames and samples the site audio streams with a frame length of 25 milliseconds and a frame shift of 10 milliseconds, extracts mel-frequency cepstral coefficients, beats, energy envelopes and emotion candidate labels. The cross-modal mapping model uses MFCC features and expert-labeled haptic patterns for contrastive learning in the training stage, adopts a structure based on contrastive learning, maps audio feature embeddings and haptic pattern embeddings to a shared semantic space, and uses positive and negative samples to constrain model convergence in the training stage, so that it can generate haptic parameters highly consistent with music emotions in the inference stage. Please refer to Figure 4 The model architecture model uses the CLIP (image-text pre-training) architecture to align the semantics of audio features and haptic parameters, and generates multi-dimensional vibration control signals through a Transformer decoder. The input is real-time audio features, and the output is a sequence of haptic parameters, including actuator position, frequency component, amplitude size, vibration envelope and spatial distribution pattern.

[0040] In further embodiments, the generated haptic parameters include at least four dimensions of amplitude, frequency, vibration envelope shape, and spatial distribution pattern, which can be adjusted independently or in combination. For example, in a rock music with strong rhythm, the system generates a pattern of high amplitude + short envelope, while in a slow classical music, it outputs a pattern of low amplitude + long envelope.

[0041] To further enhance the experience, the AI processing unit 200 also includes an optimization module that performs secondary adjustment with a weighted function between user subjective experience and energy consumption as the target. In a concert scenario, the system prioritizes immersion and allows energy consumption to increase; while in a home scenario, the system prioritizes battery life. Experimental results show that after the optimization module is enabled, user subjective satisfaction is improved by 20%, and system battery life is extended by about 35% in low-power mode.

[0042] In further embodiments, the system further comprises a multi-modal synchronization controller 300 configured to time-align the audio signal, haptic parameter, and visual signal, achieving synchronized output of multi-modal information. To ensure multi-modal consistency, the application designs a multi-modal synchronization controller 300, i.e. a precise synchronization control mechanism. All audio frames, haptic parameter frames, and VR rendering frames are attached with nanosecond-level time stamps, globally synchronized through IEEE1588PTP protocol, and synchronized to millisecond-level system runtime through a three-stage synchronization alignment mechanism, including timestamp comparison, delay estimation and compensation adjustment. The compensation method can be interpolation through jitter buffer or alignment through frame loss, ensuring overall delay control within 20 milliseconds, and synchronization error less than 1 millisecond in large-scale network environment. Comparative tests show that the average delay of traditional methods is 53 milliseconds, while the average delay of the system is only 9 milliseconds, with a standard deviation of 0.9 milliseconds. The subjective perception rate of "asynchronous" by users is reduced from 72% to less than 10%, and the immersion score is significantly improved.

[0043] In further embodiments, the system further comprises a VR interaction module 400 configured to receive user sign language input and map it to interaction instructions for adjusting haptic feedback or triggering virtual scene effects. Further, the VR interaction module 400 includes a sign language recognition component configured to map predefined sign language actions to interaction instructions to achieve haptic feedback adjustment or virtual scene triggering.

[0044] At the interaction level, the VR interaction module 400 of the present application can recognize sign language input in real time and map it to control instructions. Users can use predefined sign language actions to control haptic feedback or trigger virtual special effects. The predefined mapping table includes: three-finger horizontal sliding action for enhancing rhythm, haptic amplitude increases by 20% and superimposes pulse mode; clenched fist for one second to enter safety mode, reduce haptic intensity to 30% and shield high-frequency signals; double-click of the index finger triggers virtual scene filter changes, such as fire, starry sky or water wave effects.

[0045] In another embodiment, users can also customize the mapping relationship, and the system automatically saves personalized configurations. Compared with button input, sign language interaction is 1.4 seconds faster in operation speed, 21% higher in accuracy, and user satisfaction close to full marks. This interaction not only improves controllability, but also changes deaf-mute users from passive recipients to active participants, greatly enhancing immersion.

[0046] To ensure safety, the present application also introduces a physiological monitoring mechanism, preferably configured with an electroencephalogram (EEG) sensor and a heart rate sensor, to monitor the physiological state of the user in real time. When abnormal discharge or epilepsy risk is detected, the system triggers a safety mode within 300 milliseconds, reduces haptic output to 20%, and turns off all actuators within 1 second. Experiments show that in the simulation scenarios of 10 epilepsy patients, the system can respond in time, avoiding the risks caused by the delay of manual intervention in traditional systems. In addition, the present application supports detecting fatigue level through skin electrical signals and dynamically adjusting vibration intensity to ensure user comfort during long-term experience.

[0047] In another embodiment of the present application, please refer to Figure 6 A deaf-mute-friendly concert interaction method based on multi-modal dynamic mapping is also provided, which comprises the following steps: S1, receiving real-time audio signals and performing feature extraction; S2, inputting the features into a cross-modal mapping model to generate haptic parameters; S3, aligning the audio, haptic and visual signals through a synchronous control mechanism; S4, driving the haptic execution unit to output vibration patterns according to the haptic parameters; S5, receiving user sign language input and modifying haptic feedback or triggering virtual scenes based on the sign language input; S6, collecting physiological signals of the user in real time during the haptic execution process, and automatically reducing haptic intensity or stopping output when an abnormal state is detected to achieve safety protection.

[0048] For better understanding of the method of the present application, the method of the present application comprises: receiving real-time audio signals and extracting features, inputting the features into a cross-modal mapping model to generate haptic parameters, adjusting the haptic parameters through an optimization module, aligning the haptic parameters with audio and visual through a synchronization controller, and then driving an actuator to output a haptic pattern, while a VR module identifies user sign language and corrects feedback. When the physiological monitoring module detects abnormal signals, the safety protection is automatically triggered. The method can ensure the stability, low delay and high immersion experience of the system in different scenarios.

[0049] In terms of computer implementation, the present application also provides a program product stored in a non-transitory computer-readable medium, which, when executed by a processor, completes all the steps described above. The program can run on a central server at a concert site to distribute haptic data to multiple user terminals through a 5G network, or run on a mobile device or edge computing unit worn by a user to achieve offline processing.

[0050] In different application scenarios, the present application can show advantages. The present application is not only suitable for concert scenarios, but also suitable for home entertainment, music education and rehabilitation training. In a home scenario, the system can be used with a television or a sound system to enable a deaf-mute user to experience the rhythm and emotion of a movie or a variety show in a living room; in an education scenario, a teacher can guide students to establish a sense of musical rhythm through sign language instructions, and students can master the rhythm within three training sessions, which is 1.7 times faster than traditional teaching; in a rehabilitation training scenario, electroencephalogram monitoring and safety mechanisms can provide controllable haptic rehabilitation training for epilepsy patients; in a group scenario, multiple users can share the same haptic feedback through a network synchronization mechanism, realizing barrier-free group interaction.

[0051] In summary, the present application introduces a differentially arranged linear resonant actuator into a wearable vest, combines a cross-modal mapping model based on contrastive learning, a multi-modal synchronization control mechanism, and a sign language interaction and physiological signal monitoring module to construct a barrier-free concert interaction system for deaf-mute people. Compared with the prior art, the present application not only enables high-fidelity haptic presentation of musical rhythm and emotion, but also ensures millisecond-level synchronization of audio, visual and haptic, and gives users the ability to control individually through sign language. Through a large number of experiments, the present application has achieved significant improvement in haptic uniformity, emotional consistency, interaction efficiency, safety and energy consumption control, reaching a level that the prior art cannot achieve. Therefore, the present application has outstanding novelty and creativity in realizing immersive and barrier-free music experience, and has wide application prospects and industrial value.

[0052] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall belong to the protection scope of the appended claims of the present application.

Claims

1. A barrier-free concert interaction system for deaf and mute people based on multimodal dynamic mapping, characterized in that, The system includes: The haptic actuator is configured to output multi-dimensional vibration patterns to different areas of the user's body; The AI ​​processing unit is configured to extract features from the input music audio signal and convert the audio features into tactile parameters based on a cross-modal mapping model. A multimodal synchronization controller is configured to time-align the audio signal, tactile parameters, and visual signal to achieve synchronous output of multimodal information; The VR interaction module is configured to receive the user's sign language input and map it into interactive commands, which are used to adjust haptic feedback or trigger virtual scene effects.

2. The barrier-free concert interaction system for deaf and mute people based on multimodal dynamic mapping according to claim 1, characterized in that, The tactile actuator includes multiple linear resonant actuators arranged in the user's chest, shoulder, and waist areas, with the spacing between adjacent actuators in the chest area being smaller than that in the shoulder and waist areas, to enhance the tactile perception effect in the chest area.

3. The barrier-free concert interaction system for deaf and mute people based on multimodal dynamic mapping according to claim 1 or 2, characterized in that, The operating frequency of the haptic actuator is configured to cover the low, mid, and high frequency bands related to music perception, and preferably one or more resonant intervals are set to enhance the synesthetic experience.

4. The barrier-free concert interaction system for deaf and mute people based on multimodal dynamic mapping according to claim 1, characterized in that, The AI ​​processing unit employs a cross-modal mapping model based on contrastive learning to embed music audio features and tactile pattern embeddings into the same space to obtain tactile parameters consistent with the mood of the music.

5. The barrier-free concert interaction system for deaf and mute people based on multimodal dynamic mapping according to claim 4, characterized in that, The tactile parameters include at least one dimension, which is selected from: amplitude, frequency, vibration envelope shape, and spatial distribution pattern.

6. The barrier-free concert interaction system for deaf and mute people based on multimodal dynamic mapping according to claim 1, characterized in that, The multimodal synchronization controller achieves synchronization of audio, visual and tactile signals through three stages: timestamp comparison, delay estimation and compensation adjustment, and controls the end-to-end delay to within 20 milliseconds.

7. The barrier-free concert interaction system for deaf and mute people based on multimodal dynamic mapping according to claim 1, characterized in that, The VR interaction module includes a sign language recognition component, which is configured to map predefined sign language actions into interactive commands to achieve haptic feedback adjustment or virtual scene triggering.

8. A method for barrier-free concert interaction for deaf and mute individuals based on multimodal dynamic mapping, characterized in that, The method includes the following steps: S1. Receive real-time audio signals and extract features; S2. Input the features into the cross-modal mapping model to generate tactile parameters; S3. Align the audio, tactile, and visual signals through a synchronization control mechanism; S4. Drive the tactile actuator to output a vibration mode according to the tactile parameters; S5. Receive user sign language input and modify tactile feedback or trigger virtual scene based on the sign language input.

9. The method for barrier-free concert interaction for deaf and mute people based on multimodal dynamic mapping according to claim 8, characterized in that, The method further includes: S6. During the tactile process, the user's physiological signals are collected in real time. When an abnormal state is detected, the tactile intensity is automatically reduced or the output is stopped to achieve safety protection.

10. A computer program product, characterized in that, It includes program instructions stored on a non-transitory computer-readable medium, which, when executed by a processor, cause the processor to perform the method as described in any one of claims 8 or 9.

Images