A special population-oriented auxiliary communication method and system

Abstract

A special group-oriented auxiliary communication method and system, the method comprises: acquiring multi-modal data of a user; identifying the multi-modal data type, preprocessing, and generating original feature vectors of different modalities; mapping the original feature vectors of different modalities to the same semantic space for alignment, obtaining standard feature vectors of different modalities; setting a dynamic weighted fusion strategy to perform shared embedding space fusion on the standard feature vectors of different modalities, generating multi-modal fusion feature vectors; introducing a multi-modal intention understanding model to perform deep reasoning on the multi-modal fusion feature vectors, obtaining the real intention of the user; generating a personalized multi-modal response through a response decision engine to complete the auxiliary communication; the personalized multi-modal response includes response content and adapted modal combination. The application adopts a multi-modal fusion engine, realizes multi-modal collaborative work through semantic space alignment, and solves the problem of social isolation of special groups in all scenarios.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing technology, specifically to an auxiliary communication method and system for special groups of people. Background Technology

[0002] Currently, students with disabilities face significant communication barriers. Surveys indicate that China has approximately 206 million hearing-impaired individuals, 16.91 million visually impaired individuals, and 30 million speech-impaired individuals, with a high proportion of teenagers. Existing assistive tools are functionally limited: for example, the iFlytek Hearing App only addresses speech-to-text conversion, Braille displays are expensive and have limited functionality, and foreign solutions suffer from insufficient localization. These issues lead to low social participation, unequal access to educational resources, and exacerbated employment difficulties for students with disabilities. Although multimodal technologies have developed, a lack of deeply integrated solutions prevents real-time interaction across different groups and fails to meet the educational needs of special populations.

[0003] Existing technology, such as the patent document with publication number CN112183216A, specifically discloses an assistive system for communication among people with disabilities, including: a sign language recognition unit, an intelligent alarm unit, an event recording and reminder unit, an information conversion and processing unit, a medical assistance unit, and an interactive display unit; the sign language recognition unit is used to input the hand gesture information of deaf-mute individuals; the intelligent alarm unit is used for event alarms, and includes an alarm priority setting module, an alarm information caching and processing module, an alarm mode selection module, an online alarm module, and an offline alarm module; the event recording and reminder unit is used to record tasks that users need to complete within a certain period of time and provide timed reminders; the medical assistance unit is used to verify the legality of the user's identity, store the user's medical records and re-examination reports, output the user's health report, and assist the user with travel navigation and rehabilitation training. This system is only suitable for the simple daily communication needs of deaf-mute individuals and is not suitable for blind individuals; furthermore, it only meets the needs of simple daily communication and is not suitable for teaching environments with rich and complex communication information.

[0004] For example, patent document CN119835288B discloses an assistive system for communication among people with disabilities, relating to the field of intelligent communication technology. This system includes: a signal capture and cloud upload unit, a signal filtering and verification unit, a data fusion and storage unit, a personal profile database construction unit, a personalized communication assistance database establishment unit, a mapping rule table formulation unit, and a mapping rule display unit. However, this system only meets the needs of daily communication for people with disabilities and is not suitable for teaching environments with complex information exchange.

[0005] Therefore, it is necessary to design an auxiliary communication technology solution that can facilitate and facilitate information communication for special groups in education, social interaction and daily life. Summary of the Invention

[0006] To address the technical problems existing in the prior art that create barriers for visually, hearing, and speech impaired individuals in education, social interactions, and daily communication, this invention provides an assistive communication method for special populations, comprising the following steps: Acquire the user's multimodal data; the multimodal data includes audio streams, video streams, and Braille dot matrix codes; The multimodal data types are identified, preprocessed, and original feature vectors of different modalities are generated; the original feature vectors include original speech feature vectors, original video feature vectors, and original text feature vectors. The original feature vectors of different modalities are mapped to the same semantic space, and semantic alignment is performed to obtain standard feature vectors of different modalities. A dynamic weighted fusion strategy is set up to perform shared embedding space fusion on standard feature vectors from different modalities, generating a multimodal fused feature vector. P fused ; Introducing a multimodal intent understanding model, the multimodal fusion feature vector is processed. P fused Perform deep reasoning to understand the user's true intent; Based on the user's true intent, a personalized multimodal response is generated through a response decision engine to complete this assisted communication; the personalized multimodal response includes response content and an appropriate combination of modalities.

[0007] Furthermore, the preprocessing includes: Automatic speech recognition technology is used to convert the speech stream into an intermediate representation in the text, and then deep semantic encoding is performed to generate the original speech feature vector. Key frames are extracted from the video stream, and the scene content and optical character recognition of the key frames are analyzed to extract the text and lip-reading sequences in the key frames to form the original video feature vector. The standard text of Braille is obtained, and the tactile pressure digital signal is converted into a standardized Braille dot matrix code through a lightweight Braille reader, and then binarized for noise reduction. The standardized Braille dot matrix code is parsed and converted into the corresponding standard text to form the original text feature vector.

[0008] Furthermore, the semantic alignment includes the following: Each original feature vector of a different modality is equipped with a dedicated feature encoder. A cross-modal attention mechanism or a joint embedding model is used to project the original feature vectors of different modalities into the same semantic space to achieve semantic alignment.

[0009] Furthermore, the dynamic weighted fusion strategy is as follows: Real-time evaluation of the confidence level of the standard feature vectors for different modalities; Based on the confidence level, the weights of the standard feature vectors for different modalities are dynamically adjusted; Based on the aforementioned weights, the standard feature vectors of different modalities are weighted and fused to form a multimodal fused feature vector. P fused .

[0010] Furthermore, the dynamic adjustment of the weights of the standard feature vectors for different modalities includes increasing the weights of lip reading recognition in noisy environments; the lip reading recognition is implemented through a video modality, specifically including the following: Preset basic video and audio weights; When audio is detected in a noisy environment, the signal-to-noise ratio of the audio is calculated. SNR Or the entropy of the predicted probability; Based on the signal-to-noise ratio in a noisy environment SNR The characteristics of low and high entropy values ​​indicate a decrease in confidence. Based on the confidence level assessment results, the audio weight is reduced. w a Increase video weight w v ; The audio weight w a The calculation expression is: ; The video weight w v The calculation expression is: ; Where α is a preset signal-to-noise ratio adjustment parameter.

[0011] Furthermore, the response content includes: speech with the desired emotion; explanatory text, highlighted key information or dynamic charts; and Braille vibration signals.

[0012] This invention also provides an auxiliary communication system for special populations, implemented using the aforementioned auxiliary communication method for special populations, including: a multimodal data acquisition and preprocessing module, a feature extraction and semantic space alignment module, a dynamic fusion and intent understanding module, and a personalized multimodal response generation module; The modal data acquisition and preprocessing module is used to capture the user's multimodal data in real time, generate multimodal raw feature vectors, and input them into the feature extraction and semantic space alignment module; the multimodal data includes speech streams, video streams, and Braille dot matrix codes; The feature extraction and semantic space alignment module is used to map the received original feature vectors to the same shared embedding space, perform semantic alignment, form standard feature vectors of different modalities, and input them to the dynamic fusion and intent understanding module. The dynamic fusion and intent understanding module is used to dynamically weight and fuse the received standard feature vectors of different modalities, perform deep reasoning, obtain the user's true intent, and input it into the personalized multimodal response generation module. The personalized multimodal response generation module is used to output a personalized multimodal response based on the true intent and the contextual information of the multimodal data; the personalized multimodal response includes response content and an adapted modality combination.

[0013] Furthermore, the modal data acquisition and preprocessing module includes an automatic speech recognition unit, a speech synthesis unit, a computer vision, optical character recognition, and lip reading integration unit, and a Braille translation unit; the automatic speech recognition unit and the speech synthesis unit are used to realize the mutual conversion between speech streams and text; the computer vision, optical character recognition, and lip reading integration unit is used to realize the recognition of video streams and images; the Braille translation unit is used to realize the mutual conversion between Braille dot matrix encoding and standard text.

[0014] Furthermore, the semantic alignment includes audio, video, and camera calibration binding lip-reading region alignment; the semantic alignment of audio and video is achieved by marking timestamps and dynamically adjusting the playback sequence; the camera calibration binding lip-reading region alignment is achieved by establishing a mapping relationship between 2D image pixels and 3D facial physical space through camera calibration and coordinate transformation.

[0015] Furthermore, the context information is formed through the following process: The standard feature vectors of the different modalities are mapped to the same dimension of the shared embedding space; The standard feature vectors of different modalities are concatenated or interleaved and input into a Transformer-based natural language processing model. The standard feature vectors of different modalities interact through a cross-modal attention layer, so that each standard vector contains context from other modalities, forming the contextual information of the multimodal data.

[0016] The beneficial effects of this invention are as follows: This invention employs a multimodal fusion engine, including multimodal data acquisition and preprocessing, feature extraction and semantic space alignment, and dynamic fusion and intent understanding. By unifying the semantic space, it enables the collaborative work of speech, visual, and text / tactile modalities, forming personalized multimodal responses and addressing the isolation issues faced by students with disabilities in communication, learning, and social interaction. It can intelligently recommend dominant modalities, emotion recognition and expression, and data closure based on the environment. Through temporal, spatial, and semantic alignment strategies and fusion mechanisms, it maintains high robustness under noisy or low-light conditions. It overcomes the limitations of single-modal communication, enabling efficient communication between students with disabilities and the outside world, and is applicable to all scenarios of education, work, and social interaction. Attached Figure Description

[0017] Figure 1 This is a flowchart of the auxiliary communication method for special groups provided by the present invention; Figure 2 This is an architecture diagram of the auxiliary communication system for special groups provided by the present invention; Figure 3 This is an integrated flowchart of the computer vision, optical character and lip reading recognition integrated unit provided by the present invention; Figure 4 This is a flowchart of semantic alignment to dynamic fusion provided by the present invention. Detailed Implementation

[0018] The technical solution of the present invention is further described below, but the scope of protection is not limited to what is described.

[0019] This invention provides an auxiliary communication method for special populations, such as... Figure 1 As shown, it includes the following steps: Step S100: Obtain the user's multimodal data; the multimodal data includes audio stream, video stream, and Braille dot matrix encoding; The multimodal data are obtained from raw, uncompressed audio waveform signals acquired through user interaction tools, such as microphone signals; RGB video frame sequences or single images continuously captured by a camera; and point-pressed digital signals from a Braille keyboard. Step S200: Identify the multimodal data type, perform preprocessing, and generate original feature vectors for different modalities; the original feature vectors include original speech feature vectors, original video feature vectors, and original text feature vectors; The preprocessing involves transforming the raw, irregular multimodal data into a clean, regular, and standardized data format, including the following: Automatic speech recognition technology is used to convert speech streams into intermediate text representations, and then perform deep semantic encoding to generate original speech feature vectors; this achieves high-precision speech recognition and emotional speech synthesis; the intermediate text refers to the preliminary text result recognized from the speech stream, and the original speech feature vector is a high-dimensional numerical vector after deep semantic encoding of the "intermediate text representation"; Keyframes of the video stream are extracted, and the scene content and optical character recognition of the keyframes are analyzed to extract text and lip-reading sequences from the keyframes, forming the original video feature vector; thus realizing environmental information parsing. The standard Braille text is obtained, and the tactile pressure digital signal is converted into a standardized Braille dot matrix code using a lightweight Braille reader, followed by binarization and noise reduction. The standardized Braille dot matrix code is then parsed and converted back into the corresponding standard text to form the original text feature vector. This enables real-time Braille conversion and supports tactile feedback.

[0020] Step S300 involves mapping the original feature vectors of different modalities to the same semantic space for semantic alignment, thereby obtaining standard feature vectors for different modalities. Semantic alignment integrates speech, vision, and tactile modalities, resolving the "curse of dimensionality" and "data heterogeneity." The original data has extremely high dimensionality and completely different formats; direct processing would be computationally intensive and impossible to align. Feature extraction is compressed into low-dimensional vectors, significantly reducing computational complexity and providing a unified data format for subsequent alignment. Simultaneously, the feature extraction process is an abstraction and refinement process; it filters out irrelevant details and retains the core semantic content, enabling comparison of data from different modalities at the semantic level rather than the signal level.

[0021] The semantic alignment includes the following steps: Each original feature vector of a different modality is equipped with a dedicated feature encoder. A cross-modal attention mechanism or a joint embedding model is used to project the original feature vectors of different modalities into the same semantic space to achieve semantic alignment.

[0022] Step S400: Set a dynamic weighted fusion strategy to perform shared embedding space fusion on standard feature vectors of different modalities to generate multimodal fusion feature vectors. P fused ; The dynamic weighted fusion strategy is as follows: Real-time evaluation of the confidence level of the standard feature vectors for different modalities; Based on the confidence level, the weights of the standard feature vectors for different modalities are dynamically adjusted; Based on the aforementioned weights, the standard feature vectors of different modalities are weighted and fused to form a multimodal fused feature vector. P fused .

[0023] For example, lip-reading recognition is weighted more heavily in noisy environments; the lip-reading recognition is implemented through a video modality, specifically including the following: Preset basic video and audio modal weights; When audio is detected in a noisy environment, the signal-to-noise ratio of the audio mode is calculated. SNR Or the entropy of the predicted probability; Based on the signal-to-noise ratio of audio modes in a noisy environment SNR The characteristics of low and high entropy values ​​indicate a decrease in confidence. Based on the evaluation results of the confidence level, the audio modal weights are reduced. w a Increase video modal weights w v ; The audio modal weights w a The calculation expression is: (1) The video modal weights w v The calculation expression is: (2) Wherein, α is a preset signal-to-noise ratio adjustment parameter (a positive smoothing constant / reference signal-to-noise ratio threshold), used to normalize and smooth the audio modal weights, controlling... Follow SNR Sensitivity to change and avoidance SNR When the value is very small, the denominator is too small, leading to unstable weights. α can be obtained through system calibration, experimental training, or empirical setting, and α > 0.

[0024] The prediction results of each modality (such as character probability sequences) are weighted and summed, as shown in the expression: (3) in, The standard feature vector of the video modality; The standard feature vector of the audio modality; The core of dynamic weighted fusion is to dynamically adjust weights based on confidence levels to optimize the performance of a multimodal system. Mathematically, it is based on linear combination and weight normalization. In practice, it requires calculating confidence levels, updating weights, and performing fusion step by step. The increased weighting of lip reading in noisy environments demonstrates the value of modal complementarity—when audio is unreliable, the system adaptively relies on visual information.

[0025] Step S500: Introduce a multimodal intent understanding model to process the multimodal fusion feature vector. P fused Perform deep reasoning to understand the user's true intent; The deep reasoning is achieved through a Transformer-based natural language processing model in the multimodal intent understanding model.

[0026] Step S600: Based on the user's true intent, a personalized multimodal response is generated through the response decision engine to complete this assisted communication; the personalized multimodal response includes response content and an adapted modal combination.

[0027] The response content includes: voice with the desired emotion; explanatory text, highlighted key information or dynamic charts; and Braille vibration signals.

[0028] This invention also provides an auxiliary communication system for special populations, implemented using the aforementioned auxiliary communication method for special populations, such as... Figure 2 As shown, it includes: a multimodal data acquisition and preprocessing module P100, a feature extraction and semantic space alignment module P200, a dynamic fusion and intent understanding module P300, and a personalized multimodal response generation module P400; The modal data acquisition and preprocessing module P100 is used to capture the user's multimodal data in real time, generate multimodal raw feature vectors, and input them into the feature extraction and semantic space alignment module; the multimodal data includes speech streams, video streams, and Braille dot matrix codes; The modal data acquisition and preprocessing module P100 includes an automatic speech recognition unit, a speech synthesis unit, a computer vision, optical character recognition, and lip-reading integration unit, and a Braille translation unit. The automatic speech recognition unit and speech synthesis unit are used to convert between speech streams and text. The computer vision, optical character recognition, and lip-reading integration unit is used to recognize video streams and images. The Braille translation unit is used to convert between Braille dot matrix encoding and standard text. The standard text includes multilingual standard text.

[0029] The automatic speech recognition unit employs ASR and TTS technologies. ASR converts the speech stream into text in real time, while TTS synthesizes the text into speech. The ASR technology, based on the FunASR and Whisper modules, supports multiple dialects and languages. The TTS technology, based on the OpenTTS tool and PaddleSpeech, achieves lightweight emotional speech synthesis. OpenTTS is an integrated platform of numerous open-source TTS engines that converts text into speech based on different contexts and application scenarios. PaddleSpeech provides lightweight end-to-end emotional processing to meet evolving user emotional needs. The FunASR and Whisper modules complement each other in speech recognition. When the system detects that the audio is in Chinese or requires high real-time interaction, FunASR is used first; when the audio may contain multiple languages ​​or requires translation, it is routed to Whisper for processing.

[0030] The computer vision, optical character recognition, and lip reading integration unit integrates computer vision (CV), OCR, and lip reading algorithms, such as... Figure 3 As shown, the computer vision (CV) algorithm uses the YOLO series for real-time object detection; the OCR algorithm extracts text information based on EasyOCR or Tesseract; the lip-reading algorithm infers pronunciation through lip movement sequences, providing visual redundancy for ASR; the YOLO series, with its extremely high speed, treats object detection as a single regression problem, simultaneously predicting the bounding boxes and class probabilities of all objects upon image recognition. This enables the system to process video streams at extremely high frame rates; the functions of EasyOCR and Tesseract in extracting text information in natural scenes and plain printed text, respectively, are complementary. The lip-reading recognition accurately detects and tracks lip regions from video sequences, then extracts the spatiotemporal feature sequences of lip shape changes. An end-to-end sequence model is used to map these visual features to possible phonemes or directly generate text. Computer vision (CV) is responsible for macroscopic environmental perception, while lip-reading recognition provides a second layer of protection against ASR noise failure; the combination of the two provides strong support for contextual awareness of multimodal data.

[0031] The Braille translation unit supports bidirectional conversion between standard text and Braille. Taking Chinese as an example, the output direction drives the dot display or vibration strip to provide tactile feedback, while the input direction collects Braille keyboard input and converts it into text. This is achieved through the Orca reader and the Braille Scanner toolchain. The Orca screen reader acquires standard text via the AT-SPI auxiliary interface, and the Braille Scanner converts the input Braille into specific digital signals, accurately segments the input standard text, converts the segmented words into corresponding pinyin, and determines the tone of each syllable. For polyphonic characters, the correct pronunciation is automatically selected based on the context, combined with a dictionary and an algorithm based on reverse maximum matching word segmentation. Then, the pinyin and tone are mapped to the corresponding Braille dot codes. When the text is presented in Braille form, the system sends the generated Braille dot codes to the Braille dot display. The dot display forms Braille dots by controlling the raising and lowering of tiny electromagnetic columns or piezoelectric ceramic dots, allowing visually impaired users to touch and read. When the input is Braille, the converted standard text can be displayed on the screen for sighted people to read; it can also be synthesized into speech through a TTS (text-to-speech) engine and output in an auditory way.

[0032] The feature extraction and semantic space alignment module P200 is used to map the received original feature vectors to the same shared embedding space, perform semantic alignment, form standard feature vectors of different modalities, and input them to the dynamic fusion and intent understanding module. The semantic alignment includes audio, video, and camera calibration binding lip-reading region alignment; The semantic alignment of the audio and video is achieved by marking timestamps and dynamically adjusting the playback sequence. Specifically, each audio frame and video frame is marked with a timestamp accurate to the millisecond level based on the global clock. Then, the difference between the video frame timestamp and the audio master clock is continuously compared, and the display timing of the video frame is dynamically adjusted according to the difference.

[0033] The camera calibration and lip-reading region alignment are achieved through camera calibration and coordinate transformation, establishing a mapping relationship between 2D image pixels and 3D facial physical space. Specifically: First, the camera's intrinsic and extrinsic parameters are calibrated; then, a facial key point detection model is used to detect the face in real time and locate key points around the lips; finally, through calibration parameters and perspective transformation, these 2D key points are bound to a stable 3D facial model. Even when the head rotates, the algorithm can accurately calculate the lip region (ROI) through the 3D-2D projection relationship.

[0034] The dynamic fusion and intent understanding module P300 is used to dynamically weight and fuse the received standard feature vectors of different modalities, perform deep reasoning, obtain the user's true intent, and input it into the personalized multimodal response generation module. The process of semantic alignment to dynamic fusion is as follows: Figure 4 As shown.

[0035] Millisecond-level synchronization of audio and video frames, camera calibration binding to lip-reading region alignment, and fusion of multimodal standard feature vectors (tokens) in a shared embedding space; during fusion, weighting is applied based on the confidence level of each modality to reduce the risk of single sensor failure; The personalized multimodal response generation module P400 is used to output a personalized multimodal response based on the true intent and the contextual information of the multimodal data; the personalized multimodal response includes response content and an adapted modality combination.

[0036] The context information is formed through the following process: The standard feature vectors of the different modalities are mapped to the same dimension of the shared embedding space; The standard feature vectors of different modalities are concatenated or interleaved and input into a Transformer-based natural language processing model. The standard feature vectors of different modalities interact through a cross-modal attention layer, so that each standard vector contains context from other modalities, forming the contextual information of the multimodal data.

[0037] The contextual information is associated with intent understanding, and cross-modal deep reasoning is achieved by maintaining a multi-turn dialogue state graph.

[0038] When applied to educational scenarios, this invention provides multimodal courseware and interactive exercises, supporting the fusion output of text, voice, Braille, images, and vibration to achieve a personalized teaching feedback loop. During knowledge extraction, knowledge points in curriculum standards or textbooks are deconstructed to form a knowledge graph. Based on the attributes of the knowledge points, the system calls the modal mapping rule base to schedule or generate corresponding materials from the multimedia resource library in real time. Simultaneously, adjustments are made dynamically based on students' preset needs (three types of students with disabilities) or real-time device support. After processing by the multimodal data acquisition and preprocessing module P100 and the feature extraction and semantic space alignment module P200, the data enters the dynamic fusion and intent understanding module P300, which judges the input for errors to obtain the user's true intent. Following the personalized multimodal response generation module P400, the knowledge graph is used to locate underlying knowledge gaps, highlighting correct / incorrect areas on the screen interface, dynamically generating problem-solving animations, or providing encouragement or explanation via TTS (Text-to-Speech) voice, or converting feedback text into Braille code through a Braille translator to drive the Braille display to refresh.

[0039] This invention provides a complete and universally accessible intelligent assistive solution for special groups such as students with disabilities through multimodal fusion technology. It deeply integrates voice, vision, and tactile modalities to achieve natural interaction and high robustness. It has demonstrated excellent performance in pilot trials in special education schools and possesses broad commercial prospects and social value.

[0040] The above-disclosed embodiments are merely specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for assisting communication for specific groups of people, characterized in that, Includes the following steps: Acquire the user's multimodal data; the multimodal data includes audio streams, video streams, and Braille dot matrix codes; The multimodal data types are identified, preprocessed, and original feature vectors of different modalities are generated; the original feature vectors include original speech feature vectors, original video feature vectors, and original text feature vectors. The original feature vectors of different modalities are mapped to the same semantic space, and semantic alignment is performed to obtain standard feature vectors of different modalities. A dynamic weighted fusion strategy is set up to perform shared embedding space fusion of standard feature vectors from different modalities, generating a multimodal fusion feature vector; Introducing a multimodal intent understanding model, the multimodal fusion feature vector is processed. P fused Perform deep reasoning to understand the user's true intent; Based on the user's true intent, a personalized multimodal response is generated through a response decision engine to complete this assisted communication; the personalized multimodal response includes response content and an appropriate combination of modalities.

2. The auxiliary communication method for special groups as described in claim 1, characterized in that, The preprocessing includes: Automatic speech recognition technology is used to convert the speech stream into an intermediate representation in the text, and then deep semantic encoding is performed to generate the original speech feature vector. Key frames are extracted from the video stream, and the scene content and optical character recognition of the key frames are analyzed to extract the text and lip-reading sequences in the key frames to form the original video feature vector. The standard text of Braille is obtained, and the tactile pressure digital signal is converted into a standardized Braille dot matrix code through a lightweight Braille reader, and then binarized for noise reduction. The standardized Braille dot matrix code is parsed and converted into the corresponding standard text to form the original text feature vector.

3. The auxiliary communication method for special groups as described in claim 1, characterized in that, The semantic alignment includes the following: Each original feature vector of a different modality is equipped with a dedicated feature encoder. A cross-modal attention mechanism or a joint embedding model is used to project the original feature vectors of different modalities into the same semantic space to achieve semantic alignment.

4. The auxiliary communication method for special groups as described in claim 1, characterized in that, The dynamic weighted fusion strategy is as follows: Real-time evaluation of the confidence level of the standard feature vectors for different modalities; Based on the confidence level, the weights of the standard feature vectors for different modalities are dynamically adjusted; Based on the aforementioned weights, the standard feature vectors of different modalities are weighted and fused to form a multimodal fused feature vector. P fused .

5. The auxiliary communication method for special groups as described in claim 4, characterized in that, The dynamic adjustment of the weights of the standard feature vectors for different modalities includes increasing the weights of lip reading in noisy environments; the lip reading is implemented through a video modality, specifically including the following: Preset basic video and audio modal weights; When audio is detected in a noisy environment, the signal-to-noise ratio of the audio mode is calculated. SNR Or the entropy of the predicted probability; Based on the signal-to-noise ratio of audio modes in a noisy environment SNR The characteristics of low and high entropy values ​​indicate a decrease in confidence. Based on the evaluation results of the confidence level, the audio modal weights are reduced. w a Increase video modal weights w v ; The audio modal weights w a The calculation expression is: ； The video modal weights w v The calculation expression is: ； Where α is a preset signal-to-noise ratio adjustment parameter.

6. The auxiliary communication method for special populations as described in claim 1, characterized in that, The response content includes: voice with the desired emotion; explanatory text, highlighted key information or dynamic charts; and Braille vibration signals.

7. An auxiliary communication system for special populations, characterized in that, The method for assistive communication for special populations, as described in any one of claims 1-6, includes: a multimodal data acquisition and preprocessing module, a feature extraction and semantic space alignment module, a dynamic fusion and intent understanding module, and a personalized multimodal response generation module; The modal data acquisition and preprocessing module is used to capture the user's multimodal data in real time, generate multimodal raw feature vectors, and input them into the feature extraction and semantic space alignment module; the multimodal data includes speech streams, video streams, and Braille dot matrix codes; The feature extraction and semantic space alignment module is used to map the received original feature vectors to the same shared embedding space, perform semantic alignment, form standard feature vectors of different modalities, and input them to the dynamic fusion and intent understanding module. The dynamic fusion and intent understanding module is used to dynamically weight and fuse the received standard feature vectors of different modalities, perform deep reasoning, obtain the user's true intent, and input it into the personalized multimodal response generation module. The personalized multimodal response generation module is used to output a personalized multimodal response based on the true intent and the contextual information of the multimodal data; the personalized multimodal response includes response content and an adapted modality combination.

8. The auxiliary communication system for special populations as described in claim 7, characterized in that, The modal data acquisition and preprocessing module includes an automatic speech recognition unit, a speech synthesis unit, a computer vision, optical character recognition, and lip reading integration unit, and a Braille translation unit. The automatic speech recognition unit and the speech synthesis unit are used to realize the mutual conversion between speech streams and text. The computer vision, optical character recognition, and lip reading integration unit is used to realize the recognition of video streams and images. The Braille translation unit is used to realize the mutual conversion between Braille dot matrix encoding and standard text.

9. The auxiliary communication system for special populations as described in claim 8, characterized in that, The semantic alignment includes audio, video, and camera calibration binding lip-reading region alignment; the semantic alignment of audio and video is achieved by marking timestamps and dynamically adjusting the playback sequence; the camera calibration binding lip-reading region alignment is achieved by establishing a mapping relationship between 2D image pixels and 3D facial physical space through camera calibration and coordinate transformation.

10. The auxiliary communication system for special populations as described in claim 7, characterized in that, The context information is formed through the following process: The standard feature vectors of the different modalities are mapped to the same dimension of the shared embedding space; The standard feature vectors of different modalities are concatenated or interleaved and input into a Transformer-based natural language processing model. The standard feature vectors of different modalities interact through a cross-modal attention layer, so that each standard vector contains context from other modalities, forming the contextual information of the multimodal data.

Images