Intelligent sign language auxiliary communication method, system, device and medium

Abstract

The invention discloses an intelligent sign language auxiliary communication method. The method comprises the following steps: acquiring video data and audio data of an environment; aI intelligent analysis is carried out on the audio and video data, the video data and the audio data are preprocessed, the identity of each person in the environment is identified from the preprocessed data, independent voice of each person is separated from the multi-sound-source mixed audio, the spatial position of a speaker is positioned, and the timbre feature of each separated voice stream is extracted and marked; performing cross-modal matching on the character, the voice and the timbre, converting the voice into character information, generating a corresponding sign language animation according to the character information, and sending the character information and the sign language animation to the deaf-mute terminal; and sending the text information input by the deaf-mute to the speaker terminal. In a multi-person dialogue scene, the deaf-mute can clearly distinguish different speakers and corresponding speaking contents through the terminal, the real-time performance of communication is ensured, the deaf-mute can accurately understand the intention of the speaker, and the communication efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an intelligent sign language-assisted communication method, system, device, and medium. Background Technology

[0002] With the development of technology, the popularization of smart devices, the increase in network speed, bandwidth, and computing power, 5G networks are ushering in the era of the Internet of Things and the Internet of Models. However, for deaf and mute people, communication in daily life is often done through writing or sign language, while ordinary people often communicate through speech. Deaf and mute people face communication difficulties when communicating with multiple people via speech. Summary of the Invention

[0003] The purpose of this invention is to provide an intelligent sign language-assisted communication method, system, device, and medium, which solves the problem of communication difficulties when deaf-mute people communicate with multiple people via voice in the prior art.

[0004] This invention is achieved through the following technical solution: In a first aspect, the first embodiment of the present invention provides an intelligent sign language-assisted communication method, comprising: Acquire video and audio data of the environment sent by the speaker's terminal; AI intelligent analysis is performed on video and audio data. The video and audio data are preprocessed to identify the identities of people in the environment from the preprocessed data. The independent voice of each person is separated from the mixed audio from multiple sound sources, and the spatial location of the speaker is located. The timbre features of each separated voice stream are extracted and labeled. Cross-modal matching of people, voice and timbre is performed. The voice is converted into text information and the text information is used to generate corresponding sign language animations. The text information and sign language animations are sent to the terminal of the deaf and mute. The system acquires text information input by the deaf-mute user's terminal and sends the input text information to the speaker's terminal.

[0005] Furthermore, it also includes: Retrieve data sent by a deaf or mute user's terminal indicating whether they want to view a specific person or time. Query the corresponding video and audio data based on a specified person or time; Generate text information and sign language animations for a specified person or time based on the queried video and audio data. Send the corresponding text information and sign language animation to the deaf and mute terminal at a specified person or time.

[0006] Furthermore, the specific methods for preprocessing the video and audio data include: By using timestamps, audio and video data are synchronized to obtain synchronized audio and video data. An adaptive noise suppression algorithm is used to process the synchronized audio data to filter out environmental noise and equipment noise. Keyframes are extracted from the synchronized video data, and face frames are preserved. The processed audio data and video keyframes are converted into their respective formats to obtain preprocessed data.

[0007] Furthermore, the specific method for identifying the identities of individuals in the environment from the preprocessed data includes: A multi-task convolutional neural network is used to detect faces in keyframes of video, outputting the bounding box and key points of each face. The detected faces are tracked by a kernel correlation filtering algorithm, and a temporary ID is assigned to each tracked face. After the tracked face image is cropped, it is input into the pre-trained ArcFace model based on the ResNet-50 backbone network, and the model outputs a 128-dimensional face feature vector through the feature extraction layer. The similarity of the facial feature vectors is calculated using cosine similarity. Each person is assigned a person label, and the mapping relationship between the person label, the facial feature vector, and the keyframe image is stored.

[0008] Furthermore, the specific method for separating each person's individual speech from multi-source mixed audio and locating the speaker's spatial position includes: The preprocessed audio data is divided into frames, and the temporal audio is converted into a frequency domain feature map by the Conv-TasNet encoder; The frequency domain feature map is input into the separation network of Conv-TasNet. The separation network learns the differences in spectral features of different sound sources and outputs a separation feature map corresponding to the number of sound sources. The separate feature maps are converted into temporal audio by the Conv-TasNet decoder, resulting in an independent speech stream for each speaker; The localization branch estimates the time delay of the separated individual voice streams, combines the array layout of the terminal device's microphones, calculates the spatial orientation angle of each speaker, and binds each separated voice stream with its corresponding spatial orientation angle to form a voice stream spatial location pair.

[0009] Furthermore, the specific method for extracting and labeling the timbre features of each separated speech stream includes: Each separated individual vocal stream is preprocessed to extract timbre-related features and form a feature matrix. The features include Mel frequency cepstral coefficients, fundamental frequency, spectral flatness, and spectral entropy. The feature matrix is ​​input into a CNN model, which includes three convolutional layers and two fully connected layers. The first convolutional layer extracts local timbre features, the second convolutional layer extracts mid-level timbre features, and the third convolutional layer extracts global timbre features. The dimensionality is reduced by using the ReLU activation function and pooling layers, and a 256-dimensional timbre feature vector is output by the fully connected layers. Assign a timbre label to the timbre feature vector of each separated speech stream.

[0010] Furthermore, the specific method for generating corresponding sign language animations from text information includes: The Stable Diffusion image model is trained using sign language action videos and text descriptions to obtain a trained Stable Diffusion image model; Text information is input into a trained Stable Diffusion image model to generate corresponding sign language animations.

[0011] Secondly, another embodiment of the present invention provides an intelligent sign language-assisted communication system for implementing the intelligent sign language-assisted communication method described in the above embodiments. The system includes: The data acquisition module is used to acquire video and audio data of the environment sent by the speaker's terminal; The data processing module is used for AI intelligent analysis of video and audio data, preprocessing video and audio data, identifying the identities of people in the environment from the preprocessed data, separating the independent speech of each person from the mixed audio of multiple sound sources, locating the spatial position of the speaker, extracting and labeling the timbre features of each separated speech stream, performing cross-modal matching of people, speech and timbre, converting speech into text information, generating corresponding sign language animation from the text information, and sending the text information and sign language animation to the terminal for the deaf and mute. The data transmission module is used to acquire the text information input by the deaf-mute user's terminal and send the input text information to the speaker's terminal.

[0012] Thirdly, another embodiment of the present invention provides an electronic device comprising: a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to invoke the program instructions to execute the method described in the first embodiment above.

[0013] Fourthly, another embodiment of the present invention provides a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in the first embodiment above.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects: This invention provides an intelligent sign language-assisted communication method, system, device, and medium. Through facial recognition, multi-source sound separation and localization technology, it accurately achieves one-to-one correspondence marking of speaker, voice, and timbre. In multi-person dialogue scenarios, deaf and mute people can clearly distinguish different speakers and corresponding speech content through the terminal, ensuring real-time communication, enabling deaf and mute people to accurately understand the speaker's intentions, and improving communication efficiency. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating an intelligent sign language-assisted communication method provided in the first embodiment of the present invention; Figure 2 This is a structural block diagram of an intelligent sign language-assisted communication system provided in another embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0017] like Figure 1 As shown, the first embodiment of the present invention provides an intelligent sign language-assisted communication method, which includes the following steps: Acquire video and audio data of the environment sent by the speaker's terminal; AI intelligent analysis is performed on video and audio data. The video and audio data are preprocessed to identify the identities of people in the environment from the preprocessed data. The independent voice of each person is separated from the mixed audio from multiple sound sources, and the spatial location of the speaker is located. The timbre features of each separated voice stream are extracted and labeled. Cross-modal matching of people, voice and timbre is performed. The voice is converted into text information and the text information is used to generate corresponding sign language animations. The text information and sign language animations are sent to the terminal of the deaf and mute. The system acquires text information input by the deaf-mute user's terminal and sends the input text information to the speaker's terminal.

[0018] This invention provides an intelligent sign language-assisted communication method. By preprocessing audio and video data, the method identifies the speaker in the environment from the preprocessed data, marks the identified person, records their voice timbre, and matches the timbre with the person's signature. The audio is converted into text, and the text is then converted into sign language animation and sent to a deaf-mute terminal for playback. During playback, the current speaker is marked. If the deaf-mute user receives feedback, they can input text and send it to the speaker's terminal. In multi-person dialogue scenarios, the deaf-mute user can clearly distinguish different speakers and their corresponding content through the terminal, ensuring real-time communication, enabling them to accurately understand the speaker's intentions, and improving communication efficiency.

[0019] In another embodiment of the present invention, a specific method for preprocessing video data and audio data includes: By using timestamps, audio and video data are precisely synchronized. The audio data sampling rate is uniformly set to 16kHz with a bit depth of 16bit, while the video data resolution is adjusted to 1080P with a fixed frame rate of 30fps. After processing, synchronized audio and video data are obtained with an error controlled within 10ms to avoid timing deviations during subsequent face and voice matching. An adaptive noise suppression algorithm is used to process the synchronized audio data to filter out environmental noise (such as wind noise, background music, etc.) and equipment noise (such as microphone background noise) while preserving the purity of the human voice signal. Keyframe extraction is performed on the synchronized video data. Specifically, one frame is retained for every two frames. Frames with severe blurriness or motion blur are removed, and clear face frames are retained for subsequent recognition, thereby reducing the computational load of the algorithm. The processed audio data and video keyframes are converted to different formats to obtain preprocessed data. The processed audio data is converted to PCM format, and the video keyframes are converted to RGB format.

[0020] The above steps are used to preprocess the audio and video data to ensure that the data format and quality meet the input requirements of the AI ​​algorithm.

[0021] In this embodiment of the invention, the specific method for identifying the identities of individuals in the environment from preprocessed data includes: Multi-task convolutional neural network (MTCNN) is used to detect faces in keyframes of video, outputting bounding boxes (x, y, w, h) and key points (5 key points in total, including eyes, nose tip, and corners of mouth) for each face. Kernel correlation filtering (KCF) algorithm is used to track the detected faces, and a temporary ID is assigned to each tracked face to ensure the consistency of temporary IDs for the same person in different frames and poses. After the tracked face image is cropped, it is input into the pre-trained ArcFace model based on the ResNet-50 backbone network. The model outputs a 128-dimensional face feature vector through the feature extraction layer. In this process, the ArcFace model enhances the distinguishability of different face feature vectors by introducing Angular Margin Loss. Even if the person has a side profile (pose angle ≤ 45°), head down / head up (pitch angle ≤ 30°), or sudden changes in lighting (such as from indoors to outdoors), it can still extract features stably. The similarity of the facial feature vectors is calculated using cosine similarity. If the similarity of the facial feature vectors of the same temporary ID is ≥0.85, they are determined to be the same person. If the similarity of the facial feature vectors of different temporary IDs is ≥0.9, they are determined to be duplicate persons. The IDs are merged, a person tag is assigned to each person, and the mapping relationship between the person tag, the facial feature vector and the keyframe image is stored.

[0022] Specific methods for separating individual speech from multi-source mixed audio and locating the speaker's spatial position include: The preprocessed audio data is divided into frames of 20ms each with a 50% overlap rate. The temporal audio is converted into a frequency domain feature map by the encoder of Conv-TasNet (fully convolutional temporal audio separation network). The encoder consists of 8 layers of one-dimensional convolution with a kernel size of 3. The frequency domain feature map is input into the separation network of Conv-TasNet. The separation network contains 16 separation blocks. Each separation block consists of causal convolution, layer normalization, and gated activation units. The separation network learns the differences in spectral features of different sound sources and outputs a separation feature map corresponding to the number of sound sources. For example, if there are 3 people speaking in the scene, 3 separation feature maps will be output. The Conv-TasNet decoder converts the separated feature maps into temporal audio, obtaining an independent speech stream for each speaker, thus achieving the separation of mixed audio into a single voice. The decoder is symmetrical to the encoder and consists of 8 layers of one-dimensional deconvolution. While Conv-TasNet separates the audio, the localization branch estimates the time delay of the separated individual voice streams. Combined with the array layout of the terminal device's microphones (assuming dual microphones on a mobile phone with a spacing of 5cm), it calculates the spatial orientation angle of each speaker: horizontal angle θ: -90°~90°, vertical angle φ: -30°~30°. Each separated speech stream is bound to its corresponding spatial orientation angle to form a speech stream spatial location pair, providing spatial constraints for subsequent association with the people labeled by the ArcFace model.

[0023] Through the above steps, the individual voices of each person can be accurately separated from the mixed audio from multiple sources, and the spatial location of the speaker can be located, thus solving the problems of who is speaking and where the speaker is.

[0024] In this embodiment, the specific method for extracting and labeling the timbre features of each separated speech stream includes: Each separated individual vocal stream is preprocessed to extract timbre-related features and form a feature matrix. The features include Mel frequency cepstral coefficients, fundamental frequency, spectral flatness, and spectral entropy. The feature matrix is ​​input into a CNN model, which includes three convolutional layers and two fully connected layers. The first convolutional layer extracts local timbre features with a 3×3 kernel, a stride of 1, and 64 output channels. The second convolutional layer extracts mid-level timbre features with a 3×3 kernel, a stride of 1, and 128 output channels. The third convolutional layer extracts global timbre features with a 5×5 kernel, a stride of 2, and 256 output channels. The dimensionality is reduced by using a ReLU activation function and a 2×2 max pooling layer. Finally, a 256-dimensional timbre feature vector is output through a fully connected layer. A timbre label is assigned to the timbre feature vector of each separated speech stream, and the association between the timbre label, speech stream and spatial location is established and stored.

[0025] Through the above steps, the features of each separated speech stream can be accurately extracted and labeled, and the speaker's speech features can be obtained, providing a speech basis for accurate matching of face and speech.

[0026] Specific methods for cross-modal matching of people, voices, and timbres include: Convert the center point coordinates of the facial bounding box of each character into their relative position on the screen; Match the relative position of the image with the spatial orientation angle of the speaker. If the relative position of a person's image deviates from the orientation angle of a certain speech stream by ≤15°, it is initially determined to be a candidate matching pair. A deep learning model is used to train a correlation model between facial features and timbre features (based on the Siamese network). The facial feature vector and timbre feature vector of the candidate matching pair are input, and the correlation score between the two is calculated. If the score is ≥0.75, the match is considered successful. After a successful match, a complete mapping relationship is established for person tagging, speech stream, timbre tagging, spatial location, and temporary ID, and output to subsequent modules. If there is an unmatched speech stream (such as someone speaking in the scene but not being detected by face), it is marked as "unknown speaker" and its speech stream and timbre features are preserved.

[0027] Through the above steps, a one-to-one correspondence between person, voice, and timbre is achieved. The complete mapping relationship after cross-modal matching (person tag, voice stream, timbre feature vector, spatial location, matching confidence) is associated with the original audio and video data (timestamp, keyframe) and stored in the database. A timestamp range is added to each mapping relationship to support subsequent queries by specified person or specified time period. The correspondence between person tag and current speaker is sent to the deaf and mute terminal in real time for visual marking of the current speaker on the deaf and mute terminal screen, such as displaying a "speaking" indicator next to the person's portrait.

[0028] In this embodiment, the specific method for generating corresponding sign language animations from text information includes: The Stable Diffusion image model is trained using sign language action videos and text descriptions to obtain a trained Stable Diffusion image model; Text information is input into a trained Stable Diffusion image model to generate corresponding sign language animations.

[0029] Specific methods for training Stable Diffusion-based image models using sign language action videos and text descriptions include: First, sign language video and image data are collected, keyframes are extracted and labeled to form an image-text description pairing dataset. Data augmentation is performed simultaneously. Then, a pre-trained Stable Diffusion base model (such as SD 1.5) and CLIP text encoder are loaded.

[0030] The core training phase employs a fine-tuning method, inputting sign language images and text descriptions into the model. The UNet network learns the denoising process, predicting noise under textual guidance. The parameters are optimized using a diffusion model loss function, enabling the model to learn the visual features and semantic associations of sign language gestures.

[0031] Configure inference parameters: Sampler: Euler a, balancing generation speed and quality; Number of iterations: 30 steps, balancing real-time performance and animation smoothness; Guidance Scale (text guidance weight): 7.5, which ensures that the generated results strictly follow the Prompt semantics; Rendered resolution: 1080P, frame rate: 30fps; Output format: PNG sequence frames, each frame corresponding to a sign language action keyframe.

[0032] The CLIP text encoder converts the composite Prompt into a 768-dimensional text embedding vector, which is then input into the StableDiffusion model. Simultaneously, it initializes a random noise tensor with dimensions consistent with the generated image: [1, 3, 1080, 1920], serving as the starting point for generation.

[0033] Guided by text embedding vectors, the model's UNet network performs 30 steps of denoising iterations on the random noise tensor. In each step, the noise is predicted and subtracted from the tensor, gradually generating a sign language image that closely approximates the description in the Prompt.

[0034] During the iteration process, the model calls the sign language action semantic association library to ensure that each keyword corresponds to a standard sign language action. After generating PNG sequence frames, the model uses an interpolation algorithm to fill in the transition frames between adjacent keyframes, increasing the action frame rate from 30fps to 60fps and eliminating motion stuttering. At the same time, the model uses a posture optimization algorithm to adjust the angle of finger joints and the trajectory of arm movements to ensure that the movements are natural and smooth.

[0035] After training, the quality of the generated data is evaluated, including image sharpness, gesture accuracy, and semantic consistency. The final model can be used to generate sign language teaching materials, accessible communication aids, and diverse sign language action images, supporting sign language education and related application scenarios. The entire process achieves targeted optimization from a general text-to-image model to a sign language-specific generative model.

[0036] By training a customized Stable Diffusion sign language model and combining it with a sign language action semantic association library, the generated sign language animation strictly follows the Chinese standard sign language specifications, ensuring the accuracy of gestures, the continuity of actions, and the consistency of semantics. This solves the problems of non-standard sign language actions and semantic deviations in general models, enabling deaf and mute people to accurately understand the communication intentions.

[0037] The intelligent sign language-assisted communication method provided in this embodiment of the invention further includes: Retrieve data sent by a deaf or mute user's terminal indicating whether they want to view a specific person or time. Query the corresponding video and audio data based on a specified person or time; Generate text information and sign language animations for a specified person or time based on the queried video and audio data. Send the corresponding text information and sign language animation to the deaf and mute terminal at a specified person or time.

[0038] The above steps enable deaf and mute individuals to review the content of a speech given by a specific person or during a specific time period. They can easily retrieve any missed information, preventing the loss of communication content, which is especially useful in important communication scenarios.

[0039] This invention provides an intelligent sign language-assisted communication method that uses facial recognition, multi-source sound separation and localization technology to accurately identify the speaker, voice, and timbre. In multi-person dialogue scenarios, deaf and mute individuals can clearly distinguish different speakers and their corresponding speech content through the terminal, ensuring real-time communication, enabling them to accurately understand the speaker's intentions, and improving communication efficiency.

[0040] like Figure 2 As shown, another embodiment of the present invention provides an intelligent sign language-assisted communication system for implementing the intelligent sign language-assisted communication method described in the first embodiment above. The system includes: The data acquisition module is used to acquire video and audio data of the environment sent by the speaker's terminal; The data processing module is used for AI intelligent analysis of video and audio data, preprocessing video and audio data, identifying the identities of people in the environment from the preprocessed data, separating the independent speech of each person from the mixed audio of multiple sound sources, locating the spatial position of the speaker, extracting and labeling the timbre features of each separated speech stream, performing cross-modal matching of people, speech and timbre, converting speech into text information, generating corresponding sign language animation from the text information, and sending the text information and sign language animation to the terminal for the deaf and mute. The data transmission module is used to acquire the text information input by the deaf-mute user's terminal and send the input text information to the speaker's terminal.

[0041] The execution process of each module can be carried out according to the steps of the intelligent sign language assisted communication method provided in the first embodiment, and will not be described in detail in this embodiment.

[0042] The intelligent sign language-assisted communication system and the intelligent sign language-assisted communication method provided in this embodiment of the invention are based on the same inventive concept and have the same beneficial effects, and will not be described in detail here.

[0043] Another embodiment of the present invention provides an electronic device, which includes a processor, an input device, an output device, and a memory. The processor, the input device, the output device, and the memory are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to call the program instructions to execute the intelligent sign language assisted communication method described in the first embodiment above.

[0044] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0045] Input devices may include touchpads, microphones, etc., and output devices may include displays (LCDs, etc.), speakers, etc.

[0046] The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information about the device type.

[0047] In specific implementations, the processor, input device, and output device described in the embodiments of the present invention can execute the implementation of the method embodiments described in the embodiments of the present invention, or they can execute the implementation of the system embodiments described in the embodiments of the present invention, which will not be repeated here.

[0048] The present invention also provides an embodiment of a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the intelligent sign language-assisted communication method described in the first embodiment above.

[0049] The computer-readable storage medium can be an internal storage unit of the terminal described in the foregoing embodiments, such as the terminal's hard drive or memory. The computer-readable storage medium can also be an external storage device of the terminal, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the terminal. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0050] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0051] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the terminals and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0052] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A smart sign language-assisted communication method, characterized in that, include: Acquire video and audio data of the environment sent by the speaker's terminal; AI intelligent analysis is performed on video and audio data. The video and audio data are preprocessed to identify the identities of people in the environment from the preprocessed data. The independent voice of each person is separated from the mixed audio from multiple sound sources, and the spatial location of the speaker is located. The timbre features of each separated voice stream are extracted and labeled. Cross-modal matching of people, voice and timbre is performed. The voice is converted into text information and the text information is used to generate corresponding sign language animations. The text information and sign language animations are sent to the terminal of the deaf and mute. The system acquires text information input by the deaf-mute user's terminal and sends the input text information to the speaker's terminal.

2. The intelligent sign language-assisted communication method according to claim 1, characterized in that, Also includes: Retrieve data sent by a deaf or mute user's terminal indicating whether they want to view a specific person or time. Query the corresponding video and audio data based on a specified person or time; Generate text information and sign language animations for a specified person or time based on the queried video and audio data. Send the corresponding text information and sign language animation to the deaf and mute terminal at a specified person or time.

3. The intelligent sign language-assisted communication method according to claim 1, characterized in that, The specific methods for preprocessing video and audio data include: By using timestamps, audio and video data are synchronized to obtain synchronized audio and video data. An adaptive noise suppression algorithm is used to process the synchronized audio data to filter out environmental noise and equipment noise. Keyframes are extracted from the synchronized video data, and face frames are preserved. The processed audio data and video keyframes are converted into their respective formats to obtain preprocessed data.

4. The intelligent sign language-assisted communication method according to claim 3, characterized in that, The specific method for identifying the identities of individuals in the environment from preprocessed data includes: A multi-task convolutional neural network is used to detect faces in keyframes of video, outputting the bounding box and key points of each face. The detected faces are tracked by a kernel correlation filtering algorithm, and a temporary ID is assigned to each tracked face. After the tracked face image is cropped, it is input into the pre-trained ArcFace model based on the ResNet-50 backbone network, and the model outputs a 128-dimensional face feature vector through the feature extraction layer. The similarity of the facial feature vectors is calculated using cosine similarity. Each person is assigned a person label, and the mapping relationship between the person label, the facial feature vector, and the keyframe image is stored.

5. The intelligent sign language-assisted communication method according to claim 4, characterized in that, The specific method for separating each person's individual speech from multi-source mixed audio and locating the speaker's spatial position includes: The preprocessed audio data is divided into frames, and the temporal audio is converted into a frequency domain feature map by the Conv-TasNet encoder; The frequency domain feature map is input into the separation network of Conv-TasNet. The separation network learns the differences in spectral features of different sound sources and outputs a separation feature map corresponding to the number of sound sources. The separate feature maps are converted into temporal audio by the Conv-TasNet decoder, resulting in an independent speech stream for each speaker; The localization branch estimates the time delay of the separated individual voice streams, combines the array layout of the terminal device's microphones, calculates the spatial orientation angle of each speaker, and binds each separated voice stream with its corresponding spatial orientation angle to form a voice stream spatial location pair.

6. The intelligent sign language-assisted communication method according to claim 5, characterized in that, The specific methods for extracting and labeling the timbre features of each separated speech stream include: Each separated individual vocal stream is preprocessed to extract timbre-related features and form a feature matrix. The features include Mel frequency cepstral coefficients, fundamental frequency, spectral flatness, and spectral entropy. The feature matrix is ​​input into a CNN model, which includes three convolutional layers and two fully connected layers. The first convolutional layer extracts local timbre features, the second convolutional layer extracts mid-level timbre features, and the third convolutional layer extracts global timbre features. The dimensionality is reduced by using the ReLU activation function and pooling layers, and a 256-dimensional timbre feature vector is output by the fully connected layers. Assign a timbre label to the timbre feature vector of each separated speech stream.

7. The intelligent sign language-assisted communication method according to claim 6, characterized in that, The specific method for generating corresponding sign language animations from text information includes: The Stable Diffusion image model is trained using sign language action videos and text descriptions to obtain a trained Stable Diffusion image model; Text information is input into a trained Stable Diffusion image model to generate corresponding sign language animations.

8. An intelligent sign language-assisted communication system, characterized in that, For implementing the intelligent sign language-assisted communication method as described in any one of claims 1-7, the system comprises: The data acquisition module is used to acquire video and audio data of the environment sent by the speaker's terminal; The data processing module is used for AI intelligent analysis of video and audio data, preprocessing video and audio data, identifying the identities of people in the environment from the preprocessed data, separating the independent speech of each person from the mixed audio of multiple sound sources, locating the spatial position of the speaker, extracting and labeling the timbre features of each separated speech stream, performing cross-modal matching of people, speech and timbre, converting speech into text information, generating corresponding sign language animation from the text information, and sending the text information and sign language animation to the terminal for the deaf and mute. The data transmission module is used to acquire the text information input by the deaf-mute user's terminal and send the input text information to the speaker's terminal.

9. An electronic device, comprising: The system includes a processor, an input device, an output device, and a memory interconnected thereto, the memory storing a computer program comprising program instructions, wherein the processor is configured to invoke the program instructions to execute the intelligent sign language-assisted communication method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the intelligent sign language-assisted communication method as described in any one of claims 1-7.

Images