A wake-up assistive robot based on generative deep neural networks

Abstract

A robot for promoting awakening and assisting patients based on a generative deep neural network relates to the field of cognitive therapy. The robot can realize the function of virtual relative image accompanying and is suitable for fields such as cognitive therapy. The robot uses artificial intelligence technologies such as natural language processing and deep learning to assist in improving the consciousness disorders of patients with craniocerebral injury and promote their awakening. Steps: 1: Collect audio, video or photos of the patient's relatives as conditional input to obtain the data to be processed; 2: Perform pre-processing steps such as denoising, target detection and face recognition on the data to be processed; 3: Use a deep learning model to generate awakening-promoting accompanying audio with a similar timbre to the target; 4: Based on the generated audio, use image processing algorithms to match and adjust the character posture, lip shape and expression in the original video frame; 5: Recombine the synthesized video frames into a video sequence, output it to the user and save it. Provide more convenient and effective rehabilitation assistance.

Description

Technical Field

[0001] The present invention relates to the field of cognitive therapy, and in particular to cognitive stimulation technology for patients with craniocerebral injury and impaired consciousness. It adopts artificial intelligence technologies such as natural language processing and deep learning, and realizes an awakening-assisting robot based on a generative deep neural network through a front-end and back-end separated architecture. Background Art

[0002] Impaired consciousness refers to a state of loss of consciousness caused by various severe brain injuries, such as coma, vegetative state, and minimally conscious state. Due to severe damage to brain tissue, patients often experience impaired consciousness or even fall into a deep coma, with loss of cognitive ability and inability to move actively. The clinical treatment for impaired consciousness is usually sensory stimulation therapy, which is the systematic stimulation of one or more of the senses of vision, hearing, smell, taste, touch, and movement. Among them, family-centered sensory stimulation is more accepted than traditional sensory stimulation, but due to work or other reasons, many family members find it difficult to accompany the patient for a long time.

[0003] Therefore, the awakening-assisting robot can assist medical staff in treatment by generating sensory stimulation videos targeting the patient's family members, through both auditory and visual stimulation. Summary of the Invention

[0004] The purpose of the present invention is to provide a robot that promotes awakening and assists patients based on a generative deep neural network, which can realize the function of virtual relative image care and is suitable for fields such as cognitive therapy. The robot uses artificial intelligence technologies such as natural language processing and deep learning to assist in improving the consciousness disorders of patients with craniocerebral injury and promote their awakening. The robot uses artificial intelligence technologies such as natural language processing and deep learning to generate sounds, images and scenes that are familiar to patients with consciousness disorders, and provides sensory stimulation therapy and clinical care to patients. The robot system adopts a front-end and back-end separated architecture, using Vue for the front end, Spring Boot for the back end, and Flask for the algorithm end. Users upload videos through the front end, the back end calls the algorithm for processing, and the processing results are returned to the front end.

[0005] The present invention provides a wake-up assistance robot based on a generative deep neural network, comprising a front-end module, a back-end module, and an algorithm microservice module;

[0006] The front-end module is used to receive audio, image, video and other data of the target person uploaded by the user, and provide users with functional options such as content modification and pre-processing;

[0007] The backend module includes a storage module for receiving data to be processed, a function selection module that can call the corresponding algorithm microservice module for processing, and an EEG signal monitoring module;

[0008] The algorithm microservice module includes a speech generation submodule, a face detection submodule, a video generation submodule, and an output processing submodule;

[0009] The speech generation submodule is used to perform speech synthesis on the target text based on the uploaded video, extract the spectrum, tone, and speech speed of the target person's speech signal, and convert the text into speech output;

[0010] The face detection submodule is used to locate and segment the human area in the video frame and output the corresponding coordinates and masks;

[0011] The video generation submodule uses the synthesized speech and the character area information output by the face detection submodule to accurately drive facial expressions and lip shapes, seamlessly integrate the synthesized face image into the input video stream, and reassemble the synthesized video frames into a video sequence;

[0012] The output processing submodule is used to integrate and optimize the video frames processed by the speech synthesis, face detection, and video generation modules so that they can be played correctly by the front end;

[0013] Furthermore, the front-end module is also used to receive the processing results sent by the back-end module, display them on the interface, and provide various functional options, such as saving, sharing, printing, etc.

[0014] Furthermore, the backend module is also used to manage multiple algorithm microservice modules and select appropriate algorithm microservice modules according to different generation projects and scenarios.

[0015] Furthermore, the EEG monitoring module in the back-end module integrates an EEG signal analysis algorithm to analyze EEG activities in different frequency bands (alpha waves, beta waves) to determine the level of brain activity, thereby identifying the content type that can stimulate the most active EEG signals by comparing the EEG signal responses under different playback contents.

[0016] Furthermore, the speech generation submodule includes:

[0017] A preprocessing submodule that removes background noise from the input audio based on the UVR5 model to improve the purity of the audio.

[0018] An audio recognition submodule, based on the FunASR model, converts the input speech into corresponding text, enabling subsequent text processing modules to operate based on accurate text information.

[0019] A character embedding submodule converts discrete character data into high-dimensional vector representations, enabling the model to process text information and capture the semantic relationships between characters, making the model more natural and coherent when generating speech.

[0020] An encoder submodule, implemented by Conformer, which extracts contextual information of character embeddings and combines convolution and self-attention mechanisms to generate a series of feature vectors that can better capture long-term and short-term dependencies.

[0021] A decoder submodule recursively processes the context vector and previous output to construct a complete mel-spectrogram, which provides input to the subsequent vocoder module.

[0022] A vocoder submodule that uses a high-fidelity vocoder HIFI-GAN based on a generative adversarial network to generate more natural audio waveforms.

[0023] A post-processing enhancement module that uses SSRN (Super Resolution Network) technology to improve the clarity and quality of audio.

[0024] Furthermore, the face detection submodule processes the video frame using the S3FD model to obtain the coordinates and mask representation of the face area.

[0025] Furthermore, the video generation submodule includes:

[0026] Two encoder submodules perform feature extraction on the input through multiple convolutional layers and residual blocks, extracting high-level audio and video features layer by layer;

[0027] A feature fusion submodule, implemented through feature concatenation, is used to fuse audio features with facial features. This submodule concatenates the high-level features extracted by the audio encoder and the facial encoder to form a joint feature representation, providing rich information for the subsequent decoding process;

[0028] A decoder submodule, including multiple layers of deconvolution layers and residual blocks, is used to decode the fused features and generate feature maps;

[0029] An output submodule, which converts the decoded feature map into a pixel-level image representation through a convolutional layer and uses a Sigmoid activation function to normalize the output pixel values ​​to the range [0, 1] to ensure that the pixel values ​​of the generated image are within a reasonable range;

[0030] A discriminative submodule evaluates the quality and similarity of the generated results by comparing the generated images with the real images of the target person, ensuring that the generated images are highly visually consistent with the target person.

[0031] The present invention also provides a method for using a wake-up assistance robot based on a generative deep neural network, comprising the following steps:

[0032] Step 1: Through the input of the awakening assistance robot camera or uploading to the file system of the awakening assistance robot through the cloud, collect the audio, video or photos of the patient's family as conditional input to obtain the data to be processed;

[0033] Step 2: Preprocess the data to be processed, including denoising, target detection, face recognition and other preprocessing steps to improve the effect of subsequent processing;

[0034] Step 3: Use natural language processing algorithms and deep learning models to extract acoustic features from the input audio data, and generate awakening-promoting and accompanying audio with a similar timbre to the target based on the input speech stimulus text;

[0035] Step 4: Based on the generated audio, a variety of deep learning algorithms are integrated to process the speech signal and video frames to synchronize the generated speech with the target person's lip shape;

[0036] Step 5: Integrate and optimize the video frames processed by the speech synthesis, face detection, and video generation modules, reassemble them into a video sequence, output it to the user through the robot display and audio equipment, and save the record on the back end.

[0037] Step 6: Based on the analysis results of the EEG signals of the awakening assistance robot's external EEG headset, dynamically adjust the playback content to enhance patient participation and training effect.

[0038] The beneficial effects of the present invention are:

[0039] This invention provides a resuscitation-assistance robot based on a generative deep neural network. This robot utilizes artificial intelligence technologies such as natural language processing and deep learning to generate sounds, images, and scenes familiar to patients with impaired consciousness, providing sensory stimulation therapy. This can effectively reduce the burden of accompanying caregivers, improve the efficiency of impaired consciousness treatment, and provide more comprehensive rehabilitation services for patients.

[0040] The robotics system utilizes a separate front-end and back-end architecture, using Vue for the front-end, Spring Boot for the back-end, and Flask for the algorithm side. Users upload audio and video data through the front-end, which then processes the data using algorithms and returns the results to the front-end. The system is efficient, flexible, scalable, and easy to use.

[0041] This paper uses the deep learning S3FD model to process video frames and obtain the coordinates and mask representation of the facial region. The S3FD model possesses multi-scale feature detection capabilities, accurately locating the facial region within a video frame and outputting the coordinates and corresponding mask representation of the region. This process offers the advantages of high accuracy, robustness, and low computational complexity for face detection, providing a reliable foundation for subsequent processing steps.

[0042] Based on the target person's speech signal's speech characteristics, such as spectrum, intonation, and speech rate, this method uses natural language processing algorithms and deep learning network models to process input speech, generating generated speech based on the speech stimulus text and possessing the target person's vocal characteristics. Based on the generated speech, the method processes video frames using algorithms such as depthwise separable convolution and attention mechanisms to generate a perceptually stimulating video similar to the target person. This video generation method offers advantages such as high similarity, high controllability, and high practicality.

[0043] The present invention integrates and optimizes the generated video frames, sends them to the front-end user, and provides a function of saving them in the back-end.

[0044] The present invention dynamically adjusts the playback content based on the analysis results of the EEG signal module, so that medical staff and family members can better stimulate the perception of patients with impaired consciousness and provide targeted training, adjustment and treatment. The visual display method of the present invention has the advantages of being intuitive, clear and easy to understand. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the overall architecture diagram of the present invention.

[0046] Figure 2 It is the overall flow chart of the system of the present invention.

[0047] Figure 3 It is a flow chart of the front-end module in the present invention.

[0048] Figure 4 It is a flow chart of the back-end module in the present invention.

[0049] Figure 5 It is a flow chart of the algorithm module in the present invention.

[0050] Figure 6 It is a schematic diagram of the audio generation method in the present invention.

[0051] Figure 7 It is a schematic diagram of the video generation method in the present invention.

[0052] Figure 8 It is a schematic diagram of the output processing method in the present invention.

[0053] Figure 9 Schematic diagram of the EEG monitoring method of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the following embodiments will be further described with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention. On the contrary, the present invention encompasses any alternatives, modifications, equivalent methods, and solutions made within the spirit and scope of the present invention as defined by the claims.

[0055] Figure 1 This is the overall architecture diagram of the present invention. Figure 1 As shown, the present invention provides a wake-up assisting robot based on a generative deep neural network, comprising:

[0056] The front-end module is used to receive audio, image, video and other data of the target person uploaded by the user, and provide users with functional options such as content modification and pre-processing. The user collects audio, video or photos of the patient's family as conditional input through the input of the awakening assistance robot's camera or uploads them to the awakening assistance robot's file system through the cloud to obtain the data to be processed.

[0057] The backend module involves the processing and management of user information, including a storage module for receiving data to be processed, a function selection module that can call the corresponding algorithm microservice module for processing, and an EEG signal monitoring module; the EEG signal monitoring module is used to monitor and analyze the user's EEG signal data in real time;

[0058] The algorithm microservice module includes a speech generation submodule, a face detection submodule, a video generation submodule, and an output processing submodule;

[0059] The speech generation submodule is used to perform speech synthesis on the target text based on the uploaded video, extract the speech features of the target person's speech signal, such as spectrum, tone, and speaking speed, and convert the text into speech output;

[0060] The face detection submodule is used to locate and segment the human area in the video frame and output the corresponding coordinates and masks;

[0061] The video generation submodule uses the synthesized speech and the character area information output by the face detection submodule to accurately drive facial expressions and lip shapes, seamlessly integrate the synthesized face image into the input video stream, and reassemble the synthesized video frames into a video sequence;

[0062] The output processing submodule is used to integrate and optimize the video frames processed by the speech synthesis, face detection, and video generation modules so that they can be played correctly by the front end;

[0063] like Figure 2As shown, the user uploads the target person's audio, video or photo from the cloud through the robot touch screen, selects the preset speech stimulus text and preprocessing method, clicks the start generation button, and the front-end module sends the uploaded data to the back-end module.

[0064] like Figure 3 As shown, after receiving the uploaded data, the back-end module calls the corresponding algorithm microservice for processing. In this embodiment, the algorithm microservice called by the back-end module includes a speech generation submodule, a face detection submodule, a video generation submodule, and an output processing submodule.

[0065] like Figure 4 As shown in the figure, after receiving the uploaded data, the algorithm microservice first confirms the data type. If audio data is detected, the system will call the speech generation submodule to perform a series of processing on the speech signal in the audio data.

[0066] like Figure 5 As shown in the figure, the speech generation submodule achieves the goal of speech synthesis by converting audio data into high-quality speech output. This module consists of multiple submodules, each of which has a specific task and works together to ensure the final speech output quality. Specifically:

[0067] After receiving audio data, the speech generation submodule first calls the preprocessing module. Based on the UVR5 model and the processing algorithm selected by the front-end user, this module performs voice separation and reverberation noise reduction. Voice separation isolates the human voice from background noise or other sound sources, while reverberation noise reduction uses algorithms to reduce echoes and ambient noise, making the human voice clearer and ensuring audio signal quality.

[0068] The system calls the speech recognition model to extract text information from the preprocessed audio. The FunASR model, based on the Transformer architecture, analyzes the speech signals in the audio and generates corresponding text transcriptions, which lay the foundation for subsequent semantic processing.

[0069] The character embedding submodule uses an embedding layer to convert each character into a fixed-size vector, transforming text data into a vector representation that captures semantic relationships. Specifically, the embedding layer uses an embedding matrix, where each row corresponds to a character in the vocabulary and the contents of the row are the vector representation of that character. When processing input text, each character is first converted to its index in the vocabulary. These indices are then used to search the embedding matrix to obtain the corresponding vector representation. These vector representations preserve the semantic information of the characters, enabling the model to better understand and process text data.

[0070] The encoder submodule is a key component of the speech generation submodule. Its primary function is to convert the input character embedding vector into a contextual feature vector, providing rich semantic information for the subsequent attention mechanism and decoder. This encoder submodule is implemented using the Conformer (Convolution-augmented Transformer) architecture. The Conformer is an advanced architecture that combines a convolutional neural network (CNN) with a self-attention mechanism. It is designed to capture both local and global features, resulting in excellent performance when processing long sequences of data, such as text or speech.

[0071] Among them, the convolution operation can be expressed as:

[0072]

[0073] Among them, x is the input feature map, W is the convolution kernel, b is the bias term, f is the activation function, and x' is the output feature map.

[0074] The self-attention mechanism is used to calculate the relationship between each position in the sequence and other positions, and generate a new feature representation by weighted summation. The formula is expressed as:

[0075]

[0076] The given dimension is The input sequence ,in is the sequence length, is the feature dimension, and the input sequence is mapped to the query sequence through three different linear transformations , key matrix K and value matrix V:

[0077]

[0078] in, , , is the weight matrix. This is the dimension of the key, used for scaling to avoid large values.

[0079] The decoder submodule receives the context vector passed from the encoder and the currently generated speech features, and converts the context feature vector generated by the encoder into a mel-spectrogram by combining the self-attention layer, the encoder-decoder attention layer, the feedforward network, and the layer normalization.

[0080] The vocoder submodule uses HiFi-GAN, a high-fidelity vocoder based on a generative adversarial network. HiFi-GAN converts mel-spectrograms into high-quality audio waveforms. Compared to traditional vocoders, HiFi-GAN's main components include a generator and a discriminator. By leveraging a generative adversarial network, it produces more natural and realistic speech waveforms, significantly improving the overall quality of speech synthesis.

[0081] The post-processing enhancement module uses SSRN (Super-Resolution Speech Network) technology to improve audio clarity and quality. Through multi-layer convolution and residual learning techniques, the generated audio waveform is post-processed to remove noise, reduce distortion, and enhance audio detail and layering.

[0082] The face detection submodule uses the S3FD model to process input data, obtaining the coordinates and mask representation of the facial region. This information serves as the basis for subsequent processing. The S3FD model extracts features from the input data using a multi-layer convolutional neural network, identifying facial features layer by layer.

[0083] Through this layer-by-layer processing, the S3FD model is able to accurately locate the face region under various complex background and lighting conditions. The detected coordinate information includes the bounding box position of the face, while the mask representation provides a more detailed outline of the face region.

[0084] Based on the generated audio and facial area information, a variety of deep learning algorithms are integrated to process the voice signal and video frames to achieve synchronization between the generated voice and the target person's lip shape; this step is performed by the video generation submodule, which is a face reconstruction and fusion network based on deep learning.

[0085] The video generation submodule receives the synthesized speech output by the speech generation submodule, the video frame sequence input by the user, and the facial feature information output by the face detection submodule. It concatenates the speech feature vector and the facial feature vector, processes them through the audio and video codec, and generates a high-quality target speech video that is highly aligned with the input audio and video content. This video is then output to the output processing submodule.

[0086] The speech generation submodule and its various submodules work together to transform audio data into high-quality speech output. Each submodule performs a specific task and utilizes deep learning algorithms and techniques to ensure the final speech output quality. The module also features face detection and video generation capabilities, enabling synchronization of generated speech with the target person's lip movements.

[0087] like Figure 6 As shown, the video generation submodule includes the following submodules:

[0088] The two encoder sub-modules perform feature extraction on the input audio and video frames, extracting high-level audio and video features layer by layer; through multi-layer convolution, residual blocks and multi-head attention mechanism, deep features in audio and video sequences are extracted, and depth-separable convolution is used to reduce the amount of computation.

[0089] Multi-Head Attention is a technique widely used in Transformer models to improve the model's ability to capture different feature subspaces. Its main idea is to compute different feature representations using multiple sets of attention heads in parallel and combine these representations to better capture complex relationships in sequential data.

[0090] The multi-head attention mechanism computes multiple self-attention mechanisms in parallel, concatenates their outputs and linearly transforms them. The formula is:

[0091]

[0092]

[0093] in, is the concatenated linear transformation weight matrix.

[0094] By introducing a multi-head attention mechanism in the audio and video encoder module, richer and more accurate feature representations can be obtained in the process of extracting and fusing audio and facial features, which helps to generate more realistic and natural lip-sync videos and improve the quality and visual consistency of the video.

[0095] Depthwise separable convolution is an efficient convolution operation that decomposes the standard convolution into two independent operations: depthwise convolution and pointwise convolution. This method reduces the amount of computation and parameters while retaining the ability to extract features.

[0096] The specific formula is as follows:

[0097]

[0098] Assume that the dimension of the input feature map is ,in is the height, is the width, is the number of input channels, and the dimension of the filter is Then, each output channel The value of Represents the input feature map The pixel value of each channel, Indicates the The convolution kernel parameters on the channels, is the size of the convolution kernel, Represents a point-wise convolution kernel.

[0099] Through this decomposition, depthwise separable convolution reduces the computational effort of standard convolution from Reduce to , greatly reducing the computational complexity and the number of parameters, and increasing the computing speed.

[0100] This approach aims to alleviate the vanishing gradient problem in deep neural networks, improve information flow efficiency, and promote feature reuse. During the audio and video feature extraction process, the output of each layer can be directly passed to all subsequent layers, thereby reusing the features of previous layers and improving feature expression capabilities.

[0101] The feature fusion submodule is implemented through feature concatenation and is used to fuse audio features with facial features. This submodule concatenates the high-level features extracted by the audio encoder and the facial encoder to form a joint feature representation, which provides rich information for the subsequent decoding process. The concatenation operation can be expressed as:

[0102]

[0103] is the concatenated feature map, and its dimension is .

[0104] The joint feature representation is input into the decoder submodule, and PixelShuffle is used to replace the traditional deconvolution layer. Upsampling is achieved by channel rearrangement, which has higher computational efficiency and smoother results, avoiding the checkerboard effect that may occur during the deconvolution process. The specific operation can be expressed as follows:

[0105]

[0106] in, and are the indices of the width and height of the output image, respectively. is the channel index. More specifically, PixelShuffle converts each The blocks are rearranged into the spatial locations of the output image.

[0107] The output module converts the decoded feature map into a pixel-level image representation through a convolutional layer. The output pixel values ​​are normalized to the range [0, 1] using the Sigmoid activation function to generate the final output image.

[0108] The output image is input to the discriminant submodule, which evaluates the authenticity and similarity of the generated result by comparing the generated image with the real image of the target person. The discriminant submodule contains multiple encoder blocks, each of which consists of a series of convolutional layers, batch normalization layers, and activation functions. These layers extract image features layer by layer and gradually compress them to a lower-dimensional feature representation. Then, through a fully connected layer and a Sigmoid activation function, the image features are mapped to a binary classification result, which represents the probability of the image's authenticity. Finally, the discriminant submodule judges the quality of the generated image based on the output probability, and guides the generator optimization through the binary cross entropy loss function to improve the realism and similarity of the generated image. For a single sample, the formula of the binary cross entropy loss function can be expressed as:

[0109]

[0110] The discriminated output image is input into the video enhancement submodule, which uses the deep learning model ESRGAN to enlarge the low-resolution image to a high-resolution image, thereby improving the resolution and details of the generated video, ensuring that the final output video is clearer and more realistic, meeting the requirements of high-quality video generation.

[0111] like Figure 7 As shown in the figure, the output processing submodule synchronizes and integrates the audio and video data generated by each processing step. This module uses the convolutional neural network (DnCNN) and inter-frame interpolation technology (DAIN) to denoise the image, reducing noise, artifacts, and jitter during the generation process. Tools such as ffmpeg are used to convert the video format to ensure that the final output video is compatible with a variety of playback devices and platforms. Through this series of processing, the output processing submodule can generate high-quality, highly compatible, and visually excellent final video files, ensuring good performance in various playback environments.

[0112] like Figure 8 As shown in the figure, after the algorithm microservice processes the video data, it returns the results to the backend module. The backend module then sends the results to the frontend module. After receiving the results, the frontend module plays them for the user on the robot's touch screen and provides various options, such as saving, sharing, and printing. When the user wears the robot's external EEG headset, the backend signal monitoring module analyzes the user's EEG signal data while the frontend plays the content and determines the most effective wake-inducing method.

[0113] The electroencephalogram (EEG) is a comprehensive reflection of the electrophysiological activity of brain nerve cells on the cerebral cortex or scalp surface. EEG signals contain a large amount of physiological and disease information. In clinical medicine, EEG signal processing can provide a diagnostic basis for the effectiveness of cognitive rehabilitation. Figure 9As shown, the back-end EEG signal monitoring module of the present invention monitors and analyzes the user's EEG signal data in real time while the front-end plays content for the user, identifying the content types that are most likely to stimulate brain activity. The system collects the user's EEG signal data in real time via an EEG headset and transmits it to the back-end. Because raw EEG signals are often subject to various interferences (such as electromagnetic interference, muscle activity, and eye movement artifacts), preprocessing is required. This involves using blind source separation (ICA) to remove artifacts and noise, and applying bandpass filters to decompose the EEG signals into different frequency bands (such as delta waves from 0.5 to 4 Hz, theta waves from 4 to 8 Hz, alpha waves from 8 to 13 Hz, and beta waves from 13 to 30 Hz) to extract specific frequency bands associated with different EEG activities.

[0114] After preprocessing, the EEG signal undergoes spectral analysis to understand its frequency characteristics. The Fourier transform is a key algorithm in this process, converting the time-domain EEG signal into a frequency-domain signal, facilitating analysis of the energy distribution within each frequency band. The Fast Fourier Transform (FFT) efficiently decomposes continuous time-domain signals into their constituent frequency components. Its core concept is to leverage the symmetry and periodicity of the Discrete Fourier Transform (DFT) to accelerate computation. Given a discrete signal sequence x[n] of length N, where n=0,1,2,...,N−1, its DFT is defined as:

[0115]

[0116] in, is the signal in the frequency domain Frequency components, is the signal in the time domain samples, is the rotation factor, also known as the DFT kernel. The FFT reduces computational complexity by recursively decomposing a signal sequence (decomposing a DFT of length N into two DFTs of length N / 2, processing the even and odd terms of the signal respectively). In EEG signal processing, the FFT helps identify the signal intensity at different frequencies, thereby extracting spectral features associated with brain activity. These features include the energy and power spectral density (PSD) of each frequency band (such as alpha waves and beta waves), as well as the relative intensity of each frequency band. The system can quantify the signal intensity in different frequency bands to assess brain activity when viewing different content. High PSD values ​​generally indicate significant brain activity in that frequency band. The system also meticulously tags the content being played. These tags include characteristics of the target person, the type of generated content, and variations in voice and intonation. By comparing tags for different content with the user's EEG signal response, the system can identify specific content elements (such as certain characters or specific voice tones) that stimulate the user's EEG signal activity.

[0117] Ultimately, based on multi-dimensional EEG signal characteristics and the persistence of brain responses when viewing specific content, the system dynamically adjusts the type and presentation of content to achieve the best wake-inducing effect. For example, if it recognizes that the user reacts most strongly to a specific tone, the system will prioritize content with similar tonal characteristics.

[0118] The present invention can provide more convenient and effective rehabilitation assistance services. It uses the generated audio and video to simulate the voices and life scenes of the patient's relatives, provide family-centered emotional stimulation to the patient, reduce the repetitive interactive tasks that family members and medical staff must perform, reduce the burden of accompanying care, and improve the efficiency of consciousness disorder treatment, providing patients with more comprehensive rehabilitation services.

[0119] The above embodiments are only preferred embodiments of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A wake-up assistance robot based on a generative deep neural network, characterized in that: Including front-end module, back-end module and algorithm microservice module; The front-end module is used to receive audio, image, and video data of the target person uploaded by the user, and provide the user with functional options, including content modification and pre-processing; The backend module includes a storage module for receiving data to be processed, a function selection module that can call the corresponding algorithm microservice module for processing, and an EEG signal monitoring module; The algorithm microservice module includes a speech generation submodule, a face detection submodule, a video generation submodule, and an output processing submodule; The speech generation submodule is used to perform speech synthesis on the target text based on the uploaded video, extract the spectrum, tone, and speech speed of the target person's speech signal, and convert the text into speech output; The speech generation submodule includes: A preprocessing submodule that removes background noise from the input audio based on the UVR5 model to improve the purity of the audio; An audio recognition submodule, based on the FunASR model, converts the input speech into corresponding text, so that the subsequent text processing module can operate based on accurate text information; A character embedding submodule converts discrete character data into high-dimensional vector representations, enabling the model to process text information and capture the semantic relationships between characters, making the model more natural and coherent when generating speech; An encoder submodule, implemented as a Conformer, which extracts contextual information from character embeddings and combines convolution and self-attention mechanisms to generate a series of feature vectors that can better capture long-term and short-term dependencies; A decoder submodule that recursively processes the context vector and previous output to construct a complete mel-spectrogram, which provides input to the subsequent vocoder module; A vocoder submodule that uses a high-fidelity vocoder (HIFI-GAN) based on a generative adversarial network to generate more natural audio waveforms; A post-processing enhancement module that uses SSRN technology to improve audio clarity and quality; The face detection submodule is used to locate and segment the human area in the video frame and output the corresponding coordinates and masks; The video generation submodule uses the synthesized speech and the character area information output by the face detection submodule to accurately drive facial expressions and lip shapes, seamlessly integrate the synthesized face image into the input video stream, and reassemble the synthesized video frames into a video sequence; The output processing submodule is used to integrate and optimize the video frames processed by the speech synthesis, face detection, and video generation modules so that they can be played correctly by the front end.

2. The awakening assistance robot based on a generative deep neural network according to claim 1, characterized in that: The front-end module is also used to receive the processing results sent by the back-end module and display them on the interface, providing various functional options, including saving, sharing, and printing.

3. The awakening assistance robot based on a generative deep neural network according to claim 1, characterized in that: The backend module is also used to manage multiple algorithm microservice modules and select appropriate algorithm microservice modules according to different generation projects and scenarios.

4. The awakening assistance robot based on a generative deep neural network according to claim 1, characterized in that: The EEG monitoring module in the back-end module integrates an EEG signal analysis algorithm to analyze EEG activities in different frequency bands to determine the level of brain activity, thereby identifying the content type that can stimulate the most active EEG signals by comparing the EEG signal responses under different playback contents.

5. The awakening assistance robot based on a generative deep neural network according to claim 1, characterized in that: The face detection submodule uses the S3FD model to process the video frame to obtain the coordinates and mask representation of the face area.

6. The awakening assistance robot based on a generative deep neural network according to claim 1, characterized in that: The video generation submodule includes: Two encoder submodules perform feature extraction on the input through multiple convolutional layers and residual blocks, extracting high-level audio and video features layer by layer; A feature fusion submodule, implemented through feature concatenation, is used to fuse audio features with facial features. This submodule concatenates the high-level features extracted by the audio encoder and the facial encoder to form a joint feature representation, providing rich information for the subsequent decoding process. A decoder submodule, including multiple layers of deconvolution layers and residual blocks, is used to decode the fused features and generate feature maps; An output submodule, which converts the decoded feature map into a pixel-level image representation through a convolutional layer and uses a Sigmoid activation function to normalize the output pixel values ​​to the range [0, 1] to ensure that the pixel values ​​of the generated image are within a reasonable range; A discriminative submodule evaluates the quality and similarity of the generated results by comparing the generated images with the real images of the target person, ensuring that the generated images are highly visually consistent with the target person.

Images