Revival-promoting auxiliary robot based on generative deep neural network

Abstract

The invention discloses a revival-promoting auxiliary robot based on a generative deep neural network, and relates to the field of cognitive therapy. The robot can achieve the function of virtual relative image accompanying and is suitable for the fields of cognitive therapy and the like. The robot utilizes artificial intelligence technologies such as natural language processing and deep learning to assist in improving disturbance of consciousness of a patient suffering from craniocerebral injury and promoting the patient to wake up. The method comprises the following steps: 1, collecting audios and videos or photos of relatives of a patient as condition input, and obtaining to-be-processed data; 2, performing preprocessing steps of denoising, target detection, face recognition and the like on to-be-processed data; 3, using a deep learning model to generate a wake-up promotion accompanying audio similar to the target tone; 4, based on the generated audio, matching and adjusting figure postures, lip shapes, expressions and the like in the original video frame by using an image processing algorithm; and 5, recombining the synthesized video frames into a video sequence, outputting the video sequence to a user and storing the video sequence. And more convenient and effective rehabilitation assistance is provided.

Description

Technical Field

[0001] The present invention relates to the field of cognitive therapy, and in particular to cognitive stimulation technology for patients with craniocerebral injury and impaired consciousness. The present invention adopts artificial intelligence technologies such as natural language processing and deep learning, and realizes an awakening-assisting robot based on a generative deep neural network through a front-end and back-end separated architecture. Background Art

[0002] Disorders of consciousness refer to states of loss of consciousness caused by various severe brain injuries, such as coma, vegetative state, and minimally conscious state. Due to severe damage to brain tissue, patients often experience disorders of consciousness or even fall into a deep coma, with loss of cognitive ability and inability to actively move. The clinical treatment for disorders of consciousness is usually sensory stimulation therapy, which is systematic stimulation of one or more of the senses of vision, hearing, smell, taste, touch, and movement. Among them, family-centered sensory stimulation is more accepted than traditional sensory stimulation, but due to work or other reasons, many family members find it difficult to accompany the patient for a long time.

[0003] Therefore, the awakening-assisting robot can assist medical staff in treatment by generating sensory stimulation videos targeting the patient's family members, through both auditory and visual stimulation. Summary of the invention

[0004] The purpose of the present invention is to provide a robot for promoting awakening assistance based on a generative deep neural network, which can realize the function of virtual relative image care and is suitable for cognitive therapy and other fields. The robot uses artificial intelligence technologies such as natural language processing and deep learning to assist in improving the consciousness disorders of patients with craniocerebral injury and promote their awakening. The robot uses artificial intelligence technologies such as natural language processing and deep learning to generate sounds, images and scenes familiar to patients with consciousness disorders, and performs sensory stimulation therapy and clinical care on patients. The robot system adopts a front-end and back-end separation architecture, using vue for the front end, spring boot for the back end, and flask for the algorithm end. Users upload videos through the front end, the back end calls the algorithm for processing, and the processing results are returned to the front end.

[0005] The present invention provides a wake-up assistance robot based on a generative deep neural network, comprising a front-end module, a back-end module and an algorithm microservice module;

[0006] The front-end module is used to receive audio, picture, video and other data of the target person uploaded by the user, and provide the user with functional options such as content modification and pre-processing;

[0007] The backend module includes a storage module for receiving data to be processed, a function selection module capable of calling the corresponding algorithm microservice module for processing, and an EEG signal monitoring module;

[0008] The algorithm microservice module includes a speech generation submodule, a face detection submodule, a video generation submodule, and an output processing submodule;

[0009] The speech generation submodule is used to perform speech synthesis on the target text based on the uploaded video, extract the spectrum, tone, and speech speed characteristics of the target person's speech signal, and convert the text into speech output;

[0010] The face detection submodule is used to locate and segment the human area in the video frame and output the corresponding coordinates and masks;

[0011] The video generation submodule uses the synthesized speech and the character area information output by the face detection submodule to accurately drive the facial expression and lip shape, seamlessly integrate the synthesized face image into the input video stream, and reassemble the synthesized video frames into a video sequence;

[0012] The output processing submodule is used to integrate and optimize the video frames processed by the speech synthesis, face detection, and video generation modules so that they can be correctly played by the front end;

[0013] Furthermore, the front-end module is also used to receive the processing results sent by the back-end module, display them on the interface, and provide various functional options, such as saving, sharing, printing, etc.

[0014] Furthermore, the backend module is also used to manage multiple algorithm microservice modules and select appropriate algorithm microservice modules according to different generation projects and scenarios.

[0015] Furthermore, the EEG monitoring module in the back-end module integrates an EEG signal analysis algorithm to analyze EEG activities in different frequency bands (alpha waves, beta waves) to determine the activity level of the brain, thereby identifying the type of content that can stimulate the most active EEG signals by comparing the EEG signal responses under different playback contents.

[0016] Furthermore, the speech generation submodule includes:

[0017] A preprocessing submodule that removes background noise from the input audio based on the UVR5 model to improve the purity of the audio.

[0018] An audio recognition submodule, based on the FunASR model, converts the input speech into corresponding text, so that the subsequent text processing module can operate based on accurate text information.

[0019] A character embedding submodule converts discrete character data into high-dimensional vector representation, enabling the model to process text information and capture the semantic relationship between characters, making the model more natural and coherent when generating speech.

[0020] An encoder submodule, implemented through Conformer, which extracts contextual information of character embeddings and combines convolution and self-attention mechanisms to generate a series of feature vectors that can better capture long-term and short-term dependencies.

[0021] A decoder submodule recursively processes the context vector and previous output to construct a complete mel-spectrogram, which provides input to the subsequent vocoder module.

[0022] A vocoder submodule that uses a high-fidelity vocoder HIFI-GAN based on a generative adversarial network to generate more natural audio waveforms.

[0023] A post-processing enhancement module that uses SSRN (Super Resolution Network) technology to improve the clarity and quality of audio.

[0024] Furthermore, the face detection submodule processes the video frame using the S3FD model to obtain the coordinates and mask representation of the face area.

[0025] Furthermore, the video generation submodule includes:

[0026] Two encoder submodules perform feature extraction on the input through multiple convolutional layers and residual blocks, extracting high-level audio and video features layer by layer;

[0027] A feature fusion submodule, implemented through feature concatenation, is used to fuse audio features with facial features. This submodule concatenates the high-level features extracted by the audio encoder and the facial encoder to form a joint feature representation, providing rich information for the subsequent decoding process;

[0028] A decoder submodule, including multiple layers of deconvolution layers and residual blocks, is used to decode the fused features and generate feature maps;

[0029] An output submodule, which converts the decoded feature map into a pixel-level image representation through a convolutional layer and uses a Sigmoid activation function to normalize the output pixel value to the range of [0, 1] to ensure that the pixel value of the generated image is within a reasonable range;

[0030] A discriminative submodule evaluates the quality and similarity of the generated results by comparing the generated images with the real images of the target person, ensuring that the generated images are highly visually consistent with the target person.

[0031] The present invention also provides a method for using a wake-up assist robot based on a generative deep neural network, comprising the following steps:

[0032] Step 1: Through the input of the camera of the awakening assistance robot or uploading to the file system of the awakening assistance robot through the cloud, collect the audio, video or photos of the patient's family as conditional input to obtain the data to be processed;

[0033] Step 2: Preprocess the data to be processed, including denoising, target detection, face recognition and other preprocessing steps to improve the effect of subsequent processing;

[0034] Step 3: Use natural language processing algorithms and deep learning models to extract acoustic features from the input audio data, and generate awakening and care audio with a similar timbre to the target based on the input speech stimulus text;

[0035] Step 4: Based on the generated audio, a variety of deep learning algorithms are integrated to process the voice signal and video frames to achieve synchronization between the generated voice and the target person’s lip shape;

[0036] Step 5: Integrate and optimize the video frames processed by the speech synthesis, face detection, and video generation modules, reassemble them into video sequences, output them to the user through the robot display and audio equipment, and save the records on the back end.

[0037] Step 6: Based on the analysis results of the EEG signals of the external EEG headset of the awakening assistance robot, dynamically adjust the playback content to enhance the patient's participation and training effect.

[0038] The beneficial effects of the present invention are:

[0039] The present invention provides a generative deep neural network-based awakening assistance robot, which uses artificial intelligence technologies such as natural language processing and deep learning to generate sounds, images and scenes familiar to patients with consciousness disorders, and provides sensory stimulation therapy for patients. It can effectively reduce the burden of accompanying care, improve the efficiency of consciousness disorder treatment, and provide patients with more comprehensive rehabilitation services.

[0040] The robot system adopts a front-end and back-end separation architecture, with Vue used in the front end, Spring Boot used in the back end, and Flask used in the algorithm end. Users upload audio and video data through the front end, and the back end calls the algorithm for processing, and then returns the processing results to the front end. The system of the present invention has the characteristics of high efficiency, flexibility, scalability, and ease of use.

[0041] The present invention uses the deep learning model S3FD model to process the video frame to obtain the coordinates and mask representation of the face area. The S3FD model has the ability of multi-scale feature detection, can accurately locate the face area in the video frame, and output the coordinates of the area and the corresponding mask representation. This process has the advantages of high accuracy, robustness, and low computational complexity of face detection, providing a reliable basis for subsequent processing steps.

[0042] The present invention processes the input speech based on the speech characteristics of the target person's speech signal, such as the spectrum, tone, and speech speed, using a natural language processing algorithm and a deep learning network model to obtain a generated speech based on the speech stimulus text and having the sound characteristics of the target person. Based on the generated speech, the present invention processes the video frames using algorithms such as deep separable convolution and attention mechanism to generate a perceptual stimulation video similar to the target person. The video generation method of the present invention has the advantages of high similarity, high controllability, high practicality, and the like.

[0043] The present invention integrates and optimizes the generated video frames, sends them to the front-end user, and provides a function of saving them in the back-end.

[0044] The present invention dynamically adjusts the playback content based on the analysis results of the EEG signal module, so that medical staff and family members can better perceive and stimulate patients with consciousness disorders and make targeted training, adjustments and treatments. The visualization display method of the present invention has the advantages of being intuitive, clear and easy to understand. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the overall architecture diagram of the present invention.

[0046] Figure 2 It is the overall flow chart of the system of the present invention.

[0047] Figure 3 It is a flow chart of the front-end module in the present invention.

[0048] Figure 4 It is a flow chart of the back-end module in the present invention.

[0049] Figure 5 It is a flow chart of the algorithm module in the present invention.

[0050] Figure 6 It is a schematic diagram of the audio generation method in the present invention.

[0051] Figure 7 It is a schematic diagram of the video generation method in the present invention.

[0052] Figure 8 It is a schematic diagram of the output processing method in the present invention.

[0053] Fig. 9 Schematic diagram of the EEG monitoring method of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following embodiments will further illustrate the present invention in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims.

[0055] Figure 1 It is the overall architecture diagram of the present invention. Figure 1 As shown, the present invention provides a wake-up assisting robot based on a generative deep neural network, comprising:

[0056] The front-end module is used to receive audio, picture, video and other data of the target person uploaded by the user, and provide the user with functional options such as content modification and preprocessing; the user collects the audio, video or photos of the patient's family as conditional input through the input of the camera of the awakening assistance robot or uploads them to the file system of the awakening assistance robot through the cloud, and obtains the data to be processed;

[0057] The backend module involves the processing and management of user information, including a storage module for receiving data to be processed, a function selection module that can call the corresponding algorithm microservice module for processing, and an EEG signal monitoring module; the EEG signal monitoring module is used to monitor and analyze the user's EEG signal data in real time;

[0058] The algorithm microservice module includes a speech generation submodule, a face detection submodule, a video generation submodule, and an output processing submodule;

[0059] The speech generation submodule is used to perform speech synthesis on the target text based on the uploaded video, extract the speech features such as the spectrum, tone, and speech speed of the target person's speech signal, and convert the text into speech output;

[0060] The face detection submodule is used to locate and segment the human area in the video frame and output the corresponding coordinates and masks;

[0061] The video generation submodule uses the synthesized speech and the character area information output by the face detection submodule to accurately drive the facial expression and lip shape, seamlessly integrate the synthesized face image into the input video stream, and reassemble the synthesized video frames into a video sequence;

[0062] The output processing submodule is used to integrate and optimize the video frames processed by the speech synthesis, face detection, and video generation modules so that they can be correctly played by the front end;

[0063] like Figure 2As shown, the user uploads the audio, video or photo of the target person from the cloud through the robot touch screen, selects the preset speech stimulus text and preprocessing method, clicks the start generation button, and the front-end module sends the uploaded data to the back-end module.

[0064] like Figure 3 As shown, after receiving the uploaded data, the backend module calls the corresponding algorithm microservice for processing. In this embodiment, the algorithm microservice called by the backend module includes a speech generation submodule, a face detection submodule, a video generation submodule and an output processing submodule.

[0065] like Figure 4 As shown in the figure, after receiving the uploaded data, the algorithm microservice first confirms the data type. If audio data is detected, the system will call the speech generation submodule to perform a series of processing on the speech signal in the audio data.

[0066] like Figure 5 As shown, the speech generation submodule achieves the goal of speech synthesis by converting audio data into high-quality speech output. This module consists of multiple submodules, each of which has a specific task and works together to ensure the final speech output quality. Specifically:

[0067] After receiving the audio data, the speech generation submodule first calls the preprocessing module. This module is based on the UVR5 model and performs voice separation and reverberation noise reduction operations according to the processing algorithm selected by the front-end user. Voice separation extracts the human voice in the audio from the background noise or other sound sources, while reverberation noise reduction reduces the echo and environmental noise in the audio through an algorithm, making the human voice clearer and ensuring the quality of the audio signal.

[0068] The system calls the speech recognition model to extract text information from the preprocessed audio. The FunASR model based on the Transformer architecture generates corresponding text transcription by analyzing the speech signal in the audio, and the text transcription lays the foundation for subsequent semantic processing.

[0069] The character embedding submodule uses an embedding layer to convert each character into a fixed-size vector, converting text data into a vector representation that can capture semantic relationships. Specifically, the embedding layer uses an embedding matrix, where each row corresponds to a character in the vocabulary, and the content of the row is the vector representation of the character. When processing the input text, each character is first converted to its index in the vocabulary, and then the embedding matrix is ​​searched through these indexes to obtain the corresponding vector representation. These vector representations retain the semantic information of the characters, allowing the model to better understand and process text data.

[0070] The encoder submodule is one of the key components in the speech generation submodule. Its main function is to convert the input character embedding vector into a context feature vector to provide rich semantic information for the subsequent attention mechanism and decoder. The encoder submodule is implemented through the Conformer (Convolution-augmented Transformer) architecture. Conformer is an advanced architecture that combines convolutional neural networks (CNNs) and self-attention mechanisms. It is designed to capture local and global features at the same time, so that it performs well when processing long sequence data (such as text or speech).

[0071] Among them, the convolution operation can be expressed as:

[0072]

[0073] Among them, x is the input feature map, W is the convolution kernel, b is the bias term, f is the activation function, and x' is the output feature map.

[0074] The self-attention mechanism is used to calculate the relationship between each position in the sequence and other positions, and generate a new feature representation by weighted summation. The formula is expressed as:

[0075]

[0076] The given dimension is The input sequence ,in is the sequence length, is the feature dimension, and the input sequence is mapped to the query sequence through three different linear transformations , key matrix K and value matrix V:

[0077]

[0078] in, , , is the weight matrix. is the dimension of the key, used for scaling to avoid values ​​that are too large.

[0079] The decoder submodule receives the context vector passed from the encoder and the currently generated speech features, and converts the context feature vector generated by the encoder into a Mel-spectrogram by combining the self-attention layer, encoder-decoder attention layer, feedforward network, and layer normalization.

[0080] The vocoder submodule uses HiFi-GAN, a high-fidelity vocoder based on a generative adversarial network. HiFi-GAN can convert Mel-spectrograms into high-quality audio waveforms. Compared with traditional vocoders, the main components of HiFi-GAN include generators and discriminators. Through the generative adversarial network, more natural and realistic speech waveforms are generated, greatly improving the overall quality of speech synthesis.

[0081] The post-processing enhancement module uses SSRN (Super-Resolution Speech Network) technology to improve the clarity and quality of audio. Through multi-layer convolution and residual learning technology, the generated audio waveform is post-processed to remove noise, reduce distortion, and enhance audio details and layering.

[0082] The face detection submodule uses the S3FD model to process the input data and obtain the coordinates and mask representation of the face area, which will serve as the basis for subsequent processing. The S3FD model extracts features from the input data through a multi-layer convolutional neural network and identifies the feature information of the face layer by layer.

[0083] Through this layer-by-layer processing, the S3FD model is able to accurately locate the face region under various complex background and lighting conditions. The detected coordinate information includes the bounding box position of the face, while the mask representation provides a more refined outline of the face region.

[0084] Based on the generated audio and facial area information, a variety of deep learning algorithms are integrated to process the voice signal and video frames to achieve synchronization between the generated voice and the target person's lip shape; this step is performed by the video generation submodule, which is a face reconstruction and fusion network based on deep learning.

[0085] The video generation submodule receives the synthesized speech output by the speech generation submodule, the video frame sequence input by the user, and the facial feature information output by the face detection submodule. It concatenates the speech feature vector and the facial feature vector, processes them through the audio and video codec, generates a high-quality target task speech video that is highly aligned with the input audio and video content, and outputs it to the output processing submodule.

[0086] The speech generation submodule and its various submodules work together to achieve the conversion from audio data to high-quality speech output. Each submodule has a specific task and uses deep learning algorithms and techniques to ensure the final speech output quality. At the same time, the module also has face detection and video generation functions, which can synchronize the generated speech with the target person's lip shape.

[0087] like Figure 6 As shown, the video generation submodule includes the following submodules:

[0088] The two encoder sub-modules perform feature extraction on the input audio and video frames, extracting high-level audio and video features layer by layer; extracting deep features in audio and video sequences through multi-layer convolution, residual blocks and multi-head attention mechanism, and using depth-separable convolution to reduce the amount of computation.

[0089] Multi-Head Attention is a technique widely used in Transformer models to improve the model's ability to capture different feature subspaces. The main idea is to calculate different feature representations through multiple sets of attention heads in parallel, and combine these representations to better capture the complex relationships in sequence data.

[0090] The multi-head attention mechanism calculates multiple self-attention mechanisms in parallel, concatenates and linearly transforms their outputs, and the formula is expressed as:

[0091]

[0092]

[0093] in, is the concatenated linear transformation weight matrix.

[0094] By introducing a multi-head attention mechanism in the audio and video encoder module, richer and more accurate feature representation can be obtained in the process of extracting and fusing audio and facial features, which helps to generate more realistic and natural lip-sync videos and improve the quality and visual consistency of the video.

[0095] Depthwise separable convolution is an efficient convolution operation that decomposes the standard convolution into two independent operations: depthwise convolution and pointwise convolution. This method reduces the amount of computation and parameters while retaining the ability to extract features.

[0096] The specific formula is as follows:

[0097]

[0098] Assume that the dimension of the input feature map is ,in is the height, is the width, is the number of input channels, and the dimension of the filter is Then, each output channel The value of is expressed as Represents the input feature map The pixel value of each channel, Indicates The convolution kernel parameters on the channels, is the size of the convolution kernel, Represents a point-wise convolution kernel.

[0099] Through this decomposition, depthwise separable convolution reduces the computational effort of standard convolution from Reduce to , greatly reducing the computational complexity and the amount of parameters, and increasing the computing speed.

[0100] This approach aims to alleviate the vanishing gradient problem in deep neural networks, improve the efficiency of information flow, and promote feature reuse. In the process of audio and video feature extraction, the output of each layer can be directly passed to all subsequent layers, thereby reusing the features of the previous layers and improving feature expression capabilities.

[0101] The feature fusion submodule is implemented through feature concatenation and is used to fuse audio features with facial features. This submodule concatenates the high-level features extracted by the audio encoder and the facial encoder to form a joint feature representation, which provides rich information for the subsequent decoding process. The concatenation operation can be expressed as:

[0102]

[0103] is the concatenated feature map, and its dimension is .

[0104] The joint feature representation is input into the decoder submodule, and PixelShuffle is used to replace the traditional deconvolution layer. Upsampling is achieved by rearranging channels, which has higher computational efficiency and smoother results, avoiding the chessboard effect that may occur during the deconvolution process. The specific operation can be expressed as:

[0105]

[0106] in, and are the indices of the width and height of the output image, respectively. is the channel index. More specifically, PixelShuffle converts each The blocks are rearranged into the spatial locations of the output image.

[0107] The output module converts the decoded feature map into a pixel-level image representation through a convolution layer. The output pixel value is normalized to the range of [0, 1] using the Sigmoid activation function to generate the final output image.

[0108] The output image is input to the discriminant submodule, and the authenticity and similarity of the generated result are evaluated by comparing the generated image with the real image of the target person. The discriminant submodule contains multiple encoder blocks, each of which consists of a series of convolutional layers, batch normalization layers, and activation functions. These layers extract image features layer by layer and gradually compress them to a lower-dimensional feature representation. Then, through a fully connected layer and a Sigmoid activation function, the image features are mapped to a binary classification result, representing the probability of the authenticity of the image. Finally, the discriminant submodule judges the quality of the generated image based on the output probability, and guides the generator optimization through the binary cross entropy loss function to improve the realism and similarity of the generated image. For a single sample, the formula of the binary cross entropy loss function can be expressed as:

[0109]

[0110] The discriminated output image is input into the video enhancement submodule, which uses the deep learning model ESRGAN to enlarge the low-resolution image to a high-resolution image, thereby improving the resolution and details of the generated video, ensuring that the final output video is clearer and more realistic, meeting the needs of high-quality video generation.

[0111] like Figure 7 As shown in the figure, the output processing submodule synchronizes and integrates the audio and video data generated by each processing step. This module uses the convolutional neural network DnCNN and the inter-frame interpolation technology DAIN to denoise the image and reduce the noise, artifacts and jitter in the generation process. The video format is converted using tools such as ffmpeg to ensure that the final output video is compatible with a variety of playback devices and platforms. Through this series of processing, the output processing submodule can generate a high-quality, compatible and visually excellent final video file to ensure its good performance in various playback environments.

[0112] like Figure 8 As shown in the figure, after the algorithm microservice processes the video data, it returns the processing results to the back-end module. The back-end module sends the processing results to the front-end module. After receiving the processing results, the front-end module plays them for the user on the robot touch screen and provides various functional options, such as saving, sharing, and printing. When the user wears the robot's external EEG headset, the back-end signal monitoring module obtains the most effective way to promote awakening based on the analysis results of the user's EEG signal data when the front-end plays the content.

[0113] The electroencephalogram is a general reflection of the electrophysiological activity of brain nerve cells on the cerebral cortex or scalp surface. EEG signals contain a large amount of physiological and disease information. In clinical medicine, EEG signal processing can provide a diagnostic basis for the effect of cognitive rehabilitation. Fig. 9As shown, the back-end EEG signal monitoring module of the present invention monitors and analyzes the user's EEG signal data in real time when the front-end plays content for the user to identify the type of content that can best stimulate brain activity. The system collects the user's EEG signal data in real time through the EEG headset and transmits it to the back-end. Since the original EEG signal is usually subject to various interferences (such as electromagnetic interference, muscle activity, and eye movement artifacts), it is necessary to pre-process it, that is, to use blind source separation technology (ICA) to remove artifacts and noise, and to apply a bandpass filter to decompose the EEG signal into different frequency bands (such as 0.5-4Hz delta wave, 4-8Hz theta wave, 8-13Hz alpha wave, 13-30Hz beta wave) to extract specific frequency bands related to different EEG activities.

[0114] After preprocessing, the EEG signal is subjected to spectrum analysis to understand the frequency characteristics of the signal. Fourier transform is the key algorithm in this process, which converts the EEG signal in the time domain into a frequency domain signal, making it easier to analyze the energy distribution of each frequency band. Fast Fourier transform (FFT) can efficiently decompose continuous time domain signals into their constituent frequency components. Its core idea is to use the symmetry and periodicity of discrete Fourier transform (DFT) to accelerate calculations. Given a discrete signal sequence x[n] of length N, n=0,1,2,...,N−1, its DFT is defined as:

[0115]

[0116] in, is the signal in the frequency domain Frequency components, is the signal in the time domain samples, is the rotation factor, also known as the DFT kernel. FFT reduces the amount of computation by recursively decomposing the signal sequence (decomposing a DFT of length N into two DFTs of length N / 2, processing the even and odd terms of the signal respectively). In EEG signal processing, FFT helps identify the intensity of the signal at different frequencies, thereby extracting spectral features related to brain activity. These features include the energy of each frequency band (such as alpha waves, beta waves, etc.), power spectral density (PSD), and the relative intensity between frequency bands. The system can quantify the signal intensity in different frequency bands to assess the activity of the brain when watching different content. High PSD values ​​usually indicate that the brain activity in this frequency band is more significant. The system finely marks the content played. These marks include the characteristics of the target person, the type of generated content, changes in voice intonation, etc. By comparing the marks of different content with the response of the user's EEG signal, the system can identify which specific content elements (such as a certain character or a specific voice intonation) can stimulate the maximum activity of the user's EEG signal.

[0117] Ultimately, based on the multi-dimensional EEG signal characteristics and the persistence of the brain's response when watching specific content, the type and presentation of the content played can be dynamically adjusted to achieve the best awakening effect. For example, if it is recognized that the user reacts most strongly to a specific tone, the system will give priority to playing content with similar tone characteristics.

[0118] The present invention can provide more convenient and effective rehabilitation assistance services. It uses the generated audio and video to simulate the voices and life scenes of the patient's relatives, provide family member-centered emotional stimulation to the patient, reduce the repetitive interactive tasks that family members and medical staff must perform, reduce the burden of accompanying care, and will improve the efficiency of consciousness disorder treatment and provide patients with more comprehensive rehabilitation services.

[0119] The above embodiments are only preferred embodiments of the present invention and cannot be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A wake-up assisting robot based on a generative deep neural network, characterized in that: Including front-end module, back-end module and algorithm microservice module; The front-end module is used to receive audio, picture, and video data of the target person uploaded by the user, and provide the user with functional options, including content modification and pre-processing; The backend module includes a storage module for receiving data to be processed, a function selection module capable of calling the corresponding algorithm microservice module for processing, and an EEG signal monitoring module; The algorithm microservice module includes a speech generation submodule, a face detection submodule, a video generation submodule, and an output processing submodule; The speech generation submodule is used to perform speech synthesis on the target text based on the uploaded video, extract the spectrum, tone, and speech speed characteristics of the target person's speech signal, and convert the text into speech output; The face detection submodule is used to locate and segment the human area in the video frame and output the corresponding coordinates and masks; The video generation submodule uses the synthesized speech and the character area information output by the face detection submodule to accurately drive the facial expression and lip shape, seamlessly integrate the synthesized face image into the input video stream, and reassemble the synthesized video frames into a video sequence; The output processing submodule is used to integrate and optimize the video frames processed by the speech synthesis, face detection, and video generation modules so that they can be correctly played by the front end.

2. According to claim 1, a wake-up assistance robot based on a generative deep neural network is characterized in that: The front-end module is also used to receive the processing results sent by the back-end module and display them on the interface, providing various functional options, including saving, sharing, and printing.

3. According to claim 1, a wake-up assistance robot based on a generative deep neural network is characterized in that: The backend module is also used to manage multiple algorithm microservice modules and select appropriate algorithm microservice modules according to different generation projects and scenarios.

4. According to claim 1, a generative deep neural network-based awakening assistance robot is characterized in that: The EEG monitoring module in the back-end module integrates an EEG signal analysis algorithm to analyze EEG activities in different frequency bands to determine the activity level of the brain, thereby identifying the content type that can stimulate the most active EEG signals by comparing the EEG signal responses under different playback contents.

5. The awakening assistance robot based on a generative deep neural network according to claim 1, characterized in that: The speech generation submodule comprises: A preprocessing submodule that removes background noise from the input audio based on the UVR5 model to improve the purity of the audio; An audio recognition submodule, based on the FunASR model, converts the input speech into corresponding text, so that the subsequent text processing module can operate based on accurate text information; A character embedding submodule that converts discrete character data into high-dimensional vector representations, enabling the model to process text information and capture the semantic relationship between characters, making the model more natural and coherent when generating speech; An encoder submodule, implemented as a Conformer, which extracts contextual information from character embeddings and combines convolution and self-attention mechanisms to generate a series of feature vectors that can better capture long-term and short-term dependencies; A decoder submodule that recursively processes the context vector and previous output to construct a complete mel-spectrogram, which provides input to the subsequent vocoder module; A vocoder submodule that uses a high-fidelity vocoder based on a generative adversarial network, HIFI-GAN, to generate more natural audio waveforms; A post-processing enhancement module that uses SSRN technology to improve the clarity and quality of audio.

6. According to claim 1, a generative deep neural network-based awakening assistance robot is characterized in that: The face detection submodule processes the video frame using the S3FD model to obtain the coordinates and mask representation of the face area.

7. The awakening assistance robot based on a generative deep neural network according to claim 1, characterized in that: The video generation submodule includes: Two encoder submodules perform feature extraction on the input through multiple convolutional layers and residual blocks, extracting high-level audio and video features layer by layer; A feature fusion submodule, implemented by feature concatenation, is used to fuse audio features with facial features. This submodule concatenates the high-level features extracted by the audio encoder and the facial encoder to form a joint feature representation, which provides rich information for the subsequent decoding process. A decoder submodule, including multiple layers of deconvolutional layers and residual blocks, is used to decode the fused features and generate feature maps; An output submodule, which converts the decoded feature map into a pixel-level image representation through a convolutional layer and uses a Sigmoid activation function to normalize the output pixel value to the range [0, 1] to ensure that the pixel value of the generated image is within a reasonable range; A discriminative submodule evaluates the quality and similarity of the generated results by comparing the generated images with the real images of the target person, ensuring that the generated images are highly visually consistent with the target person.

Images