Device for rehabilitation of facial paralysis

Abstract

The invention provides a device for rehabilitation of facial paralysis. The device comprises a camera for capturing a facial image of a patient; the image processing unit based on the deep learning model is used for analyzing and processing the facial image of the patient in real time: generating the facial image of the health state based on the calibrated facial key points; migrating the facial expression of the health state to the facial image of the patient; and the display screen is used for displaying the image processed by the image processing unit and executing electron mirror exercise training combined with mirror image therapy and motor imagery therapy. By means of the device, a facial paralysis patient can do normal expression training on the vision and the brain of the patient when the patient feels that the affected side is the same as the healthy side through exercise training facing the electron mirror, and rehabilitation of the facial paralysis patient is facilitated.

Description

Technical Field

[0001] The invention relates to the technical field of medical auxiliary equipment, and in particular to a device for rehabilitation of facial nerve paralysis. Background Art

[0002] The most powerful explanation for the improvement of motor learning by "motor imagery" therapy is still the psychoneuromuscular theory (PM theory). PM theory is based on the concept that the individual's central nervous system has stored a motor plan or "flow chart" for movement. It assumes that the movement "flow chart" involved in actual activities can be strengthened and improved in the process of "motor imagery" because imagination involves the same movement "flow chart" as the actual movement. Imagination helps to learn or complete movement by improving the coordination pattern in the process of motor skill formation and giving muscles additional opportunities to practice skills. Simply put, it is through the continuous imagination of the brain, activating neurons and forming electrical activities, etc., to transmit neural electrical signals to the periphery and help peripheral muscles complete activities. "Mirror therapy" is a mirror visual feedback therapy. It only uses the principle of plane mirror visual imaging to copy the picture of the healthy side activity to the affected side, allowing patients to imagine the affected side activity. It is a treatment method completed through visual illusion, visual feedback and virtual reality, combined with rehabilitation training programs. The dual-layer combination of "mirror therapy" and "motor imagery therapy" helps to complete the closed-loop rehabilitation training of "periphery-central" and "central-periphery". Currently, the motor imagery therapy method based on plane mirror "mirror therapy" in clinical practice is mainly used for limb function training, such as hand function training. However, it has not yet been possible for patients with facial nerve paralysis (facial palsy) to avoid visual deceptive feedback and complete "mirror therapy" and "motor imagery" therapy.

[0003] Facial nerve paralysis (facial palsy) is a disease that affects the movement of facial expression muscles. Traditional exercise rehabilitation methods have many limitations. For example, when training on the paralyzed side, the patient's vision can still observe the paralyzed face on the affected side, and it is impossible to complete mirror motor imagery therapy by "deceiving" the vision. Summary of the invention

[0004] In view of the gap in the prior art, the purpose of the present invention is to provide a device for rehabilitation of facial nerve paralysis, namely, an electronic mirror device based on artificial intelligence, so that patients with facial nerve paralysis can perform exercise training facing the electronic mirror, so that the patient's vision and brain feel that the affected side is also performing normal expression training like the healthy side, which is helpful for the rehabilitation of patients with facial paralysis. It aims to combine deep fake technology and an artificial intelligence electronic mirror system to improve the rehabilitation effect of patients with facial nerve paralysis.

[0005] The specific technical solutions are as follows: A device for rehabilitation of facial nerve paralysis, comprising: a camera for capturing an image of the patient's face; An image processing unit based on a deep learning model is used to analyze and process the patient's facial image in real time: based on the calibrated facial key points, a healthy facial image is generated; and the healthy facial expression is transferred to the patient's facial image; A display screen is used to display the image processed by the image processing unit and is used to perform electronic mirror movement training combined with mirror therapy and motor imagery therapy.

[0006] Furthermore, the image processing unit performs key point calibration on the patient's facial image captured by the camera through a facial key point detection network.

[0007] Furthermore, the image processing unit uses a generative adversarial network to synthesize a facial image in a healthy state based on a healthy facial image dataset and calibrated facial key points, and then uses a conditional generative adversarial network to transfer the facial expression in a healthy state to the patient's facial image; when training the conditional generative adversarial network, the synthesized healthy state image and the patient's facial image are used as input to train the model to achieve expression transfer.

[0008] Furthermore, the image processing unit uses a deep learning model to perform real-time processing of the patient's facial image, including face detection and key point positioning; this is achieved through a pre-trained neural network.

[0009] Furthermore, it also includes a user interaction unit for performing interaction of electronic mirror movement training.

[0010] Furthermore, the process of synthesizing healthy facial images using a generative adversarial network includes: Data collection: Obtain a dataset of healthy facial images at different angles and expressions, as well as a dataset of facial images with facial paralysis at different angles and expressions; then annotate facial key points, and design a loss function for training the model by annotating the hemifacial paralysis area; then perform preprocessing including data enhancement; the generator of the generative adversarial network is used to generate a healthy full-face image, the discriminator is used to distinguish the generated image from the real healthy face image, and the adversarial loss is used to optimize the generator and the discriminator, and the reconstruction loss is used to ensure the quality and authenticity of the generated image; the input of the generator is the hemifacial paralysis image of the patient, and a convolutional neural network is used to extract features and generate a healthy full-face image; the input is a real healthy face image and a generated image, and a convolutional neural network is used to determine whether the image is real; during the training process, the generator and the discriminator are trained alternately.

[0011] Furthermore, the image processing unit uses an autoencoder to generate a facial image in a healthy state, including an encoder for encoding a facial image of hemifacial paralysis into a latent feature vector, and a decoder for generating a healthy full-face image from the latent feature vector. The loss function uses reconstruction loss to ensure that the image generated by the decoder is as close to the healthy full-face image as possible; the input of the encoder is the patient's hemifacial paralysis image, and a convolutional neural network is used to extract features and encode the image into a latent feature vector; the input of the decoder is the latent feature vector, and a deconvolutional neural network is used to generate a healthy full-face image. During the training process, the encoder and decoder are trained simultaneously to optimize the reconstruction loss.

[0012] Furthermore, the trained deep learning model is deployed in the image processing unit, the camera is used to capture the patient's facial image in real time, the model is used to generate a healthy full-face image, and it is rendered in real time to the display screen for executing electronic mirror movement training combined with mirror therapy and motor imagery therapy.

[0013] Furthermore, the image processing unit is connected to a cloud server, and the deep learning model is deployed on the cloud server. This preferred design takes into account that the processing power and computing power of local computer devices may be limited, so a cloud-based setting can also be adopted to place the main calculation and image generation process of the model on the cloud. At this time, the main function of the image processing unit is to transmit data, including uploading the patient image captured by the camera to the cloud and receiving the simulated healthy facial image fed back by the cloud, and transmitting it to the display.

[0014] Furthermore, the deep learning model is optimized and adjusted, and the model parameters are updated through the patient images and feedback collected by the image processing unit during use.

[0015] Compared with the prior art, the present invention and its preferred solution combine deep fake technology and artificial intelligence technology for rehabilitation training of patients with facial nerve paralysis. It can capture the patient's facial image in real time and provide device assistance and virtual feedback for rehabilitation training to improve the rehabilitation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments: Figure 1 Schematic diagram of the processing process of the image processing unit using the deep learning model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail with the accompanying drawings as follows: It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0018] First, the device design provided in the embodiment of the present invention can be regarded as an artificial intelligence-assisted electronic mirror system, which at least includes: a camera for capturing images of the patient’s face; An image processing unit based on a deep learning model to analyze and process the patient’s facial images in real time; A display for showing processed images and virtual feedback.

[0019] In order to facilitate control and adjustment, in a typical implementation, it generally also includes: A user interface (unit) for patient interaction with the system.

[0020] At the hardware level, cameras and display screens are both commonly used technologies. In this embodiment, the key points generally only include the selection and installation location for achieving better acquisition or display effects, so no further details are given. The image processing unit and the user interaction unit are mainly implemented through a computer system, and the key points of their design are actually the specific design of the software module. Therefore, the following mainly introduces the deep learning model through several embodiments: In the first embodiment of the present invention, the following design is provided: Facial image acquisition: Use a high-resolution camera or depth camera to capture the patient's face in real time, ensuring that the captured image is clear and can accurately capture facial details.

[0021] Landmark feature point calibration: Using deep learning models, such as facial key point detection networks (such as Dlib, OpenFace, etc.), key point calibration is performed on the collected facial images. These key points usually include facial feature points such as eyes, nose, mouth, etc., which are used for subsequent facial expression analysis and processing.

[0022] Generative Adversarial Networks (GANs) synthesize healthy facial images: Based on the calibrated facial key points, a healthy facial image is synthesized using a generative adversarial network. The training of the GANs model should be based on a large number of healthy facial image datasets to ensure that the synthesized images are highly realistic.

[0023] Conditional Generative Adversarial Networks (cGANs) for Expression Transfer: Conditional generative adversarial networks are used to transfer healthy facial expressions to patient facial images. When training cGANs, synthetic healthy state images and patient facial images are used as input to train the model to achieve expression transfer.

[0024] Deep learning models for real-time image processing: Use deep learning models to process the patient's facial images in real time, including face detection, key point location, etc. These models can be implemented using pre-trained neural networks, such as the convolutional neural network-based face detector provided in the OpenCV library.

[0025] User interface design: Develop a user-friendly interactive interface so that patients can easily interact with the system. The interface should include functions such as rehabilitation training program selection, facial expression feedback, rehabilitation progress tracking, etc., and ensure that the interface operation is simple and intuitive.

[0026] For the most critical virtual healthy facial image generation of the present invention, the above embodiment adopts a combination of a generative adversarial network and a conditional generative adversarial network. The following is a further introduction to the design through a second embodiment. Figure 1 As shown: 1. Data Preparation Data Collection: Set up high-definition high-speed cameras and seating environments in treatment rooms Therapists guide healthy people to make various expressions at different angles, and record and collect a large amount of healthy facial video data in real time. Therapists guide people with facial paralysis to make various expressions at different angles, and collect a large amount of facial paralysis video data in real time. Data annotation: Manually or semi-automatically annotate facial key points, including eyes, nose, and mouth; In the process of face detection and tracking, the following are some key typical feature points: Eye: 1. Inner corner of eye (left and right) 2. Outer canthus (left and right) 3. Upper eyelid center (left and right) 4. Center of lower eyelid (left and right) 5. Pupil center (left, right) Eyebrow: 1. Eyebrows (left and right) 2. Brow peak (left and right) 3. Eyebrow tail (left, right) nose: 1. Tip of nose 2. Nose (left and right) 3. Root of the nose 4. Center of nose base mouth: 1. Center of upper lip 2. Center of lower lip 3. Left corner of mouth 4. Right corner of mouth 5. Upper lip peak (left and right) 6. Lower lip peak (left and right) cheek: 1. Zygomatic bone (left and right) 2. Mandible (left and right) jaw: 1. Pointy chin 2. Chin edge (left and right) Mark the hemiplegic facial area for loss function design when training the model.

[0027] Data preprocessing: Normalize image size (512 × 512 pixels).

[0028] Data augmentation such as rotation, flipping, cropping, and brightness adjustment.

[0029] 2. Deep Learning Model Selection Generative Adversarial Networks (GANs) Architecture: Generator: Generates healthy full face images.

[0030] Discriminator: Distinguish between generated images and real healthy face images.

[0031] Loss function: Use Adversarial Loss to optimize the generator and discriminator.

[0032] Add reconstruction loss to ensure the quality and authenticity of the generated images.

[0033] Implementation steps: Generator: The input is an image of a patient with hemifacial paralysis.

[0034] A convolutional neural network (CNN) is used to extract features and generate healthy full-face images.

[0035] Discriminator: The input is a real healthy face image and a generated image.

[0036] Use CNN to determine whether the image is real.

[0037] Training process: The generator and discriminator are trained alternately, the generator generates healthy images that are as realistic as possible, and the discriminator distinguishes between real and generated images.

[0038] Use the Adam optimizer and adjust hyperparameters such as learning rate and batch size.

[0039] Code example (using TensorFlow / Keras): import tensorflow as tffrom tensorflow.keras import layers # Build generator def build_generator(): model = tf.keras.Sequential() model.add(layers.Dense(128 * 128 * 128, activation="relu", input_dim=100)) model.add(layers.Reshape((128, 128, 128))) model.add(layers.Conv2DTranspose(128, kernel_size=4, strides=2,padding="same")) model.add(layers.BatchNormalization()) model.add(layers.ReLU()) model.add(layers.Conv2D(3, kernel_size=3, padding="same",activation="tanh")) return model # Build the discriminator def build_discriminator(): model = tf.keras.Sequential() model.add(layers.Conv2D(64, kernel_size=3, strides=2, padding="same", input_shape=(256, 256, 3))) model.add(layers.LeakyReLU(alpha=0.2)) model.add(layers.Conv2D(128, kernel_size=3, strides=2, padding="same")) model.add(layers.LeakyReLU(alpha=0.2)) model.add(layers.Flatten()) model.add(layers.Dense(1, activation="sigmoid")) return model # Compile the model generator = build_generator() discriminator = build_discriminator() discriminator.compile(loss="binary_crossentropy", optimizer="adam") # Build GAN z = layers.Input(shape=(100,)) img = generator(z) discriminator.trainable = False real_or_fake = discriminator(img) gan = tf.keras.Model(z, real_or_fake) gan.compile(loss="binary_crossentropy", optimizer="adam") When it comes to the design of a generative adversarial network (GAN), it is necessary to define the generator, discriminator, and loss function. The following is the detailed design of these parts: Generator The goal of the generator is to convert the input hemifacial paralysis image into a healthy full face image. The generator can be implemented using a convolutional neural network (CNN). Assume that the input of the generator is z , represents a random noise vector, and the output is , represents the generated healthy full face image.

[0040] Discriminator The goal of the discriminator is to distinguish between the generated healthy full face images and the real healthy face images. Similarly, a convolutional neural network (CNN) can be used to implement the discriminator. Assume that the input of the discriminator is x , represents the image, and the output is , represents the authenticity score of the discriminator for the input image.

[0041] Adversarial Loss Adversarial loss is used to train the generator and discriminator to make the generated healthy full face images more realistic while making it more difficult for the discriminator to distinguish between real and generated images. The adversarial loss trains the generator by minimizing the probability that images generated by the generator are judged as fake, while minimizing the probability that the discriminator correctly classifies real and generated images.

[0042] The adversarial loss can be expressed as: Reconstruction Loss The reconstruction loss is used to ensure that the generated healthy full face image is as close as possible to the real healthy face image. The reconstruction loss usually adopts a pixel-level loss function, such as mean square error (MSE) or mean absolute error (MAE).

[0043] The reconstruction loss can be expressed as: By combining the adversarial loss and the reconstruction loss, the total loss function of the generator can be formed as follows: In this formula, and is a hyperparameter that controls the weights of adversarial loss and reconstruction loss in the total loss.

[0044] Further considering that the generation of adversarial networks is not the only option for achieving the purpose of the present invention, a third embodiment is provided, which implements the function through an autoencoder: Autoencoder Architecture: Encoder: Encodes the facial image of hemiplegia into a latent feature vector.

[0045] Decoder: Generates a healthy full face image from the latent feature vector.

[0046] Loss function: A reconstruction loss (such as mean squared error) is used to ensure that the image generated by the decoder is as close as possible to a healthy full face image.

[0047] Implementation steps: Encoder: The input is an image of a patient with hemifacial paralysis.

[0048] Features are extracted using CNN and encoded into a latent feature vector.

[0049] Decoder: The input is the latent feature vector.

[0050] Use a transposed CNN to generate healthy full face images.

[0051] Training process: The encoder and decoder are trained simultaneously to optimize the reconstruction loss.

[0052] Use the Adam optimizer and adjust hyperparameters such as learning rate and batch size.

[0053] Code example (using TensorFlow / Keras): Python from tensorflow.keras.models import Modelfrom tensorflow.keras.layersimport Input, Conv2D, MaxPooling2D, UpSampling2D # Build the encoder input_img = Input(shape=(256, 256, 3)) x = Conv2D(64, (3, 3), activation='relu', padding='same')(input_img) x = MaxPooling2D((2, 2), padding='same')(x) x = Conv2D(128, (3, 3), activation='relu', padding='same')(x) encoded = MaxPooling2D((2, 2), padding='same')(x) # Build the decoder x = Conv2D(128, (3, 3), activation='relu', padding='same')(encoded) x = UpSampling2D((2, 2))(x) x = Conv2D(64, (3, 3), activation='relu', padding='same')(x) x = UpSampling2D((2, 2))(x) decoded = Conv2D(3, (3, 3), activation='sigmoid', padding='same')(x) # Compile the autoencoder model autoencoder = Model(input_img, decoded) autoencoder.compile(optimizer='adam', loss='binary_crossentropy') In addition, in the fourth embodiment of the present invention, the following preferred design is provided based on the above model design: Real-time image generation and rendering Deploy the model: The trained model is deployed into a real-time system, and the camera is used to collect the patient's facial images in real time.

[0054] The model is used to generate healthy full-face images and rendered to the display in real time.

[0055] Optimization and debugging: Optimize the model's reasoning speed to ensure real-time performance.

[0056] Tune and improve the quality of generated images, reducing artifacts and unnatural effects.

[0057] User Interface and Feedback user interface: Designed with a simple and easy-to-use interface, it provides real-time images and generated healthy full-face images.

[0058] Display training progress and feedback information to help patients understand the training effect.

[0059] User feedback: Collect patient feedback to improve the system's usability and functionality.

[0060] Update models and software regularly to improve user experience and effects.

[0061] Thus, an electronic mirror system for patients with hemifacial paralysis was obtained, which used GAN or autoencoder technology to generate healthy full-face images to help patients with rehabilitation training.

[0062] Through the above implementations, the electronic mirror system of the present invention can capture the patient's facial image in real time, use deep learning technology to calibrate facial feature points and transfer expressions, and provide personalized rehabilitation training programs and real-time virtual feedback, thereby improving the rehabilitation effect of patients with facial nerve paralysis. The present invention can be applied to the field of facial nerve paralysis rehabilitation, and is also applicable to other rehabilitation and training scenarios involving the face.

[0063] It should be pointed out that the algorithm model involving deep learning and the software part of the device provided in the embodiment of the present invention can be stored in a coded form in a computer-readable storage medium, and implemented in the form of a computer program, and the basic parameter information required for the calculation can be input through computer hardware, and the calculation results can be output.

[0064] The algorithm model provided in this embodiment can be stored in a computer-readable storage medium in a coded form and implemented in the form of a computer program. The basic parameter information required for calculation is input through computer hardware, and the calculation results are output.

[0065] The present invention may be implemented as a computer program on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0066] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device that implements the specified functions.

[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing device, so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

[0069] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other various forms of devices for the rehabilitation of facial nerve paralysis under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. A device for rehabilitation of facial nerve paralysis, characterized in that: include: a camera for capturing an image of the patient's face; An image processing unit based on a deep learning model is used to analyze and process the patient's facial image in real time: based on the calibrated facial key points, a healthy facial image is generated; and the healthy facial expression is transferred to the patient's facial image; A display screen is used to display the image processed by the image processing unit and is used to perform electronic mirror movement training combined with mirror therapy and motor imagery therapy.

2. The device for rehabilitation of facial nerve paralysis according to claim 1, characterized in that: The image processing unit performs key point calibration on the patient's facial image captured by the camera through a facial key point detection network.

3. The device for rehabilitation of facial nerve paralysis according to claim 2, characterized in that: The image processing unit uses a generative adversarial network to synthesize a facial image in a healthy state based on a healthy facial image dataset and calibrated facial key points, and then uses a conditional generative adversarial network to transfer the facial expression in a healthy state to the facial image of the patient; when training the conditional generative adversarial network, the synthesized healthy state image and the patient's facial image are used as input to train the model to achieve expression transfer.

4. The device for rehabilitation of facial nerve paralysis according to claim 1, characterized in that: The image processing unit uses a deep learning model to process the patient's facial image in real time, including face detection and key point positioning; this is achieved through a pre-trained neural network.

5. The device for rehabilitation of facial nerve paralysis according to claim 1, characterized in that: Also included is a user interaction unit for performing interaction of electronic mirror motion training.

6. The device for rehabilitation of facial nerve paralysis according to claim 1, characterized in that: The process of synthesizing healthy facial images using a generative adversarial network includes: Data collection: Obtain a dataset of healthy facial images at different angles and expressions, as well as a dataset of facial images with facial paralysis at different angles and expressions; then annotate facial key points, and design a loss function for training the model by annotating the hemifacial paralysis area; then perform preprocessing including data enhancement; the generator of the generative adversarial network is used to generate a healthy full-face image, the discriminator is used to distinguish the generated image from the real healthy face image, and the adversarial loss is used to optimize the generator and the discriminator, and the reconstruction loss is used to ensure the quality and authenticity of the generated image; the input of the generator is the hemifacial paralysis image of the patient, and a convolutional neural network is used to extract features and generate a healthy full-face image; the input is a real healthy face image and a generated image, and a convolutional neural network is used to determine whether the image is real; during the training process, the generator and the discriminator are trained alternately.

7. The device for rehabilitation of facial nerve paralysis according to claim 1, characterized in that: The image processing unit uses an autoencoder to generate a facial image in a healthy state, including an encoder for encoding a facial image of hemifacial paralysis into a potential feature vector, and a decoder for generating a healthy full-face image from the potential feature vector. The loss function uses a reconstruction loss to ensure that the image generated by the decoder is as close as possible to the healthy full-face image. The input of the encoder is the patient's hemifacial paralysis image, and a convolutional neural network is used to extract features and encode the image into a potential feature vector. The input of the decoder is the potential feature vector, and a deconvolutional neural network is used to generate a healthy full-face image. During the training process, the encoder and the decoder are trained simultaneously to optimize the reconstruction loss.

8. The device for rehabilitation of facial nerve paralysis according to claim 6 or 7, characterized in that: The trained deep learning model is deployed in the image processing unit, and the camera is used to collect the patient's facial image in real time. The model is used to generate a healthy full-face image, which is then rendered to the display screen in real time for electronic mirror movement training combined with mirror therapy and motor imagery therapy.

9. The device for rehabilitation of facial nerve paralysis according to claim 1, characterized in that: The image processing unit is connected to a cloud server, and the deep learning model is deployed on the cloud server.

10. The device for rehabilitation of facial nerve paralysis according to claim 1, characterized in that: The deep learning model is optimized and adjusted, and the model parameters are updated through the patient images and feedback collected by the image processing unit during use.

Images