Levelface muscle strength assessment method and system based on low-speed instant-eye video analysis

Abstract

The invention discloses a levator face muscle strength assessment method based on slow-speed instant-eye video analysis. The method comprises the following steps: inputting a slow-speed instant-eye video and physiological information of a subject; dividing the video clips by using a face mark detection method to obtain monocular instant video clips, and forming a data set by the physiological information, the slow instant video, the corresponding monocular instant video clips and the labels; constructing a dual-path recognition network which comprises a text feature extraction module, a low-speed channel module, a high-speed channel module, a fusion module and a prediction module, wherein the low-speed channel module and the high-speed channel module are parallel; training the dual-path identification network by using the data set to obtain a video classification model; and inputting the low-speed instant video to be identified and the physiological data into the video classification model to output a corresponding prediction result. The invention further provides a system for evaluating the muscle strength of the upper face muscles. According to the method provided by the invention, the muscle strength of the lifting facial muscle can be quickly evaluated, and a more perfect reference is provided for a subsequent treatment scheme.

Description

Technical Field

[0001] The present invention belongs to the field of medical image-assisted analysis, and in particular relates to a method and system for evaluating the strength of children's facial levator muscles based on slow blink video analysis. Background Art

[0002] Ptosis is a common and frequently occurring ophthalmic disease. It refers to the incomplete or loss of function of the levator palpebrae superioris muscle and Müller smooth muscle, which results in partial or complete drooping of the upper eyelid. In mild cases, the pupil is partially covered, and in severe cases, the pupil is completely covered. Severe congenital ptosis can lead to form deprivation, causing amblyopia and astigmatism. However, such complications can be alleviated or avoided through early diagnosis and surgical correction.

[0003] Blepharoptosis surgery can be briefly divided into three categories: percutaneous approach, transconjunctival approach and frontalis suspension. The choice of surgical approach depends largely on the levator function, which is the offset distance between the upper eyelid edge from the bottom to the upper gaze with the eyebrows fixed. However, due to the differences in the understanding and cooperation level of children, especially infants and young children, standard clinical preoperative evaluation becomes extremely difficult.

[0004] Patent document CN117523633A discloses an automatic recognition method for ptosis of the eye based on image deep learning, comprising: collecting a frontal face image and extracting an eye image; inputting the eye image into a human eye segmentation model to obtain a segmented eye image; performing Hough transform detection on the segmented eye image to obtain an iris contour and perform circle fitting to obtain an iris circle, and then determining the iris center; performing image correction; performing contour extraction detection on the segmented eye image to determine the upper eyelid margin contour, the lower eyelid margin contour, and the intersection point of the iris margin and the upper eyelid margin; using an iris diameter ruler to measure to obtain MRD1 and MRD2; determining the iris eyelid margin intersection angle, the inner intersection angle, and the outer intersection angle formed by the intersection point of the iris and the upper eyelid margin; and identifying the degree of ptosis of the eye.

[0005] Patent document CN116386114A discloses a facial feature recognition system based on visual detection, including an image storage module for storing preoperative facial images, surgical plan images and postoperative facial images of several historical patients; a neural network module for storing a neural network model; an image acquisition module for acquiring preoperative facial images of the patient to be operated; an image input module, which is respectively connected to the neural network module and the image acquisition module, and is used to input the preoperative facial image of the patient acquired by the image acquisition module into the neural network model; a data processing module, which is connected to the neural network module, and is used to analyze and process the mask data output by the neural network model; a feature calculation module, which is connected to the data processing module, and is used to calculate the double eyelid surgery plan for the patient to be operated on according to the analysis and processing results of the data processing module; a plan adjustment module, which is respectively connected to the image storage module and the feature calculation module, and is used to compare the double eyelid surgery plan for the patient to be operated on with the surgery plans of historical patients to adjust the double eyelid surgery plan for the patient to be operated on. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for evaluating the strength of the levator facial muscles based on slow blink video analysis. The method can quickly evaluate the strength of the levator facial muscles to provide a more complete reference for subsequent treatment plans.

[0007] In order to achieve the first object of the present invention, the following technical solution is provided: a method for evaluating the strength of the facial levator muscles in children based on slow blink video analysis, comprising the following steps: Inputting the slow blink video and physiological information of the subject, dividing the slow blink video into multiple video segments along the time frame, and labeling the multiple video segments based on eye classification and levator muscle strength level; The facial landmark detection method is used to divide the video clips into eye regions of interest to obtain the corresponding monocular blink video clips, and the physiological information, slow blink videos, corresponding monocular blink video clips and labels are combined into a dataset; Constructing a dual-path recognition network based on deep learning, wherein the dual-path recognition network includes a text feature extraction module, a parallel slow channel module and a fast channel module, and a fusion module and a prediction module for summarizing the slow channel module and the fast channel module; The text feature extraction module is used to extract text features of physiological information and expand the text features in the visual feature dimension to obtain a corresponding text matrix; The slow channel module is used to extract semantic information about each frame of the slow blink video to obtain a corresponding visual feature sequence, and perform an element-by-element multiplication operation on the visual feature sequence and the text matrix to obtain a spatiotemporal feature sequence; The fast channel module is used to extract information about rapid eye changes in each frame of the slow blink video to obtain a temporal feature sequence; The fusion module is used to horizontally splice the spatiotemporal feature sequence with the time domain feature sequence to obtain a fused feature sequence; The prediction module performs prediction according to the input fusion feature sequence to output a prediction result, wherein the prediction result includes eye classification and levator muscle strength level; The dual-path recognition network is trained using the dataset to obtain a video classification model for identifying the strength status of the facial levator muscles. The slow blink video and physiological data to be identified are input into the video classification model to output the corresponding prediction results.

[0008] The present invention utilizes a multimodal data fusion approach to enhance the data features for the final prediction, thereby obtaining a more accurate and complete reference.

[0009] Specifically, the slow blink video is a spontaneous blink video of a subject captured in a 240FPS slow-motion shooting mode.

[0010] Specifically, the levator strength level is divided into three levels, including good muscle strength, moderate muscle strength and poor muscle strength; When the levator function is greater than 10 mm, it is defined as good muscle strength; When the levator function was greater than or equal to 5 mm and less than or equal to 10 mm, the muscle strength was defined as good; When the levator function is less than or equal to 4 mm, it is defined as poor muscle strength.

[0011] Specifically, the eye region of interest includes the eye area including the inner canthus, outer canthus, upper and lower eyelids, palpebral fissures and eyebrows.

[0012] Specifically, the slow channel module uses a feature extraction method with a low sampling rate and a high number of channels to obtain spatiotemporal features with rich semantic information in the slow blink video.

[0013] Specifically, the fast channel module uses a feature extraction method with a high sampling rate and a low number of channels to obtain fast-changing time domain features in slow-blink videos.

[0014] Specifically, the dual-path recognition network is constructed using a 3D ResNet101 convolutional neural network model as a network framework.

[0015] Specifically, the physiological data include the subject's gender, age, palpebral fissure height, and upper eyelid corneal light reflection distance.

[0016] In order to achieve the second object of the present invention, the following technical scheme is provided: a facial levator muscle strength assessment system, which is implemented by the above-mentioned child facial levator muscle strength assessment method based on slow blink video analysis, and comprises a data acquisition unit, a video analysis unit and an inference assessment unit; The data acquisition unit is used to obtain slow blink videos of the subject; The video analysis unit analyzes the input slow blink video to output a prediction result; The reasoning evaluation unit performs similarity matching with the historical case data according to the prediction result, so as to output the historical case data with the highest similarity value as the reasoning evaluation result; The historical case data includes the patient's eye type, levator muscle strength level and corresponding medical record information.

[0017] Specifically, the reasoning evaluation results include: When the levator palpebrae superioris muscle strength is good, regular follow-up is recommended; When the levator palpebrae superioris muscle strength is good, it is recommended to comprehensively evaluate the timing of surgery, and the surgical procedure can be either percutaneous or transconjunctival. When the levator palpebrae superioris muscle strength is poor, surgery is recommended as soon as possible, and the surgical procedure can be frontalis muscle suspension.

[0018] Compared with the prior art, the present invention has the following beneficial effects: Based on the dual-path video recognition network, the visual features of the video data are extracted, and the text information is fused with the visual features in the slow channel and spliced ​​with the visual features in the fast channel, so as to provide a more comprehensive fusion feature sequence for subsequent prediction work; At the same time, compared with the clinical standardized levator palpebrae superioris function test, blink video is easy to obtain and transmit, and the intelligent analysis system has low technical requirements for the operator and the degree of cooperation of the children being filmed. It is expected to be applied in the field of telemedicine, optimize regional medical resource allocation, and greatly improve the clinical management efficiency of children with ptosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a method for evaluating the strength of the facial levator muscles based on slow blink video analysis provided in this embodiment; Figure 2 A schematic diagram of the framework of the dual-path identification network provided in this embodiment; Figure 3 A framework diagram of the facial muscle strength assessment system provided in this embodiment; Figure 4 The inference logic diagram of the inference evaluation unit provided in this embodiment. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] like Figure 1 As shown in FIG. 1 , a method for evaluating the strength of the levator facialis muscles based on slow blink video analysis provided in this embodiment includes the following steps: Inputting the slow blink video and physiological information of the subject, dividing the slow blink video into multiple video segments along the time frame, and labeling the multiple video segments based on eye classification and levator muscle strength level; The facial landmark detection method is used to divide the video clips into eye regions of interest to obtain the corresponding monocular blink video clips, and the physiological information, slow blink videos, corresponding monocular blink video clips and labels are combined into a dataset; Constructing a dual-path recognition network based on deep learning, wherein the dual-path recognition network includes a text feature extraction module, a parallel slow channel module and a fast channel module, and a fusion module and a prediction module for summarizing the slow channel module and the fast channel module; The text feature extraction module is used to extract text features of physiological information and expand the text features in the visual feature dimension to obtain a corresponding text matrix; The slow channel module is used to extract semantic information about each frame of the slow blink video to obtain a corresponding visual feature sequence, and perform an element-by-element multiplication operation on the visual feature sequence and the text matrix to obtain a spatiotemporal feature sequence; The fast channel module is used to extract information about rapid eye changes in each frame of the slow blink video to obtain a temporal feature sequence; The fusion module is used to horizontally splice the spatiotemporal feature sequence with the time domain feature sequence to obtain a fused feature sequence; The prediction module performs prediction according to the input fusion feature sequence to output a prediction result, wherein the prediction result includes eye classification and levator muscle strength level; The dual-path recognition network is trained using the dataset to obtain a video classification model for identifying the strength status of the facial levator muscles. The slow blink video and physiological data to be identified are input into the video classification model to output the corresponding prediction results.

[0022] Furthermore, in this embodiment, a slow blinking video of a child is used to illustrate the specific process: A total of 1606 blink video clips of children visiting the ophthalmology center of a hospital were collected. The subjects included children with ptosis and children without ptosis. The levator palpebrae superioris muscle strength included good muscle strength, moderate muscle strength, and poor muscle strength.

[0023] In this embodiment, a common smart phone is used to shoot the above-mentioned blinking video clip in 240FPS slow motion mode.

[0024] The physiological data of the subjects include the gender, age, palpebral fissure height and corneal reflection distance of the upper eyelid margin. The captured blink video clips need to be screened for quality, including: (1) poorly shot videos, which refer to videos with defects or images such as focus or lighting; (2) Poorly coordinated videos refer to videos in which the subject’s entire face is not in the frame or the subject is not looking straight ahead during the video acquisition process.

[0025] In this embodiment, after filtering out 38 blink videos of poor quality, the remaining 1568 blink videos that meet the conditions are used as a data set.

[0026] Based on the results of clinical standardized levator palpebrae superioris muscle function test, manual labeling was performed on the dataset, including annotation of the levator palpebrae superioris muscle strength level, which includes three levels: good muscle strength, fair muscle strength and poor muscle strength. Specifically, levator muscle function >10mm is defined as good muscle strength, 5-10mm is defined as fair muscle strength, and levator muscle function ≤4mm is defined as poor muscle strength.

[0027] Finally, in this dataset, 704 video clips were classified as good muscle strength, 458 video clips were classified as fair muscle strength, and the remaining 406 video clips were classified as poor muscle strength.

[0028] In addition, in order to highlight the target area around the eyes in the dataset, based on the characteristics of the position of the eye region of interest, the human facial landmark detection method is used to locate and select the area around the eyes including the inner canthus, outer canthus, upper and lower eyelids, palpebral fissures and eyebrows in the video frame.

[0029] After locating the eye region of interest, the video clips of monocular blinks are obtained by cropping and reorganizing frame by frame. In this embodiment, according to the Pareto principle, the preprocessed data set is randomly split in a ratio of 8:2, of which 1254 (80%) video clips are used as training sets, and the remaining 314 (20%) video clips are used as test sets.

[0030] A dual-path recognition network based on deep learning was constructed and trained using the preprocessed training set data. The slow and fast channels respectively extracted the spatial-temporal features and temporal features in the slow blink video to identify the abnormal strength state of the levator palpebrae superioris muscle and obtain a trained video classification model.

[0031] like Figure 2 As shown in the figure, the dual-path recognition network proposed in this embodiment uses the 3D ResNet-101 convolutional neural network model as the backbone of the dual paths, which includes a slow pathway, which uses a low sampling rate and a high number of channels, mainly extracts the spatiotemporal features in the video, and is used to analyze the semantically rich static content; a fast pathway, which uses a high sampling rate and a low number of channels, mainly extracts the temporal features in the video, and is used to analyze the rapidly changing dynamic content. Set the slow pathway parameters: time step τ = 4, sampling frame T = 3; Set the fast pathway parameters: time step τ / α = τ / 4, sampling frame αT = 24; The channel ratio β between the fast and slow pathways is 8; the learning rate = 0.01; the batch size is set to 16, and the model is trained for 256 iterations.

[0032] In addition, in order to better obtain richer features, text information is added to the visual features. The process is as follows: First, the text features are extracted from the input physiological information. Then, the text features are transformed through a linear layer to map the text features to a dimension that is more compatible with the visual features. Then, an activation function Sigmoid is applied to introduce nonlinearity to the text features. Finally, the text features are reshaped into a 7x7 spatial dimension to construct a text matrix with the same dimension as the visual features.

[0033] In the slow channel module, the visual feature sequence of the input video frame is extracted through standard operations such as convolution, batch normalization, ReLU, max pooling, and residual blocks. The dimension of the above text matrix is ​​expanded to make its shape (-1, 1, 1,7, 7) to align with the visual features in the first visual feature sequence for element-by-element multiplication. Then the softmax function is applied on the channel dimension (`dim=1`) to convert the values ​​on each 7x7 spatial grid of the text feature into a probability distribution. Finally, the visual features in the visual feature sequence are multiplied element-by-element with the text matrix, which enables the model to weight the different spatial positions of the visual features according to the text information to obtain the final spatiotemporal feature sequence.

[0034] The fast-path module directly extracts the temporal feature sequence from the input video frames.

[0035] In the fusion module, the final spatiotemporal feature sequence and the time domain feature sequence are horizontally spliced ​​to output a fused feature sequence.

[0036] In order to evaluate the classification performance of the video classification model, this embodiment calculates the precision, recall, accuracy, and F1 score of the grading assessment of the upper eyelid levator muscle strength of children in the test set. The results are shown in Table 1: Table 1 like Figure 3 As shown, this embodiment also provides a levator facial muscle strength assessment system, which is implemented by the levator facial muscle strength assessment method based on slow blink video analysis provided in the above embodiment, and includes a data acquisition unit, a video analysis unit and an inference assessment unit.

[0037] The data acquisition unit is used to obtain the slow blink video and physiological data of the subject.

[0038] A video analysis unit, which performs analysis based on the input slow blink video and physiological data to output a prediction result; The reasoning evaluation unit performs similarity matching with the historical case data according to the prediction result to output the historical case data with the highest similarity value as the reasoning evaluation result, wherein the historical case data includes the patient's eye type, levator muscle strength level and corresponding medical record information.

[0039] like Figure 4 As shown, the reasoning evaluation results in this embodiment include: When the levator palpebrae superioris muscle strength is good, regular follow-up is recommended.

[0040] When the levator palpebrae superioris muscle strength is good, it is recommended to comprehensively evaluate the timing of surgery, and the surgical procedure can be selected through a percutaneous approach or a transconjunctival approach.

[0041] When the levator palpebrae superioris muscle strength is poor, surgery is recommended as soon as possible, and the surgical procedure can be frontalis muscle suspension.

[0042] In addition, the terms "upper", "lower", "inner", "outer", "front", "rear" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps described in these embodiments do not limit the scope of the present invention.

[0043] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of implementation of the present invention. All equivalent changes or modifications made according to the structure, characteristics and principles described in the patent application scope of the present invention should be included in the patent application scope of the present invention.

[0044] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for evaluating the strength of the facial levator muscles based on slow blink video analysis, characterized in that: The following steps are involved: Inputting the slow blink video and physiological information of the subject, dividing the slow blink video into multiple video segments along the time frame, and labeling the multiple video segments based on eye classification and levator muscle strength level; The facial landmark detection method is used to divide the video clips into eye regions of interest to obtain the corresponding monocular blink video clips, and the physiological information, slow blink videos, corresponding monocular blink video clips and labels are combined into a dataset; Constructing a dual-path recognition network based on deep learning, wherein the dual-path recognition network includes a text feature extraction module, a parallel slow channel module and a fast channel module, and a fusion module and a prediction module for summarizing the slow channel module and the fast channel module; The text feature extraction module is used to extract text features of physiological information and expand the text features in the visual feature dimension to obtain a corresponding text matrix; The slow channel module is used to extract semantic information about each frame of the slow blink video to obtain a corresponding visual feature sequence, and perform an element-by-element multiplication operation on the visual feature sequence and the text matrix to obtain a spatiotemporal feature sequence; The fast channel module is used to extract information about rapid eye changes in each frame of the slow blink video to obtain a temporal feature sequence; The fusion module is used to horizontally splice the spatiotemporal feature sequence with the time domain feature sequence to obtain a fused feature sequence; The prediction module performs prediction according to the input fusion feature sequence to output a prediction result, wherein the prediction result includes eye classification and levator muscle strength level; The dual-path recognition network is trained using the dataset to obtain a video classification model for identifying the strength status of the facial levator muscles. The slow blink video and physiological data to be identified are input into the video classification model to output the corresponding prediction results.

2. The method for evaluating facial levator muscle strength based on slow blink video analysis according to claim 1, characterized in that: The slow blink video is a spontaneous blink video of the subject captured by a 240FPS slow-motion shooting method.

3. The method for evaluating facial levator muscle strength based on slow blink video analysis according to claim 1, characterized in that: The levator strength level is divided into three levels, including good strength, fair strength and poor strength; When the levator function is greater than 10 mm, it is defined as good muscle strength; When the levator function was greater than or equal to 5 mm and less than or equal to 10 mm, the muscle strength was defined as good; When the levator function is less than or equal to 4 mm, it is defined as poor muscle strength.

4. The method for evaluating the strength of the facial levator muscles based on slow blink video analysis according to claim 1, characterized in that: The eye region of interest includes the eye area including the inner canthus, outer canthus, upper and lower eyelids, palpebral fissures and eyebrows.

5. The method for evaluating the strength of the facial levator muscles based on slow blink video analysis according to claim 1, characterized in that: The slow channel module adopts a feature extraction method with low sampling rate and high number of channels.

6. The method for evaluating facial levator muscle strength based on slow blink video analysis according to claim 1, characterized in that: The fast channel module adopts a feature extraction method with high sampling rate and low channel number.

7. The method for evaluating facial levator muscle strength based on slow blink video analysis according to claim 1, characterized in that: The dual-path recognition network is constructed using a 3D ResNet101 convolutional neural network model as a network framework.

8. The method for evaluating facial levator muscle strength based on slow blink video analysis according to claim 1, characterized in that: The physiological data include the subject's gender, age, palpebral fissure height, and upper eyelid corneal light reflection distance.

9. A facial muscle strength assessment system, characterized in that: The method for evaluating the strength of the facial levator muscles based on slow blink video analysis as claimed in any one of claims 1 to 8 is implemented, and comprises a data acquisition unit, a video analysis unit and an inference evaluation unit; The data acquisition unit is used to obtain slow blink videos and physiological data of the subject; The video analysis unit performs analysis based on the input slow blink video and physiological data to output a prediction result; The reasoning evaluation unit performs similarity matching with the historical case data according to the prediction result, so as to output the historical case data with the highest similarity value as the reasoning evaluation result; The historical case data includes the patient's eye type, levator muscle strength level and corresponding medical record information.

Images