A video-based muscle contraction measurement and facial muscle synergy analysis system

Abstract

The present invention discloses a video-based muscle contraction measurement and facial muscle synergy analysis system, which includes a data acquisition module, a data processing module, a muscle contraction signal extraction module, a facial muscle synergy analysis module and a comprehensive analysis module. The video-based muscle contraction measurement method extracts muscle contraction strength and direction information through video. In the normal feature entry stage of facial muscle synergy analysis, the synergy feature vectors of the normal user group are stored in a local server, and the standard feature ranges in different modes are obtained as a basis for judging whether the muscle synergy function is normal or abnormal. In the detection stage, the user's synergy feature vector is extracted and compared with the synergy feature vector in the local server. The facial synergy analysis is used to determine whether the current muscle function is normal, and the position of the pathological muscle is located according to the direction of the synergy feature vector.

Description

Technical Field

[0001] The present invention relates to the field of instrument measurement, and in particular to a video-based muscle contraction measurement and facial muscle coordination analysis system. Background Art

[0002] Research on the diagnosis, prediction and rehabilitation guidance of diseases related to facial muscle dysfunction, such as facial paralysis and stroke, is of great significance at the medical level. Early warning of such diseases can not only avoid some irreversible losses, but also help with the diagnosis and treatment of the disease. Analyzing facial muscle synergy with the help of computer vision methods can identify pathological muscle locations that are invisible to the human eye.

[0003] Analysis of facial muscle synergy requires further analysis and judgment by detecting muscle contraction signals. Numerous methods exist for detecting muscle contraction signals in humans. Studying muscle contraction signals can identify numerous pathological processes and related abnormalities within the neuromuscular system. Electromyography (EMG) remains the gold standard technique for assessing muscle activity and contraction. EMG signals enable analysis of many important parameters, including amplitude and duration, motor unit engagement, and functional properties related to factors such as force production and fatigue. EMG is used for traditional diagnostic applications as well as in ergonomics, exercise physiology, rehabilitation, motion analysis, biofeedback, and myoelectric control of prostheses. EMG signals are acquired using electrodes, either needle electrodes or skin electrodes; the latter is commonly referred to as surface electromyography (sEMG). sEMG is not always easy to perform, as it requires patient compliance with the application procedure, including skin preparation (hair removal, skin decontamination, and abrasions), and careful attention to electrode placement and cable positioning.

[0004] Although there are many methods for measuring muscle contraction, there are always many problems such as cumbersome measurement process, limited usage conditions, and expensive measurement equipment.

[0005] A video-based muscle contraction measurement and facial muscle coordination analysis system uses a camera to extract human skin surface activity information and, with the help of computer vision tools, can effectively obtain human muscle contraction signals. It also has the advantages of simple actual measurement methods, low price, and easy promotion.

[0006] Furthermore, combining this method to obtain facial muscle contraction signals and then performing facial synergy analysis can provide diagnosis, prediction and rehabilitation guidance for diseases related to facial muscle dysfunction such as facial paralysis and stroke, which is of great significance in medicine. Summary of the Invention

[0007] The present invention aims to address the deficiencies of the existing technology and propose a video-based muscle contraction measurement and facial muscle coordination analysis system.

[0008] The object of the present invention is achieved through the following technical solutions: a video-based muscle contraction measurement and facial muscle synergy analysis system, the system comprising: a data acquisition module, a data processing module, a muscle contraction signal extraction module, a facial muscle synergy analysis module and a comprehensive analysis module;

[0009] The data acquisition module is used to obtain face videos;

[0010] The data processing module includes a video micro-motion amplification module and a feature vector extraction module; the video micro-motion amplification module is used to amplify the motion signal of the face video acquired by the data acquisition module to obtain micro-motions that are difficult for the human eye to observe in the original face video; the feature vector extraction module is used to extract the optical flow vector of the face surface in the face video, which includes the muscle movement trend;

[0011] The muscle contraction signal extraction module includes a local frame selection module, a muscle contraction signal acquisition module and a contraction abnormality judgment module;

[0012] The local frame selection module is used to frame the muscle area of ​​interest in the face video; the muscle contraction signal acquisition module is used to obtain a one-dimensional muscle contraction signal of the muscle area of ​​interest; the contraction abnormality judgment module is used to judge the current facial muscle contraction status based on the comparison result with the standard feature range; the standard feature range is derived from the one-dimensional muscle contraction signal obtained by the muscle contraction signal acquisition module, and is used to judge whether the current facial muscle contraction function is normal;

[0013] The facial muscle collaborative analysis module includes a facial feature point positioning module, an expression judgment module, a muscle frame selection module, a feature vector combination module and an overall collaborative analysis module;

[0014] The facial feature point positioning module is used to detect facial feature points in a facial video; the expression judgment module is used to judge the expression of the current facial video; the muscle selection module selects the muscles mainly involved in the expression according to the result of the expression judgment module, and selects the specified muscles according to the facial feature points provided by the facial feature point positioning module;

[0015] The feature vector combination module is used to combine the optical flow vectors in several areas selected by the muscle frame selection module to obtain the combined feature vectors of each area; the overall collaborative analysis module superimposes the combined feature vectors to obtain a collaborative feature vector, which is compared with the collaborative feature vector set of the corresponding expressions of healthy people to obtain the overall collaborative analysis result.

[0016] The comprehensive analysis module comprehensively judges whether the facial muscle coordination function is normal based on the joint results of the muscle contraction signal extraction module and the facial muscle coordination analysis module. If the judgment results of both the muscle contraction signal extraction module and the facial muscle coordination analysis module are abnormal, it is judged that the user's facial muscle coordination function is abnormal.

[0017] Furthermore, the video acquisition module includes a near-infrared LED lamp and a near-infrared camera; the near-infrared LED lamp serves as the light source for the near-infrared camera to shoot; the near-infrared camera is used to record user and face videos, further measure muscle contraction signals, analyze muscle synergy and make pathological judgments.

[0018] Furthermore, the video micro-motion amplification module is specifically:

[0019] Perform a first-order Taylor expansion of the image of f(x+δ(t)) at time t about position x to obtain:

[0020]

[0021] Where I(x,t) is the image intensity, δ(t) is the displacement function, and the bandpass signal is obtained after time domain filtering. Assuming that the motion signal δ(t) is within the passband of the temporal bandpass filter:

[0022]

[0023] Let δ k (t) is indexed by k to represent the different time spectrum components of δ(t), each δ k (t) will be attenuated by a factor γ by temporal filtering k , which leads to the general case that the motion signal δ(t) is not completely within the passband of the time filter:

[0024]

[0025] Amplify the bandpass signal B(x,t) by α times and add it to I(x,t) to get the processed signal

[0026]

[0027] Furthermore, the processed signal Further processing yields:

[0028]

[0029] Accordingly, the case where the motion signal δ(t) is not completely within the passband of the temporal filter is processed to obtain:

[0030]

[0031] This indicates that the spatial displacement δ(t) of the local image f(x) at time t has been magnified by (1+α) times, where α is the magnification factor.

[0032] Furthermore, the feature vector extraction module specifically performs optical flow extraction on the amplified video. The constraint equation of the optical flow method is:

[0033] I x u+I y v+I t =0

[0034] in is the partial derivative of the grayscale of the pixel in the image along the x, y, and t directions, and u and v are the components of the optical flow along the x and y directions respectively;

[0035] The energy function is obtained as follows:

[0036]

[0037] Among them, (I x u+I y v+I t ) 2 As the grayscale change factor, I x u+I y v=-I t is the grayscale invariance assumption, As a smoothing factor, it is used to limit the speed of change of the u and v components. By solving the Euler-Lagrange equation, let L be the sum of the grayscale change factor and the smoothing factor, and find the minimum value of the two factors. By taking the derivative, we can get:

[0038] I x (I x u+I y v+I t )-α 2 Δu=0

[0039] I y (I x u+I y v+I t )-α 2 Δv=0

[0040] Where Δ is the Laplace operator,

[0041] use After approximate substitution, the above formula is approximately replaced by:

[0042]

[0043]

[0044] The system of equations can be solved by iterative formula:

[0045]

[0046]

[0047] Thus, the optical flow vector u in the k+1 frame is obtained k+1 ,v k+1 Thus, the muscle activity feature vector is obtained.

[0048] Furthermore, the local frame selection module creates a rectangle in the specified area (x, y, w, h), where x, y, w, and h are the center coordinates, length, and width of the rectangle, respectively; the area outside the rectangle is dyed black to create a mask to shield areas of no interest, retaining only the feature vectors inside the rectangle, thereby eliminating light interference outside the specified area.

[0049] Furthermore, the muscle contraction signal acquisition module specifically calculates the average value of the matrix elements in the framed area, thereby converting the two-dimensional feature vector into a one-dimensional time series, denoted by t i The signal strength corresponding to the moment is P i , i∈(1,n), n is the maximum time point corresponding to the current signal, t i The video human muscle contraction signal V corresponding to the moment i Use the following formula to calculate:

[0050]

[0051] Furthermore, the facial feature point positioning module specifically performs real-time 3D coordinate detection of 468 facial feature points and generates a 3D model, or uses a dynamic method to identify facial features.

[0052] Furthermore, the muscle selection module specifically comprises: according to the expression judgment output by the expression judgment module, combining prior knowledge to obtain the muscles mainly involved in the judgment, and selecting the muscle areas A1, A2, ... A involved in the judgment according to the 468 feature points of the face. n , only retain the feature vectors in the selected area; then select the selected areas A1, A2, ..., A n All the characteristic vectors in the region are superimposed to obtain the combined characteristic vectors M1, M2, ..., M of each region. n ; The obtained n combined feature vectors M1, M2, ..., M nThe synergistic feature vectors are superimposed and synthesized, and compared with the synergistic feature vector set of healthy people. If they are within the synergistic feature vector set of healthy people, the analysis result is normal. If they are not within the synergistic feature vector set of healthy people, the analysis result is abnormal. Pathological research and judgment are continued, and compared with the synergistic feature vector (standard vector) of a healthy person for analysis to locate the specific area of ​​functional attenuation and obtain the corresponding pathological muscle mass.

[0053] Furthermore, the collaborative feature vector refers to a vector indicator that reflects the overall coordination of the face, which is used to locate areas where abnormal muscle function occurs in the face, and together with the standard feature range, serves as a basis for judging whether the facial muscle coordination function is normal or not.

[0054] The beneficial effects of the present invention are:

[0055] (1) The present invention uses video amplification combined with optical measurement to quantify muscle activity information in the form of a vector field for the first time. The obtained muscle activity vector field is the basis for muscle synergy analysis and pathological muscle detection.

[0056] (2) The present invention combines computer vision tools for the first time to measure muscle contraction signals through video, realizing non-contact muscle contraction measurement in a completely new way, while solving the shortcomings of traditional measurement, such as high cost, multiple measurement tools, and complicated measurement steps.

[0057] (3) After the present invention completes the extraction of facial feature vectors, the facial muscle synergy analysis results are obtained through muscle selection and frame selection and feature vector integration, combined with the overall synergy analysis of facial muscles, so as to perform pathological analysis and judgment on facial muscles and locate specific functionally impaired muscles. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is the overall structural block diagram of a video-based muscle contraction measurement and facial muscle coordination analysis system.

[0059] Figure 2 This is a detection flow chart of a video-based muscle contraction measurement and facial muscle coordination analysis system.

[0060] Figure 3 This is a specific structural block diagram of a video-based muscle contraction measurement and facial muscle coordination analysis system.

[0061] Figure 4 Flowchart of the algorithm for muscle contraction signal acquisition.

[0062] Figure 5 Schematic diagram of muscle selection corresponding to smile in Mode 1 of a video-based muscle contraction measurement and facial muscle synergy analysis system.

[0063] Figure 6Schematic diagram of muscle selection corresponding to sadness in mode 2 of a video-based muscle contraction measurement and facial muscle coordination analysis system.

[0064] Figure 7 Schematic diagram of muscle selection corresponding to fear in mode 3 of a video-based muscle contraction measurement and facial muscle coordination analysis system.

[0065] Figure 8 Schematic diagram of a video-based muscle contraction measurement and facial muscle synergy analysis system for facial muscle synergy analysis.

[0066] Figure 9 Schematic diagram of a video-based muscle contraction measurement and facial muscle synergy analysis system for measuring facial muscle contraction signals. DETAILED DESCRIPTION

[0067] In order to illustrate in more detail a video-based muscle contraction measurement and facial muscle synergy analysis system of the present invention.

[0068] Example 1

[0069] like Figure 1 and Figure 3 As shown, this embodiment provides a video-based muscle contraction measurement and facial muscle synergy analysis system, which includes: a data acquisition module, a data processing module, a muscle contraction signal extraction module, a facial muscle synergy analysis module and a comprehensive analysis module; the data acquisition module is used to obtain facial videos; in order to obtain a standard feature range, a near-infrared camera is used to collect facial videos of n user groups (n=1, 2, ...20) with normal muscle synergy functions; the data acquisition module includes a near-infrared LED lamp, a near-infrared camera and a standard USB interface; the near-infrared LED lamp is a light source; the near-infrared camera is used to record the user's facial image, further obtain the muscle contraction signal motion feature vector, analyze muscle synergy and perform pathological judgment; the standard USB interface is used to transmit facial videos.

[0070] The data processing module includes a video micro-motion amplification module and a feature vector extraction module; the video micro-motion amplification module is used to amplify the motion signal of the facial video acquired by the data acquisition module, amplifying the two-dimensional local translational motion in the video, thereby obtaining muscle contraction motion that can be captured by the computer, and obtaining micro-motions that are difficult for the human eye to observe in the original facial video; the feature vector extraction module is used to extract the optical flow vector of the facial surface in the facial video, which includes the muscle movement trend, that is, to obtain the two-dimensional movement information of the muscle as the feature vector of the muscle activity;

[0071] The video micro-motion amplification module is specifically:

[0072] Perform a first-order Taylor expansion of the image of f(x+δ(t)) at time t about position x to obtain:

[0073]

[0074] Where I(x,t) is the image intensity, δ(t) is the displacement function, and the bandpass signal is obtained after time domain filtering. Assuming that the motion signal δ(t) is within the passband of the temporal bandpass filter:

[0075]

[0076] Let δ k (t) is indexed by k to represent the different time spectrum components of δ(t), each δ k (t) will be attenuated by a factor γ by temporal filtering k , which leads to the general case that the motion signal δ(t) is not completely within the passband of the time filter:

[0077]

[0078] Amplify the bandpass signal B(x,t) by α times and add it to I(x,t) to get the processed signal

[0079]

[0080] Processed signal Further processing yields:

[0081]

[0082] Accordingly, the case where the motion signal δ(t) is not completely within the passband of the temporal filter is processed to obtain:

[0083]

[0084] This indicates that the spatial displacement δ(t) of the local image f(x) at time t has been magnified by (1+α) times, where α is the magnification factor.

[0085] The feature vector extraction module specifically performs optical flow extraction on the amplified video. The constraint equation of the optical flow method is:

[0086] I x u+I y v+I t =0

[0087] in is the partial derivative of the grayscale of the pixel in the image along the x, y, and t directions, and u and v are the components of the optical flow along the x and y directions respectively;

[0088] The energy function is obtained as follows:

[0089]

[0090] Among them, (I X u+I y v+I t ) 2 As the grayscale change factor, I x u+I y v=-I t is the grayscale invariance assumption, As a smoothing factor, it is used to limit the speed of change of the u and v components. By solving the Euler-Lagrange equation, let L be the sum of the grayscale change factor and the smoothing factor, and find the minimum value of the two factors. By taking the derivative, we can get:

[0091] I x (I x u+I y v+I t )-α 2 Δu=0

[0092] I y (I x u+I y v+I t )-α 2 Δv=0

[0093] Where Δ is the Laplace operator,

[0094] use After approximate substitution, the above formula is approximately replaced by:

[0095]

[0096]

[0097] The system of equations can be solved by iterative formula:

[0098]

[0099]

[0100] Thus, the optical flow vector u in the k+1 frame is obtained k+1 ,v k+1 Thus, the muscle activity feature vector is obtained.

[0101] The muscle contraction signal extraction module includes a local frame selection module, a muscle contraction signal acquisition module and a contraction abnormality judgment module;

[0102] The local frame selection module is used to frame the muscle area of ​​interest in the face video, create a rectangle in the specified area (x, y, w, h), where x, y, w, and h are the center coordinates of the rectangle and the length and width of the rectangle respectively; the outer area of ​​the rectangle is dyed black to create a mask to shield the uninterested area, retaining only the feature vector inside the rectangle, thereby eliminating light interference outside the specified area; the muscle contraction signal acquisition module is used to obtain a one-dimensional muscle contraction signal of the muscle area of ​​interest. The one-dimensional muscle contraction signal is essentially a one-dimensional time series with the horizontal axis being time t and the vertical axis being the muscle contraction intensity q, and the upper and lower limits of the muscle contraction intensity q are taken. As the characteristic range of user n, the standard characteristic range refers to the characteristic range of the user group with normal muscle synergy function, taking the upper and lower limits of the characteristic range of the healthy population {q bottom ,q top};

[0103] like Figure 4 As shown, the muscle contraction signal acquisition module specifically calculates the average value of the matrix elements in the framed area, thereby converting the two-dimensional feature vector into a one-dimensional time series, denoted by t i The signal strength corresponding to the moment is P i , i∈(1,n), n is the maximum time point corresponding to the current signal, t i The facial muscle contraction signal in the video corresponding to the moment is calculated using the following formula:

[0104]

[0105] The contraction abnormality judgment module is used to judge the current facial muscle contraction status based on the comparison result with the standard feature range; the standard feature range is obtained from the one-dimensional muscle contraction signal obtained by the muscle contraction signal acquisition module, and is used to judge whether the current facial muscle contraction function is normal;

[0106] The facial muscle collaborative analysis module includes a facial feature point positioning module, an expression judgment module, a muscle frame selection module, a feature vector combination module and an overall collaborative analysis module;

[0107] The facial feature point positioning module is used to detect facial feature points in a facial video, perform real-time 3D coordinate detection of 468 facial feature points, and generate a 3D model (KARTYNNIK Y, ABLAVATSKI A, GRISHCHENKO I, et al. Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs[J / OL]. 2019: 2–5.. http: / / arxiv.org / abs / 1907.06724.), or use dynamic methods and other facial feature recognition methods to identify facial features; to obtain a collaborative feature vector set for healthy people, the expression judgment module is used to collect three expressions of user n: smile, sadness, and fear. The muscle selection module selects the muscles mainly involved in the expression based on the judgment result of the expression judgment module (ROOT AA, STEPHENS JA. Organization of the central control of muscles of facial expression in man[J]. Journal of Physiology, 2003, 549(1): 289–298. DOI: 10.1113 / jphysiol.2002.035691.), and select the specified muscles based on the facial feature points provided by the facial feature point positioning module. According to the pattern output by the expression judgment module, combined with prior knowledge, the muscle areas A1, A2, ... A involved in this pattern are selected based on the 468 facial feature points. n Only the feature vectors in the selected area are retained. After data processing and feature vector combination of all the facial videos of the normal user group, the collaborative feature vector set of the healthy population is obtained as the basis for analyzing and judging whether the muscle coordination function is normal or abnormal under different expressions;

[0108] The muscle selection module specifically: according to the expression judgment output by the expression judgment module, the muscle areas A1, A2, ... A involved in the judgment are selected based on the 468 feature points of the face. n , only retain the feature vectors in the selected area; then select the selected areas A1, A2, ..., A n All the characteristic vectors in the region are superimposed to obtain the combined characteristic vectors M1, M2, ..., M of each region. n ; The obtained n combined feature vectors M1, M2, ..., M nSuperposition and synthesis of collaborative feature vectors are performed, and then compared with the collaborative feature vector set of healthy people. If the vector is within the collaborative feature vector set of healthy people, the analysis result is normal. If it is not within the collaborative feature vector set of healthy people, the analysis result is abnormal. Pathological research and judgment are continued, and the specific area of ​​functional attenuation is located by comparison and analysis with the system feature vector (standard vector) of a healthy person, and the corresponding pathological muscle mass is obtained.

[0109] The analysis and judgment basis is obtained by performing data processing and muscle contraction signal extraction operations on the facial video of user 1 with expression 1, and obtaining the standard vector V under user 1 expression 1 11 , perform data processing and muscle contraction signal extraction on the facial video of user 1 with expression 2, and obtain the standard vector V under user 1 with expression 2 12 , perform data processing and muscle contraction signal extraction on the facial video of user 1 expression 3, and obtain the standard vector V under user 1 expression 3 13 , user 1's standard vector set V1={V 11 ,V 12 ,V 13}, similarly, the above operation is performed on user 2 to obtain the standard vector set V2 of user 2 = {V 21 ,V 22 ,V 23}, and so on to derive the standard vector set V for user n n ={V n1 ,V n2 ,V n3}. Based on this, the expression vector {V 11 ,V 21 ,…,V n1}Integrate to obtain the collaborative feature vector Area1 of expression 1. Similarly, obtain the collaborative feature vectors Area2 and Area3 of expressions 2 and 3. If n≤20, repeat the above steps. If n>20, integrate the collaborative feature vector sets under the three expressions of the healthy population respectively as the basis for the subsequent overall collaborative analysis.

[0110] The feature vector combination module is used to combine the optical flow vectors in several areas selected by the muscle frame selection module to obtain the combined feature vectors of each area, that is, to select the frame areas A1, A2, ..., A n All the characteristic vectors in the region are superimposed to obtain the combined characteristic vectors M1, M2, ..., M of each region. n The overall collaborative analysis module obtains the n combined feature vectors M1, M2, ..., M nThe resulting synergistic feature vector is then superimposed and compared with the synergistic feature vector set of the corresponding facial expressions of healthy individuals to obtain the overall synergistic analysis result. The synergistic feature vector is a vector indicator reflecting the overall synergy of the face. It is used to locate areas of abnormal muscle function in the face and, together with the standard feature range, serves as a basis for determining whether facial muscle synergy is normal. If the overall synergistic feature vector falls within the range of the synergistic feature vector set of healthy individuals, the analysis result is normal; if it does not, the analysis result is abnormal.

[0111] The comprehensive analysis module comprehensively judges whether the facial muscle coordination function is normal based on the joint results of the muscle contraction signal extraction module and the facial muscle coordination analysis module. If the judgment results of both the muscle contraction signal extraction module and the facial muscle coordination analysis module are abnormal, it is judged that the user's facial muscle coordination function is abnormal.

[0112] Example 2

[0113] like Figure 2 As shown, during the use of the present invention, the specific operation process is as follows:

[0114] S1: Use a near-infrared camera to collect facial videos of the user under test according to three expressions: smiling, sad, and frightened. Three facial videos of the same user will be obtained.

[0115] S2: The three obtained face videos are subjected to motion amplification processing and feature vector extraction, and then facial feature point detection is performed;

[0116] S3: Based on the current expression judgment result, the muscles involved in the activity under the expression are framed, the feature vector combination of the framed area is performed, and the face area of ​​the current video is framed to extract the one-dimensional muscle contraction signal; the feature vector combination is specifically: based on the current expression judgment result, combined with the prior knowledge setting, the muscle areas A1, A2, ... A involved in this mode are framed n , only retain the feature vectors in the selected area; select the selected areas A1, A2, ..., A n All the characteristic vectors in the region are superimposed to obtain the combined characteristic vectors M1, M2, ..., M of each region. n .

[0117] S4: The obtained combined feature vectors are superimposed to obtain a collaborative feature vector and compared with the collaborative feature vector set of healthy people to perform an overall collaborative analysis. According to the result, a pathological analysis result is generated to determine whether the current muscle function is abnormal. If abnormal, the dysfunctional muscle is obtained. The overall collaborative analysis is specifically as follows: the n combined feature vectors M1, M2, ..., M obtained in step S3 are superimposed to obtain a collaborative feature vector. nBy superposition, the overall collaborative feature vector is obtained, which is compared with the collaborative feature vector set of healthy people to obtain the facial muscle collaborative analysis result. If the overall collaborative feature vector is within the collaborative feature vector set of healthy people, the analysis result is normal; if it is outside the collaborative feature vector set of healthy people, the analysis result is abnormal.

[0118] S5: Compare the obtained one-dimensional muscle contraction signal with the standard feature range {q bottom ,q top} for comparison. If it is not within the standard feature range, it is judged that the current muscle contraction function is abnormal;

[0119] S6: Comprehensively analyze the judgment results. If all the judgment results are abnormal, the facial muscle coordination analysis result of the system is abnormal;

[0120] Example 3

[0121] like Figure 5 、 Figure 6 and Figure 7 As shown in the figure, the two most involved muscle groups in each expression are selected for overall coordination analysis. For expression 1, the orbicularis oculi and zygomaticus muscles are selected for smiling. For expression 2, the corrugator supercilii and risorius muscles are selected. For expression 3, the frontalis and mentalis muscles are selected for fright. Taking expression 3, fright, as an example, a video-based muscle contraction measurement and facial muscle coordination analysis system is described in combination with the method of locating 468 feature points on the face.

[0122] The specific steps for collaborative analysis of users with normal facial muscle contraction function during Mode 3 panic are as follows:

[0123] (1) Make the user look frightened for 30 seconds;

[0124] (2) Using a camera to collect facial video information of the user, and processing the facial video containing facial feature vectors under a frightened expression;

[0125] (3) The facial feature points of 468 points are located in the face video obtained in the above steps. The frontalis and mentalis muscles are selected based on the 468 feature points. The feature vectors of each region are combined to obtain the following: Figure 7 The four combined feature vectors of;

[0126] (4) The four combined feature vectors obtained in the above steps are superimposed to obtain the overall collaborative feature vector Acosθ, A∈[A min ,A max ],θ∈[θ min ,θ max ], in the healthy population collaborative feature vector set [A MIn cosθ min ,A max cosθ max] The results of facial muscle synergy analysis were normal.

[0127] Example 4

[0128] like Figure 8 As shown in the figure, taking left frontalis muscle dysfunction as an example, a video-based muscle contraction measurement and facial muscle coordination analysis system is explained in combination with the 468 feature point positioning method of the face.

[0129] For users with abnormal contraction of the left frontalis muscle, collaborative analysis is performed in Mode 3 when they are in panic mode. At this point, the user does not know which muscle is contracting abnormally. The specific steps are as follows:

[0130] (1) Make the user look frightened for 30 seconds;

[0131] (2) using a camera to collect facial video information of the user, and processing the facial video containing facial feature vectors in the panic mode;

[0132] (3) The facial feature points of 468 points are located in the face video obtained in the above steps. The frontalis and mentalis muscles are selected based on the 468 feature points. The feature vectors of each region are combined to obtain the following: Figure 8 The four combined feature vectors of;

[0133] (4) Superimpose the four combined feature vectors obtained in the above steps to obtain the overall collaborative feature vector Acosθ, Not in the healthy population collaborative feature vector set [A min cosθ min ,A max cosθ max ], the results of facial muscle synergy analysis were abnormal.

[0134] (5) Randomly obtain the collaborative feature vector A of the expression 3 of the healthy population n cosθ n , compared with the overall collaborative feature vector Acosθ obtained in the above steps, as shown Figure 9 , according to A n cosθ n -Directional analysis of Acosθ can locate the position of the pathological muscle, thereby knowing that the muscle with dysfunction is the left frontalis muscle.

[0135] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A video-based muscle contraction measurement and facial muscle coordination analysis system, characterized in that: The system includes: a data acquisition module, a data processing module, a muscle contraction signal extraction module, a facial muscle coordination analysis module and a comprehensive analysis module; The data acquisition module is used to obtain face videos; The data processing module includes a video micro-motion amplification module and a feature vector extraction module; the video micro-motion amplification module is used to amplify the motion signal of the face video acquired by the data acquisition module to obtain micro-motions that are difficult for the human eye to observe in the original face video; the feature vector extraction module is used to extract the optical flow vector of the face surface in the face video, which includes the muscle movement trend; The muscle contraction signal extraction module includes a local frame selection module, a muscle contraction signal acquisition module and a contraction abnormality judgment module; The local frame selection module is used to frame the muscle area of ​​interest in the face video; the muscle contraction signal acquisition module is used to obtain a one-dimensional muscle contraction signal of the muscle area of ​​interest; the contraction abnormality judgment module is used to judge the current facial muscle contraction status based on the comparison result with the standard feature range; the standard feature range is derived from the one-dimensional muscle contraction signal obtained by the muscle contraction signal acquisition module, and is used to judge whether the current facial muscle contraction function is normal; The facial muscle collaborative analysis module includes a facial feature point positioning module, an expression judgment module, a muscle frame selection module, a feature vector combination module and an overall collaborative analysis module; The facial feature point positioning module is used to detect facial feature points in a facial video; the expression judgment module is used to judge the expression of the current facial video; the muscle selection module selects the muscles mainly involved in the expression according to the result of the expression judgment module, and selects the specified muscles according to the facial feature points provided by the facial feature point positioning module; The feature vector combination module is used to combine the optical flow vectors in several areas selected by the muscle frame selection module to obtain a combined feature vector for each area; the overall collaborative analysis module superimposes the combined feature vectors to obtain a collaborative feature vector, which is compared with the collaborative feature vector set of the corresponding expressions of healthy people to obtain an overall collaborative analysis result; The comprehensive analysis module comprehensively judges whether the facial muscle coordination function is normal based on the joint results of the muscle contraction signal extraction module and the facial muscle coordination analysis module. If the judgment results of both the muscle contraction signal extraction module and the facial muscle coordination analysis module are abnormal, the facial muscle coordination function is judged to be abnormal.

2. A video-based muscle contraction measurement and facial muscle synergy analysis system according to claim 1, characterized in that: The video acquisition module includes a near-infrared LED lamp and a near-infrared camera; the near-infrared LED lamp serves as the light source for the near-infrared camera to shoot; the near-infrared camera is used to record user and face videos, further measure muscle contraction signals, analyze muscle synergy and make pathological judgments.

3. A video-based muscle contraction measurement and facial muscle coordination analysis system according to claim 1, characterized in that: The video micro-motion amplification module is specifically: Will time, Image about location Performing a first-order Taylor expansion yields: in, is the image intensity, is the displacement function, and the bandpass signal is obtained after time domain filtering. Assuming that the motion signal In the passband of the time bandpass filter: make by The index represents the different time spectrum components of δ(t), each will be attenuated by a factor of time filtering , from which the motion signal is derived The general case of not being completely within the passband of the time filter: The bandpass signal enlarge times, with Add together to get the processed signal : 。 4. A video-based muscle contraction measurement and facial muscle coordination analysis system according to claim 3, characterized in that: Processed signal Further processing yields: Accordingly, the motion signal The situation that is not completely within the passband of the time filter is processed to obtain: show Partial image at a moment spatial displacement Has been magnified times, is the amplification factor.

5. A video-based muscle contraction measurement and facial muscle coordination analysis system according to claim 1, characterized in that: The feature vector extraction module specifically performs optical flow extraction on the amplified video. The constraint equation of the optical flow method is: in is the partial derivative of the grayscale of the pixel in the image along the x, y, and t directions, are the components of the optical flow along the x and y directions respectively; The energy function is obtained as follows: in, As the grayscale change factor, is the grayscale invariance assumption, As a smoothing factor, it is used to limit the speed of change of u and v components; by solving the Euler-Lagrange equation, let As the sum of the grayscale change factor and the smoothing factor, the minimum value of the two factors is obtained and the derivative is obtained: in is the Laplace operator, ; use After approximate substitution, the above formula is approximately replaced by: The system of equations can be solved by iterative formula: Thus, the optical flow vector in the k+1th frame is obtained Thus, the muscle activity feature vector is obtained.

6. A video-based muscle contraction measurement and facial muscle synergy analysis system according to claim 1, characterized in that: The local selection module creates a rectangle in the specified area (x, y, w, h), where x, y, w, and h are the center coordinates, length, and width of the rectangle, respectively. The area outside the rectangle is then colored black to create a mask to block out areas of no interest, retaining only the feature vectors inside the rectangle, thereby eliminating light interference outside the specified area.

7. A video-based muscle contraction measurement and facial muscle coordination analysis system according to claim 1, characterized in that: The muscle contraction signal acquisition module specifically calculates the average value of the matrix elements in the framed area, thereby converting the two-dimensional feature vector into a one-dimensional time series. The signal strength corresponding to the time is , , n is the maximum time point corresponding to the current signal, Video of human muscle contraction signal corresponding to the moment Use the following formula to calculate: 。 8. The video-based muscle contraction measurement and facial muscle coordination analysis system according to claim 1, characterized in that: The facial feature point positioning module specifically performs real-time 3D coordinate detection of 468 facial feature points and generates a 3D model, or uses a dynamic method to identify facial features.

9. The video-based muscle contraction measurement and facial muscle coordination analysis system according to claim 1, characterized in that: The muscle selection module specifically: according to the expression judgment output by the expression judgment module, the muscle involved in the judgment is obtained by combining prior knowledge, and the muscle area involved in the judgment is selected according to the 468 feature points of the face. , only retain the feature vectors in the selected area; then All the characteristic vectors in the vector superposition are obtained The combined feature vector of each region ; The obtained n combined feature vectors The synergistic feature vectors are superimposed and synthesized, and compared with the synergistic feature vector set of healthy people. If they are within the synergistic feature vector set of healthy people, the analysis result is normal. If they are not within the synergistic feature vector set of healthy people, the analysis result is abnormal. Pathological research and judgment are continued, and the specific area of ​​functional attenuation is located by comparative analysis with the synergistic feature vector of a healthy person, corresponding to the pathological muscle mass.

10. The video-based muscle contraction measurement and facial muscle coordination analysis system according to claim 1, characterized in that: The collaborative feature vector refers to a vector indicator that reflects the overall coordination of the face. It is used to locate areas in the face where muscle function is abnormal, and together with the standard feature range, serves as a basis for judging whether the facial muscle coordination function is normal or not.

Images