System and method for identifying dysphagia based on computer vision

Abstract

The invention relates to a dysphagia recognition system and method based on computer vision, and relates to the technical field of computer vision, and the system comprises a data processing module which is used for collecting multi-modal data of a user and carrying out the normalization preprocessing, and obtaining the preprocessed multi-modal data; the feature fusion module is used for generating time sequence sensitive fusion features; the time sequence modeling module is used for performing dynamic time sequence modeling on the fusion features based on a recursive attention mechanism, capturing time sequence dependence of swallowing actions, and obtaining an attention distribution value as a time sequence modeling result; the classification detection module is used for performing anomaly detection and classification on the time sequence modeling result according to a dynamic threshold discrimination function to obtain a discrimination score and a classification result of the dysphagia; and the result identification module is used for carrying out comprehensive evaluation on the classification result through a multi-scale optimization integration algorithm, outputting evaluation indexes of the dysphagia and obtaining a dysphagia identification result. The system can improve the accuracy of dysphagia identification.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a swallowing disorder recognition system, method, electronic device and non-transitory computer-readable storage medium based on computer vision. Background Art

[0002] Dysphagia is a significant complication of various diseases and aging, and is common in patients with stroke, Parkinson's disease, and postoperative head and neck tumors. With the recent advancement of computer vision technology, researchers have begun using deep learning models to automatically analyze VFSS image sequences or facial video data, extracting features such as tongue movement, laryngeal elevation, and tracheal closure to assist in the assessment of swallowing function.

[0003] However, most current dysphagia recognition methods rely heavily on manually annotated keyframes or feature points, resulting in poor model generalization and difficulty adapting to individual anatomical differences and changes in swallowing patterns. Sequential modeling often uses static time windows, lacking sufficient awareness of the dynamics of the swallowing process and easily missing critical abnormal moments. Summary of the Invention

[0004] In response to the technical problems existing in the prior art, the present invention provides a dysphagia recognition system, method, electronic device and non-transitory computer-readable storage medium based on computer vision, which can improve the accuracy of dysphagia recognition.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] The present invention provides a dysphagia recognition system based on computer vision, the system comprising:

[0007] A data processing module is used to collect multimodal data of the user through VFSS image sequences and facial videos, and perform normalization preprocessing on the multimodal data to obtain preprocessed multimodal data;

[0008] The feature fusion module is used to extract and fuse dynamic features of pre-processed multimodal data to generate time-sensitive fusion features;

[0009] A temporal modeling module is used to perform dynamic temporal modeling on the fused features based on a recursive attention mechanism, capture the temporal dependency of swallowing movements, and obtain an attention distribution value as a temporal modeling result;

[0010] A classification detection module, configured to perform abnormality detection and classification on the time series modeling results according to a dynamic threshold discriminant function, and obtain a discrimination score and classification result of dysphagia;

[0011] The result recognition module is used to comprehensively evaluate the classification results through a multi-scale optimization integration algorithm, output the evaluation index of dysphagia, and obtain the dysphagia recognition result.

[0012] Furthermore, the data processing module is also used to:

[0013] Nonlinear mapping of a hyperbolic tangent function performs nonlinear normalization processing on the multimodal data;

[0014] Obtaining a scale factor for adjusting the data range and a dynamic modulation term of a sinusoidal fluctuation over time for synchronously adapting to changes in data amplitude and timing;

[0015] The normalization processing result of the multimodal data is adjusted by using the scale factor and the dynamic modulation item to obtain the preprocessed multimodal data.

[0016] Furthermore, the feature fusion module is also used to:

[0017] Obtaining feature maps of each mode of the preprocessed multimodal data;

[0018] Applying a time-varying window function and a modality weight to the feature maps of each modality to extract and fuse local window features;

[0019] After dynamically adjusting the integration result of the time series kernel function and the historical state vector through the adaptive coefficient, it is spliced ​​with the fused window features to generate the fused features associated with the global time series.

[0020] Furthermore, the time series modeling module is also used to:

[0021] Calculate the attention score and map it to a probability distribution through the activation function;

[0022] Combining the probability distribution with a Gaussian time decay function to suppress noise interference far away from the reference time;

[0023] After utilizing the second-order derivative information in the feature space to enhance the sensitivity to the mutation region of the fused feature, the information is superimposed on the processed probability distribution to obtain the attention distribution value.

[0024] Furthermore, the classification detection module is also used to:

[0025] Calculate the distance metric between the sample and the dynamic cluster center based on the time-varying weight;

[0026] fusing the attention distribution value and the fusion feature through an attention fusion coefficient;

[0027] The distance metric, the fused attention distribution value, the fused feature, and the adaptive error term are superimposed to compensate for model uncertainty to obtain a discrimination score for the dysphagia.

[0028] Furthermore, the dynamic cluster center is iteratively updated by the sample mean within the sliding time window, and smoothly transitioned in combination with the momentum term of the historical cluster center.

[0029] Furthermore, the result identification module is also used to:

[0030] Assign weight factors of each scale to the classification results of different time scales;

[0031] Dynamically adjusting the dysphagia discrimination score according to the weight factors of each scale and the system entropy value;

[0032] The reliability metric and the reliability modulation coefficient are introduced to perform probability calibration on the adjusted discrimination score, and the evaluation index of the dysphagia is obtained as the dysphagia identification result.

[0033] The present invention also provides a method for identifying dysphagia based on computer vision, the method comprising:

[0034] Collecting multimodal data of the user through VFSS image sequences and facial videos, and performing normalization preprocessing on the multimodal data to obtain preprocessed multimodal data;

[0035] Perform dynamic feature extraction and fusion on pre-processed multimodal data to generate time-sensitive fusion features;

[0036] Based on the recursive attention mechanism, dynamic temporal modeling is performed on the fusion features to capture the temporal dependency of the swallowing action, and the attention distribution value is obtained as the temporal modeling result;

[0037] Performing abnormality detection and classification on the time series modeling results according to a dynamic threshold discriminant function to obtain a discriminant score and classification result of dysphagia;

[0038] The classification results are comprehensively evaluated through a multi-scale optimization integration algorithm, and the evaluation index of dysphagia is output to obtain the dysphagia recognition result.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also proposes an electronic device, comprising: a memory for storing a computer software program; a processor for reading and executing the computer software program, thereby implementing a swallowing disorder identification method based on computer vision as described above.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes a non-transitory computer-readable storage medium, in which a computer software program is stored. When the computer software program is executed by a processor, it implements a swallowing disorder identification method based on computer vision as described above.

[0041] The beneficial effects of the present invention are:

[0042] (1) By introducing multimodal data (such as VFSS images and facial videos) and designing dynamic feature fusion and attention mechanisms, this method can capture subtle changes in the swallowing process from multiple angles and dimensions, effectively improving the detection accuracy of swallowing disorders.

[0043] (2) The dynamic normalization function and dynamic feature extraction operator of the present invention have time modulation capabilities, which can adapt to individual differences and dynamic changes in the swallowing process and improve the temporal perception ability of the model.

[0044] (3) The recursive attention mechanism of the present invention, combined with temporal annealing and second-order feature enhancement, can detect subtle but critical abnormal signs at an early stage, providing an earlier diagnostic basis for the intervention and treatment of dysphagia. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A scene diagram of a dysphagia recognition method based on computer vision provided by the present invention;

[0046] Figure 2 A schematic diagram of the structure of a swallowing disorder recognition system based on computer vision provided by the present invention;

[0047] Figure 3 A flowchart of a method for identifying dysphagia based on computer vision provided by the present invention;

[0048] Figure 4 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0049] Figure 5 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] See also Figure 1 , Figure 1 This is a scene diagram of a swallowing disorder recognition method based on computer vision provided by the present invention. Figure 1 As shown, the terminal and server are connected via a network, such as a wired or wireless network. Terminals include, but are not limited to, portable devices such as mobile phones and tablets installed with various network platform applications, as well as fixed devices such as computers, kiosks, and advertising machines. The server provides various business services to users, including service push servers and user recommendation servers.

[0052] It should be noted that Figure 1 The scene diagram of a swallowing disorder identification method based on computer vision is only an example. The terminal, server and application scenario described in the embodiment of the present invention are intended to more clearly illustrate the technical solution of the embodiment of the present invention, and do not generate any limitation on the technical solution provided by the embodiment of the present invention. Ordinary technicians in this field know that with the evolution of the system and the emergence of new business scenarios, the technical solution provided by the embodiment of the present invention is also applicable to similar technical problems.

[0053] Among them, the terminal can be used to:

[0054] Collecting multimodal data of the user through VFSS image sequences and facial videos, and performing normalization preprocessing on the multimodal data to obtain preprocessed multimodal data;

[0055] Perform dynamic feature extraction and fusion on pre-processed multimodal data to generate time-sensitive fusion features;

[0056] Based on the recursive attention mechanism, dynamic temporal modeling is performed on the fusion features to capture the temporal dependency of the swallowing action, and the attention distribution value is obtained as the temporal modeling result;

[0057] Performing abnormality detection and classification on the time series modeling results according to a dynamic threshold discriminant function to obtain a discriminant score and classification result of dysphagia;

[0058] The classification results are comprehensively evaluated through a multi-scale optimization integration algorithm, and the evaluation index of dysphagia is output to obtain the dysphagia recognition result.

[0059] See also Figure 2 , Figure 2 This is a structural schematic diagram of a swallowing disorder recognition system based on computer vision provided by the present invention.

[0060] like Figure 2 As shown, a computer vision-based swallowing disorder recognition system proposed in an embodiment of the present invention includes:

[0061] The data processing module 201 is used to collect multimodal data of the user through VFSS image sequences and facial videos, and perform normalization preprocessing on the multimodal data to obtain preprocessed multimodal data;

[0062] The feature fusion module 202 is used to extract and fuse dynamic features of the pre-processed multimodal data to generate time-sensitive fusion features;

[0063] A time series modeling module 203 is configured to perform dynamic time series modeling on the fused features based on a recursive attention mechanism, capture the time series dependency of the swallowing action, and obtain an attention distribution value as a time series modeling result;

[0064] A classification detection module 204 is configured to perform abnormality detection and classification on the time series modeling results according to a dynamic threshold discriminant function to obtain a dysphagia discrimination score and classification result;

[0065] The result recognition module 205 is used to comprehensively evaluate the classification results through a multi-scale optimization integration algorithm, output the evaluation index of dysphagia, and obtain the dysphagia recognition result.

[0066] Furthermore, the data processing module 201 is further configured to:

[0067] Nonlinear mapping of a hyperbolic tangent function performs nonlinear normalization processing on the multimodal data;

[0068] Obtaining a scale factor for adjusting the data range and a dynamic modulation term of a sinusoidal fluctuation over time for synchronously adapting to changes in data amplitude and timing;

[0069] The normalization processing result of the multimodal data is adjusted by using the scale factor and the dynamic modulation item to obtain the preprocessed multimodal data.

[0070] Furthermore, the processed multimodal data is represented as:

[0071]

[0072] in, is the processed multimodal data, is the scale factor, is the nonlinear coefficient, is the bias term, is the dynamic modulation amplitude, is the time-varying frequency, is the time variable, is the input multimodal data.

[0073] In the specific implementation, the nonlinear normalization part , realize nonlinear compression of the original input x (the range is approximately [-1,1]), Controls the "steepness" of the compression, larger → closer to a step function, smaller → smoother. Offsets the original input, controlling the normalized center position.

[0074] Dynamic modulation items , oscillates with time t, giving the normalized output a “timing-aware” modulation. Controls the modulation intensity (amplitude). The larger the value, the more obvious the dynamic changes. Frequency coefficient, adjusts the speed of periodicity (unit is the same as t).

[0075] It is the scale factor, which is the amplification factor of the overall product and sets the final amplitude range of the output, such as expanding the normalized data to a certain model input range.

[0076] The following calculation Assume that the parameters are as follows:

[0077] α=2.0 (scale factor);

[0078] β = 1.5 (nonlinear coefficient);

[0079] γ = 0.2 (bias term);

[0080] δ=0.3 (modulation amplitude);

[0081] ω = π (frequency is π, period is 2 seconds);

[0082] t = 1 (current time);

[0083] x=0.5 (original input value).

[0084] The calculation process is as follows:

[0085] .

[0086] The control effects of each parameter are summarized in Table 1 below.

[0087]

[0088] Table 1

[0089] Furthermore, the feature fusion module 202 is further configured to:

[0090] Obtaining feature maps of each mode of the preprocessed multimodal data;

[0091] Applying a time-varying window function and a modality weight to the feature maps of each modality to extract and fuse local window features;

[0092] After dynamically adjusting the integration result of the time series kernel function and the historical state vector through the adaptive coefficient, it is spliced ​​with the fused window features to generate the fused features associated with the global time series.

[0093] Furthermore, the fusion feature is expressed as:

[0094]

[0095] in, is the fusion feature, is the weight of the i-th mode, is a time-varying window function, is the feature map, is the adaptive coefficient, is the time series kernel function, is the historical state vector.

[0096] In the specific implementation, is the final feature representation at the current time t, Characterize the importance of each mode to the final feature, Extract the active interval of mode i at the current time, is the feature value of the original input (such as VFSS / video) after processing, Dynamically adjust the influence weight of historical integral items, Measure the similarity between t and τ, Incorporate temporal context information to enhance time series modeling.

[0097] The calculation process of is as follows, assuming that the system contains two modalities: modality 1 is a VFSS image sequence, and modality 2 is a facial video frame. Assume that time point t = 1.0 and set the following parameters:

[0098] , 0.4, 0.9, 0.7, 1.2, 0.8, 0.5, , , 0, 0.5, 1, the value is used to approximate the integral term using numerical integration (trapezoidal method).

[0099] 0.6·0.9·1.2+0.4·0.7·0.8=0.648+0.224=0.872.

[0100] 0.5·(K(1,0)·0+K(1,0.5)·0.5+K(1,1.0)·1.0)=0.5·(0+0.7788·0.5+1.0·1.0)=0.5·(0.3894+1.0)=0.5·1.3894=0.6947.

[0101] Multiply : 0.5·0.6947=0.3474.

[0102] 0.872+0.3474=1.2194.

[0103] In summary, the present invention fully integrates different types of information (visual, physiological, behavioral) through multimodal fusion, introduces contextual memory through time-varying windows and temporal kernel functions through time perception, and dynamically adjusts the influence of different parts through learning / scheduling through adaptive regulation. Each item has physical or semantic meaning, which is conducive to analysis and optimization.

[0104] Furthermore, the time series modeling module 203 is further configured to:

[0105] Calculate the attention score and map it to a probability distribution through the activation function;

[0106] Combining the probability distribution with a Gaussian time decay function to suppress noise interference far away from the reference time;

[0107] After utilizing the second-order derivative information in the feature space to enhance the sensitivity to the mutation region of the fused feature, the information is superimposed on the processed probability distribution to obtain the attention distribution value.

[0108] In some embodiments, the attention distribution value is represented as:

[0109]

[0110] in, is the attention distribution value, is the attention score, is the time decay coefficient, is the reference time point, is the second-order derivative weight coefficient, is the Laplace operator, is the fusion feature, is the normalized activation function.

[0111] In the specific implementation, is the weighted value of the importance of the current moment t to the decision, Normalize the attention scores to the range [0,1], Control away from the reference point The rate of attention decay, It is the square of the distance between the current time and the reference time. The farther the distance between the current time and the reference time, the smaller the value (Gaussian attenuation). Controls the sensitivity to feature changes (curvature), Measure the trend of feature changes (such as local peaks / valleys), Derived from the feature fusion result of the previous step.

[0112] The term is to normalize the attention score so that it can be used as a weight, Item The closer the distance is to the reference time , the more focused the attention is, the farther the distance between the two is, the attenuation will be. If the feature At this time, drastic changes (such as sudden peaks / troughs) are reflected through the second-order derivative, which increases attention sensitivity.

[0113] As an example only, assume 0.8, 0.5, t=2, 1, 0.6, then 0.689, 1, -0.3, indicating that this point is a slight valley.

[0114] Continue calculation: exp(-0.5·1)=exp(-0.5)≈0.6065;

[0115] 1+0.6·(-0.3)=1-0.18=0.82;

[0116] 0.689·0.6065·0.82≈0.343.

[0117] Based on the model / network’s direct attention estimate of the current input, The closer the time is to the reference point, the more important it is. Enhance attention focus at points of feature mutation.

[0118] Through the above methods, the present invention enables attention distribution to have both global temporal structure perception and sensitive response to local feature changes, which is particularly suitable for capturing swallowing moments, abnormal rhythms, and other time points in VFSS.

[0119] Furthermore, the classification detection module 204 is further configured to:

[0120] Calculate the distance metric between the sample and the dynamic cluster center based on the time-varying weight;

[0121] fusing the attention distribution value and the fusion feature through an attention fusion coefficient;

[0122] The distance metric, the fused attention distribution value, the fused feature, and the adaptive error term are superimposed to compensate for model uncertainty to obtain a discrimination score for the dysphagia.

[0123] Furthermore, the dynamic cluster center is iteratively updated by the sample mean within the sliding time window, and smoothly transitioned in combination with the momentum term of the historical cluster center.

[0124] In some embodiments, the dysphagia discrimination score is expressed as:

[0125]

[0126] in, is the discriminant score of dysphagia, is the time-varying weight, is the dynamic cluster center, is the attention fusion coefficient, is the adaptive error term, is the attention distribution value, It is a fusion feature.

[0127] In the specific implementation, Controls the sensitivity of anomaly detection to sample deviation. X is the input feature, such as the extracted multimodal feature vector. Equivalent to the center of the normal sample distribution, dynamically learned, Control the proportion of attention in the judgment, Capture system residual errors, dynamic disturbances or modeling deviations, Represents the importance of the current moment, Derived from the feature fusion module.

[0128] Deviation detection items , evaluate the distance between the current input x and the normal state center c(t). Large deviation → suspected abnormality. Attention fusion term , introduces contextual attention modulation to improve the accuracy of temporal discrimination (such as paying special attention to key moments). Error correction term , compensate for model errors and fluctuation interference of nonlinear systems.

[0129] Now assume: x=[1.5,2.0], c(t)=[1.0,1.2], μ(t)=0.8, A(t)=0.6, Φ(t)=1.4, η=0.5, ε(t)=0.1.

[0130] 0.8·0.89=0.712;

[0131] 0.5·0.84=0.42;

[0132] 0.712+0.42+0.1=1.232.

[0133] The distance deviation term sensitively captures input that deviates from normal swallowing patterns, the attention modulation term focuses on critical moments (such as swallow initiation and delay) to improve discrimination, and the adaptive error term accounts for individual patient differences and measurement errors. This invention is suitable for use in automated swallowing disorder detection systems, combined video analysis with physiological data for identification, and real-time clinical risk alert mechanisms.

[0134] Furthermore, the result identification module 205 is further configured to:

[0135] Assign weight factors of each scale to the classification results of different time scales;

[0136] Dynamically adjusting the dysphagia discrimination score according to the weight factors of each scale and the system entropy value;

[0137] The reliability metric and the reliability modulation coefficient are introduced to perform probability calibration on the adjusted discrimination score, and the evaluation index of the dysphagia is obtained as the dysphagia identification result.

[0138] In some embodiments, the dysphagia assessment index is expressed as:

[0139]

[0140] in, It is an evaluation indicator of swallowing disorder. is the weight factor of each scale, is the discriminant score of dysphagia, H is the system entropy value, is the entropy weight coefficient, R is the reliability measure, is the reliability modulation coefficient.

[0141] In the specific implementation, Characterize the overall risk score as the system output. is the importance of different time windows, modality resolutions, or feature spaces. Used for anomaly detection. H represents the uncertainty or confusion of the model for the current sample set. Controls the degree to which entropy affects the score. R represents the confidence level of the current judgment or data (such as data completeness). Control the moderating effect of 𝑅 on the total score.

[0142] Multi-scale ensemble scoring ∑[ π k ·D( x k ,t)] , which is a weighted aggregation of scores calculated at multiple scales (e.g., different resolutions, modalities, or time windows). Entropy penalty , the more uncertain the system is (the higher the entropy), the lower the overall score will be, thus avoiding misjudgment when the information is chaotic. If the current judgment is very credible (high R), the factor will be >1, which will amplify the scoring result; if it is low credible, it will approach 1 or even weaken it.

[0143] Assume the following parameters: The values ​​of are 0.5, 0.3 and 0.2 respectively, The values ​​of are 1.2, 0.9 and 1.5 respectively, H=0.8, 0.6, 0.75, Q=0.4.

[0144] (0.5·1.2)+(0.3·0.9)+(0.2·1.5)=0.6+0.27+0.3=1.17;

[0145] exp(-0.6·0.8)=exp(-0.48)≈0.619;

[0146] 1+0.4·0.75=1+0.3=1.3;

[0147] E = 1.17 · 0.619 · 1.3 ≈ 0.902;

[0148] In this formula, Aggregate perceptions at different levels to resist single-scale errors and improve discrimination generalization. Suppress high uncertainty scenarios and improve the model's stability to chaotic states. Adjust output confidence to enhance responses to high-quality judgments and reduce the impact of low-quality judgments.

[0149] The present invention is suitable for scenarios such as swallowing assessment based on integrated multi-source (such as VFSS, facial video) data, cross-time scale (frame level, cycle level) fusion analysis, and the design of highly robust medical auxiliary diagnosis systems.

[0150] In some embodiments, multiple recognition thresholds can be set, such as the first threshold z1 and the second threshold z2. If D(x,t) < z1, it is determined as normal swallowing; if z1 ≤ D(x,t) < z2, it is determined as mild dysphagia; if D(x,t) ≥ z2, it is determined as severe dysphagia. Thus, the present invention has the recognition advantages that the score adapts to changes over time and modalities; the distance from the normal center and the attention weight of the score are both visualized; it supports binary classification, ternary classification, or fine-grained grading, etc.

[0151] Please refer to Figure 3 , which provides a flowchart of a method for identifying dysphagia based on computer vision according to the present invention, including the following steps:

[0152] Step 301: Collect multi-modal data of the user through VFSS image sequences and facial videos, and perform normalization preprocessing on the multi-modal data to obtain preprocessed multi-modal data;

[0153] Extract and fuse dynamic features from the preprocessed multi-modal data to generate temporally sensitive fused features;

[0154] Perform dynamic temporal modeling on the fused features based on a recursive attention mechanism to capture the temporal dependence of swallowing actions, and obtain attention distribution values as the results of temporal modeling;

[0155] Perform anomaly detection and classification on the results of temporal modeling according to a dynamic threshold discrimination function to obtain the discrimination score and classification result of dysphagia;

[0156] Comprehensively evaluate the classification results through a multi-scale optimization integration algorithm, output evaluation indicators of dysphagia, and obtain the identification result of dysphagia.

[0157] It should be noted that for the specific embodiments and beneficial effects of steps 301-305 above, please refer to the description of modules 201-205 above, and will not be elaborated here.

[0158] Please refer to Figure 4 , Figure 4 which is a schematic diagram of an embodiment of an electronic device provided by an embodiment of the present invention. As Figure 4 shown, an embodiment of the present invention provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, the following steps are implemented:

[0159] Collect multi-modal data of the user through VFSS image sequences and facial videos, and perform normalization preprocessing on the multi-modal data to obtain preprocessed multi-modal data;

[0160] Perform dynamic feature extraction and fusion on pre-processed multimodal data to generate time-sensitive fusion features;

[0161] Based on the recursive attention mechanism, dynamic temporal modeling is performed on the fusion features to capture the temporal dependency of the swallowing action, and the attention distribution value is obtained as the temporal modeling result;

[0162] Performing abnormality detection and classification on the time series modeling results according to a dynamic threshold discriminant function to obtain a discriminant score and classification result of dysphagia;

[0163] The classification results are comprehensively evaluated through a multi-scale optimization integration algorithm, and the evaluation index of dysphagia is output to obtain the dysphagia recognition result.

[0164] See also Figure 5 , Figure 5 Schematic diagram of an embodiment of a computer-readable storage medium provided by an embodiment of the present invention. Figure 5 As shown, this embodiment provides a computer-readable storage medium 500 on which a computer program 411 is stored. When the computer program 411 is executed by a processor, the following steps are implemented:

[0165] Collecting multimodal data of the user through VFSS image sequences and facial videos, and performing normalization preprocessing on the multimodal data to obtain preprocessed multimodal data;

[0166] Perform dynamic feature extraction and fusion on pre-processed multimodal data to generate time-sensitive fusion features;

[0167] Based on the recursive attention mechanism, dynamic temporal modeling is performed on the fusion features to capture the temporal dependency of the swallowing action, and the attention distribution value is obtained as the temporal modeling result;

[0168] Performing abnormality detection and classification on the time series modeling results according to a dynamic threshold discriminant function to obtain a discriminant score and classification result of dysphagia;

[0169] The classification results are comprehensively evaluated through a multi-scale optimization integration algorithm, and the evaluation index of dysphagia is output to obtain the dysphagia recognition result.

[0170] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0171] Those skilled in the art will appreciate that embodiments of the present invention may be provided as systems, methods, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or boxes.

[0173] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system that is implemented in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0175] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0176] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A swallowing disorder recognition system based on computer vision, characterized in that: The system comprises: A data processing module is used to collect multimodal data of the user through VFSS image sequences and facial videos, and perform normalization preprocessing on the multimodal data to obtain preprocessed multimodal data; The feature fusion module is used to extract and fuse dynamic features of pre-processed multimodal data to generate time-sensitive fusion features; A temporal modeling module is used to perform dynamic temporal modeling on the fused features based on a recursive attention mechanism, capture the temporal dependency of swallowing movements, and obtain an attention distribution value as a temporal modeling result; A classification detection module, configured to perform abnormality detection and classification on the time series modeling results according to a dynamic threshold discriminant function, and obtain a discrimination score and classification result of dysphagia; The result recognition module is used to comprehensively evaluate the classification results through a multi-scale optimization integration algorithm, output the evaluation index of dysphagia, and obtain the dysphagia recognition result.

2. The computer vision-based swallowing disorder recognition system according to claim 1, characterized in that: The data processing module is further configured to: Nonlinear mapping of a hyperbolic tangent function performs nonlinear normalization processing on the multimodal data; Obtaining a scale factor for adjusting the data range and a dynamic modulation term of a sinusoidal fluctuation over time for synchronously adapting to changes in data amplitude and timing; The normalization processing result of the multimodal data is adjusted by using the scale factor and the dynamic modulation item to obtain the preprocessed multimodal data.

3. The computer vision-based swallowing disorder recognition system according to claim 2, characterized in that: The feature fusion module is further used for: Obtaining feature maps of each mode of the preprocessed multimodal data; Applying a time-varying window function and a modality weight to the feature maps of each modality to extract and fuse local window features; After dynamically adjusting the integration result of the time series kernel function and the historical state vector through the adaptive coefficient, it is spliced ​​with the fused window features to generate the fused features associated with the global time series.

4. The computer vision-based swallowing disorder recognition system according to claim 3, characterized in that: The timing modeling module is further configured to: Calculate the attention score and map it to a probability distribution through the activation function; Combining the probability distribution with a Gaussian time decay function to suppress noise interference far away from the reference time; After utilizing the second-order derivative information in the feature space to enhance the sensitivity to the mutation region of the fused feature, the information is superimposed on the processed probability distribution to obtain the attention distribution value.

5. The computer vision-based swallowing disorder recognition system according to claim 4, characterized in that: The classification detection module is also used for: Calculate the distance metric between the sample and the dynamic cluster center based on the time-varying weight; fusing the attention distribution value and the fusion feature through an attention fusion coefficient; The distance metric, the fused attention distribution value, the fused feature, and the adaptive error term are superimposed to compensate for model uncertainty to obtain a discrimination score for the dysphagia.

6. The computer vision-based swallowing disorder recognition system according to claim 5, characterized in that: The dynamic cluster center is iteratively updated by the sample mean within the sliding time window, and the momentum term of the historical cluster center is combined to smoothly transition.

7. The computer vision-based swallowing disorder recognition system according to claim 6, characterized in that: The result identification module is also used for: Assign weight factors of each scale to the classification results of different time scales; Dynamically adjusting the dysphagia discrimination score according to the weight factors of each scale and the system entropy value; The reliability metric and the reliability modulation coefficient are introduced to perform probability calibration on the adjusted discrimination score, and the evaluation index of the dysphagia is obtained as the dysphagia identification result.

8. A method for identifying dysphagia based on computer vision, the method implementing the system according to claim 1, characterized in that: The method comprises: Collecting multimodal data of the user through VFSS image sequences and facial videos, and performing normalization preprocessing on the multimodal data to obtain preprocessed multimodal data; Perform dynamic feature extraction and fusion on pre-processed multimodal data to generate time-sensitive fusion features; Based on the recursive attention mechanism, dynamic temporal modeling is performed on the fusion features to capture the temporal dependency of the swallowing action, and the attention distribution value is obtained as the temporal modeling result; Performing abnormality detection and classification on the time series modeling results according to a dynamic threshold discriminant function to obtain a discriminant score and classification result of dysphagia; The classification results are comprehensively evaluated through a multi-scale optimization integration algorithm, and the evaluation index of dysphagia is output to obtain the dysphagia recognition result.

Images