Method and system for swallowing disorder risk identification based on machine vision

Abstract

The application discloses a method and system for identifying swallowing disorder risk based on machine vision, which comprises the following steps: receiving video acquisition data, loading a target identification model, the target identification model being pre-configured with multiple sets of quantization sub-parameters, different quantization sub-parameters corresponding to different quantization bit widths; obtaining the hardware state information of the video acquisition data source terminal, selecting target sub-parameters for the network blocks of the target identification model from the multiple sets of quantization sub-parameters based on the hardware state information; extracting features from the video acquisition data by using the target identification model configured with the target sub-parameters to obtain throat movement features, facial movement features and head posture features; generating a swallowing stability parameter based on the throat movement features, the facial movement features and the head posture features; comparing the swallowing stability parameter with a preset risk threshold to determine a risk level and performing differentiated message pushing. The method significantly improves the universal deployment reliability and continuous monitoring stability of the swallowing disorder risk identification.

Description

Technical Field

[0001] This application relates to the fields of computer vision and pattern recognition technology, and in particular to a method and system for identifying swallowing disorders based on machine vision. Background Technology

[0002] In machine vision-based dysphagia risk identification, video analysis of laryngeal, facial, and head movement features enables non-contact assessment and long-term monitoring. However, existing edge-side model deployment methods have significant shortcomings when dealing with heterogeneous terminal environments. Traditional deployment schemes often rely solely on a single preset computational accuracy configuration for inference, failing to consider the significant differences in hardware computing power and real-time load between actual deployment terminals. This results in the same model losing subtle laryngeal movement information due to excessive accuracy compression on resource-constrained devices, while on devices with sufficient computing power, it is limited by the predetermined configuration and cannot further improve the discrimination sensitivity. Consequently, it becomes difficult to achieve both real-time continuous monitoring and fine feature capture, affecting the overall reliability of dysphagia risk identification. Summary of the Invention

[0003] The main technical problem addressed in this application is to provide a machine vision-based method and system for identifying swallowing disorders, which can effectively improve the reliability of universal deployment and the stability of continuous monitoring of swallowing disorder risk identification.

[0004] To address the aforementioned technical problems, this application adopts the following technical solution: A machine vision-based method for identifying swallowing disorder risks is provided. This method includes: receiving video acquisition data; loading a target recognition model, where the target recognition model is pre-configured with multiple sets of quantization sub-parameters, each corresponding to a different quantization bit width; acquiring hardware status information of the terminal from which the video acquisition data originates; selecting target sub-parameters for the network blocks of the target recognition model from the multiple sets of quantization sub-parameters based on the hardware status information; extracting features from the video acquisition data using the target recognition model configured with target sub-parameters to obtain laryngeal movement features, facial movement features, and head posture features; generating swallowing stability parameters based on the laryngeal movement features, facial movement features, and head posture features; comparing the swallowing stability parameters with a preset risk threshold; determining the risk level of the swallowing event based on the comparison result; and implementing differentiated message push control based on the risk level.

[0005] Multiple sets of quantization sub-parameters are obtained through elastic quantization training. Elastic quantization training includes: acquiring a calibration dataset containing neck and face video samples labeled with swallowing action stages; for each network block in the target recognition model, acquiring its output features under multiple candidate quantization bit widths; using the output features of the network block under the highest candidate quantization bit width as a benchmark, calculating the mean square error between the output features under other candidate quantization bit widths and the benchmark, and constructing a training loss based on the mean square error; adjusting the quantization sub-parameters of the network block under each candidate quantization bit width based on the training loss through backpropagation until the training loss converges, and storing the converged quantization sub-parameters in association with the corresponding candidate quantization bit widths.

[0006] The process includes: acquiring the hardware status information of the video acquisition data source terminal; selecting target sub-parameters for network blocks of the target recognition model from multiple sets of quantization sub-parameters based on the hardware status information; acquiring the processor model information and the current available memory capacity of the terminal; determining the hardware computing power level identifier based on the processor model information and the current available memory capacity; querying a preset mapping table between the hardware computing power level identifier and the target quantization bit width of each network block; determining the target quantization bit width corresponding to each network block in the target recognition model according to the hardware computing power level identifier; and for each network block, selecting a set of quantization sub-parameters whose quantization bit width is equal to the target quantization bit width from multiple sets of quantization sub-parameters corresponding to that network block as the target sub-parameter.

[0007] The process involves generating swallowing stability parameters based on laryngeal movement features, facial movement features, and head posture features. This includes: segmenting the swallowing cycle based on laryngeal movement features to obtain the start and end frame indices for each swallowing action; calculating the laryngeal elevation velocity deviation, swallowing action asymmetry coefficient, and average head sway amplitude for each swallowing action based on the start and end frame indices; standardizing the laryngeal elevation velocity deviation, swallowing action asymmetry coefficient, and average head sway amplitude; and summing the standardized three indicators by their respective preset weighting coefficients to obtain the swallowing stability parameters.

[0008] The calculation of the laryngeal elevation velocity deviation includes: obtaining vertical displacement data of key laryngeal points within the swallowing cycle based on laryngeal movement characteristics, and calculating the average laryngeal elevation velocity within the cycle; obtaining the pre-stored average laryngeal elevation velocity for reference in healthy individuals and the maximum deviation range of laryngeal elevation velocity; and dividing the difference between the average laryngeal elevation velocity and the average laryngeal elevation velocity for reference in healthy individuals by the maximum deviation range of laryngeal elevation velocity to obtain the laryngeal elevation velocity deviation value.

[0009] The calculation of the swallowing asymmetry coefficient includes: obtaining the coordinate subsequence of the left preset key point and the coordinate subsequence of the right symmetrical key point within the swallowing cycle based on facial movement features; calculating the cumulative displacement amplitude of the left preset key point and the cumulative displacement amplitude of the right symmetrical key point within the cycle; and dividing the absolute value of the difference between the left and right cumulative displacement amplitudes by the sum of the left and right cumulative displacement amplitudes to obtain the swallowing asymmetry coefficient.

[0010] The calculation of the average head sway amplitude includes: obtaining the yaw angle and pitch angle of each frame within the swallowing action cycle based on head posture features; for each frame, calculating the square root of the sum of the square of the change in yaw angle and the square of the change in pitch angle as the instantaneous head sway amplitude of that frame; and calculating the arithmetic mean of all instantaneous head sway amplitudes within the swallowing action cycle to obtain the average head sway amplitude.

[0011] The standardization process is Z-Score standardization, where the mean and standard deviation are obtained and stored in advance through statistical calculations on a healthy population sample dataset.

[0012] The process involves comparing swallowing stability parameters with preset risk thresholds, determining the risk level of a swallowing event based on the comparison results, and implementing differentiated message push control based on the risk level. This includes: comparing the swallowing stability parameters with a first risk threshold and a second risk threshold, where the first risk threshold is less than the second risk threshold; if the swallowing stability parameters are greater than or equal to the first risk threshold and less than the second risk threshold, determining the current swallowing event as a low-risk event, generating a message push identifier for a pending filtering state, and determining whether to update the message push identifier to a direct push state or cancel the push state based on the similarity between the alarm keyframe image of the current swallowing event and the alarm keyframe images of historical swallowing events; if the swallowing stability parameters are greater than or equal to the second risk threshold, determining the current swallowing event as a high-risk event, generating a message push identifier for a direct push state; and executing message push according to the status of the message push identifier.

[0013] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a machine vision-based swallowing disorder risk identification system. This system includes a loading module, a selection module, an extraction module, a generation module, and a control module. The loading module receives video acquisition data and loads a target recognition model. The target recognition model is pre-configured with multiple sets of quantization sub-parameters, each corresponding to a different quantization bit width. The selection module acquires the hardware status information of the video acquisition data source terminal and selects target sub-parameters for the network blocks of the target recognition model from the multiple sets of quantization sub-parameters based on the hardware status information. The extraction module uses the target recognition model configured with target sub-parameters to extract features from the video acquisition data, obtaining laryngeal movement features, facial movement features, and head posture features. The generation module generates swallowing stability parameters based on the laryngeal movement features, facial movement features, and head posture features. The control module compares the swallowing stability parameters with a preset risk threshold, determines the risk level of the swallowing event based on the comparison result, and executes differentiated message push control based on the risk level.

[0014] Unlike existing technologies, the advantages of this application are as follows: By pre-configuring multiple sets of quantization sub-parameters with different quantization bit widths for the target recognition model, the real-time hardware status information of the video acquisition terminal is obtained, and the appropriate target sub-parameters are dynamically selected for each network block of the model accordingly. Based on the configured model, laryngeal movement, facial movement, and head posture features are extracted, and swallowing stability parameters are jointly generated to determine the risk level and perform differentiated push. In this way, based on the actual hardware computing power and operating load of the terminal, the internal calculation accuracy of the model is adaptively adjusted, so that the same swallowing recognition algorithm can achieve the optimal match between feature extraction accuracy and real-time processing efficiency on heterogeneous terminals. It fully considers the individual differences and dynamic fluctuations of terminal hardware capabilities, and significantly improves the reliability of universal deployment and continuous monitoring stability of swallowing disorder risk identification. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating one implementation of the machine vision-based swallowing disorder risk identification method of this application.

[0016] Figure 2 This is a schematic diagram of one embodiment of the machine vision-based swallowing disorder risk identification system of this application. Detailed Implementation

[0017] To make the purpose, technical solution and effects of this application clearer and more explicit, the following describes this application in further detail with reference to the accompanying drawings and embodiments.

[0018] The terms "first" and "second" used herein are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments.

[0019] See Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the machine vision-based swallowing disorder risk identification method of this application. The method includes:

[0020] Step S11: Receive video acquisition data and load the target recognition model. The target recognition model is pre-configured with multiple sets of quantization sub-parameters, and different quantization sub-parameters correspond to different quantization bit widths.

[0021] Specifically, video acquisition data refers to a sequence of multiple frames of images generated by an optical sensor array at continuous time points and encapsulated in a specific encoding format. This data not only contains a pixel brightness matrix reflecting the contours and brightness variations of objects within the shooting scene, but also implicitly contains temporal motion correlation information. In practical deployments, video acquisition data can take one of two forms: Real-time streaming data: H.264 or H.265 format bitstreams pushed by the camera via real-time streaming protocols (such as RTSP or SRT), which the executing entity receives and processes as independent image groups frame by frame. Offline file data: MP4 or AVI container files pre-stored on the terminal device, which the executing entity extracts each decoded frame from the video track by parsing the file index table. For example, in monitoring the eating process of an elderly subject, the video acquisition data could be a sequence of color images approximately 15 seconds long, with a resolution of 1280×720 pixels and a frame rate of 30 frames per second. This sequence completely records the subject's entire movement from raising the water glass to lowering it back down, with each frame accompanied by timestamp metadata generated by the camera hardware.

[0022] An object recognition model refers to a parameterized deep neural network computation graph structure used to predict the two-dimensional or three-dimensional spatial coordinates of specific anatomical landmarks from an input pixel array. In this application, the object recognition model possesses a multi-task parallel topology architecture, specifically including: a backbone feature extraction network: composed of alternating stacked convolutional computation layers, batch normalization layers, and nonlinear activation units, responsible for mapping the original RGB three-channel image into a high-level semantic feature tensor with spatial resolution downsampling characteristics; and a multi-branch task head network: connected in parallel to the end of the backbone network, containing at least a first branch for laryngeal keypoint heatmap regression, a second branch for predicting the coordinate offsets of dense facial keypoints, and a third branch for head Euler angle pose estimation. This object recognition model was pre-trained on a publicly available human pose dataset before deployment and fine-tuned on the swallowing disorder-specific case database constructed in this application. The weight parameters of each operator within the model are organized and stored in floating-point matrix form.

[0023] Considering the differences in inference latency and memory usage of the target recognition model when running on terminal devices with different computing power levels, this embodiment introduces an elastic parameter scheduling mechanism during the model loading stage. Specifically, "pre-configuring multiple sets of quantization sub-parameters" means that during the offline model preparation stage, multiple sets of selectable weight representation schemes are pre-calculated and persistently stored for each network block with independent computational function in the target recognition model (e.g., a residual bottleneck unit or a depthwise separable convolutional layer). Each set of quantization sub-parameters corresponds to a specific weight tensor compressed representation, whose components include: Rounding value: an integer scalar used to indicate the rounding direction adjustment amount used when mapping the original floating-point weights to a low-bit-width discrete grid. The introduction of this rounding value can compensate for the distribution offset caused by direct rounding. Quantization step size: a floating-point scalar representing the floating-point physical interval corresponding to two adjacent discrete values ​​in the low-bit-width integer representation domain.

[0024] Taking a convolutional kernel weight matrix of size [3×3×64×128] in a network block as an example, its pre-configured quantization sub-parameters can include three sets, corresponding to three candidate quantization bit widths. The rounding value in the first set of sub-parameters is an integer tensor of the same size as the weight matrix, and the quantization step size is a floating-point number close to 0.0039; the second set of sub-parameters corresponds to a lower bit width representation, and its quantization step size is increased accordingly, further narrowing the range of values ​​for the rounding value tensor.

[0025] Quantization bit width refers to the number of binary bits required to represent a single weight or activation value. Different quantization bit widths directly determine the numerical resolution and storage overhead of the target recognition model during computation. Typical supported quantization bit widths in common edge computing hardware instruction sets (such as ARM NEON or NPU tensor accelerators) include: 8-bit integer (INT8): Values ​​range from -128 to 127, suitable for achieving twice the inference throughput of single-precision floating-point on general-purpose CPUs or low-power DSPs. 4-bit integer (INT4): Values ​​range from -8 to 7, which can further compress the model size and reduce memory access bandwidth requirements on some accelerator chips with dedicated matrix multiplication units. Mixed precision: Different quantization bit widths are assigned to different network blocks in the model; for example, the error-sensitive first convolutional layer is kept at 8 bits, while the deeply redundant fully connected layers are reduced to 4 bits.

[0026] In step S11 above, the loaded target recognition model does not directly fix all weights to a specific quantization bit width expression. Instead, it reads all the pre-configured quantization sub-parameter sets into memory for later use. This design allows the model to dynamically select a combination of target sub-parameters that is suitable for the current operating environment from each set of sub-parameters based on the query results of the terminal hardware computing power level identifier before executing subsequent inference tasks, thereby achieving a flexible trade-off between recognition accuracy and real-time response.

[0027] In some embodiments, multiple sets of quantization sub-parameters are obtained through elastic quantization training. Elastic quantization training includes: acquiring a calibration dataset containing neck and face video samples labeled with swallowing action stages; for each network block in the target recognition model, acquiring its output features under multiple candidate quantization bit widths; using the output features of the network block under the highest candidate quantization bit width as a benchmark, calculating the mean square error between the output features under other candidate quantization bit widths and the benchmark, and constructing a training loss based on the mean square error; adjusting the quantization sub-parameters of the network block under each candidate quantization bit width through backpropagation based on the training loss until the training loss converges, and storing the converged quantization sub-parameters associated with the corresponding candidate quantization bit widths.

[0028] Specifically, elastic quantization training is performed during the offline preparation phase before deploying the target recognition model. First, a calibration dataset is constructed, containing multiple sequences of continuous video frames from the neck and face labeled with tags for the swallowing preparation phase, oral phase, pharyngeal phase, and the undetermined esophageal phase. Each frame's annotation data includes the ground-valued spatial coordinates of key anatomical landmarks such as the thyroid cartilage notch, hyoid bone projection point, and bilateral corners of the mouth. During training, for each network block within the target recognition model, divided topologically, the forward inference output tensors of that block under multiple candidate quantization bit widths are temporarily stored. Candidate quantization bit widths can include any two or more values ​​from 2 bits, 4 bits, 8 bits, and 16 bits, with the highest candidate quantization bit width serving as the fidelity reference for that network block in the current training epoch. For any non-highest candidate quantization bit width, the element-wise mean squared error between the output tensor corresponding to that bit width and the output tensor corresponding to the highest candidate quantization bit width is calculated, and this mean squared error is used as the accuracy loss term for that network block under that candidate quantization bit width. A joint training loss function for the network block is constructed based on the weighted sum of the accuracy loss term and the original task loss function of the model. During the backpropagation adjustment phase, the quantization sub-parameters corresponding to the network block at the current candidate quantization bit width are updated using gradients based on the joint training loss function. During the update process, the floating-point basic weight parameters corresponding to the highest candidate quantization bit width are kept frozen. After multiple iterations, when the fluctuation range of the joint training loss function value is less than the preset convergence threshold, the parameter adjustment of the network block at the current candidate quantization bit width is stopped. The quantization step size parameter and the quantization zero point parameter obtained at this time are used as a set of converged quantization sub-parameters, and a mapping relationship is established with the numerical identifier of the candidate quantization bit width before being written into the model parameter pool file.

[0029] In some specific embodiments, the target recognition model includes a first convolutional network block for extracting shallow texture features of the throat region. The calibration dataset contains lateral neck video frames taken under two different bolus viscosities: 5 ml of liquid water and 10 ml of semi-solid food. Quantization forward inference with candidate quantization bit widths of 8 bits, 6 bits, and 4 bits is performed on the first convolutional network block to obtain three sets of feature maps with progressively decreasing resolution. Using the feature map corresponding to the 8-bit quantization bit width as a benchmark, the mean squared errors (MSEs) of the 6-bit and 4-bit feature maps relative to this benchmark are calculated. The MSE between the 4-bit feature map and the benchmark is greater than that between the 6-bit feature map and the benchmark. After backpropagation adjustment of the 4-bit and 6-bit quantization sub-parameters, the MSE corresponding to the 6-bit quantization sub-parameter converges to below 0.012, and the MSE corresponding to the 4-bit quantization sub-parameter converges to below 0.038. The two sets of quantization sub-parameters after convergence are associated with the identifiers "6bit" and "4bit" and stored in the model file, respectively, for subsequent inference stages to select one to call based on the real-time available memory capacity of the terminal and the processor model information.

[0030] Step S12: Obtain the hardware status information of the video acquisition data source terminal, and select target sub-parameters for the network block of the target recognition model from multiple sets of quantization sub-parameters based on the hardware status information.

[0031] Specifically, after receiving the start frame of the video acquisition data, the system application programming interface deployed at the operating system kernel layer or application framework layer is invoked to obtain the hardware status information of the terminal from which the video acquisition data originates. This hardware status information characterizes the availability of computing resources of the terminal within the current time window. The obtained hardware status information includes at least a processor model identifier string and a value of currently available physical memory. In some implementations, the hardware status information also includes at least one of the following: the processor's current core operating frequency, a device thermal status level identifier, or a value of the remaining capacity of the graphics processor's video memory.

[0032] In some embodiments, the hardware status information of the video acquisition data source terminal is obtained, and target sub-parameters are selected for the network blocks of the target recognition model from multiple sets of quantization sub-parameters based on the hardware status information. This includes: obtaining the processor model information and the current available memory capacity value of the terminal; determining the hardware computing power level identifier based on the processor model information and the current available memory capacity value; querying a preset mapping table between the hardware computing power level identifier and the target quantization bit width of each network block, and determining the target quantization bit width corresponding to each network block in the target recognition model according to the hardware computing power level identifier; for each network block, selecting a set of quantization sub-parameters with a quantization bit width equal to the target quantization bit width from multiple sets of quantization sub-parameters corresponding to that network block as the target sub-parameter.

[0033] Specifically, the processor model information is first read from the terminal's system configuration file or device registry key, and the current available memory capacity value is extracted from the memory management unit's status statistics structure. The processor model information can be a combination of a chip manufacturer's identifier and a chip generation number, and the current available memory capacity value is stored in megabytes. Next, the processor model information and the current available memory capacity value are input into a pre-built computing power level discrimination function, which outputs a hardware computing power level identifier. This discrimination function can include keyword matching rules for the processor model string and segmented comparison rules for available memory capacity thresholds. For example, when the processor model string contains a specific suffix and the current available memory capacity value exceeds 2048 megabytes, the hardware computing power level identifier is determined to be a first-level value.

[0034] After obtaining the hardware computing power level identifier, the mapping table stored in the model configuration file or resource index area is accessed. This mapping table records the correspondence between different hardware computing power level identifiers and the target quantization bit widths of each network block in the target recognition model in key-value pairs. For the divided throat feature extraction block, facial motion coding block, and head pose regression block, the target quantization bit widths corresponding to each network block under the same hardware computing power level identifier can be the same or different. Based on the hardware computing power level identifier, a key matching query is performed on the mapping table to obtain the target quantization bit width corresponding to each network block in this inference session.

[0035] For any network block, iterate through the pre-loaded sets of quantization sub-parameters associated with that network block, and perform an equivalence comparison between the quantization bit width identifier associated with each set of quantization sub-parameters and the queried target quantization bit width. Select the set of quantization sub-parameters that matches the comparison result as the target sub-parameter of that network block, and replace it in the operator attribute field of the corresponding target recognition model computation graph. Repeat the above selection process until the target sub-parameters of all specified network blocks in the target recognition model have been replaced.

[0036] Step S13: Use the target recognition model configured with target sub-parameters to extract features from the video acquisition data to obtain throat movement features, facial movement features, and head posture features.

[0037] Specifically, after the target sub-parameters corresponding to each network block in the target recognition model are replaced, each consecutive frame of the video acquisition data is sequentially input into the input layer of the target recognition model. For a single frame, the network block configured with the target sub-parameters in the target recognition model performs convolution and pooling operations on the pixel array of the region of interest on the front side of the neck in the image, outputting a heatmap tensor of laryngeal anatomical landmarks. Depth-separable convolution operations are performed on the pixel arrays of the perioral region and bilateral cheek regions in the image, outputting a sequence of facial symmetry keypoint offset vectors. A fully connected transformation is performed on the pixel array of the overall head and face contour region in the image, outputting Euler angle triples representing the spatial orientation of the head. The heatmap tensors of laryngeal anatomical landmarks corresponding to multiple consecutive frames are concatenated in the time dimension, and a spatial maximum indexing operation is performed on each frame's heatmap to obtain the two-dimensional pixel coordinates of the laryngeal keypoints within that frame. The inter-frame sequence of these two-dimensional pixel coordinates of the laryngeal keypoints is output as the laryngeal motion feature. The facial symmetry keypoint offset vector sequences corresponding to multiple consecutive frames are grouped into left-side and right-side preset keypoints. The Euclidean distance changes of both the left-side and right-side preset keypoint sets within each frame are calculated. These Euclidean distance changes within each frame are then used as the facial motion feature output. The yaw, pitch, and roll components of the Euler angle triplet sequences corresponding to multiple consecutive frames are arranged in frame order. The yaw and pitch component sequences are then used as the head pose feature output.

[0038] Step S14: Generate swallowing stability parameters based on laryngeal movement features, facial movement features, and head posture features.

[0039] Specifically, after obtaining laryngeal movement features, facial movement features, and head posture features, these three features are input into the swallowing stability parameter generation module. First, motion start point detection based on amplitude thresholds is performed on the laryngeal movement features. The frame index where the vertical displacement amplitude of the laryngeal key point first exceeds a preset start threshold is determined as the starting frame index of the swallowing action, and the frame index where the vertical displacement amplitude of the laryngeal key point falls back below a preset end threshold is determined as the ending frame index of the swallowing action. Based on the starting and ending frame indices, feature segments within the swallowing action cycle are extracted from the laryngeal movement features, facial movement features, and head posture features, respectively.

[0040] For the extracted feature segments, the laryngeal elevation velocity deviation value is calculated based on the laryngeal movement feature segment, the swallowing asymmetry coefficient is calculated based on the facial movement feature segment, and the mean head sway amplitude is calculated based on the head posture feature segment. These three indicators—laryngeal elevation velocity deviation value, swallowing asymmetry coefficient, and mean head sway amplitude—are input into a standardization function to obtain three standardized indicator values. Each of these standardized indicator values ​​is multiplied by a pre-set weighting coefficient for laryngeal elevation velocity, swallowing asymmetry, and head sway, respectively. The three products are then summed, and the sum is output as the swallowing stability parameter corresponding to that swallowing cycle.

[0041] In some embodiments, swallowing stability parameters are generated based on laryngeal movement features, facial movement features, and head posture features, including: segmenting the swallowing cycle of the laryngeal movement features to obtain the start frame index and end frame index of each swallowing action; for each swallowing action, calculating the laryngeal elevation velocity deviation value, swallowing action asymmetry coefficient, and average head sway amplitude based on the start frame index and end frame index; standardizing the laryngeal elevation velocity deviation value, swallowing action asymmetry coefficient, and average head sway amplitude; and summing the three standardized indicators by their respective preset weight coefficients to obtain the swallowing stability parameters.

[0042] In some embodiments, calculating the laryngeal elevation velocity deviation value includes: obtaining vertical displacement data of key laryngeal points within the swallowing cycle based on laryngeal movement characteristics, and calculating the average laryngeal elevation velocity within the cycle; obtaining pre-stored reference average laryngeal elevation velocity and maximum deviation range value of laryngeal elevation velocity for healthy individuals; and dividing the difference between the average laryngeal elevation velocity and the reference average laryngeal elevation velocity for healthy individuals by the maximum deviation range value of laryngeal elevation velocity to obtain the laryngeal elevation velocity deviation value.

[0043] Specifically, when segmenting the swallowing cycle based on laryngeal motion features, the inter-frame sequence of two-dimensional pixel coordinates of key laryngeal points is converted into a vertical displacement amplitude sequence. A sliding window smoothing filter is applied to this vertical displacement amplitude sequence to eliminate high-frequency jitter noise. In the smoothed vertical displacement amplitude sequence, the first sampling point index where the amplitude rises continuously from the baseline level with an upward slope exceeding a preset slope threshold is marked as the start frame index. The first sampling point index where the amplitude falls back from the peak level to within a preset tolerance range of the baseline level is marked as the end frame index. The frame interval between the start and end frame indices corresponds to one complete swallowing action.

[0044] In some embodiments, calculating the swallowing asymmetry coefficient includes: obtaining a left-side preset key point coordinate subsequence and a right-side symmetrical key point coordinate subsequence within the swallowing action cycle based on facial motion features; calculating the cumulative displacement amplitude of the left-side preset key point within the cycle and the cumulative displacement amplitude of the right-side symmetrical key point within the cycle; and dividing the absolute value of the difference between the left-side cumulative displacement amplitude and the right-side cumulative displacement amplitude by the sum of the left-side cumulative displacement amplitude and the right-side cumulative displacement amplitude to obtain the swallowing asymmetry coefficient.

[0045] In some embodiments, calculating the average head sway amplitude includes: obtaining the yaw angle and pitch angle of each frame within the swallowing action cycle based on head posture features; for each frame, calculating the square root of the sum of the square of the change in yaw angle and the square of the change in pitch angle as the instantaneous head sway amplitude of that frame; and calculating the arithmetic mean of all instantaneous head sway amplitudes within the swallowing action cycle to obtain the average head sway amplitude.

[0046] Specifically, for each swallowing action, vertical displacement data of key laryngeal points within the interval from the start frame index to the end frame index is extracted from the laryngeal motion features. The first derivative sequence of the vertical displacement data of key laryngeal points within this interval is calculated and the arithmetic mean is taken to obtain the average laryngeal elevation velocity. From facial motion features, sequences of Euclidean distance changes of preset key points on the left and symmetrical key points on the right within the same frame interval are extracted, and the cumulative displacement amplitudes on the left and right are calculated respectively. From head posture features, sequences of yaw angle components and pitch angle components within the same frame interval are extracted, and the yaw angle and pitch angle changes are calculated for each frame. The laryngeal elevation velocity deviation value is calculated based on the average laryngeal elevation velocity, the reference average laryngeal elevation velocity for healthy individuals, and the maximum deviation range of laryngeal elevation velocity. The swallowing action asymmetry coefficient is calculated based on the cumulative displacement amplitudes on the left and right. The average head sway amplitude is calculated based on the yaw angle and pitch angle changes for each frame.

[0047] The calculated deviation values ​​of laryngeal elevation velocity, swallowing asymmetry coefficient, and mean head tremor amplitude are subtracted from the statistical mean of the corresponding indicators in the healthy population, and then divided by the statistical standard deviation of the corresponding indicators in the healthy population to complete Z-Score standardization. The standardized laryngeal elevation velocity deviation value is multiplied by the laryngeal elevation velocity weighting coefficient, the standardized swallowing asymmetry coefficient is multiplied by the swallowing asymmetry weighting coefficient, and the standardized mean head tremor amplitude is multiplied by the head tremor weighting coefficient. The sum of these three products yields the swallowing stability parameter corresponding to the swallowing action.

[0048] In some embodiments, the standardization process is Z-Score standardization, where the mean and standard deviation used in Z-Score standardization are obtained and stored in advance through statistical calculations on a healthy population sample dataset.

[0049] Specifically, the Z-Score normalization process is pre-configured before calculating the swallowing stability parameters for the current swallowing action. First, a healthy population sample dataset is constructed, containing video recordings of multiple individuals with no history of swallowing disorders and assessed as normal by a clinical swallowing function assessment scale, during swallowing actions. For each video recording sample, samples of laryngeal elevation velocity deviation, swallowing action asymmetry coefficients, and mean head movement amplitude are extracted according to the method provided in the foregoing embodiments of this application, forming sets of laryngeal elevation velocity deviation, swallowing action asymmetry coefficients, and mean head movement amplitude, respectively.

[0050] The arithmetic mean of the laryngeal elevation velocity deviation values ​​is calculated by taking the arithmetic mean of all sample values ​​in the sample set. The standard deviation of the laryngeal elevation velocity deviation is then calculated. Similarly, the arithmetic mean of the swallowing asymmetry coefficient is calculated by taking the arithmetic mean of all sample values ​​in the sample set. The standard deviation of the swallowing asymmetry coefficient is then calculated. Finally, the statistical mean of the head movement amplitude is calculated by taking the arithmetic mean of all sample values ​​in the sample set. The standard deviation of the head movement amplitude is then calculated. These three sets of means and standard deviations are stored as standardization parameters in the system configuration file or model resource index area for subsequent standardization transformations of the laryngeal elevation velocity deviation values, swallowing asymmetry coefficients, and head movement amplitude values ​​calculated for each swallowing action.

[0051] In some application scenarios, three consecutive swallowing action cycles were detected from video capture data. The start and end frame indices of each swallowing action cycle were frames 145-178, 312-348, and 476-509, respectively. For the first swallowing action cycle, the average laryngeal elevation speed was calculated to be 28.5 pixels per second, while the average laryngeal elevation speed for healthy individuals was 22.0 pixels per second. The maximum deviation range of laryngeal elevation speed was 8.0 pixels per second, resulting in a laryngeal elevation speed deviation value of 0.8125. The cumulative displacement amplitude on the left side was 45.2 pixels, and the cumulative displacement amplitude on the right side was 38.6 pixels, resulting in a swallowing action asymmetry coefficient of 0.0788. The calculated average head sway amplitude for the yaw and pitch angle changes in each frame was 2.15 degrees. The deviation value of laryngeal elevation velocity after Z-score standardization is 0.95, the swallowing asymmetry coefficient after standardization is 0.52, and the mean head sway amplitude after standardization is 0.68. The weighting coefficients for laryngeal elevation velocity, swallowing asymmetry, and head sway are preset to 0.4, 0.35, and 0.25, respectively. The weighted sum of these three standardized indicators yields a swallowing stability parameter value of 0.732 for this swallowing cycle.

[0052] Step S15: Compare the swallowing stability parameters with the preset risk threshold, determine the risk level of the swallowing event based on the comparison results, and perform differentiated message push control based on the risk level.

[0053] In some embodiments, a swallowing stability parameter is compared with a preset risk threshold, the risk level of the swallowing event is determined based on the comparison result, and differentiated message push control is performed based on the risk level, including: comparing the swallowing stability parameter with a first risk threshold and a second risk threshold, wherein the first risk threshold is less than the second risk threshold; if the swallowing stability parameter is greater than or equal to the first risk threshold and less than the second risk threshold, the current swallowing event is determined to be a low-risk event, a message push identifier for a pending filtering state is generated, and based on the similarity between the alarm keyframe image of the current swallowing event and the alarm keyframe images of historical swallowing events, it is determined whether to update the message push identifier to a direct push state or cancel the push state; if the swallowing stability parameter is greater than or equal to the second risk threshold, the current swallowing event is determined to be a high-risk event, a message push identifier for a direct push state is generated; and message push is performed according to the state of the message push identifier.

[0054] Specifically, after generating the swallowing stability parameters corresponding to the current swallowing cycle, these parameters are compared with preset risk thresholds stored in the system configuration parameter area. The preset risk thresholds include a first risk threshold and a second risk threshold, where the value of the first risk threshold is less than the value of the second risk threshold. The risk level of the current swallowing event is determined based on the relationship between the swallowing stability parameters and the values ​​of the first and second risk thresholds.

[0055] After determining the risk level, the message push control process is triggered. If the swallowing stability parameter is less than the first risk threshold, a message push flag for cancellation is generated, and no alarm message is pushed for this swallowing event. If the swallowing stability parameter is greater than or equal to the first risk threshold and less than the second risk threshold, the current swallowing event is determined to be a low-risk event, a message push flag for pending filtering is generated, and the alarm keyframe similarity discrimination routine is called. Based on the discrimination result, the message push flag is updated to either a direct push status or a canceled push status. If the swallowing stability parameter is greater than or equal to the second risk threshold, the current swallowing event is determined to be a high-risk event, and a message push flag for direct push status is generated. The message push service process polls and checks the status value of the message push flag. For message push flags in the direct push status, an alarm notification message body is immediately constructed and sent to the preset target user terminal or monitoring center platform. For message push flags in the pending filtering status, they are temporarily stored in the message waiting queue until the similarity discrimination result is returned. For message push flags in the canceled push status, the alarm data buffer corresponding to this swallowing event is released.

[0056] In some application scenarios, users monitor their daily eating habits using a tablet monitoring terminal equipped with the method described in this application during dinner time. During a single meal, the system detects a swallowing action cycle and calculates the swallowing stability parameter corresponding to that cycle as 0.85. The first risk threshold is preset to 0.62, and the second risk threshold is preset to 0.91. If the swallowing stability parameter 0.85 falls within the range of being greater than or equal to the first risk threshold and less than the second risk threshold, the system determines the current swallowing event as a low-risk event and generates a message push identifier for the pending filtering state. The system extracts the frame with the largest peak displacement of the laryngeal key points from frames 156 to 189 of the swallowing action cycle as the alarm keyframe image. The system calculates the structural similarity index against historical alarm keyframe images stored in the circular buffer, with the maximum structural similarity index being 0.41. The preset similarity threshold is 0.70. Since the current maximum structural similarity index is lower than the preset similarity threshold, the system updates the message push identifier for the pending filtering state to the direct push state. Once the message push service detects the direct push status, it immediately constructs an alarm notification message body containing an alarm keyframe image, a swallowing stability parameter value of 0.85, a low-risk level identifier, and an event timestamp, and sends it to the monitoring center platform for follow-up processing by on-duty medical staff.

[0057] The above scheme pre-configures multiple sets of quantization sub-parameters with different quantization bit widths for the target recognition model, obtains real-time hardware status information of the video acquisition terminal, and dynamically selects suitable target sub-parameters for each network block of the model accordingly. Based on the configured model, it extracts laryngeal movement, facial movement, and head posture features, jointly generates swallowing stability parameters to determine the risk level, and performs differentiated push. In this way, based on the actual hardware computing power and operating load of the terminal, the internal calculation accuracy of the model is adaptively adjusted, so that the same swallowing recognition algorithm can achieve the optimal match between feature extraction accuracy and real-time processing efficiency on heterogeneous terminals. It fully considers the individual differences and dynamic fluctuations of terminal hardware capabilities, and significantly improves the reliability of universal deployment and continuous monitoring stability of swallowing disorder risk identification.

[0058] Please see Figure 2 , Figure 2 This is a schematic diagram of one embodiment of the machine vision-based swallowing disorder risk identification system of this application.

[0059] The machine vision-based swallowing disorder risk identification system 20 includes: a loading module 21, a selection module 22, an extraction module 23, a generation module 24, and a control module 25. The loading module 21 receives video acquisition data and loads a target recognition model. The target recognition model is pre-configured with multiple sets of quantization sub-parameters, each corresponding to a different quantization bit width. The selection module 22 acquires the hardware status information of the video acquisition data source terminal and selects target sub-parameters for the network blocks of the target recognition model from the multiple sets of quantization sub-parameters based on the hardware status information. The extraction module 23 uses the target recognition model configured with target sub-parameters to extract features from the video acquisition data, obtaining laryngeal movement features, facial movement features, and head posture features. The generation module 24 generates swallowing stability parameters based on the laryngeal movement features, facial movement features, and head posture features. The control module 25 compares the swallowing stability parameters with a preset risk threshold, determines the risk level of the swallowing event based on the comparison result, and executes differentiated message push control based on the risk level.

[0060] In some embodiments, multiple sets of quantization sub-parameters are obtained through elastic quantization training. Elastic quantization training includes: acquiring a calibration dataset containing neck and face video samples labeled with swallowing action stages; for each network block in the target recognition model, acquiring its output features under multiple candidate quantization bit widths; using the output features of the network block under the highest candidate quantization bit width as a benchmark, calculating the mean square error between the output features under other candidate quantization bit widths and the benchmark, and constructing a training loss based on the mean square error; adjusting the quantization sub-parameters of the network block under each candidate quantization bit width through backpropagation based on the training loss until the training loss converges, and storing the converged quantization sub-parameters associated with the corresponding candidate quantization bit widths.

[0061] In some embodiments, the selection module 22 obtains the hardware status information of the video acquisition data source terminal, and selects target sub-parameters for the network blocks of the target recognition model from multiple sets of quantization sub-parameters based on the hardware status information, including: obtaining the processor model information and the current available memory capacity value of the terminal; determining the hardware computing power level identifier based on the processor model information and the current available memory capacity value; querying the preset mapping relationship table between the hardware computing power level identifier and the target quantization bit width of each network block, and determining the target quantization bit width corresponding to each network block in the target recognition model according to the hardware computing power level identifier; for each network block, selecting a set of quantization sub-parameters whose quantization bit width is equal to the target quantization bit width from multiple sets of quantization sub-parameters corresponding to the network block as the target sub-parameters.

[0062] The generation module 24 generates swallowing stability parameters based on laryngeal movement features, facial movement features, and head posture features. This includes: segmenting the swallowing cycle of the laryngeal movement features to obtain the start frame index and end frame index of each swallowing action; for each swallowing action, calculating the laryngeal elevation speed deviation value, swallowing action asymmetry coefficient, and average head sway amplitude based on the start frame index and end frame index; standardizing the laryngeal elevation speed deviation value, swallowing action asymmetry coefficient, and average head sway amplitude; and summing the three standardized indicators by their respective preset weight coefficients to obtain the swallowing stability parameters.

[0063] In some embodiments, the generation module 24 calculates the laryngeal elevation velocity deviation value, including: obtaining vertical displacement data of key laryngeal points within the swallowing cycle based on laryngeal movement characteristics, calculating the average laryngeal elevation velocity within the cycle; obtaining the pre-stored average laryngeal elevation velocity for healthy individuals and the maximum deviation range of laryngeal elevation velocity; dividing the difference between the average laryngeal elevation velocity and the average laryngeal elevation velocity for healthy individuals by the maximum deviation range of laryngeal elevation velocity to obtain the laryngeal elevation velocity deviation value.

[0064] The generation module 24 calculates the swallowing action asymmetry coefficient, including: obtaining the left preset key point coordinate subsequence and the right symmetrical key point coordinate subsequence within the swallowing action cycle based on facial motion features; calculating the cumulative displacement amplitude of the left preset key point within the cycle and the cumulative displacement amplitude of the right symmetrical key point within the cycle; and dividing the absolute value of the difference between the left cumulative displacement amplitude and the right cumulative displacement amplitude by the sum of the left cumulative displacement amplitude and the right cumulative displacement amplitude to obtain the swallowing action asymmetry coefficient.

[0065] In some embodiments, the generation module 24 calculates the average head sway amplitude, including: obtaining the yaw angle and pitch angle of each frame in the swallowing action cycle based on head posture features; for each frame, calculating the square root of the sum of the square of the change in yaw angle and the square of the change in pitch angle as the instantaneous head sway amplitude of that frame; and calculating the arithmetic mean of all instantaneous head sway amplitudes in the swallowing action cycle to obtain the average head sway amplitude.

[0066] In some embodiments, the normalization process of the generation module 24 is Z-Score normalization, and the mean and standard deviation used in the Z-Score normalization are obtained and stored in advance by statistical calculation of a healthy population sample dataset.

[0067] In some embodiments, the control module 25 compares the swallowing stability parameter with a preset risk threshold, determines the risk level of the swallowing event based on the comparison result, and performs differentiated message push control based on the risk level, including: comparing the swallowing stability parameter with a first risk threshold and a second risk threshold, wherein the first risk threshold is less than the second risk threshold; if the swallowing stability parameter is greater than or equal to the first risk threshold and less than the second risk threshold, determining that the current swallowing event is a low-risk event, generating a message push identifier for a filtering state, and determining whether to update the message push identifier to a direct push state or cancel the push state based on the similarity between the alarm keyframe image of the current swallowing event and the alarm keyframe images of historical swallowing events; if the swallowing stability parameter is greater than or equal to the second risk threshold, determining that the current swallowing event is a high-risk event, generating a message push identifier for a direct push state; and performing message push according to the state of the message push identifier.

[0068] The above scheme pre-configures multiple sets of quantization sub-parameters with different quantization bit widths for the target recognition model, obtains real-time hardware status information of the video acquisition terminal, and dynamically selects suitable target sub-parameters for each network block of the model accordingly. Based on the configured model, it extracts laryngeal movement, facial movement, and head posture features, jointly generates swallowing stability parameters to determine the risk level, and performs differentiated push. In this way, based on the actual hardware computing power and operating load of the terminal, the internal calculation accuracy of the model is adaptively adjusted, so that the same swallowing recognition algorithm can achieve the optimal match between feature extraction accuracy and real-time processing efficiency on heterogeneous terminals. It fully considers the individual differences and dynamic fluctuations of terminal hardware capabilities, and significantly improves the reliability of universal deployment and continuous monitoring stability of swallowing disorder risk identification.

[0069] The above are merely embodiments of this application and do not limit the scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the protection scope of this application.

Claims

1. A method for identifying swallowing disorder risk based on machine vision, characterized in that, The method includes: receiving video acquisition data and loading a target recognition model, wherein the target recognition model is pre-configured with multiple sets of quantization sub-parameters, and different quantization sub-parameters correspond to different quantization bit widths; Obtain the hardware status information of the video acquisition data source terminal, and select target sub-parameters for the network block of the target recognition model from the multiple sets of quantization sub-parameters based on the hardware status information; The target recognition model configured with the target sub-parameters is used to extract features from the video acquisition data to obtain throat movement features, facial movement features, and head posture features. Swallowing stability parameters are generated based on the laryngeal movement features, facial movement features, and head posture features. The swallowing stability parameter is compared with a preset risk threshold, the risk level of the swallowing event is determined based on the comparison result, and differentiated message push control is performed based on the risk level.

2. The method according to claim 1, characterized in that, The multiple sets of quantization sub-parameters are obtained through elastic quantization training, which includes: Obtain a calibration dataset containing neck and face video samples labeled with swallowing phases; For each network block in the target recognition model, obtain its output features under multiple candidate quantization bit widths; Using the output features of the network block at the highest candidate quantization bit width as a benchmark, calculate the mean square error between the output features of other candidate quantization bit widths and the benchmark, and construct the training loss based on the mean square error; Based on the training loss, the quantization sub-parameters of the network block under each candidate quantization bit width are backpropagated and adjusted until the training loss converges. The converged quantization sub-parameters are then associated with and stored with the corresponding candidate quantization bit width.

3. The method according to claim 1, characterized in that, The step of obtaining the hardware status information of the video acquisition data source terminal, and selecting target sub-parameters for the network blocks of the target recognition model from the multiple sets of quantization sub-parameters based on the hardware status information, includes: Obtain the processor model information and current available memory capacity of the terminal; The hardware computing power level identifier is determined based on the processor model information and the current available memory capacity value; Query the preset mapping table between hardware computing power level identifiers and target quantization bit widths of each network block, and determine the target quantization bit widths corresponding to each network block in the target recognition model based on the hardware computing power level identifiers; For each network block, select a set of quantization sub-parameters from the multiple sets of quantization sub-parameters corresponding to that network block, such that the quantization bit width is equal to the target quantization bit width, as the target sub-parameter.

4. The method according to claim 1, characterized in that, The process of generating swallowing stability parameters based on the laryngeal movement features, facial movement features, and head posture features includes: The larynx motion features are segmented into swallowing cycles to obtain the start frame index and end frame index of each swallowing action; For each swallowing action, based on the start frame index and the end frame index, the deviation value of laryngeal elevation speed, the asymmetry coefficient of swallowing action, and the average value of head shaking amplitude are calculated respectively. The deviation values ​​of the laryngeal elevation speed, the asymmetry coefficient of the swallowing action, and the average value of the head shaking amplitude are standardized. The swallowing stability parameter is obtained by multiplying the three standardized indicators by their respective preset weighting coefficients and then summing them.

5. The method according to claim 4, characterized in that, The calculation of the throat elevation velocity deviation includes: Based on the laryngeal movement characteristics, obtain the vertical displacement data of key laryngeal points within the swallowing cycle, and calculate the average laryngeal elevation velocity within the cycle. Obtain the pre-stored average laryngeal elevation velocity and the maximum deviation range of laryngeal elevation velocity for reference in healthy individuals; The difference between the average laryngeal elevation speed and the average laryngeal elevation speed of the healthy population is divided by the maximum deviation range of the laryngeal elevation speed to obtain the laryngeal elevation speed deviation value.

6. The method according to claim 4, characterized in that, The calculation of the swallowing asymmetry coefficient includes: Based on the facial movement features, obtain the left-side preset key point coordinate subsequence and the right-side symmetrical key point coordinate subsequence within the swallowing action cycle; Calculate the cumulative displacement amplitude of the preset key point on the left side during the cycle, and the cumulative displacement amplitude of the symmetrical key point on the right side during the cycle; The asymmetry coefficient of the swallowing action is obtained by dividing the absolute value of the difference between the cumulative displacement amplitude on the left and the cumulative displacement amplitude on the right by the sum of the cumulative displacement amplitudes on the left and the right.

7. The method according to claim 4, characterized in that, The calculation of the average head movement amplitude includes: Based on the head posture features, the yaw angle and pitch angle of each frame within the swallowing action cycle are obtained; For each frame, calculate the square root of the sum of the squares of the yaw angle change and the squares of the pitch angle change, and use it as the instantaneous head sway amplitude for that frame. The arithmetic mean of the amplitude of head movement during all instantaneous moments within the swallowing cycle is calculated to obtain the mean amplitude of head movement.

8. The method according to claim 4, characterized in that, The standardization process is Z-Score standardization, and the mean and standard deviation used in the Z-Score standardization process are obtained and stored in advance through statistical calculations on a healthy population sample dataset.

9. The method according to claim 1, characterized in that, The step of comparing the swallowing stability parameter with a preset risk threshold, determining the risk level of the swallowing event based on the comparison result, and performing differentiated message push control based on the risk level includes: The swallowing stability parameter is compared with a first risk threshold and a second risk threshold, wherein the first risk threshold is less than the second risk threshold; If the swallowing stability parameter is greater than or equal to the first risk threshold and less than the second risk threshold, the current swallowing event is determined to be a low-risk event, a message push identifier for the filtering state is generated, and based on the similarity between the alarm keyframe image of the current swallowing event and the alarm keyframe image of the historical swallowing events, it is determined whether to update the message push identifier to the direct push state or cancel the push state. If the swallowing stability parameter is greater than or equal to the second risk threshold, the current swallowing event is determined to be a high-risk event, and a message push identifier for direct push status is generated. Message push is executed based on the status of the message push identifier.

10. A machine vision-based swallowing disorder risk identification system, characterized in that, include: The loading module is used to receive video acquisition data and load the target recognition model. The target recognition model is pre-configured with multiple sets of quantization sub-parameters, and different quantization sub-parameters correspond to different quantization bit widths. The selection module is used to obtain the hardware status information of the video acquisition data source terminal, and select target sub-parameters for the network block of the target recognition model from the multiple sets of quantization sub-parameters based on the hardware status information; The extraction module is used to extract features from the video acquisition data using a target recognition model configured with the target sub-parameters, to obtain throat movement features, facial movement features, and head posture features; The generation module is used to generate swallowing stability parameters based on the laryngeal movement features, the facial movement features, and the head posture features; The control module is used to compare the swallowing stability parameters with a preset risk threshold, determine the risk level of the swallowing event based on the comparison result, and perform differentiated message push control based on the risk level.

Images