Pharyngeal time estimation method based on swallowing contrast video

Abstract

The invention provides a pharyngeal time estimation method based on a swallowing contrast video. According to the method, a fuzzy theory and a space-time modeling thought are combined, continuous modeling is carried out on a swallowing process by constructing a membership function of a pharyngeal period starting frame and a pharyngeal period ending frame, and fuzziness of a pharyngeal period boundary and continuity of time sequence changes are explicitly embodied. According to the model, a three-dimensional convolutional neural network (3D-CNN) is adopted to extract space and time features of a video, and a space-time attention mechanism is introduced to guide the model to pay attention to dynamic changes of a contrast agent in a swallowing process, so that correlation of feature representation is enhanced. Finally, the model outputs the membership degree value of the pharyngeal period starting frame and the pharyngeal period ending frame of each frame, a continuous time curve is formed, and accurate estimation of the pharyngeal period starting and ending moments and the duration time is achieved.

Description

Technical Field

[0001] This invention relates to the technical field of estimating pharyngeal time through swallowing contrast imaging video, and more particularly to a method for estimating pharyngeal time based on a combination of deep learning and fuzzy theory.

[0002] The entire swallowing process can be divided into the oral phase, pharyngeal phase, and esophageal phase, with the pharyngeal phase being the most crucial. Accurate assessment of the pharyngeal phase time, and the output of quantitative data, can help doctors determine the level of swallowing disorders or further observe whether other diseases are present, such as aspiration or leakage.

[0003] Several other studies have been conducted on determining the pharyngeal phase during swallowing. In reference 1, B Riebold et al. explored using electromyography (EMG) and bioelectrical impedance analysis (BIA) combined with machine learning to segment the swallowing initiation process. During oral swallowing, the movements of the tongue and jaw cause EMG activity and fluctuations in bioelectrical impedance. They focused on this characteristic, placing electrodes on the larynx and chest to collect EMG and BIA data, then combining this with a random forest classifier to distinguish between swallowing and non-swallowing processes. Experimental results demonstrated that the combination of EMG and bioelectrical impedance analysis significantly improved the recognition rate of swallowing initiation, but this method requires standardized values.

[0004] Swallowing contrast imaging videos are an important tool for assisting in the assessment of swallowing disorders. They are the most intuitive and considered the gold standard for clinical assessment of the severity of swallowing disorders. Swallowing contrast imaging videos refer to special video data obtained through X-ray fluoroscopy of the mouth, pharynx, and esophagus during the swallowing of contrast agents. Currently, the analysis of swallowing contrast imaging videos in clinical practice mainly relies on professional physicians analyzing the video frame by frame. Due to the ambiguous boundaries of the pharyngeal phase, this work is very challenging, requiring a high level of expertise from medical staff and is extremely time-consuming. Therefore, proposing an automated detection method based on swallowing contrast imaging videos to accurately output the quantitative time of the pharyngeal phase is of great significance. Background Technology

[0005] For swallowing angiography videos, Andrea Bandini in Reference 2 used a method based on a two-dimensional convolutional neural network and classification tasks to estimate the duration of the pharyngeal phase. The model structure is simple and training efficiency is high. They divided all frames of the swallowing angiography into two categories: pharyngeal phase frames and non-pharyngeal phase frames. During model estimation, each frame was independently binary classified, outputting a binary classification result indicating whether it belonged to the pharyngeal phase or not. Subsequently, they used a "three consecutive frames" rule to determine the final classification result for each frame. The advantage of this method is that when labeling samples, only the start and end frames of the pharyngeal phase need to be identified to provide binary labels for all frames, resulting in high labeling efficiency. However, these labels do not reflect the temporal continuity of the swallowing angiography video, and the independent classification of each frame makes it difficult to capture the dynamic correlation between frames. Furthermore, the discretized output of the binary classification may lead to a "step-like" change at the pharyngeal phase boundary, lacking a smooth transition and failing to fully reflect the continuous changes in image features during swallowing.

[0006] Some studies have employed three-dimensional convolutional neural networks (3D convolutional neural networks) to model swallowing contrast images. This type of method, by simultaneously introducing spatial and temporal dimensions into the convolutional kernel, can capture the dynamic changes in bolus movement within the contrast image to some extent, thereby improving the model's ability to distinguish different stages of swallowing. Compared to two-dimensional convolution, 3D convolution is better at capturing dynamic information within a time window and has certain advantages in recognizing various events during swallowing. However, most studies using this method still focus on frame binary classification as the core task, failing to explicitly model the temporal continuity of the swallowing action. The output is discretized and cannot smoothly reflect the dynamic changes of the swallowing action. For example, reference 3 used a 3D convolutional neural network to process swallowing contrast images and estimate the time of the pharyngeal reflex; they still used classification methods.

[0007] The views expressed in this application are: (1) Three-dimensional convolutional neural networks can model simultaneously in time and space dimensions, which is an effective means of processing swallowing contrast imaging videos; (2) The swallowing process is a continuously changing process, and the position of the contrast agent in the human body changes continuously over time, with the boundaries of each time period being fuzzy; (3) For the task of estimating the pharyngeal period time, the start and end frames of the learning phase should be expressed using fuzzy sets, and the corresponding model output is naturally also a fuzzy set, which can be further clarified to achieve estimation; (4) By splitting the task of estimating the pharyngeal period time into two tasks, namely determining the start frame and the end frame of the pharyngeal period, the overall interpretability of the system is stronger.

[0008] References:

[0009] [1] Riebold, Benjamin, Rainer O. Seidl, and Thomas Schauer. "Electromyography-and Bioimpedance-Based Detection of Swallow Onset for the Control of Dysphagia Treatment." Sensors 24.20 (2024): 6525.

[0010] [2] Bandini, Andrea, Sana Smaoui, and Catriona M. Steele. "Automatedpharyngeal phase detection and bolus localization in videofluoroscopicswallowing study: Killing two birds with one stone?." Computer methods and programs in biomedicine 225 (2022): 107058.

[0011] [3] Lee, Jong Taek, et al. "Machine learning analysis to automatically measure response time of pharyngeal swallowing reflex in videofluoroscopicswallowing study." Scientific Reports 10.1 (2020): 14735. Summary of the Invention

[0012] Purpose of the invention

[0013] This invention proposes a method for estimating pharyngeal phase time based on swallowing contrast imaging video. This method combines fuzzy theory and spatiotemporal modeling, continuously modeling the swallowing process by constructing membership functions for the start and end frames of the pharyngeal phase, thus explicitly reflecting the fuzziness of the pharyngeal phase boundary and the continuity of temporal changes. The model uses a three-dimensional convolutional neural network (3D-CNN) to extract the spatial and temporal features of the video and introduces a spatiotemporal attention mechanism to guide the model to focus on the dynamic changes of the contrast agent during swallowing, thereby enhancing the relevance of the feature representation. Finally, the model outputs the membership values ​​of the start and end frames of the pharyngeal phase for each frame, forming a continuous time curve, achieving accurate estimation of the start and end times and duration of the pharyngeal phase.

[0014] Technical solution

[0015] A method for estimating pharyngeal period time based on swallowing contrast imaging video is characterized by the following steps for a swallowing contrast imaging video data: (1)(a) A sliding window of length L is used to slide on the swallowing contrast imaging video, with the window moving by a step size S each time, generating several video slices; (b) For each video slice, it is first grayscaled, and the grayscale slice is input into a trained pharyngeal period start point estimation model, outputting the preliminary membership value of the pharyngeal period start point of each frame in the slice; (c) The preliminary membership values ​​of the pharyngeal period start point of each frame in the entire video are averaged in each slice covering it, i.e., the overlap average, to obtain the final membership value of the pharyngeal period start point of the frame, thereby obtaining the pharyngeal period start point membership curve of the complete video domain; (d) The pharyngeal period start point membership curve of the complete video domain is sharpened to obtain a sharp estimate of the pharyngeal period start time; (2)(a) Using a sliding window of length L, slide it across the swallowing contrast video. The window moves by step S each time to generate several video slices. (b) For each video slice, first convert it to grayscale, and input the grayscale slice into the trained pharyngeal end point estimation model. Output the preliminary membership value of the pharyngeal end point of each frame in the slice. (c) Average the preliminary membership values ​​of the pharyngeal end point of each frame in the entire video in the slices covering it, i.e., the overlap average, to obtain the final pharyngeal end point membership value of the frame image, thus obtaining the pharyngeal end point membership curve of the complete video domain. (d) Combine with prior knowledge, sharpen the pharyngeal end point membership curve of the complete video domain to obtain a sharp estimate of the pharyngeal end time. (3) Subtract the sharp estimate of the pharyngeal start time from the sharp estimate of the pharyngeal end time. If divided by the video frame rate in units of frames, the pharyngeal time estimate is obtained. The overall flowchart is as follows. Figure 1 As shown, the video slices are generated by sliding. Figure 2 As shown.

[0016] The above-described method for estimating pharyngeal duration based on swallowing contrast imaging video, specifically the preliminary membership value estimation model for the pharyngeal duration start point, is characterized by the following: its input is a video slice with dimensions L×H×W, where W is the width of the image frame, H is the height of the image frame, and L is the number of image frames in the slice; the output is the preliminary membership value estimation of the pharyngeal duration start point for each image frame in the slice, forming a membership curve. The preliminary membership value estimation model for the pharyngeal duration start point includes, in sequence, a Conv3D convolutional layer with a kernel size of 3×7×7 (output size C×L×H / 2×W / 2, where C is the number of channels, the same below), a BatchNorm3D batch normalization layer (output size C×L×H / 2×W / 2), and a ReL... The system consists of a U-activated layer (output size C×L×H / 2×W / 2), a MaxPool3D layer with a 1×3×3 kernel (output size C×L×H / 4×W / 4), four residual layers (output size 8C×L×H / 32×W / 32), a 3D average pooling layer (AdaptiveAvgPool) (output size 8C×L×1×1), a Dropout layer (output size 8C×L), and a one-dimensional convolutional layer Conv1D (output size lxL) that weights and sums each channel. The four residual layers are primarily composed of 3D convolutional units. Each residual layer (input size C...)... in ×L in ×H in ×W out Each branch has two branches: the main branch (output size is C). out ×L out ×H out ×W out ) and residual branch (output size C) out ×L out ×H out ×W out The main branch sequentially includes a 3×3×3 convolutional layer (Conv3D), batch normalization (BatchNorm3D), ReLU activation function, and a 3×3×3 convolutional layer (Conv3D) and batch normalization (BatchNorm3D). The residual branch includes an optional 1×1×1 3D convolutional layer (Optional Conv3D). in ×L in ×H in ×W out The intermediate output size C obtained through the main branch out ×L out ×H out ×W outWhen there is a mismatch, this optional convolution adjusts the dimensions, adds the output of the main branch and the output of the residual branch, and then performs ReLU activation to obtain the output of the residual structure (output size C). out ×L out ×H out ×W out The preliminary membership estimation model structure for the pharyngeal phase initiation point is as follows: Figure 3 As shown, the detailed structure diagram of the residual structure is as follows: Figure 4 As shown.

[0017] The above-described method for estimating pharyngeal duration based on swallowing contrast imaging video, and its preliminary membership value estimation model for the pharyngeal duration termination point, are characterized by the following: The input is a video slice with dimensions L×H×W, where W is the width of the video image frame, H is the height of the video image frame, and L is the number of image frames in the slice. The output is the preliminary membership value estimation of the pharyngeal duration termination point for each image frame in the slice, forming a membership curve. The preliminary membership value estimation model for the pharyngeal duration termination point includes, in sequence: a Conv3D convolutional layer with a kernel size of 3×7×7 (output size C×L×H / 2×W / 2, where C is the number of channels, the same below), a BatchNorm3D batch normalization layer (output size C×L×H / 2×W / 2), ReLU activation (output size C×L×H / 2×W / 2), and a MaxPool3D max pooling layer with a kernel size of 1×3×3 (output size...). The input structure consists of four residual layers (8C×L×H / 32×W / 32), a spatial attention module (8C×L×H / 32×W / 32), a temporal attention module (8C×L×H / 32×W / 32), a 3D average pooling layer (AdaptiveAvgPool, 8C×L×1×1), a dropout random deactivation layer (8C×L), a one-dimensional convolutional layer (ConvlD, 1×L, weighted summation for each channel), and a sigmoid activation function (1×L). The four residual layers are primarily composed of 3D convolutional units. Each residual layer (input size C) has a length of 8C×L×H / 4×W / 4. in ×L in ×H in ×W out Each branch has two branches: the main branch (output size is C). out ×L out ×H out ×W out ) and residual branch (output size C) out ×L out ×H out ×W outThe main branch sequentially includes a 3×3×3 convolutional layer (Conv3D), batch normalization (BatchNorm3D), ReLU activation function, and a 3×3×3 convolutional layer (Conv3D) and batch normalization (BatchNorm3D). The residual branch includes an optional 1×1×1 3D convolutional layer (Optional Conv3D). in ×L in ×H in ×W out The intermediate output size C obtained through the main branch out ×L out ×H out ×W out When there is a mismatch, this optional convolution adjusts the dimensions, adds the output of the main branch and the output of the residual branch, and then performs ReLU activation to obtain the output of the residual structure (output size C). out ×L out ×H out ×W out The spatial attention module processes the extracted features. in The mean pooling (size 8C×L×H / 32×W / 32) and max pooling (size 8C×L×H / 32×W / 32) of each channel are calculated. After concatenation, feature fusion is performed through a 7×7 2D convolutional layer (Conv2D). The spatial attention weight matrix (size 8C×L×H / 32×W / 32) is obtained by passing the sigmoid activation function. After obtaining the spatial attention weight matrix, feature enhancement is achieved through residual gating to obtain the feature. out (Dimensions are 8C×L×H / 32×W / 32), the calculation formula is:

[0018] feature out =feature in ×(1+SpatialAttentionWeight)

[0019] The temporal attention module focuses on the extracted features. inThe mean over spatial (size 8C×L) is calculated across the height and width dimensions. Then, a temporal attention weight matrix (size 1×L) is obtained by performing a one-dimensional convolution Conv1D with weighted summation across each channel, followed by ReLU activation, and another one-dimensional convolution Conv1D with weighted summation across each channel, followed by Sigmoid activation. Temporal feature enhancement is achieved through residual gating to obtain the feature. out (Dimensions are 8C×L×H / 32×W / 32), the calculation formula is:

[0020] feature out =feature in ×(1+TemporalAttentionWeight)

[0021] The preliminary membership estimation model structure for the end of the pharyngeal phase is as follows: Figure 5 As shown, the detailed structure diagram of the residual structure is as follows: Figure 4 As shown, the detailed structure diagram of the spatial attention module is as follows: Figure 6 As shown, the detailed structure diagram of the time attention module is as follows: Figure 7 As shown.

[0022] The above-described method for estimating pharyngeal time based on swallowing contrast imaging video requires training the model parameters to obtain a well-trained estimation model. Its key feature is that the input to the estimation model for the membership function curve of the start frame is a video slice, and the output label is the corresponding slice of the membership curve of the start point. The objective function uses Smooth L1 Loss, and the parameters are updated using the backpropagation algorithm (BP algorithm) and the stochastic gradient descent (SGD) algorithm. The definition of Smooth L1 Loss is as follows:

[0023]

[0024] Where δ is the segmentation threshold, and the SGD optimizer formula is:

[0025]

[0026] Where θ t Here are the model parameters for the t-th iteration, and η is the current learning rate. Let be the gradient of the loss function with respect to the parameters. During training, a momentum term is introduced to accelerate the convergence process and suppress oscillations. Its update formula is:

[0027]

[0028] θ t+1 =θ t +v t+1

[0029] Where μ is the momentum coefficient, V t This is the momentum term at the t-th iteration.

[0030] The above-mentioned method for estimating pharyngeal time based on swallowing contrast imaging video requires training the model parameters to obtain a well-trained estimation model. Its key feature is that the input to the estimation model for the membership function curve of the end frame is a video slice, and the output label is the corresponding end-point membership curve slice. The objective function, backpropagation algorithm, and other configurations are the same as those used in training the estimation model for the membership function curve of the start frame.

[0031] The above method for estimating pharyngeal time based on swallowing contrast imaging video requires data annotation of the original swallowing contrast imaging video to obtain training samples for the membership function curve estimation model of the start frame. Its features include: (1) using a self-made annotation tool under the guidance of experienced clinical medical staff. The annotation tool is as follows: Figure 8 As shown, during the annotation process, the video frame that best matches the characteristics of the beginning of the pharyngeal phase is selected and its sequence number is denoted as a, and the video frame that best matches the characteristics of the end of the pharyngeal phase is selected and its sequence number is denoted as b. (2) The Gaussian membership function is used to perform fuzzy modeling on the pharyngeal phase beginning frame. The formula of the Gaussian membership function is:

[0032]

[0033] Where c and σ are the parameters of the Gaussian membership function, set as follows:

[0034] C = a

[0035]

[0036] Here, k is a positive value less than 1 and close to 0. Based on the above formula, a starting frame membership value is given to each frame in the video.

[0037] The above method for estimating pharyngeal time based on swallowing contrast imaging video requires data annotation of the original swallowing contrast imaging video to obtain training samples for the membership function curve estimation model of the end frame. Its features include: (1) using a self-made annotation tool under the guidance of experienced clinical medical staff. The annotation tool is as follows: Figure 8 As shown, during the annotation process, the video frame that best matches the characteristics of the beginning of the pharyngeal phase is selected and its serial number is denoted as a, and the video frame that best matches the characteristics of the end of the pharyngeal phase is selected and its serial number is denoted as b. (2) The Gaussian membership function is used to perform fuzzy modeling on the end of the pharyngeal phase frame. The formula of the Gaussian membership function is:

[0038]

[0039] Where c and σ are the parameters of the Gaussian membership function, set as follows:

[0040] c = b

[0041]

[0042] Here, k is a positive value less than 1 and close to 0. Based on the above formula, each frame in the video is given an end-frame membership value.

[0043] The above method for estimating pharyngeal time based on swallowing contrast imaging video, in step (d) of (1), sharpens the membership curve of the pharyngeal start point in the complete video domain to obtain a sharp estimate of the pharyngeal start time. The feature is that the frame number corresponding to the maximum value point of the curve is found and used as the sharp estimate of the start time.

[0044] The above method for estimating pharyngeal time based on swallowing contrast imaging video, specifically for step (d) in (2), which involves clarifying the membership curve of the pharyngeal end point in the complete video domain to obtain a clear estimate of the pharyngeal end time, is characterized by the following: In order to ensure that the estimation result conforms to the factual laws of swallowing physiology, i.e., the frame number of the pharyngeal start frame should be less than the frame number of the end frame, a frame order constraint should be introduced in the post-processing of the membership curve. Specifically, when estimating the pharyngeal end time, the frame number corresponding to the maximum value of the membership curve of the pharyngeal end point is first found as a candidate for the pharyngeal end frame. If the candidate frame is greater than the clear estimate of the pharyngeal start time mentioned above, i.e., it conforms to the prior knowledge of physiology, then the candidate frame is used as the estimation result of the pharyngeal end frame. If the candidate frame is less than the estimated value of the pharyngeal start time mentioned above, then the clear value of the pharyngeal start frame is fixed, and the frame number of the maximum value of the membership curve of the pharyngeal end point is found again on its right side as a clear estimate of the pharyngeal end time.

[0045] Beneficial effects

[0046] In the experiment, a pharyngeal phase estimation system based on swallowing contrast imaging videos was built according to the proposed scheme. Under the guidance of experienced medical staff, the raw data was labeled, including swallowing contrast imaging videos of 60 individuals and 72 raw videos. In addition, complete videos from 12 different individuals were used to test the generalization performance of the model. Modeling and estimation were performed according to the proposed scheme. It was found that the model fits the membership functions of the pharyngeal phase start frame and end frame well for patients with normal swallowing and patients with mild swallowing disorders. On the test data, the estimation accuracy of the pharyngeal phase start frame within three frames was 90.9%, which is better than the best result reported in Reference 2. The estimation accuracy of the end frame within three frames was 91.67%, which is basically the same as the best result in Reference 2. The average boundary error of the pharyngeal phase start frame and the average boundary error of the pharyngeal phase end frame were both 1.33 frames. While keeping the average boundary error of the pharyngeal phase end frame basically the same as that in Reference 3, the average boundary error of the pharyngeal phase start frame is better than the result reported in Reference 3. Attached Figure Description

[0047] Figure 1 The overall flowchart of this application scheme.

[0048] Figure 2 A diagram illustrating a sliding window.

[0049] Figure 3 The structure of the preliminary membership estimation model for the pharyngeal phase in this application.

[0050] Figure 4 , Figure 3 , Figure 5 The diagram shows four residual layers. The Input in the diagram refers to... Figure 3 , Figure 5 The features extracted after MaxPool3D are 5 in dimension. Output refers to the features further extracted from the Input after the ResidualLayers structure, which also has a feature tensor dimension of 5.

[0051] Figure 5 The structure of the preliminary membership estimation model for the end point of the pharyngeal phase in this application.

[0052] Figure 6 , Figure 5 SpatialAttention module structure diagram, features in the diagram in It refers to Figure 5 The features extracted by the ResidualLayers module have a feature tensor dimension of 5. outThis refers to the features extracted by the SpatialAttention module, with a feature tensor dimension of 5.

[0053] Figure 7 , Figure 5 TemporalAttention module structure diagram, features in the diagram in It refers to Figure 5 The features extracted by the SpatialAttention module have a feature tensor dimension of 5. out This refers to the features extracted by the TemporalAttention module, with a feature tensor dimension of 5.

[0054] Figure 8 A schematic diagram of the annotation software used in this application.

[0055] Figure 9 The membership curve of the pharyngeal start point of the complete video domain estimated in the embodiments of this application.

[0056] Figure 10 The membership curve of the pharyngeal end point of the complete video domain estimated in the embodiments of this application. Detailed Implementation

[0057] Examples. The examples described herein are some, but not all, of the embodiments of the present invention. The present invention proposes a method for evaluating the pharyngeal period time based on swallowing contrast imaging, and particularly relates to a pharyngeal period time estimation method based on a combination of deep learning and fuzzy theory. Specifically, the present invention is characterized by estimating the pharyngeal period time during swallowing based on swallowing contrast imaging video, that is, by using X-ray fluoroscopy to perform special contrast imaging on the mouth, pharynx, esophagus and other locations. The example estimates the pharyngeal period time of a swallowing contrast imaging video with a duration of 80 frames, including the following steps: (1) (a) Using a sliding window of length L = 16, slide it across the swallowing contrast video, with each window moving in a step of S = 4, generating 17 video slices. (b) For each video slice, first convert it to grayscale, then input the grayscale slice into the trained pharyngeal phase initiation estimation model, outputting the preliminary membership value of the pharyngeal phase initiation of each frame in the slice. (c) Average the preliminary membership values ​​of the pharyngeal phase initiation of each frame in the entire video across all slices to obtain the final membership value of the pharyngeal phase initiation of that frame, thus obtaining the pharyngeal phase initiation membership curve of the complete video universe, such as... Figure 9As shown in the figure, the horizontal axis Frame Number represents the video frame number, and the vertical axis Fuzzy Value represents the membership value of the final pharyngeal start point for each frame. The predicted start fuzzy curve is the membership curve of the pharyngeal start point obtained above, with max at frame. 64 indicates that the x-coordinate of the maximum value of this membership curve is 64. (d) The membership curve of the pharyngeal phase start point in the complete video domain is sharpened to obtain the sharp estimate of the pharyngeal phase start time as frame 64. (2)(a) Using a sliding window with a length of L=16, slide it on the swallowing contrast video. The window moves by a step size of S=4 each time to generate 17 video slices. (b) For each video slice, first grayscale it and input the grayscale slice into the trained pharyngeal phase end point estimation model. Output the preliminary membership value of the pharyngeal phase end point of each frame in the slice. (c) Average the preliminary membership values ​​of the pharyngeal phase end point of each frame in the entire video in each slice to obtain the final pharyngeal phase membership value of the frame image. Thus, the pharyngeal phase end point membership curve of the complete video domain is obtained, such as Figure 10 As shown, the horizontal axis Frame Number represents the video frame number, the vertical axis Fuzzy Value represents the membership value of the final pharyngeal end point of each frame, the predicted end fuzy curve is the membership curve of the pharyngeal end point obtained above, and max at frame 68 means that the horizontal coordinate of the maximum value point of this membership curve is 68. (d) Combined with prior knowledge, the membership curve of the pharyngeal end point of the complete video domain is sharpened to obtain the sharp estimate of the pharyngeal end time as the 68th frame; (3) Subtract the sharp estimate of the pharyngeal start time from the sharp estimate of the pharyngeal end time, divide by the video frame rate fps=13, and obtain the estimate of the pharyngeal time as 0.31s.

[0058] The above is a method for estimating pharyngeal phase time based on swallowing contrast imaging video. The preliminary membership value estimation model for the pharyngeal phase start point, in practice, takes a video slice as input, with dimensions L×H×W, where W is the width of the video frame (W=512 in this embodiment), H is the height of the video frame (H=512 in this embodiment), and L is the number of image frames in the slice (L=16 in this embodiment). The output is the preliminary membership value estimation of the pharyngeal phase start point for each image frame in the slice, with a size of 16×1, forming a membership curve. (The preliminary membership value estimation model for the pharyngeal phase start point has an input size of 1×16.) The 512×512 array includes, in sequence: a Conv3D convolutional layer with a kernel size of 3×7×7 and 64 kernels (output size 64×16×256×256); a BatchNorm3D batch normalization layer (output size 64×16×256×256); a ReLU activation layer (output size 64×16×256×256); a MaxPool3D max pooling layer with a kernel size of 1×3×3 (output size 64×16×128×128); and four residual structures. Layers (output size 512×16×16×16), a 3D average pooling layer AdaptiveAvgPool (output size 512×16×1×1), a Dropout random deactivation layer (output size 512×16), and a one-dimensional convolution ConvlD layer with weighted summation of each channel (output size 1×16). The four residual structures are primarily composed of 3D convolutional units, and each residual structure has two branches (input size C). in ×L in ×H in ×W out ), main branch (output size is C) out ×L out ×H out ×W out ) and residual branch (output size C) out ×L out ×H out ×W out The main branch sequentially includes a 3×3×3 convolutional layer (Conv3D), batch normalization (BatchNorm3D), ReLU activation function, and a 3×3×3 convolutional layer (Conv3D) and batch normalization (BatchNorm3D). The residual branch includes an optional 1×1×1 3D convolutional layer (Optional Conv3D). in ×L in ×H in ×W out The intermediate output size C obtained through the main branch out ×L out ×Hout ×W out When there is a mismatch, this optional convolution adjusts the dimensions, adds the output of the main branch and the output of the residual branch, and then performs ReLU activation to obtain the output of the residual structure (output size C). out ×L out ×H out ×W out In implementation, the two convolutional layers in the main branch of the first residual structure (ResidualLayer) both have 64 kernels, which is equal. The residual branch does not use optional convolution operations. Conv3D (residual structure output size is 64×16×128×128). The second residual structure, ResidualLayer2, has two convolutional layers with 128 kernels each in its main branch and 128 kernels in its residual branch (residual structure output size is 128×16×64×64). The third residual structure, ResidualLayer3, has two convolutional layers with 256 kernels each in its main branch and 256 kernels in its residual branch (residual structure output size is 256×16×32×32). The fourth residual structure, ResidualLayer4, has two convolutional layers with 512 kernels each in its main branch and 512 kernels in its residual branch (residual structure output size is 512×16×16×16). Because the number of input and output channels is inconsistent in the second, third, and fourth residual structures, optional convolution operations are used. Conv3D was adjusted, and the preliminary membership estimation model structure for the pharyngeal phase initiation point is as follows: Figure 3 As shown, the detailed structure diagram of the residual structure is as follows: Figure 4 As shown.

[0059] The above-described method for estimating pharyngeal duration based on swallowing contrast imaging video, specifically the preliminary membership value estimation model for the pharyngeal duration termination point, takes as input a video slice of dimension L×H×W, where W is the width of the video frame, H is the height of the video frame, and L is the number of image frames in the slice. The output is the preliminary membership value estimate of the pharyngeal duration termination point for each image frame in the slice, with a size of 16×1, forming a membership curve. The preliminary membership value estimation model for the pharyngeal duration termination point (input size 1×16×512×512) sequentially includes convolution... The system consists of a 3×7×7 kernel-size Conv3D convolutional layer with 64 kernels (output size 64×16×256×256), a batch normalization layer BatchNorm3D (output size 64×16×256×256), ReLU activation (output size 64×16×256×256), a 1×3×3 kernel-size maxpooling layer MaxPool3D (output size 64×16×128×128), and four residual structures. Layers (output size 512×16×16×16), SpatialAttention module (output size 512×16×16×16), TemporalAttention module (output size 512×16×16×16), 3D average pooling layer AdaptiveAvgPool (output size 512×16×1×1), Dropout random deactivation layer (output size 512×16), one-dimensional convolution Conv1D with weighted summation of each channel (output size 1×16), and Sigmoid activation function (output size 1×16). The four residual structures are primarily composed of 3D convolutional units. Each residual structure (input size C) in ×L in ×H in ×W out Each branch has two branches: the main branch (output size is C). out ×L out ×H out ×W out ) and residual branch (output size C) out ×L out ×H out ×W out The main branch sequentially includes a 3×3×3 convolutional layer (Conv3D), batch normalization (BatchNorm3D), ReLU activation function, and a 3×3×3 convolutional layer (Conv3D) and batch normalization (BatchNorm3D). The residual branch includes an optional 1×1×1 3D convolutional layer (OptionalConv3D). in ×L in×H in ×W out The intermediate output size C obtained through the main branch out ×L out ×H out ×W out When there is a mismatch, this optional convolution adjusts the dimensions, adds the output of the main branch and the output of the residual branch, and then performs ReLU activation to obtain the output of the residual structure (output size C). out ×L out ×H out ×W out In implementation, the first residual structure, ResidualLayer1, has two convolutional layers with 64 kernels each in its main branch, and the residual branch does not use convolution (the output size of the residual structure is 64×16×128×128). The second residual structure, ResidualLayer2, has two convolutional layers with 128 kernels each in its main branch, and the residual branch also has 128 kernels (the output size of the residual structure is 128×16×64×64). The third residual structure, ResidualLayer3, has... The two convolutional layers in the first branch each have 256 kernels, and the residual branch also has 256 kernels (the output size of the residual structure is 256×16×32×32). The second residual structure, ResidualLayer4, has two convolutional layers in its main branch each have 512 kernels, and the residual branch also has 512 kernels (the output size of the residual structure is 512×16×16×16). Since the number of input and output channels is inconsistent between the second, third, and fourth residual structures, Optional Conv3D is used for adjustment. The spatial attention module processes the extracted features. in (Size 512×16×16×16) The mean pooling (size 16×1×16×16) and max pooling (size 16×1×16×16) of each channel are calculated and concatenated (size 16×2×16×16). This concatenation is then performed using a 7×7 Conv2D convolutional layer with one kernel for feature fusion (output size 16×1×16×16). The spatial attention weight matrix (size 16×1×16×16) is obtained through a sigmoid activation function. After obtaining the spatial attention weight matrix, feature enhancement is achieved through residual gating to obtain the final feature. out (Dimensions are 512×16×16×16), the calculation formula is:

[0060] feature out=feature in ×(1+SpatialAttention Weight)

[0061] The temporal attention module focuses on the extracted features. in The mean over spatial (output size 512×16×16×16) is calculated along the height and width dimensions. Then, it is sequentially passed through a one-dimensional convolution Conv1D (output size 128×16) with weighted summation of each channel, ReLU activation (output size 128×16), another one-dimensional convolution Conv1D with weighted summation of each channel (output size 1×16), and a Sigmoid activation to obtain the temporal attention weight matrix TemporalAttentionWeight (size 1×16). Temporal feature enhancement is achieved through residual gating, resulting in the final feature. out (Dimensions are 512×16×16×16), the calculation formula is:

[0062] feature out =feature in ×(1+TemporalAttentionWeight)

[0063] The preliminary membership estimation model structure for the end of the pharyngeal phase is as follows: Figure 5 As shown, the detailed structure diagram of the residual structure is as follows: Figure 4 As shown, the detailed structure diagram of the spatial attention module is as follows: Figure 6 As shown, the detailed structure diagram of the time attention module is as follows: Figure 7 As shown.

[0064] The above method for estimating pharyngeal time based on swallowing contrast imaging video requires training the model parameters to obtain a well-trained estimation model. In implementation, the input of the estimation model for the membership function curve of the start frame is a video slice, and the output label is the corresponding slice of the membership curve of the start point. The objective function adopts Smooth L1 Loss, and the backpropagation algorithm (BP algorithm) is used for parameter updates. The stochastic gradient descent (SGD) algorithm is used to calculate the updated parameters. The definition of Smooth L1 Loss is as follows:

[0065]

[0066] Where δ is the segmentation threshold, which is set to 1 in this embodiment, and the SGD optimizer formula is:

[0067]

[0068] Where θ t Here are the model parameters for the t-th iteration, and η is the current learning rate. Let be the gradient of the loss function with respect to the parameters. During training, a momentum term is introduced to accelerate the convergence process and suppress oscillations. Its update formula is:

[0069]

[0070] θ t+1 =θ t +v t+1

[0071] Where μ is the momentum coefficient, which is set to 0.9 in this embodiment, V t Let be the momentum term at iteration t. During training, the initial learning rate is 0.0001. An early stopping strategy is adopted. If the learning rate does not decrease for 100 consecutive iterations, early stopping is triggered and training is stopped. A dynamic learning rate adjustment strategy is adopted. If the validation loss does not decrease for 40 consecutive iterations, the learning rate is automatically multiplied by a decay factor of 0.1. The lower limit of the learning rate is set to 0.0000001.

[0072] The above-described method for estimating pharyngeal time based on swallowing contrast imaging video requires training the model parameters to obtain a well-trained estimation model. In practice, the input of the estimation model for the membership function curve of the end frame is a video slice, and the output label is the corresponding end point membership curve slice. In this embodiment, the initial learning rate is 0.0003, and the objective function, training strategy, backpropagation algorithm, and other configurations are the same as those used for training the estimation model for the membership function curve of the start frame.

[0073] The above method for estimating pharyngeal time based on swallowing contrast imaging video requires data annotation of the original swallowing contrast imaging video to obtain training samples for the membership function curve estimation model of the start frame. In implementation, (1) annotation is performed using a self-made annotation tool under the guidance of experienced clinical medical staff. The annotation tool is as follows: Figure 8 As shown, during the annotation process, the video frame that best matches the characteristics of the beginning of the pharyngeal phase is selected, and its sequence number is denoted as a. In this embodiment, a = 64. The video frame that best matches the characteristics of the end of the pharyngeal phase is selected, and its sequence number is denoted as b. In this embodiment, b = 68. (2) The Gaussian membership function is used to model the pharyngeal phase beginning frame. The formula of the Gaussian membership function is:

[0074]

[0075] Where c and σ are the parameters of the Gaussian membership function, and they are set as follows:

[0076] c = a = 64

[0077]

[0078] Based on the above formula, a starting frame membership value is assigned to each frame in the video.

[0079] The above method for estimating pharyngeal time based on swallowing contrast imaging video requires data annotation of the original swallowing contrast imaging video to obtain training samples for the membership function curve estimation model of the end frame. In implementation, (1) annotation is performed using a self-made annotation tool under the guidance of experienced clinical medical staff. The annotation tool is as follows: Figure 8 As shown, during the annotation process, the video frame that best matches the characteristics of the beginning of the pharyngeal phase is selected and its sequence number is denoted as a. In this embodiment, a = 64. The video frame that best matches the characteristics of the end of the pharyngeal phase is selected and its sequence number is denoted as b. In this embodiment, b = 68. (2) The Gaussian membership function is used to model the end of the pharyngeal phase frame. The formula of the Gaussian membership function is:

[0080]

[0081] Where c and σ are the parameters of the Gaussian membership function, and they are set as follows:

[0082] c = b = 68

[0083]

[0084] Based on the above formula, each frame in the video is given an end-frame membership value.

[0085] The above method for estimating pharyngeal time based on swallowing contrast imaging video involves clarifying the membership curve of the pharyngeal start point in the complete video domain in step (d) of (1) to obtain a clear estimate of the pharyngeal start time. In practice, the frame number corresponding to the maximum value point of the curve is found and used as the clear estimate of the start time. In this embodiment, the clear estimate of the start time is the 64th frame.

[0086] The above method for estimating pharyngeal time based on swallowing contrast imaging video, in step (d) of (2), the membership curve of the pharyngeal end point of the complete video domain is sharpened to obtain a sharp estimate of the pharyngeal end time. In practice, in order to ensure that the estimation result conforms to the factual laws of swallowing physiology, that is, the frame number of the pharyngeal start frame should be less than the frame number of the end frame, a frame order constraint should be introduced in the post-processing of the membership curve. Specifically, when estimating the pharyngeal end time, the frame number corresponding to the maximum value of the membership curve of the pharyngeal end point is first found as a candidate for the pharyngeal end frame. If the candidate frame is greater than the sharp estimate of the pharyngeal start time mentioned above, that is, it conforms to the prior knowledge of physiology, then the candidate frame is used as the estimation result of the pharyngeal end frame. If the candidate frame is less than the estimated value of the pharyngeal start time mentioned above, then the sharp value of the pharyngeal start frame is fixed, and the frame number of the maximum value of the membership curve of the pharyngeal end point is found again on its right side as the sharp estimate of the pharyngeal end time. In this embodiment, the sharp estimate of the end time is the 68th frame.

Claims

1. A method of swallow phase time estimation based on a swallow contrast video, characterized by For a swallowing radiography video data, the method comprises the following steps: (1) (a) using a sliding window with a length of L to slide on the swallowing radiography video, and generating a plurality of video slices by moving the window by a step size S each time, (b) for each video slice, first grayscale, inputting the grayscaled slice into a trained pharyngeal phase start point estimation model, and outputting a preliminary membership value of the pharyngeal phase start point of each frame of image in the slice, (c) averaging the preliminary membership values of the pharyngeal phase start point of each frame of image in the whole video in each slice covering the frame of image, i.e., overlap averaging, obtaining a final membership value of the pharyngeal phase start point of the frame of image, thereby obtaining a pharyngeal phase start point membership curve of the complete video domain, (d) performing clear processing on the pharyngeal phase start point membership curve of the complete video domain to obtain a clear estimation of the pharyngeal phase start time; (2) (a) using a sliding window with a length of L to slide on the swallowing radiography video, and generating a plurality of video slices by moving the window by a step size S each time, (b) for each video slice, first grayscale, inputting the grayscaled slice into a trained pharyngeal phase end point estimation model, and outputting a preliminary membership value of the pharyngeal phase end point of each frame of image in the slice, (c) averaging the preliminary membership values of the pharyngeal phase end point of each frame of image in the whole video in each slice covering the frame of image, i.e., overlap averaging, obtaining a final membership value of the pharyngeal phase of the frame of image, thereby obtaining a pharyngeal phase end point membership curve of the complete video domain, (d) combining prior knowledge, performing clear processing on the pharyngeal phase end point membership curve of the complete video domain to obtain a clear estimation of the pharyngeal phase end time; (3) subtracting the clear estimation of the pharyngeal phase start time from the clear estimation of the pharyngeal phase end time, and dividing by the video frame rate in frame units to obtain an estimation of the pharyngeal phase time.

2. The method of claim 1, wherein the input of the swallowing onset point preliminary membership value estimation model is a video slice with a dimension of LxHxW, W is the width of the image frame, H is the height of the image frame, and L is the number of image frames in the slice, and the output is a preliminary membership value estimation of each image frame in the slice, which constitutes a membership curve, and the swallowing onset point preliminary membership value estimation model sequentially comprises a convolution layer Conv3D with a kernel size of 3x7x7 (the output size is CxLxH / 2xW / 2, where C is the number of channels, and the same below), a batch normalization layer BatchNorm3D (the output size is CxLxH / 2xW / 2), a ReLU activation (the output size is CxLxH / 2xW / 2), a max pooling layer MaxPool3D with a kernel size of 1x3x3 (the output size is CxLxH / 4xW / 4), four residual structures Residual Layers (the output size is 8CxLxH / 32xW / 32), a three-dimensional average pooling layer AdaptiveAvgPool (the output size is 8CxLx1x1), a Dropout random inactivation layer (the output size is 8CxL), and a one-dimensional convolution Conv1D (the output size is 1xL) of each channel weighted summation, wherein the four residual structures are mainly composed of three-dimensional convolution units, each residual structure (the input size is C in ×L in ×H in ×W out ) has two branches, a main branch (the output size is C out ×L out ×H out ×W out ) and a residual branch (the output size is C out ×L out ×H out ×W out ), the main branch sequentially comprises a convolution layer Conv3D with a kernel size of 3x3x3, a batch normalization BatchNorm3D, a ReLU activation function, a convolution layer Conv3D with a kernel size of 3x3x3, and a batch normalization BatchNorm3D, and the residual branch comprises an optional three-dimensional convolution Optional Conv3D with a kernel size of 1x1x1, when the size of the input Input C in ×L in ×H in ×W out is equal to the size of the intermediate output obtained by the main branch C out ×L out ×H out ×W out When not matched, the output of the main branch and the output of the residual branch are added after adjusting the dimensions with this optional convolution, and then passed through a ReLU activation to obtain the output of the residual structure (output size: C out x L out x H out x W out ).

3. The method of claim 1, wherein the preliminary membership value estimation model of the end of pharyngeal phase point is configured to receive a video slice with a dimension of LxHxW, wherein W is a width of a video image frame, H is a height of the video image frame, and L is a number of image frames in the slice, and output a preliminary membership value estimation of the end of pharyngeal phase point of each image frame in the slice, and the preliminary membership value estimation model of the end of pharyngeal phase point comprises, in sequence, a convolution layer Conv3D with a kernel size of 3x7x7 (with an output size of CxLxH / 2xW / 2, wherein C is a number of channels, and the same applies hereinafter), a batch normalization layer BatchNorm3D (with an output size of CxLxH / 2xW / 2), a ReLU activation (with an output size of CxLxH / 2xW / 2), a max pooling layer MaxPool3D with a kernel size of 1x3x3 (with an output size of CxLxH / 4xW / 4), four residual layers Residual Layers (with an output size of 8CxLxH / 32xW / 32), a spatial attention module SpatialAttention (with an output size of 8CxLxH / 32xW / 32), a temporal attention module TemporalAttention (with an output size of 8CxLxH / 32xW / 32), an adaptive average pooling layer AdaptiveAvgPool (with an output size of 8CxLx1x1), a Dropout random deactivation layer (with an output size of 8CxL), a one-dimensional convolution Conv1D (with an output size of 1xL) for weighted summation of each channel, and a Sigmoid activation function (with an output size of 1xL), wherein the four residual layers are mainly composed of three-dimensional convolution units, each residual layer (with an input size of CinxLinxHinxWout) has two branches, a main branch (with an output size of CoutxLoutxHoutxWout) and a residual branch (with an output size of CoutxLoutxHoutxWout), the main branch comprises, in sequence, a convolution layer Conv3D with a kernel size of 3x3x3, a batch normalization processing BatchNorm3D, a ReLU activation function, a convolution layer Conv3D with a kernel size of 3x3x3, and a batch normalization processing BatchNorm3D, and the residual branch comprises an optional three-dimensional convolution Optional Conv3D with a kernel size of 1x1x1, which is used to adjust the dimension when the input Input size CinxLinxHinxWout does not match the intermediate output size CoutxLoutxHoutxWout obtained through the main branch, and the output of the main branch and the output of the residual branch are added and then subjected to ReLU activation to obtain the output of the residual layer (with an output size of CoutxLoutxHoutxWout).The spatial attention module calculates the mean pooling over channel (size Lx1xH / 32xW / 32) and the maximum pooling over channel (size Lx1xH / 32xW / 32) on the extracted feature featurein (size 8CxLxH / 32xW / 32), performs feature fusion through a two-dimensional convolution layer Conv2D with a size of 7x7 after splicing, and obtains a spatial attention weight matrix SpatialAttentionWeight (size Lx1xH / 32xW / 32) through a Sigmoid activation function. After obtaining the spatial attention weight matrix, feature enhancement is realized through residual connection Residual gating, and a feature featureout (size 8CxLxH / 32xW / 32) is obtained. The calculation formula is: feature out = feature in x (1 + SpatialAttentionWeight) The time attention module calculates the mean over spatial (size: 8C*L) of the extracted feature featurein (size: 8C*L*H / 32*W / 32) in the height and width dimensions, and then obtains a temporal attention weight matrix TemporalAttentionWeight (size: 1*L) through one-dimensional convolution Conv1D of weighted summation of channels, ReLU activation, one-dimensional convolution Conv1D of weighted summation of channels and Sigmoid activation, and obtains the feature featureout (size: 8C*L*H / 32*W / 32) through residual gating, and the calculation formula is: feature out = feature in x (1 + TemporalAttentionWeight).

4. The pharyngeal period time estimation method based on swallowing contrast imaging video according to claim 1, in order to obtain a well-trained estimation model, it is necessary to train the above model parameters, characterized in that, The input part of the estimation model of the start frame membership function curve is a video slice, the output label is the corresponding start point membership curve slice, the objective function adopts Smooth L1 Loss, the parameter update is performed by using a back propagation algorithm (BP algorithm), and a stochastic gradient descent (SGD) algorithm is used to calculate the updated parameters, and the definition of Smooth L1 Loss is as follows: Wherein, δ is a segmentation threshold, and the SGD optimizer formula is where θ t is the model parameter at the tth iteration , η is the current learning rate, is the gradient of the loss function with respect to the parameters, during the training process, a momentum term is introduced to accelerate the convergence process and suppress oscillations, its update formula is θ t+1 = θ t + v t+1 where μ is the momentum coefficient, V t is the momentum term at the tth iteration.

5. The method of claim 1, wherein the trained estimation model is obtained by training the model parameters, and wherein the method further comprises: obtaining a plurality of training data sets, each of the training data sets comprising a plurality of frames of a swallowing video, each of the frames being associated with a corresponding time stamp; and training the model parameters based on the plurality of training data sets. The input part of the estimation model of the end frame membership function curve is a video slice, and the output label is the corresponding end point membership curve slice. The objective function, the back propagation algorithm used, and other configurations are the same as those used in training the estimation model of the start frame membership function curve.

6. The method of claim 1, wherein, to obtain the training sample of the start frame membership function curve estimation model, data labeling is performed on the original swallowing radiography video. (1) The labeling was performed under the guidance of experienced clinical medical staff using a self-made labeling tool. In the labeling process, the video frame most consistent with the characteristics of the start of the pharyngeal phase was selected, and its serial number was recorded as a. The video frame most consistent with the characteristics of the end of the pharyngeal phase was selected, and its serial number was recorded as b. (2) The start frame of the pharyngeal phase was modeled using a Gaussian membership function. The formula of the Gaussian membership function is: where c and σ are the parameters of the Gaussian membership function, and they are set as: c = a where k is a positive value less than 1 and close to 0. Based on the above formula, each frame in the video is given a start frame membership value.

7. The method of claim 1, wherein, to obtain the training sample of the ending frame membership function curve estimation model, data labeling is performed on the original swallowing radiography video. (1) The labeling was performed under the guidance of experienced clinical medical staff using a self-made labeling tool. In the labeling process, the video frame most consistent with the characteristics of the start of the pharyngeal phase was selected, and its serial number was recorded as a. The video frame most consistent with the characteristics of the end of the pharyngeal phase was selected, and its serial number was recorded as b. (2) The end frame of the pharyngeal phase was modeled using a Gaussian membership function. The formula of the Gaussian membership function is: where c and σ are the parameters of the Gaussian membership function, and they are set as: c = b where k is a positive value less than 1 and close to 0. Based on the above formula, each frame in the video is given an end frame membership value.

8. The method of claim 1, wherein the clear estimation of the start time of the pharyngeal phase is obtained by clearing the pharyngeal phase start point membership curve of the complete video domain for step (d) of (1), and is characterized in that The frame number corresponding to the maximum value of the curve is found, and it is taken as the clear estimate of the start time.

9. The method of claim 1, wherein the clear estimation of the end time of the pharyngeal phase is obtained by clearing the pharyngeal phase end point membership curve of the complete video domain for the step (d) in (2). To ensure that the estimation result conforms to the physiological fact that the start frame frame number should be less than the end frame frame number, a frame sequence constraint should be introduced in the post-processing of the membership curve. Specifically, when estimating the end time of the pharyngeal phase, first find the frame number corresponding to the maximum value of the end point membership curve of the pharyngeal phase, which is the candidate for the end frame of the pharyngeal phase. If the candidate frame is greater than the clear estimate of the start time of the pharyngeal phase, it conforms to the physiological prior knowledge, and the candidate frame is taken as the estimation result of the end frame of the pharyngeal phase. If the candidate frame is less than the estimation value of the start time of the pharyngeal phase, the clear value of the start frame of the pharyngeal phase is fixed, and the frame number corresponding to the maximum value of the end point membership curve of the pharyngeal phase is found on the right side, which is taken as the clear estimate of the end time of the pharyngeal phase.

Images