A semi-supervised key point positioning method and device for swallowing radiography analysis

Abstract

The application discloses a semi-supervised key point positioning method and equipment for swallowing radiography analysis, and the method comprises the following steps: acquiring and labeling swallowing radiography images to obtain labeled data; a joint optimization strategy is designed, and a supervised loss of the labeled data and a self-supervised consistency loss of unlabeled data are integrated in a unified training framework; pseudo labels are generated by using the unlabeled data, and the pseudo labels are dynamically updated in the model training process; the model is jointly trained by using supervised data and unsupervised data; a semantic guidance module is constructed and used for assisting the model to more accurately capture the features of the key point region; in the video processing stage, a Kalman filtering algorithm is introduced, and the key point prediction results of multiple frames are fused to realize key point timing calibration. By introducing the semantic guidance module, the model's capturing ability of the key point region features is enhanced, and the timing consistency is optimized by the Kalman filtering, so that the precision and robustness of the key point positioning are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, computer vision, medical image analysis, and the like, and in particular to a semi-supervised key point positioning method and device for swallowing radiography analysis. BACKGROUND

[0002] Swallowing disorder is a major health problem affecting human eating function, and its accurate diagnosis and treatment are of great significance to improve the quality of life of patients. Swallowing radiography examination, as the gold standard for clinical evaluation of swallowing function, can comprehensively record the structural and functional changes of the larynx and esophagus region by dynamically collecting X-ray videos of the oropharyngeal-esophageal swallowing process, and provide reliable basis for the diagnosis of swallowing disorder. However, due to the complexity of the anatomical structure and the problems such as blurred target boundary and serious noise interference in the radiography video, it is a significant challenge to accurately measure the time and kinematic parameters involved in the swallowing process. Key points, as the core features of anatomical structures, not only focus on and locate the moving targets, but also provide the basis for the calculation of kinematic parameters (such as hyoid displacement) and key indicators. Therefore, the accurate positioning of key points has important value for the quantitative evaluation of swallowing function and the optimization of clinical diagnosis and treatment. Further, the automatic key point positioning method will provide an efficient solution for the objective analysis of swallowing disorder and large-scale clinical application.

[0003] Currently, key point positioning technology is widely used in the field of computer vision, which extracts structured features in images or videos through deep learning models to achieve accurate positioning. In fully supervised key point positioning, the model relies on a large amount of manually annotated high-quality key point coordinates as training data. Such methods usually extract features from images or video frames through convolutional neural networks (CNN) or models based on self-attention mechanisms (such as Transformer), and predict the positions of key points through coordinate regression or heat map regression. These methods can achieve high positioning accuracy when there is sufficient labeled data. However, the anatomical structure in swallowing radiography video is complex, the labeling cost is high, and the subjectivity is strong, which makes it difficult to scale the acquisition of training data, thereby limiting the applicability of fully supervised methods.

[0004] To solve the problem of insufficient annotation, semi-supervised key point positioning methods have emerged. These methods combine a small amount of labeled data with a large amount of unlabeled data to improve model performance by exploiting the potential structural information in unlabeled data. Semi-supervised methods usually include strategies such as generating pseudo-labels, contrastive learning, or consistency constraints, which can reduce the dependence on manual annotation by exploiting the characteristics of unlabeled data. Especially in swallowing radiography videos, it is relatively easy to obtain unlabeled data, and semi-supervised methods can improve the accuracy and robustness of key point positioning while controlling the cost of annotation. In addition, semi-supervised methods can better utilize temporal information and complementary multi-modal features when dealing with the ambiguous boundaries and complex dynamic changes of specific anatomical structures in videos, thereby enhancing the model's ability to capture key points. These characteristics make semi-supervised key point positioning a promising research direction in swallowing radiography analysis.

[0005] However, existing semi-supervised key point positioning techniques have limitations in the field of swallowing radiography analysis. Current methods lack specific domain semantic guidance strategies for swallowing radiography, making it difficult to fully utilize background knowledge to improve the robustness of the model in complex backgrounds and small target regions, especially when the swallowing action target is ambiguous or the background is complex. At the same time, these techniques do not effectively consider the temporal consistency between video frames, which can lead to large fluctuations in key point prediction results in the time dimension, making it difficult to meet the strict requirements of clinicians for high accuracy and temporal stability of the action trajectory. SUMMARY

[0006] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a semi-supervised key point positioning method, device and medium for swallowing radiography analysis.

[0007] The first technical solution adopted by the present application is:

[0008] A semi-supervised key point positioning method for swallowing radiography analysis, comprising the following steps:

[0009] Obtaining a swallowing radiography image, annotating anatomical key points in the swallowing radiography image, and obtaining labeled data;

[0010] Designing a joint optimization strategy to integrate the supervised loss of labeled data and the self-supervised consistency loss of unlabeled data in a unified training framework; generating pseudo-labels from unlabeled data and dynamically updating the pseudo-labels during model training; and jointly training the model using supervised data and unsupervised data;

[0011] Constructing a semantic guidance module to assist the model in more accurately capturing the features of the key point region, thereby improving the positioning performance of complex backgrounds and ambiguous key points;

[0012] A Kalman filter algorithm is introduced in the video processing stage, and the key point prediction results of multiple frames are fused to realize key point timing calibration.

[0013] Further, the swallowing radiography image is acquired, and the anatomical key points in the swallowing radiography image are labeled to obtain labeled data, including:

[0014] According to professional medical anatomy knowledge and the characteristics of the swallowing radiography image, key positions accurately reflecting the changes of the anatomical structure during swallowing are selected as the anatomical key points to be labeled;

[0015] Professional image labeling tools are used, and medical image professionals accurately label the selected key points on the swallowing radiography image to form a small amount of labeled data set, providing a reliable supervised data basis for subsequent model training.

[0016] Further, the joint optimization strategy is designed, and the supervised loss of the labeled data and the self-supervised consistency loss of the unlabeled data are integrated in a unified training framework; pseudo labels are generated from the unlabeled data, and the pseudo labels are dynamically updated during the model training process, and the model is trained jointly using supervised data and unsupervised data, including:

[0017] A unified training framework including a labeled data supervision branch and an unlabeled data self-supervised branch is constructed, in the labeled data supervision branch, a regression loss function is used to measure the difference between the model prediction result and the true label of the labeled data, to guide the model to learn the key features in the labeled data;

[0018] In the unlabeled data self-supervised branch, a consistency loss function based on data enhancement is designed to mine the potential features in the unlabeled data;

[0019] During the model training process, pseudo labels are generated from the prediction results of the unlabeled data, high-confidence prediction results are selected as pseudo labels by setting a confidence threshold, and these pseudo labels are used together with the labeled data to participate in the next round of model training. At the same time, according to the iterative training of the model, the confidence evaluation and selection strategy of the pseudo labels are constantly updated, and the reliability of the pseudo labels is dynamically improved, so as to realize the joint and effective training of supervised data and unlabeled data, and improve the generalization ability and adaptability of the model to complex situations.

[0020] Further, for the labeled data, a heat map is used as the supervision signal of the key points, and a labeled image x s and its corresponding labeled key point coordinate set where K is the number of key points, and a Gaussian heat map is generated for each key point as a supervision signal:

[0021]

[0022] where (u, v) is the image pixel coordinate, σ is the standard deviation of Gaussian distribution, and controls the diffusion range of the heat map;

[0023] The regression loss function is a mean square error loss function; the key point heat map predicted by the model is The supervised loss of a single sample is calculated by a mean square error (MSE) loss function:

[0024]

[0025] where H k represents the labeled key points.

[0026] Further, for unlabeled data, a teacher-student training paradigm is adopted to construct a teacher model f T and a student model f S ; for unlabeled images x u , input data and

[0027] Data is input into the teacher model f T , and the corresponding heat map prediction is output. The heat map prediction is then converted from weak to strong to generate pseudo labels Data is input into the student model f S , and the output heat map is

[0028] Only when the maximum response value of the pseudo label exceeds the confidence threshold τ, it is used for student model training; otherwise, the pseudo label does not participate in training, and the expression is:

[0029]

[0030] where is an indicator function, when , the pseudo label is set to zero and does not participate in training; τ(t) is a dynamic confidence threshold.

[0031] Further, the dynamic confidence threshold τ(t) is linearly improved with the training process, and the expression is:

[0032]

[0033] where t is the current training round, T is the total training round, τ init and τ end represent the initial value and final value of the confidence threshold.

[0034] The loss function for training the student model with unlabeled data is defined as:

[0035]

[0036] Further, the semantic guidance module is configured to assist the model in more accurately capturing features of the key point region, and the semantic guidance module comprises:

[0037] The pre-trained DINOv2 model is used to extract semantic features from the swallowing radiography image.

[0038] The semantic guidance module is designed to align the semantic features using the features extracted by the key point positioning model from the swallowing radiography image, so that the model focuses more on the semantic information of the key point region, thereby accurately positioning the key anatomical structure in a complex background and improving the positioning accuracy and robustness of the model.

[0039] Further, the expression of the extracted semantic features is:

[0040] F sec =f DINOv2 (x)

[0041] In the formula, f DINOv2 represents the DINOv2 model;

[0042] The key point positioning model extracts the features F enc of the swallowing radiography image through its own encoder f model :

[0043] F model =f enc (x)

[0044] The semantic guidance module aligns the semantic features by calculating the cosine similarity of F sec and F model , so as to guide the model to focus more on the semantic information of the key point region; the calculation formula of the cosine similarity is:

[0045]

[0046] In the formula, ‖·‖ represents the two-norm of the vector.

[0047] Further, the Kalman filter algorithm is introduced in the video processing stage to realize key point timing calibration by fusing the key point prediction results of multiple frames, and the Kalman filter algorithm comprises:

[0048] After performing key point positioning on the swallowing radiography video frame by frame, the key point prediction result of each frame is input into the Kalman filter algorithm.

[0049] The Kalman filter algorithm estimates and predicts the state of key points by establishing a state space model. It combines the observation values ​​of the current frame (i.e., the key point prediction results of the model) and the state estimation values ​​of the previous frame to calculate a more accurate and smooth key point position estimate for the current frame, thereby effectively reducing the jitter and discontinuity of the positioning trajectory and improving the timing consistency and accuracy of key point detection in video sequences.

[0050] The second technical solution adopted by the present invention is:

[0051] An electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement a semi-supervised key point positioning method for swallowing radiography analysis as described above.

[0052] The third technical solution adopted by the present invention is:

[0053] A computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a semi-supervised key point localization method for swallowing radiography analysis as described above.

[0054] The fourth technical solution adopted by the present invention is:

[0055] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, so that the computer device performs the above method.

[0056] The present invention has the following beneficial effects: By combining training with labeled and unlabeled data, it constructs an efficient and accurate keypoint localization framework. Furthermore, by introducing a semantic guidance module, the present invention enhances the model's ability to capture keypoint regional features and optimizes temporal consistency through Kalman filtering, effectively improving the accuracy and robustness of keypoint localization, particularly in complex backgrounds and with blurred targets. This invention will provide more accurate keypoint kinematic parameters for clinical scenarios such as dysphagia diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and those skilled in the art can obtain other drawings according to these drawings without paying any creative labor.

[0058] Figure 1 is a step flow chart of a semi-supervised key point positioning method for swallowing radiography analysis in the embodiments of the present application.

[0059] Figure 2 is an anatomical key point schematic diagram in the embodiments of the present application. DETAILED DESCRIPTION

[0060] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0061] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0062] In the description of the present application, the meaning of several is one or more, and the meaning of multiple is more than two. Greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0063] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solutions.

[0064] To solve the technical problems of the prior art, the present application provides a semi-supervised key point positioning scheme for swallowing radiography analysis, which combines supervised data and unsupervised data for joint training to build an efficient and accurate key point positioning framework, aiming to solve the technical problems of complex anatomical structure and key point annotation scarcity in swallowing radiography videos. Specifically, first, anatomical key points in swallowing radiography pictures are labeled based on clinical knowledge to form a small amount of high-quality labeled data set; second, a joint optimization strategy is designed to integrate the supervised loss of labeled data and the self-supervised consistency loss of unlabeled data in a unified training framework, and pseudo-labels are generated using unlabeled data to dynamically update the pseudo-labels during model training to improve their reliability; then, with the semantic feature extraction capability of the DINOv2 model, a semantic guidance module is constructed to assist the model in more accurately capturing the features of the key point region and improving the positioning performance of complex backgrounds and fuzzy key points; finally, in the video processing stage, the Kalman filter algorithm is introduced to smooth the positioning trajectory by fusing the key point prediction results of multiple frames, effectively improving the temporal consistency and accuracy of key point detection.

[0065] Embodiment 1

[0066] As shown in the figure, the present embodiment provides a semi-supervised key point positioning method for swallowing radiography analysis, comprising the following steps: Figure 1

[0067] S1, acquiring swallowing radiography images, labeling anatomical key points in the swallowing radiography images, and obtaining labeled data.

[0068] The anatomical key points in the swallowing radiography pictures are labeled based on clinical knowledge to form a small amount of high-quality labeled data set, providing a basis for subsequent model training. Specifically, step S1 comprises the following steps:

[0069] S11, according to professional medical anatomy knowledge and the characteristics of swallowing radiography images, select key positions that can accurately reflect the changes of anatomical structure during swallowing, such as soft palate, hyoid bone, etc., as the anatomical key points to be labeled.

[0070] S12, using professional image labeling tools, experienced medical image professionals accurately label the selected key points on the swallowing radiography pictures, record their coordinate positions and other information, and after strict quality audit, form a small amount of labeled data set with high accuracy and representativeness, providing a reliable supervised data basis for subsequent model training.

[0071] ​S2. Design a joint optimization strategy to integrate the supervised loss of labeled data and the self-supervised consistency loss of unlabeled data in a unified training framework; use unlabeled data to generate pseudo-labels, and dynamically update the pseudo-labels during model training, and jointly train the model using supervised and unsupervised data.

[0072] In some embodiments, step S2 specifically includes the following steps:

[0073] S21. Build a unified training framework that includes a labeled data supervision branch and an unlabeled data self-supervision branch. In the labeled data supervision branch, use a common regression loss function (such as mean squared error loss) to measure the difference between the model prediction results and the true labels of the labeled data, guiding the model to learn the key features in the labeled data.

[0074] S22. In the unlabeled data self-supervision branch, a consistency loss function based on data enhancement is designed. For example, after performing random transformations (such as rotation, flipping, etc.) on the unlabeled data, the model is required to maintain consistency in the prediction results of the data before and after the transformation, so as to explore the potential features in the unlabeled data.

[0075] S23. During the model training process, pseudo labels are generated using the prediction results of unlabeled data. By setting a certain confidence threshold, high-confidence prediction results are selected as pseudo labels, and these pseudo labels are used together with the labeled data in the next round of model training. At the same time, according to the iterative training of the model, the confidence evaluation and selection strategy of the pseudo labels are continuously updated to dynamically improve the reliability of the pseudo labels, thereby realizing the joint effective training of supervised data and unlabeled data, and improving the model's generalization ability and adaptability to complex situations.

[0076] S3. Build a semantic guidance module to assist the model in capturing the features of key point areas more accurately, thereby improving the positioning performance of complex backgrounds and blurred key points.

[0077] Specifically, by leveraging the semantic feature extraction capabilities of the DINOv2 model, a semantic guidance module is constructed to assist the model in more accurately capturing the features of key point areas, thereby improving the positioning performance for complex backgrounds and blurred key points. Step S3 specifically includes the following steps:

[0078] S31. The DINOv2 model is introduced as a pre-trained model into the key point localization framework, and its excellent semantic feature extraction capability is used to extract semantic features from swallowing radiography images.

[0079] S32, based on the extracted semantic features, design a semantic guidance module, align the features extracted by the key point positioning model encoder with the semantic features extracted by DINOv2, so that the model focuses more on the semantic information of the key point area, thereby accurately positioning the key anatomical structure in a complex background and improving the positioning accuracy and robustness of the model.

[0080] S4, introduce Kalman filtering algorithm in the video processing stage, and fuse the key point prediction results of multiple frames to realize key point timing calibration.

[0081] In the video processing stage, Kalman filtering algorithm is introduced, and the key point prediction results of multiple frames are fused to smooth the positioning trajectory, thereby effectively improving the timing consistency and accuracy of key point detection. This method constructs an efficient and accurate key point positioning framework, solves the technical problems such as complex anatomical structure and key point annotation scarcity in swallowing radiography video. In some embodiments, step S4 specifically includes the following steps:

[0082] S41, after performing key point positioning on the swallowing radiography video frame by frame, the key point prediction results of each frame are taken as the input of the Kalman filtering algorithm.

[0083] S42, the Kalman filtering algorithm estimates and predicts the position, velocity and other states of the key points by establishing a state space model, combines the observation value of the current frame (i.e. the key point prediction result of the model) and the state estimation value of the previous frame, and calculates the more accurate and smooth key point position estimation of the current frame, thereby effectively reducing the positioning trajectory jitter and discontinuity caused by factors such as video frame noise and key point detection error, improving the timing consistency and accuracy of key point detection in video sequence, and making the key point positioning results of the entire swallowing radiography video more stable and reliable, providing more accurate data support for subsequent clinical analysis.

[0084] The above method is explained and described in detail in combination with the accompanying drawings and specific embodiments.

[0085] The semi-supervised key point positioning method for swallowing radiography analysis provided in this embodiment specifically includes the following steps:

[0086] Step 1: Determine and label the anatomical key points in the swallowing radiography image.

[0087] In this step, according to medical professional knowledge and clinical experience, the present application determines the anatomical parts closely related to the swallowing process as key points. These parts usually include soft palate, hyoid bone, etc. in the swallowing video. The specific key point categories are: hyoid bone upper convex point, hyoid bone lower convex point, hyoid bone left convex point, soft palate left end point, soft palate right end point, soft palate peak point, second cervical vertebra left lower vertex, fourth cervical vertebra left lower vertex, etc.Figure 2 shown.

[0088] Using professional image annotation tools, experienced medical imaging professionals accurately annotated selected key points on the swallowing radiographs, recording their coordinates and other information. After rigorous quality review, 13,895 annotated images were obtained, providing a reliable supervised data foundation for subsequent model training. In addition, 31,536 unannotated images were collected, providing a richer basis for feature learning in the model.

[0089] Step 2: Design a joint optimization strategy for labeled and unlabeled data.

[0090] First, for the labeled data, we use the heat map as the supervision signal of the key points. Given a labeled image x s and its corresponding annotation key point coordinate set Where K is the number of key points. In this example, K=8. A Gaussian heat map is generated for each key point as a supervision signal:

[0091]

[0092] Among them, (u, v) is the image pixel coordinate, σ is the standard deviation of the Gaussian distribution, which controls the diffusion range of the heat map. The key point heat map predicted by the model is The supervised loss of a single sample is calculated using the mean squared error (MSE) loss function:

[0093]

[0094] This loss guides the model to learn key point features in the labeled data and optimize the accuracy of heat map prediction.

[0095] For unlabeled data, a teacher-student training paradigm is adopted to build a teacher model f T and student model f S For unlabeled images x u Generating input data using weak data enhancement (such as simple preprocessing) and strong data enhancement (such as affine transformation such as rotation) and Bundle Input to the teacher model f T , output the corresponding heat map prediction Then to Perform weak to strong transformation, such as generating pseudo labels through the same affine transformation Correspondingly, the input is strongly enhanced Give the student model f S , the heat map output by the student model is

[0096] To ensure the quality of pseudo labels, only when the maximum response value of the pseudo label exceeds the confidence threshold τ, it is used for student model training. Otherwise, the pseudo label does not participate in training, which can be mathematically represented as:

[0097]

[0098] Here, is an indicator function, when the pseudo label is set to zero and does not participate in training. τ(t) is a dynamic confidence threshold that can be constantly improved with the number of training iterations. Optionally, τ(t) here is linearly improved during the training process, such as:

[0099]

[0100] where t is the current training round, T is the total training round, τ init and τ end represent the initial value and final value of the confidence threshold. Then the loss function for training the student model with unlabeled data is defined as:

[0101]

[0102] Pseudo labels can reflect the complex characteristics of unlabeled data and ensure the reliability of high confidence, thereby effectively supervising the student model to learn key point features under complex transformations.

[0103] Step 3: Construct a semantic guidance module to assist the model in capturing key point region features.

[0104] To improve the model's ability to capture key point region features in swallowing radiography images, this embodiment proposes a semantic guidance module that leverages the semantic feature extraction capabilities of the DINOv2 model to guide the key point positioning model to focus more accurately on key regions. The pre-trained DINOv2 model extracts high-dimensional semantic features F sec from swallowing radiography images x (both labeled and unlabeled images are applicable):

[0105] F sec = f DINOv2 (x)

[0106] where f DINOv2 represents the DINOv2 model. At the same time, the key point positioning model extracts features F enc from swallowing radiography images through its own encoder f model :

[0107] F model = f enc (x)

[0108] The semantic guidance module aligns the semantic features by calculating the cosine similarity of F sec and F model , thereby guiding the model to focus more on the semantic information of the key point region. The calculation formula of the cosine similarity is:

[0109]

[0110] where ||·|| represents the two-norm of the vector. Then the loss function L sec of the semantic auxiliary is defined as:

[0111]

[0112] where N represents the number of pixels on the feature map, and respectively represent the feature vectors of the i-th position.

[0113] Combining the two loss functions in step 2, the total loss function L is:

[0114]

[0115] where λ1 and λ2 are hyperparameters that adjust the weights of the loss function. The stochastic gradient descent algorithm (SGD) is used to optimize the loss function, thereby training an accurate key point positioning model.

[0116] Step 4: Implement key point timing calibration using Kalman filter guided inter-frame key point estimation method.

[0117] To improve the timing consistency and accuracy of video key point positioning, this embodiment introduces a Kalman filter algorithm in the key point detection process, effectively reducing noise interference and positioning errors through multi-frame information fusion and trajectory smoothing processing.

[0118] First, use the key point positioning model to perform frame-by-frame prediction on each frame of the swallowing contrast video, obtaining the two-dimensional coordinate prediction results of n key points in each frame where t represents the frame number and n is the number of key points. z t is input as the observation value of the Kalman filter algorithm. Define the state vector of the key point where the state of each key point includes two-dimensional position and velocity:

[0119]

[0120] where p 1x ,p 1y and v 1x ,v 1yrespectively represent the two-dimensional position and velocity of the i-th keypoint. The state space model consists of a state transition model and an observation model. The state transition model is:

[0121] x t = Fx t-1 + w t-1

[0122] where F is the state transition matrix describing the relationship between position and velocity, e.g.

[0123]

[0124] At represents the time interval between adjacent frames, I 2n and 0 2n are the identity matrix and zero matrix of dimension 2n x 2n, respectively, w t-1 is the process noise satisfying Q is the covariance matrix of the process noise. The observation model is:

[0125] z t = Hx t + v t

[0126] where H is the observation matrix used to extract position information from the state vector, e.g. H = [I 2n 0 2n ], v t is the observation noise satisfying R is the covariance matrix of the observation noise.

[0127] At each frame, the keypoint state is estimated and calibrated using the following steps. The prediction step is:

[0128]

[0129]

[0130] where x is the state estimate of the previous frame, P t-1|t-1 is its covariance matrix. The update step is:

[0131]

[0132] P t|t = (I - K t H)P t|t-1

[0133] where K t is the Kalman gain used to balance the weights of the observation and prediction values.

[0134] The state estimate output by the Kalman filter Position information in the image That is, the key point position after smoothing calibration. Through the time smoothing effect of Kalman filtering, the inter-frame jitter and discontinuity problem is effectively reduced, and the time consistency and accuracy of key point detection can be significantly improved under complex background and noise conditions, providing high-quality key point kinematic parameters for subsequent clinical analysis.

[0135] To sum up, the application provides a semi-supervised key point positioning method for swallowing radiography analysis, which combines joint training of labeled data and unlabeled data to construct an efficient and accurate key point positioning framework. Compared with existing methods, the application introduces a semantic guidance module based on the DINOv2 model to enhance the model's ability to capture key point region features, and optimizes the temporal consistency through Kalman filtering to effectively improve the accuracy and robustness of key point positioning, especially in complex background and blurred target conditions. The application will provide more accurate key point kinematic parameters for clinical scenarios such as swallowing disorder diagnosis.

[0136] Embodiment 2

[0137] The application also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a semi-supervised key point positioning method for swallowing radiography analysis as shown in Figure 1

[0138] It can be understood that the memory can include random access memory (RAM) and read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing various method embodiments described above, etc.; the data storage area can store data created according to the use of the server, etc.

[0139] ​The processor can include one or more processing cores. The processor connects various parts within the entire server by various interfaces and lines, performs various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Alternatively, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with one or several combinations of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs; the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but be implemented by a separate chip.

[0140] Since the electronic device is an electronic device corresponding to the semi-supervised key point positioning method for swallowing radiography analysis of the embodiments of the application, and the principle of solving problems of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described.

[0141] Embodiment 3

[0142] The embodiments of the application also provide a computer readable storage medium, wherein at least one instruction, at least one program, a code set or an instruction set are stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement a semi-supervised key point positioning method for swallowing radiography analysis as shown in Figure 1

[0143] ​Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0144] Since the storage medium is a storage medium corresponding to the semi-supervised key point positioning method for swallowing radiography analysis of the embodiments of the present application, and the problem solving principle of the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0145] Embodiment 4

[0146] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the semi-supervised key point positioning method for swallowing radiography analysis according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Among them, the executable computer program codes or "codes" for executing various embodiments can be written in high-level programming languages such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0147] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so forth.

[0148] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0149] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A semi-supervised keypoint localization method for swallow contrast analysis, characterized in that, The method comprises the following steps: Obtaining a swallowing radiography image, labeling anatomical key points in the swallowing radiography image, and obtaining labeled data; Designing a joint optimization strategy, integrating the supervised loss of the labeled data and the self-supervised consistency loss of the unlabeled data in a unified training framework; generating pseudo labels from the unlabeled data, and dynamically updating the pseudo labels during the model training process; and jointly training the model using supervised data and unsupervised data; A semantic guidance module is constructed to assist the model in more accurately capturing the features of the key point region; In the video processing stage, a Kalman filter algorithm is introduced to fuse the key point prediction results of multiple frames to realize key point timing calibration. The construction of the semantic guidance module for assisting the model in more accurately capturing the features of the key point region comprises: A pre-trained DINOv2 model is used to extract semantic features from the swallowing radiography image; A semantic guidance module is designed to align the semantic features using the features extracted from the swallowing radiography image by the key point positioning model, so that the model focuses more on the semantic information of the key point region, thereby accurately positioning the key anatomical structures in complex backgrounds, improve the positioning accuracy and robustness of the model; The expression of the extracted semantic features is: F sec = f DINOv2 (x) where f DINOv2 represents the DINOv2 model; x is the swallowing contrast image; The key point positioning model encodes the image I through its own encoder f enc extracting features F of the swallowing radiography image model : F model = f enc (x) The semantic guidance module aligns the semantic features by calculating cosine similarity of F sec and F model , so as to guide the model to focus more on the semantic information of the key point area; the calculation formula of the cosine similarity is: where ‖·‖ represents the two-norm of a vector; The semantic-auxiliary loss function L sec is defined as: where N represents the number of pixels on the feature map, and respectively represent the feature vector of the i-th position.

2. The semi-supervised keypoint localization method for swallow radiography analysis according to claim 1, wherein, The acquisition of the swallowing radiography image, the labeling of the anatomical key points in the swallowing radiography image, and the obtaining of the labeled data comprise: According to professional medical anatomical knowledge and the characteristics of the swallowing radiography image, key positions that can accurately reflect the changes in the anatomical structures during swallowing are selected as the anatomical key points that need to be labeled; Using professional image labeling tools, medical image professionals accurately label the selected key points on the swallowing radiography image to form a small amount of labeled data set, providing a reliable supervised data basis for subsequent model training.

3. The semi-supervised keypoint localization method for swallow radiography analysis according to claim 1, wherein, The design of the joint optimization strategy, which integrates the supervised loss of the labeled data and the self-supervised consistency loss of the unlabeled data in a unified training framework, generates pseudo labels from the unlabeled data, and dynamically updates the pseudo labels during the model training process, comprises: A unified training framework containing a labeled data supervision branch and an unlabeled data self-supervision branch is constructed, in the labeled data supervision branch, a regression loss function is used to measure the difference between the model prediction result and the true label of the labeled data to guide the model to learn the key features in the labeled data; In the unlabeled data self-supervision branch, a consistency loss function based on data augmentation is designed to mine the potential features in the unlabeled data; During the model training process, pseudo labels are generated from the prediction results of the unlabeled data, by setting a confidence threshold, the high-confidence prediction results are selected as the pseudo labels, and these pseudo labels are used together with the labeled data to participate in the next round of model training, while the confidence evaluation and selection strategy of the pseudo labels are constantly updated according to the iterative training of the model, dynamically improving the reliability of the pseudo labels, thereby realizing the joint and effective training of the supervised data and the unlabeled data, and improving the generalization ability and adaptability of the model to complex situations.

4. The semi-supervised keypoint localization method for swallow radiography analysis according to claim 3, wherein, For the labeled data, a heat map is used as the supervision signal for the key points, and a labeled image x is given s and its corresponding labeled key point coordinate set where K is the number of key points, and a Gaussian heat map is generated for each key point as a supervision signal: Wherein, (u, v) is the image pixel coordinates, sigma is the standard deviation of Gaussian distribution, and controls the diffusion range of the heat map; The regression loss function is a mean square error loss function; the key point heat map predicted by the model is Calculate the supervised loss of a single sample by a mean square error loss function: In the formula, H k denotes the annotated key points.

5. The semi-supervised keypoint localization method for swallow radiography analysis according to claim 3, wherein, For unlabeled data, the teacher-student training paradigm is adopted, and a teacher model f T and a student model f S are constructed. For unlabelled pictures x u Generating input data using weak data augmentation and strong data augmentation, respectively and data is input into a teacher model f T , and a corresponding heat map prediction is output The heat map prediction is then converted from weak to strong, generating pseudo labels data is input into a student model f S , and the output heat map is Only when the maximum response value of the pseudo label exceeds the confidence threshold τ, it is used for student model training; otherwise, the pseudo label does not participate in training, and the expression is: wherein, is an indicator function, when the pseudo label is set to zero and does not participate in training; τ(t) is a dynamic confidence threshold, t is the current training round.

6. The semi-supervised keypoint localization method for swallow radiography analysis according to claim 5, wherein, The dynamic confidence threshold τ(t) is linearly improved with the training process, and the expression is: In the formula, t is the current training round, T is the total training round, τ init and τ end represent the initial value and final value of the confidence threshold value; The loss function for training the student model using unlabeled data is defined as: Wherein, K is the number of key points.

7. The semi-supervised keypoint localization method for swallow radiography analysis according to claim 1, wherein, The Kalman filtering algorithm is introduced in the video processing stage, and the key point prediction results of multiple frames are fused to realize key point timing calibration, including: After frame-by-frame key point positioning of the swallowing contrast video, the key point prediction result of each frame is taken as the input of the Kalman filtering algorithm; The Kalman filtering algorithm estimates and predicts the state of the key point by establishing a state space model, combines the observation value of the current frame and the state estimation value of the previous frame, calculates the more accurate and smooth key point position estimation of the current frame, thereby effectively reducing the positioning trajectory jitter and discontinuity problem, and improving the timing consistency and accuracy of key point detection in video sequence.

8. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of any one of claims 1 to 7.

Images