A swallowing radiography video micro-motion recognition and positioning method and electronic equipment

Abstract

The application discloses a swallowing radiography video micro-motion recognition and positioning method and electronic equipment, wherein the method comprises the following steps: determining anatomical key points in the swallowing radiography video and extracting a key point sequence, obtaining the spatial position and motion trend of the throat, and providing structured guidance for micro-motion recognition and positioning; extracting appearance features of the swallowing radiography video; fusing the key point sequence and the appearance features, complementing multi-modal information to improve the discrimination ability of a micro-motion recognition and positioning model for fine-grained actions; designing a joint optimization strategy for action classification and positioning, predicting the micro-motion category and its time sequence boundary, and realizing accurate recognition and positioning of swallowing micro-motion. The application constructs a unified analysis framework by fusing video time sequence features and anatomical key point sequence information, guides the model to focus on the key area and period by using the key point sequence, and then realizes accurate micro-motion recognition and positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, computer vision, medical video analysis, etc., and in particular to a swallowing radiography video micro-motion recognition and positioning method and an electronic device. BACKGROUND

[0002] Content understanding of medical videos has a wide range of application scenarios and important application value in real life. With the development of artificial intelligence technology, micro-motion recognition and positioning has important research value and application prospect in the medical field, especially in the diagnosis and treatment of swallowing disorders. Swallowing radiography video, as an important tool for clinical evaluation of swallowing function, can directly record and analyze the swallowing process of patients. However, due to the rapidity and complexity of swallowing motion, manually labeling micro-motions in swallowing videos is both time-consuming and susceptible to subjective interference, making it difficult to meet clinical needs. Therefore, it is inevitable to develop an automatic micro-motion recognition and positioning method for swallowing radiography videos.

[0003] Currently, action recognition technology based on videos usually adopts a deep learning model to extract spatio-temporal features in videos by combining convolutional neural networks (CNN) and recurrent neural networks (RNN) or models based on self-attention mechanisms (such as Transformer), so as to realize the classification and positioning of actions. This kind of method usually decomposes videos into continuous frame sequences, extracts spatial features from them to describe the key information of static images, and captures the dynamic changes of actions by modeling the temporal sequence relationship between frames. In addition, action positioning is usually achieved through time series regression or bounding box regression techniques to accurately locate the time range of action occurrence. These methods have achieved remarkable results in standard action recognition and positioning tasks, but in swallowing radiography videos, they still face specific technical difficulties. On the one hand, swallowing actions are accompanied by complex anatomical structure changes, with small target volumes, large background / noise interference, and blurred contours of organ radiography targets, making it difficult for existing technologies to accurately capture key dynamic features and action boundaries. On the other hand, swallowing micro-motions have fuzzy time sequence boundaries, short duration, and multiple actions occurring simultaneously, further increasing the difficulty of modeling temporal information and fine-grained features. SUMMARY

[0004] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a swallowing radiography video micro-motion recognition and positioning method based on key point sequences, an electronic device, and a medium.

[0005] The first technical solution adopted by the present application is:

[0006] A swallowing radiography video micro-motion recognition and positioning method, comprising the following steps:

[0007] Determine the anatomical key points in the swallowing radiography video and extract the key point sequence, obtain the spatial position and motion trend of the larynx, and provide structured guidance for micro-motion recognition and positioning.

[0008] Extract the appearance features (RGB and optical flow features) of the swallowing radiography video.

[0009] Fuse the key point sequence with the appearance features, and improve the recognition ability of the micro-motion recognition and positioning model for fine-grained actions through multi-modal information complementation.

[0010] Design a joint optimization strategy for action classification and positioning, predict the micro-motion category and its timing boundary, and realize accurate recognition and positioning of swallowing micro-motion.

[0011] Further, the determination of the anatomical key points in the swallowing radiography video and the extraction of the key point sequence to obtain the spatial position and motion trend of the larynx include:

[0012] According to professional medical anatomical atlas and clinical swallowing function evaluation related standards, the key anatomical sites closely related to swallowing action are selected as key points;

[0013] Through image recognition algorithm, the pixel coordinates of the key points are located in each frame of swallowing radiography video image, and then these coordinate information is arranged in sequence according to the time sequence of the video, to obtain the key point sequence;

[0014] Analyze the coordinate changes reflected by the key point sequence to obtain the spatial position changes and motion trend characteristics of the key parts in the swallowing process, which provides detailed and orderly guidance for subsequent recognition and positioning operations.

[0015] Further, the extraction of the appearance features of the swallowing radiography video includes:

[0016] The swallowing radiography video is input into a convolutional neural network by frame, and the RGB features corresponding to each frame of image are extracted to capture the appearance information such as color and texture of the image;

[0017] The optical flow algorithm is used to calculate the pixel motion information between adjacent frames to construct the optical flow field, and then the deep neural network is used to extract the optical flow features to represent the motion of objects in the video;

[0018] The extracted RGB features and optical flow features are spliced in feature dimension to obtain the fused appearance features.

[0019] Further, for the swallowing radiography video, a sliding window method is used to construct the data set.

[0020] Let the total frame number corresponding to the total time length of the swallowing video be N, the length of the sliding window (the number of frames contained) be L, the step (the number of frames each time) be S, and the number of video segments divided by the sliding window n be calculated according to the following formula:

[0021]

[0022] In the formula, represents the floor operation;

[0023] Each divided video segment will be used as a data unit for subsequent extraction of RGB features, and then the RGB features and optical flow features of each frame image in each segment will be obtained.

[0024] Further, the fusion of the key point sequence and the appearance feature improves the recognition ability of the micro-motion recognition and positioning model for fine-grained actions through multi-modal information complementation, including:

[0025] An attention mechanism is adopted to assign different weights to different parts of the appearance feature based on the key region guide information provided by the key point sequence, so that the micro-motion recognition and positioning model focuses more on the features related to the key regions of the swallowing action during the analysis process, and realizes deep fusion of multi-modal information.

[0026] By constructing a training sample set containing a fusion feature vector, the micro-motion recognition and positioning model is trained using a supervised learning method, so that the micro-motion recognition and positioning model fully learns the advantages brought by multi-modal information complementation during the training process, and then improves the recognition ability of the micro-motion details.

[0027] Further, the attention mechanism is adopted to assign different weights to different parts of the appearance feature based on the key region guide information provided by the key point sequence, including:

[0028] For the key point sequence, a graph convolutional neural network (GCN) or a convolutional neural network (CNN) is used to extract key point sequence features

[0029] A cross-attention mechanism is used to fuse the key point sequence features and the appearance feature F i to obtain enhanced key point sequence features and the appearance feature

[0030] Further, the joint optimization strategy of action classification and positioning is designed to predict the micro-motion category and its time boundary, and to realize accurate recognition and positioning of swallowing micro-motions, including:

[0031] A joint loss function is constructed, which includes a classification loss function and a positioning loss function, the classification loss function is used to measure the difference between the predicted micro-motion category and the true category, and the positioning loss function is used to measure the error between the predicted action timing boundary and the actual timing boundary;

[0032] An optimization algorithm is used to optimize the constructed joint loss function, and in the model training process, the parameters of the model are adjusted according to the feedback of the loss value, so that the model gradually converges, and the accuracy of micro-motion category prediction and the precision of timing boundary positioning are improved;

[0033] In the test stage, the swallowing radiography video to be analyzed is input into the trained model, and the category of micro-motion and the corresponding timing boundary information are directly obtained through the model output, so as to realize the accurate identification and positioning of swallowing micro-motion.

[0034] Further, the joint loss function is constructed by the following method:

[0035] The enhanced appearance feature and the key point sequence feature are respectively sent into the same structure of the action recognition and positioning backbone network, and the expression is:

[0036]

[0037] In the formula, Y i 、 is the final output of the model, including the action classification result and the positioning result, Y i 、 corresponds to the output obtained by the enhanced appearance feature and the enhanced key point sequence feature respectively; φ TAL represents the ActionMamba model;

[0038] Focal Loss is used as the classification loss function L cls , and the expression is:

[0039]

[0040] In the formula, p i is the predicted micro-motion category probability distribution, y i is the true category, γ is the focusing parameter, α is the balance factor, and C is the total number of micro-motion categories:

[0041] The distance intersection over union loss function is used to measure the error between the predicted action timing boundary and the actual timing boundary, and the expression of the positioning loss function is:

[0042]

[0043] In the formula, DIoU is a distance intersection ratio; s p is a predicted action start time, e p is an end time, s g is a real action start time, e g is an end time, p is a center point distance, and c is a duration;

[0044] The joint loss function is:

[0045] L = L cls + λL loc

[0046] In the formula, λ is a weight coefficient for balancing the classification loss and the positioning loss.

[0047] Further, the training step of the micro-action recognition and positioning model comprises:

[0048] In each training round, the model parameters are updated using a stochastic gradient descent algorithm according to the loss function value; at the same time, the exponential moving average and gradient clipping techniques are used to stabilize the training process and prevent gradient explosion, and the expression is:

[0049]

[0050] θ ema,t = γθ ema,t-1 + (1-γ)θ t

[0051] In the formula, θ t is the model parameter at the tth iteration, is the gradient of the loss function L with respect to the parameter θ t , G is a gradient threshold, θ ema,t is the exponential moving average parameter at the tth iteration, and γ is a decay rate.

[0052] The second technical solution adopted by the present application is:

[0053] An electronic device, comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement a swallowing contrast video micro-action recognition and positioning method as described above.

[0054] The third technical solution adopted by the present application is:

[0055] A computer readable storage medium, the storage medium stores at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to realize a swallowing radiography video micro-motion recognition and positioning method as described above.

[0056] The fourth technical solution adopted by the present application is:

[0057] A computer program product or computer program, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer readable storage medium. The processor of the computer equipment can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer equipment executes the above method.

[0058] The beneficial effects of the present application are: the present application constructs a unified analysis framework by fusing video timing features and anatomical key point sequence information, uses key point sequences to guide the model to focus on key areas and time periods, and then realizes accurate micro-motion recognition and positioning. In addition, the present application not only focuses on appearance information, but also fully excavates anatomical key points and kinematic information in the swallowing process, effectively solves the problem of small motion target volume and blurred organ contrast target contour in swallowing radiography video. The present application will provide more accurate time parameter analysis for clinical scenes such as swallowing disorder diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments in the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0060] Figure 1 is a step flow chart of a swallowing radiography video micro-motion recognition and positioning method in an embodiment of the present application;

[0061] Figure 2 is an anatomical key point schematic diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0062] Embodiments of the present application are described below in detail with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary only, and are used only for the purpose of explaining the present application, and are not to be understood as limiting the present application. For the step numbers in the following embodiments, they are only set for the convenience of setting out the description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0063] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by the upper, lower, front, rear, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and is not to indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0064] In the description of the present application, the meaning of several is one or more, and the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of indicated technical features.

[0065] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0066] Currently, swallowing radiography video analysis faces specific technical difficulties. On the one hand, swallowing movements are accompanied by complex anatomical structure changes, and the moving target volume is small, the background / noise interference is large, and the organ radiography target contour is blurred, making it difficult for existing technologies to accurately capture key dynamic features and action boundaries. On the other hand, the timing boundary of swallowing micro-movement is fuzzy, the duration is short, and there are multiple actions concurrent, further increasing the difficulty of modeling the timing information and fine-grained features. Existing methods focus on using the RGB modality of the video, only focusing on appearance information, and fail to fully exploit the key anatomical points and kinematics information in the swallowing process, making it difficult to meet the strict requirements of clinical practice for accurate identification and positioning of micro-movements in swallowing radiography videos.

[0067] Based on this, this invention proposes a micro-motion recognition and localization scheme based on key point sequences in swallowing angiography videos. By fusing video temporal features with anatomical key point sequence information to construct a unified analysis framework, the key point sequences guide the model to focus on key regions and time periods, thereby achieving accurate micro-motion recognition and localization. Specifically, firstly, based on clinical knowledge, anatomical key points in the swallowing angiography video are identified and key point sequences are extracted to obtain the spatial location and movement trend of the larynx, providing structured guidance for micro-motion recognition and localization. Secondly, a deep learning model is used to extract the appearance features (RGB and optical flow features) of the swallowing angiography video. Then, the key point sequences and appearance features are fused, and the model's ability to distinguish fine-grained movements is improved through multimodal information complementarity. Finally, a joint optimization strategy for movement classification and localization is designed to predict the micro-motion category and its temporal boundary, achieving accurate recognition and localization of swallowing micro-motions.

[0068] Example 1

[0069] like Figure 1 As shown, this embodiment provides a method for recognizing and locating micro-movements in swallowing contrast imaging videos, to achieve accurate recognition and location of micro-movements in swallowing contrast imaging videos, overcoming the shortcomings of existing technologies in processing such videos, and improving the diagnostic and analytical effects in clinical applications. The method includes the following steps:

[0070] S1. Based on clinical knowledge, identify key anatomical points in swallowing imaging videos and extract key point sequences to obtain the spatial location and movement trend of the larynx, providing structured guidance for micro-movement recognition and localization.

[0071] In some embodiments, step S1 specifically includes the following steps:

[0072] S11. Based on professional medical anatomical atlases and relevant standards for clinical swallowing function assessment, key anatomical sites closely related to swallowing movements are selected as key points. These key points cover the pharynx, esophageal inlet, and other areas.

[0073] S12. Using an image recognition algorithm, locate the pixel coordinates of the above key points in each frame of the swallowing contrast video image, and then arrange these coordinate information sequentially according to the time sequence of the video to extract the key point sequence.

[0074] S13. Analyze the coordinate changes reflected by the key point sequence to obtain the spatial position changes and movement trends of key parts such as the larynx during swallowing, so as to provide detailed and systematic guidance for subsequent identification and positioning operations.

[0075] S2. Use a deep learning model to extract the appearance features (RGB and optical flow features) of the swallowing imaging video.

[0076] In some embodiments, step S2 specifically comprises the following steps:

[0077] S21, select a suitable convolutional neural network (CNN) architecture, input the swallowing contrast video frame by frame into the network, extract the RGB feature map corresponding to each frame of image to capture the color, texture and other appearance information of the image.

[0078] S22, calculate the pixel motion information between adjacent frames using the optical flow algorithm, construct the optical flow field, and then extract the optical flow features based on another deep neural network to represent the motion of objects in the video.

[0079] S23, concatenate the RGB features and optical flow features in the feature dimension to obtain the fused appearance features.

[0080] S3, fuse the key point sequence with the appearance features to improve the recognition ability of the micro-motion recognition and positioning model for fine-grained actions through multi-modal information complementation.

[0081] In some embodiments, step S3 specifically comprises the following steps:

[0082] S31, use an attention mechanism to assign different weights to different parts of the appearance features based on the key region guidance information provided by the key point sequence, so that the model focuses more on the features related to the key regions of the swallowing action during analysis, and realizes deep fusion of multi-modal information.

[0083] S32, construct a training sample set containing fused feature vectors, and use supervised learning to train the classification and positioning model, so that it learns the advantages brought by multi-modal information complementation during training, and further improves the recognition ability of micro-motion details.

[0084] S4, design a joint optimization strategy for action classification and positioning to predict the micro-motion category and its timing boundary, and realize accurate recognition and positioning of swallowing micro-motions.

[0085] In some embodiments, step S4 specifically comprises the following steps:

[0086] S41, construct a joint loss function containing a classification loss function and a positioning loss function, the classification loss function is used to measure the difference between the predicted micro-motion category and the true category, and the positioning loss function is used to measure the error between the predicted action timing boundary and the actual timing boundary;

[0087] S42, use an optimization algorithm (such as stochastic gradient descent) to optimize the constructed joint loss function, and during the model training process, continuously adjust the parameters of the model according to the feedback of the loss value, so that the model gradually converges, and improve the accuracy of micro-motion category prediction and the precision of timing boundary positioning.

[0088] S43. During the testing phase, the swallowing contrast video to be analyzed is input into the trained model. The model output directly obtains the category of micro-movements and the corresponding temporal boundary information, thereby achieving accurate identification and localization of swallowing micro-movements.

[0089] The above method will be explained in detail below with reference to the accompanying drawings and specific embodiments.

[0090] This embodiment provides a method for micro-motion recognition and localization in swallowing angiography videos based on key point sequences, which specifically includes the following steps:

[0091] Step 1: Identify and extract the sequence of anatomical key points from the swallowing imaging video.

[0092] In this step, based on medical expertise and clinical experience, the present invention identifies anatomical locations closely related to the swallowing process as key points. These locations typically include the soft palate and hyoid bone in swallowing videos. Specific key point categories include: suprahyoid prominence, infrahyoid prominence, left hyoid prominence, left end point of the soft palate, right end point of the soft palate, soft palate peak, lower left apex of the second vertebra, and lower left apex of the fourth vertebra, such as... Figure 2 As shown.

[0093] For each frame of the swallowing contrast imaging video, a keypoint localization model is used to locate the positions of the aforementioned keypoints. Specifically, this invention constructs a keypoint dataset for a swallowing contrast imaging dataset based on labeled data, which is used to train the existing keypoint localization model HRNet. Then, this model is used to extract the keypoints for each frame of the swallowing contrast imaging video. Let the i-th keypoint in the j-th frame be keypoint p. ij =(x ij ,y ij ), where x and y represent the horizontal and vertical coordinates of the image, respectively. All keypoints in the j-th frame can be represented as:

[0094] k j =[P 1j ,P 2j ,…,P 8j ]

[0095] The keypoint sequence of an N-frame swallowing angiography video can be represented as:

[0096] k = [k1, k2, ..., k N ]

[0097] Step 2: Extract appearance features (RGB and optical flow features) from swallowing imaging video.

[0098] First, for the available swallowing video, a sliding window method is used to build a data set. Let the total time length of the swallowing video correspond to the frame number N, the length of the sliding window (including the frame number) L, and the step (the frame number of each sliding) S. Then the number of video segments divided by the sliding window n can be calculated according to the following formula:

[0099]

[0100] Where represents the downward rounding operation. Each divided video segment will be used as a data unit for subsequent extraction of RGB features, and then the RGB features and optical flow features of each frame image in each segment can be obtained. Optionally, an I3D model is used as a feature extractor to extract video features. Wherein, the pre-trained RGB I3D model RGB and the pre-trained optical flow I3D model Flow In the process of processing the video segment C i , the RGB feature g i and the optical flow feature f i can be extracted accordingly, and the specific formula is as follows:

[0101] g i = φ RGB (C i ), f i = φ Flow (C i )

[0102] In order to better fuse these two kinds of features, the present application adopts the commonly used feature splicing operation to fuse the RGB and optical flow features to obtain the fused appearance feature F i , and the specific formula is as follows:

[0103] F i = [C i ; f i ]

[0104] Step 3: design a multi-modal information fusion mechanism for fusing the key point sequence and the appearance feature.

[0105] This embodiment uses a graph convolutional neural network (GCN) or a convolutional neural network (CNN) to extract key point sequence features.

[0106] 1) If GCN is adopted, the keypoint sequence needs to be constructed into a graph structure data first, each anatomical keypoint as a node of the graph, and the edges are constructed according to the anatomical correlation and the mutual influence relationship in the swallowing action. Optionally, an ST-GCN model is used to extract the keypoint sequence features. Before extracting the features, the sliding window operation mentioned in step 2 also needs to be performed on the keypoint sequence, and the originally continuous keypoint sequence is divided into multiple sub-sequence segments by sliding window according to the set window length and step size. Then the keypoint sub-sequence segment X i The features are extracted by the ST-GCN model The specific calculation formula is as follows:

[0107]

[0108] 2) If CNN is adopted, the keypoint sequence needs to be arranged into a format suitable for CNN input, such as converting the keypoint coordinates into a heat map form, and then stacking them in time sequence into a format similar to a video. Then the features are extracted by a 3D-CNN network. Optionally, an I3D model φ HM is used to extract X i The corresponding heat map sequence feature H i The specific calculation formula is as follows:

[0109]

[0110] After obtaining the keypoint sequence feature X , the cross-attention mechanism is used to fuse it with the previously obtained appearance feature F i . Specifically, the keypoint sequence feature can be taken as the query (denoted as Q), and the appearance feature as the key (denoted as K) and the value (denoted as V), and vice versa, to build a bidirectional cross-attention path to mine the dual-modal correlation information. Through the use of cross-attention-based Transformer module, the enhanced keypoint sequence feature can be represented as The specific calculation formula is as follows:

[0111]

[0112] Wherein, F Q , W K , and W V are all learnable linear projection layer parameters. Similarly, the enhanced appearance feature F The specific formula is as follows:

[0113]

[0114] Step 4: Design a joint optimization strategy for micro-motion classification and positioning.

[0115] Enhanced appearance features will be achieved and key point sequence features The data are fed into action recognition and localization backbone networks with the same structure, respectively. Optionally, the ActionMamba model φ can be used. TAL The specific calculation formula for obtaining the prediction results is as follows:

[0116]

[0117] Among them, Y i , It is the final output of the model, containing action classification results and localization results, Y i , These correspond to the outputs obtained from enhanced appearance features and enhanced keypoint sequence features, respectively.

[0118] To alleviate class imbalance and focus more on hard-to-classify samples, Focal Loss is used as the classification loss function. Let the predicted probability distribution of micro-action classes be p. i (i represents the category), the true category is y. i (Using one-hot encoding), γ is the focusing parameter (usually taken as 2), α is the balancing factor, and the classification loss is as follows, where C is the total number of micro-action categories:

[0119]

[0120] The Distance Intersection over Union (DIoU) loss function is used to measure the error between the predicted action temporal boundary and the actual temporal boundary. Let the predicted action start time be s. p The end time is e p The actual start time of the action is s. g The end time is e g Given the distance from the center point ρ and the duration c, then:

[0121]

[0122] The positioning loss is expressed as:

[0123] L loc =1-DIoU

[0124] The joint loss function used for the final training is L = L cls +λL loc , where λ is the weighting coefficient that balances the classification loss and the localization loss, which can be determined through experimental adjustment.

[0125] Step 5: Train the micro-motion recognition and localization model.

[0126] In each round of training, the model parameters are updated according to the loss function value of step 4 using the stochastic gradient descent algorithm. At the same time, the exponential moving average (EMA) and gradient clipping techniques are used to stabilize the training process and prevent gradient explosion, and the specific formula is as follows:

[0127]

[0128] θ ema,t =γθ ema,t-1 +(1-γ)θ t

[0129] Where θ t is the model parameter at the tth iteration, is the gradient of the loss function L with respect to the parameter θ t , G is the gradient threshold, θ ema,t is the exponential moving average parameter at the tth iteration, and γ is the decay rate. In the test phase, the swallowing contrast video to be analyzed is input into the trained model, and the class of micro-movement and the corresponding timing boundary information are directly obtained through the model output, realizing accurate identification and positioning of swallowing micro-movement.

[0130] In summary, the swallowing contrast video micro-movement recognition and positioning method based on key point sequence proposed by the present application constructs a unified analysis framework by fusing video timing features and anatomical key point sequence information, and uses key point sequence to guide the model to focus on the key area and period, and then realizes accurate micro-movement recognition and positioning. Compared with the existing method, the present application not only focuses on appearance information, but also fully excavates anatomical key points and kinematic information in the swallowing process, effectively dealing with the problem of small motion target volume and blurred outline of organ contrast targets in the swallowing contrast video. The present application will provide more accurate time parameter analysis for clinical scenes such as swallowing disorder diagnosis.

[0131] Embodiment 2

[0132] The embodiment of the present application also provides an electronic device, which comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize the swallowing contrast video micro-movement recognition and positioning method as shown in Figure 1 .

[0133] It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing various method embodiments described above, and the like; and the data storage area can store data created according to the use of the server, and the like.

[0134] The processor can include one or more processing cores. The processor connects various parts within the entire server through various interfaces and lines, performs various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with a combination of one or more of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs, etc.; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be implemented by a separate chip.

[0135] Since the electronic device is an electronic device corresponding to the swallowing radiography video micro-motion recognition and positioning method of the embodiments of the present application, and the principle of solving problems of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0136] Embodiment 3

[0137] The embodiments of the present application also provide a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement a swallowing radiography video micro-motion recognition and positioning method as shown in Figure 1 ​

[0138] Those skilled in the art can understand that all or part of the steps of various methods of the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0139] Since the storage medium is a storage medium corresponding to the swallowing radiography video micro-motion recognition and positioning method of the embodiments of the present application, and the problem solving principle of the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be described again.

[0140] Embodiment 4

[0141] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the swallowing radiography video micro-motion recognition and positioning method according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Among them, the executable computer program code or "code" for executing various embodiments can be written in a high-level programming language such as Python, C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0142] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so forth.

[0143] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0144] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A method for swallowing contrast video micro-motion recognition and localization, the method comprising: The method comprises the following steps: determine the anatomical key points in the swallowing radiography video and extract the key point sequence, obtain the spatial position and motion trend of the larynx, and provide structured guidance for micro-motion recognition and positioning; extract the appearance features of the swallowing radiography video; fuse the key point sequence and the appearance features, complement the multi-modal information to improve the discrimination ability of the micro-motion recognition and positioning model for fine-grained actions; design a joint optimization strategy for action classification and positioning, predict the micro-motion category and its time sequence boundary, and realize accurate recognition and positioning of swallowing micro-motion; construct a joint loss function comprising a classification loss function and a positioning loss function; use the distance intersection over union loss function to measure the error between the predicted action time sequence boundary and the actual time sequence boundary, and the expression of the positioning loss function is: L loc = 1 - DIoU where DIoU is the distance intersection over union; s p is the predicted action start time, e p is the predicted action end time, s g is the real action start time, e g is the real end time, p is the center point distance, and c is the duration.

2. The method of claim 1, wherein, The method for determining the anatomical key points in the swallowing radiography video and extracting the key point sequence, obtaining the spatial position and motion trend of the larynx, comprises: According to professional medical anatomical atlas and clinical swallowing function evaluation related standards, the key anatomical sites closely related to swallowing action are selected as key points; By using image recognition algorithm, the pixel coordinates of the key points are located in each frame of the swallowing radiography video image, and then these coordinate information is arranged in sequence according to the time sequence of the video to obtain the key point sequence; analyze the coordinate change reflected by the key point sequence, obtain the spatial position change and motion trend characteristics of the key parts in the swallowing process, and provide detailed and orderly guidance basis for subsequent recognition and positioning operation.

3. The method of claim 1, wherein, The method for extracting the appearance features of the swallowing radiography video comprises: input the swallowing radiography video frame by frame into the convolutional neural network to extract the RGB features corresponding to each frame of image; use the optical flow algorithm to calculate the pixel motion information between adjacent frames, construct the optical flow field, and then use the deep neural network to extract the optical flow features to represent the motion of the object in the video; splicing the feature dimensions of the extracted RGB features and optical flow features to obtain the fused appearance features.

4. The method of claim 3, wherein, For the swallowing radiography video, a sliding window method is used to construct the data set; Let the frame number corresponding to the total time length of the swallowing video be N, the length of the sliding window be L, and the step length be S. The number n of video segments divided by the sliding window is calculated according to the following formula: In the formulae, denotes a floor operation; Each divided video segment is used as a data unit for subsequent extraction of RGB features, and then the RGB features and optical flow features of each frame of image in each segment are obtained.

5. The method of claim 1, wherein, The method for fusing the key point sequence and the appearance features, complementing the multi-modal information to improve the discrimination ability of the micro-motion recognition and positioning model for fine-grained actions, comprises: use the attention mechanism to guide the information based on the key region provided by the key point sequence, assign different weights to different parts of the appearance features, make the micro-motion recognition and positioning model focus more on the features related to the key region of the swallowing action in the analysis process, and realize deep fusion of multi-modal information; By constructing a training sample set containing a fusion feature vector, a micro-motion recognition and positioning model is trained in a supervised learning manner, so that the micro-motion recognition and positioning model fully learns the advantages brought by the complementarity of multi-modal information during training, thereby improving the discrimination ability of micro-motion detail features.

6. The method of claim 5, wherein, The attention mechanism is used to assign different weights to different parts of the appearance feature based on the key region guidance information provided by the key point sequence, including: For the key point sequence, a graph convolutional neural network or a convolutional neural network is used to extract key point sequence features Fusing keypoint sequence features with cross attention mechanism with appearance features F i , obtaining enhanced keypoint sequence features and appearance features 7. The method of claim 1, wherein, The joint optimization strategy of action classification and positioning is designed to predict the micro-motion category and its timing boundary, realize accurate identification and positioning of swallowing micro-motions, including: An optimization algorithm is used to optimize the constructed joint loss function, and during the model training process, the parameters of the model are continuously adjusted according to the feedback of the loss value, so that the model gradually converges, and the accuracy of micro-motion category prediction and the precision of timing boundary positioning are improved; In the test stage, the swallowing radiography video to be analyzed is input into the trained model, and the category and corresponding timing boundary information of the micro-motion are directly obtained through the model output, realizing accurate identification and positioning of swallowing micro-motions.

8. The method of claim 7, wherein, The joint loss function is constructed in the following way: The enhanced appearance features and keypoint sequence features are fed into the same structure of action recognition and pose backbone network respectively, expressed as: In the formula, Y i , is the final output of the model, containing the action classification result and the positioning result, Y i , correspond to the outputs obtained by the enhanced appearance features and the enhanced key point sequence features respectively; φ TAL represents the ActionMamba model; Focal Loss is used as the classification loss function L cls , the expression is: where p i is the predicted micro-motion class probability distribution, y i is the true class, γ1is a focus parameter, α is a balancing factor, and C is the total number of micro-motion classes: The joint loss function is: L = L cls + λL loc Where λ is the weight coefficient for balancing the classification loss and the positioning loss.

9. The method of claim 7, wherein, The training steps of the micro-motion recognition and positioning model include: In each round of training, the model parameters are updated using the stochastic gradient descent algorithm according to the loss function value; at the same time, the exponential moving average and gradient clipping techniques are used to stabilize the training process and prevent gradient explosion, and the expression is: θ ema,t = γ2θ ema,t-1 + (1 - γ2)θ t where θtis the model parameter at the tth iteration, t is the model parameter at the tth iteration, is the gradient of the loss function L with respect to the parameter θt, G is the gradient threshold, θtis the model parameter at the tth iteration, t is the model parameter at the tth iteration, ema,t is the exponential moving average parameter at the tth iteration, and γ2is the decay rate.

10. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores at least one program, which is loaded and executed by the processor to implement the method of any one of claims 1-9.

Images