A micro-motion time parameter acquisition method, device and medium

Abstract

The application discloses a kind of micro-motion time learning parameter acquisition method, device and medium, belong to medical video understanding and identification field.Therein method includes: obtaining swallowing radiography video data;Video features are extracted according to swallowing radiography video data;The video features include RGB feature and optical flow feature;The video features extracted are input into attention mechanism of variable window size, and feature enhancement processing is carried out, and visual enhancement feature is output;Visual enhancement feature is input into two-stage micro-motion positioning network from coarse to fine, and the result of swallowing radiography micro-motion positioning and identification is output.The application realizes the automatic positioning on time sequence to swallowing radiography video by using the time sequence action positioning technology in video understanding, obtains the time sequence information of each micro-motion in swallowing process, to realize swallowing radiography intelligent quantitative analysis, provides reliable basis.

Description

Technical Field

[0001] This invention relates to the field of medical video understanding and recognition, and in particular to a method, apparatus and medium for acquiring micro-motion temporal parameters. Background Technology

[0002] Content comprehension of medical videos has wide-ranging applications and significant value in real life. These videos include specialized imaging videos used to assist in diagnosis and assessment, videos used for rehabilitation medical behavior assessment, and surgical videos used as educational materials. Swallowing imaging, as an important examination method for assessing swallowing function and diagnosing swallowing disorders, can currently be broadly divided into three types: qualitative analysis, semi-quantitative analysis, and quantitative analysis. Among these, the temporal parameters of swallowing imaging are one of the important indicators of quantitative analysis, mainly achieved by analyzing the movement trajectory of the swallowing organs and accurately measuring the relevant temporal parameters of the swallowing process.

[0003] However, due to the lack of effective and targeted analysis techniques, current swallowing angiography video-assisted diagnosis mainly relies on doctors' manual frame-by-frame observation and qualitative analysis, which is both time-consuming and laborious, and makes it difficult to guarantee objectivity and accuracy. In addition, swallowing angiography micro-motion recognition is a fine-grained motion recognition task. The range of organ movements is small and there is some overlap between movements, making them difficult to distinguish. Furthermore, each swallowing movement is short in duration (the duration of the movement itself accounts for a small percentage of the entire video) and the length of the movements varies.

[0004] In summary, how to utilize temporal action localization technology in video understanding to automatically locate swallowing angiography videos in time, obtain the temporal information of various micro-movements during swallowing, and achieve automatic measurement of chronological parameters to realize intelligent quantitative analysis of swallowing angiography is a problem that urgently needs to be solved by technicians in this field. Summary of the Invention

[0005] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, device and medium for obtaining micro-motion timing parameters.

[0006] The technical solution adopted in this invention is:

[0007] A method for obtaining micro-motion temporal parameters includes the following steps:

[0008] Acquire swallowing contrast imaging video data;

[0009] Video features are extracted from swallowing contrast imaging video data; the video features include RGB features and optical flow features.

[0010] The extracted video features are input into a variable window size attention mechanism for feature enhancement processing, and visual enhancement features are output.

[0011] Visual enhancement features are input into a two-stage micro-motion localization network, from coarse to fine, and the output is the result of micro-motion localization and recognition during swallowing imaging.

[0012] Furthermore, the acquisition of swallowing contrast imaging video data includes:

[0013] The acquired swallowing contrast imaging video data was cleaned, and video segments were extracted from the swallowing contrast imaging video data using a preset sliding window.

[0014] If a video clip contains at least one complete swallowing process, save the video clip as a valid dataset file; otherwise, discard the video clip.

[0015] Furthermore, the step of extracting video features from swallowing contrast imaging video data includes:

[0016] A 13D model was used as a video feature extractor to extract video features v from swallowing contrast video data.

[0017] Among them, the 13D model, based on the image classification model, dilates 2D convolution to 3D convolution and extracts spatiotemporal features from RGB and optical flow information in two feature streams.

[0018] Furthermore, the step of inputting the extracted video features into a variable-window-size attention mechanism for feature enhancement processing and outputting visually enhanced features includes:

[0019] For each extracted video feature frame, a window of a preset size is used to extract the features of neighboring segments. The features of the video feature frame and the features of the neighboring segments are used as key-value pairs K and V, and the neighboring features are used as query Q. An attention function is used to perform a mapping to obtain the video feature F enhanced with local features. l The calculation formula is as follows:

[0020]

[0021] The LTE algorithm obtains features enhanced with neighboring features through local temporal self-attention coding. The calculation process is as follows:

[0022]

[0023] In the formula, f i Let f be the feature of the i-th feature frame in the video features. w For the feature segment of the neighboring window corresponding to the i-th feature frame, f i l γ represents the feature of the i-th feature frame after enhancement using neighboring features. l ρ l and All are learnable linear mapping functions;

[0024] The features enhanced with global features are obtained through global temporal self-attention encoding (GTE). The calculation process is as follows:

[0025]

[0026] In the formula, f v For the characteristics of the entire video, f i g Let γ be the feature of the i-th feature frame after global feature enhancement. g ρ g and All are learnable linear mapping functions;

[0027] The enhanced features from LTE and GTE are concatenated through Local-Global Temporal Feature Encoding (LGTE) to obtain the visually enhanced feature F, which is enhanced with both local and global features. v .

[0028] Furthermore, a preset variable window size generation module is used to predict the window size threshold th of the current feature frame. i The variable window size generation module consists of an average pooling layer, LeakyReLU, a fully connected layer, and a Sigmoid layer.

[0029] The expression for the variable window size generation module is:

[0030] th i =(W+1) / / 2·Sigmoid(Linear(LeakyReLU(AveragePool(f i ))))

[0031] p=[(W-1) / / 2,,...,1,0,1,...,,(W-1) / / 2]

[0032] e i =th i -p

[0033] W i v =Sigmoid(e i / T)

[0034] Where W is the preset maximum window size, th i The window size threshold with values ​​ranging from [0, (W+1) / / 2] is used to determine the weighted sequence W. i v ; p is the absolute position code, e iThis indicates whether each position within the window is within the predicted window size threshold; T represents the control parameter of the sigmoid function, which is used to evaluate e. i Thresholding is performed to obtain the final weight sequence W. i v Where T is smaller, W i v The closer the encoding is to one-hot encoding.

[0035] Furthermore, the micro-motion localization network is an A2Net model;

[0036] The process of inputting visual enhancement features into a two-stage micro-motion localization network (from coarse to fine) and outputting the results of swallowing angiography micro-motion localization and recognition includes:

[0037] For the input visual enhancement feature F v The A2Net model first uses a convolutional module, which consists of two convolutional layers and a pooling layer. The A2Net model simultaneously predicts the action localization results of both the unanchored and anchored bounding box modules. The unanchored bounding box module generates classification and regression losses, while the anchored bounding box module generates classification, overlap, and regression losses. The action localization results of the unanchored and anchored bounding box modules are then combined, and the final localization result is obtained through a non-maximum suppression algorithm.

[0038] Furthermore, the method for obtaining micro-motion time parameters also includes the following steps:

[0039] For evaluation in the completed dataset, the located candidate boxes are ranked according to their confidence scores, and the top n are used to calculate the final mAP; where AP is the evaluation metric for each action category, and mAP is the average result of AP across multiple classes; n is a positive integer.

[0040] Furthermore, the calculation method for the evaluation index AP is as follows:

[0041] Based on the located anchor boxes, the overlap rate between each anchor box and all ground truth values ​​is calculated. Anchor boxes with an overlap rate greater than a certain threshold are considered positive samples (TP), and those that do not reach the threshold are considered negative samples (FP). All anchor boxes are sorted from highest to lowest confidence level. The number of positive and negative samples in the top n anchor boxes is counted, and the recall and precision are calculated.

[0042] recall = TP / (TP + FN)

[0043] precision = TP / (TP + FP)

[0044] The recall-precision curve is obtained by plotting recall on the horizontal axis and precision on the vertical axis. The area between the recall-precision curve and the horizontal and vertical axes is calculated as the AP calculation result.

[0045] Another technical solution adopted in this invention is:

[0046] A device for acquiring micro-motion time parameters, comprising:

[0047] At least one processor;

[0048] At least one memory for storing at least one program;

[0049] When the at least one program is executed by the at least one processor, the at least one processor performs the method as described above.

[0050] Another technical solution adopted in this invention is:

[0051] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0052] The beneficial effects of this invention are: by utilizing the temporal action localization technology in video understanding, this invention achieves automatic temporal localization of swallowing contrast imaging videos, obtains the temporal information of each micro-movement during swallowing, and provides a reliable basis for realizing intelligent quantitative analysis of swallowing contrast imaging. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the steps of a method for obtaining micro-motion temporal parameters in swallowing contrast imaging video detection according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of a network structure framework for acquiring micro-motion temporal parameters in swallowing contrast imaging video detection according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram illustrating the enhanced attention mechanism features with variable window size in an embodiment of the present invention;

[0057] Figure 4This is a schematic diagram of the variable window size module in an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of the coarse-to-fine motion localization network structure in an embodiment of the present invention.

[0059] Figure 6 This is a visualization result of the acquisition of micro-motion time parameters in a swallowing contrast imaging video detection according to an embodiment of the present invention. Detailed Implementation

[0060] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0061] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0062] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0063] Furthermore, in the description of this invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0064] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0065] Terminology Explanation:

[0066] LGTE: Local-Global Temporal Feature Encoding

[0067] LTE: Localized Temporal Self-Attention Coding

[0068] GTE: Global Temporal Self-Attention Encoding

[0069] like Figure 1 and Figure 2 As shown, this embodiment provides a method for obtaining micro-motion temporal parameters in swallowing contrast imaging video detection, including the following steps:

[0070] S1. Obtain swallowing contrast imaging video data.

[0071] First, the swallowing angiography videos were cleaned, and then a dataset was created using a sliding window method. This yielded the RGB features needed for feature extraction. Next, optical flow features were extracted from the segmented video files. The optical flow data required for feature extraction was then used to recalculate the micro-movements and the start and end times of the complete swallowing process for each segmented video, based on the annotation data from the doctor's data annotation file, using the starting point of the original video for each segmented video. This new data was used as annotation information. The annotation information for each video file was stored as a dictionary in an npz file.

[0072] In this embodiment, the swallowing contrast imaging videos are first cleaned, and then the available swallowing videos are used to create a dataset using a sliding window method. Specifically, a sliding window length of 32 seconds and 64 seconds is used, and a sliding step of 1 / 4 of the sliding window length is used as the sliding step size. Video segments are extracted from the original video files. If a video segment contains at least one complete swallowing process, it is considered a valid dataset file; otherwise, it is discarded. This yields the RGB input data and optical flow input data needed for feature extraction.

[0073] S2. Extract RGB features and optical flow features from the extracted swallowing imaging video clips.

[0074] The swallowing contrast imaging video clip data is input into a pre-trained convolutional neural network for feature extraction. For the pre-trained convolutional neural network, any existing typical convolutional neural network can be selected as the video RGB feature extractor and optical flow feature extractor for feature extraction.

[0075] As an optional implementation, the I3D model is used as the video feature extractor to extract video features v. This model is based on the image classification model, dilating 2D convolutions into 3D convolutions and extracting spatiotemporal features from RGB and optical flow information in two feature streams. During the computation of the 3D convolution, the 2D filter weights are repeated N times along the time dimension, and the final result is normalized by dividing by N. Regarding the temporal and spatial receptive field settings, in the InceptionV1 module, the stride of the first two pooling layers is set to 1 in the temporal dimension and 2x2 in the spatial dimension, while the operator of the last pooling layer is set to 2x7x7. Furthermore, except for the last convolutional layer, normalization and ReLU activation are added after all convolutional operations.

[0076] S3. Input visual features into a variable window size attention mechanism feature enhancement process and output visually enhanced features.

[0077] The extracted features are input into a variable window size attention mechanism module for enhancement. Considering that the swallowing contrast micro-motion recognition task is shorter in duration than the temporal action recognition task on public datasets, and is more sensitive to changes between neighboring frames of the video, a self-attention mechanism is adopted to enhance the video features so that they can better reflect the changes between neighboring frames and the differences with the global data. This enhances the expressive power of the features.

[0078] Enhanced features for attention mechanisms with variable window sizes. Figure 3 The diagram illustrates its structure. For each extracted video feature frame, a window of a specific size is used to extract features from neighboring segments. The features of the feature frame and the features of neighboring segments are treated as key-value pairs K and V, and the neighboring features are treated as queries Q. An attention function is used to perform a mapping, resulting in the video feature F enhanced with local features. l .

[0079]

[0080] Its LTE calculation process is shown in formula (2) below. Where f i Let f be the feature of the i-th feature frame in the video features. w For the feature segment of the neighboring window corresponding to the i-th feature frame, f i l γ represents the feature of the i-th feature frame after enhancement using neighboring features. l ρ l and It is a learnable linear mapping function.

[0081]

[0082] The GTE global feature enhancement feature frame feature is calculated in a similar manner to LTE. The calculation process is shown in the following formula (3), where f v For the characteristics of the entire video, f i g Let γ be the feature of the i-th feature frame after global feature enhancement. g ρ g and It is a learnable linear mapping function.

[0083]

[0084] Ultimately, LGTE will connect the enhanced features from LTE and GTE to obtain feature F, which is enhanced with local and global features. v .

[0085] Considering the varying durations of different micro-actions, using a fixed-size window would limit the ability to model different micro-actions locally. This can be optimized by using a variable-size window module, such as... Figure 4 As shown, this method dynamically learns the length of the region of interest for each feature frame from the features by utilizing the context information of the feature frame. Specifically, for each feature frame f of the video features... i The appropriate unidirectional window size threshold th for the current feature frame is predicted using a module consisting of an average pooling layer, LeakyReLU, a fully connected layer, and a sigmoid function. i The specific calculation formula is as follows:

[0086] th i =(W+1) / / 2·Sigmoid(Linear(LeakyReLU(AveragePool(f i (4)

[0087] Where W is the preset maximum window size, th i The value of W is a window size threshold ranging from [0, (W+1) / / 2]. The weighted sequence W is used for localization. i v The specific calculation expression is as follows:

[0088] p=[(W-1) / / 2,,...,1,0,1,...,,(W-1) / / 2]

[0089] e i =th i -p

[0090] W i v =Sigmoid(e i / T)

[0091] In the above formula, p is the absolute position code, and th is the window size threshold. i Subtracting them, we get e. i This is used to indicate whether each position within the window is within the predicted window size threshold; then the Sigmoid function is used to evaluate e. i Thresholding is performed to obtain the final weight sequence W. i v .

[0092] S4. Input the enhanced features of the swallowing contrast imaging video into a two-stage micro-motion localization network from coarse to fine, and output the results of swallowing contrast imaging micro-motion localization and recognition.

[0093] In swallowing angiography video temporal motion localization, the micro-motions that need to be located are very short, averaging less than 1 second, and there are overlapping areas between these micro-motions, making it difficult to locate their temporal information from the complete video. A coarse-to-fine motion localization mechanism is employed to improve the effectiveness of micro-motion localization; the flowchart is shown below. Figure 5 As shown. Specifically, firstly, a good single-class localizer is trained for each micro-motion category on a training set with relatively short video durations using a pre-trained model. Based on this, a coarse-to-fine motion localization method is adopted: firstly, the complete swallowing process is located from the video, and swallowing candidate boxes are filtered using a confidence threshold. Then, features are re-extracted from the segments corresponding to the swallowing candidate boxes, and the extracted features are input into the single-class localizer to predict the temporal information of the corresponding micro-motion category. Adding the offset of the swallowing segment itself to the predicted candidate box position yields the localization result of the coarse-to-fine two-stage method.

[0094] The A2Net model is mainly used for micro-motion localization in swallowing angiography videos, given the input video features F. v The model first uses a convolutional module. The basic convolutional module consists of two convolutional layers and a pooling layer to reduce channel size and fuse temporal information. The model simultaneously predicts both anchorless and anchored bounding box modules; the former generates classification and regression losses, while the latter generates classification, overlap, and regression losses. Finally, the action localization results from the anchorless and anchored bounding box modules are combined, and a non-maximum suppression algorithm is used to obtain the final localization result.

[0095] The evaluation metric used is mAP. AP is used as the evaluation metric for each action category, while mAP is the average result of AP across multiple classes. For evaluation on the completed dataset, the located candidate boxes are ranked by confidence score, and the top 100 are used to calculate the final mAP. The IOU threshold ranges from 0.1 to 0.7, and is taken in increments of 0.1.

[0096] Specifically, the calculation method for the evaluation index AP is as follows:

[0097] Using the anchor boxes located by the temporal localization model, the overlap rate between each anchor box and all ground truth values ​​is calculated. Anchor boxes with an overlap rate greater than a certain threshold are considered positive samples (TP), and anchor boxes that do not reach the threshold are considered negative samples (FP). All anchor boxes are sorted from high to low according to their confidence level. The number of positive samples and the number of negative samples in the top 100 are counted to calculate the recall and precision sequences.

[0098] recall = TP / (TP + FN)

[0099] precision = TP / (TP + FP)

[0100] Where TP represents the number of positive samples currently counted, FP represents the number of negative samples, and FN represents the number of undetected positive samples currently counted. Finally, by plotting recall on the horizontal axis and precision on the vertical axis, a recall-precision curve can be obtained. The area under each curve, along with the horizontal and vertical axes, is the final AP calculation result.

[0101] Experimental numerical results:

[0102] First, a single classifier is trained for each action category. The localization performance of each single classifier on the test set is shown in Table 1 below.

[0103] Table 1 shows the localization effect of a single-class temporal action in the first stage.

[0104]

[0105] To verify the effectiveness of the coarse-to-fine two-stage localization method, a two-stage experiment was conducted based on the one-stage method. The feature extraction process, hyperparameter settings, and evaluation metrics were the same as in the first stage. The specific data results are shown in Table 2.

[0106] Table 2 shows the positioning effect of the temporal action framework in the two stages from coarse to fine.

[0107]

[0108] To verify the effectiveness of the variable-window-size attention mechanism enhancement method, experiments were conducted by introducing a variable-window-size attention mechanism enhancement method into the two-stage approach of this embodiment. Experimental results show that the method of this embodiment can enhance video features and improve model performance. Specific numerical results are shown in Table 3.

[0109] Table 3 shows the localization effect of the two-stage temporal action in the attention mechanism with a variable window size.

[0110]

[0111] Visualization results

[0112] To further illustrate the effectiveness of this method, the following figure shows the visualization results.

[0113] exist Figure 6 The paper demonstrates the effectiveness of the two-stage localization method using a variable window size attention mechanism. The results show that the method in this embodiment can effectively obtain the start time of each micro-movement for quantitative assessment in swallowing contrast imaging videos, thus achieving the acquisition of micro-movement temporal parameters in swallowing contrast imaging video detection.

[0114] In summary, the method of the present invention can automatically locate swallowing contrast imaging videos in time by utilizing the temporal action localization technology in video understanding, and obtain the temporal information of each micro-movement during swallowing, thus providing a reliable basis for realizing intelligent quantitative analysis of swallowing contrast imaging.

[0115] This embodiment also provides a device for acquiring micro-motion time parameters, including:

[0116] At least one processor;

[0117] At least one memory for storing at least one program;

[0118] When the at least one program is executed by the at least one processor, the at least one processor performs the following: Figure 1 The method shown.

[0119] This embodiment of the micro-motion time parameter acquisition device can execute the micro-motion time parameter acquisition method provided in the method embodiment of the present invention, and can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0120] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0121] This embodiment also provides a storage medium storing instructions or programs that can execute the micro-motion timing parameter acquisition method provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0122] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0123] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0124] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0126] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0127] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0128] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0129] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0130] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for obtaining micro-motion temporal parameters, characterized in that, Includes the following steps: Acquire swallowing contrast imaging video data; Video features are extracted from swallowing contrast imaging video data; the video features include RGB features and optical flow features. The extracted video features are input into a variable window size attention mechanism for feature enhancement processing, and visual enhancement features are output. Visual enhancement features are input into a two-stage micro-motion localization network, from coarse to fine, and the output is the result of micro-motion localization and recognition during swallowing imaging. The micro-motion localization network is the A2Net model; The process of inputting visual enhancement features into a two-stage micro-motion localization network (from coarse to fine) and outputting the results of swallowing angiography micro-motion localization and recognition includes: Visual enhancement features for input The A2Net model first uses a convolutional module, which consists of two convolutional layers and a pooling layer. The A2Net model simultaneously predicts the action localization results of both the unanchored and anchored bounding box modules. The unanchored bounding box module generates classification and regression losses, while the anchored bounding box module generates classification, overlap, and regression losses. The action localization results of the unanchored and anchored bounding box modules are then combined, and the final localization result is obtained through a non-maximum suppression algorithm.

2. The method for obtaining micro-motion time parameters according to claim 1, characterized in that, The acquisition of swallowing contrast imaging video data includes: The acquired swallowing contrast imaging video data was cleaned, and video segments were extracted from the swallowing contrast imaging video data using a preset sliding window. If a video clip contains at least one complete swallowing process, save the video clip as a valid dataset file; otherwise, discard the video clip.

3. The method for obtaining micro-motion time parameters according to claim 1, characterized in that, The extraction of video features based on swallowing contrast imaging video data includes: The I3D model was used as the video feature extractor to extract video features from swallowing contrast imaging video data. ; Among them, the I3D model, based on the image classification model, dilates 2D convolution to 3D convolution and extracts spatiotemporal features from RGB and optical flow information in two feature streams.

4. The method for obtaining micro-motion time parameters according to claim 1, characterized in that, The process of inputting the extracted video features into a variable-window-size attention mechanism for feature enhancement processing and outputting visually enhanced features includes: For each extracted video feature frame, a window of a preset size is used to extract the features of neighboring segments. The features of the video feature frame and the features of the neighboring segments are used as key-value pairs K and V, and the neighboring features are used as query Q. An attention function is then used to perform a mapping to obtain the video features enhanced with local features. The calculation formula is as follows: The LTE algorithm obtains features enhanced with neighboring features through local temporal self-attention coding. The calculation process is as follows: In the formula, The first feature in the video i Features of each feature frame For the first i Each feature frame corresponds to a feature segment of a neighboring window. To enhance the first by using neighbor features i Features of each feature frame , and All are learnable linear mapping functions; The features enhanced with global features are obtained through global temporal self-attention encoding (GTE). The calculation process is as follows: In the formula, Features of the entire video To enhance the first part using global features i Features of each feature frame , and All are learnable linear mapping functions; By using Local-Global Temporal Feature Encoding (LGTE), the features enhanced by LTE and GTE are concatenated to obtain visually enhanced features augmented with local and global features. .

5. The method for obtaining micro-motion time parameters according to claim 4, characterized in that, The window size threshold of the current feature frame is predicted using a preset variable window size generation module. The variable window size generation module consists of an average pooling layer, LeakyReLU, a fully connected layer, and a Sigmoid layer. The expression for the variable window size generation module is: in The preset maximum window size, The range of values ​​is Window size threshold, weighted localization sequence ; For absolute position encoding, It is used to indicate whether each position within the window is within the predicted window size threshold; The control parameters of the sigmoid function are represented by... Sigmoid function pairs Thresholding is performed to obtain the final weight sequence. .

6. The method for obtaining micro-motion time parameters according to claim 1, characterized in that, The method for obtaining micro-motion time parameters also includes the following steps: For evaluation in the completed dataset, the located candidate boxes are ranked according to their confidence scores, and the top n are used to calculate the final mAP; where AP is the evaluation metric for each action category, and mAP is the average result of AP across multiple classes; n is a positive integer.

7. The method for obtaining micro-motion time parameters according to claim 6, characterized in that, The calculation method for the evaluation metric AP is as follows: Based on the located anchor boxes, the overlap rate between each anchor box and all ground truth values ​​is calculated. Anchor boxes with an overlap rate greater than a certain threshold are considered positive samples (TP), and those that do not reach the threshold are considered negative samples (FP). All anchor boxes are sorted from highest to lowest confidence level, and the number of positive and negative samples in the top n anchor boxes is counted to calculate the recall rate. precision : The recall-precision curve is obtained by plotting recall on the horizontal axis and precision on the vertical axis. The area between the recall-precision curve and the horizontal and vertical axes is calculated as the AP calculation result.

8. A device for acquiring micro-motion time parameters, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-7.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-7.

Images