Diaphragm recognition and motion abnormality recognition method based on transformer

Abstract

The application discloses a diaphragm recognition and motion abnormality recognition method based on a Transformer, and comprises the following steps: S1, video sequences of an intercostal attached area linear array acoustic window and a subcostal convex array acoustic window are respectively preprocessed to obtain two acoustic window sequences; S2, the two preprocessed acoustic window sequences are respectively taken as inputs of a diaphragm recognition network, frame-by-frame segmentation results, frame-by-frame key point coordinates, standard acoustic window category distribution, acoustic window quality scores, motion monitoring curves and motion abnormalities are taken as outputs of the diaphragm recognition network, and the diaphragm recognition network is trained to obtain an optimal diaphragm recognition network; and S3, the optimal diaphragm recognition network is adopted to complete diaphragm recognition and motion abnormality recognition. The diaphragm recognition and motion abnormality recognition method can reduce operator dependence by forming an end-to-end closed loop through acoustic window recognition and quality gating, time sequence consistency modeling, displacement field and contour propagation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for diaphragm recognition and motion abnormality recognition based on Transformer. Background Technology

[0002] Diaphragmatic ultrasound is a non-invasive, bedside assessment tool for evaluating diaphragmatic movement and thickness changes. It is an important tool for assessing weaning from mechanical ventilation, stratifying perioperative pulmonary complication risks, and following up on neuromuscular diseases. Two types of complementary standard acoustic windows are commonly used clinically: one is the zone of apposition (ZOA) acoustic window, typically using a linear array probe to measure diaphragmatic thickness and thickness fraction (TF); the other is the subcostal acoustic window, typically using a convex array probe to visualize the diaphragmatic fornix and measure diaphragmatic excursion through the liver (or spleen) acoustic window. In actual acquisition, the diaphragmatic boundaries are easily blurred and the curve is easily jittery due to the selection of the acoustic window, liver / spleen acoustic window obstruction, lung air interference, and differences in operator angle. This leads to insufficient stability in the automatic identification of diaphragmatic dysfunctions (such as weakened movement, paradoxical movement, and asynchrony).

[0003] Diaphragmatic ultrasound is of high frequency and essential use in intensive care, anesthesia and perioperative management, and assessment of respiratory and neuromuscular diseases, offering advantages such as real-time, non-invasive, and bedside availability. In real-world clinical acquisition, to balance anatomical visibility and motion measurement conditions under different acoustic windows, it is often necessary to simultaneously acquire two complementary video sequences: a linear array sequence of the intercostal attachment area for thickness / TF assessment, and a subcostal convex array sequence for dome displacement assessment, with structural identification, curve generation, and abnormality alerts completed within the same workflow.

[0004] The first type of bottleneck stems from the imaging mechanism and noise structure of ultrasound. The inherent speckle noise, occlusion, and artifacts in ultrasound can lead to insufficient contrast at tissue boundaries. The grayscale transitions between the diaphragm and the thoracic / abdominal cavity interface, the lung base / liver and spleen acoustic windows, and adjacent tissues exhibit soft boundary characteristics. This soft boundary characteristic makes segmentation highly dependent on local texture and gradient, further causing boundary breaks, voids, and local missegmentation, making it difficult to maintain a stable output morphology over the long term.

[0005] The second type of bottleneck stems from the non-rigid motion and temporal consistency requirements within the respiratory cycle. Within a single respiratory cycle, the diaphragm simultaneously encompasses inspiratory / expiratory phases, overall dome displacement, and local thickness changes in the attachment zone, with amplitude differences and phase asynchrony between different regions. If frame-by-frame independent inference or weak temporal constraints are employed, the segmentation results are prone to jitter and drift on the time axis. When segmentation and keypoint trajectories are used for motion calculations, these drifts are misinterpreted as actual displacement, resulting in non-physiological peaks, baseline drift, and phase misalignment in displacement / thickness changes and displacement curves, ultimately amplifying into unreliable anomaly warnings.

[0006] The third type of bottleneck stems from the hard constraints of clinical workflow. Motion monitoring relies on a standard acoustic window and measurable quality. The segment definition, direction decomposition, and reference interval of motion monitoring are all assumed to be established under the standard acoustic window coordinate system; when the acoustic window deviates or key structures are not fully displayed, segment projection mismatch and amplitude deviation will be systematically amplified, making it difficult to support the clinical demand for simultaneous data acquisition and prompting.

[0007] The fourth type of bottleneck stems from the lack of complementary utilization and consistency constraints between the two acoustic windows. The two acoustic windows differ in terms of field of view, scale, and noise distribution, and there may be situations where one acoustic window is of insufficient quality while the other is of acceptable quality. Without cross-acoustic window collaborative modeling and quality gating, problems such as inconsistent structure recognition, unclear curve sources, and unstable anomaly prompts with acoustic window switching can easily occur. Summary of the Invention

[0008] Purpose of the invention: The purpose of this invention is to provide a method for diaphragm recognition and motion anomaly recognition based on Transformer. It constructs a spatiotemporal modeling backbone using a Transformer architecture based on an attention mechanism, and realizes the output of diaphragm segmentation and key points, and output of sound window category and quality score for two sound windows respectively. Under the constraints of quality gating and cross-sound window collaboration, it generates a unified motion monitoring curve and anomaly prompts, thereby forming a technical closed loop that is trainable, interpretable and can provide real-time prompts.

[0009] Technical solution: A method for diaphragm recognition and motion abnormality recognition based on Transformer, including the following steps:

[0010] S1, preprocess the video sequences of the linear array acoustic window in the interrib attachment area and the convex array acoustic window under the rib edge to obtain two acoustic window sequences;

[0011] S2, the preprocessed two acoustic window sequences are used as inputs to the diaphragm recognition network, and the frame-by-frame segmentation results, frame-by-frame key point coordinates, standard acoustic window category distribution, acoustic window quality score, motion monitoring curve and motion abnormality are used as outputs to train the diaphragm recognition network to obtain the optimal diaphragm recognition network.

[0012] S3 employs the optimal diaphragm recognition network to complete diaphragm recognition and motion abnormality recognition.

[0013] Furthermore, the diaphragm recognition network includes a dual-path spatiotemporal feature extraction backbone module, an anatomical prior multi-task diaphragm recognition module, a standard acoustic window recognition and quality scoring module, and a curve generation and anomaly scoring module connected in sequence.

[0014] The dual-channel spatiotemporal feature extraction backbone module performs unified time sampling and length alignment on the two acoustic window sequences, and performs intensity standardization on each, so that the two inputs are consistent in grayscale scale and dynamic range, and outputs a global representation.

[0015] The anatomical prior multi-task diaphragm recognition module takes the fused encoded features of the two channels as input, outputs the diaphragm segmentation results and key point coordinates to the two acoustic windows respectively, and applies anatomical priors and context constraints to each channel.

[0016] The standard sound window recognition and quality scoring module takes the global representation output by the dual-channel spatiotemporal feature extraction backbone module as input, and outputs the standard sound window category distribution and sound window quality score of the two sound windows respectively; and generates a fused condition vector based on the standard sound window category distribution and sound window quality score of the two channels, and conditionally modulates the two-channel coded features to achieve sound window compliance judgment, quality gating prompts and cross-sound window feature alignment.

[0017] The curve generation and anomaly scoring module takes the segmentation results, key point coordinates, coding features, and sound window quality scores of the two sound windows as inputs, estimates the displacement fields of the two sound windows respectively, and performs contour and key point recognition to generate two motion monitoring curves. The module then performs curve fusion output based on quality gating weights. Simultaneously, it outputs motion anomaly probability, grading results, and risk scores based on the fused curves, enabling anomaly and risk conclusions to be traced back to the segmentation, key point trajectory, displacement field, and curve evidence chain.

[0018] Furthermore, the specific steps for obtaining the global representation from the dual-path spatiotemporal feature extraction backbone module include:

[0019] S11, divide each frame of image into The patch has the following number of tokens per frame: , No. The road is always there The Each patch is embedded through a linear mapping. and spatiotemporal location encoding Encoding in spatial location Time and location coding Based on this, the sound window identifier is embedded. , and obtain a token with the identity of the voice window; To explicitly distinguish the sources of the two audio windows, we have:

[0020] ,

[0021] ,

[0022] in, The input image is standardized; For the first input image One patch; To flatten; and For embedded parameters; For dimension The token represents;

[0023] S12, perform spatiotemporal consistency modeling for each acoustic window to obtain the spatiotemporal features of the two single acoustic windows. , ;

[0024] S13, receiving information from viewpoint 2 from viewpoint 1, constructing cross-sound window attention. The fusion and update are performed using a residual method to obtain the fused encoded features. The expression is as follows:

[0025] ,

[0026] ,

[0027] ,

[0028] in, , , , It is a learnable matrix; The dimension of the key vector

[0029] Similarly, by receiving information from viewpoint 1 from viewpoint 2, a cross-sound window attention mechanism is constructed. The fused coding features are obtained. .

[0030] Furthermore, the diaphragm segmentation result is the pixel-by-pixel category Softmax probability output by the segmentation head; the process of obtaining the key point coordinates is as follows: first, the key point detection head... The system identifies key anatomical points and outputs a heatmap; then, soft-argmax is used to map the heatmap to continuous coordinates.

[0031] The segmentation loss used by the segmentation head. Cross-entropy loss With Dice loss The combined loss forms the final segmentation loss. Cross-entropy loss characterizes pixel classification consistency, and Dice loss characterizes the overlap between predicted and ground truth regions; the expression is as follows:

[0032] ,

[0033] ,

[0034] ,

[0035] in, For frame indexing; For pixel position; For category; For pixels, the category they belong to The probability of; It is a one-hot truth value; Number of pixels; Number of categories; It is a smoothing constant; , For weights.

[0036] Furthermore, when the segmentation head processes the diaphragm segmentation, it calculates the anatomical context constraint loss within each acoustic window to ensure that the structural output of each acoustic window satisfies the basic anatomical relationship.

[0037] The anatomical context constraints include:

[0038] Non-overlapping constraints The probability of overlap between the diaphragm and the thoracic / abdominal cavity interface is used as a penalty term.

[0039] Includes constraints To ensure that the thoracic / abdominal cavity interface is within the diaphragm's envelope, first perform maximum pooling on the diaphragm probability to form an expanded envelope, and then penalize the probability that the thoracic / abdominal cavity interface falls outside the envelope;

[0040] Key point—segment alignment constraints By binding keypoints and boundary geometry to the same coordinate system, the absolute value of the sign distance field SDF at keypoints is used to penalize deviations, thereby promoting consistency between keypoints and boundary geometry and continuously aligning the segmentation with the keypoints.

[0041] ,

[0042] ,

[0043] ,

[0044] in, For the probability of diaphragm; The probability of the thoracic / abdominal cavity interface; The pooling kernel size controls the tolerance for boundary gray regions; Obtained through bilinear sampling.

[0045] Furthermore, the standard sound window recognition and quality scoring module uses sound window category discrimination and sound window quality quantification as network outputs to form real-time prompts and quality gating at the acquisition end; the sound window category solves the problem of whether the current sound window is in the target sound window and which type of sound window it belongs to, while the sound window quality scoring solves the problem of whether the measurable quality is achieved under the target sound window. Together, they reduce the systemic risk of curve distortion and false alarms.

[0046] The standard acoustic window recognition and quality scoring module maps the acoustic window probability v to a conditional vector and performs channel modulation on the features, enabling automatic alignment of segmentation and motion representation under different acoustic windows. The acoustic window probability v is obtained by fusing the two acoustic window category distributions, and is used to generate unified conditional features and modulate the two encoded features.

[0047] ,

[0048] in, The original feature; For conditional features; ⊙ represents channel-by-channel multiplication; For scaling and bias vectors.

[0049] Furthermore, the curve generation and anomaly scoring module simultaneously receives the segmentation and key point results of two sound windows, and performs gating and fusion by combining the quality scores of the two sound windows: when both channels meet the threshold, two geometric measurement sequences are generated and fused to obtain a unified curve; when only one channel meets the quality standard, it automatically degenerates to outputting the curve using that channel; the steps include the following:

[0050] S41, segmentation guides motion representation;

[0051] For each input acoustic window, the features are weighted and pooled according to the diaphragm probability to obtain the motion token for each frame. Subsequently, the two motion sets were analyzed separately. Temporal modeling is performed to obtain cross-frame latent representations. Simultaneously, a fusion weight determined by the quality scores of the two sound windows is introduced for weighted fusion. When the quality of one channel is lower than the threshold or is a non-standard sound window, its weight is reset to zero and the other channel dominates the output.

[0052] S42, Displacement field estimation;

[0053] The diaphragm region is output through both acoustic windows. arrive Displacement field: First define feature similarity Establish a candidate set of displacements; then use soft-argmax to obtain continuous displacements. Ultimately, the key points are updated with displacement. Continuous displacement of key points Obtained through bilinear sampling;

[0054] S43, monitoring curve generation;

[0055] Geometric measurements are constructed using key points and then mapped to displacement / thickness variation curves: First, the midpoint of the base is defined, then the longitudinal length is defined; the baseline length is selected, and the longitudinal displacement / thickness variation is defined as the value of the curve over time, outputting the curve over the time length; the unified curve output by dual-window fusion is obtained by two quality scoring gates, and when one is invalid, the curve automatically degenerates into the other curve.

[0056] Compared with the prior art, the significant advantages of this invention are as follows:

[0057] 1. This invention forms an end-to-end closed loop through sound window recognition and quality gating, temporal consistency modeling, displacement field and contour propagation, which can reduce operator dependence: it automatically distinguishes between standard sound windows and measurable quality and provides prompts at the acquisition end;

[0058] 2. This invention reduces non-physiological spikes by suppressing frame-by-frame jitter and baseline drift, making periodic measurements of excursion / thickness / TF more consistent across repeated acquisitions and different device domains; it can suppress jitter and drift and improve repeatability and curve stability, while enabling abnormal and risk outputs to be traced back to the evidence chain of segmentation, key points, displacement fields and curves, enhancing interpretability and clinical verifiability. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the overall architecture of the present invention;

[0060] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0061] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0062] This invention addresses the integrated ultrasound acquisition and analysis scenario, proposing an end-to-end jointly trainable method. Using dual-window video sequences as input, the goal is to simultaneously satisfy four engineering requirements within the same learnable representation: structural reliability, window compliance, curve credibility, and anomaly interpretability. Specifically, this includes:

[0063] Dual-sound-window collaborative anatomical prior multi-task diaphragm recognition: Video sequences of two sound windows are preprocessed separately and then fed into a shared spatiotemporal feature extraction backbone. Based on the shared encoded features, the segmentation and key point monitoring results of the sound windows are output separately. Clinical common sense is incorporated into the training objective through shape priors and anatomical relationship constraints, and cross-sound-window consistency constraints are introduced to ensure that the segmentation and key points of the two sound windows remain mutually calibrated under noise, occlusion, and boundary ambiguity conditions, thereby improving structural stability and cross-sound-window consistency from the source.

[0064] Dual-window standard acoustic window recognition and endogenous and conditional modulation of quality scores: Acoustic window classification and quality scores are output for two separate acoustic windows. The quality score serves as a prerequisite for motion monitoring and curve output, forming a gating mechanism that provides real-time prompts at the acquisition end regarding whether each acoustic window meets monitoring conditions. Simultaneously, unified conditional features are generated from the two acoustic window categories and quality scores. Conditional modulation of shared encoded features enables automatic alignment of segmentation, keypoints, and motion estimation across different acoustic windows, supporting interpretable decisions in the quality dimension for subsequent curve fusion and anomaly detection.

[0065] Linking Differentiable Curve Generation and Anomaly Scoring Based on Displacement Field Estimation and Contour Propagation: Under the constraints of two-way structural output and conditional features, a displacement field estimation module is constructed to output the frame-by-frame displacement field of the diaphragm region. In the contour propagation module, the displacement field is used to achieve consistent cross-frame propagation of the diaphragm contour / key points, forming a traceable geometric measurement sequence. Subsequently, displacement / thickness change curves are generated from the propagated geometric quantities, and anomaly probabilities and scores are output on the basis of curve stabilization. At the same time, risk outputs consistent with clinical endpoints (such as the risk of weaning failure and the risk of perioperative pulmonary complications) are given, so that abnormal conclusions can be traced back to displacement field, contour propagation and curve evidence, improving interpretability and clinical trust.

[0066] In this embodiment, the input is defined as a diaphragmatic ultrasound video sequence. Network output is defined as Includes frame-by-frame segmentation results Frame-by-frame key point coordinates Standard section category distribution Cross-section quality score Motion monitoring curve and abnormal motion output .

[0067] ,

[0068] ,

[0069] in, For frame number, Represents the set of real numbers. , These are the dimensions of a single frame image; These are the learnable parameters of the network. This indicates two audio window video sequence inputs, where This indicates the sequence of linear acoustic windows in the intercostal attachment area. This indicates the sub-ribbed convex acoustic window sequence; , This indicates the timing of the first and second audio windows. A single frame image. Indicates Diaphragm recognition network mapping for parameters; , This represents the frame-by-frame segmentation result sequence of the first and second audio windows. , This represents the frame-by-frame keypoint coordinate sequence of the first and second audio windows. , This indicates the standard acoustic window category distribution for the first and second acoustic windows. , This indicates the quality score for the first and second audio windows. This represents the fused motion monitoring curve sequence. This indicates an abnormal motion output.

[0070] like Figure 1 The diagram shown is an overall architecture diagram of the diaphragm recognition and motion abnormality recognition method based on Transformer provided by the present invention. Figure 2 The flowchart of this invention is as follows:

[0071] Step 1: Intensity normalization processing is performed on the diaphragm ultrasound video sequence to obtain the input sequence;

[0072] In this embodiment, the input diaphragmatic ultrasound video sequence is subjected to intensity normalization frame by frame, compressing the grayscale drift caused by differences in device gain and dynamic range to a uniform numerical scale. Simultaneously, numerically stable terms are retained during the normalization process to avoid instability caused by the denominator approaching zero. After this processing, the sensitivity of subsequent spatiotemporal feature extraction to noise and device domain differences is significantly reduced, providing a consistent input distribution for structure recognition and motion monitoring.

[0073] To reduce the impact of differences in equipment, operators, and gain settings on feature assignment, intensity normalization is performed on the two input sequences separately, and acoustic window identification information is introduced into the shared spatiotemporal backbone to achieve joint representation learning of the two sequences.

[0074] Step 2: Input the two input sequences into the constructed diaphragm recognition network. Use the frame-by-frame segmentation results, frame-by-frame key point coordinates, standard acoustic window category distribution, acoustic window quality score, motion monitoring curve and motion abnormality as the output of the diaphragm recognition network to train the diaphragm recognition network and obtain the optimal diaphragm recognition network.

[0075] The diaphragm recognition network consists of a dual-channel spatiotemporal feature extraction backbone module (preferably a dual-channel Transformer), an anatomical prior multi-task diaphragm recognition module, a standard acoustic window recognition and quality scoring module, and a curve generation and anomaly scoring module, all connected sequentially. These modules form a closed loop through feature sharing and conditional modulation. The acoustic window and quality not only output cue signals but also inversely modulate segmentation and motion representation, ensuring consistent alignment of anatomical appearance and motion patterns under different acoustic windows.

[0076] (a) Dual-path spatiotemporal feature extraction backbone module (preferably Transformer)

[0077] This module is designed for dual-window ultrasound video sequence input. It addresses the issues of missing local structural cues when a single-window ultrasound is affected by speckle noise and occlusion; insufficient cross-frame motion continuity within a single-window ultrasound leads to jitter in the structural output on the time axis; and the lack of cross-window information interaction, as different ultrasound windows contain complementary anatomical ultrasound windows and artifact distributions, preventing complementary information from entering the same representation space. In a preferred embodiment, this module uses a Spatio-temporal Transformer as its core and introduces a cross-view fusion module within the backbone. This module outputs shared encoded features based on a progressive logic of first stabilizing the spatio-temporal representation of a single-window ultrasound, then performing cross-window complementary fusion, providing a unified and stable spatio-temporal representation for subsequent segmentation, keypoint estimation, displacement estimation, and curve generation. In other embodiments, the spatio-temporal sequence modeling backbone can also be implemented using convolutional-temporal networks, RNNs / TCNs, or other attention networks, as long as they can perform cross-frame consistency modeling of multi-frame sequences and support cross-window feature interaction.

[0078] The input is defined as a pair of video sequences with two acoustic windows. To ensure temporal consistency in cross-window fusion, this module performs unified time sampling and length alignment on both sequences before they enter the backbone, and performs intensity normalization on each sequence. This ensures that the two inputs are consistent in grayscale and dynamic range, providing a stable numerical basis for subsequent attention calculations. The specific implementation steps are as follows:

[0079] Step 11, Dual-path token construction and sound window identification;

[0080] First, divide each frame of the image into... The patch has the following number of tokens per frame: , No. The road is always there The Each patch is embedded through a linear mapping. The token after adding position encoding is represented as a token sequence. Encoding in spatial location Time and location coding Based on this, the sound window identifier is embedded. You will receive a token with a voice window identity. This is used to explicitly distinguish the sources of two acoustic windows within the dual-path spatiotemporal feature extraction backbone module, avoiding acoustic window confusion during the cross-window fusion stage, and enabling the network to learn the appearance differences and complementary relationships of the same anatomical target in different acoustic windows. The expressions for each parameter are as follows:

[0081] (1)

[0082] (2)

[0083] in, The input image is standardized; For the first input image One patch; To flatten; and For embedded parameters; For dimension The token sequence represents the embedded patch content and embedded spatial, temporal, and sound window identifier information.

[0084] Step 12, Monophonic window spatiotemporal modeling;

[0085] Before cross-window fusion, spatiotemporal consistency modeling is performed within each acoustic window to stabilize the structural semantics and motion continuity of each path. For the... The path sequence is obtained by first performing spatial modeling within each frame. Subsequently, cross-frame temporal modeling is performed at the same spatial location to obtain a temporally enhanced feature representation of that spatial location aggregated along the time dimension, which is used to characterize continuous motion changes within the cardiac cycle. By adopting an organization method of first converging the intra-frame structure and then constraining the inter-frame continuity, each output has strong structural stability and temporal consistency before entering cross-sound window fusion, reducing the risk of noise amplification during cross-sound window interaction.

[0086] (3)

[0087] (4)

[0088] in, This represents spatial attention in Multi-Head Self-Attention, which performs self-attention on all spatial tokens within the same frame to learn the spatial dependencies of structural boundaries. This represents the temporal attention of Multi-HeadSelf-Attention, which performs self-attention on tokens across frames at the same spatial location n, learning temporal consistency and periodic motion dependence. Indicates the first Road sound window at all times Spatial location Input token at the location; Indicates the first Road sound window at all times Intra-frame feature set enhanced with spatial attention Indicates the first Characteristics of a spatial location; Indicates a fixed spatial location The cross-frame enhanced feature sequence obtained by temporal attention aggregation; the superscript in the formula This indicates transpose, used to rearrange the time-dimensional aggregation results back into the feature organization form required for subsequent calculations.

[0089] Step 13: The cross-view information fusion module outputs shared encoded features;

[0090] Obtaining the spatiotemporal characteristics of two mono windows , Subsequently, the diaphragm recognition network introduces a cross-view information fusion module, enabling learnable information exchange between the two audio windows. Among these features... , These represent the single-window spatiotemporal features of acoustic windows 1 and 2 obtained in step 12, respectively, i.e., the two feature sequences entering the cross-view information fusion module. Taking the reception of acoustic window 2 information by acoustic window 1 as an example, a cross-window attention mechanism is constructed. And the fusion update is completed using a residual method to obtain .

[0091] (5)

[0092] (6)

[0093] (7)

[0094] in, , , , It is a learnable matrix; The dimension of the key vector.

[0095] Symmetrically, the cross-window attention of acoustic window 2 receiving information from acoustic window 1 is calculated and obtained. This cross-view information fusion module explicitly injects complementary information—that one acoustic window is visible while the other is occluded or affected by artifacts—into the other's representation, enabling the two features to share key anatomical and motion cues while maintaining their respective acoustic window identities. This provides a consistent and complementary coding basis for subsequent dual-path segmentation and keypoint output.

[0096] After processing by the cross-view information fusion module, the fused encoded features of the two channels are output. , Furthermore, shared coding features are formed for subsequent modules to use uniformly. The organization principle of shared coding features is: cross-sound window complementary information is fused, and the identity of each sound window remains traceable. On the one hand, it provides coding features of corresponding sound windows for two-way segmentation and key points; on the other hand, it provides a more robust spatiotemporal representation of cross-sound windows for displacement estimation, contour propagation and curve generation, thereby achieving a joint improvement of structure recognition and motion monitoring under dual-sound window input conditions.

[0097] (II) Anatomical Priority Multi-Task Diaphragm Recognition Module

[0098] This module employs multi-task learning as its core organizational approach, binding diaphragm segmentation and keypoint detection to the same spatiotemporal features. It further introduces shape priors and anatomical relationship constraints, enabling the two tasks to continuously calibrate against each other during training. The segmentation branch provides continuous boundary contours and regional semantic context constraints, while the keypoint branch provides stable geometric anchors and reference frames. Together, they enhance robustness against noise, occlusion, and blurred boundaries. This anatomical prior multi-task diaphragm recognition module takes the dual-channel encoded features output from the cross-view information fusion module as input, outputting diaphragm segmentation results and keypoint coordinates through two acoustic windows respectively. Anatomical priors and contextual constraints are applied within each channel to ensure the structural stability of the two outputs under noise and occlusion conditions.

[0099] Step 21, Diaphragmatic segmentation head;

[0100] The segmentation head first outputs the pixel-by-pixel category Softmax probability. To simultaneously ensure pixel-level discrimination and region overlap quality, the segmentation loss... Cross-entropy loss With Dice loss The two loss functions are combined to form the final segmentation loss. The cross-entropy loss characterizes the consistency of pixel classification, and the Dice loss characterizes the overlap between the predicted and ground truth regions.

[0101] (8)

[0102] (9)

[0103] (10)

[0104] in, For frame indexing; For pixel position; For category; For pixels, the category they belong to The probability of; It is a one-hot truth value; Number of pixels; Number of categories; It is a smoothing constant; , For weights.

[0105] Step 22, Key Point Detection Head;

[0106] Key point detection points include the diaphragm apex, the left and right attachment points of the diaphragm, and the base of the diaphragm crura. A heat map is generated from each key anatomical point.

[0107] (11)

[0108] To enable key point coordinates to participate in curve generation and end-to-end backpropagation, soft-argmax is used to map the heat dissipation force map into continuous coordinates.

[0109] (12)

[0110] True Value Heatmap By generating a Gaussian distribution from the labeled points, the heatmap loss can be calculated. .

[0111] (13)

[0112] in, For the first Frame features; For key point indexing; pixel position The predicted thermal value; Key point coordinates; It is a pixel coordinate vector; This is the temperature coefficient. Indicates the first The detection head mapping function corresponding to each key point; This represents the true heatmap at the pixel location. The value at that location.

[0113] Step 23, contour smoothing constraint;

[0114] To avoid unreasonable shapes such as breaks, cavities, and jagged edges in diaphragm segmentation, this embodiment transforms the prior assumption that the diaphragm boundary should be continuous and its shape reasonable into an optimizable objective. A signed distance field (SDF) is introduced as an implicit geometric representation, and the SDF regression loss is calculated. :

[0115] (14)

[0116] in, This represents the predicted symbol distance field value of the network's additional output, the true value. Calculated from mask.

[0117] In addition to geometric consistency, a total variational smoothing term is added as a loss. Furthermore, the total variation term is used to suppress high-frequency noise and jagged edges at the boundary. The symbolic distance field SDF is responsible for elevating the boundary from the classification result to a geometric function, while TV (i.e., the smoothing term) is responsible for converging the geometric function to a smooth shape. Together, they reduce breaks, voids, and jagged edges.

[0118] (15)

[0119] in, Number of frames in the sequence; For frame indexing; For pixel position; For the first Frame at pixel position The predicted symbol distance field value at the location; It is a spatial gradient operator used to calculate local changes in the horizontal and vertical directions; It represents the L1 norm, used to constrain the total variation of the sign distance field in space.

[0120] Step 24, dissect the contextual constraints;

[0121] Considering the anatomical relationships of the diaphragm, such as the thoracic / abdominal cavity interface being enveloped by the diaphragm and different tissues not overlapping, three types of probabilistic constraints are introduced to ensure that the output satisfies basic anatomical relationships: non-overlapping constraints, containment constraints, and keypoint-segment alignment constraints. The probability of overlap between the diaphragm and the thoracic / abdominal cavity interface is used as a penalty term; constraints are included. To ensure the thoracic / abdominal cavity interface lies within the diaphragm's envelope, first perform max pooling on the diaphragm probability to form an expanded envelope, then penalize the probability of the thoracic / abdominal cavity interface falling outside the envelope; key point—segment alignment constraint. By binding keypoints and boundary geometry to the same coordinate system, and using the absolute value of the signed distance field SDF at keypoints to penalize deviations, this term promotes the alignment of keypoints with boundary geometry, making the segmentation and keypoints continuously align with each other.

[0122] (16)

[0123] (17)

[0124] (18)

[0125] in, For the probability of diaphragm; The probability of the thoracic / abdominal cavity interface; The pooling kernel size controls the tolerance for boundary gray regions. Obtained through bilinear sampling.

[0126] The aforementioned anatomical context constraint losses (i.e., formulas (16), (17), and (18)) are calculated within each acoustic window by default to ensure that the structural output of each acoustic window satisfies the basic anatomical relationship. The losses related to structural recognition, including segmentation loss, key point detection loss, contour smoothing constraint loss, and anatomical context constraint loss, are calculated separately on the output of the two acoustic windows and are summed into the total loss by summation or weighted summation.

[0127] (III) Standard Sound Window Recognition and Quality Scoring Module

[0128] The reliability of diaphragmatic motion monitoring relies on a standard acoustic window. The standard acoustic window recognition and quality scoring module outputs acoustic window category determination and acoustic window quality quantification as network outputs, forming real-time prompts and quality gating at the acquisition end. Acoustic window category addresses the question of "whether the current location is within the target acoustic window and which category it belongs to," while quality scoring addresses the question of "whether measurable quality is achieved within the target acoustic window." Both work together to reduce the systemic risks of curve distortion and false alarms. This invention performs acoustic window classification and quality assessment separately on the two acoustic windows in a dual-channel video sequence input scenario. The acoustic window categories include: intercostal attachment zone linear array acoustic window and subcostal convex array acoustic window, and outputs acoustic window classification and quality assessment scores respectively for subsequent gating and fusion decisions.

[0129] Standard acoustic window recognition extracts global representations from the spatiotemporal backbone to classify intercostal linear array acoustic windows and subcostal convex array acoustic windows. The acoustic window classification loss is calculated using cross-entropy loss. Based on the acoustic window category classification, the acoustic window quality score outputs a quantified value between [0,1] representing the current sequence's support for motion monitoring. This quality score directly serves workflow gating; if it falls below a threshold, it prompts the physician to adjust the acoustic window, avoiding misleading output curves. The acoustic window quality score loss employs MSE loss, particularly suitable for regression problems. The acoustic window classification loss and quality score loss are calculated on the two outputs respectively and summed or weighted summation into the total loss.

[0130] To create a mutually reinforcing closed loop, the standard acoustic window recognition and quality scoring module maps the acoustic window probability v to a conditional vector and modulates the features through channels, enabling automatic alignment of segmentation and motion representation under different acoustic windows. The acoustic window probability v is obtained by fusing the two acoustic window category distributions, used to generate unified conditional features and modulate the two encoded features.

[0131] (19)

[0132] in, The original feature; For conditional features; ⊙ represents channel-by-channel multiplication; For scaling and bias vectors.

[0133] This modulation mechanism transforms the semantics of acoustic windows into a direct constraint on feature distribution, enabling unified modeling of anatomical appearance priors under different acoustic windows, and further improving segmentation stability and consistency of motion estimation.

[0134] (iv) Curve generation and anomaly scoring module

[0135] The curve generation and anomaly scoring module writes the tracking-measurement-curve process into a differentiable computational graph, ensuring that the curve output has a geometric source and an interpretable path. Unlike the single-window scenario, in the dual-window input scenario, this module simultaneously receives the segmentation and key point results of two windows and combines the quality scores of the two windows for gating and fusion: when both channels meet the threshold, two geometric measurement sequences are generated and fused to obtain a unified curve; when only one channel meets the quality standard, it automatically degenerates to using that channel's curve output, thus ensuring that the curve source is clear and does not jitter when switching windows.

[0136] Step 41, segment and guide the motion representation;

[0137] To focus motion modeling on the diaphragm region, the curve generation and anomaly scoring module performs weighted pooling on the features of each input acoustic window according to the diaphragm probability, obtaining motion tokens for each frame. Subsequently, the two motion sets were analyzed separately. Temporal modeling is performed to obtain cross-frame latent representations. To obtain a unified motion representation, this module introduces a weighted fusion method based on the fusion weights determined by the quality scores of the two sound windows. When the quality of one channel is below the threshold or is a non-standard sound window, its weight is reset to zero and the other channel takes over as the dominant output, thereby achieving a reliable closed loop of quality gating at the acquisition end.

[0138] (20)

[0139] (twenty one)

[0140] in, Location features; For the probability of diaphragm; It is a stable term; It is a time encoder.

[0141] Step 42, displacement field estimation;

[0142] To give the curve an interpretable geometric origin, the diaphragm region is output on both acoustic windows. arrive The displacement field. First, define the feature similarity. Establish a candidate set of displacements. Then use soft-argmax to obtain continuous displacements. Ultimately, the key points are updated with displacement. Continuous displacement of key points It is obtained through bilinear sampling.

[0143] (twenty two)

[0144] (twenty three)

[0145] (twenty four)

[0146] in, These are pixel coordinates; For the candidate set of displacements; The inner product similarity; This is the temperature coefficient.

[0147] Step 43, monitoring curve generation;

[0148] Geometric measurements are constructed using key points and then mapped to displacement / thickness change curves. Taking the midpoint between the diaphragm apex and the diaphragm attachment point as an example, the midpoint of the base is first defined, then the longitudinal length is defined. A reference time such as the end of expiration is selected as the baseline length, and the longitudinal displacement / thickness change is defined as the value of the curve over time, allowing the output of the curve over the specified time length. The unified curve output by dual-window fusion is also obtained by two-way quality scoring gating. When one channel is invalid, the curve automatically degenerates into the other, avoiding noise introduced during fusion.

[0149] (25)

[0150] (26)

[0151] in, It is a moment The longitudinal length, It is a time index. It is a moment Predicted coordinates of the diaphragm vertex and It is a moment The predicted coordinates of the two diaphragm attachment points It is a vector norm, used to calculate the distance between two points. It is a moment The displacement change curve values ​​are taken from the values. It is a stabilizing term used to avoid the denominator being zero or close to zero.

[0152] To output segmental curves, samples are taken from the two diaphragm contours. Contour points Segment set Calculate arc length And define segment displacement / thickness variation This design ensures a one-to-one correspondence between multiple curves and segment contours, facilitating the location of anomaly sources and the generation of interpretable reports.

[0153] (27)

[0154] (28)

[0155] in, It is a moment The The arc length corresponding to each segment It is a segment index. It is a time index. It is the contour sampling point index. It is a moment The Coordinates of each contour sampling point It is the L2 norm, used to calculate the distance between adjacent contour points. It is a moment The The values ​​of the displacement variation curves for each segment are taken. It is a moment The Arc length of each segment Reference time The Arc length of each segment.

[0156] Supervision and morphological constraints are introduced at the curve layer, and regression loss can be applied to each curve and the fusion curve. Constrain the numerical values, then use second-order difference constraints. To suppress jagged edges and noise, ensure physiological continuity, and finally add baseline consistency constraints. The physical meaning of locking the reference time.

[0157] (29)

[0158] (30)

[0159] (31)

[0160] Where N is the number of curve sampling points, The predicted curve at time The value of , The GT curve at time... The truth value of , Absolute operation. , , These are the predicted curve values ​​at adjacent time points.

[0161] Step 44, anomaly scoring;

[0162] Based on the output motion curve, abnormal detection and risk assessment results are provided to indicate the risk of diaphragmatic dysfunction such as weakened diaphragmatic movement, paradoxical movement, and asynchrony. Furthermore, it outputs explainable endpoint-related risks for two types of clinical applications: firstly, risks related to weaning from mechanical ventilation (e.g., weaning failure, re-intubation after extubation, or prolonged ventilation time); secondly, risks related to perioperative complications (e.g., the risk of perioperative pulmonary complications). Abnormality and risk assessments are based on the curve morphology and can be traced back to the displacement field, contour propagation, and curve evidence, forming a traceable chain of "abnormal / risk conclusion - curve evidence - geometric source".

[0163] (32)

[0164] (33)

[0165] in, An abnormal encoder; For pooling operators; For truth labels; This represents the probability of an anomaly. , These are the classification header parameters. It is the Sigmoid function. It is the transpose of the class head weight vector. From time 1 to time 2 The predicted curve sequence, It is the classification head bias parameter.

[0166] This embodiment employs multi-task joint optimization, co-converging structure, acoustic window, curve, and anomaly under the same training objective. The total loss is defined as follows: weight allocation follows the training priority of first ensuring structural reliability, then curve reliability, and finally anomaly reliability. The structure term ensures the stability of boundaries and keypoints; the acoustic window and quality terms ensure that the output curve is within the effective coordinate system and effective quality range; the curve term ensures morphological and physiological continuity; and the anomaly term forms a reliable distinction after the curve stabilizes.

[0167] (34)

[0168] in, , , , , , , Assign task weights; , For curve smoothing and baseline term weights.

Claims

1. A method for diaphragm recognition and motion abnormality recognition based on Transformer, characterized in that, The steps include the following: S1, preprocess the video sequences of the linear array acoustic window in the interrib attachment area and the convex array acoustic window under the rib edge to obtain two acoustic window sequences; S2, the preprocessed two acoustic window sequences are used as inputs to the diaphragm recognition network, and the frame-by-frame segmentation results, frame-by-frame key point coordinates, standard acoustic window category distribution, acoustic window quality score, motion monitoring curve and motion abnormality are used as outputs to train the diaphragm recognition network to obtain the optimal diaphragm recognition network. S3 employs the optimal diaphragm recognition network to complete diaphragm recognition and motion abnormality recognition.

2. The method for diaphragm recognition and motion abnormality recognition based on Transformer according to claim 1, characterized in that, The diaphragm recognition network includes a dual-path spatiotemporal feature extraction backbone module, an anatomical prior multi-task diaphragm recognition module, a standard acoustic window recognition and quality scoring module, and a curve generation and anomaly scoring module connected in sequence. The dual-channel spatiotemporal feature extraction backbone module performs unified time sampling and length alignment on the two acoustic window sequences, and performs intensity standardization on each, so that the two inputs are consistent in grayscale scale and dynamic range, and outputs a global representation. The anatomical prior multi-task diaphragm recognition module takes the fused encoded features of the two channels as input, outputs the diaphragm segmentation results and key point coordinates to the two acoustic windows respectively, and applies anatomical priors and context constraints to each channel. The standard sound window recognition and quality scoring module takes the global representation output by the dual-channel spatiotemporal feature extraction backbone module as input, and outputs the standard sound window category distribution and sound window quality score of the two sound windows respectively; and generates a fused condition vector based on the standard sound window category distribution and sound window quality score of the two channels, and conditionally modulates the two-channel coded features to achieve sound window compliance judgment, quality gating prompts and cross-sound window feature alignment. The curve generation and anomaly scoring module takes the segmentation results, key point coordinates, coding features, and sound window quality scores of the two sound windows as inputs, estimates the displacement fields of the two sound windows respectively, and performs contour and key point recognition to generate two motion monitoring curves. The module then performs curve fusion output based on quality gating weights. Simultaneously, it outputs motion anomaly probability, grading results, and risk scores based on the fused curves, enabling anomaly and risk conclusions to be traced back to the segmentation, key point trajectory, displacement field, and curve evidence chain.

3. The method for diaphragm recognition and motion abnormality recognition based on Transformer according to claim 2, characterized in that, The specific steps for obtaining the global representation from the dual-path spatiotemporal feature extraction backbone module include: S11, divide each frame of image into The patch has the following number of tokens per frame: , No. The road is always there The Each patch is embedded through a linear mapping. and spatiotemporal location encoding Encoding in spatial location Time and location coding Based on this, the sound window identifier is embedded. , and obtain a token with the identity of the voice window; To explicitly distinguish the sources of the two audio windows, we have: ， ， in, The input image is standardized; For the first input image One patch; To flatten; and For embedded parameters; For dimension The token represents; S12, perform spatiotemporal consistency modeling for each acoustic window to obtain the spatiotemporal features of the two single acoustic windows. , ; S13, receiving information from viewpoint 2 from viewpoint 1, constructing cross-sound window attention. The fusion and update are performed using a residual method to obtain the fused encoded features. The expression is as follows: ， ， ， in, , , , It is a learnable matrix; The dimension of the key vector Similarly, by receiving information from viewpoint 1 from viewpoint 2, a cross-sound window attention mechanism is constructed. The fused coding features are obtained. .

4. The method for diaphragm recognition and motion abnormality recognition based on Transformer according to claim 2, characterized in that, The diaphragm segmentation result is the pixel-by-pixel category Softmax probability output by the segmentation head; the process of obtaining keypoint coordinates is as follows: first, the keypoint detection head... Each key anatomical point is analyzed, and a heatmap is output. Then, soft-argmax is used to map the heatmap to continuous coordinates. The segmentation loss used by the segmentation head. Cross-entropy loss With Dice loss The combined loss forms the final segmentation loss. Cross-entropy loss characterizes pixel classification consistency, and Dice loss characterizes the overlap between predicted and ground truth regions; the expression is as follows: ， ， ， in, For frame indexing; For pixel position; For category; For pixels, the category they belong to The probability of; It is a one-hot truth value; Number of pixels; Number of categories; It is a smoothing constant; , For weights.

5. The method for diaphragm recognition and motion abnormality recognition based on Transformer according to claim 4, characterized in that, When the segmentation head processes the diaphragm segmentation, it calculates the anatomical context constraint loss within each acoustic window to ensure that the structural output of each acoustic window satisfies the basic anatomical relationship. The anatomical context constraints include: Non-overlapping constraints The probability of overlap between the diaphragm and the thoracic / abdominal cavity interface is used as a penalty term. Includes constraints To ensure that the thoracic / abdominal cavity interface is within the diaphragm's envelope, first perform maximum pooling on the diaphragm probability to form an expanded envelope, and then penalize the probability that the thoracic / abdominal cavity interface falls outside the envelope; Key point—segment alignment constraints By binding keypoints and boundary geometry to the same coordinate system, the absolute value of the sign distance field SDF at keypoints is used to penalize deviations, thereby promoting consistency between keypoints and boundary geometry and continuously aligning the segmentation with the keypoints. ， ， ， in, For the probability of diaphragm; The probability of the thoracic / abdominal cavity interface; The pooling kernel size controls the tolerance for boundary gray regions; Obtained through bilinear sampling.

6. The method for diaphragm recognition and motion abnormality recognition based on Transformer according to claim 2, characterized in that, The standard sound window recognition and quality scoring module outputs sound window category identification and sound window quality quantification as network outputs, forming real-time prompts and quality gating at the acquisition end; the sound window category solves the problem of whether the current sound window is in the target sound window and which type of sound window it belongs to, while the sound window quality scoring solves the problem of whether the measurable quality is achieved under the target sound window. Together, they reduce the systemic risk of curve distortion and false alarms. The standard acoustic window recognition and quality scoring module maps the acoustic window probability v to a conditional vector and performs channel modulation on the features, enabling automatic alignment of segmentation and motion representation under different acoustic windows. The acoustic window probability v is obtained by fusing the two acoustic window category distributions, and is used to generate unified conditional features and modulate the two encoded features. ， in, The original feature; For conditional features; ⊙ represents channel-by-channel multiplication; , These are the scaling and bias vectors, respectively.

7. The method for diaphragm recognition and motion abnormality recognition based on Transformer according to claim 2, characterized in that, The curve generation and anomaly scoring module simultaneously receives the segmentation and key point results of two sound windows, and performs gating and fusion by combining the quality scores of the two sound windows: when both channels meet the threshold, two geometric measurement sequences are generated and fused to obtain a unified curve; when only one channel meets the quality standard, it automatically degenerates to outputting the curve of that channel. The steps include the following: S41, segmentation guides motion representation; For each input acoustic window, the features are weighted and pooled according to the diaphragm probability to obtain the motion token for each frame. ; Subsequently, the two motion sets were analyzed separately. Temporal modeling is performed to obtain cross-frame latent representations. Simultaneously, a fusion weight determined by the quality scores of the two sound windows is introduced for weighted fusion. When the quality of one channel is lower than the threshold or is a non-standard sound window, its weight is reset to zero and the other channel dominates the output. S42, Displacement field estimation; The diaphragm region is output through both acoustic windows. arrive Displacement field: First define feature similarity Establish a candidate set of displacements; then use soft-argmax to obtain continuous displacements. Ultimately, the key points are updated with displacement. Continuous displacement of key points Obtained through bilinear sampling; S43, monitoring curve generation; Geometric measurements are constructed using key points and then mapped to displacement / thickness variation curves: First, the midpoint of the base is defined, then the longitudinal length is defined; the baseline length is selected, and the longitudinal displacement / thickness variation is defined as the value of the curve over time, outputting the curve over the time length; the unified curve output by dual-window fusion is obtained by two quality scoring gates, and when one is invalid, the curve automatically degenerates into the other curve.

Images