Anatomy-based thyroid ultrasound scan video stream analysis method

Abstract

The application aims to provide an anatomic thyroid ultrasound scanning video flow analysis method, which comprises the following steps: modeling a graph neural network according to an ultrasound image anatomic knowledge; performing image segmentation on a thyroid and peripheral organs by using an adaptive thyroid anatomic reasoning segmentation network; and judging a scanning stage of a current frame by using a time convolution network. The application adaptively learns an anatomic relationship modeling based on a self-attention mechanism of an input feature, thereby better capturing the connection between complex anatomic structures, and initiatively performs ultrasound scanning flow analysis based on a video stream, realizes accurate understanding and stage segmentation of the ultrasound scanning flow, introduces a time-space information fusion stage, and uses a time convolution network (TCN) to enable the model to not only capture spatial features, but also understand dynamic changes in a time sequence, thereby improving the recognition ability of the scanning stage in the ultrasound video stream.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an anatomy-based thyroid ultrasound scanning video process analysis method. Background Art

[0002] Ultrasound has extensive medical applications, including diagnosis and treatment of various diseases. Due to its non-invasive nature, lack of ionizing radiation, high reliability, and low cost, it is one of the most widely used medical imaging technologies worldwide. However, the demand for ultrasound examinations and ultrasound physicians is high, and training an ultrasound physician is time-consuming. This project uses artificial intelligence to analyze the ultrasound scanning process of experienced physicians, enabling young physicians to complete standardized ultrasound scanning procedures and paving the way for the subsequent implementation of autonomous ultrasound scanning.

[0003] With the advent of the big data era, research on the application of machine learning to ultrasound scanning process analysis has rapidly developed. Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) have been particularly extensively studied in this area. For example, CNNs have strong image feature extraction capabilities, making them suitable for spatial feature analysis of ultrasound images. They also feature a simple architecture, high model efficiency, and ease of implementation. RNNs can process sequential data, making them suitable for dynamic process analysis of ultrasound scans. They also preserve long-term dependencies, facilitating diagnosis. However, a thorough analysis of existing technologies reveals that both models have limitations when processing graph-structured data. First, CNNs rely on a grid structure, making it difficult to effectively model the complex and irregular topological positional relationships between organs, making them difficult to directly apply. Second, while RNNs perform well when processing long sequences of data, they struggle to effectively learn and process graphs, which have complex topologies and highly variable edge-node relationships. Summary of the Invention

[0004] The purpose of this invention is to provide an anatomically-based method for analyzing the thyroid ultrasound scan video workflow. This method does not rely on a predefined adjacency matrix, but instead adaptively learns anatomical relationship modeling through a self-attention mechanism based on input features. This method allows the network to dynamically adjust the connection strengths between nodes to reflect the actual anatomical spatial relationships between them, thereby better capturing the connections between complex anatomical structures. It also pioneers video stream-based ultrasound scan workflow analysis, achieving a precise understanding and stage segmentation of the ultrasound scan workflow. This invention introduces a spatiotemporal information fusion stage and employs a temporal convolutional network (TCN), enabling the model to not only capture spatial features but also understand dynamic changes in the temporal sequence, thereby improving the ability to identify scan stages in ultrasound video streams.

[0005] A thyroid ultrasound scanning video process analysis method based on anatomy, comprising:

[0006] Graph neural network modeling based on ultrasound image anatomical knowledge;

[0007] Adopting an adaptive thyroid anatomy reasoning segmentation network to optimize and fine-tune the ultrasound image segmentation results based on the graph neural nodes;

[0008] A temporal convolutional network is used to determine the scanning phase of the current frame.

[0009] Preferably, the graph neural network modeling based on ultrasound image anatomical knowledge includes:

[0010] The ultrasound image is represented as G(V,E), where G represents the ultrasound image, V is the set of points, and E is the set of edges;

[0011] The geometric centers of anatomical structures in ultrasound images are represented as nodes in a graph neural network, and the anatomical spatial position relationships between nodes are represented as edges in the graph neural network.

[0012] Each node i has a feature vector , combining the eigenvectors into a eigenmatrix , where N is the number of nodes and F is the feature dimension of each node;

[0013] Create an N×N adjacency matrix A whose elements Indicates whether there is an edge between node i and node j. If so, it is 1, otherwise it is 0.

[0014] Preferably, the graph neural network modeling based on ultrasound image anatomical knowledge includes:

[0015] Multiply the neighbor node information of the current node A by a coefficient and add it to node A as the supplementary information of node A, which is regarded as a feature update of node A.

[0016] Perform aggregation operations on each node in the ultrasound image and update the features of all ultrasound image nodes;

[0017] Node features are updated by continuously exchanging neighborhood information until a stable equilibrium is reached, at which point the feature vectors of all nodes contain information about their neighboring nodes;

[0018] Use the aggregation function to take the adjacent matrix A to represent the connection between nodes, and take the node's feature vector X as input. The output can capture the relationship by performing relational reasoning on different nodes:

[0019] ;

[0020] Among them, W is a learnable parameter, It is a nonlinear operation.

[0021] Preferably, the step of using an adaptive thyroid anatomy reasoning segmentation network to perform image segmentation according to nodes of a graph neural network comprises:

[0022] The adaptive thyroid anatomy reasoning and segmentation network includes: a backbone module, a neck module, an adaptive relationship graph reasoning module and a segmentation head module;

[0023] Inputting the nodes of the graph neural network into the adaptive thyroid anatomy reasoning and segmentation network;

[0024] The adaptive thyroid anatomy reasoning and segmentation network outputs a thyroid segmentation result.

[0025] Preferably, the neck module is composed of upsampling on a feature pyramid network and downsampling on a path aggregation network for multi-scale feature fusion;

[0026] The adaptive relationship graph reasoning module is embedded between the backbone module and the segmentation head module, and is used to identify and integrate the relationship of local topological structures and describe the local topological relationship blocks of structured constraints;

[0027] The fully convolutional neural network module takes the output feature map of the adaptive relationship graph reasoning module as input, performs pixel-by-pixel classification, and outputs the thyroid segmentation result.

[0028] The backbone module uses CSPDarknet53 as the backbone for feature extraction;

[0029] The ultrasound image of size H×W×C is input into the backbone module, and the backbone network introduces the focus module in the initial stage;

[0030] Overlay width and height information into the channel;

[0031] After the focus module, the feature map passes through the BaseConv, CSP layer, and SPPBottleneck network layer, and outputs three feature maps with sizes of H / 8×W / 8×256, H / 16×W / 16×512, and H / 32×W / 32×1024.

[0032] Preferably, the neck module performs feature fusion on the obtained feature map after the backbone network extracts the features of the image;

[0033] The neck module consists of upsampling on a feature pyramid network and downsampling on a path aggregation network;

[0034] The feature pyramid network fuses deep semantic information features with high-resolution shallow information from top to bottom;

[0035] The path aggregation network supplements the feature pyramid network and introduces a path aggregation method to aggregate shallow feature maps and deep feature maps, and transmit feature information along a specific path to pass the strong positioning features of the low layer up.

[0036] Preferably, the adaptive relationship graph reasoning module includes:

[0037] Message functions and aggregation functions, expressed as:

[0038] ,

[0039] ;

[0040] The adjacency matrix Combine the messages from neighboring nodes v to generate an aggregate representation of the information received by node u, where N is the total number of node vertices and the connection between two adjacent nodes is represented by an edge weight Control, the node feature matrix X is composed of the center point features, denoted as Where C is the feature dimension, and represents the learnable weight matrix;

[0041] Feature concatenation is performed between two node vectors and is calculated as the weighted sum of the node feature representations. The message function and aggregation function formulas are expressed as:

[0042] ,

[0043] ;

[0044] Among them, || represents the connection operation, and the shared and learnable linear transformation weight matrices are applied to each pair of node vectors respectively. ,if and is the feature learned from the closer area, then The value of is very large;

[0045] Feature updates are organized as follows via graph reasoning:

[0046] ;

[0047] in, is a normalized matrix, ReLU is an activation function, K is calculated by a nonlinear transformation function;

[0048] The output of the local topological relation block is sent to the location branch supervised by Dice Loss, and the loss is calculated using the ground truth value of the geometric center and the predicted segmentation value, which is expressed as:

[0049] ;

[0050] and They represent the true sample label value and the model prediction value respectively, and N is the total number of center points.

[0051] Preferably, the scanning stage of using a temporal convolutional network to determine the current frame includes:

[0052] Each frame of the video (x_1, x_2, ..., x_T, T is the number of frames) is encoded in sequence using an encoder to obtain the feature vector E = (e_1, e_2, ..., e_T) of each frame;

[0053] The temporal convolutional network fuses spatial and temporal features through multiple dilated convolutional layers, and performs feature fusion through residual connections and normalization layers. Expressed as:

[0054] ;

[0055] in is the spatial feature at time point t, are the weight, bias and dilation rate of the i-th layer convolution, f is the convolution operation, is the feature at time point t after fusion;

[0056] After each convolutional layer, layer normalization and ReLU activation function are used, and a fully connected layer is used to map the learned features to the final output sequence, which is expressed as:

[0057] ;

[0058] Where F is the combination of convolutional layer and activation function.

[0059] Preferably, the scanning stage of using a temporal convolutional network to determine the current frame includes:

[0060] Use causal convolutional layers to process the encoded feature sequence;

[0061] After each convolutional layer, batch normalization and ReLU activation function are applied;

[0062] The stacked TCN blocks output a feature sequence that incorporates rich spatiotemporal information;

[0063] These feature vectors are then fed into the classification layer for the final classification task, using a softmax function to classify each frame into different stages of the doctor's scan:

[0064] ;

[0065] in is the output of the TCN module, and are the weights and biases of the classification layer, is the classification probability;

[0066] The cross entropy loss function is used to calculate the difference between the predicted probability of each stage and the true probability of the actual stage of the adaptive thyroid anatomy inference segmentation network, which is expressed as:

[0067] ;

[0068] in, is the true label of the i-th category, is the probability that the model predicts the i-th category.

[0069] The present invention provides the following benefits: 1. Topological modeling of anatomical structures in ultrasound images: The geometric centers of anatomical structures in ultrasound images are represented as nodes in a graph neural network (GNN), effectively modeling inter-organ positional information and complex anatomical relationships. The anatomical spatial positional relationships between nodes are represented as edges in the GNN, directly reflecting the true spatial relationships between anatomical structures. 2. Graph Attention Segmentation Network Based on Topological Structure: An adaptive graph attention network is proposed that adaptively learns anatomical relationship modeling based on input features, independent of a predefined adjacency matrix. Through this adaptive learning mechanism, the aggregation process focuses more on related nodes with stronger relationships, enhancing the model's ability to capture complex relationships between anatomical structures. Furthermore, through feature concatenation and weighted summation, the representation of node features is enhanced, enabling the model to more accurately describe the structural constraints between two center points. 3. Scanning process analysis based on segmented videos: A stage for spatiotemporal information fusion, centered on a temporal convolutional network (TCN), is introduced, enabling the model to not only capture spatial features in the image but also understand temporal series features that change over time. This approach enables the model to more accurately identify the various scanning phases in ultrasound video streams, significantly improving its ability to identify the entire ultrasound scanning process. This not only enhances the model's perception of the spatial features of ultrasound images but also enables it to understand dynamic changes in the temporal dimension, providing a powerful tool for automated analysis of ultrasound scanning processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] Figure 1 This is a flow chart of a thyroid ultrasound scanning video process analysis method based on anatomy of the present invention;

[0073] Figure 2 This is a schematic diagram of the structure of a thyroid ultrasound scanning video process analysis system based on anatomy of the present invention;

[0074] Figure 3 Schematic diagram of the topological structure of the thyroid ultrasound image of the present invention;

[0075] Figure 4 Schematic diagram of the structure of the adaptive relationship graph reasoning module of the present invention. DETAILED DESCRIPTION

[0076] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0077] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0078] In addition, the descriptions of "first", "second", etc. in the present invention are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0079] CNNs rely on a grid structure, making it difficult to effectively model the complex and irregular topological positional relationships between organs, making them difficult to apply directly. Secondly, while RNNs are effective at processing long sequences of data, they also struggle to effectively learn and process graphs, which have complex topologies and highly variable relationships between edges and nodes.

[0080] This invention provides an anatomically-based method for analyzing the thyroid ultrasound scan video workflow. This method does not rely on a predefined adjacency matrix, but instead adaptively learns anatomical relationship modeling through a self-attention mechanism based on input features. This method allows the network to dynamically adjust the connection strengths between nodes to reflect the actual anatomical spatial relationships between them, thereby better capturing the connections between complex anatomical structures. It also pioneers video stream-based ultrasound scan workflow analysis, achieving precise understanding and stage segmentation of the ultrasound scan workflow. This invention introduces a spatiotemporal information fusion stage and employs a temporal convolutional network (TCN), enabling the model to not only capture spatial features but also understand dynamic changes in the temporal sequence, thereby improving the ability to identify scan stages in ultrasound video streams.

[0081] A thyroid ultrasound scan video process analysis method based on anatomy, reference Figure 1 and Figure 2 ,include:

[0082] S100, graph neural network modeling based on ultrasound image anatomical knowledge;

[0083] S200, uses an adaptive thyroid anatomy reasoning segmentation network to perform image segmentation based on the nodes of the graph neural network;

[0084] S300, using a temporal convolutional network to determine the scanning stage of the current frame.

[0085] In graph neural networks (GNNs), the neighbor relationships between nodes and edges are considered crucial information, enabling them to capture the spatial structure and relationships within images. This approach is particularly well-suited for analyzing complex network structures in ultrasound images, effectively leveraging the topological information within them. In the context of autonomous thyroid ultrasound examination process analysis, inspired by the GNN architecture, characteristic organs or tissues in ultrasound images (such as the thyroid gland, trachea, muscles, and aorta) are used as nodes, and their anatomical relationships as edges within the graph to model the thyroid anatomy. This approach overcomes the limitations of CNNs and RNNs in processing complex graph-structured data and adapts to complex and irregular topologies.

[0086] Preferably, reference Figure 3, S100, graph neural network modeling based on ultrasound image anatomical knowledge includes:

[0087] The ultrasound image is represented as G(V,E), where G represents the ultrasound image, V is the set of points, and E is the set of edges;

[0088] The geometric centers of anatomical structures in ultrasound images are represented as nodes in a graph neural network, and the anatomical spatial position relationships between nodes are represented as edges in the graph neural network.

[0089] Each node i has a feature vector , combining the eigenvectors into a eigenmatrix , where N is the number of nodes and F is the feature dimension of each node;

[0090] Create an N×N adjacency matrix A whose elements Indicates whether there is an edge between node i and node j. If so, it is 1, otherwise it is 0.

[0091] A graph is composed of nodes and edges, typically represented as G(V,E), where G represents the graph, V is the set of nodes, and E is the set of edges. To effectively model the positional information and complex anatomical relationships between organs, the geometric centers of anatomical structures in ultrasound images are represented as nodes in a graph neural network, and the anatomical spatial positional relationships between nodes are represented as edges in the graph neural network. 1: Sternocleidomastoid muscle; 2: Internal jugular vein; 3: Common carotid artery; 4: Left and right lobes of the thyroid gland; 5: Isthmus of the thyroid gland; 6, 7: Anterior cervical muscles (sternohyoid and sternothyroid); 8: Trachea; 9: Esophagus; 10: Left and right longus colli muscles.

[0092] Preferably, S100, performing graph neural network modeling based on ultrasound image anatomical knowledge includes:

[0093] Multiply the neighbor node information of the current node A by a coefficient and add it to node A as the supplementary information of node A, which is regarded as a feature update of node A.

[0094] Perform aggregation operations on each node in the ultrasound image and update the features of all ultrasound image nodes;

[0095] Node features are updated by continuously exchanging neighborhood information until a stable equilibrium is reached, at which point the feature vectors of all nodes contain information about their neighboring nodes;

[0096] Use the aggregation function to take the adjacent matrix A to represent the connection between nodes, and take the node's feature vector X as input. The output can capture the relationship by performing relational reasoning on different nodes:

[0097] ;

[0098] Among them, W is a learnable parameter, It is a nonlinear operation.

[0099] In graph neural networks, node representations are updated by aggregating information from its neighboring nodes. The aggregation operation multiplies the neighboring node information of the current node A by a coefficient and adds it to node A as supplementary information, representing a feature update for node A. The aggregation operation is performed on each node in the graph, updating the features of all nodes. Node features are updated by continuously exchanging neighborhood information until a stable equilibrium is reached, at which point the feature vectors of all nodes include information about their neighboring nodes.

[0100] Preferably, reference Figure 4 S200 uses an adaptive thyroid anatomy reasoning segmentation network to perform image segmentation based on the nodes of the graph neural network, including:

[0101] The adaptive thyroid anatomy reasoning and segmentation network includes: a backbone module, a neck module, an adaptive relationship graph reasoning module, and a segmentation head module;

[0102] The nodes of the graph neural network are input into the adaptive thyroid anatomy reasoning and segmentation network;

[0103] The adaptive thyroid anatomy reasoning segmentation network outputs the thyroid segmentation result.

[0104] The adaptive thyroid anatomy reasoning segmentation network consists of the backbone, neck, adaptive relationship diagram graph (ARGR) module, and segmentation head. The backbone primarily uses CSPDarknet53 for feature extraction. The neck consists of a feature pyramid network (FPN) upsampling and a path aggregation network (PAN) downsampling, primarily for multi-scale feature fusion. The adaptive relationship diagram graph module is embedded between the neck and head to identify and integrate local topological relationships: local topological relationship blocks that describe structured constraints. The FCN module uses the output feature map of the ARGR module as input and performs pixel-by-pixel classification to ultimately produce the segmentation result.

[0105] Preferably, the neck module consists of upsampling on a feature pyramid network and downsampling on a path aggregation network for multi-scale feature fusion;

[0106] The adaptive relationship graph reasoning module is embedded between the backbone module and the segmentation head module to identify and integrate the relationship of local topological structures and describe the local topological relationship blocks of structured constraints;

[0107] The fully convolutional neural network module takes the output feature map of the adaptive relationship graph reasoning module as input, performs pixel-by-pixel classification, and outputs the thyroid segmentation result.

[0108] The backbone module uses CSPDarknet53 as the backbone for feature extraction;

[0109] The ultrasound image of size H×W×C is input into the backbone module, and the backbone network introduces the focus module in the initial stage;

[0110] Overlay width and height information into the channel;

[0111] After the focus module, the feature map passes through the BaseConv, CSP layer, and SPPBottleneck network layer, and outputs three feature maps with sizes of H / 8×W / 8×256, H / 16×W / 16×512, and H / 32×W / 32×1024.

[0112] CSPDarknet53 was used as the backbone for feature extraction. A thyroid ultrasound image of size H × W × C was fed into the backbone network. A Focus Module was introduced at the initial stage of the backbone network. This module processes the input image in the first step of the network to adapt it for feature extraction. The Focus Module operates by taking a value from every other pixel in the horizontal and vertical directions of the image matrix, generating four separate feature matrices. This operation is equivalent to downsampling the image while simultaneously adding width and height information to the channels, increasing the number of channels and improving information extraction efficiency. After the Focus Module, the feature map passes through several layers, including BaseConv, CSP, and SPPBottleneck. Finally, three feature maps of sizes H / 8 × W / 8 × 256, H / 16 × W / 16 × 512, and H / 32 × W / 32 × 1024 are generated for input to the Neck layer.

[0113] Preferably, after the backbone network extracts the features of the image, the neck module performs feature fusion on the obtained feature map;

[0114] The neck module consists of upsampling on a feature pyramid network and downsampling on a path aggregation network;

[0115] The feature pyramid network integrates deep semantic information features with high-resolution shallow information from top to bottom;

[0116] The path aggregation network complements the feature pyramid network and introduces a path aggregation method to aggregate shallow feature maps and deep feature maps, and transmit feature information along a specific path to pass the strong positioning features of the lower layer.

[0117] After the backbone network extracts the features of the image, the obtained feature map is input to the neck for feature fusion. In our method, the neck part is upsampled by feature pyramid networks (FPN) and path aggregation network (

[0118] The FPN is composed of downsampling from a deep layer aggregation network (PAN). The FPN works top-down, fusing deep semantic features with high-resolution shallow layers, enhancing semantic expression at multiple scales but failing to convey positioning information. PAN addresses this by adding a bottom-up pyramid behind the FPN to complement it. This approach introduces path aggregation, aggregating shallow and deep feature maps and transferring feature information along specific paths to pass strong positioning features from lower layers up. The repeated downsampling of PAN significantly improves model detection accuracy.

[0119] Preferably, the adaptive relationship graph reasoning module includes:

[0120] Message functions and aggregation functions, expressed as:

[0121] ,

[0122] ;

[0123] The adjacency matrix Combine the messages from neighboring nodes v to generate an aggregate representation of the information received by node u, where N is the total number of node vertices and the connection between two adjacent nodes is represented by an edge weight Control, the node feature matrix X is composed of the center point features, denoted as Where C is the feature dimension, and represents the learnable weight matrix;

[0124] In the local topological relationship block, the relationship between anatomical structures in the image is learned to further enhance localization. To ensure sufficient representation power to describe the structural constraints between two center points, we perform feature concatenation between two node vectors, calculated as the weighted sum of the node feature representations. The message function and aggregation function formulas are expressed as:

[0125] ,

[0126] ;

[0127] Among them, || represents the connection operation, and the shared and learnable linear transformation weight matrices are applied to each pair of node vectors respectively. ,if and is the feature learned from the closer area, then The value of is very large;

[0128] To capture pairwise topological information from the multi-head graph attention layer in the local topological relation block, the feature update is organized as follows via graph reasoning:

[0129] ;

[0130] in, is a normalized matrix, ReLU is an activation function, K is calculated using a nonlinear transformation function; to do so, messages are passed by treating the learned correspondences as an adjacency matrix. This enhances relevant features and embeds alignment information into the feature evolution. This enables aggregation to focus more on relevant nodes with "close" relationships.

[0131] The output of the local topological relationship block is sent to the location branch supervised by Dice Loss, which corresponds to the segmentation geometric center and can maximize the similarity between the predicted segmentation result and the true label. The output of the local topological relationship block is sent to the location branch supervised by Dice Loss, and the loss is calculated using the ground truth value and the predicted segmentation value of the geometric center, which is expressed as:

[0132] ;

[0133] and They represent the true sample label value and the model prediction value respectively, and N is the total number of center points.

[0134] To better capture anatomical relationships and improve segmentation accuracy, a graph attention mechanism is introduced to constrain the topological relationships between anatomical structures. It can adaptively learn anatomical relationship modeling through self-attention based on input features, without relying on the predefined adjacency matrix in the main graph convolution layer. The feature maps output by the backbone and neck networks divide the input image into a series of pixel regions, where the center point of each region is responsible for detecting potential anatomical structures and generating corresponding segmentation masks based on the center point features. This structure enables the model to accurately capture anatomical details, thereby achieving high-quality image segmentation. The visual information transmitted between anatomical structures is modeled by measuring the influence from one center point to another based on their visual similarity and calculated as a normalized embedded Gaussian function.

[0135] A fully convolutional network (FCN) converts the last fully connected layer into a convolutional layer, outputting a heatmap of the same size as the feature map and predicting the class probability for each pixel. The heatmap is then upsampled to the original image size. The FCN uses a skip-level architecture to combine deep global information with shallow detail information, and enhances segmentation details through element-wise addition or concatenation. Finally, a 1x1 convolution is used for pixel-by-pixel classification to produce the segmentation result.

[0136] Preferably, in S300, using a temporal convolutional network to determine the scanning stage of the current frame includes:

[0137] Each frame of the video (x_1, x_2, ..., x_T, T is the number of frames) is encoded in sequence using an encoder to obtain the feature vector E = (e_1, e_2, ..., e_T) of each frame;

[0138] The temporal convolutional network fuses spatial and temporal features through multiple dilated convolutional layers, and performs feature fusion through residual connections and normalization layers. Expressed as:

[0139] ;

[0140] in is the spatial feature at time point t, are the weight, bias and dilation rate of the i-th layer convolution, f is the convolution operation, is the feature at time point t after fusion;

[0141] After each convolutional layer, layer normalization and ReLU activation function are used, and a fully connected layer is used to map the learned features to the final output sequence, which is expressed as:

[0142] ;

[0143] Where F is the combination of convolutional layer and activation function.

[0144] In order to achieve accurate understanding and stage segmentation of the ultrasound scanning process, the spatiotemporal information fusion stage was introduced, and the temporal convolutional network (TCN) was used to fuse spatial and temporal features.

[0145] Preferably, in S300, using a temporal convolutional network to determine the scanning stage of the current frame includes:

[0146] Use causal convolutional layers to process the encoded feature sequence;

[0147] After each convolutional layer, batch normalization and ReLU activation function are applied;

[0148] The stacked TCN blocks output a feature sequence that incorporates rich spatiotemporal information;

[0149] These feature vectors are then fed into the classification layer for the final classification task, using a softmax function to classify each frame into different stages of the doctor's scan:

[0150] ;

[0151] in is the output of the TCN module, and are the weights and biases of the classification layer, is the classification probability;

[0152] The cross entropy loss function is used to calculate the difference between the predicted probability of each stage and the true probability of the actual stage of the adaptive thyroid anatomy inference segmentation network, which is expressed as:

[0153] ;

[0154] in, is the true label of the i-th category, is the probability that the model predicts the i-th category.

[0155] The encoded feature sequence is processed using causal convolutional layers to ensure that the model does not utilize future information when predicting the current frame, but is determined only by the current and previous frames. Next, the receptive field is expanded by dilated convolutional layers, allowing the output of each time step to rely on historical information over a longer time range. After each convolutional layer, batch normalization and ReLU activation functions are applied to stabilize the training process and introduce nonlinearity. Residual connections are added to facilitate the training of deep networks, allowing the network to learn more complex features. By stacking multiple such TCN blocks, the network is able to capture more complex spatiotemporal features. Ultimately, the output of the TCN is a feature sequence that incorporates rich spatiotemporal information, providing a strong feature representation for subsequent classification and stage segmentation. These feature vectors are then fed into the classification layer for the final classification task.

[0156] The present invention provides the following benefits: 1. Topological modeling of anatomical structures in ultrasound images: The geometric centers of anatomical structures in ultrasound images are represented as nodes in a graph neural network (GNN), effectively modeling inter-organ positional information and complex anatomical relationships. The anatomical spatial positional relationships between nodes are represented as edges in the GNN, directly reflecting the true spatial relationships between anatomical structures. 2. Graph Attention Segmentation Network Based on Topological Structure: An adaptive graph attention network is proposed that adaptively learns anatomical relationship modeling based on input features, independent of a predefined adjacency matrix. Through this adaptive learning mechanism, the aggregation process focuses more on related nodes with stronger relationships, enhancing the model's ability to capture complex relationships between anatomical structures. Furthermore, through feature concatenation and weighted summation, the representation of node features is enhanced, enabling the model to more accurately describe the structural constraints between two center points. 3. Scanning process analysis based on segmented videos: A stage for spatiotemporal information fusion, centered on a temporal convolutional network (TCN), is introduced, enabling the model to not only capture spatial features in the image but also understand temporal series features that change over time. This approach enables the model to more accurately identify the various scanning phases in ultrasound video streams, significantly improving its ability to identify the entire ultrasound scanning process. This not only enhances the model's perception of the spatial features of ultrasound images but also enables it to understand dynamic changes in the temporal dimension, providing a powerful tool for automated analysis of ultrasound scanning processes.

[0157] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A thyroid ultrasound scanning video process analysis method based on anatomy, characterized in that: include: Graph neural network modeling based on ultrasound image anatomical knowledge; Adaptive thyroid anatomy reasoning and segmentation network is used to segment the thyroid gland and surrounding organs. Use temporal convolutional networks to determine the scanning phase of the current frame; The adaptive thyroid anatomy reasoning and segmentation network includes: a backbone module, a neck module, an adaptive relationship graph reasoning module and a segmentation head module; Inputting the nodes of the graph neural network into the adaptive thyroid anatomy reasoning and segmentation network; The adaptive thyroid anatomy reasoning and segmentation network outputs a thyroid segmentation result; The neck module is composed of upsampling on a feature pyramid network and downsampling on a path aggregation network for multi-scale feature fusion; The adaptive relationship graph reasoning module is embedded between the backbone module and the segmentation head module, and is used to identify and integrate the relationship of local topological structures and describe the local topological relationship blocks of structured constraints; The fully convolutional neural network module takes the output feature map of the adaptive relationship graph reasoning module as input, performs pixel-by-pixel classification, and outputs the thyroid segmentation result; Among them, the backbone module uses CSPDarknet53 as the backbone of feature extraction; The ultrasound image of size H×W×C is input into the backbone module, and the backbone network introduces the focus module in the initial stage; Overlay width and height information into the channel; After the focus module, the feature map passes through the BaseConv, CSP layer, and SPPBottleneck network layer, and outputs three feature maps with sizes of H / 8×W / 8×256, H / 16×W / 16×512, and H / 32×W / 32×1024; The adaptive relationship graph reasoning module includes: Message functions and aggregation functions, expressed as: ， ； In the formula, the adjacency matrix Combine the messages from neighboring nodes v to generate an aggregate representation of the information received by node u, where N is the total number of node vertices and the connection between two adjacent nodes is represented by an edge weight Control, the node feature matrix X is composed of the center point features, denoted as Where C is the feature dimension, and represents the learnable weight matrix; Feature concatenation is performed between two node vectors and is calculated as the weighted sum of the node feature representations. The message function and aggregation function formulas are expressed as: ， ； Where || represents the connection operation, and the sharable and learnable linear transformation weight matrices are applied to each pair of node vectors respectively. ,if and is the feature learned from the closer area, then The value of is very large; Feature updates are organized as follows via graph reasoning: ； Where, is a normalized matrix, ReLU is an activation function, K is calculated by a nonlinear transformation function; The output of the local topological relation block is sent to the location branch supervised by Dice Loss, and the loss is calculated using the ground truth value of the geometric center and the predicted segmentation value, which is expressed as: ； and They represent the true sample label value and the model prediction value respectively, and N is the total number of center points.

2. The anatomy-based thyroid ultrasound scanning video process analysis method according to claim 1, characterized in that: The graph neural network modeling based on ultrasound image anatomical knowledge includes: The ultrasound image is represented as G(V,E), where G represents the ultrasound image, V is the set of points, and E is the set of edges; The geometric centers of anatomical structures in ultrasound images are represented as nodes in a graph neural network, and the anatomical spatial position relationships between nodes are represented as edges in the graph neural network. Each node i has a feature vector , combining the eigenvectors into a eigenmatrix , where N is the number of nodes and F is the feature dimension of each node; Create an N×N adjacency matrix A whose elements Indicates whether there is an edge between node i and node j. If so, it is 1, otherwise it is 0.

3. The anatomy-based thyroid ultrasound scan video process analysis method according to claim 2, characterized in that: The graph neural network modeling based on ultrasound image anatomical knowledge includes: Multiply the neighbor node information of the current node A by a coefficient and add it to node A as the supplementary information of node A, which is regarded as a feature update of node A. Perform aggregation operations on each node in the ultrasound image and update the features of all ultrasound image nodes; Node features are updated by continuously exchanging neighborhood information until a stable equilibrium is reached, at which point the feature vectors of all nodes contain information about their neighboring nodes; Use the aggregation function to take the adjacent matrix A to represent the connection between nodes, and take the node's feature vector X as input. The output can capture the relationship by performing relational reasoning on different nodes: ； Among them, W is a learnable parameter, It is a nonlinear operation.

4. The method for analyzing the thyroid ultrasound scan video process based on anatomy according to claim 1, characterized in that: After the backbone network extracts the features of the image, the neck module performs feature fusion on the obtained feature map; The neck module consists of upsampling on a feature pyramid network and downsampling on a path aggregation network; The feature pyramid network fuses deep semantic information features with high-resolution shallow information from top to bottom; The path aggregation network supplements the feature pyramid network and introduces a path aggregation method to aggregate shallow feature maps and deep feature maps, and transmit feature information along a specific path to pass the strong positioning features of the low layer up.

5. The method for analyzing the thyroid ultrasound scan video process based on anatomy according to claim 1, characterized in that: The scanning stage of using the temporal convolutional network to determine the current frame includes: Each frame of the video (x_1, x_2, ..., x_T, T is the number of frames) is encoded in sequence using an encoder to obtain the feature vector E = (e_1, e_2, ..., e_T) of each frame; The temporal convolutional network fuses spatial and temporal features through multiple dilated convolutional layers, and performs feature fusion through residual connections and normalization layers. Expressed as: ； in is the spatial feature at time point t, are the weight, bias and dilation rate of the i-th layer convolution, f is the convolution operation, is the feature at time point t after fusion; After each convolutional layer, layer normalization and ReLU activation function are used, and a fully connected layer is used to map the learned features to the final output sequence, which is expressed as: ； Where F is the combination of convolutional layer and activation function.

6. The method for analyzing the thyroid ultrasound scan video flow based on anatomy according to claim 5, characterized in that: The scanning stage of using the temporal convolutional network to determine the current frame includes: Use causal convolutional layers to process the encoded feature sequence; After each convolutional layer, batch normalization and ReLU activation function are applied; The stacked TCN blocks output a feature sequence that incorporates rich spatiotemporal information; These feature vectors are then fed into the classification layer for the final classification task, using a softmax function to classify each frame into different stages of the doctor's scan: ； in is the output of the TCN module, and are the weights and biases of the classification layer, is the classification probability; The cross entropy loss function is used to calculate the difference between the predicted probability of each stage and the true probability of the actual stage of the adaptive thyroid anatomy inference segmentation network, which is expressed as: ； in, is the true label of the i-th category, is the probability that the model predicts the i-th category.

Images