Anatomy-based thyroid ultrasound scanning video process analysis method

Abstract

The invention aims to provide an anatomy-based thyroid ultrasound scanning video process analysis method. The method comprises the following steps: carrying out graph neural network modeling according to ultrasound image anatomy knowledge; carrying out image segmentation on the thyroid and peripheral organs by adopting a self-adaptive thyroid anatomy inference segmentation network; and judging the scanning stage of the current frame by adopting the time convolutional network. According to the method, anatomical relation modeling is adaptively learned through a self-attention mechanism based on input features, so that the relation between complex anatomical structures is better captured, ultrasonic scanning process analysis based on video streams is creatively carried out, accurate understanding and stage segmentation of the ultrasonic scanning process are realized, a spatio-temporal information fusion stage is introduced, and the analysis efficiency is improved. A time convolutional network (TCN) is adopted, so that the model not only can capture spatial features, but also can understand dynamic changes on a time sequence, and therefore, the recognition capability of a scanning stage in an ultrasonic video stream is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an anatomy-based thyroid ultrasound scanning video flow analysis method. Background Art

[0002] Ultrasound has a wide range of applications in medicine and can be used to diagnose and treat various diseases. It is one of the most widely used medical imaging technologies in the world because it is non-invasive, has no ionizing radiation, is highly reliable and low-cost. However, there is a large demand for ultrasound examinations and ultrasound doctors, and it takes a long time to train an ultrasound doctor. Artificial intelligence methods are used to analyze experienced ultrasound scanning processes so that young doctors can complete standardized ultrasound scanning processes, and at the same time pave the way for the subsequent realization of autonomous ultrasound scanning.

[0003] With the advent of the big data era, the application of machine learning in the analysis of ultrasound scanning process has developed rapidly. Among them, convolutional neural networks (CNN) and recurrent neural networks (RNN) are more widely studied in the analysis of ultrasound scanning process. For example, CNN has a strong ability to extract image features, which is suitable for the spatial feature analysis of ultrasound images, and has a simple architecture, high model efficiency, and is easy to implement; RNN can process sequence data, which is suitable for the dynamic process analysis of ultrasound scanning, and can save long-term dependency information, which is conducive to diagnosis. However, it is not difficult to find that these two models have certain shortcomings in processing graph structure data through in-depth analysis of existing technologies. First, CNN relies on a grid structure and it is difficult to effectively model the complex and irregular topological structure position relationship between organs, which makes CNN difficult to apply directly. Secondly, although RNN works well in processing long sequence data, it is also difficult for RNN to effectively learn and process data structures such as graph structures with complex topology and variable relationships between edges and nodes. Summary of the invention

[0004] The purpose of the present invention is to provide an anatomically based thyroid ultrasound scanning video process analysis method, which does not rely on a predefined adjacency matrix, but adaptively learns anatomical relationship modeling through a self-attention mechanism based on input features. This method allows the network to dynamically adjust the connection strength between nodes to reflect the actual anatomical spatial relationship between them, so as to better capture the connection between complex anatomical structures, and pioneers the analysis of ultrasound scanning process based on video streams, so as to achieve accurate understanding and stage segmentation of ultrasound scanning process. The present invention introduces a spatiotemporal information fusion stage and adopts a temporal convolutional network (TCN) so that the model can not only capture spatial features, but also understand dynamic changes in time series, thereby improving the recognition ability of scanning stages in ultrasound video streams.

[0005] A thyroid ultrasound scanning video process analysis method based on anatomy, comprising: Graph neural network modeling based on anatomical knowledge of ultrasound images; Adopting adaptive thyroid anatomy reasoning segmentation network to optimize and fine-tune ultrasound image segmentation results according to graph neural nodes; A temporal convolutional network is used to determine the scanning phase of the current frame.

[0006] Preferably, the graph neural network modeling based on ultrasound image anatomical knowledge includes: The ultrasound image is represented as G(V,E), where G represents the ultrasound image, V is a set of points, and E is a set of edges; The geometric center of the anatomical structure in the ultrasound image is represented as a node in the graph neural network, and the anatomical spatial position relationship between the nodes is represented as an edge in the graph neural network; Each node i has a feature vector , combining the eigenvectors into a eigenmatrix , where N is the number of nodes and F is the feature dimension of each node; Create an N×N adjacency matrix A whose elements Indicates whether there is an edge between node i and node j. If so, it is 1, otherwise it is 0.

[0007] Preferably, the graph neural network modeling based on ultrasound image anatomical knowledge includes: Multiply the neighbor node information of the current node A by the coefficient and add it to the node A as the supplementary information of the node A, which is regarded as a feature update of the node A. Perform aggregation operation on each node in the ultrasound image and update the features of all ultrasound image nodes; Node features are updated by continuously exchanging neighborhood information until a stable equilibrium is reached, at which point the feature vectors of all nodes contain information about their neighboring nodes; Using the aggregation function to take the adjacent matrix A to represent the connection between nodes, and taking the node's feature vector X as input, the output can capture the relationship by performing relational reasoning on different nodes: ; Among them, W is a learnable parameter, It is a non-linear operation.

[0008] Preferably, the step of using an adaptive thyroid anatomy reasoning segmentation network to perform image segmentation according to nodes of a graph neural network comprises: The adaptive thyroid anatomy reasoning and segmentation network includes: a backbone module, a neck module, an adaptive relationship graph reasoning module and a segmentation head module; Inputting the nodes of the graph neural network into the adaptive thyroid anatomy reasoning segmentation network; The adaptive thyroid anatomical reasoning segmentation network outputs a thyroid segmentation result.

[0009] Preferably, the neck module is composed of up-sampling on a feature pyramid network and down-sampling on a path aggregation network for multi-scale feature fusion; The adaptive relationship graph reasoning module is embedded between the backbone module and the segmentation head module, and is used to identify and integrate the relationship of local topological structures and describe the local topological relationship blocks of structured constraints; The fully convolutional neural network module takes the output feature map of the adaptive relationship graph reasoning module as input, performs pixel-by-pixel classification, and outputs the thyroid segmentation result.

[0010] The backbone module uses CSPDarknet53 as the backbone for feature extraction; The ultrasound image of size H×W×C is input into the backbone module, and the backbone network introduces the focus module in the initial stage; Overlay width and height information into the channel; After the focus module, the feature map passes through BaseConv, CSP layer, SPBottleneck network layer, and outputs three feature maps with sizes of H / 8×W / 8×256, H / 16×W / 16×512, and H / 32×W / 32×1024.

[0011] Preferably, the neck module performs feature fusion on the obtained feature map after the backbone network extracts the features of the image; The neck module consists of upsampling on a feature pyramid network and downsampling on a path aggregation network; The feature pyramid network fuses deep semantic information features with high-resolution shallow information from top to bottom; The path aggregation network supplements the feature pyramid network and introduces a path aggregation method to aggregate shallow feature maps and deep feature maps, and transmit feature information along a specific path to pass the strong positioning features of the low layer.

[0012] Preferably, the adaptive relationship graph reasoning module includes: Message functions and aggregation functions, expressed as: , ; The adjacency matrix Combine the messages from neighboring nodes v to generate an aggregate representation of the information received by node u, where N is the total number of node vertices and the connection between two adjacent nodes is represented by an edge weight Control, the node feature matrix X is composed of the center point features, denoted as Where C is the characteristic dimension, and represents the learnable weight matrix; Feature concatenation is performed between two node vectors and calculated as the weighted sum of node feature representations. The message function and aggregation function formulas are expressed as: , ; Among them, || represents the connection operation, and a shareable and learnable linear transformation weight matrix is ​​applied to each pair of node vectors. ,if and is the feature learned from the closer region, then The value of is very large; The feature update is organized as follows through graph reasoning: ; in, is a normalized matrix, ReLU is an activation function, K is calculated by a nonlinear transformation function; The output of the local topological relation block is sent to the location branch supervised by Dice Loss, and the loss is calculated using the ground truth value of the geometric center and the predicted segmentation value, expressed as: ; and They represent the true sample label value and the model prediction value respectively, and N is the total number of center points.

[0013] Preferably, the scanning stage of using a temporal convolutional network to determine the current frame includes: Each frame of the video (x_1, x_2, ..., x_T, T is the number of frames) is encoded in sequence using an encoder to obtain the feature vector E = (e_1, e_2, ..., e_T) of each frame; The temporal convolutional network fuses spatial-temporal features through multiple dilated convolutional layers, and performs feature fusion through residual connections and normalization layers. It is expressed as: ; in is the spatial feature at time point t, are the weight, bias and dilation rate of the i-th layer of convolution, f is the convolution operation, is the feature at time point t after fusion; After each convolutional layer, a fully connected layer is used to map the learned features to the final output sequence using layer normalization and ReLU activation function, expressed as: ; Where F is the combination of the convolutional layer and the activation function.

[0014] Preferably, the scanning stage of using a temporal convolutional network to determine the current frame includes: Use causal convolutional layers to process the encoded feature sequence; After each convolutional layer, batch normalization and ReLU activation function are applied; The stacked TCN blocks output feature sequences that incorporate rich spatiotemporal information; These feature vectors are then fed into the classification layer for the final classification task, using a softmax function to classify each frame into different stages of the doctor’s scan: ; in is the output of the TCN module, and are the weights and biases of the classification layer, is the classification probability; The cross entropy loss function is used to calculate the difference between the predicted probability of each stage and the true probability of the actual stage for the adaptive thyroid anatomy reasoning segmentation network, expressed as: ; in, is the true label of the i-th category, is the probability that the model predicts the i-th category.

[0015] The beneficial effects of the present invention are as follows: 1. Topological modeling of anatomical structures under ultrasound images: The geometric center of the anatomical structure in the ultrasound image is represented as a node in the graph neural network (GNN), which effectively models the position information and complex anatomical relationship between organs, and the anatomical spatial position relationship between nodes is represented as an edge in the GNN, which directly reflects the real spatial relationship between anatomical structures. 2. Graph attention segmentation network based on topological structure: An adaptive graph attention network is proposed, which can adaptively learn anatomical relationship modeling based on input features without relying on a predefined adjacency matrix. Through the adaptive learning mechanism, the aggregation process can pay more attention to related nodes with stronger relationships, enhancing the model's ability to capture complex relationships between anatomical structures. And through feature splicing and weighted sum, the representation ability of node features is enhanced, so that the model can more accurately describe the structured constraints between two center points. 3. Scanning process analysis based on segmented video: A stage of spatiotemporal information fusion is introduced, the core of which is the temporal convolutional network (TCN), which enables the model to not only capture spatial features in the image, but also understand time series features that change over time. In this way, the model can more accurately identify the various scanning stages in the ultrasound video stream, thereby significantly improving the ability to identify the entire ultrasound scanning process. It not only enhances the model's perception of the spatial characteristics of ultrasound images, but also enables it to gain insight into dynamic changes in the time dimension, providing a powerful tool for the automatic analysis of ultrasound scanning processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A flow chart of a thyroid ultrasound scanning video process analysis method based on anatomy of the present invention; Figure 2 It is a structural schematic diagram of a thyroid ultrasound scanning video process analysis system based on anatomy of the present invention; Figure 3 It is a schematic diagram of the topological structure of the thyroid ultrasound image of the present invention; Figure 4 It is a schematic diagram of the structure of the adaptive relationship graph reasoning module of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0021] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0022] CNN relies on a grid structure and is difficult to effectively model the complex and irregular topological positional relationships between organs, which makes it difficult to directly apply CNN. Secondly, although RNN works well when processing long sequence data, it is also difficult for RNN to effectively learn and process data structures such as graph structures with complex topology and variable relationships between edges and nodes.

[0023] The present invention provides a thyroid ultrasound scanning video process analysis method based on anatomy, which does not rely on a predefined adjacency matrix, but adaptively learns anatomical relationship modeling through a self-attention mechanism based on input features. This method allows the network to dynamically adjust the connection strength between nodes to reflect the actual anatomical spatial relationship between them, so as to better capture the connection between complex anatomical structures, and pioneers the analysis of ultrasound scanning process based on video streams, achieving accurate understanding and stage segmentation of ultrasound scanning process. The present invention introduces a spatiotemporal information fusion stage and adopts a temporal convolutional network (TCN) so that the model can not only capture spatial features, but also understand dynamic changes in time series, thereby improving the recognition ability of scanning stages in ultrasound video streams.

[0024] A thyroid ultrasound scanning video process analysis method based on anatomy, reference Figure 1 and Figure 2 ,include: S100, graph neural network modeling based on anatomical knowledge of ultrasound images; S200, uses an adaptive thyroid anatomy reasoning segmentation network to perform image segmentation based on the nodes of the graph neural network; S300, using a temporal convolutional network to determine the scanning stage of the current frame.

[0025] In the Graph Neural Network (GNN), the neighbor relationship of nodes and edges is regarded as important information, which can capture the spatial structure and relationship in the image. It is particularly suitable for complex network structure analysis in ultrasound images and can effectively utilize the topological information in the image. In the scenario of autonomous thyroid ultrasound examination process analysis, inspired by the GNN structure, the characteristic organs or tissues (thyroid, trachea, muscle, aorta, etc.) in the ultrasound image can be used as nodes, and their anatomical position relationships can be used as edges on the graph to model the thyroid anatomical structure. This method makes up for the shortcomings of CNN and RNN in processing complex graph structure data and adapts to complex and irregular topological structures.

[0026] Preferably, reference Figure 3 , S100, graph neural network modeling based on ultrasound image anatomical knowledge includes: The ultrasound image is represented as G(V,E), where G represents the ultrasound image, V is a set of points, and E is a set of edges; The geometric center of the anatomical structure in the ultrasound image is represented as a node in the graph neural network, and the anatomical spatial position relationship between the nodes is represented as an edge in the graph neural network; Each node i has a feature vector , combining the eigenvectors into a eigenmatrix , where N is the number of nodes and F is the feature dimension of each node; Create an N×N adjacency matrix A whose elements Indicates whether there is an edge between node i and node j. If so, it is 1, otherwise it is 0.

[0027] A graph is composed of points and edges, usually represented as G(V,E), where G represents a graph, V is a set of points, and E is a set of edges. In order to effectively model the position information and complex anatomical relationships between organs, the geometric center of the anatomical structure in the ultrasound image is represented as a node in the graph neural network, and the anatomical spatial position relationship between nodes is represented as an edge in the graph neural network. 1: sternocleidomastoid muscle; 2: internal jugular vein; 3: common carotid artery; 4: left and right lobes of the thyroid gland; 5: isthmus of the thyroid gland; 6, 7: anterior neck muscles (sternohyoid muscle and sternothyroid muscle); 8: trachea; 9: esophagus; 10: left and right longus colli muscles.

[0028] Preferably, S100, performing graph neural network modeling based on ultrasound image anatomical knowledge includes: Multiply the neighbor node information of the current node A by the coefficient and add it to the node A as the supplementary information of the node A, which is regarded as a feature update of the node A. Perform aggregation operation on each node in the ultrasound image and update the features of all ultrasound image nodes; Node features are updated by continuously exchanging neighborhood information until a stable equilibrium is reached, at which point the feature vectors of all nodes contain information about their neighboring nodes; Using the aggregation function to take the adjacent matrix A to represent the connection between nodes, and taking the node's feature vector X as input, the output can capture the relationship by performing relational reasoning on different nodes: ; Among them, W is a learnable parameter, It is a non-linear operation.

[0029] In graph neural networks, the representation of a node is updated by aggregating the information of its neighboring nodes. The aggregation operation multiplies the neighboring node information of the current node A by a coefficient and adds it to node A as the supplementary information of node A, which is considered as a feature update of node A. Aggregation operation is performed on each node in the graph to update the features of all graph nodes. Node features are updated by continuously exchanging neighborhood information until a stable equilibrium is reached, at which point the feature vectors of all nodes contain the information of their neighboring nodes.

[0030] Preferably, reference Figure 4 , S200, uses an adaptive thyroid anatomical reasoning segmentation network to perform image segmentation according to the nodes of the graph neural network, including: The adaptive thyroid anatomy reasoning and segmentation network includes: a backbone module, a neck module, an adaptive relationship graph reasoning module, and a segmentation head module; The nodes of the graph neural network are input into the adaptive thyroid anatomy reasoning segmentation network; The adaptive thyroid anatomy inference segmentation network outputs the thyroid segmentation result.

[0031] The overview of the adaptive thyroid anatomy reasoning segmentation network includes the backbone, neck, adaptive relationship diagram reasoning (ARGR) module and segmentation head. The backbone part mainly uses CSPDarknet53 for feature extraction. The neck part consists of feature pyramid network (FPN) upsampling and path aggregation network (PAN) downsampling, which is mainly used for multi-scale feature fusion. The adaptive relationship graph reasoning module is embedded between the neck and the head to identify and integrate the relationship of the local topological structure: the local topological relationship block that describes the structured constraints. The FCN module takes the output feature map of the ARGR module as input, performs pixel-by-pixel classification, and finally produces the segmentation result.

[0032] Preferably, the neck module consists of up-sampling on a feature pyramid network and down-sampling on a path aggregation network for multi-scale feature fusion; The adaptive graph reasoning module is embedded between the backbone module and the segmentation head module to identify and integrate the relationships of local topological structures and describe the local topological relationship blocks of structured constraints; The fully convolutional neural network module takes the output feature map of the adaptive relationship graph reasoning module as input, performs pixel-by-pixel classification, and outputs the thyroid segmentation result.

[0033] The backbone module uses CSPDarknet53 as the backbone for feature extraction; The ultrasound image of size H×W×C is input into the backbone module, and the backbone network introduces the focus module in the initial stage; Overlay width and height information into the channel; After the focus module, the feature map passes through BaseConv, CSP layer, SPBottleneck network layer, and outputs three feature maps with sizes of H / 8×W / 8×256, H / 16×W / 16×512, and H / 32×W / 32×1024.

[0034] CSPDarknet53 was used as the backbone for feature extraction. Thyroid ultrasound images of size H×W×C were input into the backbone network. The backbone network introduced the Focus Module in the initial stage. The function of this module is to process the input image in the first step of the network so that it can adapt to the network for feature extraction. The working principle of the Focus Module is to take a value from every other pixel in the horizontal and vertical directions of the image matrix, so that four separate feature matrices can be generated. This operation is equivalent to downsampling the image, while superimposing the width and height information into the channel, increasing the number of channels and thus improving the efficiency of information extraction. After the Focus Module, the feature map will pass through several layers, including BaseConv, CSP layer, SPPBottleneck and other network layers. Finally, three feature maps of sizes H / 8×W / 8×256, H / 16×W / 16×512 and H / 32×W / 32×1024 are made for input to Neck.

[0035] Preferably, after the backbone network extracts the features of the image, the neck module performs feature fusion on the obtained feature map; The neck module consists of upsampling on a feature pyramid network and downsampling on a path aggregation network; The feature pyramid network integrates deep semantic information features with high-resolution shallow information from top to bottom; The path aggregation network supplements the feature pyramid network and introduces a path aggregation method to aggregate shallow feature maps and deep feature maps, and transmit feature information along a specific path to pass the strong positioning features of the low layer.

[0036] After the backbone network extracts the features of the image, the obtained feature map is input to the neck for feature fusion. In our method, the neck part is upsampled by feature pyramid networks (FPN) and path aggregation network ( The FPN combines deep semantic information features with high-resolution shallow information from top to bottom, enhancing the semantic expression of multiple scales, but does not transmit positioning information. PAN aims at this point, adding a bottom-up pyramid behind FPN to supplement FPN and introduce path aggregation. By aggregating shallow feature maps and deep feature maps and transmitting feature information along a specific path, the strong positioning features of the low layer are transmitted. The repeated downsampling of PAN completely improves the accuracy of model detection.

[0037] Preferably, the adaptive relationship graph reasoning module includes: Message functions and aggregation functions, expressed as: , ; The adjacency matrix Combine the messages from neighboring nodes v to generate an aggregate representation of the information received by node u, where N is the total number of node vertices and the connection between two adjacent nodes is represented by an edge weight Control, the node feature matrix X is composed of the center point features, denoted as Where C is the characteristic dimension, and represents the learnable weight matrix; In the local topological relationship block, the relationship between anatomical structures in the image is learned to further enhance localization. In order to ensure sufficient representation ability to describe the structured constraints between two center points, we perform feature concatenation between two node vectors, calculated as the weighted sum of node feature representations. Feature concatenation between two node vectors is calculated as the weighted sum of node feature representations. The message function and aggregation function formulas are expressed as: , ; Among them, || represents the connection operation, and a shareable and learnable linear transformation weight matrix is ​​applied to each pair of node vectors. ,if and is the feature learned from the closer region, then The value of is very large; In order to capture pairwise topological information from the multi-head graph attention layer in the local topological relation block, the feature update is organized as follows via graph reasoning: ; in, is a normalized matrix, ReLU is an activation function, The K is calculated by a nonlinear transformation function; for this purpose, the message is passed by treating the learned correspondences as an adjacency matrix. This allows the relevant features to be enhanced and the alignment information to be embedded into the feature evolution. This enables the aggregation to focus more on relevant nodes with "close" relationships.

[0038] The output of the local topological relationship block is sent to the location branch supervised by Dice Loss, which corresponds to the segmentation geometric center and can maximize the similarity between the predicted segmentation result and the true label. The output of the local topological relationship block is sent to the location branch supervised by Dice Loss, and the loss is calculated using the ground truth value and predicted segmentation value of the geometric center, expressed as: ; and They represent the true sample label value and the model prediction value respectively, and N is the total number of center points.

[0039] In order to better capture anatomical relationships and improve segmentation accuracy, a graph attention mechanism is introduced to constrain the topological relationships between anatomical structures. It can adaptively learn anatomical relationship modeling through self-attention based on input features, without relying on the predefined adjacency matrix in the main graph convolution layer. The feature map output by the trunk and neck network divides the input image into a series of pixel regions, where the center point of each region is responsible for detecting potential anatomical structures and generating corresponding segmentation masks based on the center point features. This structure enables the model to accurately capture anatomical details, thereby achieving high-quality image segmentation. The visual information transmitted between anatomical structures is modeled by measuring the influence from one center point to another according to their visual similarity and calculated as a normalized embedded Gaussian function.

[0040] The fully convolutional network (FCN) outputs a heat map of the same size as the feature map by converting the last fully connected layer into a convolutional layer, predicting the class probability of each pixel. Then, the heat map is enlarged to the original image size by upsampling. FCN uses a skip-level structure to combine deep global information and shallow detail information, and enhances segmentation details by element addition or splicing. Finally, a 1x1 convolution is used for pixel-by-pixel classification to obtain the segmentation result.

[0041] Preferably, S300, using a temporal convolutional network to determine the scanning stage of the current frame includes: Each frame of the video (x_1, x_2, ..., x_T, T is the number of frames) is encoded in sequence using an encoder to obtain the feature vector E = (e_1, e_2, ..., e_T) of each frame; The temporal convolutional network fuses spatial-temporal features through multiple dilated convolutional layers, and performs feature fusion through residual connections and normalization layers. It is expressed as: ; in is the spatial feature at time point t, are the weight, bias and dilation rate of the i-th layer of convolution, f is the convolution operation, is the feature at time point t after fusion; After each convolutional layer, a fully connected layer is used to map the learned features to the final output sequence using layer normalization and ReLU activation function, expressed as: ; Where F is the combination of the convolutional layer and the activation function.

[0042] In order to achieve accurate understanding and stage segmentation of the ultrasound scanning process, the spatiotemporal information fusion stage was introduced, and the temporal convolutional network (TCN) was used to fuse spatial and temporal features.

[0043] Preferably, S300, using a temporal convolutional network to determine the scanning stage of the current frame includes: Use causal convolutional layers to process the encoded feature sequence; After each convolutional layer, batch normalization and ReLU activation function are applied; The stacked TCN blocks output feature sequences that incorporate rich spatiotemporal information; These feature vectors are then fed into the classification layer for the final classification task, using a softmax function to classify each frame into different stages of the doctor’s scan: ; in is the output of the TCN module, and are the weights and biases of the classification layer, is the classification probability; The cross entropy loss function is used to calculate the difference between the predicted probability of each stage and the true probability of the actual stage for the adaptive thyroid anatomy reasoning segmentation network, expressed as: ; in, is the true label of the i-th category, is the probability that the model predicts the i-th category.

[0044] The encoded feature sequence is processed using a causal convolutional layer to ensure that the model does not use future information when predicting the current frame, but is only determined by the current and previous frames. Next, the receptive field is expanded by dilating the convolutional layer so that the output of each time step can rely on historical information over a longer time range. After each convolutional layer, batch normalization and ReLU activation functions are applied to stabilize the training process and introduce nonlinearity. Residual connections are added to facilitate the training of deep networks, allowing the network to learn more complex features. By stacking multiple such TCN blocks, the network is able to capture more complex spatiotemporal features. Ultimately, the output of TCN is a feature sequence that incorporates rich spatiotemporal information, providing a strong feature representation for subsequent classification and stage segmentation. These feature vectors are then fed into the classification layer for the final classification task.

[0045] The beneficial effects of the present invention are as follows: 1. Topological modeling of anatomical structures under ultrasound images: The geometric center of the anatomical structure in the ultrasound image is represented as a node in the graph neural network (GNN), which effectively models the position information and complex anatomical relationship between organs, and the anatomical spatial position relationship between nodes is represented as an edge in the GNN, which directly reflects the real spatial relationship between anatomical structures. 2. Graph attention segmentation network based on topological structure: An adaptive graph attention network is proposed, which can adaptively learn anatomical relationship modeling based on input features without relying on a predefined adjacency matrix. Through the adaptive learning mechanism, the aggregation process can pay more attention to related nodes with stronger relationships, enhancing the model's ability to capture complex relationships between anatomical structures. And through feature splicing and weighted sum, the representation ability of node features is enhanced, so that the model can more accurately describe the structured constraints between two center points. 3. Scanning process analysis based on segmented video: A stage of spatiotemporal information fusion is introduced, the core of which is the temporal convolutional network (TCN), which enables the model to not only capture spatial features in the image, but also understand time series features that change over time. In this way, the model can more accurately identify the various scanning stages in the ultrasound video stream, thereby significantly improving the ability to identify the entire ultrasound scanning process. It not only enhances the model's perception of the spatial characteristics of ultrasound images, but also enables it to gain insight into dynamic changes in the time dimension, providing a powerful tool for the automatic analysis of ultrasound scanning processes.

[0046] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A thyroid ultrasound scanning video process analysis method based on anatomy, characterized in that: include: Graph neural network modeling based on anatomical knowledge of ultrasound images; Adopting adaptive thyroid anatomical reasoning segmentation network to segment the thyroid gland and surrounding organs; A temporal convolutional network is used to determine the scanning phase of the current frame.

2. The method for analyzing the thyroid ultrasound scan video flow based on anatomy according to claim 1, characterized in that: The graph neural network modeling based on ultrasound image anatomical knowledge includes: The ultrasound image is represented as G(V,E), where G represents the ultrasound image, V is a set of points, and E is a set of edges; The geometric center of the anatomical structure in the ultrasound image is represented as a node in the graph neural network, and the anatomical spatial position relationship between the nodes is represented as an edge in the graph neural network; Each node i has a feature vector , combining the eigenvectors into a eigenmatrix , where N is the number of nodes and F is the feature dimension of each node; Create an N×N adjacency matrix A whose elements Indicates whether there is an edge between node i and node j. If so, it is 1, otherwise it is 0.

3. The method for analyzing the thyroid ultrasound scan video flow based on anatomy according to claim 2, characterized in that: The graph neural network modeling based on ultrasound image anatomical knowledge includes: Multiply the neighbor node information of the current node A by the coefficient and add it to the node A as the supplementary information of the node A, which is regarded as a feature update of the node A. Perform aggregation operation on each node in the ultrasound image and update the features of all ultrasound image nodes; Node features are updated by continuously exchanging neighborhood information until a stable equilibrium is reached, at which point the feature vectors of all nodes contain information about their neighboring nodes; Using the aggregation function to take the adjacent matrix A to represent the connection between nodes, and taking the node's feature vector X as input, the output can capture the relationship by performing relational reasoning on different nodes: ； Among them, W is a learnable parameter, It is a non-linear operation.

4. The method for analyzing the thyroid ultrasound scan video flow based on anatomy according to claim 1, characterized in that: The use of an adaptive thyroid anatomical reasoning segmentation network to perform image segmentation on the thyroid gland and surrounding organs comprises: The adaptive thyroid anatomy reasoning and segmentation network includes: a backbone module, a neck module, an adaptive relationship graph reasoning module and a segmentation head module; Inputting the nodes of the graph neural network into the adaptive thyroid anatomy reasoning segmentation network; The adaptive thyroid anatomical reasoning segmentation network outputs a thyroid segmentation result.

5. The method for analyzing the thyroid ultrasound scan video flow based on anatomy according to claim 4, characterized in that: The neck module consists of upsampling on a feature pyramid network and downsampling on a path aggregation network for multi-scale feature fusion; The adaptive relationship graph reasoning module is embedded between the backbone module and the segmentation head module, and is used to identify and integrate the relationship of local topological structures and describe the local topological relationship blocks of structured constraints; The fully convolutional neural network module takes the output feature map of the adaptive relationship graph reasoning module as input, performs pixel-by-pixel classification, and outputs the thyroid segmentation result.

6. The method for analyzing the thyroid ultrasound scan video flow based on anatomy according to claim 4, characterized in that: The backbone module uses CSPDarknet53 as the backbone for feature extraction; The ultrasound image of size H×W×C is input into the backbone module, and the backbone network introduces the focus module in the initial stage; Overlay width and height information into the channel; After the focus module, the feature map passes through BaseConv, CSP layer, SPBottleneck network layer, and outputs three feature maps with sizes of H / 8×W / 8×256, H / 16×W / 16×512, and H / 32×W / 32×1024.

7. The method for analyzing the thyroid ultrasound scanning video flow based on anatomy according to claim 4, characterized in that: After the backbone network extracts the features of the image, the neck module performs feature fusion on the obtained feature map; The neck module consists of upsampling on a feature pyramid network and downsampling on a path aggregation network; The feature pyramid network fuses deep semantic information features with high-resolution shallow information from top to bottom; The path aggregation network supplements the feature pyramid network and introduces a path aggregation method to aggregate shallow feature maps and deep feature maps, and transmit feature information along a specific path to pass the strong positioning features of the low layer.

8. The method for analyzing the thyroid ultrasound scan video flow based on anatomy according to claim 4, characterized in that: The adaptive relationship graph reasoning module includes: Message functions and aggregation functions, expressed as: ， ； The adjacency matrix Combine the messages from neighboring nodes v to generate an aggregate representation of the information received by node u, where N is the total number of node vertices and the connection between two adjacent nodes is represented by an edge weight Control, the node feature matrix X is composed of the center point features, denoted as Where C is the feature dimension, and represents the learnable weight matrix; Feature concatenation is performed between two node vectors and calculated as the weighted sum of node feature representations. The message function and aggregation function formulas are expressed as: ， ； Among them, || represents the connection operation, and a shareable and learnable linear transformation weight matrix is ​​applied to each pair of node vectors. ,if and is the feature learned from the closer region, then The value of is very large; The feature update is organized as follows through graph reasoning: ； in, is a normalized matrix, ReLU is an activation function, K is calculated by a nonlinear transformation function; The output of the local topological relation block is sent to the location branch supervised by Dice Loss, and the loss is calculated using the ground truth value of the geometric center and the predicted segmentation value, expressed as: ； and They represent the true sample label value and the model prediction value respectively, and N is the total number of center points.

9. The method for analyzing the thyroid ultrasound scan video flow based on anatomy according to claim 1, characterized in that: The scanning stage of using the temporal convolutional network to determine the current frame includes: Each frame of the video (x_1, x_2, ..., x_T, T is the number of frames) is encoded in sequence using an encoder to obtain the feature vector E = (e_1, e_2, ..., e_T) of each frame; The temporal convolutional network fuses spatial-temporal features through multiple dilated convolutional layers, and performs feature fusion through residual connections and normalization layers. It is expressed as: ； in is the spatial feature at time point t, are the weight, bias and dilation rate of the i-th layer of convolution, f is the convolution operation, is the feature at time point t after fusion; After each convolutional layer, a fully connected layer is used to map the learned features to the final output sequence using layer normalization and ReLU activation function, expressed as: ； Where F is the combination of the convolutional layer and the activation function.

10. The method for analyzing the thyroid ultrasound scanning video flow based on anatomy according to claim 9, characterized in that: The scanning stage of using the temporal convolutional network to determine the current frame includes: Use causal convolutional layers to process the encoded feature sequence; After each convolutional layer, batch normalization and ReLU activation function are applied; The stacked TCN blocks output feature sequences that incorporate rich spatiotemporal information; These feature vectors are then fed into the classification layer for the final classification task, using a softmax function to classify each frame into different stages of the doctor’s scan: ； in is the output of the TCN module, and are the weights and biases of the classification layer, is the classification probability; The cross entropy loss function is used to calculate the difference between the predicted probability of each stage and the true probability of the actual stage for the adaptive thyroid anatomy reasoning segmentation network, expressed as: ； in, is the true label of the i-th category, is the probability that the model predicts the i-th category.

Images