Thyroid nodule classification method, system, intelligent terminal and storage medium

Abstract

The application discloses a thyroid nodule classification method and system, an intelligent terminal and a storage medium. The thyroid ultrasound image is preprocessed by a preprocessing module to obtain a target image. A multi-branch hollow convolution module is used to extract features of the target image from multiple scales and multiple directions. A feature interaction module is used for feature interaction and dense connection is adopted to fully utilize global features and local features. A multi-branch non-local self-attention fusion module is used for self-attention operation and fusion operation to enhance the features. Finally, a classification module is used for classification to obtain the category of the thyroid nodule. Compared with the prior art, the features can be fully extracted, and the thyroid nodule can be accurately classified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image recognition, and particularly relates to a thyroid nodule classification method and system, an intelligent terminal and a storage medium. BACKGROUND

[0002] Ultrasound is a fast, safe and inexpensive imaging technique that can achieve visual observation of thyroid nodules. Based on ultrasound images, accurate differentiation of thyroid nodules requires the accuracy of feature extraction.

[0003] Although deep learning models have been used to classify ultrasound images of thyroid nodules, current deep learning models can only extract local features of specific regions well, ignoring global features, and the feature extraction is not sufficient, and the classification of thyroid nodules is not accurate enough.

[0004] Therefore, the prior art needs to be improved and improved. SUMMARY

[0005] The main purpose of the present application is to provide a thyroid nodule classification method and system, an intelligent terminal and a computer readable storage medium, which aims to solve the problem of insufficient feature extraction and inaccurate thyroid nodule classification.

[0006] In order to achieve the above purpose, the present application provides a thyroid nodule classification system, which comprises:

[0007] A preprocessing module is configured to preprocess the thyroid ultrasound image to obtain a target image.

[0008] A multi-branch hollow convolution module is configured to extract features of the target image from multiple scales and multiple directions, each branch of the multi-branch hollow convolution module is provided with a hollow convolution, and the dilation rates of all hollow convolutions are different from each other.

[0009] A feature interaction module is configured to perform feature interaction on the features extracted by the multi-branch hollow convolution module, and comprises a plurality of double dense Transformer blocks, the double dense Transformer block comprises a plurality of first Transformer models and second Transformer models arranged alternately, the first Transformer model adopts a window-based multi-head self-attention mechanism, the second Transformer model adopts a shift window-based multi-head self-attention mechanism, and the connection between the Transformer models in the double dense Transformer block is dense connection.

[0010] The multi-branch non-local self-attention fusion module is used for enhancing the features after the interaction of the feature interaction module, and comprises a query matrix branch, a keyword matrix branch and a value matrix branch, and is further provided with a fusion unit used for multiplying the features of the query matrix branch with the features of the keyword matrix branch and adding the features of the value matrix branch to obtain the fused features.

[0011] The classification module is used for classifying the fused features to obtain the category of the thyroid nodule.

[0012] Optionally, the query matrix branch and the value matrix branch are provided with point-by-point convolution, the keyword matrix branch is provided with an average pooling branch and a maximum pooling branch, and different scale pooling layers are arranged on the average pooling branch and the maximum pooling branch.

[0013] Optionally, the feature interaction module further comprises a position compression and excitation module, which is used for performing compression and excitation operations on the features output by the double dense Transformer block.

[0014] Optionally, the position compression and excitation module comprises a point-by-point convolution module adopting residual connection, an average pooling branch and a maximum pooling branch in parallel, the point-by-point convolution module is used for performing convolution on the query matrix, the keyword matrix and the value matrix obtained according to the features output by the double dense Transformer block, the average pooling branch and the maximum pooling branch are used for performing pooling operations on the features output by the point-by-point convolution module, and full connection networks are arranged on the average pooling branch and the maximum pooling branch.

[0015] Optionally, the method further comprises a probability classification module, which is used for performing probability averaging on the classification results obtained from different ultrasound images of the same thyroid nodule to obtain the probability-averaged thyroid nodule classification result.

[0016] In order to achieve the above purpose, the application further provides a thyroid nodule classification method, which comprises the following steps:

[0017] Pretreating a thyroid ultrasound image to obtain a target image;

[0018] Performing feature extraction on the target image by adopting a hollow convolution on multiple branches to obtain a feature map;

[0019] The feature map is subjected to feature interaction by using a plurality of double dense Transformer blocks to obtain an interactive feature, the double dense Transformer block comprises a plurality of first Transformer models and second Transformer models arranged alternately, the first Transformer model adopts a window-based multi-head self-attention mechanism, the second Transformer model adopts a shift window-based multi-head self-attention mechanism, and the connection between the Transformer models in the double dense Transformer block is dense connection.

[0020] The interactive feature is subjected to a self-attention-based fusion operation to obtain a fused feature.

[0021] The fused feature is subjected to classification to obtain the probability of the thyroid nodule belonging to a category.

[0022] Optionally, the model is trained based on a federated learning method, a plurality of clients and a server are provided, the client is deployed with a personalized model and a local model, the personalized model is provided with a basic layer and a personalized layer, and the training steps comprise:

[0023] The thyroid ultrasound image sample is input into the local model and the personalized model, and the model is optimized according to the difference between the probability distributions of the local model and the personalized model.

[0024] The model parameters of the basic layer are uploaded to the server for aggregation, and the aggregated model parameters are obtained from the server and the model parameters of the basic layer are updated.

[0025] Optionally, the features output by the double dense Transformer block are also subjected to compression and excitation operations, and the compression and excitation operations comprise:

[0026] The features output by the double dense Transformer block are subjected to point-by-point convolution using residual connection to obtain convolution features;

[0027] The convolution features are subjected to average pooling and maximum pooling, respectively.

[0028] The features output by the average pooling and the features output by the maximum pooling are fused to obtain compressed and excited features.

[0029] In order to achieve the above-mentioned purposes, the application further provides an intelligent terminal, which comprises a memory, a processor, and a thyroid nodule classification program stored in the memory and executable on the processor, and the steps of any one of the above-mentioned thyroid nodule classification methods are realized when the thyroid nodule classification program is executed by the processor.

[0030] In order to achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a thyroid nodule classification program, and the thyroid nodule classification program realizes the steps of any one of the above thyroid nodule classification methods when executed by a processor.

[0031] From the above, the application pre-processes the thyroid ultrasound image through the pre-processing module to obtain a target image, extracts features of the target image from multiple scales and multiple directions through the multiple-branch hollow convolution module, effectively fuses global features and local features through the feature interaction module and dense connection, performs a self-attention-based fusion operation on the features through the multi-branch non-local self-attention fusion module to enhance the features, and finally classifies the features through the classification module to obtain the category of the thyroid nodule. Compared with the prior art, the application can fully extract features and accurately classify the thyroid nodule. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0033] Figure 1 is an architecture schematic diagram of the thyroid nodule classification system provided by the embodiment of the application;

[0034] Figure 2 is Figure 1 is an architecture schematic diagram of the multi-branch hollow convolution module in the embodiment;

[0035] Figure 3 is Figure 1 is an architecture schematic diagram of the Dswin block in the embodiment;

[0036] Figure 4 is Figure 1 is an architecture schematic diagram of the position compression and excitation module in the embodiment;

[0037] Figure 5 is Figure 1 is an architecture schematic diagram of the point-wise convolution module in the embodiment;

[0038] Figure 6 is Figure 1 is an architecture schematic diagram of the multi-branch non-local self-attention fusion module in the embodiment;

[0039] Figure 7 is Figure 1 is an architecture schematic diagram of the probability classification module in the embodiment;

[0040] Figure 8 is a structural schematic diagram of a thyroid nodule classification method provided by an embodiment of the present application;

[0041] Figure 9 is an internal structure principle block diagram of the intelligent terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0043] It is to be understood that the terminology "includes", "has", "holds", "contains" and / or "comprising", when used in this specification and in the following claims, indicates the presence of the described features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0044] It is also to be understood that the terminology used in the present specification and the appended claims is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0045] It will be further understood that the terms "comprises" and / or "comprising", when used in this specification and the following claims, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0046] As used in this specification and the appended claims, the term "if' can, in some instances, be interpreted as meaning "when", or "once", or "in response to a determination" or "in response to detecting". Similarly, the phrase "if determined" or "if detected [a described condition or event]" can, in some instances, be interpreted to mean "once determined" or "in response to a determination" or "once detected [a described condition or event]" or "in response to detecting [a described condition or event]".

[0047] Clearly and completely, the technical solutions in the embodiments of the present application are described in conjunction with the drawings of the embodiments of the present application in the following description. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0048] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details that are set forth in the following description, in other manners that are different from those described herein, and it is understood that persons skilled in the art can make similar substitutions without departing from the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0049] Because the appearances of various thyroid lesion tissues are not much different, and the contrast is too low compared with the surrounding muscle tissues, it is difficult to correctly distinguish the thyroid lesion tissues. The traditional convolutional neural network can only extract the features of a specific region well, ignores the global features outside the specific region, and the feature information extraction is insufficient, resulting in inaccurate thyroid nodule classification.

[0050] In view of the above problems, the present application provides a thyroid nodule classification system, which extracts the features of a specific region in a thyroid ultrasound image from multiple scales and multiple directions using a cavity convolution, uses a plurality of alternately arranged window-based multi-head self-attention mechanism Transformer models and shift window-based multi-head self-attention mechanism Transformer models for feature interaction, and uses dense connection between the Transformer models to effectively fuse global features and local features; the interacted features are subjected to self-attention operation and fusion operation to enhance the features, and then classified. The features can be fully extracted, and the thyroid nodules can be accurately classified.

[0051] Exemplary System

[0052] The embodiment of the present application provides a thyroid nodule classification system, and the network architecture of the classification system is as shown in Figure 1 Also can be called a dense Transformer model based on multi-branch feature fusion. Deployed on electronic devices such as mobile terminals, computers, notebooks and servers, used for binary classification of thyroid nodules, such as cystic category and solid category. It can also be used for multi-classification of thyroid nodules, and can also be used for classification of other images.

[0053] The classification system mainly comprises a preprocessing module, a multi-branch atrous convolution module, a feature interaction module, a multi-branch non-local self-attention fusion module and a classification module. The preprocessing module is mainly used to crop the ultrasound image according to the nodule region to obtain a target image; the multi-branch atrous convolution module is used to extract features of the target image from multiple scales and multiple directions to construct feature maps of different sizes; the feature interaction module is used to fuse global features and local features of the feature maps; the multi-branch non-local self-attention fusion module is used to perform self-attention operation and fusion operation on the features output by the feature interaction module to enhance the features; and the classification module is used to classify the enhanced features to obtain the category of the thyroid nodule.

[0054] Specifically, when the preprocessing module crops the thyroid ultrasound image, the doctor can label the thyroid nodule region in the ultrasound image with a rectangular frame, and then crop the target image according to the rectangular frame; or the thyroid ultrasound image can be edge detected to crop the rectangular frame region where the thyroid nodule region is located to obtain the target image. By cropping the target image in the ultrasound image, the subsequent feature extraction and interaction can be more effective, and more accurate features can be extracted.

[0055] In order to generate more image samples for training, the preprocessing module further comprises a data augmentation unit for data augmentation of the target image. Various operations such as flipping, rotation and mirroring can be used in the data augmentation unit to expand the image samples.

[0056] After the preprocessing module obtains the target image, the target image is input into the multi-branch atrous convolution module. The multi-branch atrous convolution module is provided with multiple branches in parallel, and atrous convolution is used in each branch, and the dilation rates of all atrous convolutions are different, i.e. the sizes of the atrous convolutions in each branch are different, and different dilation rates can capture more context information.

[0057] The multi-branch atrous convolution module (Multi-branch Atrous Convolution: MAC) architecture of the embodiment is shown in Figure 2 As shown in the figure, three branches are provided, each branch first performs 1x1 convolution, then extracts features of the target image through atrous convolution, the dilation rates of the atrous convolutions of the three branches are 2, 3 and 4 respectively, then 1x1 convolution is performed to realize multi-scale and multi-directional feature extraction, and then the features extracted by each branch are fused and average pooling operation is performed. By using different sizes of atrous convolutions in each branch, the receptive field can be supplemented. Moreover, since dense connection is adopted in the feature interaction module, the model calculation amount will be large, by splitting the convolution kernel into multiple smaller convolution kernels through the multi-branch atrous convolution, and then applying them to different input channels, the network parameters can be reduced, and the model calculation amount can be reduced.

[0058] Then the feature map output by the multi-branch cavity convolution module is input into a feature interaction module. The feature interaction module includes a plurality of double dense Transformer blocks, each double dense Transformer block includes a first Transformer model and a second Transformer model arranged alternately, and the specific number of the first Transformer model and the second Transformer model is not limited. A window-based multi-head self-attention mechanism (W-MSA) is used in the first Transformer model, and a shift window-based multi-head self-attention mechanism (SW-MSA) is used in the second Transformer model. Using W-MSA can divide the feature map into a window, and then perform internal calculation of each window respectively. Since only self-attention calculation is performed in each window, the amount of calculation can be effectively reduced. However, there is also a certain correlation between the information of different windows. If only W-MSA module is used, information transmission cannot be performed between windows, and a large amount of feature information will exist independently. By introducing shift W-MSA (SW-MSA), information exchange between different windows can be realized.

[0059] In the double dense Transformer block, the features output by each Transformer model are input into the subsequent Transformer model to form a dense connection, which can enhance the transmission of features and more effectively utilize feature information. Optionally, the features output by each Transformer model can also be input into the subsequent Transformer model in a skip manner.

[0060] In a standard convolutional network, only the highest level features are extracted for final output. In a dense connection, the model can use features of different levels, which can provide smoother decision boundaries and more accurate results.

[0061] The embodiment takes Swin Transformer (Swin-T) as the backbone, and the double dense Transformer block is also called Dswin block. The architecture of the Dswin block is shown in Figure 3 The Swin block is the first Transformer model, and the Shift-Swin block is the second Transformer model. The calculation method of each block is as follows:

[0062] X n =Swin_B(X n-1 ),

[0063] X n+1 =Shift_Swin_B(X n )+X n ,

[0064] X n+5 = D(X n-1 , Xn,..., X n+4 ).

[0065] where X n and X n+1 are the output features of Swin block and Shift-Swin block, Swin_B and Shift_Swin_B use W-MSA and SW-MSA respectively, and D represents dense connection-based calculation.

[0066] The feature interaction module composed of the Dswin block in the embodiment is shown in FIG. 1, which has four stages, each of which includes a Dswin block for feature interaction. The stage 1 further includes a linear embedding layer for mapping the feature maps of the multi-branch hole convolution module, and the stages 2-4 further include a slice layer for secondary sampling. Three feature maps of different sizes are constructed in stages 1-3; the output result of stage 4 is further linearly transformed by a linear embedding layer on the channel data of each pixel. Figure 1

[0067] Although the multi-layer features are spliced in the Dswin block using dense connection, the sliding window design cited in the SwinTransformer still breaks the connection between the features. Therefore, in stages 1-3, a position squeeze and excitation module (PSE) is further arranged after the Dswin block to solve the problems of feature alignment error and feature missing, and further strengthen the key domain information and weaken the background information. The position squeeze and excitation module can adopt a general compression and excitation network model.

[0068] The architecture of the position squeeze and excitation module in the embodiment is shown in FIG. 2. First, the query matrix, the keyword matrix and the value matrix are respectively input into a point-by-point convolution module using residual connection to make up for the information relationship between the channels. Then, two branches are input respectively: an average pooling branch and a maximum pooling branch, and a two-layer fully connected network is used on the two branches to fuse the information of each channel, so that the network can strengthen the connection between all channels, and finally perform feature fusion. The architecture of the point-by-point convolution module is shown in FIG. 3. Figure 4 Figure 5

[0069] ​​​The feature after the interaction of the feature interaction module is input into the multi-branch non-local self-attention fusion module (MNLF). First, the self-attention operation is performed, then the features of the query matrix branch, the keyword matrix branch and the value matrix branch are fused, and the fused features are output. That is, the attention based on the global and the attention based on the maximum pooling and the average pooling are reorganized to obtain the enhanced feature map.

[0070] The multi-branch non-local self-attention fusion module architecture of the embodiment is as shown in Figure 6 For the query matrix branch (Query branch), a 1x1 pointwise convolution is used to complete the global attention operation, and then Taylor expansion is performed. For the keyword matrix branch (Key branch), after using a 1x1 convolution, the average pooling branch and the maximum pooling branch are used respectively. Different scales of pooling layers are provided on each branch to obtain feature vectors containing different scale information, and the feature vectors of each layer are spliced. Then, the results obtained by the two branches are fused, and finally Taylor expansion is performed to expand them to the same shape as the features of the query matrix branch. Then, the features of the query matrix and the features of the keyword matrix are multiplied to obtain the attention based on the global and the attention based on the maximum pooling and the average pooling. After flattening, the value matrix (Value matrix) is added to obtain the fused features.

[0071] The fused features are input into the classification module, and the probability of each classification is output to obtain the category of the thyroid nodule.

[0072] Considering that different ultrasound images may differ greatly and different doctors' annotation operations differ greatly, which may affect the final prediction. Therefore, as shown in Figure 7 When classifying the features, the embodiment does not use the traditional classification prediction method, but also constructs a probabilistic classification module (Probabilistic Classification Module: PCM) to average the prediction probabilities of multiple data, so that the classification result obtained is more accurate. That is, first, different ultrasound image samples of the same case are predicted to obtain multiple prediction results, and then the probability is averaged.

[0073] Reference Figure 1The specific process of the embodiment for two-classification of thyroid nodules is as follows: the preprocessed ultrasound image is input into a slicing module, the image is segmented into a plurality of 4*4 blocks, global information is extracted by inputting the image into a multi-branch hollow convolution module, and linear transformation is performed on the channel data of each pixel by a linear embedding layer. Feature interaction is performed by a Dswin block, three feature maps of different sizes are constructed, and secondary sampling is performed by a slicing layer in stages 2-4. Linear transformation is performed on the channel data of each pixel by a linear embedding layer. Finally, the obtained result is input into a multi-branch local attention module, and local features and global features are reorganized and fused to extract the features of the thyroid nodule image, which is input into a probability classification module to obtain the classification result of the thyroid image.

[0074] In summary, the thyroid nodule classification system of the embodiment takes Swin Transformer (Swin-T for short) as the backbone, adds a multi-branch hollow convolution module, captures more context information while keeping the parameter amount unchanged, and modifies the number and connection mode of the Transformer module, so that the relevant information of a specific area can be exchanged multiple times. Finally, a multi-branch non-local self-attention fusion module is used, which can effectively fuse global features and local features again. The feature information can be fully extracted to obtain accurate classification results.

[0075] Exemplary Method

[0076] As shown in Figure 8 , the embodiment of the application also provides a thyroid nodule classification method, specifically, the above classification method comprises the following steps:

[0077] Step S100: pre-processing the thyroid ultrasound image to obtain a target image;

[0078] Step S200: performing feature extraction on the target image using a hollow convolution on multiple branches to obtain a feature map;

[0079] Step S300: performing feature interaction on the feature map using a plurality of double-dense Transformer blocks to obtain an interacted feature, the double-dense Transformer block comprising a plurality of first Transformer models and second Transformer models arranged alternately, the first Transformer model adopting a window-based multi-head self-attention mechanism, the second Transformer model adopting a shift window-based multi-head self-attention mechanism, and the connection between the Transformer models in the double-dense Transformer block being dense connection;

[0080] Step S400: performing a self-attention-based fusion operation on the interaction features to obtain fused features;

[0081] Step S500: classifying the fused features to obtain the probability of the category to which the thyroid nodule belongs and outputting.

[0082] Optionally, the features output by the dense Transformer block are also compressed and excited, and the compression and excitation operation includes: performing point-by-point convolution on the features output by the dense Transformer block using residual connection to obtain convolution features; performing average pooling and maximum pooling on the convolution features, respectively; fusing the features output by the average pooling and the features output by the maximum pooling to obtain compressed and excited features.

[0083] In the embodiment, the thyroid nodule classification method described above can refer to the corresponding description in the thyroid nodule classification system described above, and will not be described here.

[0084] The thyroid nodule classification method of the embodiment can fully utilize global features and local features and fully utilize each layer to obtain better classification results by using the double dense Transformer block with dense connection; can capture more context information by using the multi-branch dilated convolution with different expansion rates; and can fully extract feature information and achieve good classification results by performing a self-attention-based fusion operation on the interaction features to obtain global-based attention and maximum-pooling-based and average-pooling-based attention, respectively, and then reorganizing and fusing.

[0085] Before the thyroid nodule classification method is used, model training needs to be performed. Although with the development of medical technology and the process of digitization, more and more medical data are generated and accumulated, including medical images, medical records, genetic data, etc. These data contain very valuable medical information. However, due to privacy and data security issues, these data cannot be directly shared and exchanged.

[0086] The embodiment trains the model according to a federated learning method, including a plurality of clients and an aggregation server, the local client includes a personalized model and a local model, and the personalized model is provided with a basic layer and a personalized layer. First, the aggregation server allocates an initial parameter value to each client, the client trains the local model according to the thyroid ultrasound image sample, and then the client uploads the parameter of the local model to the aggregation server for aggregation. A new model parameter can be obtained after the first round of aggregation, and the local model of the client does not participate in the update after the first round, only the personalized model performs model covering. When the personalized model is trained, the image sample of the client first passes through the basic layer, and then passes through the personalized layer, and finally obtains the output, and only the parameter of the basic layer is transmitted back to the aggregation server for aggregation after the training is completed. The aggregation server obtains a new model parameter, and transmits the new model parameter to the client to update the model parameter of the basic layer in the personalized model.

[0087] In the model training process, the Wasserstein distance (optimal transport distance, used to measure the difference between two probability distributions) is used to measure the distance between the probability distributions of the local model and the personalized layer model, so that the two models can interact with each other. The formula of the Wasserstein distance is as follows:

[0088]

[0089] As described above, the data is only trained at the local client and does not need to be uploaded to the aggregation server, and the user's privacy data does not need to be collected, thereby protecting the privacy of each client. The data contained by each client also contains patient data containing the characteristics of each region, and a personalized model containing the characteristics of each region, so that the model parameters uploaded by multiple regional clients can be aggregated to obtain an aggregated model applicable to the whole country, compared with the traditional model training, a more universal model can be trained by using the data of different regional hospitals, thereby improving the prediction accuracy and reliability of the model.

[0090] In the training of the embodiment, the data used comes from 17 hospitals in different regions, including 1395 ultrasound images of thyroid nodule patients. Each patient has transverse and longitudinal images. Through preprocessing, a total of 45439 images are finally obtained. In the training, 5-fold cross-validation is used, AdamW is selected as the optimizer, the learning rate is set to 0.05, the epoch of training is set to 70, and the cross-entropy loss is calculated. The comparison between the embodiment and various existing models in various classification indicators is as follows:

[0091]

[0092]

[0093] As can be seen from the above table, the embodiment has improved in the indicators of Acc (Accuracy: accuracy), Sen (Sensitivity: sensitivity), Spe (Specificity: specificity), Prec (Precision: precision), F1 (F1-score), Bac (Balanced Accuracy: balanced accuracy), and Auc (Area under the Curve: the size of the area under the ROC curve).

[0094] Based on the above embodiment, the application further provides an intelligent terminal, and a principle block diagram thereof can be as shown in the figure. Figure 9 The above intelligent terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a thyroid nodule classification program. The internal memory provides an environment for the operation of the operating system and the thyroid nodule classification program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with external terminals through network connection. The thyroid nodule classification program, when executed by the processor, implements the steps of any one of the thyroid nodule classification methods described above. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0095] Those skilled in the art can understand that, Figure 9 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the intelligent terminal to which the application scheme is applied. The specific intelligent terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0096] The application embodiment further provides a computer readable storage medium, and the computer readable storage medium stores a thyroid nodule classification program. The thyroid nodule classification program, when executed by a processor, implements the steps of any one of the thyroid nodule classification methods provided by the application embodiment.

[0097] It should be understood that the serial numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the application embodiment.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0099] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0100] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different ways to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0101] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented by other ways. For example, the above-mentioned apparatus / terminal device embodiments are only schematic, for example, the division of the above-mentioned modules or units is only a logical function division, and other division manners can be adopted in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0102] The above integrated modules / units, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the above-mentioned computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be realized. The above-mentioned computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The above-mentioned computer readable medium can include any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the above-mentioned computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0103] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand; it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not deviate from the spirit and scope of the corresponding technical solutions, and should be included in the protection scope of the present application.

Claims

1. A thyroid nodule classification system characterized in that, The system comprises: a preprocessing module for preprocessing a thyroid ultrasound image to obtain a target image; a multi-branch hollow convolution module for extracting features of the target image from multiple scales and multiple directions, each branch of the multi-branch hollow convolution module being provided with a hollow convolution, and the dilation rates of all hollow convolutions being different from each other; a feature interaction module for performing feature interaction on the features extracted by the multi-branch hollow convolution module, comprising a plurality of double-dense Transformer blocks, the double-dense Transformer block comprising a plurality of first Transformer models and second Transformer models arranged alternately, the first Transformer model adopting a window-based multi-head self-attention mechanism, the second Transformer model adopting a shift window-based multi-head self-attention mechanism, and the connection between the Transformer models in the double-dense Transformer block being dense connection; a multi-branch non-local self-attention fusion module for enhancing the features after the feature interaction of the feature interaction module, comprising a query matrix branch, a keyword matrix branch and a value matrix branch, and further comprising a fusion unit for multiplying the features of the query matrix branch and the features of the keyword matrix branch and then adding the features of the value matrix branch to obtain fused features; a classification module for classifying the fused features to obtain the category of the thyroid nodule.

2. The thyroid nodule classification system of claim 1, wherein, The query matrix branch and the value matrix branch are provided with point-by-point convolution, the keyword matrix branch is provided with an average pooling branch and a maximum pooling branch, and different scale pooling layers are provided on the average pooling branch and the maximum pooling branch.

3. The thyroid nodule classification system of claim 1, wherein, The feature interaction module further comprises a position compression and excitation module for performing compression and excitation operations on the features output by the double-dense Transformer block.

4. The thyroid nodule classification system of claim 3, wherein, The position compression and excitation module comprises a point-by-point convolution module adopting residual connection, an average pooling branch and a maximum pooling branch in parallel, the point-by-point convolution module being used for convolution on a query matrix, a keyword matrix and a value matrix obtained according to the features output by the double-dense Transformer block; the average pooling branch and the maximum pooling branch being used for pooling operation on the features output by the point-by-point convolution module, and full connection networks being provided on the average pooling branch and the maximum pooling branch.

5. The thyroid nodule classification system of claim 1, wherein, Further comprising a probability classification module for performing probability averaging on the classification results obtained from different ultrasound images of the same thyroid nodule to obtain a probability-averaged thyroid nodule classification result.

6. A method of classifying thyroid nodules, characterized by, It comprises: preprocessing a thyroid ultrasound image to obtain a target image; performing feature extraction on the target image by using hollow convolution on multiple branches to obtain a feature map; The feature map is interacted by a plurality of double dense Transformer blocks to obtain an interaction feature, the double dense Transformer block includes a plurality of first Transformer models and second Transformer models arranged alternately, the first Transformer model adopts a window-based multi-head self-attention mechanism, the second Transformer model adopts a shift window-based multi-head self-attention mechanism, and the connection between the Transformer models in the double dense Transformer block is dense connection; The interaction feature is subjected to a self-attention-based fusion operation to obtain a fused feature; The fused feature is classified to obtain the probability of the category to which the thyroid nodule belongs and output.

7. The thyroid nodule classification method of claim 6, wherein, The model is trained based on a federated learning method, a plurality of clients and a server are provided, the client is deployed with a personalized model and a local model, the personalized model is provided with a basic layer and a personalized layer, and the training steps include: The thyroid ultrasound image sample is input into the local model and the personalized model, and the model is optimized according to the difference between the probability distributions of the local model and the personalized model; The model parameters of the basic layer are uploaded to the server for aggregation, and the aggregated model parameters are obtained from the server and the model parameters of the basic layer are updated.

8. The thyroid nodule classification method of claim 6, wherein, The feature output by the double dense Transformer block is also subjected to compression and excitation operation, and the compression and excitation operation includes: The feature output by the double dense Transformer block is subjected to point-by-point convolution with residual connection to obtain a convolution feature; The convolution feature is subjected to average pooling and maximum pooling respectively; The features output by the average pooling and the maximum pooling are fused to obtain a compressed and excited feature.

9. An intelligent terminal, characterized by The intelligent terminal includes a memory, a processor, and a thyroid nodule classification program stored on the memory and executable on the processor, and the thyroid nodule classification program is executed by the processor to realize the steps of the thyroid nodule classification method according to any one of claims 6-8.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores a thyroid nodule classification program, and the thyroid nodule classification program is executed by the processor to realize the steps of the thyroid nodule classification method according to any one of claims 6-8.

Images