Ultrasonic image thyroid nodule detection device based on prior knowledge guidance

Abstract

The invention discloses an ultrasonic image thyroid nodule detection device based on prior knowledge guidance, and the method integrates prior knowledge and nodule distribution characteristics into a deep network, and comprises two stages: in the first stage, a multi-scale coarse detection module ThyroidDetI designs a multi-scale region-based detection network to learn pyramid features; the region proposal is trained by priori knowledge about actual nodule size and shape distribution so as to detect nodules on different feature scales, and coarse detection of the nodules is carried out; in the second stage, a multi-branch fine classification module ThyroidDetII provides a multi-branch fine classification network to integrate features oriented to multi-view diagnosis, and each network branch captures and enhances a group of specific features commonly used by doctors to perform fine classification of nodules. According to the scheme, the omission ratio of small-size nodules is effectively reduced, the error detection rate of challenging nodules is reduced, the detection accuracy is improved, and subjective judgment errors in the medical diagnosis process are greatly reduced.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and medicine, and more specifically, to a thyroid nodule detection device based on ultrasonic images guided by prior knowledge. Background Art

[0002] Thyroid nodules are an endocrine disease with a high incidence rate. They are divided into benign and malignant nodules. If malignant nodules are not diagnosed and treated in time, they may further deteriorate and affect the patient's life safety. Ultrasound examination is a very important tool for thyroid nodule assessment. It can obtain information such as the boundaries, morphology, and echoes of the patient's thyroid nodules, providing support for further treatment of the patient. However, since physicians usually make diagnoses based on the ultrasonic characteristics of nodules in ultrasound images, this is relatively subjective and highly dependent on the physician's clinical experience, and misdiagnosis and missed diagnosis are prone to occur.

[0003] Deep learning models, especially convolutional neural networks (CNNs), have shown major performance improvements over traditional learning methods in various visual recognition tasks. Although existing deep learning methods are generally more powerful than traditional methods, they still have certain limitations in diagnostic performance. Similar to most traditional methods that rely on handcrafted features, existing deep learning methods tend to treat natural images and medical images equally and directly use a general CNN model for thyroid nodule classification. This approach ignores important domain knowledge and expert experience related to specific medical diagnosis tasks and has problems with improper operation.

[0004] To address this challenge, many computer-aided diagnosis (CAD) systems have been proposed for automatic objective classification of thyroid nodules. Typically, CAD systems contain three basic components, namely nodule detection, feature extraction, and classifier construction. Traditional methods usually predefine the doctor's nodule location, then extract handcrafted features (e.g., texture and morphological features), and build a classifier. However, due to the differences in shape, size, and internal features of thyroid nodules, the low-level handcrafted features used in traditional CAD methods have limited discriminative ability due to their inherent simplicity and locality, resulting in the problem of missed and over-detected nodules. Summary of the invention

[0005] In order to solve the defects in the existing technology, a thyroid nodule detection device based on ultrasound images guided by prior knowledge is proposed, including a nodule multi-scale coarse detection module ThyroidDetⅠ and a nodule multi-branch fine classification module ThyroidDetⅡ:

[0006] Nodule multi-scale coarse detection module ThyroidDetⅠ: A pyramid network is constructed to detect nodules of different scales, in which a top-down module is designed to integrate high-level semantic information with low-level feature maps for small nodule detection. In addition, in order to further improve the detection performance, the present invention refers to an important prior constraint, namely the true distribution of nodule size and shape, and designs reliable anchor points for each feature scale in proposal generation to better anchor the box. Prior knowledge helps to optimize the initialization and filtering of proposals, thereby achieving more accurate detection. The region proposal is trained by prior knowledge about the actual nodule size and shape distribution to perform coarse detection of nodules;

[0007] Nodule multi-branch fine classification module ThyroidDetⅡ: A multi-branch fine classification network is proposed to integrate features for multi-view diagnosis, in which each network branch captures and enhances a set of specific features commonly used by doctors to achieve fine classification of nodules. Specifically, in addition to the basic branch that extracts semantic features from the input patch, the multi-branch network also integrates a context branch to extract enhanced context features that are closely related to malignant nodules. In addition, an aspect ratio preservation strategy is introduced to process the input image patch, which can retain the aspect ratio of the nodule during resizing to obtain better classification performance;

[0008] The ultrasound image thyroid nodule detection device guided by prior knowledge first performs rough detection of nodules through the nodule multi-scale rough detection module ThyroidDetⅠ, and then performs fine classification of nodules through the nodule multi-branch fine classification module ThyroidDetⅡ.

[0009] The specific implementation process of the nodule multi-scale coarse detection module ThyroidDetⅠ is as follows:

[0010] 1) ResNet-50 is used as the backbone of the detection network. It consists of one convolutional layer (i.e., Conv1 using a 7×7 kernel) and four residual blocks (i.e., Conv2_x to Conv5_x), with Conv2_x to Conv5_x having 3, 4, 6, and 3 building blocks, respectively. Each building block contains 3 convolutional layers, using 1×1, 3×3, and 1×1 kernels, respectively. Each of them is followed by a maximum pooling layer that downsamples the feature map by a factor of 2. Therefore, similar to FPN, the backbone here forms a bottom-up path that computes a pyramid feature hierarchy with gradually increasing abstraction, but with decreasing spatial resolution. Conv2_x to Conv5_x have 3, 4, 6, and 3 building blocks, respectively. Each building block contains 3 convolutional layers, using 1×1, 3×3, and 1×1 kernels, respectively.

[0011] 2) In order to accurately detect objects of different sizes, the present invention designs a top-down fusion module to hierarchically fuse high-level backbone features with low-level backbone features, and then use the merged features at different scales to detect objects of different sizes. Specifically, from Conv2_x to Conv5_x, the top-down fusion module is injected between any two adjacent blocks. In each fusion module, the high-level feature map is upsampled by 2 times using a convolutional layer, and then concatenated (via 1×1 convolution) with the transferred low-level feature map. The concatenated feature map is finally processed by a 3×3 convolutional layer to generate a fused feature map with reduced aliasing effect for detecting objects at that specific scale. In this way, the network proposed in the present invention generates five region proposals of different scales (i.e., Conv2_x, Conv3_x, Conv4_x, Conv5_x and Pool5), where the five feature map sizes are 1 / 4, 1 / 8, 1 / 16, 1 / 32 and 1 / 64 of the original input size, respectively in width, height and corresponding feature maps. These region proposals are finally sent to the R-CNN block to detect nodules of different sizes.

[0012] 3) Detection anchors limited by prior knowledge: The multi-scale detection network of the present invention uses anchors with designed size and aspect ratio (i.e., height divided by width) to predict region proposals for object detection. The present invention collects statistics on the distribution of nodule size aspect ratios in the training dataset in advance, and the results are as follows: Figure 4 As shown. Figure 4 There are two observations in the figure: (1) the aspect ratio range of all nodules in the training set is (0.2, 2.5), which means that the default anchor aspect ratio range of the experiment is also (0.2, 2.5); (2) the larger the nodule size, the smaller the corresponding aspect ratio, which means that the aspect ratio range defined by the bottom layer should be larger than the aspect ratio range defined by the top layer. Therefore, in order to ensure that the size and shape of the region proposal are consistent with the true nodule distribution, the anchor box of the lower feature map is constrained to be smaller in size and larger in aspect ratio than that of the higher feature map. In general, an anchor box with 1 optional size (32, 64, 128, 256, 512) and corresponding 3 optional aspect ratios ((0.4, 1.0, 2.5), (0.4, 1.0, 2.5), (0.5, 1.0, 2.0), (0.7, 1.0, 1.5), (0.7, 1.0, 1.5)) is generated at each feature map position. The anchor box sizes and aspect ratios defined in different feature scale spaces can well cover different nodule sizes in the current image. Therefore, based on the prior knowledge about the aspect ratio and size of thyroid nodules, the auxiliary effect of region proposal on automatic nodule detection can be effectively improved.

[0013] The specific implementation process of the nodule multi-branch fine classification module ThyroidDetⅡ is as follows:

[0014] The classification results produced by the nodule multi-scale coarse detection module ThyroidDetⅠ are relatively rough, considering that the proposal classification is only a supplementary task to help the main task of detecting nodules in the first step. However, these classification results still provide valuable diagnostic suggestions. The samples misclassified by the nodule multi-scale coarse detection module ThyroidDetⅠ may be challenging nodules, which cannot be well distinguished from the relatively weak diagnostic features that are mainly learned for nodule detection. To further improve the diagnostic performance, a more complex classification network is needed, which should include the views and attention of radiologists (especially when diagnosing challenging nodules) to guide the learning of more discriminative features for automatic nodule recognition. The present invention proposes a multi-branch fine classification module ThyroidDetⅡ to learn features for multi-view classification for fine classification of nodules.

[0015] Specifically, the multi-branch refined classification ThyroidDetⅡ adopts two complementary branches to learn multi-view features of nodules from different regions according to the doctor's attention. The structure of multi-branch refined classification ThyroidDetⅡ is shown in the figure. Figure 5 As shown, the subnetwork structures in all branches are the same, but the parameters are different. The present invention refines the ZFnet architecture into the backbone of a multi-branch fine classification network. It retains 5 convolutional layers (i.e., Conv1 uses 7×7 cores, Conv2 uses 5×5 cores, and Conv3-Conv5 uses 3×3 cores), two fully connected layers containing 4096 units, and two additional fully connected layers with 1024 and 2 outputs, respectively. After Conv1, Conv2, and Conv5, a maximum pooling operation with a window size of 3×3 is applied.

[0016] The first branch inputs the ROI of the original nodule. Since convolutional neural networks usually require input of standard size, the area block where the original ROI is located needs to be adjusted first. In order to retain the original aspect ratio information of nodules of different shapes, the original ROI is adjusted using the aspect ratio preserving cropping strategy. For each nodule of size h×w, it is first cropped and filled with a square area block of size: max(h,w). In this way, the nodule will not be stretched in only one direction, and the aspect ratio information will be retained for nodule classification based on convolutional neural networks. In this way, the nodule will not be stretched in only one direction, and the aspect ratio information will be retained for nodule classification based on CNN.

[0017] The second branch inputs an image patch that is 1.5 times the size of the original nodule ROI. The second branch inputs an image patch that is 1.5 times the size of the original nodule ROI. In addition to the information inside the nodule, the characteristics of the surrounding tissue (such as diffuse sclerosis and external echo differences) also affect the classification of the nodule. In order to consider this contextual information, an image patch centered on the original ROI and 1.5 times its size is cropped as the input of the second branch, which is called the context branch.

[0018] Finally, the multi-view features from the two parallel branches (i.e., the output of FC3 in each refined ZFNet) are concatenated and fused by an FC layer (256 units) for a softmax classifier for automatic nodule classification.

[0019] The multi-task loss function for end-to-end region proposal and nodule detection is defined as follows:

[0020]

[0021] Among them, {p i} represents the classification score of the i-th anchor point, p i is the probability that the current anchor point is classified as a foreground (target), p i * is the true label; {t i} represents the bounding box prediction value of the i-th anchor point, t i are the coordinates predicted by the network, t i * is the coordinate of the real target; L cls is the classification loss, using the cross-entropy loss function; L reg is the bounding box regression loss, using smooth L1 loss; N cls is the regularization factor of the classification loss; N reg is the regularization factor of the regression loss; λ is the weight to balance the two parts of the loss.

[0022] The beneficial effects of the present invention are as follows:

[0023] The present invention applies deep learning methods, combines the tasks of thyroid nodule detection and classification, and improves the network in terms of nodule detection accuracy. It can perform real-time detection of thyroid nodules in ultrasound images and obtain the classification results of benign and malignant thyroid nodules of patients, thereby providing a reliable basis for clinical diagnosis, reducing the misdiagnosis rate, improving diagnostic efficiency, and reducing unnecessary medical intervention for patients.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the following briefly introduces the drawings required for use in the specific implementation or the prior art description. Obviously, the drawings in the following description are Figure 1 This is an implementation mode of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 This is a schematic diagram of a flow chart of a device for detecting thyroid nodules in ultrasonic images based on prior knowledge guidance according to an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the network model structure of the present invention;

[0028] Figure 3 This is an example of the nodule multi-scale coarse detection module ThyroidDetⅠ of the present invention;

[0029] Figure 4 is an example of the distribution of aspect ratios of nodules in the dataset of the present invention;

[0030] Figure 5 This is an example of the nodule multi-branch refined classification module ThyroidDetⅡ of the present invention;

[0031] Figure 6 This is a comparison of the qualitative detection results of the present invention and other methods. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings. The described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0033] Figure 1 FIG. 1 is a flow chart of a device for detecting thyroid nodules in ultrasound images based on prior knowledge according to an embodiment of the present invention. Figure 1 As shown, the method includes the following two modules.

[0034] The ultrasound image thyroid nodule detection device guided by prior knowledge first performs rough detection of nodules through the nodule multi-scale rough detection module ThyroidDetⅠ, and then performs fine classification of nodules through the nodule multi-branch fine classification module ThyroidDetⅡ.

[0035] Nodule multi-scale coarse detection module ThyroidDetⅠ: A pyramid network is constructed to detect nodules of different scales, in which a top-down module is designed to integrate high-level semantic information with low-level feature maps for small nodule detection. In addition, in order to further improve the detection performance, the present invention refers to an important prior constraint, namely the true distribution of nodule size and shape, and designs reliable anchor points for each feature scale in proposal generation to better anchor the box. Prior knowledge helps to optimize the initialization and filtering of proposals, thereby achieving more accurate detection. The region proposal is trained by prior knowledge about the actual nodule size and shape distribution for coarse detection of nodules.

[0036] The specific implementation process of the nodule multi-scale coarse detection module ThyroidDetⅠ is as follows:

[0037] 1) ResNet-50 is used as the backbone of the detection network. It consists of one convolutional layer (i.e., Conv1 using a 7×7 kernel) and four residual blocks (i.e., Conv2_x to Conv5_x), with Conv2_x to Conv5_x having 3, 4, 6, and 3 building blocks, respectively. Each building block contains 3 convolutional layers, using 1×1, 3×3, and 1×1 kernels, respectively. Each of them is followed by a maximum pooling layer that downsamples the feature map by a factor of 2. Therefore, similar to FPN, the backbone here forms a bottom-up path that computes a pyramid feature hierarchy with gradually increasing abstraction, but with decreasing spatial resolution. Conv2_x to Conv5_x have 3, 4, 6, and 3 building blocks, respectively. Each building block contains 3 convolutional layers, using 1×1, 3×3, and 1×1 kernels, respectively.

[0038] 2) In order to accurately detect objects of different sizes, the present invention designs a top-down fusion module to hierarchically fuse high-level backbone features with low-level backbone features, and then use the merged features at different scales to detect objects of different sizes. Specifically, from Conv2_x to Conv5_x, the top-down fusion module is injected between any two adjacent blocks. In each fusion module, the high-level feature map is upsampled by 2 times using a convolutional layer, and then concatenated (via 1×1 convolution) with the transferred low-level feature map. The concatenated feature map is finally processed by a 3×3 convolutional layer to generate a fused feature map with reduced aliasing effect for detecting objects at that specific scale. In this way, the network proposed in the present invention generates five region proposals of different scales (i.e., Conv2_x, Conv3_x, Conv4_x, Conv5_x and Pool5), where the five feature map sizes are 1 / 4, 1 / 8, 1 / 16, 1 / 32 and 1 / 64 of the original input size, respectively in width, height and corresponding feature maps. These region proposals are finally sent to the R-CNN block to detect nodules of different sizes.

[0039] 3) Detection anchors limited by prior knowledge: The multi-scale detection network of the present invention uses anchors with designed size and aspect ratio (i.e., height divided by width) to predict region proposals for object detection. The present invention collects statistics on the distribution of nodule size aspect ratios in the training dataset in advance, and the results are as follows: Figure 4 As shown. Figure 4 There are two observations in the figure: (1) the aspect ratio range of all nodules in the training set is (0.2, 2.5), which means that the default anchor aspect ratio range of the experiment is also (0.2, 2.5); (2) the larger the nodule size, the smaller the corresponding aspect ratio, which means that the aspect ratio range defined by the bottom layer should be larger than the aspect ratio range defined by the top layer. Therefore, in order to ensure that the size and shape of the region proposal are consistent with the true nodule distribution, the anchor box of the lower feature map is constrained to be smaller in size and larger in aspect ratio than that of the higher feature map. In general, an anchor box with 1 optional size (32, 64, 128, 256, 512) and corresponding 3 optional aspect ratios ((0.4, 1.0, 2.5), (0.4, 1.0, 2.5), (0.5, 1.0, 2.0), (0.7, 1.0, 1.5), (0.7, 1.0, 1.5)) is generated at each feature map position. The anchor box sizes and aspect ratios defined in different feature scale spaces can well cover different nodule sizes in the current image. Therefore, based on the prior knowledge about the aspect ratio and size of thyroid nodules, the auxiliary effect of region proposal on automatic nodule detection can be effectively improved.

[0040] Nodule multi-branch fine classification module ThyroidDetⅡ: A multi-branch fine classification network is proposed to integrate features for multi-view diagnosis, in which each network branch captures and enhances a set of specific features commonly used by doctors to achieve fine classification of nodules. Specifically, in addition to the basic branch that extracts semantic features from the input patch, the multi-branch network also integrates a context branch to extract enhanced context features that are closely related to malignant nodules. In addition, an aspect ratio preservation strategy is introduced to process the input image patch, which can preserve the aspect ratio of the nodule during resizing to obtain better classification performance.

[0041] The specific implementation process of the nodule multi-branch fine classification module ThyroidDetⅡ is as follows:

[0042] The classification results produced by the nodule multi-scale coarse detection module ThyroidDetⅠ are relatively rough, considering that the proposal classification is only a supplementary task to help the main task of detecting nodules in the first step. However, these classification results still provide valuable diagnostic suggestions. The samples misclassified by the nodule multi-scale coarse detection module ThyroidDetⅠ may be challenging nodules, which cannot be well distinguished from the relatively weak diagnostic features that are mainly learned for nodule detection. To further improve the diagnostic performance, a more complex classification network is needed, which should include the views and attention of radiologists (especially when diagnosing challenging nodules) to guide the learning of more discriminative features for automatic nodule recognition. The present invention proposes a multi-branch fine classification module ThyroidDetⅡ to learn features for multi-view classification for fine classification of nodules.

[0043] Specifically, the multi-branch refined classification ThyroidDetⅡ adopts two complementary branches to learn multi-view features of nodules from different regions according to the doctor's attention. The structure of multi-branch refined classification ThyroidDetⅡ is shown in the figure. Figure 5 As shown, the subnetwork structures in all branches are the same, but the parameters are different. The present invention refines the ZFnet architecture into the backbone of a multi-branch fine classification network. It retains 5 convolutional layers (i.e., Conv1 uses 7×7 cores, Conv2 uses 5×5 cores, and Conv3-Conv5 uses 3×3 cores), two fully connected layers containing 4096 units, and two additional fully connected layers with 1024 and 2 outputs, respectively. After Conv1, Conv2, and Conv5, a maximum pooling operation with a window size of 3×3 is applied.

[0044] The first branch inputs the ROI of the original nodule. Since convolutional neural networks usually require input of standard size, the area block where the original ROI is located needs to be adjusted first. In order to retain the original aspect ratio information of nodules of different shapes, the original ROI is adjusted using the aspect ratio preserving cropping strategy. For each nodule of size h×w, it is first cropped and filled with a square area block of size: max(h,w). In this way, the nodule will not be stretched in only one direction, and the aspect ratio information will be retained for nodule classification based on convolutional neural networks. In this way, the nodule will not be stretched in only one direction, and the aspect ratio information will be retained for nodule classification based on CNN.

[0045] The second branch inputs an image patch that is 1.5 times the size of the original nodule ROI. The second branch inputs an image patch that is 1.5 times the size of the original nodule ROI. In addition to the information inside the nodule, the characteristics of the surrounding tissue (such as diffuse sclerosis and external echo differences) also affect the classification of the nodule. In order to consider this contextual information, an image patch centered on the original ROI and 1.5 times its size is cropped as the input of the second branch, which is called the context branch.

[0046] Finally, the multi-view features from the two parallel branches (i.e., the output of FC3 in each refined ZFNet) are concatenated and fused by the FC layer (256 units) for automatic nodule classification by a softmax classifier. Loss Function: To train the detection network, the multi-task loss function for end-to-end region proposal and nodule detection is defined as:

[0047]

[0048] Among them, {p i} represents the classification score of the i-th anchor point, p i is the probability that the current anchor point is classified as a foreground (target), p i * is the true label; {t i} represents the bounding box prediction value of the i-th anchor point, t i are the coordinates predicted by the network, t i * is the coordinate of the real target; L cls is the classification loss, using the cross-entropy loss function; L reg is the bounding box regression loss, using smooth L1 loss; N cls is the regularization factor of the classification loss; N reg is the regularization factor of the regression loss; λ is the weight to balance the two parts of the loss.

[0049] The quantitative test results are shown in the following table

[0050] Table 1 Quantitative comparison results of multi-scale detection networks with different nodule sizes (Note: the table headers are AP when IoU = 0.50: 0.95, 0.50, 0.75, area = small, medium, large)

[0051]

[0052] The ablation experiment results of the nodule multi-branch fine classification module ThyroidDet II are shown in the following table

[0053] Table 2 Ablation experiment results

[0054]

[0055] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A device for detecting thyroid nodules using ultrasound images based on prior knowledge, characterized in that: It includes nodule multi-scale coarse detection module ThyroidDetⅠ and nodule multi-branch fine classification module ThyroidDetⅡ; Nodule multi-scale coarse detection module ThyroidDetⅠ: A pyramid network is constructed to detect nodules of different scales, in which a top-down module is designed to integrate high-level semantic information with low-level feature maps for small nodule detection. In addition, in order to further improve the detection performance, the present invention refers to an important prior constraint, namely the true distribution of nodule size and shape, and designs reliable anchor points for each feature scale in proposal generation to better anchor the box. Prior knowledge helps to optimize the initialization and filtering of proposals, thereby achieving more accurate detection. The region proposal is trained by prior knowledge about the actual nodule size and shape distribution to perform coarse detection of nodules; Nodule multi-branch fine classification module ThyroidDetⅡ: A multi-branch fine classification network is proposed to integrate features for multi-view diagnosis, in which each network branch captures and enhances a set of specific features commonly used by doctors to achieve fine classification of nodules. Specifically, in addition to the basic branch that extracts semantic features from the input patch, the multi-branch network also integrates a context branch to extract enhanced context features that are closely related to malignant nodules. In addition, an aspect ratio preservation strategy is introduced to process the input image patch, which can retain the aspect ratio of the nodule during resizing to obtain better classification performance; The ultrasound image thyroid nodule detection device guided by prior knowledge first performs rough detection of nodules through the nodule multi-scale rough detection module ThyroidDetⅠ, and then performs fine classification of nodules through the nodule multi-branch fine classification module ThyroidDetⅡ.

2. The ultrasonic image thyroid nodule detection device based on prior knowledge guidance according to claim 1 is characterized in that: The specific implementation process of the nodule multi-scale coarse detection module ThyroidDetⅠ is as follows: 1) ResNet-50 is used as the backbone of the detection network. It consists of one convolutional layer (i.e., Conv1 using a 7×7 kernel) and four residual blocks (i.e., Conv2_x to Conv5_x), with Conv2_x to Conv5_x having 3, 4, 6, and 3 building blocks, respectively. Each building block contains 3 convolutional layers, using 1×1, 3×3, and 1×1 kernels, respectively. Each of them is followed by a maximum pooling layer that downsamples the feature map by a factor of 2. Therefore, similar to FPN, the backbone here forms a bottom-up path that computes a pyramid feature hierarchy with gradually increasing abstraction, but with decreasing spatial resolution. Conv2_x to Conv5_x have 3, 4, 6, and 3 building blocks, respectively. Each building block contains 3 convolutional layers, using 1×1, 3×3, and 1×1 kernels, respectively. 2) In order to accurately detect objects of different sizes, the present invention designs a top-down fusion module to hierarchically fuse high-level backbone features with low-level backbone features, and then use the merged features at different scales to detect objects of different sizes. Specifically, from Conv2_x to Conv5_x, the top-down fusion module is injected between any two adjacent blocks. In each fusion module, the high-level feature map is upsampled by 2 times using a convolutional layer, and then concatenated (via 1×1 convolution) with the transferred low-level feature map. The concatenated feature map is finally processed by a 3×3 convolutional layer to generate a fused feature map with reduced aliasing effect for detecting objects at that specific scale. In this way, the network proposed in the present invention generates five region proposals of different scales (i.e., Conv2_x, Conv3_x, Conv4_x, Conv5_x and Pool5), where the five feature map sizes are 1 / 4, 1 / 8, 1 / 16, 1 / 32 and 1 / 64 of the original input size, respectively in width, height and corresponding feature maps. These region proposals are finally sent to the R-CNN block to detect nodules of different sizes. 3) Detection anchors constrained by prior knowledge: The multi-scale detection network of the present invention uses anchors with designed size and aspect ratio (i.e., height divided by width) to predict region proposals for object detection. The present invention statistically calculates the aspect ratio distribution of nodules in the training data set in advance, and the result is shown in FIG4 . From FIG4 , there are two observations: (1) The aspect ratio range of all nodules in the training set is (0.2, 2.5), that is, the default anchor aspect ratio range of the experiment is also (0.2, 2.5); (2) The larger the nodule size, the smaller its corresponding aspect ratio, which indicates that the range of aspect ratios defined at the bottom layer should be larger than the range of aspect ratios defined at the top layer. Therefore, in order to ensure that the size and shape of the region proposal are consistent with the true nodule distribution, the anchor box of the lower feature map is constrained to a smaller size and a larger aspect ratio than the higher feature map. In general, an anchor box with 1 optional size (32, 64, 128, 256, 512) and corresponding 3 optional aspect ratios ((0.4, 1.0, 2.5), (0.4, 1.0, 2.5), (0.5, 1.0, 2.0), (0.7, 1.0, 1.5), (0.7, 1.0, 1.5)) is generated at each feature map position. The anchor box size and aspect ratio defined in different feature scale spaces can well cover different nodule sizes in the current image. Therefore, based on the prior knowledge about the aspect ratio and size of thyroid nodules, the auxiliary function of region proposal for automatic nodule detection can be effectively improved.

3. The ultrasonic image thyroid nodule detection device based on prior knowledge guidance according to claim 1 is characterized in that: The specific implementation process of the nodule multi-branch fine classification module ThyroidDetⅡ is as follows: The classification results produced by the nodule multi-scale coarse detection module ThyroidDetⅠ are relatively rough, considering that the proposal classification is only a supplementary task to help the main task of detecting nodules in the first step. However, these classification results still provide valuable diagnostic suggestions. The samples misclassified by the nodule multi-scale coarse detection module ThyroidDetⅠ may be challenging nodules, which cannot be well distinguished from the relatively weak diagnostic features that are mainly learned for nodule detection. To further improve the diagnostic performance, a more complex classification network is needed, which should include the views and attention of radiologists (especially when diagnosing challenging nodules) to guide the learning of more discriminative features for automatic nodule recognition. The present invention proposes a multi-branch fine classification module ThyroidDetⅡ to learn features for multi-view classification for fine classification of nodules. Specifically, the multi-branch refined classification ThyroidDetⅡ adopts two complementary branches to learn multi-view features of nodules from different regions according to the doctor's attention. The structure of the multi-branch refined classification ThyroidDetⅡ is shown in Figure 5. The subnetwork structures in all branches are the same, but the parameters are different. The present invention refines the ZFnet architecture into the backbone of the multi-branch refined classification network. It retains 5 convolutional layers (i.e., Conv1 uses 7×7 cores, Conv2 uses 5×5 cores, and Conv3-Conv5 uses 3×3 cores), two fully connected layers containing 4096 units, and two additional fully connected layers with 1024 and 2 outputs, respectively. After Conv1, Conv2, and Conv5, a maximum pooling operation with a window size of 3×3 is applied. The first branch inputs the ROI of the original nodule. Since convolutional neural networks usually require input of standard size, the area block where the original ROI is located needs to be adjusted first. In order to retain the original aspect ratio information of nodules of different shapes, the original ROI is adjusted using the aspect ratio preserving cropping strategy. For each nodule of size h×w, it is first cropped and filled with a square area block of size: max(h,w). In this way, the nodule will not be stretched in only one direction, and the aspect ratio information will be retained for nodule classification based on convolutional neural networks. In this way, the nodule will not be stretched in only one direction, and the aspect ratio information will be retained for nodule classification based on CNN. The second branch inputs an image patch that is 1.5 times the size of the original nodule ROI. The second branch inputs an image patch that is 1.5 times the size of the original nodule ROI. In addition to the information inside the nodule, the characteristics of the surrounding tissue (such as diffuse sclerosis and external echo differences) also affect the classification of the nodule. In order to consider this contextual information, an image patch centered on the original ROI and 1.5 times its size is cropped as the input of the second branch, which is called the context branch. Finally, the multi-view features from the two parallel branches (i.e., the output of FC3 in each refined ZFNet) are concatenated and fused by an FC layer (256 units) for a softmax classifier for automatic nodule classification.

4. The ultrasonic image thyroid nodule detection device based on prior knowledge guidance according to claim 1, characterized in that: The multi-task loss function for end-to-end region proposal and nodule detection is defined as follows: Among them, {p i } represents the classification score of the i-th anchor point, p i is the probability that the current anchor point is classified as a foreground (target), p i * is the true label; {t i } represents the bounding box prediction value of the i-th anchor point, t i are the coordinates predicted by the network, t i * is the coordinate of the real target; L cls is the classification loss, using the cross-entropy loss function; L reg is the bounding box regression loss, using smooth L1 loss; N cls is the regularization factor of the classification loss; N reg is the regularization factor of the regression loss; λ is the weight to balance the two parts of the loss.

Images