Fine-grained automatic identification and classification system for thyroid nodules based on weakly supervised learning

Abstract

The application relates to the technical field of data processing, in particular to a thyroid nodule fine-grained automatic identification and classification system based on weak supervision learning, which comprises a server, the server is pre-set with a thyroid fine-grained identification model, the server is used for acquiring an identification image, and a fine-grained identification result output by the thyroid fine-grained identification model according to the identification image; the thyroid fine-grained identification model comprises: a feature pyramid attention module, which is used for processing the identification image according to a feature pyramid attention algorithm; a spatial attention module, which is used for acquiring a binary mask, generating an attention map according to the binary mask, and positioning and discriminating a region; and a multi-network fusion module, which is used for acquiring output results of multiple backbone networks, fusing the output results, and generating a fine-grained identification result. The scheme can solve the technical problem that, when the thyroid nodule is classified in a fine-grained manner in the prior art, the classification accuracy is low due to the multi-scale and multi-network structure of an ultrasonic image.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of data processing, and particularly relates to a thyroid nodule fine-grained automatic identification and classification system based on weak supervision learning. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the combination of medicine and artificial intelligence disciplines makes more possibilities in temporary research. For example, the benign and malignant of pathological images containing thyroid nodules can be distinguished through an artificial intelligence algorithm, but such distinction is only a relatively coarse-grained distinction, and how to realize more fine-grained classification is a current key problem.

[0003] The weak supervision method is a common way in fine-grained classification, but the differences between different types of thyroid nodules are small, and different subtypes of thyroid nodules are highly similar, which may cause error propagation in fine-grained classification. In addition, due to the problems of collection means and shooting angles, the original ultrasound images may have multiple scales, which makes it difficult to process scale information, and the weak supervision method usually uses a multi-network model structure, which needs to fuse the intermediate information in the multiple networks, which may cause difficulty in fusing the intermediate information, resulting in an increase in classification error and even classification failure. SUMMARY

[0004] The application aims to provide a thyroid nodule fine-grained automatic identification and classification system based on weak supervision learning, so as to solve the technical problem of low classification accuracy caused by multiple scales of ultrasound images and multiple network structures in the prior art when classifying thyroid nodules in a fine-grained manner.

[0005] The application provides the following basic scheme:

[0006] The thyroid nodule fine-grained automatic identification and classification system based on weak supervision learning comprises a server,

[0007] The server is provided with a thyroid fine-grained identification model, and the server is used for acquiring an identification image and outputting a fine-grained identification result output by the thyroid fine-grained identification model according to the identification image.

[0008] The thyroid fine-grained identification model comprises:

[0009] A feature pyramid attention module is used for processing the identification image according to a feature pyramid attention algorithm.

[0010] A spatial attention module is used for acquiring a binary mask and generating an attention map according to the binary mask to position and distinguish a region.

[0011] A multi-network fusion module is used for acquiring output results of multiple backbone networks, fusing the output results, and generating a fine-grained identification result.

[0012] Further, the feature pyramid attention module is used to set the convolution kernel size to establish a feature pyramid, and adjust the dilation rate, and process the identification image according to the established feature pyramid.

[0013] Further, the feature pyramid attention module processes the identification image according to the following formula:

[0014]

[0015] wherein, is the input feature, is the multi-receptive field feature pyramid, is the uppermost branch of the feature pyramid, is the convolution operation.

[0016] Further, the feature pyramid attention module is used to establish a plurality of feature pyramids, and the convolution kernel and the dilation rate of different feature pyramids are different.

[0017] Further, the spatial attention module is used to integrate the top K accuracy activation classes, and generates an attention map according to the following formula:

[0018]

[0019]

[0020] wherein, is the attention map, is the height and width of the feature map, is the th activation class, indicates the accuracy of the th activation class, is the weight of the th activation class, is a real set.

[0021] Further, the binary mask is obtained according to the following formula:

[0022]

[0023] wherein, is the binary mask, and are weight coefficients.

[0024] Further, the binary mask satisfies the following conditions:

[0025] The value range needs to satisfy , and most of the values should be 1 or 0; the part with a value of 1 is located in the discrimination area, and the part with a value of 0 is located outside the discrimination area.

[0026] Further, the multi-network fusion module is configured to use iNaturalist and / or ImageNet as pre-training parameters of the backbone network.

[0027] Further, the multi-network fusion module is configured to generate network weights of different backbone networks according to output network entropy by the following formula:

[0028]

[0029]

[0030] wherein, is network entropy of the i-th backbone network, and E is a weight, then is network weight of the i-th backbone network, represents the i-th activation class accuracy.

[0031] Further, the multi-network fusion module is configured to generate a fine-grained recognition result according to network weights of different backbone networks by the following formula:

[0032]

[0033] wherein, is the fine-grained recognition result, is the number of backbone networks.

[0034] Beneficial effects:

[0035] 1. In the scheme, according to the multi-scale characteristics of the thyroid nodule ultrasound image, the feature pyramid attention module is improved, and two ways are adopted to improve the receptive field of the feature map. One is to set the convolution kernel to extract the feature map, and the other is to set the dilution rate to expand the receptive field through the way of dilated convolution. With the help of convolution kernel of different dilution rates, the fusion of multi-receptive field is realized, the deep feature information is obtained from different receptive fields, the feature extraction of complex image information is realized, and the processing of the ultrasound image is realized.

[0036] 2. In the scheme, in order to avoid that the judgment area is only affected by the maximum activation class, the spatial attention module comprehensively considers the top K accuracy, and the weight is obtained by the cumulative ratio of the top K accuracy class and the top K class result. At the same time of fully reflecting the classification result, the influence of the last few classification information is reduced. By adopting a plurality of activation class methods, the problem of error propagation of the spatial attention mechanism caused by the small difference between the thyroid nodule classes and the characteristics of the highly similar different subtypes of thyroid nodules is solved. The positioning accuracy of the judgment area is improved, and the error propagation problem is reduced.

[0037] ​​​3、In the scheme, the information output weight of each backbone network is dynamically adjusted through the multi-network fusion module, the intermediate information fusion of the multiple backbone networks is realized, and the weighted addition method is used, so that the intermediate information fusion is more reasonable, and the accuracy and robustness of the thyroid fine-grained identification model are ensured. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The network architecture diagram of the thyroid fine-grained identification model of the thyroid nodule fine-grained automatic identification and classification system based on weakly supervised learning is shown. DETAILED DESCRIPTION

[0039] The following will be further described in detail through specific embodiments:

[0040] EMBODIMENT

[0041] The thyroid nodule fine-grained automatic identification and classification system based on weakly supervised learning comprises a server, the server is pre-provided with a thyroid fine-grained identification model, the server is used for acquiring an identification image, and outputs a fine-grained identification result output by the thyroid fine-grained identification model according to the identification image.

[0042] As shown in the accompanying Figure 1 The thyroid fine-grained identification model comprises:

[0043] A feature pyramid attention module is used for processing the identification image according to a feature pyramid attention algorithm.

[0044] A spatial attention module is used for acquiring a binary mask, generating an attention map according to the binary mask, and positioning and discriminating a region.

[0045] A multi-network fusion module is used for acquiring output results of multiple backbone networks, fusing the output results, and generating a fine-grained identification result.

[0046] The feature pyramid attention module is used for setting a convolution kernel size to establish a feature pyramid, and adjusting an inflation rate, and processing the identification image according to the established feature pyramid. The feature pyramid attention module is used for establishing multiple feature pyramids, and the convolution kernel and the inflation rate of different feature pyramids are different.

[0047] The feature pyramid attention module processes the identification image according to the following formula:

[0048]

[0049] Wherein, is an input feature, is a multi-receptive field feature pyramid, is an uppermost branch of the feature pyramid, is a convolution operation.

[0050] As attached Figure 1 As shown, first, the output of backbone network 1 Enter as input While keeping the size unchanged, the feature map dimension is reduced and the . Another part, Enter the feature pyramid composed of different convolution kernels In this example, we set different convolution kernel sizes and expansion rates to expand the receptive field. and The convolution kernel has dilation rates of 3 and 5, respectively, and Multiply them together to fuse the features of each region. Finally, perform a global pooling operation to obtain the feature pyramid output.

[0051] In this scheme, the thyroid fine-grained recognition model uses the fusion of multiple receptive fields by fusing convolution kernels with different expansion rates. This can not only achieve the goal, but also reduce parameters and prevent overfitting to a certain extent.

[0052] The class activation map is a method of locating the discriminant area by generating an attention map. The formula is as follows:

[0053]

[0054]

[0055] in, Refers to taking the maximum along the C axis, that is, the direction of the maximum activation class, It is the feature map obtained through operations such as convolution and pooling. are the parameters that need to be trained, are the height, width and number of channels of the feature map respectively.

[0056] Considering the maximum activation class, the maximum activation may not always be located within the correct discrimination region, which may lead to propagation errors. Therefore, the spatial attention module is used to integrate the top K accuracy activation classes and generate an attention map according to the following formula:

[0057]

[0058]

[0059] in, is the attention map, are the height and width of the feature map, For the Name activation class, Indicates the Name activation class accuracy, For the The weights of the activation classes, is the set of real numbers.

[0060] The comprehensive top K accuracy activation class is obtained by adding the ratio of the top K accuracy classes to the top K class results, which not only reflects the weight of the top few, but is not affected by the results of the last few.

[0061] To generate an attention map, a binary mask is required.

[0062] The binary mask satisfies the following conditions:

[0063] The value range must meet , and most of the values ​​should be 1 or 0; the part with the value of 1 is located in the discrimination area, and the part with the value of 0 is located outside the discrimination area.

[0064] In order to meet the above conditions, the binary mask is used as a trainable parameter and obtained through training. Two layers are set The convolutional layer, and through Function and Perform scale transformation, that is, obtain the binary mask according to the following formula:

[0065]

[0066] in, is a binary mask, and is the weight coefficient, specifically: is the weight coefficient of the activation class, are the weight coefficients in the scale transformation process, and the two weight coefficients are set separately in advance.

[0067] Adjusting the binary mask as the network changes during training helps to better obtain appropriate values. Next, the mask is used to find the smallest rectangle that captures all the discriminative regions in the image. This rectangle is resized to the input image ratio and used for cropping. Finally, the cropped image is sampled and mapped to the reference resolution. The resulting image becomes the input to the next network.

[0068] The multi-network fusion module is used to use iNaturalist and / or ImageNet as pre-trained parameters for the backbone network. In this embodiment, backbone network 1 uses iNaturalist pre-trained parameters, and backbone network 2 uses ImageNet pre-trained parameters. In other embodiments, backbone network 1 may use ImageNet pre-trained parameters, backbone network 2 may use iNaturalist pre-trained parameters, backbone networks may both use ImageNet pre-trained parameters, or backbone network 2 may both use iNaturalist pre-trained parameters.

[0069] To more reasonably fuse the intermediate output information, the method of weighted addition needs to be considered. Therefore, the multi-network fusion module is configured to generate network weights of different backbone networks according to the output network entropy by the following formula:

[0070]

[0071]

[0072] wherein, is the network entropy of the i-th backbone network, E is taken as the weight, and then is the network weight of the i-th backbone network, represents the i-th activation class accuracy. According to the result of the output network entropy, the confidence of different networks is determined, which is used as the basis for determining the weight. The smaller the output network entropy of the backbone network is, the higher the confidence of the network is. The multi-network fusion module is configured to generate a fine-grained recognition result according to the network weights of different backbone networks by the following formula:

[0073]

[0074]

[0075]

[0076] wherein, is the fine-grained recognition result, is the number of backbone networks.

[0077] The fine-grained recognition result includes: benign: nodular goiter, Hashimoto's thyroiditis, other benign; malignant: thyroid papillary carcinoma, follicular carcinoma.

[0078] The scheme combines the characteristics of thyroid ultrasound images, constructs a spatial attention algorithm, and improves the multi-scale multi-network information fusion strategy, finally establishes a thyroid nodule fine-grained automatic recognition classification system, realizes the organic combination of medical and artificial intelligence disciplines, reduces the artificial cost of medical treatment, and improves the accuracy and robustness of thyroid nodule recognition.

[0079] In other embodiments, the server is also configured to verify the thyroid fine-grained recognition model, and the server is also configured to obtain experimental data, i.e., ultrasound images, and the server is also configured to crop the pictures to a uniform size, and rotate, horizontally transform, and perform data enhancement processing. Through data enhancement, the number of experimental samples is expanded, and model training under small sample conditions is realized.

[0080] ​​​The above-mentioned are only embodiments of the present application, and the common knowledge of the specific structure and characteristics in the scheme is not described too much herein. The ordinary skilled person in the art knows all the ordinary technical knowledge in the field of the present application before the application date or the priority date, can know all the prior art in the field, and has the ability to apply the conventional experimental means before that date. The ordinary skilled person in the art can perfect and implement the present scheme under the guidance of the present application, combined with their own ability. Some typical known structures or known methods should not be an obstacle for the ordinary skilled person in the art to implement the present application. It should be pointed out that, for those skilled in the art, without departing from the structure of the present application, a number of modifications and improvements can be made, which should also be considered as the protection scope of the present application. The protection scope of the present application should be subject to the content of its claims, and the specific implementation mode and the like in the specification can be used to explain the content of the claims.

Claims

1. A fine-grained automatic identification and classification system for thyroid nodules based on weakly supervised learning, characterized by: Including servers, The server is pre-installed with a thyroid fine-grained recognition model, and is used to obtain a recognition image and output a fine-grained recognition result output by the thyroid fine-grained recognition model based on the recognition image; The fine-grained thyroid recognition model includes: Feature pyramid attention module, used to process the recognition image according to the feature pyramid attention algorithm; The spatial attention module is used to obtain a binary mask and generate an attention map based on the binary mask to locate and identify the area; Multi-network fusion module, used to obtain the output results of multiple backbone networks and fuse the output results to generate fine-grained recognition results; The feature pyramid attention module is used to set the convolution kernel size to build a feature pyramid and adjust the expansion rate, and process the recognition image according to the established feature pyramid; The feature pyramid attention module processes the recognition image according to the following formula: in, is the input feature, It is a feature pyramid with multiple receptive fields. It is the top branch of the feature pyramid. is the convolution operation; The spatial attention module is used to integrate the top K accuracy activation classes and generate an attention map according to the following formula: in, is the attention map, are the height and width of the feature map, For the Name activation class, Indicates the Name activation class accuracy, For the The weights of the activation classes, is the set of real numbers; The binary mask is obtained according to the following formula: in, is a binary mask, and is the weight coefficient.

2. The fine-grained automatic identification and classification system for thyroid nodules based on weakly supervised learning according to claim 1 is characterized by: The feature pyramid attention module is used to build multiple feature pyramids, and different feature pyramids have different convolution kernels and expansion rates.

3. The fine-grained automatic identification and classification system for thyroid nodules based on weakly supervised learning according to claim 1 is characterized by: The binary mask satisfies the following conditions: The value range must meet , and most of the values ​​should be 1 or 0; the part with the value of 1 is located in the discrimination area, and the part with the value of 0 is located outside the discrimination area.

4. The fine-grained automatic identification and classification system for thyroid nodules based on weakly supervised learning according to claim 1 is characterized by: The multi-network fusion module is used to use iNaturalist and / or ImageNet as pre-trained parameters of the backbone network.

5. The fine-grained automatic identification and classification system for thyroid nodules based on weakly supervised learning according to claim 1 is characterized by: The multi-network fusion module is used to generate network weights of different backbone networks based on the output network entropy using the following formula: in, For the The network entropy of the backbone network, taking E as the weight, then For the The network weight of the backbone network, Indicates the Name activation class accuracy; Indicates the number of networks.

6. The fine-grained automatic identification and classification system for thyroid nodules based on weakly supervised learning according to claim 5 is characterized by: The multi-network fusion module is used to generate fine-grained recognition results based on the network weights of different backbone networks using the following formula: in, For fine-grained recognition results, is the number of backbone networks.

Images