Thyroid disease image detection and identification method, terminal device and system

Abstract

The invention provides a thyroid disease image detection and identification method, terminal equipment and a thyroid disease image detection and identification system. The detection and identification method comprises the following steps: inputting a to-be-detected infrared image and a to-be-detected visible light image into a trained multilayer convolutional neural network based on the to-be-detected infrared image and the to-be-detected visible light image obtained at the same moment in a thyroid disease area, a first temperature distribution feature is extracted from the to-be-detected infrared image, a first morphological feature is extracted from the to-be-detected visible light image, and the first morphological feature comprises thyroid texture and thyroid color; extracting and highlighting a depth temperature distribution feature and a depth morphological feature from the first temperature distribution feature and the first morphological feature; and inputting the depth temperature distribution characteristics and the depth morphological characteristics into a multi-task learning model, and detecting the type of the thyroid disease in real time. By means of the method, infrared images and visible light images are detected in real time, and various thyroid diseases are detected.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image detection, and in particular to a detection and recognition method, terminal equipment and system for thyroid disease images. Background Art

[0002] Thyroid diseases include hyperthyroidism, hypothyroidism, thyroid nodules and other types. These diseases are common in all kinds of people and have a significant impact on the quality of life of patients. Traditional methods for detecting thyroid diseases include ultrasound, radionuclide scanning, serological tests, etc. However, radionuclide scanning and certain serological tests require injection of contrast agents or blood draws, which bring discomfort and risks to patients; high-precision ultrasound and radionuclide scanning equipment are expensive and the examination costs are high. Summary of the invention

[0003] The purpose of the present invention is to provide a detection and identification method, terminal equipment and system for thyroid disease images, so as to realize the simultaneous detection of multiple thyroid diseases, real-time detection of infrared images and visible light images, and meet the needs of clinical real-time applications.

[0004] The technical solution provided by the present invention is as follows:

[0005] The present invention provides a method for detecting and identifying thyroid disease images, comprising the following steps:

[0006] Based on an infrared image to be detected and a visible light image to be detected obtained at the same time of a thyroid diseased area, the infrared image to be detected and the visible light image to be detected are input into a trained multi-layer convolutional neural network, and a first temperature distribution feature is extracted from the infrared image to be detected, and a first morphological feature is extracted from the visible light image to be detected, wherein the first morphological feature includes thyroid texture and thyroid color.

[0007] The deep temperature distribution feature and the deep morphological feature are extracted and highlighted from the first temperature distribution feature and the first morphological feature.

[0008] The deep temperature distribution features and deep morphological features are input into a multi-task learning model to detect the type of thyroid disease in real time; thyroid diseases include hyperthyroidism, hypothyroidism, and thyroid nodules.

[0009] Specifically, through the detection and recognition method of thyroid disease images proposed in the present invention, the temperature distribution characteristics of infrared images and the morphological characteristics of visible light images are extracted, and the characteristics of various thyroid diseases (such as hyperthyroidism, hypothyroidism, and thyroid nodules) are jointly learned to improve the accuracy of real-time detection.

[0010] Furthermore, before extracting the first temperature distribution feature from the infrared image to be detected and extracting the first morphological feature from the visible light image to be detected, the method further includes:

[0011] Using data enhancement technology, the infrared image sample is preprocessed into a first infrared image sample, and the visible light image sample is preprocessed into a first visible light image sample; using a generative adversarial network, the first infrared image sample is generated into a first virtual infrared image sample, and the first visible light image sample is generated into a first virtual visible light image sample; and the multi-layer convolutional neural network is trained using the first virtual infrared image sample and the first virtual visible light image sample.

[0012] Specifically, the present invention uses data enhancement technology to increase the diversity of training data and improve the generalization ability of the model; uses a generative adversarial network to generate virtual samples of infrared images and visible light images, further expands the data set for training, and enhances the robustness of the model.

[0013] Further, extracting and highlighting the deep temperature distribution feature and the deep morphological feature from the first temperature distribution feature and the first morphological feature includes:

[0014] A residual network is used to extract deep temperature distribution features from the first temperature distribution features, and a residual network is used to extract deep morphological features from the first morphological features; the deep temperature distribution features include color temperature area change features and color temperature area boundary features, and the deep morphological features include thyroid color change features and thyroid shape change features.

[0015] The self-attention mechanism and multi-head attention mechanism are used to enhance the features of the deep temperature distribution characteristics and deep morphological characteristics, so that different thyroid diseases can highlight the corresponding deep temperature distribution characteristics and deep morphological characteristics.

[0016] Specifically, the present invention utilizes a residual network to extract deep temperature distribution features and deep morphological features, alleviates the problem of gradient vanishing, and improves the training effect of the deep network; an attention mechanism is used to highlight the features of key areas of the thyroid gland, suppress interference from irrelevant areas, and improve detection accuracy.

[0017] Furthermore, before inputting the deep temperature distribution features and deep morphological features into the multi-task learning model, the following steps are included:

[0018] A multi-task learning model is constructed, which includes a shared feature layer and a task-specific layer; the shared feature layer fuses the deep temperature distribution features and the deep morphological features to generate a comprehensive feature map of the thyroid gland; the task-specific layer learns and classifies the comprehensive feature map of the thyroid gland to predict the type of thyroid disease.

[0019] Specifically, the present invention improves the detection ability and generalization performance of the model by constructing a multi-task learning model and jointly learning multiple diseases such as hyperthyroidism, hypothyroidism, and thyroid nodules. The structure of the shared feature layer and the task-specific layer is adopted to balance the relationship between the tasks and avoid mutual interference between the tasks.

[0020] Further, extracting a first temperature distribution feature from the infrared image to be detected, and extracting a first morphological feature from the visible light image to be detected, includes:

[0021] In the input layer of the multi-layer convolutional neural network, the infrared image to be detected and the visible light image to be detected are superimposed and fused; in the middle layer of the multi-layer convolutional neural network, the first temperature distribution feature is extracted from the infrared image to be detected, and the first morphological feature is extracted from the visible light image to be detected; the middle layer includes several convolutional layers; in the fully connected layer of the multi-layer convolutional neural network, the first temperature distribution feature and the first morphological feature are output to the residual network.

[0022] Specifically, the interaction of multimodal input data (such as infrared images and visible light images) is considered in the input layer, which helps to capture the early dependencies between different modalities; the middle layer performs some independent feature extraction on each modality to retain the unique information of each modality (such as the first temperature distribution feature and the first morphological feature).

[0023] Furthermore, in the middle layer of the multi-layer convolutional neural network, extracting a first temperature distribution feature from the infrared image to be detected and extracting a first morphological feature from the visible light image to be detected includes:

[0024] In the first and second convolutional layers, primary and intermediate temperature distribution features are extracted in turn for the infrared images to be detected, and primary and intermediate morphological features are extracted in turn for the visible light images to be detected; in the third convolutional layer, advanced temperature distribution features are extracted from the intermediate temperature distribution features, and advanced morphological features are extracted from the intermediate morphological features for each thyroid disease; in the fourth convolutional layer, the advanced temperature distribution features are enhanced to form the first temperature distribution features, and the advanced morphological features are enhanced to form the first morphological features.

[0025] Specifically, the convolutional layers of the multi-layer convolutional neural network are used to extract features of infrared images and visible light images respectively, and each layer of the network captures local feature information at different levels.

[0026] Furthermore, based on the number of samples, pruning technology is used to delete neurons or convolution kernels that contribute little when the multi-layer convolutional neural network is used to extract the first temperature distribution feature and the first morphological feature; quantization technology is used to convert the parameters of the multi-layer convolutional neural network from floating point numbers to low-order integers; distillation technology is used to train a small multi-layer convolutional neural network with a small number of convolutional layers to imitate the output of a large multi-layer convolutional neural network with a large number of convolutional layers; the number of convolutional layers of the large multi-layer convolutional neural network is greater than the number of layers of the small multi-layer convolutional neural network.

[0027] Specifically, through algorithm optimization and model compression technology, the computational complexity and parameter amount of the model are reduced, and model pruning, quantization and distillation technology are used to achieve real-time detection of infrared images and visible light images, which not only meets the needs of real-time clinical applications, but also improves the detection speed and efficiency, facilitating real-time monitoring and diagnosis.

[0028] The present invention also provides a terminal device for identifying images of thyroid diseases, comprising a processor for calling instructions stored in a memory, wherein when the instructions are executed, the terminal device executes any one of the methods for detecting and identifying thyroid disease images.

[0029] The present invention also provides a thyroid disease image detection and recognition system, comprising:

[0030] The infrared thermal imaging device and the visible light camera collect infrared images and visible light images of the diseased area of ​​the thyroid gland of a patient at the same time, and send the infrared images and visible light images to the terminal device for processing.

[0031] The terminal device obtains the deep temperature distribution characteristics of the infrared image and the deep morphological characteristics of the visible light image to detect the type of thyroid disease in real time.

[0032] Through the detection and identification method, terminal device and system of thyroid disease images provided by the present invention, temperature distribution features and morphological features of infrared images and visible light images are synchronously extracted, and the characteristics of various thyroid diseases are identified based on a multi-task learning model, thereby realizing real-time detection of thyroid diseased areas in infrared images and visible light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The following will further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of a thyroid disease image detection and identification method, terminal device and system in a clear and easy-to-understand manner with reference to the accompanying drawings.

[0034] Figure 1 It is a structural schematic diagram of an embodiment of a thyroid disease image detection and recognition system of the present invention;

[0035] Figure 2It is a flow chart of an embodiment of a method for detecting and identifying thyroid disease images of the present invention;

[0036] Figure 3 is a flow chart of another embodiment of a method for detecting and identifying thyroid disease images of the present invention;

[0037] Figure 4 is a flow chart of another embodiment of a method for detecting and identifying thyroid disease images of the present invention;

[0038] Figure 5 is a flow chart of another embodiment of a method for detecting and identifying thyroid disease images of the present invention;

[0039] Figure 6 is a flow chart of another embodiment of a method for detecting and identifying thyroid disease images of the present invention;

[0040] Figure 7 is a flow chart of another embodiment of a method for detecting and identifying thyroid disease images of the present invention;

[0041] Figure 8 It is a schematic diagram of the structure of a multi-layer convolutional neural network of a method for detecting and identifying thyroid disease images of the present invention;

[0042] Fig. 9 is a flow chart of another embodiment of a method for detecting and identifying thyroid disease images of the present invention;

[0043] Fig.10 is a flow chart of another embodiment of a method for detecting and identifying thyroid disease images of the present invention;

[0044] Fig.11 It is a structural schematic diagram of a multi-task learning model of a thyroid disease image detection and recognition method of the present invention;

[0045] Fig.12 is a schematic diagram of an image of a hyperthyroidism patient identified by the present invention;

[0046] Fig.13 is a schematic diagram of an image of a hypothyroidism patient identified by the present invention;

[0047] Fig.14 It is a schematic diagram of an image of a patient with thyroid nodules identified by the present invention. DETAILED DESCRIPTION

[0048] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0049] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections.

[0050] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0051] In addition, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0053] Thyroid disease is a type of disease that affects the function of the thyroid gland, which is an endocrine gland located in the neck that is responsible for the production and secretion of thyroid hormones that affect many metabolic activities in the body. Thyroid disease mainly includes three manifestations: hyperthyroidism, hypothyroidism, and thyroid nodules.

[0054] Traditional methods for detecting thyroid diseases include ultrasound, radionuclide scanning, and serological tests, but they each have some limitations: the results of ultrasound examinations are highly dependent on the operator and it is difficult to distinguish certain types of nodules; radionuclide scanning injects contrast agents, which carries a certain risk of radiation exposure; and serological tests require blood draws, which causes discomfort to patients.

[0055] Infrared thermal imaging technology has gradually become a new choice for thyroid disease detection due to its non-contact, rapid and non-invasive advantages. Infrared thermal imaging technology has been gradually applied in the field of medical imaging in recent years. This technology detects the thermal radiation on the surface of the human body to form a temperature distribution image, and can obtain temperature information of the thyroid area non-contact and non-invasively.

[0056] Therefore, the present invention proposes a detection and recognition method, terminal device and system for thyroid disease images, which adopts a multi-layer convolutional neural network structure, combines infrared images and visible light images, and extracts the temperature features of infrared images and the morphological features of visible light images layer by layer. The attention mechanism is introduced to highlight the features of key areas, a multi-task learning model is constructed to jointly learn multiple thyroid diseases, and the model is compressed to reduce the computational complexity, so as to realize real-time detection of thyroid disease images.

[0057] By using infrared images and visible light images for multimodal feature fusion, using multi-layer convolutional neural networks and residual network structures, and combining innovative data enhancement technology, the diversity of training data is increased, the generalization ability of the model is improved, and the adaptability to individual differences of different patients is enhanced. This can effectively respond to the thyroid disease conditions of different patients and improve the accuracy and reliability of classification results.

[0058] The following is a description with reference to the accompanying drawings:

[0059] In one embodiment of the present invention, Figure 1 , which shows a schematic diagram of the structure of a thyroid disease image detection and recognition system 100 provided in some embodiments of the present disclosure. The detection and recognition system 100 includes: an infrared thermal imaging device 110, a visible light camera 120 and a terminal device 130.

[0060] The infrared thermal imaging device 110 and the visible light camera 120 collect infrared images and visible light images of a patient's thyroid diseased area at the same time, and send the infrared images and visible light images to the terminal device 130 for processing.

[0061] The terminal device 130 obtains the depth temperature distribution characteristics of the infrared image and the depth morphological characteristics of the visible light image, and detects the type of thyroid disease in real time.

[0062] Specifically, the infrared thermal imaging device 110 is an instrument that creates images by detecting infrared radiation emitted by an object. The infrared thermal imaging device 110 can provide reliable temperature data under various lighting conditions, can "see" temperature distribution maps, and provide thermal images of objects, playing an important role in many applications that require accurate temperature measurement and object detection.

[0063] The visible light camera 120 is a camera that works by capturing images in the visible light band. It is capable of taking images within the visible light range and can show the texture, subtle textures, and shape contours of the object's surface.

[0064] After the terminal device 130 acquires the infrared image of the infrared thermal imaging device 110 and the visible light image of the visible light camera 120, it extracts the temperature characteristics of the infrared image and the morphological characteristics of the visible light image, and combines the temperature and morphological characteristics to identify the characteristic pattern of a certain thyroid disease.

[0065] Through the thyroid disease image detection and recognition system 100 proposed in the present invention, images of two modes are acquired for recognition, and images of the thyroid region are detected dynamically and in real time.

[0066] In one embodiment of the present invention, Figure 2 As shown, it shows a flowchart of the method for detecting and identifying thyroid disease images performed by the terminal device 130 of the present disclosure, including the following steps:

[0067] S100 is based on an infrared image to be detected and a visible light image to be detected obtained at the same time of a thyroid diseased area, inputs the infrared image to be detected and the visible light image to be detected into a trained multi-layer convolutional neural network, extracts a first temperature distribution feature from the infrared image to be detected, and extracts a first morphological feature from the visible light image to be detected, wherein the first morphological feature includes thyroid texture and thyroid color.

[0068] S200 extracts and highlights the deep temperature distribution feature and the deep morphological feature from the first temperature distribution feature and the first morphological feature.

[0069] S300 inputs deep temperature distribution features and deep morphological features into a multi-task learning model to detect the type of thyroid disease in real time; thyroid diseases include hyperthyroidism, hypothyroidism, and thyroid nodules.

[0070] Specifically, infrared images and visible light images of the patient's thyroid region at the same time are obtained. Among them, the infrared image can reflect the metabolic activity and blood flow changes of the thyroid gland and is less affected by the environmental optical fiber; the visible light image can provide detailed information on the surface morphology of the thyroid gland, but is easily affected by light and shadow. Before feature extraction, the collected infrared images and visible light images need to be normalized to eliminate noise and interference and improve image quality.

[0071] like Figure 3The figure shows the structural block diagram of the model for identifying multiple thyroid diseases. The trained multi-layer convolutional neural network (CNN) is used to extract temperature and morphological features of the infrared image and visible light image to be detected. Different convolutional layers capture feature information at different levels, and finally output the first temperature distribution feature and the first morphological feature (including texture and color).

[0072] Through multi-layer convolution operations and attention mechanisms, the deep features in the infrared images to be detected and the visible light images to be detected are fully extracted, the model's perception of features of different sizes is enhanced, the accuracy and specificity of thyroid disease detection are improved, and misdiagnosis and missed diagnosis are reduced.

[0073] The multi-task learning model jointly learns the characteristics of multiple diseases such as hyperthyroidism, hypothyroidism, thyroid nodules, etc., and detects temperature anomalies and morphological anomalies of deep temperature distribution characteristics and deep morphological characteristics through the multi-task learning model to identify specific thyroid diseases.

[0074] In this embodiment, the detection and recognition method of thyroid disease images proposed in the present invention is used to extract the temperature distribution characteristics of infrared images and the morphological characteristics of visible light images, and jointly learn the characteristics of various thyroid diseases (such as hyperthyroidism, hypothyroidism, and thyroid nodules) to achieve real-time detection of thyroid diseases.

[0075] In one embodiment of the present invention, Figure 4 As shown, before extracting the first temperature distribution feature from the infrared image to be detected and extracting the first morphological feature from the visible light image to be detected, it is necessary to expand the training data set. The above step S100 may include:

[0076] S110 uses a data enhancement technology to preprocess the infrared image sample into a first infrared image sample, and preprocess the visible light image sample into a first visible light image sample.

[0077] S120 generates a first virtual infrared image sample from the first infrared image sample and generates a first virtual visible light image sample from the first visible light image sample by using a generative adversarial network.

[0078] S130 uses the first virtual infrared image sample and the first virtual visible light image sample to train the multi-layer convolutional neural network.

[0079] Specifically, a variety of data enhancement techniques such as random rotation, scaling, translation, and noise addition are used to increase the diversity of training data and enable the model to learn a wider range of features. Since infrared images are more sensitive to temperature changes, it is necessary to consider simulating different temperature conditions or environmental factors to generate the first infrared image sample; it is necessary to simulate natural phenomena such as illumination changes, shadows, and reflections for visible light images to generate the first visible light image sample.

[0080] The first infrared image sample and the first visible light image sample are combined with a generative adversarial network (GAN) to generate a first virtual infrared image sample and a first virtual visible light image sample, so as to further expand the training data set to help train the diagnosis model.

[0081] In this embodiment, the purpose and significance of each specific operation (such as flipping, rotating, scaling, cropping, color conversion, etc.) can be more clearly understood by using the data enhancement module, and can be better applied to the actual detection model. These operations increase the diversity of training data, simulate various possible actual situations, and improve the generalization ability and robustness of the model.

[0082] In one embodiment of the present invention, Figure 5 As shown, when the above sample is trained, the multi-layer convolutional neural network needs to be optimized, and the above step S100 also includes:

[0083] Based on the number of samples, S140 uses pruning technology to delete neurons or convolution kernels that contribute little when extracting the first temperature distribution feature and the first morphological feature in the multi-layer convolutional neural network.

[0084] S150 uses quantization technology to convert the parameters of multi-layer convolutional neural networks from floating-point numbers to low-bit integers.

[0085] S160 uses distillation technology to train a small multi-layer convolutional neural network with a small number of convolutional layers to imitate the output of a large multi-layer convolutional neural network with a large number of convolutional layers; the number of convolutional layers of the large multi-layer convolutional neural network is greater than the number of layers of the small multi-layer convolutional neural network.

[0086] Specifically, model compression technology (pruning, quantization, and distillation) is used to reduce the computational complexity and parameter quantity of the model to achieve real-time detection of infrared images and visible light images.

[0087] Pruning technology reduces the computational complexity and storage requirements of the model by deleting redundant neurons or convolution kernels that contribute little to feature extraction (for example, neurons that are not important for extracting the first temperature distribution feature or the first morphological feature). The pruned model can significantly reduce the computational complexity while ensuring accuracy.

[0088] Quantization technology converts the parameters of the model from floating-point numbers to low-bit integers (such as 8 bits), reducing the storage space and computational complexity of the model. The quantized model runs more efficiently on hardware and is suitable for embedded devices and real-time applications.

[0089] Distillation technology improves the performance of a small model by training a smaller student model to imitate the output of a large teacher model. The student model is lighter in the inference phase and is suitable for real-time detection scenarios. Small multi-layer convolutional neural networks usually have fewer convolutional layers (2-4) and fully connected layers (1-2); the convolution kernel size can be 3×3 or 5×5; the pooling layer is usually a 2×2 pooling window. Large multi-layer convolutional neural networks usually have dozens to hundreds of convolutional layers, and the number of parameters can reach hundreds of millions to billions.

[0090] In this embodiment, algorithm optimization and model compression technology are used to reduce the computational complexity and parameter quantity of the model, and model pruning, quantization and distillation technology are used to achieve real-time detection of infrared images and visible light images, which not only meets the needs of real-time clinical applications, but also improves the detection speed and efficiency, facilitating real-time monitoring and diagnosis.

[0091] In one embodiment of the present invention, Figure 6 As shown, after the model training is completed, the first temperature distribution feature and the first morphological feature can be extracted. The above step S100 also includes:

[0092] S170 superimposes and fuses the infrared image to be detected and the visible light image to be detected in the input layer of the multi-layer convolutional neural network.

[0093] S180 extracts a first temperature distribution feature from the infrared image to be detected and a first morphological feature from the visible light image to be detected in the middle layer of the multi-layer convolutional neural network; the middle layer includes several convolutional layers.

[0094] S190 outputs the first temperature distribution feature and the first morphological feature to the residual network in the fully connected layer of the multi-layer convolutional neural network.

[0095] Specifically, first, the inputs of different modalities (for example, infrared images and visible light images) are initially fused, and then the fused data is input to the next layer of the multi-layer convolutional neural network for processing. In the input layer of the convolutional network, the inputs of different modalities are first simply connected, spliced, or fused together by other methods to form a comprehensive input feature map. For example, the pixel-level features of infrared images and visible light images can be superimposed or juxtaposed, and then this combined feature map is used as the input of the multi-layer convolutional neural network (CNN).

[0096] Secondly, the features of different modalities are fused in the middle layer of the multi-layer convolutional neural network (such as the middle layer of the multi-layer convolutional neural network) to extract the first temperature distribution features of the infrared image and the first morphological features of the visible light image. In the mid-term fusion, the data of each modality is firstly extracted together for preliminary features, and then some independent features are extracted for each modality. For example, preliminary features of infrared images and visible light images are extracted through several layers of convolutional networks respectively, and then the features are fused, which can not only retain the unique information of each modality, but also combine the advantages of multi-modality at the appropriate time to improve the complexity of feature expression.

[0097] Finally, the first temperature distribution feature and the first morphological feature extracted are output to the residual network in the fully connected layer of the multi-layer convolutional neural network to further extract the deep features.

[0098] In this embodiment, by considering the interaction of multimodal input data at the input layer, it is helpful to capture the early dependencies between different modes; the middle layer performs some independent feature extraction on each mode to retain the unique information of each mode (such as the first temperature distribution feature, the first morphological feature).

[0099] In one embodiment of the present invention, Figure 7 As shown, the first temperature distribution feature and the first morphological feature need to be extracted at different convolutional layers, and the above step S180 also includes:

[0100] S181 performs primary and intermediate temperature distribution feature extraction on the infrared image to be detected in the first convolution layer and the second convolution layer, and performs primary and intermediate morphological feature extraction on the visible light image to be detected.

[0101] The third convolutional layer S182 extracts high-level temperature distribution features from intermediate temperature distribution features and high-level morphological features from intermediate morphological features for each thyroid disease.

[0102] S183 enhances the high-level temperature distribution feature in the fourth convolutional layer to form a first temperature distribution feature, and enhances the high-level morphological feature to form a first morphological feature.

[0103] Specifically, Figure 8 As shown in Figure 1, the network structure of the feature extraction model of the multi-layer convolutional neural network includes 4 convolutional layers and 4 pooling layers. The specific structure of extracting image features using a multi-layer convolutional neural network (CNN) is as follows:

[0104] Input layer: The infrared image to be detected and the visible light image to be detected, the size is H×W×C, where H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image.

[0105] The first convolution layer: 64 3×3 convolution kernels, ReLU activation function, output feature map H×W×64; the first pooling layer: 2×2 maximum pooling, output feature map H / 2×W / 2×64; the second convolution layer: 128 3×3 convolution kernels, ReLU activation function, output feature map H / 2×W / 2×128; the second pooling layer: 2×2 maximum pooling, output feature map H / 4×W / 4×128. Among them, the first and second convolution layers are used for primary and intermediate feature extraction of infrared images and visible light images of various diseases such as hyperthyroidism, hypothyroidism, and thyroid nodules. The pooling layer is used to reduce the size of the feature map and reduce the amount of calculation.

[0106] Among them, in infrared images, the first convolutional layer usually extracts primary temperature distribution features, such as edges, textures and simple temperature changes in the image. The primary temperature distribution feature extraction can identify the boundaries of the thyroid area and rough temperature differences. These features can determine the approximate area and temperature range of the disease; in visible light images, the first convolutional layer extracts basic morphological features, edges, corners and basic texture features. The primary morphological features can identify the outline of the thyroid area and simple color changes, helping to determine the basic morphology and approximate color distribution of the thyroid.

[0107] In infrared images, the second convolutional layer combines and abstracts the primary temperature distribution features to form intermediate temperature distribution features, such as local high or low temperature areas. In visible light images, the second convolutional layer detects intermediate morphological features of local texture patterns, color gradients, and structural details of the thyroid gland, and identifies the details of the thyroid gland and potential lesion areas.

[0108] The third convolution layer: 256 3×3 convolution kernels, ReLU activation function, output feature map H / 4×W / 4×256; the third pooling layer: 2×2 maximum pooling, output feature map H / 8×W / 8×256. A dedicated third convolution layer is set for each disease to extract high-level features of different diseases.

[0109] Among them, in infrared images, the high-level temperature distribution features extracted by the third convolutional layer are in-depth analysis of the intermediate temperature distribution features, identifying more abstract and complex temperature change patterns related to the disease (significant patterns of specific diseases), such as unique thermal patterns in inflammatory areas or abnormal temperature distribution characteristics; in visible light images, the third convolutional layer can extract high-level morphological features of subtle structural changes, lesion characteristics or irregular color patterns of the thyroid gland.

[0110] The fourth convolution layer: 512 3×3 convolution kernels, ReLU activation function, output feature map H / 8×W / 8×512; the fourth pooling layer: 2×2 maximum pooling, output feature map H / 16×W / 16×512. In the fourth convolution layer, the advanced temperature distribution features and advanced morphological features are further integrated and enhanced to form the first temperature distribution features and the first morphological features.

[0111] The feature extraction model of the multi-layer convolutional neural network also includes two fully connected layers and an output layer. The fully connected layer is combined with the residual network structure to solve the gradient vanishing problem in deep network training and extract the deep features of the image. Among them, the first fully connected layer: 1024 neurons, ReLU activation function; the second fully connected layer: 512 neurons, ReLU activation function; output layer: softmax function, n-class output (n is the number of categories of thyroid disease classification).

[0112] In this embodiment, feature extraction is performed on infrared images and visible light images respectively through the convolution layers of a multi-layer convolutional neural network. Each layer of the network captures local feature information at different levels. After fusing features at different levels, it has stronger nonlinear expression ability, which helps to distinguish different disease characteristics.

[0113] In one embodiment of the present invention, Fig. 9 As shown, when extracting deeper depth features from the first temperature distribution feature and the first morphological feature, the above step S200 includes:

[0114] S210 uses a residual network to extract deep temperature distribution features from the first temperature distribution features, and uses a residual network to extract deep morphological features from the first morphological features; the deep temperature distribution features include color temperature area change features and color temperature area boundary features, and the deep morphological features include thyroid color change features and thyroid shape change features.

[0115] S220 uses self-attention mechanism and multi-head attention mechanism to strengthen the features of deep temperature distribution characteristics and deep morphological characteristics, so that different thyroid diseases highlight the corresponding deep temperature distribution characteristics and deep morphological characteristics.

[0116] Specifically, the residual network is used in the fully connected layer to extract the deep features of infrared images and visible light images, and the residual network contains skip connections, which can directly add the input features to the output of the convolutional layer, retaining the original feature information. For example, the residual network can recognize that some irregular temperature distributions or specific textures are typical manifestations of certain types of thyroid diseases; it can also combine multiple basic features such as contours or abnormal shapes to distinguish normal and abnormal areas.

[0117] In infrared images, the first temperature distribution feature directly extracts the temperature data of the infrared image, which only reflects the heat distribution of the thyroid gland and displays different temperature ranges with different color codes; while the deep temperature distribution feature can better capture subtle changes and complex patterns in temperature distribution, and can extract more abstract and high-level temperature patterns from the first temperature distribution feature, and identify certain special thermal patterns that may be related to specific types of thyroid diseases.

[0118] In visible light images, the first morphological features mainly identify some basic morphological features that reflect the morphological information of the thyroid gland, including thyroid texture (such as the lines or structure on the surface of the thyroid gland) and color (such as color changes or spots of the thyroid gland); while the deep morphological features can extract more complex morphological patterns or details, such as subtle texture changes, color gradients, and morphological abnormalities of the thyroid gland. The extraction of deep morphological features is more effective for disease detection.

[0119] In infrared images, the self-attention mechanism focuses on areas with abnormally high or low temperatures; in visible light images, the self-attention mechanism focuses on areas with abnormal colors or textures. The self-attention mechanism can identify and focus on image areas that may be related to the disease, rather than treating all image pixels equally, thereby improving the accuracy of diagnosis.

[0120] Multi-Head Attention is a technology that enhances feature expression. In image processing, it uses multiple "heads" to perform attention calculations simultaneously, and can focus on image features from different angles or scales. For example, the first attention head focuses on temperature distribution, the second focuses on texture features, and the third focuses on color patterns. Each head generates different feature representations, which are integrated together to enable the network to capture multi-level and multi-angle information of the image. This diverse feature capture method enables the model to extract useful information more accurately when faced with complex and changing inputs, thereby improving the overall model's expressiveness and performance.

[0121] In this embodiment, the present invention utilizes a residual network to extract deep temperature distribution features and deep morphological features, alleviates the problem of gradient vanishing, and improves the training effect of the deep network; an attention mechanism is used to highlight the features of key areas of the thyroid gland, suppress interference from irrelevant areas, and improve detection accuracy.

[0122] In one embodiment of the present invention, Fig.10 As shown, when identifying thyroid diseases based on deep features and multi-task learning models, the above step S300 includes:

[0123] S310 builds a multi-task learning model, which includes a shared feature layer and a task-specific layer.

[0124] The S320 shared feature layer fuses the deep temperature distribution features and deep morphological features to generate a comprehensive feature map of the thyroid gland.

[0125] The S330 task-specific layer learns and classifies the comprehensive feature map of the thyroid gland and predicts the type of thyroid disease.

[0126] Specifically, Fig.11 As shown in the figure, in a multi-task learning model, there are usually some basic features that are shared by multiple tasks, and these features capture the common information of the input data. For example, there are some common visual features (such as shape, contour, and overall pattern of temperature distribution) in infrared images or visible light images. The shared feature layer integrates these common features as the basis for multiple tasks. For example, when performing recognition of infrared images and visible light images at the same time, the shared features may include structural morphology information and temperature distribution information of the thyroid gland.

[0127] The deep temperature distribution features and deep morphological features are integrated into a comprehensive thyroid feature map, from which multiple subsequent tasks can obtain useful feature information. The comprehensive thyroid feature map includes not only the deep temperature distribution features of the thermal pattern of a specific type of thyroid disease, but also the deep morphological features of the thyroid's subtle texture changes, color gradients, and morphological abnormalities.

[0128] In the shared feature layer, multiple tasks share the same set of basic features, but each task has its own specific goals and requirements, and a task-specific layer is needed to customize specific tasks for different diseases. Each task-specific layer is optimized according to its respective task goals to ensure the best performance for that task. This is usually achieved through task-specific loss functions. For example, classification tasks are optimized through the cross loss function and the Adam optimizer, while detection tasks may be optimized using IoU (intersection over union) or other target detection loss functions.

[0129] In patients with hyperthyroidism, the shared feature layer sends the deep temperature distribution features and deep morphological features in the comprehensive feature map of hyperthyroidism to the task-specific layer of hyperthyroidism. The task-specific layer can identify that the color performance of the infrared image is disordered and asymmetric, the boundaries of the color temperature area are unclear and discontinuous, the color temperature area spans from high to low, there are obvious temperature spans and color distribution abnormalities, the temperature at the thyroid position is red, and the temperature is high, such as Fig.12 As shown in the infrared thermal image of a patient with hyperthyroidism, the thyroid gland is clearly distributed in a red butterfly shape. Therefore, the characteristic pattern of hyperthyroidism can be identified by the infrared thermal imaging image showing a higher temperature and the visible light image showing abnormal enlargement or color change of the thyroid region.

[0130] In patients with hypothyroidism, the shared feature layer sends the deep temperature distribution features and deep morphological features in the comprehensive feature map of hypothyroidism to the task-specific layer of hypothyroidism, such as Fig.13 As shown, the direct manifestation of the infrared image is a decrease in the temperature of the thyroid area, and the visible light image shows that the thyroid area becomes lighter or atrophied.

[0131] In patients with thyroid nodules, the shared feature layer sends the deep temperature distribution features and deep morphological features in the comprehensive feature map of the thyroid nodules to the task-specific layer of the thyroid nodules, such as Fig.14 As shown in the figure, the infrared image shows that the overall distribution is disordered and asymmetric, the regional boundaries are unclear, discontinuous dots, and the color temperature areas are high and low. Therefore, thyroid nodules can be identified by abnormal temperature points shown in infrared images and obvious nodules or irregular shapes on the thyroid surface shown in visible light images.

[0132] In this embodiment, the present invention improves the detection ability and generalization performance of the model by constructing a multi-task learning model to jointly learn multiple diseases such as hyperthyroidism, hypothyroidism, and thyroid nodules. The structure of the shared feature layer and the task-specific layer is adopted to balance the relationship between the tasks and avoid mutual interference between the tasks.

[0133] In one embodiment of the present invention, a terminal device includes a processor and a memory, wherein the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory to implement the detection and identification method of thyroid disease images in the above-mentioned corresponding method embodiment.

[0134] The terminal device may be a desktop computer, a notebook, a PDA, a tablet computer, a mobile phone, a human-computer interaction screen, etc. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that this example does not constitute a limitation on the terminal device, and may include more or fewer components, or a combination of certain components, or different components.

[0135] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0136] The memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. The memory may also be an external storage device of the terminal device, such as a plug-in hard disk equipped on the smart device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory may include both an internal storage unit of the device terminal and an external storage device. The memory is used to store the computer program and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is to be output.

[0137] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed with hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0139] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A method for detecting and identifying thyroid disease images, characterized in that: The following steps are involved: Based on an infrared image to be detected and a visible light image to be detected acquired at the same time of a diseased area of ​​a thyroid gland, the infrared image to be detected and the visible light image to be detected are input into a multi-layer convolutional neural network that has been trained, a first temperature distribution feature is extracted from the infrared image to be detected, and a first morphological feature is extracted from the visible light image to be detected, wherein the first morphological feature includes a thyroid gland texture and a thyroid gland color; Extract and highlight the deep temperature distribution feature and the deep morphological feature from the first temperature distribution feature and the first morphological feature; The deep temperature distribution features and the deep morphological features are input into a multi-task learning model to detect the type of the thyroid disease in real time; the thyroid diseases include hyperthyroidism, hypothyroidism, and thyroid nodules.

2. The method for detecting and identifying thyroid disease images according to claim 1, characterized in that: Before extracting the first temperature distribution feature from the infrared image to be detected and extracting the first morphological feature from the visible light image to be detected, the method further includes: Using data enhancement technology, preprocessing the infrared image sample into a first infrared image sample, and preprocessing the visible light image sample into a first visible light image sample; Using a generative adversarial network, generating a first virtual infrared image sample from the first infrared image sample, and generating a first virtual visible light image sample from the first visible light image sample; The multi-layer convolutional neural network is trained using the first virtual infrared image samples and the first virtual visible light image samples.

3. The method for detecting and identifying thyroid disease images according to claim 1, characterized in that: The extracting and highlighting the deep temperature distribution feature and the deep morphological feature from the first temperature distribution feature and the first morphological feature comprises: Extracting the deep temperature distribution feature from the first temperature distribution feature using a residual network, and extracting the deep morphological feature from the first morphological feature using the residual network; the deep temperature distribution feature includes a color temperature region change feature and a color temperature region boundary feature, and the deep morphological feature includes a thyroid color change feature and a thyroid shape change feature; The self-attention mechanism and the multi-head attention mechanism are used to strengthen the features of the deep temperature distribution characteristics and the deep morphological characteristics, so that different thyroid diseases highlight the corresponding deep temperature distribution characteristics and deep morphological characteristics.

4. The method for detecting and identifying thyroid disease images according to claim 1, characterized in that: Before the deep temperature distribution feature and the deep morphological feature are input into the multi-task learning model, the method includes: Constructing the multi-task learning model, wherein the multi-task learning model includes a shared feature layer and a task-specific layer; The shared feature layer fuses the deep temperature distribution feature and the deep morphological feature to generate a comprehensive thyroid feature map; The task-specific layer learns and classifies the comprehensive characteristic map of the thyroid gland and predicts the type of the thyroid disease.

5. The method for detecting and identifying thyroid disease images according to claim 1, characterized in that: The step of extracting the first temperature distribution feature from the infrared image to be detected and extracting the first morphological feature from the visible light image to be detected includes: Superimposing and fusing the infrared image to be detected and the visible light image to be detected at the input layer of the multi-layer convolutional neural network; In the middle layer of the multi-layer convolutional neural network, the first temperature distribution feature is extracted from the infrared image to be detected, and the first morphological feature is extracted from the visible light image to be detected; the middle layer includes a plurality of convolutional layers; The first temperature distribution feature and the first morphological feature are output to the residual network in the fully connected layer of the multi-layer convolutional neural network.

6. The method for detecting and identifying thyroid disease images according to claim 5, characterized in that: The method of extracting the first temperature distribution feature from the infrared image to be detected and extracting the first morphological feature from the visible light image to be detected in the middle layer of the multi-layer convolutional neural network includes: In the first convolution layer and the second convolution layer, primary and intermediate temperature distribution feature extraction is performed on the infrared image to be detected, and primary and intermediate morphological feature extraction is performed on the visible light image to be detected; The third convolutional layer extracts high-level temperature distribution features from the intermediate temperature distribution features and high-level morphological features from the intermediate morphological features for each of the thyroid diseases; In the fourth convolutional layer, the high-level temperature distribution feature is enhanced to form the first temperature distribution feature, and the high-level morphological feature is enhanced to form the first morphological feature.

7. The method for detecting and identifying thyroid disease images according to any one of claims 1 to 6, characterized in that: Based on the number of samples, a pruning technique is used to delete neurons or convolution kernels that contribute little when the multi-layer convolutional neural network is used to extract the first temperature distribution feature and the first morphological feature; The parameters of the multi-layer convolutional neural network are converted from floating point numbers to low-order integers using a quantization technique; A distillation technique is used to train a small multi-layer convolutional neural network with a small number of convolutional layers to imitate the output of a large multi-layer convolutional neural network with a large number of convolutional layers; the number of convolutional layers of the large multi-layer convolutional neural network is greater than the number of layers of the small multi-layer convolutional neural network.

8. A terminal device, characterized in that: An image for identifying thyroid diseases, comprising a processor for calling instructions stored in a memory, wherein when the instructions are executed, a terminal device executes a method for detecting and identifying thyroid disease images as described in any one of claims 1 to 7.

9. A thyroid disease image detection and recognition system, characterized in that: include: The infrared thermal imaging device and the visible light camera collect infrared images and visible light images of the diseased area of ​​the thyroid gland of a patient at the same time, and send the infrared images and the visible light images to the terminal device for processing; The terminal device obtains the depth temperature distribution characteristics of the infrared image and the depth morphological characteristics of the visible light image, and detects the type of the thyroid disease in real time.

Images