Thyroid image processing method, device, electronic device and storage medium

Abstract

The present application provides a thyroid image processing method, device, electronic device and storage medium. The method includes: obtaining an image to be tested, the image to be tested includes an image obtained by detecting a thyroid part; inputting the image to be tested into a preset deep learning model to obtain a first detection result of the preset deep learning model on the image to be tested; when the first detection result indicates that there are thyroid nodules in the image to be tested, the lesion image area in the first detection result indicating the presence of thyroid nodules in the image to be tested is input into a tested machine learning model to obtain a second detection result of the machine learning model on the lesion image area. In this solution, the preset deep learning model and the machine learning model are combined to detect the image to be tested, and the machine learning model does not need to detect images where there are no thyroid nodules and areas that are not lesion image areas, thereby reducing the amount of calculation, improving the calculation efficiency, and helping to improve the accuracy of thyroid nodule detection.

Description

Technical Field

[0001] The present application relates to the field of computer image processing, and in particular, to a thyroid image processing method, device, electronic device and storage medium. Background Art

[0002] At present, in the process of detecting thyroid nodules in the human body, ultrasound detection is usually used to obtain images of the thyroid area, and then the detected images are manually or machine-based for thyroid nodule detection. Among them, the manual recognition method requires high professional knowledge of the operators and has low recognition efficiency. The machine recognition method is limited by the current processing method of the network model, and the recognition accuracy and efficiency need to be improved. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a thyroid image processing method, device, electronic device and storage medium, which can improve the accuracy and efficiency of thyroid nodule detection.

[0004] In order to achieve the above purpose, the embodiments of the present application are implemented in the following ways:

[0005] In a first aspect, an embodiment of the present application provides a thyroid image processing method, the method comprising: acquiring an image to be tested, the image to be tested comprising an image obtained by detecting a thyroid part; inputting the image to be tested into a preset deep learning model to obtain a first detection result of the preset deep learning model on the image to be tested; when the first detection result indicates that there are thyroid nodules in the image to be tested, inputting a lesion area in the first detection result indicating the presence of thyroid nodules in the image to be tested into a tested machine learning model to obtain a second detection result of the machine learning model on the lesion area, the second detection result indicating the severity of the thyroid nodules in the lesion area.

[0006] In the above-mentioned implementation, a preset deep learning model is used to detect whether there are thyroid nodules in the image to be tested, and when there are thyroid nodules, the lesion image area in the image to be tested is input into the machine learning model. In this way, the machine learning model does not need to detect images without thyroid nodules and areas that are not lesion image areas, thereby reducing the amount of calculation and improving the calculation efficiency. In addition, the combination of the preset deep learning model and the machine learning model is conducive to improving the accuracy of thyroid nodule detection.

[0007] In combination with the first aspect, in some optional implementations, the preset deep learning model includes a residual network and a candidate region network, and before acquiring the image to be tested, the method further includes:

[0008] Acquire a training data set and a test data set, wherein each image in the training data set and the test data set corresponds to at least one label, and the label is used to mark an image area where a thyroid nodule exists, or to mark an image area where a thyroid nodule does not exist;

[0009] Inputting the training data set into the residual network to obtain training images containing thyroid nodules in the training data set;

[0010] Inputting the training image region where the thyroid nodules exist into the candidate region network to obtain a candidate region, wherein the candidate region is a lesion image region segmented from the training image and indicating the existence of the thyroid nodules;

[0011] Using the candidate region to train a gcForest model to obtain a trained gcForest model;

[0012] The trained gcForest model is tested according to the test data set to obtain the tested machine learning model.

[0013] In the above implementation, the residual network, the candidate region network, and the gcForest model are trained by using a training data set, and the gcForest model is tested by using a test data set, so that the accuracy of model detection can be improved.

[0014] In combination with the first aspect, in some optional implementations, using the candidate region to train a gcForest model to obtain a trained gcForest model includes:

[0015] Extracting features of the candidate area through sliding windows of different specified window sizes to obtain feature maps corresponding to the different specified window sizes;

[0016] The feature map is input into the random forest in the gcForest model, the gcForest model is trained, and the trained gcForest model is obtained.

[0017] In the above implementation, features are extracted from the candidate regions through sliding windows of multiple sizes to train the gcForest model, which is beneficial for reducing training samples while achieving the purpose of model training.

[0018] In combination with the first aspect, in some optional implementations, obtaining the image to be tested includes:

[0019] According to the image type of the image to be tested, an interface corresponding to the image type is selected to acquire the image to be tested, and the image type includes at least one of PNG, JPG, BMP, and DICOM.

[0020] In the above-mentioned implementation, the image to be tested is acquired through an interface corresponding to the image type. In this way, the images to be tested of different image types can be detected and processed, which is beneficial to improving the application range of the images to be tested.

[0021] In combination with the first aspect, in some optional implementations, inputting the image to be tested into a preset deep learning model to obtain a first detection result of the preset deep learning model on the image to be tested includes:

[0022] Inputting the image to be tested into the residual network in the preset deep learning model to obtain an intermediate detection result of whether a thyroid nodule exists in the image to be tested;

[0023] When the intermediate detection result indicates that a thyroid nodule exists in the image to be tested, the image to be tested is input into the candidate area network in the deep learning model to obtain the first detection result, which includes the lesion area segmented from the image to be tested and indicating the presence of a thyroid nodule.

[0024] In combination with the first aspect, in some optional embodiments, the image to be tested input into the preset deep learning model is an image obtained after the image to be tested is preprocessed, and the preprocessing includes at least one of image denoising and image enhancement.

[0025] In combination with the first aspect, in some optional embodiments, the image to be measured includes an image obtained by magnetic resonance imaging of the thyroid gland.

[0026] In a second aspect, the present application further provides a thyroid image processing device, the device comprising:

[0027] An acquisition unit, used for acquiring an image to be tested, wherein the image to be tested includes an image obtained by detecting a thyroid gland part;

[0028] A first detection unit, configured to input the image to be tested into a preset deep learning model to obtain a first detection result of the preset deep learning model on the image to be tested;

[0029] The second detection unit is used to input the lesion image area in the first detection result indicating the presence of thyroid nodules in the image to be tested into a tested machine learning model when the first detection result indicates the presence of thyroid nodules in the image to be tested, so as to obtain a second detection result of the machine learning model on the lesion image area, wherein the second detection result indicates the severity of the thyroid nodules in the lesion image area.

[0030] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory coupled to each other, wherein the memory stores a computer program, and when the computer program is executed by the processor, the electronic device executes the above method.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed on a computer, the computer executes the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic flow chart of a thyroid image processing method provided in an embodiment of the present application.

[0034] Figure 2 A schematic diagram of the network structure of the deep learning model provided in an embodiment of the present application.

[0035] Figure 3 A schematic diagram of the network structure of the machine learning model provided in the embodiment of the present application.

[0036] Figure 4 A block diagram of a thyroid image processing device provided in an embodiment of the application.

[0037] Icons: 200 - thyroid image processing device; 210 - acquisition unit; 220 - first detection unit; 230 - second detection unit. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. It should be noted that the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0039] The present application provides an electronic device that can be used to analyze and detect captured thyroid images, and to assist users in detecting thyroid nodules.

[0040] The electronic device may include a processing module and a storage module. The storage module stores a computer program, and when the computer program is executed by the processing module, the electronic device can perform each step of the following thyroid image processing method.

[0041] In this embodiment, the electronic device may further include other modules, for example, the electronic device may further include a communication module for establishing a communication connection with other devices. The processing module, the storage module, and the communication module are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0042] The electronic device may be, but is not limited to, a personal computer, a server, or the like.

[0043] Please refer to Figure 1 The present application provides a thyroid image processing method, which can be applied to the above-mentioned electronic device, and each step of the method is executed or implemented by the electronic device. The method may include the following steps:

[0044] Step S110, obtaining an image to be tested, wherein the image to be tested includes an image obtained by detecting a thyroid gland part;

[0045] Step S120, inputting the image to be tested into a preset deep learning model to obtain a first detection result of the preset deep learning model on the image to be tested;

[0046] Step S130, when the first detection result indicates that there are thyroid nodules in the image to be tested, the lesion area in the first detection result indicating the presence of thyroid nodules in the image to be tested is input into a tested machine learning model to obtain a second detection result of the machine learning model on the lesion area, and the second detection result indicates the severity of the thyroid nodules in the lesion area.

[0047] In the above-mentioned implementation, a preset deep learning model is used to detect whether there are thyroid nodules in the image to be tested, and when there are thyroid nodules, the lesion image area in the image to be tested is input into the machine learning model. In this way, the machine learning model does not need to detect images without thyroid nodules and areas that are not lesion image areas, thereby reducing the amount of calculation and improving the calculation efficiency. In addition, the combination of the preset deep learning model and the machine learning model is conducive to improving the accuracy of thyroid nodule detection.

[0048] The following is a detailed description of each step of the method:

[0049] Before step S110, the method includes the steps of model training and testing. Figure 2 and Figure 3, the preset deep learning model can be Mask R-CNN. Mask R-CNN adds a branch of prediction segmentation Mask (which can be interpreted as "mask") on the basis of Faster R-CNN (Region Convolutional Neural Network). Mask R-CNN includes a residual network and a candidate region network (Region Proposals Networks, RPN). Before step S110, the method can include steps S101 to S105, as follows:

[0050] Step S101, obtaining a training data set and a test data set, wherein each image in the training data set and the test data set corresponds to at least one label, and the label is used to mark an image area where a thyroid nodule exists, or mark an image area where a thyroid nodule does not exist;

[0051] Step S102, inputting the training data set into the residual network to obtain training images containing thyroid nodules in the training data set;

[0052] Step S103, inputting the training image region where the thyroid nodules exist into the candidate region network to obtain a candidate region, wherein the candidate region is a lesion image region segmented from the training image and indicating the existence of the thyroid nodules;

[0053] Step S104, training a gcForest model using the candidate region to obtain a trained gcForest model;

[0054] Step S105, testing the trained gcForest model according to the test data set to obtain the tested machine learning model.

[0055] Understandably, the training data set and the test data set are image data prepared in advance by the user. Both the training data set and the test data set include multiple images, and the number of images included can be flexibly determined according to actual conditions, and is not specifically limited here. The training data set is used to train the model, and the test data set is used to test the model to improve the accuracy of model detection.

[0056] Both the training data set and the test data set are images obtained by photographing the thyroid gland of the human body, and corresponding labels can be set manually for each image. The label content can be added according to the actual situation. For example, the label indicates whether there are nodules in the thyroid gland area in the figure, and when there are thyroid nodules, the area where the thyroid nodules are present is marked. Among them, the image can be, but is not limited to, an MRI (Magnetic Resonance Imaging) image obtained by magnetic resonance imaging of the thyroid gland.

[0057] When annotating the area of ​​thyroid nodules in the images of the training data set and the test data set, users can flexibly annotate and segment according to the actual situation. For example, the user can use the QPainter tool in the PyQt5.QtGui package to directly draw a rectangle or ellipse for the thyroid nodule area on the image, and then call the rectangle and ellipse interface in the OpenCV package to directly segment the outlined area; or, use the QPainter in the PyQt5.QtGui package to select points according to mouse clicks, then connect all the points into polygons, and finally call the polygon interface in the OpenCV package to directly segment the outlined area; or, use the Livewire algorithm in the OpenCV package to implement the magnetic area segmentation operation; or based on the preset threshold range of the grayscale value of the pixel point in the image, use the threshold tool in the OpenCV package to customize the threshold size and generate the threshold segmentation result, and the preset threshold range is the grayscale threshold range corresponding to the presence of thyroid nodules. It should be noted that the software toolkits for thyroid nodule area annotation (such as the PyQt5.QtGui package, the QPainter tool, the OpenCV package, etc.) are well known to those skilled in the art.

[0058] In this embodiment, compared with ultrasonic imaging, MRI images have better tissue resolution and contrast, can perform multi-directional and multi-level scanning of the thyroid gland and its surrounding tissues at the same time, and can provide information on morphology, tumor function and biological characteristics. Therefore, using MRI images to detect thyroid micronodules is beneficial to improving the accuracy of detection.

[0059] In this embodiment, the residual network can be flexibly determined according to actual conditions. For example, the residual network can be a ResNet101 convolutional neural network. The residual network is used to extract image features of nodules or micronodules in the image, and then through the Feature Pyramid Networks (FPN), it can process multi-scale changes in thyroid nodule detection and extract more thyroid nodule features while increasing the amount of calculation very little.

[0060] The Candidate Region Network is a lightweight neural network that performs convolution operations by scanning feature maps through a sliding window. By setting different window sizes and aspect ratios, Anchors are generated. Anchors are used to define relative position reference points in the image.

[0061] Please refer again Figure 2 The candidate region network can perform convolution processing on the image output by the residual network and the optimized ADC (Apparent Diffusion Coefficient) to obtain an "intermediate image". Among them, the ADC image optimization method is well known to those skilled in the art. ADC is used to describe the parameters of the diffusion speed and range of different water molecules in magnetic resonance diffusion weighted imaging. The ADC image is a pre-acquired image and is well known to those skilled in the art.

[0062] The candidate region network can output two types of information for each Anchor: the first is the prediction of the foreground or background category of the Anchor. The foreground category represents that there is a certain probability that one or more types of targets exist in the Anchor. The background category refers to other objects other than the target to be detected, which will be filtered out later. The second is the fine-tuning of the preset bounding box. When the center of the target does not completely coincide with the center of the foreground Anchor, that is, when there is an offset, the output position information (x, y, w, h) is the percentage change to accurately adjust the Anchor position and fit the target position more correctly. Among them, x and y are the horizontal and vertical coordinates of the center of the target, w refers to the width of the preset bounding box, and h refers to the height of the preset bounding box. There is a phenomenon of overlapping in the foreground Anchor. The Anchor with a low foreground score is filtered out by non-maximum suppression, and the Anchor with the highest score is retained. Finally, the candidate region is obtained (for example, Figure 2 The box area corresponding to the "middle picture" in the figure is the Region Proposals.

[0063] Please refer again Figure 2 , the candidate region network can input the acquired Region Proposals into the ROI Align layer (referring to the custom operation name, the generated region proposal box can be mapped to the feature map, and the ROI Align mapping can be written). After the layer maps the Region Proposals to the feature map, it outputs the regions of interest (ROIs) of the same size for subsequent classification, regression, and mask generation. Among them, FC (Fully Connected Layers) is used to implement border regression, result classification, etc. FCN (Fully Convolutional Networks) is used to implement image segmentation.

[0064] Mask R-CNN generates a mask for detecting thyroid nodules by adding a mask branch to the final output (i.e., replacing all the last fully connected layers in the traditional convolutional neural network with convolutional layers).

[0065] In this embodiment, the gcForest model can optimize the diagnostic ability of the model according to the size characteristics of the thyroid nodule image area. That is, the gcForest model can extract the lesion image area with thyroid nodules through different windows, and can realize the training and testing of the model on the basis of reducing the training samples to improve the accuracy of detection. Exemplarily, step S104 may include:

[0066] Extracting features of the candidate area through sliding windows of different specified window sizes to obtain feature maps corresponding to the different specified window sizes;

[0067] The feature map is input into the random forest in the gcForest model, the gcForest model is trained, and the trained gcForest model is obtained.

[0068] In this embodiment, the gcForest model may include a multi-grained scanning unit and a cascade forest unit. The multi-grained scanning unit can generate feature maps of different sizes by setting different scanning window sizes, and then input two types of preset random forests to generate input vectors for the cascade forest unit. Among them, the two types of preset random forests refer to: random forests and completely random forests, both of which have different leaf nodes, and both generate feature maps as inputs to the cascade unit. The cascade forest unit generates a detection result by learning the feature map generated by the multi-grained scanning unit. The detection result includes content indicating the severity of the thyroid nodule. For example, the detection result may include the size of the thyroid nodule, the severity of benign or malignant, etc.

[0069] In this embodiment, the images in the training data set and the test data set input to the residual network and the gcForest model can be preprocessed images. For example, data denoising and data enhancement are performed on MRI images. For example, different wavelet basis functions are used to perform a two-dimensional wavelet transform on the thyroid MRI image to complete the denoising of the MRI image; the Gaussian Laplace operator is used to smooth the MRI image to achieve the purpose of strengthening the texture features of the MRI image. The preprocessed image is then input into the Mask R-CNN network model to train the residual network, candidate region network, gcForest model and other models in the Mask R-CNN network model, which is conducive to improving the accuracy of model recognition detection.

[0070] Since the maximum diameter of thyroid micronodule lesions is usually less than or equal to 1 cm, and the aspect ratio of the lesion area is between 0.5 and 2.0, the sliding window size in the candidate region network can be set to [4, 8, 16] (where the numbers 4, 8, and 16 refer to the length or width of the pixels of the three sliding windows of different sizes, respectively), and the window aspect ratio is set to [0.5, 1, 2] (the numbers 0.5, 1, and 1.2 correspond to the aspect ratios of the three sliding windows, respectively) to optimize the Region Proposals generated by the candidate region network and generate the final thyroid micronodule automatic detection and segmentation model, as follows:

[0071] L=L cls +λ1L box +λ2L mask (1)

[0072] In formula (1), L is the total loss function, L cls is the classification loss function, L box is the detection loss function, L Mask is the segmentation loss function, and λ1 and λ2 are the balance weight parameters. Since it is necessary to further extract the radiomics feature values ​​of the lesion area, the segmentation branch of the Mask R-CNN model can use the average binary cross entropy loss function to extract the feature values, as follows:

[0073]

[0074] In formula (2), m represents the dimension of the two-dimensional mask, i and j are loop variables, k represents the category, and y and Represent the label value and the predicted value respectively.

[0075] In this embodiment, the electronic device can detect the location of the thyroid nodule lesion area using Mask R-CNN, but it is not correctly classified (for example, the severity of the thyroid nodule is unclear). The gcForest model can be used to classify the severity of the thyroid nodule, so it is necessary to train the gcForest model to determine the severity of the thyroid nodule.

[0076] Please refer again Figure 3 , the images input by Mask R-CNN to the gcForest model can include two types of images, for example, the input of path A is a diffusion weighted image, and the input of path B is an apparent diffusion coefficient image. According to the size characteristics of thyroid nodules, managers can set the sliding window size of the multi-scale scanning process of the gcForest model to different specified window sizes, such as 3×3 and 5×5, respectively. 3×3 means that the width and height of the window are both 3 pixels, and 5×5 means that the width and height of the window are both 5 pixels.

[0077] Please refer again Figure 3 To further improve the recognition accuracy of the gcForest model, the electronic device can integrate the extracted radiomics features into the cascade process. For example, the electronic device concatenates the extracted radiomics features of the micronodules with the output value and input vector of the previous layer of random forest as the input of the next layer, and then repeats the above process until the model verification converges. Figure 3 The 120div shown refers to the feature vector composed of 120 types of radiomics features extracted.

[0078] In this embodiment, when the gcForest model is tested, the labels of the images in the test data set are marked with content corresponding to the severity level of the thyroid nodules. The gcForest model is tested by the test data set until the model converges, thereby improving the accuracy of the gcForest model in classifying the severity of thyroid nodules.

[0079] In step S110, the electronic device can obtain the image to be tested in a manner that can be flexibly determined according to actual conditions. For example, the electronic device can obtain the image to be tested from a USB flash drive or a server storing the device to be tested.

[0080] In addition, the electronic device can also select corresponding interfaces to obtain the image to be tested according to different image types. For example, step S110 may include: according to the image type of the image to be tested, selecting an interface corresponding to the image type to obtain the image to be tested, wherein the image type includes but is not limited to at least one of PNG (Portable Network Graphics), JPEG (Joint Photographic Experts Group), BMP (Bitmap), and DICOM (Digital Imaging and Communications in Medicine). Among them, PNG, JPG, BMP, and DICOM are image data formats, which are formats well known to those skilled in the art.

[0081] For example, if the image type of the image to be tested is PNG, JPG, or BMP, the OpenCV interface in the electronic device is used to read and obtain the basic information of the image to be tested, and then the image is converted from the BGR (Blue Green Red) color space to the RGB (Red Green Blue) color space, and finally the QImage interface in PyQt5 is used to display the image to be tested. PyQt5 is a framework for Python binding Digia QT5 applications, which is a module of Python and is a toolkit well known to those skilled in the art.

[0082] If the image type of the image to be tested is DICOM, the Pydicom interface is used to read the image, obtain the basic information of the image, and then convert it into JPEG format, then convert the image from the BGR color space to the RGB color space, and finally display the image using the QImage interface in PyQt5. Among them, the OpenCV interface, the Pydicom interface, and the QImage interface are interfaces for obtaining images in electronic devices, and are well-known interfaces to those skilled in the art.

[0083] In this embodiment, the image to be tested input into the preset deep learning model is an image obtained after the image to be tested is preprocessed, and the preprocessing includes at least one of image denoising and image enhancement. That is, before step S120, the image to be tested needs to be preprocessed.

[0084] Among them, image denoising includes but is not limited to Gaussian filtering denoising, median filtering denoising, PM (Perona-Malik) (the name is a combination of two names, and the PM equation algorithm uses the principle of isotropic nonlinear diffusion to achieve image filtering) equation denoising and TV (Total Variation) method denoising.

[0085] The Gaussian filtering denoising method may be: calling the GaussianBlur interface in the OpenCV toolkit pre-installed in the electronic device, setting filtering parameters to achieve filtering denoising, and the filtering parameters may be flexibly determined according to actual conditions.

[0086] The median filtering denoising method may be: calling the medianBlur interface in the OpenCV toolkit pre-installed in the electronic device, setting filtering parameters to achieve filtering denoising, and the filtering parameters may be flexibly determined according to actual conditions.

[0087] The PM equation denoising method may be: image denoising is implemented according to the nonlinear anisotropic diffusion equation (PM diffusion equation).

[0088] The TV method denoising method can be: according to the anisotropic model of the gradient descent flow to smooth the image, the image is smoothed as much as possible inside the image (the difference between adjacent pixels is small), and the image edge (image contour) is not smoothed as much as possible.

[0089] The image enhancement method may be: calling the ImageEnhance interface in the PIL package pre-installed in the electronic device to automatically enhance the image. The PIL package is a commonly used image package in Python and can be used to enhance the image to enhance the texture features of the image. The PIL package is a software tool well known to those skilled in the art.

[0090] Through the above-mentioned image preprocessing, the electronic device can filter out some interference factors in the image and enhance the corresponding image features, which is conducive to improving the accuracy of image detection of the image to be tested by the preset deep learning model and machine learning model.

[0091] In step S120, the image to be tested input into the preset deep learning model is an image obtained after the image to be tested is preprocessed, and the preprocessing includes at least one of the above-mentioned image denoising and image enhancement.

[0092] As an optional implementation, step S120 may include:

[0093] Inputting the image to be tested into the residual network in the preset deep learning model to obtain an intermediate detection result of whether a thyroid nodule exists in the image to be tested;

[0094] When the intermediate detection result indicates that a thyroid nodule exists in the image to be tested, the image to be tested is input into the candidate area network in the deep learning model to obtain the first detection result, which includes the lesion area segmented from the image to be tested and indicating the presence of a thyroid nodule.

[0095] Understandably, the residual network can be used to detect whether there are thyroid nodules in the image to be tested. For the image to be tested with thyroid nodules, the candidate region network can locate the image area of ​​the thyroid nodule in the image to be tested, and use the image area of ​​the thyroid nodule as the lesion area and segment it from the image to be tested. The segmented lesion area can be used as the first detection result and input into the gcForest model, so that the gcForest model can subsequently perform targeted detection of the severity of the lesion area without having to detect the entire image to be tested, so as to reduce the amount of calculation and improve the accuracy of the detection.

[0096] In step S130, the machine learning model includes but is not limited to the gcForest model, which is used to detect the severity of the thyroid nodules in the lesion image area. The severity can be classified according to the actual situation. For example, the severity includes a classification that characterizes the thyroid nodules as benign or malignant. In this way, the electronic device can obtain whether there are thyroid nodules and the severity of the nodules by detecting the image to be tested, thereby providing a diagnostic basis for the staff, facilitating the auxiliary staff to complete the detection of thyroid nodules and improving the efficiency and accuracy of the detection.

[0097] Please refer to Figure 4 , the embodiment of the present application also provides a thyroid image processing device 200, which can be applied to the above-mentioned electronic device to execute each step in the method. The thyroid image processing device 200 includes at least one software function module that can be stored in the storage module in the form of software or firmware (Firmware) or solidified in the operating system (OS) of the electronic device. The processing module is used to execute the executable module stored in the storage module, such as the software function module and computer program included in the thyroid image processing device 200.

[0098] The thyroid image processing device 200 may include an acquisition unit 210, a first detection unit 220, and a second detection unit 230. The following operation steps may be performed:

[0099] An acquisition unit 210 is used to acquire an image to be tested, wherein the image to be tested includes an image obtained by detecting a thyroid gland part;

[0100] A first detection unit 220 is used to input the image to be tested into a preset deep learning model to obtain a first detection result of the preset deep learning model on the image to be tested;

[0101] The second detection unit 230 is used to input the lesion image area in the first detection result indicating the presence of thyroid nodules in the image to be tested into a tested machine learning model when the first detection result indicates the presence of thyroid nodules in the image to be tested, so as to obtain a second detection result of the machine learning model on the lesion image area, wherein the second detection result indicates the severity of the thyroid nodules in the lesion image area.

[0102] Optionally, the thyroid image processing device 200 may further include a model training unit and a model testing unit. Before executing step S110, the acquisition unit 210 may also be used to acquire a training data set and a test data set, wherein each image in the training data set and the test data set corresponds to at least one label, and the label is used to mark the image area where thyroid nodules exist, or to mark the image area where thyroid nodules do not exist. The model training unit is used to input the training data set into the residual network to obtain the training image with thyroid nodules in the training data set, and to input the training image area with thyroid nodules into the candidate area network to obtain the candidate area, wherein the candidate area is the lesion area representing the presence of thyroid nodules segmented from the training image, and to train the gcForest model using the candidate area to obtain the trained gcForest model. The model testing unit is used to test the trained gcForest model according to the test data set to obtain the tested machine learning model.

[0103] Optionally, the model training unit can also be used to: extract features from the candidate area through sliding windows of different specified window sizes to obtain feature maps corresponding to the different specified window sizes; input the feature map into the random forest in the gcForest model, train the gcForest model, and obtain the trained gcForest model.

[0104] Optionally, the acquisition unit 210 may also be used to: select an interface corresponding to the image type to acquire the image to be tested according to the image type of the image to be tested, and the image type includes at least one of PNG, JPG, BMP, and DICOM.

[0105] Optionally, the first detection unit 220 can also be used to: input the image to be tested into the residual network in the preset deep learning model to obtain an intermediate detection result of whether a thyroid nodule exists in the image to be tested; when the intermediate detection result indicates that a thyroid nodule exists in the image to be tested, input the image to be tested into the candidate area network in the deep learning model to obtain the first detection result, and the first detection result includes the lesion area segmented from the image to be tested and indicating the presence of a thyroid nodule.

[0106] In this embodiment, the processing module may be an integrated circuit chip having the ability to process signals. The above-mentioned processing module may be a general-purpose processor. For example, the processor may be a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and may implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0107] The storage module may be, but is not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc. In this embodiment, the storage module may be used to store images to be tested, preset deep learning models, machine learning models, etc. Of course, the storage module may also be used to store programs, and the processing module executes the program after receiving the execution instruction.

[0108] The communication module is used to establish a communication connection between the electronic device and other devices (such as a server) through a network, and to send and receive data through the network.

[0109] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process of each step in the aforementioned method, and will not be elaborated here.

[0110] The embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the computer executes the thyroid image processing method described in the above embodiment.

[0111] Through the description of the above implementation methods, technical personnel in this field can clearly understand that the present application can be implemented by hardware, and can also be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0112] In summary, in this solution, a preset deep learning model is used to detect whether there are thyroid nodules in the image to be tested, and when there are thyroid nodules, the lesion area in the image to be tested is input into the machine learning model. In this way, the machine learning model does not need to detect images without thyroid nodules and areas that are not lesion areas, thereby reducing the amount of calculation and improving the calculation efficiency. In addition, the combination of the preset deep learning model and the machine learning model is conducive to improving the accuracy of thyroid nodule detection.

[0113] In the embodiments provided in the present application, it should be understood that the disclosed device, system and method can also be implemented in other ways. The above-described device, system and method embodiments are merely schematic, for example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0114] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A thyroid image processing method, characterized in that: The method comprises: Acquire an image to be tested, wherein the image to be tested includes an image obtained by detecting a thyroid gland portion; Inputting the image to be tested into a preset deep learning model to obtain a first detection result of the preset deep learning model on the image to be tested; When the first detection result indicates that a thyroid nodule exists in the image to be tested, inputting a lesion image region in the first detection result indicating that a thyroid nodule exists in the image to be tested into a tested machine learning model to obtain a second detection result of the machine learning model on the lesion image region, wherein the second detection result indicates the severity of the thyroid nodule in the lesion image region; The preset deep learning model includes a residual network and a candidate region network. Before acquiring the image to be tested, the method further includes: acquiring a training data set and a test data set, each image in the training data set and the test data set corresponds to at least one label, and the label is used to mark the image area where thyroid nodules exist, or mark the image area where thyroid nodules do not exist; inputting the training data set into the residual network to obtain the training image with thyroid nodules in the training data set; inputting the training image area with thyroid nodules into the candidate region network to obtain the candidate area, and the candidate area is the lesion area representing the presence of thyroid nodules segmented from the training image; training the gcForest model using the candidate area to obtain the trained gcForest model; testing the trained gcForest model according to the test data set to obtain the tested machine learning model; The residual network is used to extract image features of nodules or micronodules in the image, so as to process multi-scale changes in thyroid nodule detection through a feature pyramid network and extract thyroid nodule features; The training process of the gcForest model includes: concatenating the extracted micronodule imaging features with the output value of the previous layer of random forest and the input vector as the input of the next layer, and repeating the process until the model converges; the image input to the machine learning model by the preset deep learning model includes a diffusion weighted map and an apparent diffusion coefficient map, and the micronodule imaging features are extracted from the diffusion weighted map and the apparent diffusion coefficient map.

2. The method according to claim 1, characterized in that The gcForest model is trained using the candidate region to obtain a trained gcForest model, including: Extracting features of the candidate area through sliding windows of different specified window sizes to obtain feature maps corresponding to the different specified window sizes; The feature map is input into the random forest in the gcForest model, the gcForest model is trained, and the trained gcForest model is obtained.

3. The method according to claim 1, characterized in that Get the image to be tested, including: According to the image type of the image to be tested, an interface corresponding to the image type is selected to acquire the image to be tested, and the image type includes at least one of PNG, JPG, BMP, and DICOM.

4. The method according to claim 1, characterized in that: Inputting the image to be tested into a preset deep learning model to obtain a first detection result of the preset deep learning model on the image to be tested, including: Inputting the image to be tested into the residual network in the preset deep learning model to obtain an intermediate detection result of whether a thyroid nodule exists in the image to be tested; When the intermediate detection result indicates that a thyroid nodule exists in the image to be tested, the image to be tested is input into the candidate area network in the deep learning model to obtain the first detection result, which includes the lesion area segmented from the image to be tested and indicating the presence of a thyroid nodule.

5. The method according to claim 1, characterized in that The image to be tested input into the preset deep learning model is an image obtained after the image to be tested is preprocessed, and the preprocessing includes at least one of image denoising and image enhancement.

6. The method according to claim 1, characterized in that The image to be measured includes an image obtained by magnetic resonance imaging of the thyroid gland.

7. A thyroid image processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring an image to be tested, wherein the image to be tested includes an image obtained by detecting a thyroid gland part; A first detection unit, configured to input the image to be tested into a preset deep learning model to obtain a first detection result of the preset deep learning model on the image to be tested; a second detection unit, configured to, when the first detection result indicates that a thyroid nodule exists in the image to be tested, input a lesion image region in the first detection result indicating that a thyroid nodule exists in the image to be tested into a tested machine learning model, to obtain a second detection result of the machine learning model on the lesion image region, wherein the second detection result indicates a severity of the thyroid nodule in the lesion image region; The preset deep learning model includes a residual network and a candidate region network, and an acquisition unit is used to acquire a training data set and a test data set, each image in the training data set and the test data set corresponds to at least one label, and the label is used to mark the image area where thyroid nodules exist, or mark the image area where thyroid nodules do not exist; The model training unit is used to input the training data set into the residual network to obtain a training image with thyroid nodules in the training data set, and input the training image area with thyroid nodules into the candidate area network to obtain a candidate area, wherein the candidate area is a lesion image area representing the presence of thyroid nodules segmented from the training image, and train a gcForest model using the candidate area to obtain a trained gcForest model; A model testing unit, used to test the trained gcForest model according to the test data set to obtain the tested machine learning model; The residual network is used to extract image features of nodules or micronodules in the image, so as to process multi-scale changes in thyroid nodule detection through a feature pyramid network and extract thyroid nodule features; The model testing unit is specifically used to: concatenate the extracted micronodule imaging features with the previous layer random forest output value and input vector as the next layer input, and repeat the process until the model converges; the image input to the machine learning model by the preset deep learning model includes a diffusion weighted map and an apparent diffusion coefficient map, and the micronodule imaging features are extracted from the diffusion weighted map and the apparent diffusion coefficient map.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory coupled to each other, wherein the memory stores a computer program. When the computer program is executed by the processor, the electronic device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 6.

Images