A multi-source papillary thyroid carcinoma classification system based on federated learning

Abstract

The application relates to the technical field of thyroid papillary carcinoma classification, and discloses a multi-source thyroid papillary carcinoma classification system based on federal learning, which comprises a central server and a plurality of local devices connected with the central server; the local device is used for acquiring a thyroid ultrasound image and classifying the thyroid ultrasound image by adopting a classification model; the central server and the plurality of local devices are used for updating the classification model by federal learning based on a training set; wherein the global model in the federal learning process is updated based on the data points in the training set adopted by each local device and the local models uploaded by each local device. The accuracy of thyroid papillary carcinoma classification is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of thyroid papillary carcinoma classification, and in particular to a multi-source thyroid papillary carcinoma classification system based on federated learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Papillary thyroid carcinoma (PTC) is the most common pathological type of thyroid cancer, accounting for about 85.00%, and PTC is divided into 15 subtypes according to tumor cell shape, proportion, structure and cell nucleus characteristics, among which tall cell variant (TCV), solid variant (SV), hobnail variant (HV), diffuse sclerosing variant (DSV) and columnar cell variant are often considered as aggressive variants of PTC (AVPTC). The tall cell variant is the most common subtype of AVPTC, accounting for more than 70% of the total, followed by the diffuse sclerosing variant, the solid / small beam type, the hobnail type and the columnar cell type. It is of great significance to distinguish high-risk subtypes from the classical variant for treatment.

[0004] The AI-based disease diagnosis method essentially relies on large training data sets. Since the thyroid ultrasound imaging data stored in the ultrasound machine is scattered by itself, and the sample size of the invasive subtype of thyroid cancer is even smaller, the data volume of a single medical institution is often insufficient to train a reliable classification system. Moreover, the exchange of medical research data between different centers is often subject to data protection, privacy protection and data sovereignty regulations. Technical problems also affect data usage, such as when a large amount of data needs to be transmitted digitally, the data transmission line quickly reaches its performance limit. In view of these circumstances, many medical researches are limited to the local area, and it is difficult to use data available elsewhere.

[0005] Within a hospital, there are many different sources of data, from different machine manufacturers and different machine models of the same manufacturer, as well as different workstations, and there are also differences between the original data and the post-processing data of the machines. SUMMARY

[0006] In order to solve the problems in the prior art, the application provides a multi-source thyroid papillary carcinoma classification system based on federated learning, which forms an update iteration process of federated learning between local devices and a central server, trains a robust classification model, and improves the accuracy of thyroid papillary carcinoma classification.

[0007] In a first aspect, the application provides a multi-source thyroid papillary carcinoma classification system based on federated learning.

[0008] A multi-source thyroid papillary carcinoma classification system based on federated learning comprises a central server and a plurality of local devices connected to the central server.

[0009] The local device is configured to acquire thyroid ultrasound images and classify the thyroid ultrasound images using a classification model.

[0010] The central server and the plurality of local devices update the classification model based on a training set through federated learning.

[0011] In the federated learning process, the global model is updated based on the number of data points in the training set used by each local device and the local models uploaded by each local device.

[0012] Further, each local device is connected to a plurality of workstations.

[0013] The workstation is configured to upload thyroid ultrasound images acquired by different ultrasound devices to the local device.

[0014] Further, the workstation uses different image acquisition cards to acquire thyroid ultrasound images from different ultrasound devices.

[0015] Further, the workstation is also configured to acquire thyroid ultrasound images input by a user, add labels to the thyroid ultrasound images acquired by the ultrasound devices to form a training set, and upload the training set to the local device.

[0016] Further, the specific steps of the federated learning are as follows:

[0017] Each local device optimizes the global model sent by the central server using a local training set to obtain a local model and uploads the local model to the central server.

[0018] The central server calculates a global model based on the local models uploaded by all local devices and sends the global model to each local device as a classification model for each local device.

[0019] Further, the specific method for optimizing the global model using the local data set is as follows:

[0020] Based on the weight of the global model, the data in the local training set is fused based on multi-scale features, and the global model is optimized by random gradient descent and minimizing the loss function.

[0021] Further, the multi-scale feature fusion is realized by a plurality of convolution layers connected in sequence.

[0022] Further, the parameters of the global model are:

[0023]

[0024] Wherein, K is the number of local devices, is the parameter of the local model sent by the local device k at the tth global model update, n k is the number of data points in the local training set used by the local device k to optimize the global model, and n is the total number of data points participating in collaborative training.

[0025] In a second aspect, the present application also provides an electronic device, comprising:

[0026] a memory for non-transiently storing computer readable instructions; and

[0027] a processor for running the computer readable instructions,

[0028] wherein the computer readable instructions are run by the processor to perform the following steps:

[0029] obtaining a thyroid ultrasound image;

[0030] classifying the thyroid ultrasound image using a classification model;

[0031] wherein the classification model is updated through federated learning between the central server and the local devices based on the training set; the global model in the federated learning process is updated based on the number of data points in the training set used by each local device and the local model uploaded by each local device.

[0032] In a third aspect, the present application also provides a storage medium non-transiently storing computer readable instructions, wherein when the non-transient computer readable instructions are executed by a computer, the following steps are performed:

[0033] obtaining a thyroid ultrasound image;

[0034] classifying the thyroid ultrasound image using a classification model;

[0035] The classification model is updated based on the training set and the central server through federated learning, and the global model in the federated learning process is updated based on the number of data points in the training set of each local device and the local model uploaded by each local device.

[0036] Compared with the prior art, the beneficial effects of the present application are:

[0037] The multi-source thyroid papillary carcinoma classification system based on federated learning is repeatedly iterated between the local device and the central server, and after multiple iterations, the optimal model is finally reached in the central server and then distributed to each hospital center, so that each hospital center has the latest model; and the classification model with strong robustness is trained by fusing the common features of ultrasound images based on multiple types of ultrasound machines and different workstations, thereby improving the accuracy of thyroid papillary carcinoma classification.

[0038] The advantages of the additional aspects of the present application will be partially given in the following description, or understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0039] The drawings accompanying the specification of the present application form a part thereof and serve to provide further understanding of the present application, together with the exemplary embodiments of the present application and their descriptions, which serve to explain the present application, and do not constitute improper limitations on the present application.

[0040] Figure 1 The overall structure diagram of the multi-source thyroid papillary carcinoma classification system based on federated learning of the first embodiment;

[0041] Figure 2 The detailed structure diagram of the multi-source thyroid papillary carcinoma classification system based on federated learning of the first embodiment. DETAILED DESCRIPTION

[0042] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0043] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0044] All data acquisition of the present embodiment is based on compliance with laws and regulations and user consent, and legal application of data.

[0045] Embodiment one

[0046] The present embodiment provides a multi-source thyroid papillary carcinoma classification system based on federated learning;

[0047] As shown in Figure 1 and Figure 2 A multi-source papillary thyroid carcinoma classification system based on federated learning, comprising: several different types of ultrasound devices (ultrasound machines) set in different (no less than three) hospitals, several workstations, several local devices and a central server.

[0048] Ultrasound device, for collecting thyroid ultrasound images.

[0049] Specifically, the brands of the ultrasound devices include, but are not limited to, GE, Philips, Siemens, Mindray, VINNO, SonoScape and SIUI, etc.

[0050] The ultrasound device includes, but is not limited to, a palm-sized ultrasound device, which is different from the product form of the traditional ultrasound device with a host + probe, and the host is reduced to only a small circuit board built into the probe, becoming only a "probe" which is equivalent to a B-ultrasound, and only a mobile phone, a tablet computer installed with an ultrasound APP software are needed for display, and the image is transmitted to the mobile phone / tablet by the built-in wifi of the probe.

[0051] Each ultrasound device is connected to a workstation, and the ultrasound images from different ultrasound devices are stored in the respective workstations.

[0052] The workstation includes a data acquisition module and a label making module.

[0053] The data acquisition module of the workstation is used to connect the ultrasound device, acquire the ultrasound images collected by the ultrasound device and store them.

[0054] The workstation refers to the use of different image acquisition cards to collect thyroid ultrasound images from different ultrasound machines into the workstation in order to improve the portability of reading the images stored in different ultrasound machines and the more image storage space, so that the ultrasound images can be more conveniently stored and read.

[0055] The data acquisition module of the workstation is also used to acquire the ultrasound images input by the user.

[0056] The label making module of the workstation is used to make labels for the ultrasound images according to the pathological standard, form a training set and upload it to the local device. That is, the workstation is used to acquire the thyroid ultrasound images collected by the ultrasound device, and is also used to acquire the thyroid ultrasound images input by the user, together with the thyroid ultrasound images collected by the ultrasound device, to form a training set after adding labels, and upload it to the local device.

[0057] Specifically, based on the pathological image, the workstation-stored ultrasound image is divided into four categories: benign nodule, papillary thyroid carcinoma classic subtype, high cell subtype and other subtypes.

[0058] A hospital sets up a local device for connecting all ultrasound devices and workstations in the hospital where the connector is located.

[0059] The local device (hospital center) includes a model training module and a result output module.

[0060] The model training module of the local device is used to obtain the training set stored in different workstations, combine it into a local training set, and train the classification model using the local training set.

[0061] Taking one of the hospital centers as an example, the ultrasound images in the local training set are sent into the first convolutional layer, then the features extracted by the two convolutional layers are fused, and then the fused results and the original first convolutional features are sent into the next convolutional layer; the next convolutional layer extracts features from the first layer again, and then the features extracted by the previous layer and the features extracted by the current layer are fused again; the features extracted by the high-level convolutional layer and the features fused by each convolutional layer are obtained in turn, and are sent into the central server together, and the central server integrates the classification model, trains the classification result, and updates the classification model. The remaining hospital centers take the same classification model training steps. Finally, the central server updates the classification model and sends it to each hospital center.

[0062] Based on the training set, the central server and the local devices update the classification model through federated learning. Each local device optimizes the global model sent by the central server using the local data set (based on the weight of the global model, the data in the local data set is fused based on multi-scale features, and the global model is optimized through stochastic gradient descent and minimizing the loss function) to obtain a local model and upload it to the central server; the central server calculates the global model according to the local models uploaded by all local devices and sends it to each local device as the classification model of each local device.

[0063] Specifically:

[0064] The initial central server maintains a global central model f with initial weights w t-1 , and each local device data point set s t Share the central model and the initial weight, randomly select a given score value C, where C∈[0,1]; all local device data subsets constitute s t ;

[0065] Each local device k∈s t , receives the initial weight wt-1 First, multi-scale based feature fusion is performed, then local target minimization based, using a mini-batch stochastic gradient descent algorithm with a local learning rate minimum, performs training steps on local private data, optimizing the model by minimizing the cross-entropy loss of classification;

[0066] If the local training is completed, each hospital center sends its updated model update back to the central server,

[0067] Finally, the central server receives the updated model from all the hospital centers and calculates an average model w t According to the following formula, update each parameter of the global model f;

[0068]

[0069] Where K is the number of local devices, w t is the parameter at the tth update, is the parameter of the local model sent by the hospital center k at the tth global model update, n k is the number of data points in the local training set used by the local device k to optimize the global model at the tth time, and n is the total number of data points participating in collaborative training.

[0070] Where the first convolutional layer is c1, the second convolutional layer is c2, and the third convolutional layer is c3; different ultrasound machines are denoted as m, m ∈ [1, n], where n is the total number of ultrasound machines; different workstations are denoted as s, m ∈ [1, i], where i is the total number of workstations; the multi-scale based feature fusion step is as follows:

[0071] The fused features of the first convolutional layer are

[0072]

[0073] Where, refers to the features of the ultrasound images collected by the mth ultrasound machine learned by the convolutional network based on c1, refers to the features of the ultrasound images collected by the st workstation learned by the convolutional network based on c1. Each ultrasound device and workstation in each training batch only inputs one ultrasound image.

[0074] The fused features of the second convolutional layer are

[0075]

[0076] Similarly, the fused features of the third convolutional layer are

[0077]

[0078] At this point, the data feature fusion step of the local equipment of a hospital is completed.

[0079] The result output module of the workstation is configured to classify the thyroid ultrasound image using the classification model.

[0080] The update iteration process of federated learning is formed between the local equipment and the central server, and the process is repeated continuously. After multiple iterations, the optimal model is finally reached in the central server, and then it is distributed to each hospital center. In this way, each hospital center has the latest model, and subsequent new data can also be updated to continue to optimize the model.

[0081] The embodiment trains an AI technology based on ultrasound image features to identify the histopathological type of thyroid cancer tissue by fusing common features of ultrasound images based on multiple types of ultrasound machines and different workstation sources, which has strong robustness. The AI technology can be used to identify thyroid cancer (papillary carcinoma) and benign nodules, and further distinguish high cell subtypes and classical and other AVPTC subtypes in papillary carcinoma, providing evidence support for precise treatment of thyroid papillary carcinoma, especially early detection, early diagnosis and early treatment of high-risk subtypes of thyroid papillary carcinoma, and further providing ideas for "three early" of medullary carcinoma and undifferentiated carcinoma and other invasive subtypes of thyroid cancer.

[0082] Embodiment two

[0083] The embodiment also provides an electronic device, including one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the following steps:

[0084] Obtaining a thyroid ultrasound image;

[0085] Classifying the thyroid ultrasound image using a classification model;

[0086] The classification model is updated between the central server through federated learning based on the training set. The global model in the federated learning process is updated based on the number of data points in the training set of each local equipment and the local model uploaded by each local equipment.

[0087] It should be understood that in the embodiment, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0088] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0089] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software.

[0090] The method in the embodiment one can be directly embodied as hardware processor execution completion, or executed by hardware and software modules in the processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0091] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software mode depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0092] Embodiment three

[0093] The embodiment also provides a computer readable storage medium for storing computer instructions, which are executed by a processor to perform the following steps:

[0094] Obtaining a thyroid ultrasound image;

[0095] Classifying the thyroid ultrasound image by using a classification model;

[0096] The classification model is updated based on a training set and a central server through federated learning. A global model in the federated learning process is updated based on the number of data points in the training set of each local device and the local model uploaded by each local device.

[0097] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.

Claims

1. A multi-source papillary thyroid carcinoma classification system based on federated learning, characterized in that, It includes a central server and several local devices connected to the central server; The local device is used to acquire thyroid ultrasound images and classify the thyroid ultrasound images using a classification model. The central server and several local devices update the classification model based on the training set through federated learning; each local device connects to several workstations. The workstation is used to upload thyroid ultrasound images acquired by different ultrasound devices to the local device; The workstation is also used to acquire thyroid ultrasound images entered by the user, and together with the thyroid ultrasound images acquired by the ultrasound device, add tags to form a training set, and upload it to the local device. In the federated learning process, the global model is updated based on the number of data points in the training set used by each local device and the local model uploaded by each local device. The specific steps of the federated learning are as follows: Each local device sends a global model to the central server, optimizes the global model using its local training set, obtains a local model, and then uploads it to the central server. The specific method for optimizing the global model using a local dataset is as follows: Based on the weights of the global model, after performing multi-scale feature fusion on the data in the local training set, the global model is optimized by stochastic gradient descent and minimizing the loss function. The multi-scale feature fusion steps are as follows: The features after fusion in the first convolutional layer are in, This refers to convolutional networks based on The first thing I learned m Features of ultrasound images acquired by an ultrasound machine This refers to convolutional networks based on The first thing I learned s Features of ultrasound images acquired by each workstation; only one ultrasound image is input for each ultrasound device and workstation in each training batch; n This represents the total number of ultrasound machines. i The total number of workstations; different ultrasound machines are represented as m; different workstations are represented as s. The fused result and the original first-layer convolutional features are fed into the next convolutional layer. The next convolutional layer extracts features from the features received from the first layer again, and then merges the features fused from the previous layer with the features extracted from this layer again. This process is repeated to obtain the features extracted by higher-level convolutional layers and the features fused from each convolutional layer.

2. The multi-source papillary thyroid carcinoma classification system based on federated learning as described in claim 1, characterized in that, The workstation uses different image acquisition cards to acquire thyroid ultrasound images from different ultrasound devices.

3. The multi-source papillary thyroid carcinoma classification system based on federated learning as described in claim 1, characterized in that, The central server calculates a global model based on the local models uploaded by all local devices and sends it to each local device as the classification model for each local device.

4. The multi-source papillary thyroid carcinoma classification system based on federated learning as described in claim 1, characterized in that, The multi-scale feature fusion is achieved through several convolutional layers connected in sequence.

5. A multi-source papillary thyroid carcinoma classification system based on federated learning as described in claim 1, characterized in that, The parameters of the global model are as follows: in, K The number of local devices. It is a local device k In the t The parameters of the local model sent during the next global model update. It is a local device k Optimize the number of data points used in the local training set for the global model. n It represents the total number of data points participating in collaborative training.

6. An electronic device employing a federated learning-based multi-source papillary thyroid carcinoma classification system as described in any one of claims 1-5, characterized in that, include: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, the following steps are performed: Obtain thyroid ultrasound images; A classification model was used to classify thyroid ultrasound images; The classification model is updated through federated learning with the central server based on the training set. The global model in the federated learning process is updated based on the number of data points in the training set used by each local device and the local model uploaded by each local device.

7. A storage medium employing a federated learning-based multi-source papillary thyroid carcinoma classification system as described in any one of claims 1-5, characterized in that it is non-transitory. The non-transitory computer-readable instructions are stored in a location whereby, when executed by a computer, the following steps are performed: Obtain thyroid ultrasound images; A classification model was used to classify thyroid ultrasound images; The classification model is updated through federated learning with the central server based on the training set. The global model in the federated learning process is updated based on the number of data points in the training set used by each local device and the local model uploaded by each local device.

Images