Artificial Intelligence-Based Auxiliary Diagnosis Method and Related Equipment for Thyroid Nodules

Abstract

The present application proposes an artificial intelligence-based auxiliary diagnosis method, device, electronic device and storage medium for thyroid nodules. The artificial intelligence-based auxiliary diagnosis method for thyroid nodules includes: collecting thyroid ultrasound images with labeled data as a training set; building a first auxiliary diagnosis model based on a control vector set, the first auxiliary diagnosis model including a convolutional structure, a localization structure and a classification structure; training the first auxiliary diagnosis model to update the model parameters to obtain a second auxiliary diagnosis model and a standard vector set, the model parameters including the control vector set; collecting the ultrasound image to be diagnosed and inputting it into the second auxiliary diagnosis model, taking the output of the classification structure as the auxiliary diagnosis result, and constructing an auxiliary feature map based on the output of the convolutional structure and the standard vector set; and displaying the auxiliary feature map and the auxiliary diagnosis result on the terminal screen to assist the doctor in the diagnosis process. The present application can improve the interpretability and accuracy of the auxiliary diagnosis model, thereby improving the doctor's diagnosis efficiency.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular, to an artificial intelligence-based auxiliary diagnosis method, device, electronic device and storage medium for thyroid nodules. Background Art

[0002] In recent years, ultrasound examination has become an increasingly important means of tumor screening and diagnosis in clinical practice. Its non-invasive, safe, real-time and low-cost characteristics have made it widely used in many regions of China and the world. The thyroid gland is most suitable for ultrasound examination, and ultrasound provides important imaging evidence for clinical practice in the preoperative diagnosis and postoperative follow-up of thyroid lesions.

[0003] The Thyroid Imaging Reporting and Data System (TI-RADS) is a unified terminology and definition of thyroid imaging findings by the American College of Radiology. It finally gives a risk assessment classification of 6 categories according to the suspicious degree of malignancy of the lesion, and clarifies the corresponding clinical treatment suggestions.

[0004] At present, computer-aided diagnosis methods can provide objective diagnosis suggestions for doctors and reduce the work intensity of doctors. However, when the existing auxiliary diagnosis methods obtain diagnosis results based on medical images, the basis for making the diagnosis results is often some mixed information such as the type of imaging equipment, rather than medical-related information. At the same time, it does not explain which features in the medical images are beneficial to obtaining accurate diagnosis results, and the interpretability and accuracy of the auxiliary diagnosis process are relatively low, and it cannot well assist doctors in making accurate diagnosis results. Summary of the Invention

[0005] In view of the above, it is necessary to propose an artificial intelligence-based auxiliary diagnosis method and related devices for thyroid nodules to solve the technical problem of how to improve the interpretability and accuracy of the auxiliary diagnosis process. Among them, the related devices include an artificial intelligence-based auxiliary diagnosis device for thyroid nodules, an electronic device and a storage medium.

[0006] The present application provides an artificial intelligence-based auxiliary diagnosis method for thyroid nodules, and the method includes:

[0007] Collect thyroid ultrasound images with labeled data as a training set, where the labeled data includes the benign and malignant category labels and risk level labels of thyroid nodules in the thyroid ultrasound images;

[0008] Build a first auxiliary diagnosis model based on a control vector set, where the first auxiliary diagnosis model includes a convolutional structure, a positioning structure and a classification structure;

[0009] Train the first auxiliary diagnosis model based on the training set and a preset loss function to update the model parameters, obtaining a second auxiliary diagnosis model and a set of standard vectors. The model parameters include the parameters of the convolutional structure, the localization structure, and the classification structure. The parameters of the localization structure are the set of reference vectors.

[0010] Collect the ultrasound image to be diagnosed and input it into the second auxiliary diagnosis model. Use the output of the convolutional structure in the second auxiliary diagnosis model as the diagnostic convolutional feature map, and use the output of the classification structure in the second auxiliary diagnosis model as the auxiliary diagnosis result.

[0011] Construct an auxiliary feature map based on the diagnostic convolutional feature map and the set of standard vectors.

[0012] Display the auxiliary feature map and the auxiliary diagnosis result on the terminal screen to assist the doctor in the diagnosis process, obtaining the diagnosis result of the ultrasound image to be diagnosed.

[0013] In some embodiments, when building the first auxiliary diagnosis model based on the set of reference vectors, the first auxiliary diagnosis model includes a convolutional structure, a localization structure, and a classification structure, and includes:

[0014] Construct a set of reference vectors. The set of reference vectors includes a reference vector subset for each preset diagnosis category. The reference vector subset includes a preset number of reference feature vectors. The preset diagnosis categories include benign low risk, benign high risk, malignant low risk, and malignant high risk.

[0015] Build the first auxiliary diagnosis model based on the set of reference vectors. The first auxiliary diagnosis model is formed by connecting the convolutional structure, the localization structure, and the classification structure in series. The input of the first auxiliary diagnosis model is a thyroid ultrasound image, and the output is the probability vector of the benign or malignant nature and the probability vector of the risk level of the thyroid ultrasound image.

[0016] The input of the convolutional structure is a thyroid ultrasound image, and the output is the convolutional feature map of the thyroid ultrasound image. The convolutional feature map includes multiple convolutional feature vectors.

[0017] The localization structure is used to calculate the similarity between the convolutional feature map and each reference feature vector in the set of reference vectors, and arrange all the similarities in a fixed order along the column direction to obtain a similarity vector.

[0018] The input of the classification structure is the similarity vector, and the output is the probability vector of the benign or malignant nature and the probability vector of the risk level of the thyroid ultrasound image.

[0019] In some embodiments, the similarity satisfies the relational expression:

[0020]

[0021] Among them, z is the convolutional feature map output by the convolutional structure, is any convolutional feature vector in the convolutional feature map, p i is the control feature vector i in the control vector set, and the convolutional feature vector and the control feature vector have the same size; ∈ is the adjustment coefficient, ∑ maxk(x) X(x) represents the sum of the first k maximum values among all parameters X(x), g i (z) is the similarity between the convolutional feature map z and the control feature vector i.

[0022] In some embodiments, the first auxiliary diagnosis model is trained based on the training set and a preset loss function to update the model parameters, obtaining a second auxiliary diagnosis model and a standard vector set. The model parameters include the parameters of the convolutional structure, the localization structure, and the classification structure. The parameters of the localization structure are the control vector set, including:

[0023] Fix the parameters in the classification structure, and perform preliminary training on the convolutional structure and the localization structure in the first auxiliary diagnosis model based on the training set and the preset loss function to update the parameters of the convolutional structure and all control feature vectors in the control vector set of the localization structure;

[0024] During the preliminary training process, continuously select training batches from the training set to calculate the value of the preset loss function. When the value of the preset loss function no longer changes, stop the preliminary training, and use all control feature vectors in the localization structure as standard feature vectors, and store all standard feature vectors to obtain a standard vector set;

[0025] Fix the parameters of the convolutional structure and all control feature vectors in the localization structure, and perform secondary training on the classification structure in the first auxiliary diagnosis model based on the training set and the cross-entropy loss function to update the parameters of the classification structure;

[0026] During the secondary training process, continuously select training batches from the training set to calculate the value of the cross-entropy loss function. When the value of the cross-entropy loss function no longer changes, stop the secondary training to obtain the second auxiliary diagnosis model.

[0027] In some embodiments, the preset loss function satisfies the relational expression:

[0028]

[0029] Among them, M represents the data volume of each training batch in the preliminary training, and respectively represent the benign and malignant category labels and risk level labels of the i-th thyroid ultrasound image in a training batch, and respectively represent the benign and malignant probability vector and the risk level probability vector of the i-th thyroid ultrasound image output by the first auxiliary diagnosis model in a training batch. represents and The cross-entropy loss function, Clst is the mutual exclusion loss function, λ is the weight factor, and Loss is the value of the preset loss function.

[0030] In some embodiments, the mutual exclusion loss function satisfies the relational expression:

[0031]

[0032] where M represents the data volume of each training batch in the preliminary training. represents the preset diagnosis category of the i-th thyroid ultrasound image in a training batch. is the mutual exclusion type of the preset diagnosis category of the i-th thyroid ultrasound image obtained according to the preset mutual exclusion pair; p j is any one of the control feature vectors in the control vector subset of the preset diagnosis category of the i-th thyroid ultrasound image. is any one of the control feature vectors in the control vector subset of the mutual exclusion type of the preset diagnosis category of the i-th thyroid ultrasound image. is any one of the convolution feature vectors in the convolution feature map of the i-th thyroid ultrasound image, and ∑ mink(x) X(x) represents the sum of the first k minimum values among all the parameters X(x); Clst is the value of the mutual exclusion loss function.

[0033] In some embodiments, constructing the auxiliary feature map based on the diagnostic convolution feature map and the standard vector set includes:

[0034] Regarding each 1×1×C convolution feature vector in the diagnostic convolution feature map as a diagnostic convolution feature vector, where C is the number of image channels of the diagnostic convolution feature map;

[0035] Calculating the cosine similarity between each diagnostic convolution feature vector and each standard feature vector in the standard vector set;

[0036] Selecting the standard feature vector corresponding to the maximum value among all the cosine similarities of the same diagnostic convolution feature vector as the matching vector of the diagnostic convolution feature vector, and the matching vector corresponds to the diagnostic convolution feature vector one by one;

[0037] Replacing each diagnostic convolution feature vector in the diagnostic convolution feature map with the matching vector to construct the auxiliary feature map.

[0038] The embodiments of the present application further provide an artificial intelligence-based auxiliary diagnosis device for thyroid nodules, and the device includes:

[0039] An acquisition unit, configured to acquire thyroid ultrasound images with labeled data as a training set, where the labeled data includes benign and malignant category labels and risk level labels of thyroid nodules in the thyroid ultrasound images;

[0040] A construction unit, configured to construct a first auxiliary diagnosis model based on a control vector set, where the first auxiliary diagnosis model includes a convolutional structure, a localization structure, and a classification structure;

[0041] A training unit, configured to train the first auxiliary diagnosis model based on the training set and a preset loss function to update model parameters, and obtain a second auxiliary diagnosis model and a standard vector set, where the model parameters include the parameters of the convolutional structure, the localization structure, and the classification structure, and the parameters of the localization structure are the control vector set;

[0042] An input unit, configured to acquire an ultrasound image to be diagnosed and input it into the second auxiliary diagnosis model, use the output of the convolutional structure in the second auxiliary diagnosis model as a diagnostic convolutional feature map, and use the output of the classification structure in the second auxiliary diagnosis model as an auxiliary diagnosis result;

[0043] A construction unit, configured to construct an auxiliary feature map based on the diagnostic convolutional feature map and the standard vector set;

[0044] An auxiliary unit, configured to display the auxiliary feature map and the auxiliary diagnosis result on a terminal screen to assist the doctor's diagnosis process, and obtain the diagnosis result of the ultrasound image to be diagnosed.

[0045] The embodiments of the present application further provide an electronic device, and the electronic device includes:

[0046] A memory, storing at least one instruction;

[0047] A processor, configured to execute the instruction stored in the memory to implement the artificial intelligence-based auxiliary diagnosis method for thyroid nodules.

[0048] The embodiments of the present application further provide a computer-readable storage medium, where at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in an electronic device to implement the artificial intelligence-based auxiliary diagnosis method for thyroid nodules.

[0049] In summary, the present application introduces medical-related information by means of standard feature vectors, adds a localization structure to the auxiliary diagnosis model to calculate the similarity between medical image features and medical-related information, and determines the benignity, malignancy, and risk level of nodules based on the similarity. At the same time, the standard feature vectors are mapped onto the convolutional feature maps of medical images for visualization, which can reflect the severity of different positions on medical images, improve the interpretability and accuracy of the auxiliary diagnosis model, and thus improve the doctor's diagnosis efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flowchart of a preferred embodiment of the artificial intelligence-based thyroid nodule auxiliary diagnosis method involved in the present application.

[0051] Figure 2 is a schematic structural diagram of a first auxiliary diagnosis model involved in the present application.

[0052] Figure 3 is a functional module diagram of a preferred embodiment of the artificial intelligence-based thyroid nodule auxiliary diagnosis device involved in the present application.

[0053] Figure 4 is a schematic structural diagram of an electronic device of a preferred embodiment of the artificial intelligence-based thyroid nodule auxiliary diagnosis method involved in the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] In order to more clearly understand the purpose, features, and advantages of the present application, the present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments may be combined with each other. Many specific details are set forth in the following description in order to fully understand the present application. The described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments.

[0055] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present application, "a plurality" means two or more, unless otherwise specifically defined.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs. The terms used in the description of the present application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" as used herein includes any and all combinations of one or more of the related listed items.

[0057] The embodiments of the present application provide an artificial intelligence-based auxiliary diagnosis method for thyroid nodules, which can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0058] The electronic device can be any electronic product that can perform human-computer interaction with customers. For example, a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an Internet Protocol Television (IPTV), a smart wearable device, etc.

[0059] The electronic device may further include a network device and / or a client device. Among them, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing.

[0060] The network where the electronic device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0061] As Figure 1 shown, it is a flowchart of a preferred embodiment of the artificial intelligence-based auxiliary diagnosis method for thyroid nodules of the present application. According to different requirements, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0062] S10, collect thyroid ultrasound images with labeled data as a training set, where the labeled data includes the benign and malignant category labels and risk level labels of thyroid nodules in the thyroid ultrasound images.

[0063] In an optional embodiment, the step of collecting thyroid ultrasound images with labeled data as a training set, where the labeled data includes the benign and malignant category labels and risk level labels of thyroid nodules in the thyroid ultrasound images, includes:

[0064] Collect all thyroid ultrasound images with no disputes in the diagnostic results during the historical diagnosis process, where the diagnostic results include the benign and malignant judgment results of thyroid nodules and the TI-RADS assessment levels;

[0065] Obtain the label data for each thyroid ultrasound image according to the diagnostic results, where the label data includes the benign and malignant category labels and risk level labels of the thyroid nodules in the thyroid ultrasound image;

[0066] Store all thyroid ultrasound images and label data to obtain a training set, where the thyroid ultrasound images and the label data correspond one by one.

[0067] Among them, the TI-RADS (Thyroid imaging reporting and data system) is the unification of the terms and definitions of thyroid imaging by the American College of Radiology, and gives a risk assessment classification level of 6 categories according to the suspicious degree of lesion malignancy.

[0068] Exemplarily, if the diagnostic result of a thyroid ultrasound image is benign and the TI-RADS assessment level is 3, then the benign and malignant category label of the thyroid ultrasound image is benign, and the risk level label of the thyroid ultrasound image is 3.

[0069] In this way, the construction of the training set is completed. The training set contains multiple thyroid ultrasound images, and each thyroid ultrasound image corresponds to a label data, where the label data includes the benign and malignant category labels and risk level labels, providing a data basis for realizing the auxiliary diagnosis of thyroid nodules.

[0070] S11, build a first auxiliary diagnosis model based on the control vector set, where the first auxiliary diagnosis model includes a convolutional structure, a positioning structure, and a classification structure.

[0071] In an optional embodiment, the building of the first auxiliary diagnosis model based on the control vector set, where the first auxiliary diagnosis model includes a convolutional structure, a positioning structure, and a classification structure, includes:

[0072] Construct a control vector set, where the control vector set includes a control vector subset for each preset diagnostic category, the control vector subset includes a preset number of control feature vectors, and the preset diagnostic categories include benign low risk, benign high risk, malignant low risk, and malignant high risk;

[0073] Build a first auxiliary diagnosis model based on the control vector set, where the first auxiliary diagnosis model is formed by connecting the convolutional structure, the positioning structure, and the classification structure in series, and the input of the first auxiliary diagnosis model is a thyroid ultrasound image, and the output is the benign and malignant probability vector and risk level probability vector of the thyroid ultrasound image;

[0074] The input of the convolutional structure is a thyroid ultrasound image, and the output is a convolutional feature map of the thyroid ultrasound image, and the convolutional feature map includes a plurality of convolutional feature vectors;

[0075] The positioning structure is used to calculate the similarity between the convolutional feature map and each reference feature vector in the reference vector set, and arrange all the similarities in a fixed order along the column direction to obtain a similarity vector;

[0076] The input of the classification structure is the similarity vector, and the output is a benign and malignant probability vector and a risk level probability vector of the thyroid ultrasound image.

[0077] In an alternative embodiment, a TI-RADS assessment level or a risk level label not less than a preset level is regarded as a high risk, and a TI-RADS assessment level or a risk level label less than the preset level is regarded as a low risk, where the preset level is 4; then all diagnostic results can be divided into four preset diagnostic categories: benign low risk, benign high risk, malignant low risk, and malignant high risk according to the benign and malignant judgment results and the TI-RADS assessment level in the diagnostic results; for each preset diagnostic category, the thyroid ultrasound image corresponding to the preset diagnostic category is sent into an open-source feature extraction network to obtain a feature map, and then experts respectively select the same preset number of reference feature vectors with a preset size in the lesion internal area and the lesion external area of the feature map to obtain a reference vector subset for each preset diagnostic category, where the lesion area is the thyroid nodule area, and the open-source feature extraction network can be existing ResNet, Densenet, MobileNet, etc.; store the reference vector subsets for each preset diagnostic category to construct a reference vector set. It should be noted that the reference vector set can reflect medical-related information of different preset diagnostic categories.

[0078] Exemplarily, it is set that the preset size of the reference feature vector is 1×1×128, and the preset numbers of different preset diagnostic categories are different. It is set that the preset numbers corresponding to benign low risk, benign high risk, malignant low risk, and malignant high risk are 10, 5, 5, and 10 in sequence. Then, the reference vector subset of benign low risk contains a total of 10 reference feature vectors in the lesion internal area and 10 reference feature vectors in the lesion external area; a reference vector set is obtained in the same way, and the reference vector set contains a total of 60 reference feature vectors, and the size of each reference feature vector is 1×1×128.

[0079] In an alternative embodiment, a first auxiliary diagnosis model is built based on the reference vector set. The first auxiliary diagnosis model is composed of a convolutional structure, a positioning structure, and a classification structure connected in series. The structural schematic diagram of the first auxiliary diagnosis model is as Figure 2as shown

[0080] In an optional embodiment, the convolutional structure is used to extract features from the input thyroid ultrasound image. The convolutional structure is formed by connecting multiple convolutional layers in series. All convolutional layers continuously downsample the input thyroid ultrasound image to obtain a convolutional feature map with a size of W×H×C, where C represents the number of image channels in the convolutional feature map, and W×H represents the width and height dimensions of each image channel. The downsampling size is related to the number of convolutional layers in the convolutional structure. Each vector with a size of 1×1×C in the convolutional feature map is used as a convolutional feature vector. Then, the convolutional feature map includes a total of W×H convolutional feature vectors, and the sizes of the convolutional feature vectors are the same as those of the control feature vectors. Among them, the convolutional structure can adopt existing convolutional structures such as ResNet, DenseNet, MobileNet, etc., and this application does not make any restrictions.

[0081] It should be noted that the size of the convolutional feature vector is the same as that of the control feature vector in the control vector set.

[0082] Preferably, two 1×1 convolutional layers are added at the end of the pre-trained ResNet-34, and the activation function of the last convolutional layer is selected as the sigmoid function, while the activation functions of other convolutional layers are all ReLU functions to obtain a convolutional structure. The convolutional structure continuously downsamples the input thyroid ultrasound image to obtain a convolutional feature map with a size of 14×14×128. There are a total of 14×14 convolutional feature vectors in the convolutional feature map, and the size of each convolutional feature vector is 1×1×128.

[0083] In an optional embodiment, the positioning structure is used to calculate the similarity between the convolutional feature map and each control feature vector to introduce medical relevant information of thyroid nodules. In the positioning structure, the similarity between the convolutional feature map and each control feature vector in the control vector set is calculated, and the similarity satisfies the relational expression:

[0084]

[0085] where z is the convolutional feature map output by the convolutional structure, is any convolutional feature vector in the convolutional feature map, p i is the control feature vector i in the control vector set. The sizes of the convolutional feature vector and the control feature vector are the same; ∈ is an adjustment coefficient, and ∑ maxk(x) X(x) represents the sum of the first k maximum values among all parameters X(x), and g i(z) is the similarity between the convolutional feature map z and the control feature vector i, and the value of the adjustment coefficient is 0.001; further, all similarities are arranged in a fixed order along the column direction to obtain a similarity vector, and the size of the similarity vector is N rows by 1 column, where N is the number of control feature vectors in the control vector set.

[0086] In an alternative embodiment, the similarity vector is used as the input of the classification structure, and the classification structure includes a benign and malignant classifier and a risk level classifier, and both the benign and malignant classifier and the risk level classifier are composed of a fully connected layer and a softmax function; the benign and malignant classifier performs feature processing on the similarity vector to obtain a benign and malignant probability vector, and the benign and malignant probability vector is a vector of 2 rows by 1 column, and the 2 rows respectively represent the probability values that the thyroid ultrasound image belongs to benign and malignant, and the sum of all probability values in the benign and malignant probability vector is 1; the risk level classifier performs feature processing on the similarity vector to obtain a risk level probability vector, and the risk level probability vector is a vector of 6 rows by 1 column, and the 6 rows respectively represent the probabilities that the thyroid ultrasound image belongs to each TI-RADS assessment level, and the sum of all probability values in the risk level probability vector is 1.

[0087] In this way, the construction of the first auxiliary diagnosis model is completed. The introduction of the localization structure enables the first auxiliary diagnosis model to learn accurate medical-related information of the thyroid nodule before obtaining the benign and malignant probability vector and the risk level probability vector, improves the interpretability of the first auxiliary diagnosis model, and at the same time improves the accuracy of the benign and malignant probability vector and the risk level probability vector.

[0088] S12. Train the first auxiliary diagnosis model based on the training set and a preset loss function to update the model parameters, and obtain a second auxiliary diagnosis model and a standard vector set. The model parameters include the parameters of the convolutional structure, the localization structure, and the classification structure, and the parameter of the localization structure is the control vector set.

[0089] In an alternative embodiment, after the first auxiliary diagnosis model is constructed, the parameters in the first auxiliary diagnosis model are all initial parameters and cannot meet the requirements of thyroid nodule auxiliary diagnosis. In order to ensure the accuracy of the benign and malignant probability vector and the risk level probability vector output by the first auxiliary diagnosis model, it is necessary to train the first auxiliary diagnosis model based on the training set and a preset loss function to obtain a second auxiliary diagnosis model.

[0090] In an alternative embodiment, the first auxiliary diagnosis model is trained based on the training set and a preset loss function to update model parameters, obtaining a second auxiliary diagnosis model and a standard vector set. The model parameters include the parameters of the convolutional structure, the localization structure, and the classification structure. The parameters of the localization structure are the control vector set, including:

[0091] Fix the parameters in the classification structure, and based on the training set and the preset loss function, preliminarily train the convolutional structure and the localization structure in the first auxiliary diagnosis model to update the parameters of the convolutional structure and all control feature vectors in the control vector set in the localization structure;

[0092] During the preliminary training process, continuously select training batches from the training set to calculate the value of the preset loss function. When the value of the preset loss function no longer changes, stop the preliminary training, and use all control feature vectors in the localization structure as standard feature vectors, and store all standard feature vectors to obtain a standard vector set;

[0093] Fix the parameters of the convolutional structure and all control feature vectors in the localization structure, and based on the training set and the cross-entropy loss function, perform secondary training on the classification structure in the first auxiliary diagnosis model to update the parameters of the classification structure;

[0094] During the secondary training process, continuously select training batches from the training set to calculate the value of the cross-entropy loss function. When the value of the cross-entropy loss function no longer changes, stop the secondary training to obtain a second auxiliary diagnosis model.

[0095] In an alternative embodiment, first fix the parameters of the classification structure in the first auxiliary diagnosis model. In order to constrain the convolutional structure and the localization structure to learn image features related to the auxiliary diagnosis task in thyroid ultrasound images and the discriminative features between different preset diagnoses, based on the training set and the preset loss function, perform preliminary training on the convolutional structure and the localization structure in the first auxiliary diagnosis model to update the parameters of the convolutional structure and all control feature vectors in the localization structure. The preset loss function satisfies the relationship:

[0096]

[0097] where M represents the data volume of each training batch in the preliminary training, and respectively represent the benign / malignant category label and the risk level label of the i-th thyroid ultrasound image in a training batch, and respectively represent the benign / malignant probability vector and the risk level probability vector of the i-th thyroid ultrasound image output by the first auxiliary diagnosis model in a training batch, representation and The cross-entropy loss function, Clst is the mutual exclusion loss function, λ is the weight factor, and Loss is the value of the preset loss function. Among them, the value of the weight factor is 0.5.

[0098] Among them, the mutual exclusion loss function is used to constrain the convolutional structure and the localization structure to learn the discriminative features between different preset diagnoses, so as to improve the accuracy of the output result of the first auxiliary diagnosis model. The mutual exclusion loss function satisfies the relationship:

[0099]

[0100] Among them, M represents the amount of data in each training batch during the preliminary training, represents the preset diagnosis category of the i-th thyroid ultrasound image in a training batch, is the mutually exclusive category of the preset diagnosis category of the i-th thyroid ultrasound image obtained according to the preset mutual exclusion pair; p j is any control feature vector in the control vector subset of the preset diagnosis category of the i-th thyroid ultrasound image, is any control feature vector in the control vector subset of the mutually exclusive category of the preset diagnosis category of the i-th thyroid ultrasound image; is any convolutional feature vector in the convolutional feature map of the i-th thyroid ultrasound image, ∑ mink(x) X(x) represents the sum of the first k minimum values among all parameters X(x); Clst is the value of the mutual exclusion loss function.

[0101] Among them, the preset diagnosis category can be obtained according to the benign and malignant category labels and risk level labels of the thyroid ultrasound image; there are two preset mutual exclusion pairs in total. Benign low risk and malignant high risk form a preset mutual exclusion pair, and benign high risk and malignant low risk form another preset mutual exclusion pair.

[0102] In an alternative embodiment, during the preliminary training process, a fixed number of thyroid ultrasound images are continuously selected from the training set as a training batch, and the fixed number is the amount of data in a training batch; the thyroid ultrasound images in the training batch are sequentially input into the first auxiliary diagnosis model to calculate the value of the preset loss function, and the parameters of the convolutional structure in the first auxiliary diagnosis model and all control feature vectors in the localization structure are updated by the gradient descent method. When the value of the preset loss function no longer changes, the preliminary training is stopped, and all control feature vectors in the localization structure are used as standard feature vectors. The standard feature vectors can reflect the standard features of different preset diagnosis categories; all standard feature vectors are stored to obtain a standard vector set.

[0103] In an optional embodiment, after the preliminary training is completed, the convolutional part and the localization part of the first auxiliary diagnosis model can learn the image features related to the auxiliary diagnosis task in the thyroid ultrasound images and the discriminative features between different preset diagnoses, further fix the parameters of the convolutional structure and all the reference feature vectors in the localization structure, and perform secondary training on the classification structure in the first auxiliary diagnosis model based on the training set and the cross-entropy loss function to constrain the classification structure to learn the correct diagnosis results of benign / malignant and risk levels.

[0104] In this optional embodiment, during the secondary training process, a fixed number of thyroid ultrasound images are continuously selected from the training set as a training batch, and the fixed number is the data volume of a training batch; the thyroid ultrasound images in the training batch are sequentially input into the first auxiliary diagnosis model to calculate the value of the cross-entropy loss function, and the parameters of the classification structure in the first auxiliary diagnosis model are updated by the gradient descent method. When the value of the cross-entropy loss function no longer changes, the secondary training is stopped to obtain a second auxiliary diagnosis model, and the second auxiliary diagnosis model can obtain accurate benign / malignant probability vectors and risk level probability vectors.

[0105] In this way, the first auxiliary diagnosis model is trained twice based on the training set and the preset loss function to obtain a second auxiliary diagnosis model and a standard vector set, and the second auxiliary diagnosis model can obtain accurate benign / malignant probability vectors and risk level probability vectors.

[0106] S13, Collect the ultrasound image to be diagnosed and input it into the second auxiliary diagnosis model, use the output of the convolutional structure in the second auxiliary diagnosis model as the diagnostic convolutional feature map, and use the output of the classification structure in the second auxiliary diagnosis model as the auxiliary diagnosis result.

[0107] In an optional embodiment, the diagnostic convolutional feature map is the convolutional feature map of the ultrasound image to be diagnosed, and its size is W×H×C; further, the 1×1×C convolutional feature vectors in the diagnostic convolutional feature map are used as diagnostic convolutional feature vectors, and a total of W×H diagnostic convolutional feature vectors are obtained.

[0108] In this optional embodiment, the auxiliary diagnosis result includes the benign / malignant probability vector and the risk level probability vector in the ultrasound image to be diagnosed.

[0109] In this way, the diagnostic convolutional feature map and the auxiliary diagnosis result of the ultrasound image to be diagnosed are obtained by means of the second auxiliary diagnosis model.

[0110] S14, Construct an auxiliary feature map based on the diagnostic convolutional feature map and the standard vector set.

[0111] In an optional embodiment, constructing the auxiliary feature map based on the diagnostic convolutional feature map and the standard vector set includes:

[0112] Regarding each 1×1×C convolutional feature vector in the diagnostic convolutional feature map as a diagnostic convolutional feature vector, where C is the number of image channels of the diagnostic convolutional feature map;

[0113] Calculating the cosine similarity between each diagnostic convolutional feature vector and each standard feature vector in the standard vector set;

[0114] Selecting the standard feature vector corresponding to the maximum value among all the cosine similarities of the same diagnostic convolutional feature vector as the matching vector of the diagnostic convolutional feature vector, and the matching vector corresponds one-to-one with the diagnostic convolutional feature vector;

[0115] Replacing each diagnostic convolutional feature vector in the diagnostic convolutional feature map with the matching vector to construct the auxiliary feature map.

[0116] In this optional embodiment, the standard feature vector can reflect the standard features of different preset diagnostic categories, and the auxiliary feature map can reflect the preset diagnostic categories corresponding to different positions in the diagnostic convolutional feature map, providing auxiliary information for doctors' diagnosis.

[0117] In this way, the auxiliary feature map is obtained. The auxiliary feature map can intuitively reflect the preset diagnostic categories at different positions in the diagnostic convolutional feature map, and the preset diagnostic categories can in turn reflect the severity, providing reliable auxiliary information for doctors' diagnostic processes.

[0118] S15, Displaying the auxiliary feature map and the auxiliary diagnosis result on the terminal screen to assist the doctor's diagnostic process, and obtaining the diagnosis result of the to-be-diagnosed ultrasound image.

[0119] In an optional embodiment, when the auxiliary feature map and the auxiliary diagnosis result are displayed on the terminal screen, the doctor can understand the severity of different positions in the to-be-diagnosed ultrasound image with the help of the auxiliary feature map, and can understand the benign and malignant classification and risk level judgment of the to-be-diagnosed ultrasound image with the help of the auxiliary diagnosis result, so as to assist the doctor's diagnostic process to obtain the diagnosis result of the to-be-diagnosed ultrasound image.

[0120] In this way, based on the auxiliary feature map and the auxiliary diagnosis result, the doctor can locate the more severe areas in the to-be-diagnosed ultrasound image, and at the same time obtain the benign and malignant classification and risk level judgment, assisting the doctor's diagnostic process and improving the doctor's diagnostic efficiency.

[0121] As can be seen from the above technical solutions, the present application introduces medical-related information by means of standard feature vectors, adds a positioning structure to the auxiliary diagnosis model to calculate the similarity between medical image features and medical-related information, and determines the benignity, malignancy and risk level of nodules based on the similarity. At the same time, the standard feature vector is mapped onto the convolutional feature map of the medical image to achieve visualization, which can reflect the severity of different positions on the medical image, improve the interpretability and accuracy of the auxiliary diagnosis model, and thus improve the doctor's diagnosis efficiency.

[0122] Please refer to Figure 3 , Figure 3 FIG. is a functional module diagram of a preferred embodiment of the artificial-intelligence-based thyroid nodule auxiliary diagnosis device of the present application. The artificial-intelligence-based thyroid nodule auxiliary diagnosis device 11 includes an acquisition unit 110, a construction unit 111, a training unit 112, an input unit 113, a construction unit 114, and an auxiliary unit 115. The module / unit referred to in the present application means a series of computer-readable instruction segments that can be executed by a processor 13 and can complete a fixed function, and is stored in a memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0123] In an alternative embodiment, the acquisition unit 110 is configured to acquire thyroid ultrasound images with labeled data as a training set, and the labeled data includes benign and malignant category labels and risk level labels of thyroid nodules in the thyroid ultrasound images.

[0124] In an alternative embodiment, acquiring the thyroid ultrasound images with labeled data as a training set, where the labeled data includes benign and malignant category labels and risk level labels of thyroid nodules in the thyroid ultrasound images, includes:

[0125] Acquire all thyroid ultrasound images with no disputes in the diagnosis results during the historical diagnosis process, where the diagnosis results include the benign and malignant judgment results of thyroid nodules and the TI-RADS evaluation level;

[0126] Obtain the labeled data of each thyroid ultrasound image according to the diagnosis results, where the labeled data includes benign and malignant category labels and risk level labels of thyroid nodules in the thyroid ultrasound images;

[0127] Store all thyroid ultrasound images and labeled data to obtain a training set, where the thyroid ultrasound images and the labeled data correspond one by one.

[0128] Among them, the TI-RADS (Thyroid imaging reporting and data system) is the unification of the terms and definitions of thyroid imaging by the American College of Radiology, and a risk assessment classification level of 6 categories is given according to the suspected malignancy degree of the lesion.

[0129] Exemplarily, if the diagnostic result of the thyroid ultrasound image is benign and the TI-RADS assessment level is 3, the benign and malignant category label of the thyroid ultrasound image is benign, and the risk level label of the thyroid ultrasound image is 3.

[0130] In an optional embodiment, the building unit 111 is used to build a first auxiliary diagnosis model according to the control vector set, and the first auxiliary diagnosis model includes a convolution structure, a positioning structure and a classification structure.

[0131] In an optional embodiment, building the first auxiliary diagnosis model according to the control vector set, the first auxiliary diagnosis model includes a convolution structure, a positioning structure and a classification structure, including:

[0132] Construct a control vector set, the control vector set includes a control vector subset of each preset diagnosis category, the control vector subset includes a preset number of control feature vectors, and the preset diagnosis categories include benign low risk, benign high risk, malignant low risk and malignant high risk;

[0133] Build a first auxiliary diagnosis model according to the control vector set, the first auxiliary diagnosis model is formed by connecting the convolution structure, the positioning structure and the classification structure in series, the input of the first auxiliary diagnosis model is the thyroid ultrasound image, and the output is the benign and malignant probability vector and the risk level probability vector of the thyroid ultrasound image;

[0134] The input of the convolution structure is the thyroid ultrasound image, and the output is the convolution feature map of the thyroid ultrasound image, and the convolution feature map includes a plurality of convolution feature vectors;

[0135] The positioning structure is used to calculate the similarity between the convolution feature map and each control feature vector in the control vector set, and arrange all the similarities in a fixed order along the column direction to obtain a similarity vector;

[0136] The input of the classification structure is the similarity vector, and the output is the benign and malignant probability vector and the risk level probability vector of the thyroid ultrasound image.

[0137] In an alternative embodiment, a TI-RADS assessment level or a risk level label not less than a preset level is regarded as a high risk, and a TI-RADS assessment level or a risk level label less than the preset level is regarded as a low risk, where the preset level is 4; then all diagnostic results can be divided into four preset diagnostic categories: benign low risk, benign high risk, malignant low risk, and malignant high risk according to the benign and malignant judgment results and the TI-RADS assessment level in the diagnostic results; for each preset diagnostic category, the thyroid ultrasound image corresponding to the preset diagnostic category is sent into an open-source feature extraction network to obtain a feature map, and then experts respectively select the same preset number of control feature vectors with a preset size in the internal area and the external area of the lesion in the feature map to obtain a control vector subset for each preset diagnostic category, where the lesion area is the thyroid nodule area, and the open-source feature extraction network can be existing ResNet, Densenet, MobileNet, etc.; the control vector subset for each preset diagnostic category is stored to construct a control vector set. It should be noted that the control vector set can reflect medical-related information of different preset diagnostic categories.

[0138] Exemplarily, it is set that the preset size of the control feature vector is 1×1×128, and the preset numbers for different preset diagnostic categories are different. It is set that the preset numbers corresponding to benign low risk, benign high risk, malignant low risk, and malignant high risk are 10, 5, 5, and 10 in sequence. Then, the control vector subset of benign low risk contains a total of 10 control feature vectors in the internal area of the lesion and 10 control feature vectors in the external area of the lesion; the control vector set is obtained in the same way. The control vector set contains a total of 60 control feature vectors, and the size of each control feature vector is 1×1×128.

[0139] In an alternative embodiment, a first auxiliary diagnosis model is built based on the control vector set. The first auxiliary diagnosis model is composed of a convolution structure, a localization structure, and a classification structure connected in series. The structural schematic diagram of the first auxiliary diagnosis model is as Figure 2 shown.

[0140] In an optional embodiment, the convolution structure is used to extract features from the input thyroid ultrasound image. The convolution structure is formed by connecting multiple convolutional layers in series. All convolutional layers continuously perform downsampling on the input thyroid ultrasound image to obtain a convolutional feature map with a size of W×H×C, where C represents the number of image channels in the convolutional feature map, and W×H represents the width and height dimensions of each image channel. The size of the downsampling is related to the number of convolutional layers in the convolution structure. Each vector with a size of 1×1×C in the convolutional feature map is used as a convolutional feature vector. Then, the convolutional feature map includes a total of W×H convolutional feature vectors, and the size of the convolutional feature vector is the same as that of the control feature vector. Among them, the convolution structure can adopt existing convolution structures such as ResNet, DenseNet, MobileNet, etc., and this application does not make any restrictions.

[0141] It should be noted that the size of the convolutional feature vector is the same as that of the control feature vector in the control vector set.

[0142] Preferably, two 1×1 convolutional layers are added at the end of the pre-trained ResNet-34, and the activation function of the last convolutional layer is selected as the sigmoid function, while the activation functions of other convolutional layers are all ReLU functions to obtain a convolution structure. The convolution structure continuously performs downsampling on the input thyroid ultrasound image to obtain a convolutional feature map with a size of 14×14×128. There are a total of 14×14 convolutional feature vectors in the convolutional feature map, and the size of each convolutional feature vector is 1×1×128.

[0143] In an optional embodiment, the positioning structure is used to calculate the similarity between the convolutional feature map and each control feature vector to introduce medical relevant information of thyroid nodules. In the positioning structure, the similarity between the convolutional feature map and each control feature vector in the control vector set is calculated, and the similarity satisfies the relational expression:

[0144]

[0145] Among them, z is the convolutional feature map output by the convolution structure, is any convolutional feature vector in the convolutional feature map, p i is the control feature vector i in the control vector set. The size of the convolutional feature vector is the same as that of the control feature vector; ∈ is an adjustment coefficient, and ∑ maxk(x) X(x) represents the sum of the first k maximum values among all parameters X(x), and g i(z) is the similarity between the convolutional feature map z and the control feature vector i, and the value of the adjustment coefficient is 0.001; further, all similarities are arranged in a fixed order along the column direction to obtain a similarity vector, and the size of the similarity vector is N rows by 1 column, where N is the number of control feature vectors in the control vector set.

[0146] In an optional embodiment, the similarity vector is used as the input of the classification structure. The classification structure includes a benign and malignant classifier and a risk level classifier, and both the benign and malignant classifier and the risk level classifier are composed of a fully connected layer and a softmax function; the benign and malignant classifier performs feature processing on the similarity vector to obtain a benign and malignant probability vector. The benign and malignant probability vector is a vector of 2 rows by 1 column, and the two rows respectively represent the probability values that the thyroid ultrasound image belongs to benign and malignant, and the sum of all probability values in the benign and malignant probability vector is 1; the risk level classifier performs feature processing on the similarity vector to obtain a risk level probability vector. The risk level probability vector is a vector of 6 rows by 1 column, and the six rows respectively represent the probabilities that the thyroid ultrasound image belongs to each TI-RADS assessment level, and the sum of all probability values in the risk level probability vector is 1.

[0147] In an optional embodiment, the training unit 112 is used to train the first auxiliary diagnosis model based on the training set and a preset loss function to update the model parameters, and obtain a second auxiliary diagnosis model and a standard vector set. The model parameters include the parameters of the convolutional structure, the localization structure, and the classification structure, and the parameters of the localization structure are the control vector set.

[0148] In an optional embodiment, after the first auxiliary diagnosis model is built, the parameters in the first auxiliary diagnosis model are all initial parameters and cannot meet the requirements of thyroid nodule auxiliary diagnosis. To ensure the accuracy of the benign and malignant probability vector and the risk level probability vector output by the first auxiliary diagnosis model, it is necessary to train the first auxiliary diagnosis model based on the training set and a preset loss function to obtain a second auxiliary diagnosis model.

[0149] In an optional embodiment, training the first auxiliary diagnosis model based on the training set and a preset loss function to update the model parameters, obtaining a second auxiliary diagnosis model and a standard vector set, where the model parameters include the parameters of the convolutional structure, the localization structure, and the classification structure, and the parameters of the localization structure are the control vector set, includes:

[0150] Fix the parameters in the classification structure, and perform preliminary training on the convolutional structure and the localization structure in the first auxiliary diagnosis model based on the training set and a preset loss function to update the parameters of the convolutional structure and all control feature vectors in the control vector set of the localization structure;

[0151] During the preliminary training process, training batches are continuously selected from the training set to calculate the value of the preset loss function. When the value of the preset loss function no longer changes, the preliminary training is stopped, and all the reference feature vectors in the positioning structure are used as standard feature vectors. All the standard feature vectors are stored to obtain a standard vector set;

[0152] Fix the parameters of the convolutional structure and all the reference feature vectors in the positioning structure, and based on the training set and the cross-entropy loss function, perform secondary training on the classification structure in the first auxiliary diagnosis model to update the parameters of the classification structure;

[0153] During the secondary training process, training batches are continuously selected from the training set to calculate the value of the cross-entropy loss function. When the value of the cross-entropy loss function no longer changes, the secondary training is stopped to obtain the second auxiliary diagnosis model.

[0154] In an alternative embodiment, first fix the parameters of the classification structure in the first auxiliary diagnosis model. In order to constrain the convolutional structure and the positioning structure to learn the image features related to the auxiliary diagnosis task in thyroid ultrasound images and the discriminative features between different preset diagnoses, based on the training set and the preset loss function, perform preliminary training on the convolutional structure and the positioning structure in the first auxiliary diagnosis model to update the parameters of the convolutional structure and all the reference feature vectors in the positioning structure. The preset loss function satisfies the relation:

[0155]

[0156] where M represents the data volume of each training batch in the preliminary training, and respectively represent the benign / malignant category label and the risk level label of the i-th thyroid ultrasound image in a training batch, and respectively represent the benign / malignant probability vector and the risk level probability vector of the i-th thyroid ultrasound image output by the first auxiliary diagnosis model in a training batch, represents and is the cross-entropy loss function of, Clst is the mutual exclusion loss function, λ is the weight factor, and Loss is the value of the preset loss function. Among them, the value of the weight factor is 0.5.

[0157] Among them, the mutual exclusion loss function is used to constrain the convolutional structure and the positioning structure to learn the discriminative features between different preset diagnoses to improve the accuracy of the output result of the first auxiliary diagnosis model. The mutual exclusion loss function satisfies the relation:

[0158]

[0159] Among them, M represents the amount of data in each training batch during the preliminary training, represents the preset diagnosis category of the i-th thyroid ultrasound image in a training batch, is the mutually exclusive type of the preset diagnosis category of the i-th thyroid ultrasound image obtained according to the preset mutually exclusive pair; p j is any control feature vector in the control vector subset of the preset diagnosis category of the i-th thyroid ultrasound image, is any control feature vector in the control vector subset of the mutually exclusive type of the preset diagnosis category of the i-th thyroid ultrasound image; is any convolutional feature vector in the convolutional feature map of the i-th thyroid ultrasound image, ∑ mink(x) X(x) represents the sum of the first k minimum values among all parameters X(x); Clst is the value of the mutually exclusive loss function.

[0160] Among them, the preset diagnosis category can be obtained according to the benign and malignant category labels and risk level labels of the thyroid ultrasound image; there are two preset mutually exclusive pairs in total. Benign low risk and malignant high risk form a preset mutually exclusive pair, and benign high risk and malignant low risk form another preset mutually exclusive pair.

[0161] In an alternative embodiment, during the preliminary training process, a fixed number of thyroid ultrasound images are continuously selected from the training set as a training batch, and the fixed number is the amount of data in a training batch; the thyroid ultrasound images in the training batch are sequentially input into the first auxiliary diagnosis model to calculate the value of the preset loss function, and the parameters of the convolutional structure and all control feature vectors in the positioning structure in the first auxiliary diagnosis model are updated by the gradient descent method. When the value of the preset loss function no longer changes, the preliminary training is stopped, and all control feature vectors in the positioning structure are used as standard feature vectors, and the standard feature vectors can reflect the standard features of different preset diagnosis categories; all standard feature vectors are stored to obtain a standard vector set.

[0162] In an alternative embodiment, after the preliminary training is completed, the convolutional part and the positioning part of the first auxiliary diagnosis model can learn the image features related to the auxiliary diagnosis task in the thyroid ultrasound image and the difference features between different preset diagnoses, further fix the parameters of the convolutional structure and all control feature vectors in the positioning structure, and perform secondary training on the classification structure in the first auxiliary diagnosis model based on the training set and the cross-entropy loss function to constrain the classification structure to learn the correct diagnosis results of benign and malignant and risk levels.

[0163] In this optional embodiment, during the secondary training process, a fixed number of thyroid ultrasound images are continuously selected from the training set as a training batch, where the fixed number is the data volume of one training batch. The thyroid ultrasound images in the training batch are sequentially input into the first auxiliary diagnosis model to calculate the value of the cross-entropy loss function, and the parameters of the classification structure in the first auxiliary diagnosis model are updated by the gradient descent method. When the value of the cross-entropy loss function no longer changes, the secondary training is stopped to obtain a second auxiliary diagnosis model, which can obtain accurate benign and malignant probability vectors and risk level probability vectors.

[0164] In an optional embodiment, the input unit 113 is configured to collect the ultrasound image to be diagnosed and input it into the second auxiliary diagnosis model, use the output of the convolutional structure in the second auxiliary diagnosis model as the diagnostic convolutional feature map, and use the output of the classification structure in the second auxiliary diagnosis model as the auxiliary diagnosis result.

[0165] In an optional embodiment, the diagnostic convolutional feature map is the convolutional feature map of the ultrasound image to be diagnosed, and its size is W×H×C; further, the 1×1×C convolutional feature vector in the diagnostic convolutional feature map is used as the diagnostic convolutional feature vector, and a total of W×H diagnostic convolutional feature vectors are obtained.

[0166] In this optional embodiment, the auxiliary diagnosis result includes the benign and malignant probability vectors and the risk level probability vectors in the ultrasound image to be diagnosed.

[0167] In an optional embodiment, the construction unit 114 is configured to construct an auxiliary feature map based on the diagnostic convolutional feature map and the standard vector set.

[0168] In an optional embodiment, the constructing the auxiliary feature map based on the diagnostic convolutional feature map and the standard vector set includes:

[0169] Use each 1×1×C convolutional feature vector in the diagnostic convolutional feature map as a diagnostic convolutional feature vector, where C is the number of image channels of the diagnostic convolutional feature map;

[0170] Calculate the cosine similarity between each diagnostic convolutional feature vector and each standard feature vector in the standard vector set;

[0171] Select the standard feature vector corresponding to the maximum value among all the cosine similarities of the same diagnostic convolutional feature vector as the matching vector of the diagnostic convolutional feature vector, and the matching vector corresponds to the diagnostic convolutional feature vector one by one;

[0172] Replace each diagnostic convolutional feature vector in the diagnostic convolutional feature map with the matching vector to construct the auxiliary feature map.

[0173] In this optional embodiment, the standard feature vector can reflect the standard features of different preset diagnosis categories, and the auxiliary feature map can reflect the preset diagnosis categories corresponding to different positions in the diagnostic convolution feature map, providing auxiliary information for doctors' diagnosis.

[0174] In an optional embodiment, the auxiliary unit 115 is configured to display the auxiliary feature map and the auxiliary diagnosis result on the terminal screen to assist the doctor in the diagnosis process, and obtain the diagnosis result of the to-be-diagnosed ultrasound image.

[0175] In an optional embodiment, by displaying the auxiliary feature map and the auxiliary diagnosis result on the terminal screen, the doctor can understand the severity of different positions in the to-be-diagnosed ultrasound image with the help of the auxiliary feature map, and can understand the benign and malignant classification and risk level judgment of the to-be-diagnosed ultrasound image with the help of the auxiliary diagnosis result, so as to assist the doctor in the diagnosis process to obtain the diagnosis result of the to-be-diagnosed ultrasound image.

[0176] It can be seen from the above technical solutions that this application introduces medical-related information with the help of the standard feature vector, adds a positioning structure to the auxiliary diagnosis model to calculate the similarity between the medical image features and the medical-related information, and judges the benign and malignant nature and risk level of the nodules according to the similarity. At the same time, mapping the standard feature vector to the convolution feature map of the medical image to achieve visualization can reflect the severity of different positions on the medical image, improve the interpretability and accuracy of the auxiliary diagnosis model, and thus improve the doctor's diagnosis efficiency.

[0177] Please refer to Figure 4 , which is a schematic structural diagram of an electronic device provided by an embodiment of this application. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement the artificial intelligence-based thyroid nodule auxiliary diagnosis method described in any of the above embodiments.

[0178] In an optional embodiment, the electronic device 1 further includes a bus and a computer program stored in the memory 12 and executable on the processor 13, such as an artificial intelligence-based thyroid nodule auxiliary diagnosis program.

[0179] Figure 4 Only the electronic device 1 with the memory 12 and the processor 13 is shown. Those skilled in the art can understand that Figure 4 the shown structure does not limit the electronic device 1, and it may include fewer or more components than shown, or combine some components, or have different component arrangements.

[0180] Combined with Figure 1, the memory 12 in the electronic device 1 stores multiple computer-readable instructions to implement an artificial intelligence-based auxiliary diagnosis method for thyroid nodules. The processor 13 can execute the multiple instructions to implement:

[0181] Collect thyroid ultrasound images with labeled data as the training set. The labeled data includes the benign and malignant category labels and risk level labels of thyroid nodules in the thyroid ultrasound images;

[0182] Build a first auxiliary diagnosis model based on the control vector set. The first auxiliary diagnosis model includes a convolutional structure, a localization structure, and a classification structure;

[0183] Train the first auxiliary diagnosis model based on the training set and a preset loss function to update the model parameters, obtaining a second auxiliary diagnosis model and a standard vector set. The model parameters include the parameters of the convolutional structure, the localization structure, and the classification structure. The parameters of the localization structure are the control vector set;

[0184] Collect the ultrasound image to be diagnosed and input it into the second auxiliary diagnosis model. Use the output of the convolutional structure in the second auxiliary diagnosis model as the diagnostic convolutional feature map, and use the output of the classification structure in the second auxiliary diagnosis model as the auxiliary diagnosis result;

[0185] Build an auxiliary feature map based on the diagnostic convolutional feature map and the standard vector set;

[0186] Display the auxiliary feature map and the auxiliary diagnosis result on the terminal screen to assist the doctor's diagnosis process, obtaining the diagnosis result of the ultrasound image to be diagnosed.

[0187] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiment, which will not be elaborated here.

[0188] Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 can be a bus structure or a star structure. The electronic device 1 can also include more or fewer other hardware or software than shown, or different component arrangements. For example, the electronic device 1 can also include input / output devices, network access devices, etc.

[0189] It should be noted that the electronic device 1 is only an example. Other existing or future electronic products that can be adapted to this application should also be included in the protection scope of this application and are included herein by reference.

[0190] Among them, the memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disc, etc. The memory 12 can be an internal storage unit of the electronic device 1 in some embodiments, such as the mobile hard disk of the electronic device 1. The memory 12 can also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the electronic device 1. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the artificial intelligence-based thyroid nodule auxiliary diagnosis program, etc., but also be used to temporarily store the data that has been output or will be output.

[0191] In some embodiments, the processor 13 can be composed of integrated circuits. For example, it can be composed of a single packaged integrated circuit, or can be composed of multiple integrated circuits with the same or different functions packaged, including the combination of one or more Central Processing Units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips, etc. The processor 13 is the control core (Control Unit) of the electronic device 1, connecting various components of the entire electronic device 1 through various interfaces and lines. By running or executing the programs or modules stored in the memory 12 (such as executing the artificial intelligence-based thyroid nodule auxiliary diagnosis program, etc.), and calling the data stored in the memory 12, it can execute various functions of the electronic device 1 and process data.

[0192] The processor 13 executes the operating system of the electronic device 1 and various installed application programs. The processor 13 executes the application programs to implement the steps in the above-mentioned embodiments of various artificial intelligence-based thyroid nodule auxiliary diagnosis methods, such as Figure 1 the steps shown.

[0193] Exemplarily, the computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an acquisition unit 110, a construction unit 111, a training unit 112, an input unit 113, a construction unit 114, and an auxiliary unit 115.

[0194] The integrated units implemented in the form of software function modules as described above may be stored in a computer-readable storage medium. The software function modules stored in a storage medium include several instructions for causing a computer device (which may be a personal computer, a computer device, or a network device, etc.) or a processor (Processor) to execute a part of the artificial intelligence-based thyroid nodule auxiliary diagnosis method described in various embodiments of the present application.

[0195] If the modules / units integrated in the electronic device 1 are implemented in the form of software function units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present application, it may also be completed by a computer program instructing relevant hardware devices. The computer program may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-described various method embodiments may be implemented.

[0196] Among them, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disc, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory, and other memories, etc.

[0197] Further, the computer-readable storage medium mainly includes a program storage area and a data storage area. Among them, the program storage area may store an operating system, application programs required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0198] The blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithms. A blockchain, in essence, is a decentralized database, a series of data blocks generated by using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of the information (anti-counterfeiting) and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, an application service layer, etc.

[0199] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, in Figure 4 it is only represented by one arrow, but it does not mean that there is only one bus or one type of bus. The bus is set to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0200] The embodiments of this application also provide a computer-readable storage medium (not shown in the figure). Computer-readable instructions are stored in the computer-readable storage medium, and the computer-readable instructions are executed by a processor in an electronic device to implement the artificial intelligence-based thyroid nodule auxiliary diagnosis method described in any of the above embodiments.

[0201] In several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation.

[0202] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0203] In addition, the functional modules in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0204] In addition, it is obvious that the term "including" does not exclude other units or steps, and the singular form does not exclude the plural form. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. Terms such as first and second are used to denote names and do not denote any particular order.

[0205] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An artificial intelligence-based auxiliary diagnosis method for thyroid nodules, characterized in that, The method includes: Collecting thyroid ultrasound images with labeled data as a training set, where the labeled data includes the benign and malignant category labels and risk level labels of thyroid nodules in the thyroid ultrasound images; Constructing a first auxiliary diagnosis model based on a control vector set, where the first auxiliary diagnosis model includes a convolutional structure, a localization structure, and a classification structure; Training the first auxiliary diagnosis model based on the training set and a preset loss function to update model parameters, obtaining a second auxiliary diagnosis model and a standard vector set, where the model parameters include the parameters of the convolutional structure, the localization structure, and the classification structure, the parameters of the localization structure being the control vector set, and the preset loss function satisfying the relational expression: where M represents the data volume of each training batch in the preliminary training, and respectively represent the benign / malignant category label and the risk level label of the i-th thyroid ultrasound image in a training batch, and respectively represent the benign / malignant probability vector and the risk level probability vector of the i-th thyroid ultrasound image output by the first auxiliary diagnosis model in a training batch, represents and 's cross-entropy loss function, Clst is the mutual exclusion loss function, λ is the weight factor, Loss is the value of the preset loss function, and the mutual exclusion loss function satisfies the relational expression: where M represents the data volume of each training batch in the preliminary training, represents the preset diagnosis category of the i-th thyroid ultrasound image in a training batch, is the mutual exclusion type of the preset diagnosis category of the i-th thyroid ultrasound image obtained according to the preset mutual exclusion pair; p j is any control feature vector in the control vector subset of the preset diagnosis category of the i-th thyroid ultrasound image, is any control feature vector in the control vector subset of the mutual exclusion type of the preset diagnosis category of the i-th thyroid ultrasound image; is any convolutional feature vector in the convolutional feature map of the i-th thyroid ultrasound image, ∑ mink(x) X(x) represents the sum of the first k minimum values among all parameters X(x); Clst is the value of the mutual exclusion loss function; Collecting the ultrasound image to be diagnosed and inputting it into the second auxiliary diagnosis model, taking the output of the convolutional structure in the second auxiliary diagnosis model as the diagnostic convolutional feature map, and taking the output of the classification structure in the second auxiliary diagnosis model as the auxiliary diagnosis result; Constructing an auxiliary feature map based on the diagnostic convolutional feature map and the standard vector set; Displaying the auxiliary feature map and the auxiliary diagnosis result on the terminal screen to assist the doctor in the diagnosis process, and obtaining the diagnosis result of the ultrasound image to be diagnosed.

2. The artificial intelligence-based auxiliary diagnosis method for thyroid nodules according to claim 1, characterized in that, The constructing of the first auxiliary diagnosis model based on the control vector set, where the first auxiliary diagnosis model includes a convolutional structure, a localization structure, and a classification structure, includes: Constructing a control vector set, where the control vector set includes control vector subsets for each preset diagnosis category, the control vector subset includes a preset number of control feature vectors, and the preset diagnosis categories include benign low risk, benign high risk, malignant low risk, and malignant high risk; Constructing a first auxiliary diagnosis model based on the control vector set, where the first auxiliary diagnosis model is composed of a convolutional structure, a localization structure, and a classification structure connected in series, the input of the first auxiliary diagnosis model is a thyroid ultrasound image, and the output is the benign and malignant probability vectors and risk level probability vectors of the thyroid ultrasound image; The input of the convolutional structure is a thyroid ultrasound image, and the output is the convolutional feature map of the thyroid ultrasound image, where the convolutional feature map includes multiple convolutional feature vectors; The localization structure is used to calculate the similarity between the convolutional feature map and each control feature vector in the control vector set, and arrange all similarities in a fixed order along the column direction to obtain a similarity vector; The input of the classification structure is the similarity vector, and the output is the benign and malignant probability vectors and risk level probability vectors of the thyroid ultrasound image.

3. The artificial intelligence-based auxiliary diagnosis method for thyroid nodules according to claim 2, characterized in that, The similarity satisfies the relational expression: Among them, z is the convolutional feature map output by the convolutional structure, is any convolutional feature vector in the convolutional feature map, p i is the reference feature vector i in the reference vector set, and the sizes of the convolutional feature vector and the reference feature vector are the same; ∈ is the adjustment coefficient, ∑ maxk(x) X(x) represents the sum of the first k maximum values among all parameters X(x), g i (z) is the similarity between the convolutional feature map z and the reference feature vector i.

4. The artificial intelligence-based auxiliary diagnosis method for thyroid nodules according to claim 1, characterized in that, Training the first auxiliary diagnosis model based on the training set and a preset loss function to update the model parameters, obtaining a second auxiliary diagnosis model and a standard vector set, where the model parameters include the parameters of the convolutional structure, the localization structure, and the classification structure, and the parameter of the localization structure is the control vector set, includes: Fixing the parameters in the classification structure, and preliminarily training the convolutional structure and the localization structure in the first auxiliary diagnosis model based on the training set and a preset loss function to update the parameters of the convolutional structure and all control feature vectors in the control vector set of the localization structure; During the preliminary training process, continuously select training batches from the training set to calculate the value of the preset loss function, stop the preliminary training when the value of the preset loss function no longer changes, and take all control feature vectors in the localization structure as standard feature vectors, and store all standard feature vectors to obtain a standard vector set; Fix the parameters of the fixed convolution structure and all the reference feature vectors in the positioning structure, and perform secondary training on the classification structure in the first auxiliary diagnosis model based on the training set and the cross-entropy loss function to update the parameters of the classification structure; During the secondary training process, continuously select training batches from the training set to calculate the value of the cross-entropy loss function. When the value of the cross-entropy loss function no longer changes, stop the secondary training to obtain the second auxiliary diagnosis model.

5. The artificial intelligence-based auxiliary diagnosis method for thyroid nodules according to claim 1, characterized in that, The constructing the auxiliary feature map based on the diagnostic convolution feature map and the standard vector set includes: Regard each 1×1×C convolution feature vector in the diagnostic convolution feature map as a diagnostic convolution feature vector, where C is the number of image channels of the diagnostic convolution feature map; Calculate the cosine similarity between each diagnostic convolution feature vector and each standard feature vector in the standard vector set; Select the standard feature vector corresponding to the maximum value among all the cosine similarities of the same diagnostic convolution feature vector as the matching vector of the diagnostic convolution feature vector, and the matching vector corresponds to the diagnostic convolution feature vector one by one; Replace each diagnostic convolution feature vector in the diagnostic convolution feature map with the matching vector to construct the auxiliary feature map.

6. An artificial intelligence-based auxiliary diagnosis device for thyroid nodules, characterized in that, The device is used to implement the artificial intelligence-based thyroid nodule auxiliary diagnosis method according to any one of claims 1 to 5. The device includes: An acquisition unit, configured to acquire thyroid ultrasound images with labeled data as a training set, where the labeled data includes the benign and malignant category labels and risk level labels of thyroid nodules in the thyroid ultrasound images; A construction unit, configured to construct a first auxiliary diagnosis model based on a reference vector set, where the first auxiliary diagnosis model includes a convolution structure, a positioning structure, and a classification structure; A training unit, configured to train the first auxiliary diagnosis model based on the training set and a preset loss function to update the model parameters, and obtain a second auxiliary diagnosis model and a standard vector set. The model parameters include the parameters of the convolution structure, the positioning structure, and the classification structure, and the parameters of the positioning structure are the reference vector set; An input unit, configured to acquire the ultrasound image to be diagnosed and input it into the second auxiliary diagnosis model, use the output of the convolution structure in the second auxiliary diagnosis model as the diagnostic convolution feature map, and use the output of the classification structure in the second auxiliary diagnosis model as the auxiliary diagnosis result; A construction unit, configured to construct an auxiliary feature map based on the diagnostic convolution feature map and the standard vector set; An auxiliary unit, configured to display the auxiliary feature map and the auxiliary diagnosis result on the terminal screen to assist the doctor's diagnosis process, and obtain the diagnosis result of the ultrasound image to be diagnosed.

7. An electronic device, characterized in that, The electronic device includes: A memory, storing computer-readable instructions; and A processor, configured to execute the computer-readable instructions stored in the memory to implement the artificial intelligence-based thyroid nodule auxiliary diagnosis method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the method for computer-aided diagnosis of thyroid nodules based on artificial intelligence according to any one of claims 1 to 5 is implemented.

Images