Classification model generation method, classification method, device, equipment and medium

Abstract

The embodiments of the present disclosure provide a classification model generation method, a classification method, an apparatus, a device, and a medium. The classification model generation method includes: obtaining, for an area to be inspected of a sample object, an image data sample of the area to be inspected, and obtaining a protein sequence data sample of each sample object to construct a training sample; using the training sample as input, training a preset classification model to obtain a classification model, wherein the classification model is used to predict the property category of the area to be inspected of the sample object, wherein the preset classification model includes an image data classification model for classifying image data samples, and a protein sequence classification model for classifying protein sequence data samples, and the output of the image data classification model and the output of the protein sequence classification model are feature-concatenated and output as a classification result through a fully connected layer.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a classification model generation method, a classification method, a device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Ultrasound is a commonly used screening tool for thyroid disease and has been used for a long time. It can sensitively detect thyroid nodules, accurately assess their location, size, and number, and differentiate between benign and malignant thyroid lesions. However, statistics show that approximately 30% of thyroid nodules cannot be clearly diagnosed before surgery, and current clinical decisions often favor removal. The thyroid gland contains important immune cells and secretes essential molecules such as thyroxine, which promotes growth and development, metabolism, and regulates organ activity. People who lose their thyroid glands require lifelong medication.

[0003] The medical field has very high requirements for accuracy. The accuracy of distinguishing benign and malignant thyroid nodules is crucial to patients, but there is currently a lack of accurate identification methods. Summary of the Invention

[0004] The embodiments of the present disclosure provide a classification model generation method, a classification method, an apparatus, a device, and a storage medium, which can improve the accuracy of classification.

[0005] In one aspect, an embodiment of the present disclosure provides a classification model generation method, comprising:

[0006] For the area to be inspected of the sample object, an image data sample of the area to be inspected is obtained, and a protein sequence data sample of each sample object is obtained to construct a training sample;

[0007] The training sample is used as input to train the preset classification model to obtain a classification model, and the classification model is used to predict the property category of the area to be inspected of the sample object; the preset classification model includes an image data classification model for classifying image data samples, and a protein sequence classification model for classifying protein sequence data samples. The output of the image data classification model and the output of the protein sequence classification model are feature-concatenated and output as a classification result through a fully connected layer.

[0008] On the other hand, the present disclosure also provides a classification method, including:

[0009] Acquiring image data of the area to be examined of the object to be tested, and acquiring protein sequence data;

[0010] The image data and protein sequence data are input into a classification model, wherein the classification model is obtained according to the aforementioned classification model generation method, and the property category of the area to be checked is determined based on the output of the classification model.

[0011] On the other hand, the present disclosure also provides a classification model generation device, including:

[0012] A feature acquisition and sample construction module is used to obtain image data samples of the area to be examined for the sample object, and to obtain protein sequence data samples of each sample object to construct a training sample;

[0013] A model training module is used to train a preset classification model with the training sample as input to obtain a classification model, wherein the classification model is used to predict the property category of the area to be inspected of the sample object, wherein the preset classification model includes an image data classification model for classifying image data samples, and a protein sequence classification model for classifying protein sequence data samples. The output of the image data classification model and the output of the protein sequence classification model are feature-concatenated and output as a classification result through a fully connected layer.

[0014] On the other hand, an embodiment of the present disclosure further provides a classification device, including:

[0015] A feature acquisition module is used to obtain image data of the area to be examined of the object to be tested and to obtain protein sequence data;

[0016] A classification module is used to input the image data and protein sequence data into a classification model, wherein the classification model is obtained according to the aforementioned classification model generation method, and based on the output of the classification model, determine the property category of the area to be checked.

[0017] On the other hand, an embodiment of the present disclosure further provides an electronic device, comprising a processor and a memory storing a computer program that can be run on the processor, wherein when the processor executes the program, it implements the aforementioned classification model generation method, or implements the aforementioned classification method.

[0018] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program that can be run on a processor, and when the computer program is executed by the processor, it implements the aforementioned classification model generation method, or implements the aforementioned classification method.

[0019] By using the method of this embodiment, the properties of the region to be checked are classified by combining image data and protein sequence data, and the input information is increased to improve the accuracy of the classification.

[0020] Of course, implementing any product or method of the present disclosure does not necessarily require achieving all of the advantages described above simultaneously. Other features and advantages of the present disclosure will be described in the following embodiments and, in part, will become apparent from the embodiments or be learned through practice of the present disclosure. The objectives and other advantages of the embodiments of the present disclosure can be achieved and obtained through the structures particularly pointed out in the description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are intended to provide a further understanding of the technical solutions of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation of the technical solutions of the present disclosure. The shapes and sizes of the components in the drawings do not reflect the actual scale and are intended only to illustrate the contents of the present disclosure.

[0022] Figure 1 A flow chart of a classification model generation method provided in an embodiment of the present disclosure;

[0023] Figure 2 This is a schematic diagram of the structure of a preset classification model according to an embodiment of the present disclosure;

[0024] Figure 3 This is a schematic diagram of the structure of another preset classification model according to an embodiment of the present disclosure;

[0025] Figure 4 for Figure 3 Diagram of the internal structure of the residual network in the shown structure;

[0026] Figure 5 This is a flow chart of a classification method according to an embodiment of the present disclosure;

[0027] Figure 6 This is a schematic diagram of the structure of a classification model generating device according to an embodiment of the present disclosure;

[0028] Figure 7 This is a schematic structural diagram of a classification device according to an embodiment of the present disclosure;

[0029] Figure 8 This is a schematic diagram of the classification model structure of the application example of the present disclosure;

[0030] Figure 9 This is the ultrasound data of thyroid nodules;

[0031] Figure 10 Detailed information diagram of each layer structure in resnet models of different depths;

[0032] Figure 11 Schematic diagram of the CBAM module structure;

[0033] Figure 12 It is a schematic diagram of the CAM submodule structure;

[0034] Figure 13 It is a schematic diagram of the SAM submodule structure;

[0035] Figure 14 The following are the identifiers, symbols and Entrez library gene names of 14 proteins that are important in distinguishing the nature of thyroid nodules;

[0036] Figure 15 Schematic diagram of one-hot coding of amino acids;

[0037] Figure 16 Schematic diagram of the ViT network structure;

[0038] Figure 17 Flowchart for training a classification model that includes a segmentation model;

[0039] Figure 18 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] The present disclosure describes a plurality of embodiments, but the description is exemplary rather than restrictive, and it is apparent to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure. Although many possible feature combinations are shown in the drawings and discussed in the embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0041] The present disclosure includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The disclosed embodiments, features, and elements of the present disclosure may also be combined with any conventional features or elements to form a unique inventive solution defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other inventive solutions to form another unique inventive solution defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in this disclosure may be implemented individually or in any appropriate combination. Therefore, the embodiments are not subject to other limitations except for the limitations set forth in the appended claims and their equivalents. In addition, various modifications and changes may be made within the scope of protection of the appended claims.

[0042] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be understood by those skilled in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation on the claims. In addition, the claims to the method and / or process should not be limited to performing their steps in the order written, and those skilled in the art can readily understand that these orders can be changed and still remain within the spirit and scope of the disclosed embodiments.

[0043] Unless otherwise defined, the technical or scientific terms used in this disclosure have the usual meanings understood by persons of ordinary skill in the art to which this disclosure belongs. The words "first", "second" and similar terms used in this disclosure do not indicate any order, quantity or importance, but are merely used to distinguish different components. In this disclosure, "plurality" may refer to two or more numbers. Words such as "include" or "comprising" mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0044] In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of some known functions and components. The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures can refer to the general design.

[0045] The inventors noted that the Guo Tiannan and Li Ziqing laboratories at Westlake University collaborated with multiple clinical teams in China and, through analysis of proteomics data, identified 14 protein combinations that play a key role in distinguishing benign and malignant thyroid nodules. Ten of these combinations have been reported in existing literature. Consequently, the inventors proposed a classification method that combines ultrasound imaging data and protein data to distinguish between benign and malignant thyroid nodules. By combining data features from multiple dimensions, such as imaging and protein data, the model's input information is increased, improving the accuracy and precision of the classification model's discrimination.

[0046] Figure 1 The flow chart of the classification model acquisition method provided in the embodiment of the present disclosure includes the following steps:

[0047] Step 11, for the area to be inspected of the sample object, obtain an image data sample of the area to be inspected, and obtain a protein sequence data sample of each sample object to construct a training sample;

[0048] In an exemplary embodiment, taking the application of the classification model to the determination of properties of thyroid nodules as an example, the sample subject may be a patient with a thyroid nodule, the region to be examined may be the thyroid region, the image data sample may be, for example, thyroid ultrasound image data, and the protein sequence data sample may be obtained by measuring a tissue sample of the region to be examined. In addition to ultrasound image data, the image data may also include magnetic resonance imaging (MRI) data, computed tomography (CT) image data, or any one or more of the three.

[0049] In other embodiments, the classification model can also be used to determine the properties of other lesion areas. In this case, the image data sample of the area to be examined is image data of the lesion area, including but not limited to at least one of ultrasound images, MRI images, and CT images. The protein sequence data sample is the protein sequencing result of the sample object.

[0050] In an exemplary embodiment, the image data samples and protein sequence data samples of a sample object are combined into a sample group, and the property category of the target region of the sample object is used as the label for the sample group to construct a loss function during subsequent model training. Multiple sample objects constitute multiple sample groups, and multiple sample groups constitute training samples. Each sample group includes the image data and protein sequence data corresponding to a sample object, as well as the corresponding property category label.

[0051] Step 12: Using the training sample as input, train a preset classification model to obtain a classification model, which is used to predict the property category of the area to be inspected of the sample object.

[0052] The disclosed embodiment classifies the properties of the region to be examined (eg, thyroid nodules) by combining image data and protein sequence data. Since different types of data are used to jointly determine the property category, the accuracy of classification can be improved.

[0053] In an exemplary embodiment, as Figure 2 As shown, the preset classification model includes an image data classification model for classifying image data samples and a protein sequence classification model for classifying protein sequence data samples. The output of the image data classification model and the output of the protein sequence classification model are feature-concatenated and then passed through a fully connected layer (which can be 2-3 layers) to output the classification result. During training, the image data classification model and the protein sequence classification model can be trained separately, and then the entire preset classification model can be trained to improve the accuracy of the entire model.

[0054] In an exemplary embodiment, as Figure 3 and Figure 4As shown, the image data classification model includes a first residual neural network, a first CBAM and a first fully connected layer, wherein the first residual neural network includes five modules connected in sequence, the image data sample is input into the first layer module of the first residual neural network, the first CBAM is located after the first layer module and after the fifth layer module, the output of the first CBAM after the fifth layer module is used as the input of the first fully connected layer, and the output of the first fully connected layer is the output of the image data classification model.

[0055] In an exemplary embodiment, as Figure 3 and Figure 4 As shown, the protein sequence classification model includes an encoding splicing module, a second residual neural network, a second CBAM and a second fully connected layer, wherein the second residual neural network includes five modules connected in sequence, and the protein sequence data sample is input into the first layer module of the second residual neural network after encoding and splicing, and the second CBAM is respectively located after the first layer module and after the fifth layer module, and the output of the second CBAM after the fifth layer module is used as the input of the second fully connected layer, and the output of the second fully connected layer is the output of the protein sequence classification model. In an exemplary embodiment, the protein sequence classification model may further include: a visual converter (ViT model) and a third fully connected layer, and the protein sequence data sample is input into the visual converter after encoding and splicing, and the output of the visual converter is used as the input of the third fully connected layer.

[0056] In an exemplary embodiment, in the above step 10, after obtaining the image data sample of the area to be inspected, the method further includes segmenting the image data sample using a preset segmentation model; and training the preset segmentation model to obtain a segmentation model.

[0057] Among them, training the preset segmentation model includes: respectively using the boundary direction histogram method and the Fourier shape descriptor method to extract the contour features of the segmentation results to obtain histogram eigenvalues and Fourier eigenvalues, determining the loss value of the segmentation model based on the baseline value of the boundary direction histogram method and the baseline value of the Fourier shape descriptor method, updating the parameters of the segmentation model based on the loss value, and using the model that meets the training end condition as the segmentation model, and the training end condition is that the segmentation model converges or reaches a preset number of updates. Using the boundary direction histogram method and the Fourier shape descriptor method to extract contour features and increase calculations from the perspective of the boundary can improve accuracy. In other examples, other methods can also be used to reflect boundary information.

[0058] In the training end condition, segmentation model convergence means that the loss value is less than or equal to the preset loss value, or the loss value can no longer decrease. The preset number of updates can be set according to actual needs.

[0059] In an exemplary embodiment, training the preset segmentation model to obtain the segmentation model includes: jointly training the preset classification model and the preset segmentation model. That is, when calculating the final loss function L, the loss function Lc of the classification model and the loss function Ls of the segmentation model are simultaneously referenced.

[0060] The present disclosure also provides a classification method, such as Figure 5 Shown, including:

[0061] Step 21, obtaining image data of the area to be examined of the object to be tested, and obtaining protein sequence data;

[0062] In step 22, the image data and protein sequence data are input into a classification model, which is obtained according to the classification model generation method of the aforementioned embodiment. Based on the output of the classification model, the property category of the area to be checked is determined.

[0063] The structure of the classification model is described in the previous article and will not be repeated here.

[0064] In an exemplary embodiment, after acquiring the image data, the method further includes segmenting the image data sample using a preset segmentation model, and using the segmentation results as the image data of the area to be examined. Segmenting key areas, such as nodules and tumors, facilitates subsequent classification and prevents interference from the image that may affect the classification results.

[0065] The present disclosure also provides a classification model generation device, such as Figure 6 Shown, including:

[0066] A feature acquisition and sample construction module is used to obtain image data samples of the area to be examined for the sample object, and to obtain protein sequence data samples of each sample object to construct a training sample;

[0067] A model training module is used to train a preset classification model with the training sample as input to obtain a classification model, wherein the classification model is used to predict the property category of the area to be inspected of the sample object, wherein the preset classification model includes an image data classification model for classifying image data samples, and a protein sequence classification model for classifying protein sequence data samples. The output of the image data classification model and the output of the protein sequence classification model are feature-concatenated and output as a classification result through a fully connected layer.

[0068] The operation of each module in the classification model generation device can be found in the description of the aforementioned method and will not be repeated here.

[0069] The present disclosure also provides a classification device, such as Figure 7 Shown, including:

[0070] A feature acquisition module is used to obtain image data of the area to be examined of the object to be tested and to obtain protein sequence data;

[0071] A classification module is used to input the image data and protein sequence data into a classification model, wherein the classification model is obtained according to the aforementioned classification model generation method, and based on the output of the classification model, determine the property category of the area to be checked.

[0072] The following is an example of applying the classification model to determine the properties of thyroid nodules, where the image data is ultrasound data, to specifically illustrate the above method. The preset classification model is as follows: Figure 8 The training methods include:

[0073] Step 1: acquiring an ultrasound image data sample of a region to be examined of a sample object;

[0074] Ultrasound is a commonly used method for screening thyroid diseases. It was first used in the examination of thyroid diseases. It can sensitively detect thyroid nodules, accurately assess the location, size, and number of nodules, and distinguish between benign and malignant thyroid lesions. With the continuous development of high-frequency ultrasound probes and the development and application of various advanced computer image processing technologies, the resolution of ultrasound for thyroid imaging is getting higher and higher, and it has become the preferred examination method for thyroid diseases. The preprocessing of thyroid ultrasound data includes but is not limited to one or more of the following: denoising, smoothing, enhancement, etc. Thyroid ultrasound data such as Figure 9 shown.

[0075] Step 2: inputting the ultrasound image data into an image data classification model (hereinafter referred to as image model);

[0076] The image model is implemented using a residual neural network model and an attention model, wherein the residual neural network model can be implemented using ResNet50. The residual neural network model includes five layers: Layer0, Layer1, Layer2, Layer3, and Layer4, corresponding to conv1, conv2_x, conv3_x, conv4_x, and conv5_x of ResNet50 (with the same structure). The image model is finally implemented using the first fully connected layer for dimensionality reduction.

[0077] The major contribution of residual neural networks is the discovery of degradation and the invention of shortcut connections to address this phenomenon. This significantly alleviates the difficulty of training extremely deep neural networks. For the first time, the depth of neural networks has exceeded 100 layers, with the largest neural networks exceeding 1,000 layers. Figure 10 Detailed information on each layer structure in resnet models of different depths.

[0078] In this example, a CBAM (Convolutional Block Attention Module) connection is added after the first layer Layer0 (conv1) module and the fifth layer Layer4 (conv5_x) module of the residual neural network. After testing, it was found that this connection can obtain better classification results. CBAM is a lightweight convolutional attention module that combines channel and spatial attention mechanisms. CBAM consists of a CAM (Channel Attention Module) sub-module and a SAM (Spatial Attention Module) sub-module. The CAM module compresses the spatial dimension while keeping the channel dimension unchanged, and calculates the importance (weight) of each channel of the input image. The SAM module keeps the spatial dimension unchanged, compresses the channel dimension, and focuses on the location information of the target, thereby improving the ability of feature representation. Figure 11 As shown in Figure 1, the input feature is processed by the CAM submodule, and the processing result is multiplied with the input feature to obtain the channel correction result (F'). One channel of the channel correction result is input to the SAM submodule, and the other channel is multiplied with the processing result of the SAM submodule as the output feature of the CBAM module. Figure 12 As shown in the figure, the CAM submodule includes a maximum pooling layer (MaxPool), an average pooling layer (AvgPool), and a multi-layer perceptron (MLP) shared by the maximum pooling layer and the average pooling layer. The output result is the channel attention result Mc. Mc is multiplied by the input feature to obtain the feature corrected by the CAM submodule - the channel correction feature F', which serves as the input of the SAM submodule. Figure 13 As shown in the figure, the SAM submodule includes MaxPool, AvgPool and the convolution layer (conv layer) outputting the spatial attention result Ms. Ms is multiplied by F' to obtain the final output feature of CBAM.

[0079] Step 3: Use the cross-validation method to train the image model.

[0080] The loss function is the cross entropy loss function L c :

[0081]

[0082] is the reference value (ground truth) or true value, indicating that sample i belongs to category C or does not belong to category C. is the prediction result, which indicates the predicted probability that sample i belongs to category C, m is the number of samples, C is the category, and w c is the weight of the category.

[0083] The above steps 1-3 are the training method of the image model.

[0084] Step 4: for the region to be checked of the sample object, obtain a protein sequence data sample of the region to be checked;

[0085] The method for obtaining a protein sequence can be to perform mass spectrometry analysis on a thyroid tissue sample of a sample subject, which can include the following three steps: (1) using an enzyme to cut the protein into shorter polypeptides; (2) sequencing each of the produced polypeptides; and (3) assembling the measured peptide sequences into a complete protein sequence.

[0086] Figure 14 These are the identifiers, symbols, and Entrez library gene names of 14 proteins that play an important role in distinguishing the nature (benign or malignant) of thyroid nodules according to existing literature.

[0087] The sequence information of the above 14 proteins in the Uniprot protein database is as follows:

[0088] O75347·TBCA_HUMAN:

[0089] MADPRVRQIKIKTGVVKRLVKEKVMYEKEAKQQEEKIEKMRAEDGENYDIKKQAEILQESRMMIPDCQRRLEAAYLDLQRILENEKDLEEAEEYKEARLVLDSVKLEA

[0090] P02765 FETUA_HUMAN:

[0091] MKSLVLLLCLAQLWGCHSAPHGPGLIYRQPNCDDPETEEAALVAIDYINQNLPWGYKHTLNQIDEVKVWPQQPSGELFEIEIDTLETTCHVLDPTPVARCSVRQLKEHAVEGDCDFQLLKLDGKFSVVYAKCDSSPDSAEDVRKVCQDCPLLAPLNDTRVVHAAKAALAAFNA QNNGSNFQLEEISRAQLVPLPPSTYVEFTVSGTDCVAKEATEAAKCNLLAEKQYGFFCKATLSEKLGGAEVAVTCMVFQTQPVSSQPQPEGANEAVPTPVVDPPDAPPSPPLGAGPLPAGSPPDSHVLLAAPPGHQLHRAHYDLRHTFMGVVSLGSPSGEVSHPRKTRTVVQPS

[0092] VGAAAGPVVPPCPG

[0093] RIRHFKV

[0094] P04083·ANXA1_HUMAN:

[0095] MAMVSEFLKQAWFIENEEQEYVQTVKSSKGGPGSAVSPYPTFNPSSDVAALHKAIMVKGVDEATIIDILITKRNNAQRQQIKAAYLQETGKPLDETLKALTGHLEEVVLALLKTPAQFDADELRAAMKGLGTDETLKKALTGHLEEVVLALLKTPAQFDADELRAAMKGLGTDETLLIEILASRTNKEIRDINRVYREELKRDLAKDITSDTSGDFRNALLSLAKGDRSEDFGVNEDLADSDARALYEAGERRKGTDVNVFNTILTTRSYPQLRRVFQKYTKYSKHDMNKVLDLELKDIEKCLTAIVKCATSKPAFFAEKLHQAMKGVGTRHKALIRIMVSRSEIDMNDIKAFYQKMYGISLCQAILDETKGDYEKILVALCGGNP04216·THY1_HUMAN:

[0096] MNLAISIALLLTVLQVSRGQKVTSLTACLVDQSLRLDCRHENTSSSPIQYEFSLTRETKKHVLFGTVGVPEHTYRSRTNFTSKYNMKVLYLSAFTSKDEGTYTCALHHSGHSPPISSQNVTVLRDKLVKCEGISLLAQNTSSWQLLATLLLLDLDLMSLL

[0097] P10909·CLUS_HUMAN:

[0098] MMKTLLLFVGLLLTWESGQVLGDQTVSDNELQEMSNQGSKYVNKEIQNAVNGVKQIKTLIEKTNEERKTLSNLEEAKKKKDALN ETRESETKLKELPGVCNETMMALWEECKPCLKQTCMKFYARVCRSGSGLVGRQLEEFLNQSSPFYFWMNGDRIDSLLENDRQQTHML DVMQDHFSRASSIIDELFQDRFFTREPQDTYHYLPFSLPHRRPHFFFPKSRIVRSLMPFSPYEPLNFHAMFQPFLEMIHEAQQAMDIHFHSPAFQHPPTEFIREGDDDRTVCREIRHNSTGCLRMKDQCDKCREILSVDCSTNNPSQAKLRRELDESLQNEERLKTRKLL

[0099] YQWKMLNTSSLLE

[0100] QLNEQFNWVSRLANLTQGEDQYLRVTTVASHTCDSDVPSGVTEVVVKLFDSDPITTVPVEVSRKNPKFMETVAEKALQEYRKKHREE

[0101] P16403·H12_HUMAN:

[0102] MSETAPAAAPAAAPPAEKAPVKKKAAKKAGGTPRKASGPPVSELITKAVAASKERSGVSLAALKKALAAAGYDVEKNNSRIKLGLKLSLVSKGTLVQTKGTGASGSFKLNKKAASGEAKPKVKKAGGTKPKKPVGAAKKPKKAAGGATPKKSAKKTPKKSAKKTPKKKAKKKPAAATVTKKVAKSPKKAKVAKPKKAAKSAAKAVKPKAAKPKVKAVKPKKAAKPKAAKSAAKAVKPKAAKPKVVKPKKAAKPKKAAK

[0103] P17931·LEG3_HUMAN:

[0104] MADNFSLHDALSGSGNPNPQGWPGAWGNQPAGAGGYPGASYPGAYPGQAPPGAYPGQAPPGAYPGAPGAYPGAPAPGVYPGPPSGPGAYPSSGQPSATGAYPATGPYGAPAGPLIVPYNLPLPGGVVPRMLITILGTVKPNANRIALDFQRGNDVAFHFNPRFNENNRRVIVCNTKLDNNWGREERQSVFPFESGKPFKIQVLVEPDHFKVANDAHLLQYNHRVKKLNEISKLGISGDIDLSASYTMI

[0105] P26038·MUST_HUMAN:

[0106] MPKTISVRVTTMDAELEFAIQPNTTGKQLFDQVVKTIGLREVWFFGLQYQDTKGFSTWLKLNKKVTAQDVRKESPLLFKFRAKFYPEDVSEELIQDITQRLFFLQVKEGILNDDIYCPPETAVLLASYAVQSKYGDFNKEVHKSGYLAGDKLLPQRVLEQHKLNKDQWEERI QVWHEEHRGMLREDAVLEYLKIAQDLEMYGVNYFSIKNKKGSELWLGVDALGLNIYEQNDRLTPKIGFPWSEIRNISFNDKKFVIKPIDKKAPDFVFYAPRLRINKRILALCMGNHELYMRRRKPDTIEVQQMKAQAREEKHQKQMERAMLENEKKKREMAEKEKIKEREK

[0107] EELMERLKQIEEQTKK

[0108] AQQELEEQTRRALELEQERKRAQSEAEKLAKERQEAEEAKEALLQASRDQKKTQEQLALEMAELTARISQLEMARQKKESEAVEWQQKAQMVQEDLEKTRAELKTAMSTPHVAEPAENEQDEQDENGAEASADLRADAMAKD

[0109] RSEEERTTEAEKNERVQKHLKALTSELANARDESKKTA

[0110] NDMIHAENMRLGRDKYKTLRQIRQGNTKQRIDEFESM

[0111] P27797·CALR_HUMAN:

[0112] MLLSVPLLLGLLGLAVAEPAVYFKEQFLDGDGWTSRWIESKHKSDFGKFVLSSGKFYGDEEKDKGLQTSQDARFYALSASFEPFSNKGQTLVVQFTVKHEQNIDCGGGYVKLFPNSLDQTDMHGDSEYNIMFGPDICGPGTKKVHVIFNYKGKNVLINKDIRCKDDEFTHLYTLIVRPDNTYEVKIDNSQVESGSLEDDWDFLPPKKIKDPDASKPEDWDERAKIDDPTDSKPEDWDKPEHIPDPDAKKPEDWDEEMDGEWEPPVIQNPEYKGEWKPRQIDNPDYKGTWIHPEIDNPEYSPDPSIYAYDNFGVLGLDLWQVKSGTIFDNFLITNDEAYAEEFGNETWGVTKAAEKQMKDKQDEEQRLKEEEEDKKRKEEEEAEDKEDDEDKDEDEEDEEDKEEDEEEDVPGQAKDEL

[0113] P30086·PEBP1_HUMAN:

[0114] MPVDLSKWSGPLSLQEVDEQPQHPLHVTYAGAAVDELGKVLTPTQVKNRPTSISWDGLDSGKLYTLVLTDPDAPSRKDPKYREWHHFLVVNMK GNDISSGTVLSDYVGSGPPKGTGLHRYVWLVYEQDRPLKCDEPILSNRSGDHRGKFKVASFRKKYELRAPVAGTCYQAEWDDYVPKLYEQLSGK

[0115] P35579·MYH9_HUMAN:

[0116] MQQAADKYVDKNFINNPLAQADWAAKKLVWVPSDKSGFEPASLKEEVGEEAIVELVENGKKVKVNKDDIQKMNPPKFSKVEDM AELTCLNEASWLHNLKERYYSGLIYTYSGLFCVVINPYKNLPIYSEEIVEMYKGKKRHEMPPHIAITDTAYRSMMQDREDQSILCT GESGAGKTENTKKVIQYLAYVASSHKSKKDQGELERQLLQLLQLLQLLQNPILEAFGNAKTVKNDNSSRFGKFIRINFDVNGYIVGANIETYLLEXRAIRQAKEERTFHIFYLLSGAGEHLKTDLLLEPYNKYRFLSNGHVTIPGQQDKDMFQMRIMEQMEQMREQMLGLL

[0117] QLGNIWFKKERNT

[0118] DQASMPDNTAAQKVSHLLGINVTDFTRGILTPRIKVGRDYVQKAQTKEQADFAIEALAKATYERMFRWLVLRINKALDKTKRQGASFIGILDIAGFEIFDLNSFEQLCINYTNEKLQQLFNHTMFILEQEEYQREGIEWNFIDFGLDLQ

[0119] PCIDLIEKPAPPGILALLDEECWFPKATDK

[0120] SFVEKVMQEQGTHPKFQKPKQLKDKADFCIIHYAGKVDYKADEWLMKNMDPLNDNIATLLHQSSDKFVSELWKDVDRIIGLDQVAGMSETALPGAFKTRKGMFRTVGQLYKEQLAKLMATLRNTNPNFVRCIIPNHEKKAGKLDPHLVLDQLRCNGVLEGIRICRQGFPNRVVFQEFRQRYEILTPNSIPKGFMDGKQACVLMIKALELDSNLYRIGQSKVFFRAGVLAHLEEERDLKITDVIIGFQACCRGYLARKAFAKRQQQLTAMKVLQRNCAAYLKLRNWQWWRLFTKVKPLLQVSRQEEEMMAKEEELVKVREKQLAAENRLTEMETLQSQLMA

[0121] EKLQLQEQLQAETELCAEAE

[0122] ELRARLTAKKQELEEICHDLEARVEEEEERCQHLQAEKKKMQQNIQELEEQLEEEESARQKLQLEKVTTEAKLKKLEEEQIILEDQNCKLAKEKKLLEDRIAEFTTNLTEEEEKSKSLAKLKNKHEAMITDLEERLRREEKQRQELEKT

[0123] RRKLEGDSTDLSDQIAELQAQIAELKMQLAK

[0124] KEEELQAALARVEEEAAQKNMALKKIRELESQISELQEDLESERASRNKAEKQKRDLGEELEALKTELEDTLDSTAAQQELRSKREQEVNILKKTLEEEAKTHEAQIQEMRQKHSQAVEELAEQLEQTKRVKANLEKAKQTLENER

[0125] GELANEVKVLLQGKGDSEHKRKKVEAQLQELQVK

[0126] FNEGERVRTELADKVTKLQVELDNVTGLLSQSDSKSSKLTKDFSALESQLQDTQELLQEENRQKLSLSTKLKQVEDEKNSFREQLEEEEEAKHNLEKQIATLHAQVADMKKKMEDSVGCLETAEEVKRKLQKDLEGLSQRHEEKV

[0127] AAYDKLEKTKTRLQQELDDLLVDLDHQRQSACNLE

[0128] KKQKKFDQLLAEEKTISAKYAEERDRAEAEAREKETKALSLARALEEAMEQKAELERLNKQFRTEMEDLMSSKDDVGKSVHELEKSKRALEQQVEEMKTQLEELEDELQATEDAKLRLEVNLQAMKAQFERDLQGRDEQSEEKK

[0129] KQLVRQVREMEAELEDERKQRSMAVAARKKLEMDLK

[0130] DLEAHIDSANKNRDEAIKQLRKLQAQMKDCMRELDDTRASREEILAQAKENEKKLKSMEAEMIQLQEELAAAERAKRQAQQERDELADEIANSSGKGALALEEKRRLEARIAQLEEELEEEQGNTELINDRLKKANLQIDQINTDL

[0131] NLERSHAQKNENARQQLERQNKELKVKLQEMEGT

[0132] VKSKYKASITALEAKIAQLEEQLDNETKERQAACKQVRRTEKKLKDVLLQVDDERRNAEQYKDQADKASTRLKQLKRQLEEAEEEAQRANASRRKLQRELEDATETADAMNREVSSLKNKLRRGDLPFVVPRRMARKGAGDGSDEEVDGKADGAEAKPAE

[0133] P98164·LRP2_HUMAN:

[0134] MDRGPAAVACTLLLALVACLAPASGQECDSAHFRCGSGHCIPADWR

[0135] CDGTKDCSDDADEIGCAVVTCQQGYFKCQSEGQCIPNSWVCDQDQDCD

[0136] DGSDERQDCSQSTCSSHQITCSNGQCIPSEYRCDHVRDCPDGADENDCQY

[0137] PTCEQLTCDNGACYNTSQKCDWKVDCRDSSDEINCTEICLHNEFSCGNGE

[0138] CIPRAYVCDHDNDCQDGSDEHACNYPTCGGYQFTCPSGRCIYQNWVCDG

[0139] EDDCKDNGDEDGCESGPHDVHKCSPREWSCPESGRCISIYKVCDGILDCP

[0140] CREDENNTSTGKYCSMTLCSALNCQYQCHETPYGGACFCPPGYIINHNDS

[0141] RTCVEFDDCQIWGICDQKCESRPGRHLCHCEEGYILERGQYCKANDSFGE

[0142] ASIIFSNGRDLLIGDIHGRSFRILVESQNRGVAVGVAFHYHLQRVFWTDTV

[0143] QNKVFSVDINGLNIQEVLNVSVETPENLAVDWVNNKIYLVETKVNRIDM

[0144] VNLDGSYRVTLITENLGHPRGIAVDPTVGYLFFSDWESLSGEPKLERAFM

[0145] DGSNRKDLVKTKLGWPAGVTLDMISKRVYWVDSRFDYIETVTYDGIQRK

[0146] TVVHGGSLIPHPFGVSLFEGQVFFTDWTKMAVLKANKFTETNPQVYYQA

[0147] SLRPYGVTVYHSLRQPYATNPCKDNNGGCEQVCVLSHRTDNDGLGFRCK

[0148] CTFGFQLDTDERHCIAVQNFLIFSSQVAIRGIPFTLSTQEDVMVPVSGNPSF

[0149] FVGIDFDAQDSTIFFSDMSKHMIFKQKIDGTGREILAANRVENVESLAFDW

[0150] ISKNLYWTDSHYKSISVMRLADKTRRTVVQYLNNPRSVVVHPFAGYLFFT

[0151] DWFRPAKIMRAWSDGSHLLPVINTTLGWPNGLAIDWAASRLYWVDAYF

[0152] DKIEHSTFDGLDRRRLGHIEQMTHPFGLAIFGEHLFFTDWRLGAIIRVRKA

[0153] DGGEMTVIRSGIAYILHLKSYDVNIQTGSNACNQPTHPNGDCSHFCFPVPN

[0154] FQRVCGCPYGMRLASNHLTCEGDPTNEPPTEQCGLFSFPCKNGRCVPNYY

[0155] LCDGVDDCHDNSDEQLCGTLNNTCSSSAFTCGHGECIPAHWRCDKRNDC

[0156] VDGSDEHNCPTHAPASCLDTQYTCDNHQCISKNWVCDTDNDCGDGSDE

[0157] KNCNSTETCQPSQFNCPNHRCIDLSFVCDGDKDCVDGSDEVGCVLNCTAS

[0158] QFKCASGDKCIGVTNRCDGVFDCSDNSDEAGCPTRPPGMCHSDEFQCQE

[0159] DGICIPNFWECDGHPDCLYGSDEHNACVPKTCPSSYFHCDNGNCIHRAWL

[0160] CDRDNDCGDMSDEKDCPTQPFRCPSWQWQCLGHNICVNLSVVCDGIFDC

[0161] PNGTDESPLCNGNSCSDFNGGCTHECVQEPFGAKCLCPLGFLLANDSKTC

[0162] EDIDECDILGSCSQHCYNMRGSFRCSCDTGYMLESDGRTCKVTASESLLL

[0163] LVASQNKIIADSVTSQVHNIYSLVENGSYIVAVDFDSISGRIFWSDATQGK

[0164] TWSAFQNGTDRRVVFDSSIILTETIAIDWVGRNLYWTDYALETIEVSKIDG

[0165] SHRTVLISKNLTNPRGLALDPRMNEHLLFWSDWGHHPRIERASMDGSMR

[0166] TVIVQDKIFWPCGLTIDYPNRLLYFMDSYLDYMDFCDYNGHHRRQVIASD

[0167] LIIRHPYALTLFEDSVYWTDRATRRVMRANKWHGGNQSVVMYNIQWPL

[0168] GIVAVHPSKQPNSVNPCAFSRCSHLCLLSSQGPHFYSCVCPSGWSLSPDLL

[0169] NCLRDDQPFLITVRQHIIFGISLNPEVKSNDAMVPIAGIQNGLDVEFDDAEQ

[0170] YIYWVENPGEIHRVKTDGTNRTVFASISMVGPSMNLALDWISRNLYSTNP

[0171] RTQSIEVLTLHGDIRYRKTLIANDGTALGVGFPIGITVDPARGKLYWSDQG

[0172] TDSGVPAKIASANMDGTSVKTLFTGNLEHLECVTLDIEEQKLYWAVTGR

[0173] GVIERGNVDGTDRMILVHQLSHPWGIAVHDSFLYYTDEQYEVIERVDKAT

[0174] GANKIVLRDNVPNLRGLQVYHRRNAAESSNGCSNNMNACQQICLPVPGG

[0175] LFSCACATGFKLNPDNRSCSPYNSFIVVSMLSAIRGFSLELSDHSETMVPV

[0176] AGQGRNALHVDVDVSSGFIYWCDFSSSVASDNAIRRIKPDGSSLMNIVTH

[0177] GIGENGVRGIAVDWVAGNLYFTNAFVSETLIEVLRINTTYRRVLLKVTVD

[0178] MPRHIVVDPKNRYLFWADYGQRPKIERSFLDCTNRTVLVSEGIVTPRGLA

[0179] VDRSDGYVYWVDDSLDIIARIRINGENSEVIRYGSRYPTPYGITVFENSIIW

[0180] VDRNLKKIFQASKEPENTEPPTVIRDNINWLRDVTIFDKQVQPRSPAEVNN

[0181] NPCLENNGGCSHLCFALPGLHTPKCDCAFGTLQSDGKNCAISTENFLIFAL

[0182] SNSLRSLHLDPENHSPPFQTINVERTVMSLDYDSVSDRIYFTQNLASGVGQ

[0183] ISYATLSSGIHTPTVIASGIGTADGIAFDWITRRIYYSDYLNQMINSMAEDG

[0184] SNRTVIARVPKPRAIVLDPCQGYLYWADWDTHAKIERATLGGNFRVPIVN

[0185] SSLVMPSGLTLDYEEDLLYWVDASLQRIERSTLTGVDREVIVNAAVHAFG

[0186] LTLYGQYIYWTDLYTQRIYRANKYDGSGQIAMTTNLLSQPRGINTVVKN

[0187] QKQQCNNPCEQFNGGCSHICAPGPNGAECQCPHEGNWYLANNRKHCIVD

[0188] NGERCGASSFTCSNGRCISEEWKCDNDNDCGDGGSDEMESVCALHTCSPT

[0189] AFTCANGRCVQYSYRCDYYNDCGDGGSDEAGCLFRDCNATTFMCNNRR

[0190] CIPREFICNGVDNCHDNNTSDEKNCPDRTCQSGYTKCHNSNICIPRVYLCD

[0191] GDNDCGDNSDENPTYCTTHTCSSSEFQCASGRCIPQHWYCDQETDCFDAS

[0192] DEPASCGHSERTCLADEFKCDGGRCIPSEWICDGDNDCGDMSDEDKRHQ

[0193] CQNQNCSDSEFLCVNDRPPDRRCIPQSWVCDGDVDCTDGYDENQNCTRR

[0194] TCSENEFTCGYGLCIPKIFRCDRHNDCGDYSDERGCLYQTCQQNQFTCQN

[0195] GRCISKTFVCDEDNDCGDGSDELMHLCHTPEPTCPHEFKCDNGRCIEMM

[0196] KLCNHLDDCLDNSDEKGCGINECHDPSISGCDHNCTDTLTSFYCSCRPGY

[0197] KLMSDKRTCVDIDECTEMPFVCSQKCENVIGSYICKCAPGYLREPDGKTC

[0198] RQNSNIEPYLIFSNRYYLRNLTIDGYFYSLILEGLDNVVALDFDRVEKRLY

[0199] WIDTQRQVIERMFLNKTNKETIINHRLPAAESLAVDWVSRKLYWLDARL

[0200] DGLFVSDLNGGHRRMLAQHCVDANNTFCFDNPRGLALHPQYGYLYWAD

[0201] WGHRAYIGRVGMDGTNKSVIISTKLEWPNGITIDYTNDLLYWADAHLGYI

[0202] EYSDLEGHHRHTVYDGALPHPFAITIFEDTIYWTDWNTRTVEKGNKYDGS

[0203] NRQTLVNTTHRPFDIHVYHPYRQPIVSNPCGTNNGGCSHLCLIKPGGKGFT

[0204] CECPDDFRTLQLSGSTYCMPMCSSTQFLCANNEKCIPIWWKCDGQKDCS

[0205] DGSDELALCPQRFCRLGQFQCSDGNCTSPQTLCNAHQNCPDGSDEDRLLC

[0206] ENHHCDSNEWQCANKRCIPESWQCDTFNDCEDNSDEDSSHCASRTCRPG

[0207] QFRCANGRCIPQAWKCDVDNDCGDHSDEPIEECMSSAHLCDNFTEFSCKT

[0208] NYRCIPKWAVCNGVDDCRDNSDEQGCEERTCHPVGDFRCKNHHCIPLRW

[0209] QCDGQNDCGDNSDEENCAPRECTESEFRCVNQQCIPSRWICDHYNDCGD

[0210] NSDERDCEMRTCHPEYFQCTSGHCVHSELKCDGSADCLDASDEADCPTR

[0211] FPDGAYCQATMFECKNHVCIPPYWKCDGDDDCGDGSDEELHLCLDVPC

[0212] NSPNRFRCDNNRCIYSHEVCNGVDDCGDGTDETEEHCRKPTPKPCTEYEY

[0213] KCGNGHCIPHDNVCDDADDCGDWSDELGCNKGKERTCAENICEQNCTQ

[0214] LNEGGFICSCTAGFETNVFDRTSCLDINECEQFGTCPQHCRNTKGSYECVC

[0215] ADGFTSMSDRPGKRCAAEGSSPLLLLPDNVRIRKYNLSSERFSEYLQDEEY

[0216] IQAVDYDWDPKDIGLSVVYYTVRGEGSRFGAIKRAYIPNFESGRNNLVQE

[0217] VDLKLKYVMQPDGIAVDWVGRHIYWSDVKNKRIEVAKLDGRYRKWLIS

[0218] TDLDQPAAIAVNPKLGLMFWTDWGKEPKIESAWMNGEDRNILVFEDLG

[0219] WPTGLSIDYLNNDRIYWSDFKEDVIETIKYDGTDRRVIAKEAMNPYSLDIF

[0220] EDQLYWISKEKGEVWKQNKFGQGKKEKTLVVNPWLTQVRIFHQLRYNK

[0221] SVPNLCKQICSHLCLLRPGGYSCACPQGSSFIEGSTTECDAAIELPINLPPPC

[0222] RCMHGGNCYFDETDLPKCKCPSGYTGKYCEMAFSKGISPGTTAVAVLLTI

[0223] LLIVVIGALAIAGFFHYRRTGSLLPALPKLPSLSSLVKPSENGNGVTFRSGA

[0224] DLNMDIGVSGFGPETAIDRSMAMSEDFVMEMGKQPIIFENPMYSARDSAV

[0225] KVVQPIQVTVSENVDNKNYGSPINPSEIVPETNPTSPAADGTQVTKWNLF

[0226] KRKSKQTTNFENPIYAQMENEQKESVAATPPPSPSLPAKPKPPSRRDPTPT

[0227] YSATEDTFKDTANLVKEDSEV

[0228] Q9H299·SH3L3_HUMAN:

[0229] MSGLRVYSTSVTGSREIKSQQSEVTRILDGKRIQYQLVDISQDNALRD EMRALAGNPKATPPQIVNGDQYCGDYELFVEAVEQNTLQEFLKLA

[0230] Q9HAT2·SIAE_HUMAN:

[0231] MVAPGLVLGLVLPLILWADRRSAGIGFRFASYINNDMVLQKEPAGAVIWGFGTPGATVTVTLRQGQETIMKKVTSVKAHSDTWMVVLDPMKPGGPFEVMAQQTLEKINFTLRVHDVLFGDVWLCSGQSNMQ MTVLQIFNATRELSNTAAYQSVRILSVSPIQAEQELEDLVAVDLQWSKPTSENLGHGYFKYMSAVCWLFGRHLYDTLQYPIGLIASSWGGTPIEAWSSGRSLKACGVPKQGSIPYDSVTGPSKHSVLWNAM IHPLCNMTLKGVVWYQGESNINYNTDLYNCTFPALIEDWRETFHRGSQGQTERFFPFGLVQLSSDLSKKSSDDGFPQIRWHQTADFGYVPNPKMPNTFMAVAMDLCDRDSPFGSIHPRDKQTVAYRLHLGA RALAYGEKNLTFEGPLPEKIELLAHKGLLNLTYYQQIQVQKKDNKIFEISCCSDHRCKWLPASMNTVSTQSLTLAIDSCHGTVVALRYAWTTWPCEYKQCPLYHPSSALPAPPFIAFITDQGPGHQSNVAK

[0232] The above proteins are only presented as examples, and the protein sequences are not guaranteed to be complete. Those skilled in the art can use other types of protein sequences or other numbers of protein sequences in the same way, and this application does not impose any restrictions.

[0233] Step 5, inputting the protein sequence data into a protein sequence data classification model (hereinafter referred to as protein model);

[0234] First, use one-hot encoding to encode the amino acid sequences of 14 key proteins, and then concatenate the encoded matrices to generate an n*20 matrix, where n is the total number of amino acids in the 14 proteins, and 20 means there are 20 types of amino acids that make up proteins. One-hot encoding, also known as one-bit effective encoding, mainly uses an N-bit state register to encode N states. Each state has an independent register bit, and only one bit is effective at any time, such as Figure 15 shown.

[0235] The spliced data can be input into two sub-models respectively. The structure of the first sub-model is the same as that of the image model. It uses the same CNN module, including the residual neural network model resnet50 and the attention model CBAM. The CBAM module is added after the first layer module and the fifth layer module of resnet50, and finally the second fully connected layer is used for dimensionality reduction. The second sub-module is implemented using the ViT (Vision Transformer) model. The ViT model is the first model to introduce transformers into the field of CV (computer vision). It mainly relies on self-attention (self-attention mechanism) to model global context. It has received widespread attention since it was proposed. The ViT model structure is as follows: Figure 16 As shown in the figure, the ViT model divides the input into multiple patches (small blocks), then projects each patch into a fixed-length vector and feeds it into the Transformer. The subsequent encoder operations are the same as in the original Transformer. The first sub-model (CNN model) focuses more on local information, while the second sub-model (ViT model) focuses on global information, can model long-range dependencies, and retain more spatial information.

[0236] Step 6: Use cross-validation method to train protein model

[0237] In this example, the cross-validation method is used to train the first sub-model and the second sub-model respectively.

[0238] In other embodiments, the protein model may include only the first sub-model or only the second sub-model.

[0239] The loss function used can be found in the formula in step 3 above.

[0240] The above steps 4-6 are the training method of the protein model.

[0241] There is no order of precedence between the training of the imaging model and the training of the protein model.

[0242] Step 7: Train the entire classification model.

[0243] In this example, the classification model is used to predict the property category of a nodule in the thyroid region, for example, whether the thyroid nodule is a malignant tumor or not.

[0244] Among the malignant signs of thyroid nodules, the more specific ones are: microcalcification, irregular edges, and aspect ratio > 1; other malignant signs include: solid hypoechoic nodules, absence of halo, extrathyroidal invasion, and abnormal ultrasound signs of cervical lymph nodes. Abnormal signs of cervical lymph nodes mainly include: microcalcification, cystic changes, high echoes, and peripheral blood flow inside the lymph nodes. In addition, they also include round lymph nodes, irregular or blurred borders, uneven internal echoes, disappearance of lymph nodes, or unclear corticomedullary boundaries. These can all be identified through imaging data.

[0245] Use the trained image model to extract features from the image data to obtain the first feature f1, and use the trained protein model to extract features from the protein sequence data. Since the protein model includes two sub-models, two features are obtained after feature extraction: the second feature f21 and the third feature f22. The second feature f21 and the third feature f22 are concatenated to obtain the fourth feature f2. Fix the previous feature extraction module, concatenate the first feature f1 and the fourth feature f2 and input them into the last two fully connected layers (the third fully connected layer and the fourth fully connected layer) for training the last two layers to obtain the final classification model.

[0246] In other embodiments, three fully connected layers (a third fully connected layer, a fourth fully connected layer, and a fifth fully connected layer) may also be set, and the training method may still adopt the cross-validation method.

[0247] During training, the classification model is trained using the cross-validation method.

[0248] The loss function is the cross entropy loss function L c :

[0249]

[0250] is the benchmark value, indicating whether sample i belongs to category C or not. is the prediction result, , represents the predicted probability that sample i belongs to category C, m is the number of samples, C is the category, w c is the weight of the category.

[0251] After calculating the loss value of the classification model, the parameters of the classification model are updated based on the loss value, and the model that meets the training end conditions is used as the final classification model. The training end conditions include the classification model convergence or reaching a preset number of updates. The classification model convergence means that the loss value is less than or equal to the preset loss value, or the loss value no longer decreases. The preset number of updates can be set according to actual needs.

[0252] This embodiment proposes a method for distinguishing benign and malignant thyroid nodules based on imaging data and proteomics data. Using thyroid ultrasound data as image input, a CNN network is used in combination with spatial attention and channel attention to extract features from the ultrasound data. Using 14 key proteins as protein input, the protein data is encoded and spliced, and protein features are extracted in an image-based manner using CNN and Transformer combined with channel attention. By combining imaging data and protein data to distinguish benign and malignant thyroid nodules, the input information is increased to improve the accuracy of the distinction.

[0253] In order to obtain a better discrimination effect, in the above step 1, after obtaining the ultrasound image data of the area to be examined, the image can also be segmented. In other embodiments, if the image is an MRI image or a CT image, the following segmentation method is also applicable.

[0254] The classification model training process of the segmentation model is added as follows Figure 17 In this example, the segmentation model uses a U-shape network structure. The U-shaped network uses the underlying features to improve the lack of upsampling information and combines low-resolution information with high-resolution information to achieve better results in medical image segmentation. Figure 17 As shown in Figure 2, the U-shape network structure includes a downsampling module (Encoder) and an upsampling module (Decoder). The output of the Decoder module is the segmentation result. After obtaining the segmentation result, the boundary direction histogram method and the Fourier shape descriptor method are used to describe the contour features to obtain a more optimized edge segmentation effect, thereby better segmenting nodules, tumors, or blood vessels.

[0255] The contour is a curve composed of a series of connected points, which represents the basic shape of an object. Methods for constructing contour features include boundary feature method and Fourier shape descriptor method. The boundary feature method obtains the shape parameters of the image by describing the boundary features. Classic methods include Hough transform detection of parallel lines and boundary direction histogram method. Among them, the boundary direction histogram method first uses the differential image to obtain the image edge, and then makes a histogram of the edge size and direction. The usual method is to construct the image grayscale gradient direction matrix. The Fourier shape descriptor method uses the Fourier transform of the object boundary as a property description, and uses the closedness and periodicity of the region boundary to convert the two-dimensional problem into a one-dimensional problem.

[0256] In this example, when using the boundary direction histogram method to describe boundary features, the edges of the image must first be extracted and the direction of each edge point calculated. The number of edge points in each direction is then counted to form an edge direction histogram. The Sobel operator is used to perform edge detection on the segmentation result to obtain the dx and dy of the edge operator points. The edge direction of each pixel is calculated as θ(x,y) = argtg(dx / dy). The edge direction values are quantized at intervals of 10 degrees (in other examples, the intervals can also be set to other degrees). The quantized edge direction values are histogram-normalized to obtain Dp. The same operation is performed on Mask (the true value of the segmentation result) to obtain the normalized boundary direction histogram Dm.

[0257] In addition to the boundary direction histogram, the Fourier shape descriptor can also well describe the contour features. After a simple normalization operation on the Fourier descriptor, the descriptor can be made invariant to translation, rotation, and scale, that is, it is not affected by the position, angle, and scaling of the contour in the image. It is a robust image feature. The boundary points (x0, y0), (x1, y1), ..., (xk-1, yk-1) can be expressed as x(k) = x k , y(k)=y k , so the boundary can be represented as a coordinate sequence s(k) = [x(k), y(k), k = 0, 1, 2, ..., K-1]. Furthermore, each coordinate can be treated as a complex number, resulting in: s(k) = x(k) + jy(k), where k = 0, 1, 2, ..., K-1. The x-axis is the real axis of the complex sequence, and the y-axis is the imaginary axis of the complex sequence. This representation transforms the two-dimensional problem into a one-dimensional problem. The discrete Fourier transform (DFT) of s(k) is:

[0258]

[0259] Where u = 0, 1,…, K-1, S(u) is called the Fourier descriptor of the boundary.

[0260] In other embodiments, the contour features may be described using only the boundary direction histogram method or only the Fourier shape descriptor. The difference lies in the slight difference in the subsequent calculation of the loss function.

[0261] In this example, the loss function for the segmentation model is:

[0262] L s =L1+βL2

[0263] Among them, β is the weight of L2,

[0264]

[0265] L1 is the cross entropy loss function, is the base value of the pixel, indicating whether the sample i (here represented as a pixel) belongs to category C or not. is the prediction result, which indicates the predicted probability that sample i belongs to category C, m is the number of pixels, C is the category, and w c is the weight of the category.

[0266]

[0267] Among them, J is the number of intervals of the boundary direction histogram, dm is the normalized value of the boundary direction histogram of the mask image, dp is the normalized value of the boundary direction histogram of the prediction result, Sm is the discrete Fourier transform of the boundary point in the mask image, Sp is the discrete Fourier transform of the boundary point in the prediction result, and K is the total number of boundary points.

[0268] Based on the contour features of the image, the information of boundary feature description and Fourier shape descriptor is used to increase the attention to edges during model training, thereby improving the segmentation performance at the edges of the model.

[0269] If only the boundary direction histogram is used to describe the contour features, then:

[0270]

[0271] If only Fourier shape descriptor is used to describe the contour features, then:

[0272]

[0273] After adding the segmentation module, multi-task joint learning can be used in this example, such as Figure 17 As shown in the figure, the segmentation model and the classification model are trained together to obtain better robustness. The loss function of the classification model is:

[0274] L=L s +L c

[0275] L c is the cross entropy loss function for the classification model.

[0276] In other embodiments, the segmentation model obtained by joint learning can be used alone to perform image segmentation.

[0277] In an exemplary embodiment, the present disclosure further provides an electronic device, which may include a processor and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, it implements the classification model generation method or classification method in any of the above embodiments of the present disclosure.

[0278] In an exemplary embodiment, Figure 18 FIG. 1 is a schematic diagram of the structure of an electronic device in an embodiment of the present disclosure. Figure 18 As shown, the device 60 includes: at least one processor 601; and at least one memory 602 and a bus 603 connected to the processor 601; wherein the processor 601 and the memory 602 communicate with each other through the bus 603; the processor 601 is used to call the program instructions in the memory 602 to execute the steps of the classification model generation method or classification method in any of the above embodiments.

[0279] The processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a transistor logic device, etc., and this disclosure does not limit this.

[0280] The memory may include read-only memory (ROM) and random access memory (RAM), and provides instructions and data to the processor. Some of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0281] In addition to the data bus, the bus can also include the power bus, control bus and status signal bus. However, for the sake of clarity, Figure 18 In the text, various buses are marked as buses.

[0282] During implementation, the processing performed by the processing device can be completed by hardware integrated logic circuits in the processor or by instructions in the form of software. That is, the method steps of the embodiments of the present disclosure can be embodied as being executed by a hardware processor, or by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0283] In an exemplary embodiment, the present disclosure also provides a non-volatile computer-readable storage medium on which a computer program that can be run on a processor is stored. When the computer program is executed by the processor, the steps of the aforementioned smart branch application management method are implemented.

[0284] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the functional modules / units mentioned in the above description are not divided equally; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0285] Although the embodiments disclosed in this disclosure are as described above, the contents described are merely embodiments adopted to facilitate understanding of the disclosure and are not intended to limit the disclosure. Any person skilled in the art to which the disclosure belongs may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope of the disclosure. However, the scope of patent protection of the disclosure shall still be based on the scope defined by the attached claims.

Claims

1. A classification model generation method, characterized in that: include: For a region to be inspected of a sample object, an image data sample of the region to be inspected is obtained, and a protein sequence data sample of each sample object is obtained to construct a training sample; the training sample includes a plurality of sample groups, each of which includes an image data sample and a protein sequence data sample; Taking the training samples as input, the image data classification model and the protein sequence classification model are trained respectively, and then the entire preset classification model is trained to obtain a classification model, which is used to predict the property category of the area to be inspected of the sample object; wherein, the preset classification model includes an image data classification model for classifying image data samples, and a protein sequence classification model for classifying protein sequence data samples, and the output of the image data classification model and the output of the protein sequence classification model are feature-concatenated and output as a classification result through a fully connected layer.

2. The method according to claim 1, characterized in that The image data classification model includes a first residual neural network, a first CBAM and a first fully connected layer, wherein the first residual neural network includes five layers of modules connected in sequence, the image data sample is input into the first layer module of the first residual neural network, the first CBAM is located after the first layer module and after the fifth layer module, respectively, the output of the first CBAM after the fifth layer module is used as the input of the first fully connected layer, and the output of the first fully connected layer is the output of the image data classification model.

3. The method according to claim 1, characterized in that The protein sequence classification model includes an encoding splicing module, a second residual neural network, a second CBAM and a second fully connected layer, wherein the second residual neural network includes five layers of modules connected in sequence, the protein sequence data sample is input into the first layer module of the second residual neural network after encoding and splicing, the second CBAM is located after the first layer module and the fifth layer module respectively, the output of the second CBAM after the fifth layer module is used as the input of the second fully connected layer, and the output of the second fully connected layer is the output of the protein sequence classification model.

4. The method according to claim 3, characterized in that The protein sequence classification model also includes: a visual converter and a third fully connected layer. The protein sequence data samples are encoded and spliced and then input into the visual converter. The output of the visual converter serves as the input of the third fully connected layer.

5. The method according to any one of claims 1 to 4, characterized in that After obtaining the image data samples of the area to be inspected, the method further includes segmenting the image data samples using a preset segmentation model; and training the preset segmentation model to obtain a segmentation model.

6. The method according to claim 5, characterized in that The training of the preset segmentation model includes: using the boundary direction histogram method and the Fourier shape descriptor method to extract contour features of the segmentation results to obtain histogram eigenvalues and Fourier eigenvalues respectively, determining the loss value of the segmentation model based on the baseline value of the boundary direction histogram method and the baseline value of the Fourier shape descriptor method, updating the parameters of the segmentation model based on the loss value, and using the model that meets the training end condition as the segmentation model, and the training end condition is that the segmentation model converges or reaches a preset number of updates.

7. The method according to claim 5, characterized in that The training of the preset segmentation model to obtain the segmentation model includes: jointly training the preset classification model and the preset segmentation model.

8. A classification method, characterized in that include: Acquiring image data of the area to be examined of the object to be tested, and acquiring protein sequence data; The image data and protein sequence data are input into a classification model, wherein the classification model is obtained according to the classification model generation method according to any one of claims 1 to 7, and the property category of the area to be checked is determined based on the output of the classification model.

9. The method according to claim 8, characterized in that After acquiring the image data, the method further includes segmenting the image data samples using a preset segmentation model, and using the segmentation results as the image data of the area to be inspected.

10. A classification model generation device, characterized in that: include: A feature acquisition and sample construction module is used to obtain image data samples of the area to be examined for the sample object, and obtain protein sequence data samples of each sample object to construct training samples; the training samples include multiple sample groups, each of which includes image data samples and protein sequence data samples; A model training module is used to take the training samples as input, train the image data classification model and the protein sequence classification model respectively, and then train the entire preset classification model to obtain a classification model. The classification model is used to predict the property category of the area to be inspected of the sample object, wherein the preset classification model includes an image data classification model for classifying image data samples, and a protein sequence classification model for classifying protein sequence data samples. The output of the image data classification model and the output of the protein sequence classification model are feature-concatenated and output as a classification result through a fully connected layer.

11. A classification device, characterized in that: include: A feature acquisition module is used to obtain image data of the area to be examined of the object to be tested and to obtain protein sequence data; A classification module is used to input the image data and protein sequence data into a classification model, wherein the classification model is obtained according to the classification model generation method according to any one of claims 1 to 7, and based on the output of the classification model, determine the property category of the area to be checked.

12. An electronic device comprising a processor and a memory storing a computer program executable on the processor, wherein: When the processor executes the program, the classification model according to any one of claims 1 to 7 is implemented. A type generation method, or a classification method as described in any one of claims 8 to 9.

13. A computer-readable storage medium, characterized in that A computer program that can be run on a processor is stored thereon, and when the computer program is executed by the processor, it implements the classification model generation method as described in any one of claims 1 to 7, or implements the classification method as described in any one of claims 8 to 9.

Images