Pathology classification model training method, pathology classification method and electronic device

Abstract

The embodiments of the present invention provide a pathology classification model training method, a pathology classification method and an electronic device, which relate to the field of medical image technology. By acquiring sample data including ultrasound contrast data, blood index data and pathology diagnosis labels, the ultrasound contrast data is processed to obtain contrast index data and each ultrasound contrast image, and the ultrasound contrast image, the contrast index data and the blood index data are encoded respectively, and spliced ​​and fused to obtain a first multimodal feature map, the first multimodal feature map is input into an adaptive feature extraction module for processing to obtain a second multimodal feature map, and the second multimodal feature map is input into a classification layer to obtain the predicted pathology type of the case, and based on the predicted pathology type and the pathology diagnosis label, the loss information is calculated, and iterative training is performed until the preset conditions are met. In this way, the accuracy of pathology classification can be improved by combining different feature responses on ultrasound contrast data and blood index data.

Description

Technical Field

[0001] The present invention relates to the field of medical image technology, and in particular to a pathology classification model training method, a pathology classification method and an electronic device. Background Art

[0002] Ovarian tumors are common reproductive system diseases in women and can occur at any age. Based on biological behavior and pathological morphology, they can be divided into three categories: benign, borderline, and malignant.

[0003] Tumor markers are substances that exist in malignant tumor cells, are produced by malignant tumors, or are produced by the host in response to stimulation of malignant tumors. They can be detected in the patient's tumor tissue, excrement or body fluids, and are often used in clinical practice for differential diagnosis of malignant tumors and judgment of treatment response. Currently, the tumor markers known to be related to ovarian tumors and widely used include carbohydrate antigen CA125, human epididymis protein HE4, and carcinoembryonic antigen CEA. Since other malignant tumors such as pancreatic cancer, lung cancer, and gastric cancer also produce the same markers, the accuracy of diagnosing ovarian tumors using tumor markers alone is low. Summary of the invention

[0004] In view of this, an object of an embodiment of the present invention is to provide a pathology classification model training method, a pathology classification method and an electronic device to at least partially improve the above-mentioned problem.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a pathology classification model training method, comprising:

[0007] Acquire sample data of each case, each of the sample data includes ultrasound contrast data, blood index data and pathological diagnosis labels;

[0008] Extracting contrast index data of each of the ultrasound contrast imaging data and ultrasound contrast images of the ultrasound contrast imaging data when the contrast agent reaches a peak, and adding each of the contrast index data and each of the ultrasound contrast images to the corresponding sample data;

[0009] Input each of the sample data into a pathology classification model, encode the ultrasound contrast image, the contrast index data, and the blood index data, respectively, and obtain a contrast image feature map, a contrast index feature map, and a blood index feature map, respectively;

[0010] The angiography image feature map, the angiography index feature map and the blood index feature map are spliced ​​and fused to obtain a first multimodal feature map;

[0011] Inputting the first multimodal feature map into an adaptive feature extraction module for processing to obtain a second multimodal feature map;

[0012] Inputting the second multimodal feature map into the classification layer to obtain the predicted pathological type of the case, including: using full connection to perform feature conversion on the second multimodal feature map to obtain a multimodal feature vector; using a SoftMax function to process the multimodal feature vector to obtain three predicted probability values; the three predicted probability values ​​respectively represent the probability values ​​of the case being benign, borderline, and malignant; and taking the largest value of the three predicted probability values ​​as the predicted pathological type of the case;

[0013] Based on the predicted pathology type and the pathology diagnosis label, loss information is calculated, and the pathology classification model is iteratively trained according to the loss information until a preset condition is met.

[0014] Optionally, the contrast index data and the blood index data each include at least one characteristic index. Before the step of respectively encoding the ultrasound contrast image, the contrast index data and the blood index data to respectively obtain a contrast image characteristic graph, a contrast index characteristic graph and a blood index characteristic graph, the method further includes:

[0015] Performing information gain calculation on the angiography index data and the blood index data in each of the sample data, eliminating the characteristic indexes whose correlation is less than a first preset threshold, and obtaining processed sample data;

[0016] The importance of the angiography index data and the blood index data in each processed sample data is evaluated, and the characteristic indexes whose importance is less than a second preset threshold are eliminated to obtain the processed sample data.

[0017] Optionally, the information gain calculation is performed on the angiography index data and the blood index data in each of the sample data, and the characteristic indexes whose correlation is less than a first preset threshold are eliminated to obtain the processed sample data, including:

[0018] Put all the feature indicators in each sample data into a total feature set;

[0019] Eliminate any of the feature indicators to obtain the remaining feature subset, and calculate the information gain value of the remaining feature subset; the calculation formula of the information gain value is:

[0020]

[0021]

[0022]

[0023] in, is the remaining feature subset, is the information gain value of the remaining feature subset, m is the number of cases, For the i of the cases described n Features, is the pathological diagnosis label of the sample data, , 0 means benign, 1 means borderline, and 2 means malignant;

[0024] Eliminate the feature indicators whose information gain values ​​are less than a first preset threshold to obtain processed sample data;

[0025] The formula for evaluating the importance of the angiography index data and the blood index data in the processed sample data is:

[0026]

[0027] in, represents the characteristic index to be evaluated, represents the total set of feature indicators, express The characteristic dimension of Indicates except Any subset of the remaining feature indicators other than express The characteristic dimension of express The weight value of express The predicted value of express and The predicted value of the union.

[0028] Optionally, respectively encoding the ultrasound contrast image, the contrast index data, and the blood index data to obtain a contrast image feature graph, a contrast index feature graph, and a blood index feature graph, respectively, includes:

[0029] The ultrasound contrast image is subjected to standardization processing, and a convolutional neural network is used to extract and encode features of the ultrasound contrast image after the standardization processing to obtain a contrast image feature map;

[0030] The contrast index data and the blood index data are respectively normalized, and the normalized contrast index data and blood index data are respectively encoded using a multi-layer perceptron to obtain a contrast index characteristic graph and a blood index characteristic graph.

[0031] Optionally, the adaptive feature extraction module includes an adaptive attention submodule and a feedforward layer submodule, and the step of inputting the first multimodal feature map into the adaptive feature extraction module for processing to obtain a second multimodal feature map includes:

[0032] Inputting the first multimodal feature map into the adaptive attention submodule, and performing adaptive attention processing to obtain an attention output feature vector;

[0033] The attention output feature vector is input into the feed-forward layer submodule for feature conversion to obtain a second multimodal feature map.

[0034] Optionally, the adaptive attention submodule includes three weight matrices , as well as , the step of inputting the first multimodal feature map into the adaptive attention submodule, and obtaining an attention output feature vector after adaptive attention processing, comprises:

[0035] Multiply each of the weight matrices by the first multimodal feature map to obtain query vectors , key vector Sum value vector ;

[0036] For the query vector and the key vector Perform dot multiplication calculation to obtain the similarity score matrix;

[0037] Standardizing the similarity score matrix according to the number of dimensions of the first multimodal feature graph to obtain a standard similarity score matrix;

[0038] The standard similarity score matrix is ​​input into the adaptive SoftMax to calculate the similarity weights between the modal feature parameters;

[0039] The similarity weight and the value vector Multiply them together to get the weighted eigenvector;

[0040] The weighted feature vector is added to the first multimodal feature map, and subjected to layer normalization processing to obtain an attention output feature vector.

[0041] Optionally, inputting the attention output feature vector into the feed-forward layer submodule for feature conversion to obtain a second multimodal feature map includes:

[0042] Inputting the attention output feature vector into the feedforward layer submodule to obtain a feedforward output feature vector;

[0043] The feedforward output feature vector is added to the attention output feature vector, and subjected to layer normalization processing to obtain a second multimodal feature map.

[0044] Optionally, inputting the second multimodal feature map into a classification layer to obtain a predicted pathological type of the case includes:

[0045] Performing feature transformation on the second multimodal feature map using full connection to obtain a multimodal feature vector;

[0046] The multimodal feature vector is processed using a SoftMax function to obtain three predicted probability values; the three predicted probability values ​​respectively represent the probability values ​​of the case being benign, borderline, and malignant;

[0047] The largest value among the three predicted probability values ​​is taken as the predicted pathological type of the case.

[0048] In a second aspect, an embodiment of the present invention provides a pathology classification method, the method comprising:

[0049] Obtain ultrasound contrast data and blood index data of the case;

[0050] The ultrasound contrast imaging data and the blood index data are input into a pathology classification model to obtain the pathology type of the case.

[0051] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements any of the above-described methods when executing the program.

[0052] The embodiment of the present invention provides a pathological classification model training method, a pathological classification method and an electronic device, which obtains sample data including ultrasound contrast data, blood index data and pathological diagnosis labels, processes the ultrasound contrast data, obtains contrast index data and each ultrasound contrast image, respectively encodes the ultrasound contrast image, contrast index data and blood index data, and performs splicing and fusion to obtain a first multimodal feature map; inputs the first multimodal feature map into an adaptive feature extraction module for processing to obtain a second multimodal feature map, inputs the second multimodal feature map into a classification layer, obtains the predicted pathological type of the case, calculates loss information based on the predicted pathological type and pathological diagnosis label, and performs iterative training until the preset conditions are met. Thus, the accuracy of pathological classification can be improved by combining different feature responses on ultrasound contrast data and blood index data. The included feature indicators are screened by using information gain and importance evaluation, and feature indicators with high correlation with the pathological type are selected, which can improve the diagnostic efficiency of the model. The adaptive feature extraction module can utilize case data that lacks certain modalities, thereby enhancing the model's adaptability to data. At the same time, it can adaptively select various feature parameters in ultrasound contrast images, contrast index data, and blood index data.

[0053] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of a process flow of a pathology classification model training method provided by an embodiment of the present invention;

[0057] Figure 3 Another schematic diagram of a process flow of a pathology classification model training method provided by an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of data processing of a pathology classification model provided by an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of a process flow of a pathology classification method provided by an embodiment of the present invention.

[0060] Icon: 100 - electronic device; 101 - memory; 102 - communication interface; 103 - processor; 104 - bus. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0064] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0065] As described in the background technology, ovarian tumors are common reproductive system diseases in women and can occur at any age. According to biological behavior and pathological morphology, they can be divided into three categories: benign, borderline, and malignant. Ovarian malignant tumors are the fifth most common tumors among female malignant tumors and have the highest mortality rate. They have always seriously endangered the health of women all over the world. The ovaries are located deep in the pelvic cavity, with a deep anatomical position, complex histological types, and insidious onset. Ovarian malignant tumors often have no obvious symptoms in the early stages. About 70% of malignant patients are already in the late stage when diagnosed, and the 5-year survival rate of late-stage patients is only 30%, far lower than the 70%-90% in the early stage.

[0066] Ultrasound angiography is a new diagnostic technology that uses contrast agents to display the condition of microvessels based on conventional ultrasound. Doctors can diagnose lesions by observing the distribution, perfusion time, perfusion pattern and other characteristics of the microvessels displayed by the contrast agents. This greatly improves the level of exploration of the scope and depth of infiltration of tumors, and provides a reliable basis for differential diagnosis of the nature of ovarian tumors.

[0067] Tumor markers are substances that exist in malignant tumor cells, are produced by malignant tumors, or are produced by the host in response to stimulation of malignant tumors. They can be detected in the patient's tumor tissue, excrement or body fluids, and are often used in clinical practice for differential diagnosis, staging, and treatment response judgment of malignant tumors. Currently, the tumor markers known to be related to ovarian tumors and widely used include carbohydrate antigen CA125, human epididymis protein HE4, and carcinoembryonic antigen CEA. Since other malignant tumors such as pancreatic cancer, lung cancer, and gastric cancer also produce the same markers, the specificity of using tumor markers alone to diagnose ovarian tumors is low.

[0068] Based on the above situation, the embodiment of the present invention provides a pathology classification model training method, a pathology classification method and an electronic device, which obtains sample data including ultrasound contrast data, blood index data and pathology diagnosis labels, processes the ultrasound contrast data, obtains contrast index data and each ultrasound contrast image, and uses the sample data to iteratively train the pathology classification model. Thus, the pathology classification model can be trained by combining different characteristic responses on ultrasound contrast data and blood index data, thereby improving the accuracy of pathology classification.

[0069] To implement the process steps and functions of each example of the present invention, please refer to Figure 1 , Figure 1A schematic structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, and the memory 101 and the processor 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0070] The electronic device 100 may be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understandable that the electronic device 100 is not limited to a physical server, but may also be a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine. The operating system of the electronic device 100 may be, but is not limited to, a Windows system, a Linux system, etc.

[0071] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0072] The communication connection between the electronic device 100 and an external device is achieved through at least one communication interface 102 (which can be wired or wireless).

[0073] The processor 103 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the embodiment of the present invention can be completed by the hardware integrated logic circuit in the processor 103 or the instructions in the form of software. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0074] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0075] The following is an exemplary description of the pathology classification model training method provided by the present invention. Specifically, Figure 2 A flow chart of a pathology classification model training method provided by an embodiment of the present invention is shown in FIG. Figure 2 , the execution subject of this method can be the above Figure 1 The electronic device 100 shown in FIG. 1 is used to train a pathology classification model. The method includes: Figure 2 The following steps are shown:

[0076] S210: Obtain sample data for each case, where each sample data includes ultrasound imaging data, blood index data, and pathological diagnosis labels.

[0077] Obtain a large amount of basic sample data. Taking ovarian tumor detection as an example, one patient corresponds to one case and one sample data. Each sample data includes ultrasound angiography data, blood index data and pathological diagnosis labels.

[0078] Among them, ultrasound angiography data is obtained by performing ultrasound angiography on the patient's ovaries. Ultrasound angiography is a new diagnostic technology that uses contrast agents to display the condition of microvessels on the basis of conventional ultrasound. The lesions can be diagnosed by observing the distribution of microvessels, perfusion time, perfusion pattern and other characteristics displayed by the contrast agent.

[0079] Blood index data may include carbohydrate antigens CA125, CA199, CA153, human epididymis protein HE4, carcinoembryonic antigen CEA, alpha-fetoprotein AFP, human chorionic gonadotropin HCG, etc.

[0080] The pathological diagnosis label is the patient's pathological biopsy result, which can include benign, borderline, and malignant.

[0081] S220: extracting contrast index data of each ultrasound contrast data and ultrasound contrast images of the ultrasound contrast data when the contrast agent reaches the peak, and adding each contrast index data and each ultrasound contrast image to corresponding sample data.

[0082] Each ultrasound contrast data is processed to extract and record contrast index data, which may include perfusion characteristics, contrast agent enhancement start time, peak time, disappearance time, peak intensity, maximum lesion diameter, and area under the enhancement curve and other related parameters.

[0083] When the contrast agent reaches its peak, the key frame at that moment is extracted to obtain the ultrasound contrast image.

[0084] Each contrast index data and each ultrasound contrast image are added to the corresponding sample data, that is, one sample data includes ultrasound contrast data, ultrasound contrast image, contrast index data, blood index data and pathological diagnosis label.

[0085] S250: Input each sample data into the pathology classification model, encode the ultrasound contrast image, contrast index data and blood index data respectively, and obtain a contrast image feature map, a contrast index feature map and a blood index feature map respectively.

[0086] For each sample data, the ultrasound contrast image is encoded to obtain a contrast image feature map, the contrast index data is encoded to obtain a contrast index feature map, and the blood index data is encoded to obtain a blood index feature map.

[0087] S260: splicing and fusing the angiography image feature map, the angiography index feature map, and the blood index feature map to obtain a first multimodal feature map.

[0088] The feature maps of contrast images, contrast indexes, and blood indexes are concatenated in the channel dimension to increase the dimension of the features, and then further fused through a convolutional layer. This convolutional layer can include batch normalization and activation functions to enhance the expressiveness of features and introduce nonlinearity. Such a convolutional layer helps to learn the complex relationships between different features and extract more useful representations for subsequent tasks.

[0089] S270: Input the first multimodal feature map into the adaptive feature extraction module for processing to obtain a second multimodal feature map.

[0090] The adaptive feature extraction module can dynamically adjust the feature extraction process, improve the model's ability to recognize lesion features, enhance the model's generalization ability and computational efficiency, and improve the model's training efficiency and performance.

[0091] S280: Input the second multimodal feature map into the classification layer to obtain the predicted pathological type of the case.

[0092] Step S280 includes the following steps:

[0093] S281: Use full connection to perform feature transformation on the second multimodal feature map to obtain a multimodal feature vector.

[0094] S282: Use the SoftMax function to process the multimodal feature vector to obtain three predicted probability values; the three predicted probability values ​​respectively represent the probability values ​​of the case being benign, borderline, and malignant.

[0095] S283: The largest value among the three predicted probability values ​​is used as the predicted pathological type of the case.

[0096] In the classification layer, the second multimodal feature map is first transformed by full connection and the dimension of the parameters is reduced. Then, the SoftMax function is used to process and output three values, which respectively represent the probability values ​​of the ovarian space-occupying lesion case being benign, borderline, and malignant. The three probability values ​​representing the three properties are compared, and the one with the largest probability is used as the output result of the final model, that is, the ovarian space-occupying lesion being considered to belong to this type of property.

[0097] S290: Based on the predicted pathological type and the pathological diagnosis label, the loss information is calculated, and the pathological classification model is iteratively trained according to the loss information until the preset conditions are met.

[0098] Optionally, the sample data can be divided into a training set and a test set. The pathology classification model is trained using the training set data. The Adam optimizer is used during the training process. The loss function uses the cross entropy loss function. The batch-size of each batch of training images is set according to the size of the computing resources. At the same time, the number of iterations, initial learning rate, learning rate change strategy and other hyperparameters are set according to the training situation. The formula of the Adam optimizer is as follows:

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] in express The gradient of the moment; express The exponential moving average of the gradient at time, initial value ; express The squared exponential moving average of the gradient at time, initial value ; represents the learning rate; , which can avoid division by 0; express The model parameters at time t, express The model parameters at time.

[0105] The model generated during the training process is tested using the test set data. When the test results reach the preset target, the model training is completed. If the test results do not reach the preset target, the parameters need to be adjusted and the model training needs to be repeated.

[0106] The pathology classification model training method obtains sample data including ultrasound contrast data, blood index data and pathology diagnosis labels, processes the ultrasound contrast data, obtains contrast index data and each ultrasound contrast image, respectively encodes the ultrasound contrast image, contrast index data and blood index data, and performs splicing and fusion to obtain a first multimodal feature map, and iteratively trains the pathology classification model using the first multimodal feature map. In this way, the pathology classification model can be trained by combining different feature responses on ultrasound contrast data and blood index data, thereby improving the accuracy of pathology classification.

[0107] In a possible implementation, in order to improve the diagnostic efficiency of the pathology classification model, the angiographic index data and the blood index data can be screened, see Figure 3 , Figure 3 Another flow chart of a pathology classification model training method provided by an embodiment of the present invention, wherein the angiography index data and the blood index data each include at least one characteristic index, and before the above step S250, the pathology classification model training method may further include: Figure 3 The following steps are shown:

[0108] S230: Calculate information gain for the angiography index data and the blood index data in each sample data, remove feature indexes whose correlation is less than a first preset threshold, and obtain processed sample data.

[0109] Eliminate feature indicators with low correlation and retain data with high correlation with the original features. Optionally, this step may include the following steps:

[0110] S231: Put all feature indicators in each sample data into a total feature set.

[0111] For example, For all sample data, is the number of cases, For the i Cases n features, then the total feature set contains i*n features, which can be expressed as .

[0112] S232: Eliminate any feature index to obtain the remaining feature subset, and calculate the information gain value of the remaining feature subset; the calculation formula of the information gain value is:

[0113]

[0114]

[0115]

[0116] in, is the remaining feature subset, is the information gain value of the remaining feature subset, m is the number of cases, For the i Cases n Features, is the pathological diagnosis label of the sample data, , 0 represents benign, 1 represents borderline, and 2 represents malignant.

[0117] For example, calculate The information gain value of excluding the characteristic index After that, the remaining feature subset , which can be calculated by this formula Similarly, the information gain value of each feature index can be calculated in turn.

[0118] S233: Eliminate feature indicators whose information gain values ​​are less than a first preset threshold to obtain processed sample data.

[0119] S240: Evaluate the importance of the angiography index data and the blood index data in each processed sample data, and remove the characteristic indexes whose importance is less than a second preset threshold to obtain the processed sample data.

[0120] The characteristic indicators may be further screened by importance evaluation. Optionally, the formula for importance evaluation of the angiographic indicator data and the blood indicator data in the processed sample data is:

[0121]

[0122] in, represents the characteristic index to be evaluated, represents the total set of feature indicators, express The characteristic dimension of Indicates except Any subset of the remaining feature indicators other than express The characteristic dimension of express The weight value of express The predicted value of express and The predicted value of the union.

[0123] For example, if 10 characteristic indicators are removed in S230, , we are now To conduct a materiality assessment, for Substitute all subsets into the above formula for calculation, and we get The feature indicators whose importance is less than the second preset threshold are eliminated, and the data is restored to the structure of the sample data in step S220.

[0124] By screening the angiography index data and blood index data, the impact of redundant information on model performance is reduced.

[0125] There are many ways to encode the ultrasound contrast image, the contrast index data, and the blood index data. In one possible implementation, step S250 may include the following steps:

[0126] S251: Standardize the ultrasound contrast image, and use a convolutional neural network to extract and encode features of the standardized ultrasound contrast image to obtain a contrast image feature map.

[0127] Optionally, the formula for normalizing the ultrasound contrast imaging image may be:

[0128]

[0129] in, Represents the pixel value after normalization. represents the original pixel value, represents the mean of all pixel values, Indicates the standard deviation of pixel values.

[0130] Convolutional neural networks can adopt classic convolutional structures, such as ResNet, DenseNet, GoogLeNet, etc.

[0131] S252: Normalizing the contrast index data and the blood index data respectively, and encoding the normalized contrast index data and blood index data respectively using a multi-layer perceptron to obtain a contrast index feature map and a blood index feature map respectively.

[0132] Optionally, the normalization formula may be:

[0133]

[0134] in, represents the eigenvalue after normalization, represents the original eigenvalue, represents the minimum value among all eigenvalues, Represents the maximum value among all eigenvalues.

[0135] In a possible implementation, the adaptive feature extraction module in step S270 is a repeatedly stackable adaptive feature extraction module, the minimum number of stacking is set to 5, and the maximum number is set to 10. The adaptive feature extraction module may include an adaptive attention submodule and a feedforward layer submodule. Step S270 may include the following steps:

[0136] S271: Input the first multimodal feature map into the adaptive attention submodule, and obtain an attention output feature vector after adaptive attention processing.

[0137] Optionally, the adaptive attention submodule may include three weight matrices , as well as , , as well as The initial value of is randomly generated by a Gaussian function and updated during the pathology classification model training process. Step S271 may include the following steps:

[0138] S2711: Multiply each weight matrix by the first multimodal feature map to obtain query vectors , key vector Sum value vector .

[0139] S2712: Query vector and the key vector Perform dot multiplication to obtain the similarity score matrix.

[0140] S2713: Standardize the similarity score matrix according to the number of dimensions of the first multimodal feature graph to obtain a standard similarity score matrix.

[0141] The similarity score matrix is ​​scaled using the number of dimensions of the first multimodal feature map. The purpose of scaling is to ensure that the values ​​of the similarity score matrix are not too large or too small, which helps to stabilize the training of the model. The scaling operation can be achieved by dividing by the square root of the number of dimensions:

[0142]

[0143] Among them, score is the original similarity score, and d is the dimension of the feature map.

[0144] S2714: Input the standard similarity score matrix into the adaptive SoftMax to calculate the similarity weights between the modal feature parameters.

[0145] In Adaptive SoftMax, masks are used to select modal attention. When a modal parameter is missing, the modal attention area at the corresponding position on the mask is set to 0, that is, this area has no similarity and does not participate in the attention calculation process.

[0146] S2715: Similarity weight and value vector Multiply them together to get the weighted eigenvector.

[0147] Pathology classification model in computing and value vector Before multiplication, it will automatically check whether there is a missing mode. If there is a missing mode, the value of the corresponding area on the mask will be set to 0, so that only the valid mode can be converted from attention to similarity weight through the SoftMax function. The similarity weight vector is converted by the value vector By multiplying, the features with high correlation with the pathological type are enhanced, while the influence of the features with low correlation on the model output is suppressed, so as to realize the function of adaptively selecting each characteristic index in the ultrasound contrast image, contrast index data and blood index data.

[0148] S2716: Add the weighted feature vector to the first multimodal feature map, and perform layer normalization processing to obtain an attention output feature vector.

[0149] This method can be implemented using residual connection, that is, adding the weighted feature vector to the first multimodal feature map, and then using layer normalization to process the added result, which can not only maintain the distribution stability of the data, but also improve the training efficiency and performance of the pathology classification model.

[0150] S272: Input the attention output feature vector into the feed-forward layer submodule for feature conversion to obtain a second multimodal feature map.

[0151] Similar to the logic of step S2716, step S272 may include the following steps:

[0152] S2721: Input the attention output feature vector into the feedforward layer submodule to obtain the feedforward output feature vector.

[0153] S2722: Add the feedforward output feature vector and the attention output feature vector, and perform layer normalization processing to obtain a second multimodal feature map.

[0154] In this embodiment, in the classification layer, the second multimodal feature map is first converted by full connection and the dimension of the parameters is reduced, and then the SoftMax function is used to process and output three values, which respectively represent the probability values ​​of the ovarian space-occupying lesion case of the input test being benign, borderline and malignant. The three probability values ​​representing the three properties are compared, and the one with the largest probability is used as the output result of the final model, that is, it is considered that the ovarian space-occupying lesion of the input test belongs to this type of property.

[0155] The following is an overall description of the sample data processing process, see Figure 4 , Figure 4 The data processing diagram of the pathology classification model provided in the embodiment of the present invention is as follows: after obtaining the sample data, the sample data is processed. Each sample data includes an ultrasound contrast image, contrast index data, and blood index data. For a sample data, Figure 4 First, the ultrasound contrast image, contrast index data and blood index data in the sample data are encoded respectively to obtain the contrast image feature map, the contrast index feature map and the blood index feature map, and then the three are spliced ​​and fused to obtain the first multimodal feature map, and the first multimodal feature map is input into the adaptive attention submodule, and the output obtained is added to the input first multimodal feature map to obtain the attention output feature vector, and the attention output feature vector is input into the feedforward layer submodule, and the output obtained is added to the input attention output feature vector to obtain the second multimodal feature map, and finally the second multimodal feature map is input into the classification layer to obtain the predicted pathological type.

[0156] Furthermore, based on the above pathology classification model, the present invention also provides a pathology classification method, see Figure 5 , Figure 5A schematic diagram of a process flow of a pathology classification method provided by an embodiment of the present invention, the pathology classification method includes: Figure 5 The following steps are shown:

[0157] S310: Obtain ultrasound contrast data and blood index data of the case.

[0158] S320: Inputting the ultrasound contrast data and the blood index data into the pathology classification model to obtain the pathology type of the case.

[0159] Among them, the pathology classification model is trained by the above-mentioned pathology classification model training method.

[0160] In summary, the pathological classification model training method, pathological classification method and electronic device provided by the embodiment of the present invention obtain sample data including ultrasound contrast data, blood index data and pathological diagnosis labels, process the ultrasound contrast data, obtain contrast index data and each ultrasound contrast image, encode the ultrasound contrast image, contrast index data and blood index data respectively, and perform splicing and fusion to obtain a first multimodal feature map; input the first multimodal feature map into the adaptive feature extraction module for processing to obtain a second multimodal feature map, input the second multimodal feature map into the classification layer, obtain the predicted pathological type of the case, calculate the loss information based on the predicted pathological type and pathological diagnosis label, and perform iterative training until the preset conditions are met. Thus, the accuracy of pathological classification can be improved by combining different feature responses on ultrasound contrast data and blood index data. The use of the adaptive feature extraction module can utilize case data with some modalities missing, enhance the adaptability of the model to the data, and at the same time, it can adaptively select each feature parameter in the ultrasound contrast image, contrast index data and blood index data.

[0161] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0163] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0164] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0165] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A pathology classification model training method, characterized in that: include: Acquire sample data of each case, each of the sample data includes ultrasound contrast data, blood index data and pathological diagnosis labels; Extracting contrast index data of each of the ultrasound contrast data and ultrasound contrast images of the ultrasound contrast data when the contrast agent reaches a peak, and adding each of the contrast index data and each of the ultrasound contrast images to the corresponding sample data; the contrast index data and the blood index data each include at least one characteristic index; The information gain calculation is performed on the angiography index data and the blood index data in each of the sample data, and the feature indexes whose correlation is less than a first preset threshold are eliminated to obtain the processed sample data, including: putting all the feature indexes in each sample data into a total feature set; excluding any of the feature indexes to obtain the remaining feature subsets, and calculating the information gain value of the remaining feature subsets; the calculation formula of the information gain value is: in, is the remaining feature subset, is the information gain value of the remaining feature subset, m is the number of cases, For the i of the cases described n Features, is the pathological diagnosis label of the sample data, , 0 means benign, 1 means borderline, and 2 means malignant; Eliminate the feature indicators whose information gain values ​​are less than a first preset threshold to obtain processed sample data; The importance of the contrast index data and the blood index data in each processed sample data is evaluated, and the characteristic index whose importance is less than the second preset threshold is eliminated to obtain the processed sample data; the formula for evaluating the importance of the contrast index data and the blood index data in each processed sample data is: in, represents the characteristic index to be evaluated, represents the total set of feature indicators, express The characteristic dimension of Indicates except Any subset of the remaining feature indicators other than express The characteristic dimension of express The weight value of express The predicted value of express and The predicted value of the union; Input each of the sample data into a pathology classification model, encode the ultrasound contrast image, the contrast index data, and the blood index data, respectively, and obtain a contrast image feature map, a contrast index feature map, and a blood index feature map, respectively; The angiography image feature map, the angiography index feature map and the blood index feature map are spliced ​​and fused to obtain a first multimodal feature map; The first multimodal feature map is input into an adaptive feature extraction module for processing to obtain a second multimodal feature map; the adaptive feature extraction module includes an adaptive attention submodule and a feedforward layer submodule; the adaptive attention submodule includes three weight matrices , as well as ; The step of inputting the first multimodal feature map into an adaptive feature extraction module for processing to obtain a second multimodal feature map comprises: multiplying each of the weight matrices by the first multimodal feature map to obtain query vectors , key vector Sum value vector ; For the query vector and the key vector Perform a dot multiplication calculation to obtain a similarity score matrix; standardize the similarity score matrix according to the number of dimensions of the first multimodal feature map to obtain a standard similarity score matrix; input the standard similarity score matrix into the adaptive SoftMax to calculate the similarity weights between the modal feature parameters; and compare the similarity weights with the value vector The method comprises: multiplying the weighted feature vector and the first multimodal feature map to obtain a weighted feature vector; adding the weighted feature vector and the first multimodal feature map, and performing layer normalization processing to obtain an attention output feature vector; inputting the attention output feature vector into the feedforward layer submodule for feature conversion to obtain a second multimodal feature map; inputting the second multimodal feature map into the classification layer to obtain the predicted pathological type of the case, including: using full connection to perform feature conversion on the second multimodal feature map to obtain a multimodal feature vector; using the SoftMax function to process the multimodal feature vector to obtain three predicted probability values; the three predicted probability values ​​respectively represent the probability values ​​of the case being benign, borderline and malignant; taking the largest value among the three predicted probability values ​​as the predicted pathological type of the case; Based on the predicted pathology type and the pathology diagnosis label, loss information is calculated, and the pathology classification model is iteratively trained according to the loss information until a preset condition is met.

2. The method according to claim 1, characterized in that The step of respectively encoding the ultrasound contrast image, the contrast index data and the blood index data to obtain a contrast image characteristic graph, a contrast index characteristic graph and a blood index characteristic graph respectively comprises: The ultrasound contrast image is subjected to standardization processing, and a convolutional neural network is used to extract and encode features of the ultrasound contrast image after the standardization processing to obtain a contrast image feature map; The contrast index data and the blood index data are respectively normalized, and the normalized contrast index data and blood index data are respectively encoded using a multi-layer perceptron to obtain a contrast index characteristic graph and a blood index characteristic graph.

3. The method according to claim 1, characterized in that The step of inputting the attention output feature vector into the feed-forward layer submodule for feature conversion to obtain a second multimodal feature map includes: Inputting the attention output feature vector into the feedforward layer submodule to obtain a feedforward output feature vector; The feedforward output feature vector is added to the attention output feature vector, and subjected to layer normalization processing to obtain a second multimodal feature map.

4. A pathology classification method, characterized in that: The method comprises: Obtain ultrasound contrast data and blood index data of the case; The ultrasound contrast imaging data and the blood index data are input into a pathology classification model to obtain the pathology type of the case; the pathology classification model is trained by the method according to any one of claims 1-3.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 or claim 4 is implemented.

Images