Multi-modal ovarian tumor database construction method and system and medium
By constructing a multimodal ovarian tumor database and integrating image, table and text data, the problem of multimodal data integration in the existing technology is solved, the accuracy and efficiency of ovarian tumor diagnosis is improved, and it is expanded to other pathological predictions.
Patent Information
- Application Number
- CN202410964305.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
AI Technical Summary
The existing technology is difficult to effectively integrate multiple data modes, resulting in insufficient diagnostic efficiency and accuracy in the diagnosis of diseases such as ovarian tumors.
Using the multimodal database construction method, by obtaining image, table and text modal data, preprocessing and oversampling of data, standard images, tables and text databases are constructed, and combined with machine learning technology, a balanced multimodal ovarian tumor database is generated. .
It improves the diagnostic accuracy and efficiency of ovarian tumors, and uses migration to other pathological predictions, and realizes auxiliary decision support for medical diagnosis.
Smart Images

Figure CN118942720A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent health management, and more specifically, relates to a method, system and medium for constructing a multimodal ovarian tumor database. Background Art
[0002] Artificial intelligence and machine learning systems are expected to become essential tools for the next generation of clinical practice and medical diagnostics. This expected utility, particularly in AI systems designed to improve clinical efficiency and patient outcomes, will require knowledge from multiple data sources and a variety of input modalities. Multimodal architectures for AI systems are attractive because they can emulate the input conditions that clinicians and healthcare administrators currently use to perform predictions and responses in their complex decision-making environments.
[0003] Typical clinical practice uses a set of different information formats contained in patients' Electronic Health Records, such as tabular data (such as age, sex, height and weight, etc.), image data (such as photographs, X-rays, computed tomography, magnetic resonance imaging, etc.), time series data (such as intermittent pulse oximetry, respiratory analysis, electrocardiogram, ultrasound and wearable sensors, etc.) and structured sequence data (such as genomics, proteomics and metabolomics, etc.) and other sources (Soenksen LR, Ma Y et al.).
[0004] Recently, artificial intelligence models that use multiple data modes have been proven to be applicable to fields such as cardiology, dermatology, gastroenterology, gynecology, and immunology. How to better integrate data from different modalities and use machine learning models and artificial intelligence frameworks based on multimodal databases to better assist medical diagnosis and prediction has always been a key research topic for experts and scholars in various fields. Summary of the invention
[0005] The purpose of the present invention is to provide a multimodal ovarian tumor database construction method, system and medium, which uses machine learning technology and multimodal data to construct a patient ovarian tumor classification database to assist doctors in making diagnoses and improve the efficiency and accuracy of diagnosis. It is based on the current development of multimodal data in the medical field and is in line with the current trend of artificial intelligence being applied to medical diagnosis.
[0006] In view of the development and demand for multimodal data in the current medical field, and referring to the importance and necessity of artificial intelligence-assisted diagnosis and prediction in the medical field in today's society, by establishing a tumor database based on machine learning, it is not only possible to analyze and predict ovarian tumor data, but also to migrate and apply it to the prediction of other pathologies.
[0007] In order to achieve the above object, the present invention adopts the following technical scheme:
[0008] According to a first aspect of the present invention, a method for constructing a multimodal ovarian tumor database is provided, the method comprising:
[0009] Acquire image modality data, table modality data, and text modality data, wherein the image modality data, table modality data, and text modality data all contain patient personal information, form a multimodality data with the same patient personal information, assign a classification label to the multimodality data, and the classification label is used to indicate the ovarian tumor category of the multimodality data;
[0010] Processing the image modality data to obtain a standard image database;
[0011] Processing the table modal data to obtain a standard table database;
[0012] Processing the text modality data to obtain a standard text database;
[0013] A multimodal ovarian tumor database is constructed, and different degrees of oversampling are performed on samples of different categories according to the distribution density of different samples in the multimodal ovarian tumor database. The multimodal ovarian tumor database includes a standard image database, a standard table database, and a standard text database. The samples include processed image modality data, processed table modality data, processed text modality data, or processed multimodality data.
[0014] Furthermore, the image modal data includes at least one ultrasound image of different patients, the table modal data includes a clinical information table of ovarian tumors of different patients, and the text modal data includes three characteristic fields contained in the clinical information table of ovarian tumors of different patients, and the three characteristic fields are operation name, postoperative pathology and pathology summary.
[0015] Furthermore, the image modality data is processed to obtain a standard image database, specifically including:
[0016] Based on the image modality data, at least one image modality data of a patient is named in the form of XN, where X represents the patient number, which is determined according to the patient's personal information, and N represents the serial number of the patient's image modality data;
[0017] When one image modality data includes a plurality of ultrasound images, the image modality data is cropped to obtain a plurality of ultrasound images, each ultrasound image being used as one image modality data of the patient;
[0018] In the case where the patient's privacy information exists in the image modality data, the image modality data is collected to remove the patient's privacy information in the image modality data;
[0019] The named image modality data is input into an image feature extractor for extraction to obtain standard image data of a set image size. The standard image database is composed of a plurality of standard image data of different patients.
[0020] Furthermore, the named image modality data is input into the image feature extractor for extraction to obtain standard image data of a set image size, specifically including:
[0021] Determine the reduction factors k1 and k2 of the width and length according to the size W×H of the image input to the image feature extractor, where W is the width of the image and H is the length of the image;
[0022] The sampling interval is determined as W / k1, H / k2 according to the reduction factors k1 and k2 of the width and length.
[0023] The image input to the image feature extraction is downsampled according to the sampling interval to output standard image data of a set image size.
[0024] Furthermore, the table modal data is processed to obtain a standard table database, which specifically includes:
[0025] If there are missing data in the table data, the missing data is replaced by the average value of the field where the missing data is located;
[0026] Directly save the numerical information in the table data, convert the data recorded in yes / no to 0 / 1 for saving, and remove the three characteristic fields contained in the table data;
[0027] If the table data contains a patient identifier, de-identify the patient identifier, wherein the patient identifier includes the patient's name and surgery date, and use an unsequenced number as a unique identifier to mark the patient's personal information corresponding to the table data according to the patient's personal information;
[0028] The table data is normalized to obtain standard table data, each characteristic field in the standard table data is a numerical value, and the standard table data of different patients constitute the standard table database. The normalization process is expressed as:
[0029]
[0030] In the formula, is the mean of the original data in the table data, is the standard deviation of the original data in the table data, x is the original data in the table data, x new is the normalized value, x i is the i-th original data in the table data, and n is the total number of original data in the table data.
[0031] Furthermore, the text modality data is processed to obtain a standard text database, specifically including:
[0032] Determine the text length according to the number of words in the text modality data of each patient;
[0033] Based on the text length, the text data in the text modality data is processed by adding uniform invalid characters to embed and fill or deleting useless characters to obtain standard text data. The length of the text data in the standard text data is equal to the text length. The standard text data of different patients constitute the standard text database.
[0034] Furthermore, according to the distribution density of different samples in the multimodal ovarian tumor database, different categories of samples are oversampled to different degrees, specifically including:
[0035] Determine the number of minority class samples N to be generated + ;
[0036] Find each minority class sample in the original training set S i=1,2,…,N + The number of K nearest neighbors of the i-th minority class sample is The proportion parameter Γ of each minority class sample is determined by the following formula i :
[0037]
[0038] In the formula, Z is the normalization factor, K is the number of neighbors of the i-th minority class sample;
[0039] The frequency g of each minority class sample being selected as the main sample is determined by the following formula i :
[0040] g i =Γ i ×N + ×SR
[0041] Where SR is the sampling rate.
[0042] Furthermore, according to the distribution density of different samples in the multimodal ovarian tumor database, oversampling of different categories of samples to different degrees is performed, further comprising:
[0043] When a minority class sample is selected as the main sample, the distance between the minority class sample and its K nearest neighbor samples is calculated.
[0044] The weight of the corresponding nearest neighbor sample is determined according to the distance between each nearest neighbor sample and the minority class sample.
[0045] Based on the weights, generating synthetic samples;
[0046] The synthetic samples are merged with the original data set to obtain a balanced data set.
[0047] According to a second aspect of the present invention, a multimodal ovarian tumor database construction system is provided, the system comprising:
[0048] A data acquisition module is configured to acquire image modality data, table modality data, and text modality data, wherein the image modality data, table modality data, and text modality data all contain patient personal information, and to form a multimodal data with the same patient personal information, and to assign a classification label to the multimodal data, wherein the classification label is used to indicate the ovarian tumor category of the multimodal data;
[0049] A first data processing module is configured to process the image modality data to obtain a standard image database;
[0050] A second data processing module is configured to process the table modal data to obtain a standard table database;
[0051] A third data processing module is configured to process the text modality data to obtain a standard text database;
[0052] The database construction and oversampling module is configured to construct a multimodal ovarian tumor database, and to oversample different categories of samples to different degrees according to the distribution density of different samples in the multimodal ovarian tumor database. The multimodal ovarian tumor database includes a standard image database, a standard table database, and a standard text database. The samples include processed image modality data, processed table modality data, processed text modality data, or processed multimodality data.
[0053] According to a third aspect of the present invention, there is provided a readable storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the method as described above.
[0054] The present invention has at least the following beneficial effects:
[0055] The advantages of the present invention are to establish a large-scale patient-centered multimodal database, provide a collection and preprocessing paradigm for medical multimodal databases, build a multimodal database based on machine learning artificial intelligence, assist in the medical diagnosis of ovarian tumors, and improve the accuracy and efficiency of disease diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 A flowchart of a method for constructing a multimodal ovarian tumor database according to an embodiment of the present invention is shown.
[0057] Figure 2 A flow chart of constructing a standard image database according to an embodiment of the present invention is shown.
[0058] Figure 3 A standard image data acquisition flow chart according to an embodiment of the present invention is shown.
[0059] Figure 4 A flow chart of constructing a standard table database according to an embodiment of the present invention is shown.
[0060] Figure 5 A flow chart of constructing a standard text database according to an embodiment of the present invention is shown.
[0061] Figure 6 A flow chart of sample oversampling according to an embodiment of the present invention is shown.
[0062] Figure 7 A further flow chart of sample oversampling according to an embodiment of the present invention is shown.
[0063] Figure 8 A structural diagram of a multimodal ovarian tumor database construction system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0064] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and specific embodiments, but are not intended to limit the present invention. For the various steps described herein, if there is no necessity for a causal relationship between each other, the order in which they are described as examples herein should not be regarded as a limitation, and those skilled in the art should know that they can be adjusted in order, as long as the logic between them is not destroyed, resulting in the inability to implement the entire process.
[0065] The present invention proposes a method for constructing a multimodal ovarian tumor database. Figure 1 , is a flow chart of the method for constructing a multimodal ovarian tumor database, and the method for constructing a multimodal ovarian tumor database includes the following steps:
[0066] Step S100, acquiring image modal data, table modal data and text modal data, wherein the image modal data, table modal data and text modal data all contain patient personal information, combining the image modal data, table modal data and text modal data with the same patient personal information into a multimodal data, assigning a classification label to the multimodal data, and the classification label is used to indicate the ovarian tumor category of the multimodal data.
[0067] In some embodiments, the image modality data includes at least one ultrasound image of different patients, the table modality data includes a clinical information table of ovarian tumors of different patients, and the text modality data includes three characteristic fields contained in the clinical information table of ovarian tumors of different patients, and the three characteristic fields are surgery name, postoperative pathology and pathology summary.
[0068] For example, the above data can be images and tabular data provided by medical staff. The image data is based on patients and named by patient numbers, and includes several two-dimensional ultrasound images and ultrasound contrast images of patients, as well as related ultrasound diagnosis information reports. The diagnosis report includes text information such as "clinical diagnosis" (clinical cases), "diagnosis site" (specific site of ultrasound examination, such as vaginal uterus, transvaginal, etc.), "ultrasound diagnosis conclusion" (radiologist's preliminary diagnosis of patient nodules and possible tumors based on ultrasound images) and "ultrasound diagnosis description" (radiologist's specific description of the shape and size of nodules and possible tumors, as well as blood flow signals based on ultrasound images and ultrasound diagnosis conclusions). Tabulated data comes from the clinical information form of ovarian tumors provided by medical staff, including patient personal information (date of surgery, name, age and patient number), "menstruation and childbearing" (age of menarche, number of days of menstruation, menstrual cycle, number of pregnancies, number of births and whether menopause), basic physical signs data (height, weight, age and BMI), "course of disease and symptoms" (course of disease, whether there is abdominal pain and whether there is abdominal distension), "tumor history" (family history of malignant tumors, family history of ovarian cancer, personal history of breast cancer, personal history of reproductive system malignant tumors and personal history of malignant tumors other than breast and reproductive system), "comorbidities" (whether there is hypertension and diabetes), "tumor markers" (CA125, CEA, CA199, AFP and CA153, etc.), "surgery and course of disease" (name of surgery, postoperative pathology and pathology summary).
[0069] The "Pathology Summary" field in the original table is the final ovarian tumor pathology type of the patient independently derived by at least three professional doctors and experts based on imaging and table data and surgery. "Pathology Summary" is intended to be used as the gold standard for measurement. There are more than 30 types of ovarian tumors provided. Through statistical analysis, the 7 most numerous categories are selected as classification labels, and the remaining categories are classified as "others", thus obtaining 8 types of ovarian tumors.
[0070] Step S200: Process the image modality data to obtain a standard image database.
[0071] In some embodiments, see Figure 2 , is a flow chart for constructing a standard image database, wherein the image modality data is processed to obtain a standard image database, specifically including:
[0072] Step S201, based on the image modality data, at least one image modality data of a patient is named in the form of XN, where X represents the patient number, which is determined according to the patient's personal information, and N represents the serial number of the patient's image modality data.
[0073] Exemplarily, for image data, several ultrasound images of a patient are named in the form of "patient number-0N, N=1, 2, 3..." (such as "0010286156-01"), stored in jpg format, and correspond to clinical information according to the patient number.
[0074] Step S202 , when one image modality data includes multiple ultrasound images, the image modality data is cropped to obtain multiple ultrasound images, each of which serves as one image modality data of the patient.
[0075] Step S203: When the image modality data contains the patient's privacy information, the image modality data is collected to remove the patient's privacy information in the image modality data.
[0076] For example, in the special case where there are multiple ultrasound images (two, four or nine) in one picture, use XnConvert software to manually crop them into single images losslessly and name them in a unified format. The original image may contain private information such as the patient's name and examination date. In order to de-privacy, such images are also cropped. There are 2497 original image data in total, and 3349 after processing. Due to the particularity of ultrasound examination, the same patient often takes several ultrasound examination images from multiple angles and positions during the examination. The relationship between ultrasound images and tables and result factors is many-to-one, that is, one patient corresponds to one table data, one text data and several ultrasound images.
[0077] Step S204, inputting the named image modality data into an image feature extractor for extraction to obtain standard image data of a set image size, wherein the standard image database is composed of a plurality of standard image data of different patients.
[0078] In some embodiments, see Figure 3, which is a flowchart for obtaining standard image data, inputs the named image modality data into the image feature extractor for extraction, and obtains standard image data of a set image size, specifically including:
[0079] Step S2041, determining the reduction factors k1 and k2 of the width and length according to the size W×H of the image input into the image feature extractor, where W is the width of the image and H is the length of the image.
[0080] Step S2042: Determine the sampling interval as W / k1, H / k2 according to the reduction factors k1 and k2 of the width and length.
[0081] Step S2043, downsampling the image input to the image feature extraction according to the sampling interval to output standard image data of a set image size.
[0082] For example, image reduction is achieved by reducing the number of pixels. Therefore, appropriate pixels should be selected from the original image according to the reduced size so that the image can maintain the characteristics of the original image after reduction. Here, the equal-interval sampling method is mainly used, which is to take pixels at a certain distance in the original image and put them on the output image. Suppose the size of the original image is W×H, and the reduction factors of width and length are k1 and k2 respectively, then the sampling intervals are W / k1 and H / k2 respectively. That is to say, a pixel is taken every W / k1 in the horizontal direction of the original image and every H / k2 in the vertical direction. When the reduction factors k1 and k2 of length and width are equal, the image is reduced in equal proportion; when they are not equal, they are reduced in unequal proportions, and the ratio of the length and width of the reduced image will change. The "resize()" function in the "opencv" library is used to implement image downsampling, and the output image size is defined as 224×224.
[0083] Step S300: Process the table modal data to obtain a standard table database.
[0084] In some embodiments, see Figure 4 , is a flow chart for constructing a standard table database, wherein the table modal data is processed to obtain a standard table database, specifically including:
[0085] Step S301: when there is missing data in the table data, the missing data is replaced by the average value of the field where the missing data is located.
[0086] Step S302, directly save the numerical information in the table data, convert the data recorded in yes / no to 0 / 1 for storage, and remove the three characteristic fields contained in the table data.
[0087] For example, for table data, considering the actual significance of medical data, the default data is intended to be replaced by the average value of the field. The numerical information is directly saved, and the data recorded in "yes / no" is converted to "0 / 1" for storage, and the three fields of "surgery and pathology" (surgery name, postoperative pathology and pathology summary) stored in natural language are removed. At this time, the table data is converted into a standard table database, and each feature field is a numerical value.
[0088] Step S303, if there is a patient identifier in the table data, the patient identifier is de-identified, the patient identifier includes the patient's name and operation date, and an unsequenced number is used as a unique identifier based on the patient's personal information to mark the patient's personal information corresponding to the table data.
[0089] It should be noted that, considering the particularity of medical data, in order to better provide patients with health insurance rights and protect the privacy of their health information, the patient identifier will be de-identified in accordance with the "HIPAA Act", and the patient's name, surgery date and other information in the original form will be hidden. A special unordered number will be used as the unique identifier for the patient. In order to limit the pre-processed data to a certain range and eliminate the adverse effects caused by singular sample data, the original table data needs to be normalized. This operation can speed up the gradient descent to find the optimal solution, and thus may improve the accuracy.
[0090] Step S304, normalizing the table data to obtain standard table data, wherein each characteristic field in the standard table data is a numerical value, and the standard table data of different patients constitute the standard table database, and the normalization process is expressed as:
[0091]
[0092] In the formula, is the mean of the original data in the table data, is the standard deviation of the original data in the table data, x is the original data in the table data, x new is the normalized value, x i is the i-th original data in the table data, and n is the total number of original data in the table data.
[0093] Step S400: Process the text modality data to obtain a standard text database.
[0094] In some embodiments, see Figure 5 , is a flow chart for constructing a standard text database, wherein the text modality data is processed to obtain a standard text database, specifically including:
[0095] Step S401, determining the text length according to the number of words in the text modality data of each patient.
[0096] It should be noted that the text data in the text modal data of each patient can be the three characteristic fields of "surgery and pathology" (surgery name, postoperative pathology and pathology summary) in the table, and the text information such as "clinical diagnosis", "diagnosis site", "ultrasound diagnosis conclusion" and "ultrasound diagnosis description" in the diagnosis report, which are organized into a standardized whole paragraph of text data. In this way, the number of words in the text data of each patient can be counted. After counting the number of words in the text data of all patients, the mean, median or maximum value or the ratio of the maximum value of the number of words in the text data of all patients can be used as the text length, where the number of words in the text data refers to the number of all characters in the text data.
[0097] Step S402, based on the text length, the text data in the text modality data is processed by adding uniform invalid characters to embed and fill or deleting useless characters to obtain standard text data, the length of the text data in the standard text data is equal to the text length, and the standard text data of different patients constitute the standard text database.
[0098] Step S500, constructing a multimodal ovarian tumor database, oversampling different categories of samples to different degrees according to the distribution density of different samples in the multimodal ovarian tumor database, wherein the multimodal ovarian tumor database includes a standard image database, a standard table database, and a standard text database, and the samples include processed image modality data, processed table modality data, processed text modality data, or processed multimodality data.
[0099] It should be noted that the purpose of oversampling different categories of samples to different degrees according to the distribution density of different samples in the multimodal ovarian tumor database is to use weighted distribution for samples of different minority categories (such as simple cysts and high-grade serous cystadenoma) according to the different learning difficulties, so that the minority class samples that are easier for the model to learn, and more comprehensive data are generated for the minority class samples that are difficult to learn; the model's learning of data distribution is improved by reducing the bias introduced by class imbalance and adaptively shifting the classification decision boundary to difficult samples.
[0100] Different samples can be oversampled to solve the problem of unbalanced distribution of sample data in each database included in the multimodal ovarian tumor database.
[0101] In some embodiments, see Figure 6, is a flow chart of sample oversampling, which performs oversampling to different degrees on samples of different categories according to the distribution density of different samples in the multimodal ovarian tumor database, specifically including:
[0102] Step S501, determine the number of minority class samples N to be generated + ;
[0103] Step S502, find each minority class sample in the original training set S i=1,2,…,N + The number of K nearest neighbors of the i-th minority class sample is The proportion parameter Γ of each minority class sample is determined by the following formula i :
[0104]
[0105] In the formula, Z is the normalization factor, K is the number of neighbors of the i-th minority class sample;
[0106] Step S503: determine the frequency g of each minority class sample being selected as the main sample by the following formula: i :
[0107] g i =Γ i ×N + ×SR
[0108] Where SR is the sampling rate.
[0109] In some embodiments, see Figure 7 , is a further flow chart of sample oversampling, in Figure 6 Based on the process shown, different categories of samples are oversampled to different degrees according to the distribution density of different samples in the multimodal ovarian tumor database, and further include:
[0110] Step S504, when a minority class sample is selected as a main sample, calculating the distance between the minority class sample and its K nearest neighbor samples;
[0111] Step S505, determining the weight of the corresponding nearest neighbor sample according to the distance between each nearest neighbor sample and the minority class sample;
[0112] Step S506, generating a synthetic sample based on the weight;
[0113] Step S507: merging the synthesized sample with the original data set to obtain a balanced data set.
[0114] It should be noted that the role of step S500 is to process unbalanced data sets, and its main idea is to make full use of the density distribution information of the samples to determine the frequency of each minority class sample as a majority sample, and synthesize more training data for a few difficult-to-learn class samples, so as to correct the negative impact of the imbalanced distribution of the class as much as possible. The principle of the ADASYN algorithm is to generate synthetic samples based on the density of each minority class sample. It first calculates the distance between each minority class sample and its nearest neighbor sample, and then calculates a weight based on the distance of each sample to guide the generation of synthetic samples. Minority class samples with a longer distance will obtain a higher weight, so more synthetic samples will be generated, while minority class samples with a closer distance will generate fewer synthetic samples. In this way, synthetic samples can be generated more adaptively, so that the synthetic samples are closer to the real data distribution, thereby improving the model's learning ability for minority class samples. In other words, the data distribution of each sample in the multimodal ovarian tumor database constructed by this embodiment is finally balanced.
[0115] Figure 8 FIG. 1 shows a structural diagram of a multimodal ovarian tumor database construction system according to an embodiment of the present invention. The embodiment of the present invention provides a multimodal ovarian tumor database construction system, such as Figure 8 As shown, the multimodal ovarian tumor database construction system 800 includes:
[0116] The data acquisition module 801 is configured to acquire image modality data, table modality data, and text modality data, wherein the image modality data, table modality data, and text modality data all contain patient personal information, and form a multimodal data with the same patient personal information, and assign a classification label to the multimodal data, wherein the classification label is used to indicate the ovarian tumor category of the multimodal data;
[0117] A first data processing module 802 is configured to process the image modality data to obtain a standard image database;
[0118] The second data processing module 803 is configured to process the table modal data to obtain a standard table database;
[0119] A third data processing module 804 is configured to process the text modality data to obtain a standard text database;
[0120] The database construction and oversampling module 805 is configured to construct a multimodal ovarian tumor database, and oversample different categories of samples to different degrees according to the distribution density of different samples in the multimodal ovarian tumor database. The multimodal ovarian tumor database includes a standard image database, a standard table database, and a standard text database. The samples include processed image modality data, processed table modality data, processed text modality data, or processed multimodality data.
[0121] In some embodiments, the image modality data includes at least one ultrasound image of different patients, the table modality data includes a clinical information table of ovarian tumors of different patients, and the text modality data includes three characteristic fields contained in the clinical information table of ovarian tumors of different patients, and the three characteristic fields are surgery name, postoperative pathology and pathology summary.
[0122] In some embodiments, the first data processing module is further configured to:
[0123] Based on the image modality data, at least one image modality data of a patient is named in the form of XN, where X represents the patient number, which is determined according to the patient's personal information, and N represents the serial number of the patient's image modality data;
[0124] When one image modality data includes a plurality of ultrasound images, the image modality data is cropped to obtain a plurality of ultrasound images, each ultrasound image being used as one image modality data of the patient;
[0125] In the case where the patient's privacy information exists in the image modality data, the image modality data is collected to remove the patient's privacy information in the image modality data;
[0126] The named image modality data is input into an image feature extractor for extraction to obtain standard image data of a set image size. The standard image database is composed of a plurality of standard image data of different patients.
[0127] In some embodiments, the first data processing module is further configured to:
[0128] Determine the reduction factors k1 and k2 of the width and length according to the size W×H of the image input to the image feature extractor, where W is the width of the image and H is the length of the image;
[0129] The sampling interval is determined as W / k1, H / k2 according to the reduction factors k1 and k2 of the width and length.
[0130] The image input to the image feature extraction is downsampled according to the sampling interval to output standard image data of a set image size.
[0131] In some embodiments, the second data processing module is further configured to:
[0132] If there are missing data in the table data, the missing data is replaced by the average value of the field where the missing data is located;
[0133] Directly save the numerical information in the table data, convert the data recorded in yes / no to 0 / 1 for saving, and remove the three characteristic fields contained in the table data;
[0134] If the table data contains a patient identifier, de-identify the patient identifier, wherein the patient identifier includes the patient's name and surgery date, and use an unsequenced number as a unique identifier to mark the patient's personal information corresponding to the table data according to the patient's personal information;
[0135] The table data is normalized to obtain standard table data, each characteristic field in the standard table data is a numerical value, and the standard table data of different patients constitute the standard table database. The normalization process is expressed as:
[0136]
[0137] In the formula, is the mean of the original data in the table data, is the standard deviation of the original data in the table data, x is the original data in the table data, x new is the normalized value, x i is the i-th original data in the table data, and n is the total number of original data in the table data.
[0138] In some embodiments, the third data processing module is further configured to:
[0139] Determine the text length according to the number of words in the text modality data of each patient;
[0140] Based on the text length, the text data in the text modality data is processed by adding uniform invalid characters to embed and fill or deleting useless characters to obtain standard text data. The length of the text data in the standard text data is equal to the text length. The standard text data of different patients constitute the standard text database.
[0141] In some embodiments, the database construction and oversampling module is further configured to:
[0142] Determine the number of minority class samples N to be generated + ;
[0143] Find each minority class sample in the original training set S i=1,2,…,N + The number of K nearest neighbors of the i-th minority class sample is The proportion parameter Γ of each minority class sample is determined by the following formula i :
[0144]
[0145] In the formula, Z is the normalization factor, K is the number of neighbors of the i-th minority class sample;
[0146] The frequency g of each minority class sample being selected as the main sample is determined by the following formula i :
[0147] g i =Γ i ×N + ×SR
[0148] Where SR is the sampling rate.
[0149] In some embodiments, the database construction and oversampling module is further configured to:
[0150] When a minority class sample is selected as the main sample, the distance between the minority class sample and its K nearest neighbor samples is calculated.
[0151] The weight of the corresponding nearest neighbor sample is determined according to the distance between each nearest neighbor sample and the minority class sample.
[0152] Based on the weights, generating synthetic samples;
[0153] The synthetic samples are merged with the original data set to obtain a balanced data set.
[0154] It should be noted that the various system structures described in this embodiment belong to the same technical concept as the previously described method, and achieve the same technical effect through the same principle, which will not be repeated here.
[0155] An embodiment of the present invention further provides a readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the methods described in the above embodiments.
[0156] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.
Claims
1. A method for constructing a multimodal ovarian tumor database. It is characterized in that The method comprises: Acquire image modality data, table modality data, and text modality data, wherein the image modality data, table modality data, and text modality data all contain patient personal information, form a multimodality data with the same patient personal information, assign a classification label to the multimodality data, and the classification label is used to indicate the ovarian tumor category of the multimodality data; Processing the image modality data to obtain a standard image database; Processing the table modal data to obtain a standard table database; Processing the text modality data to obtain a standard text database; A multimodal ovarian tumor database is constructed, and different degrees of oversampling are performed on samples of different categories according to the distribution density of different samples in the multimodal ovarian tumor database. The multimodal ovarian tumor database includes a standard image database, a standard table database, and a standard text database. The samples include processed image modality data, processed table modality data, processed text modality data, or processed multimodality data.
2. The method according to claim 1, It is characterized in that The image modal data includes at least one ultrasound image of different patients, the table modal data includes clinical information tables of ovarian tumors of different patients, and the text modal data includes three characteristic fields contained in the clinical information tables of ovarian tumors of different patients, and the three characteristic fields are operation name, postoperative pathology and pathology summary.
3. The method according to claim 2, It is characterized in that The image modality data is processed to obtain a standard image database, specifically including: Based on the image modality data, at least one image modality data of a patient is named in the form of XN, where X represents the patient number, which is determined according to the patient's personal information, and N represents the serial number of the patient's image modality data; When one image modality data includes a plurality of ultrasound images, the image modality data is cropped to obtain a plurality of ultrasound images, each ultrasound image being used as one image modality data of the patient; In the case where the patient's privacy information exists in the image modality data, the image modality data is collected to remove the patient's privacy information in the image modality data; The named image modality data is input into an image feature extractor for extraction to obtain standard image data of a set image size. The standard image database is composed of a plurality of standard image data of different patients.
4. The method according to claim 3, It is characterized in that The named image modality data is input into the image feature extractor for extraction to obtain the standard image data of the set image size, including: According to the size W×H of the image input to the image feature extractor, the reduction factor k of width and length is determined 1 and k 2 , where W is the width of the image and H is the length of the image; The reduction factor k according to the width and length 1 and k 2 Determine the sampling interval as W / k 1 , H / k 2 , The image input to the image feature extraction is downsampled according to the sampling interval to output standard image data of a set image size.
5. The method according to claim 2, It is characterized in that The table modal data is processed to obtain a standard table database, specifically including: If there are missing data in the table data, the missing data is replaced by the average value of the field where the missing data is located; Directly save the numerical information in the table data, convert the data recorded in yes / no to 0 / 1 for saving, and remove the three characteristic fields contained in the table data; If the table data contains a patient identifier, de-identify the patient identifier, wherein the patient identifier includes the patient's name and surgery date, and use an unsequenced number as a unique identifier to mark the patient's personal information corresponding to the table data according to the patient's personal information; The table data is normalized to obtain standard table data, each characteristic field in the standard table data is a numerical value, and the standard table data of different patients constitute the standard table database. The normalization process is expressed as: In the formula, is the mean of the original data in the table data, is the standard deviation of the original data in the table data, x is the original data in the table data, x new is the normalized value, x i is the i-th original data in the table data, and n is the total number of original data in the table data.
6. The method according to claim 1, It is characterized in that The text modality data is processed to obtain a standard text database, specifically including: Determine the text length according to the number of words in the text modality data of each patient; Based on the text length, the text data in the text modality data is processed by adding uniform invalid characters to embed and fill or deleting useless characters to obtain standard text data. The length of the text data in the standard text data is equal to the text length. The standard text data of different patients constitute the standard text database.
7. The method according to claim 1, It is characterized in that According to the distribution density of different samples in the multimodal ovarian tumor database, different categories of samples are oversampled to different degrees, specifically including: Determine the number of minority class samples N to be generated + ; Find each minority class sample in the original training set S The number of K nearest neighbors of the i-th minority class sample is The proportion parameter Γ of each minority class sample is determined by the following formula i : In the formula, Z is the normalization factor, K is the number of neighbors of the i-th minority class sample; The frequency g of each minority class sample being selected as the main sample is determined by the following formula i : g i =C i ×N + ×SR Where SR is the sampling rate.
8. The method according to claim 7, It is characterized in that According to the distribution density of different samples in the multimodal ovarian tumor database, different categories of samples are oversampled to different degrees, and further comprising: When a minority class sample is selected as the main sample, the distance between the minority class sample and its K nearest neighbor samples is calculated. Determine the weight of the corresponding nearest neighbor sample according to the distance between each nearest neighbor sample and the minority class sample, Based on the weights, generating synthetic samples; The synthetic samples are merged with the original data set to obtain a balanced data set.
9. A multimodal ovarian tumor database construction system, It is characterized in that The system comprises: A data acquisition module is configured to acquire image modality data, table modality data, and text modality data, wherein the image modality data, table modality data, and text modality data all contain patient personal information, and to form a multimodal data with the same patient personal information, and to assign a classification label to the multimodal data, wherein the classification label is used to indicate the ovarian tumor category of the multimodal data; A first data processing module is configured to process the image modality data to obtain a standard image database; A second data processing module is configured to process the table modal data to obtain a standard table database; A third data processing module is configured to process the text modality data to obtain a standard text database; The database construction and oversampling module is configured to construct a multimodal ovarian tumor database, and to oversample different categories of samples to different degrees according to the distribution density of different samples in the multimodal ovarian tumor database. The multimodal ovarian tumor database includes a standard image database, a standard table database, and a standard text database. The samples include processed image modality data, processed table modality data, processed text modality data, or processed multimodality data.
10. A readable storage medium, It is characterized in that The readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 8.
