Multi-source heterogeneous data fusion method and system based on neural network

Abstract

The invention relates to the technical field of data fusion, and particularly discloses a multi-source heterogeneous data fusion method and system based on a neural network, and the method comprises the steps: carrying out the data amplification of all text medical data and all image medical data, and obtaining the amplification data of corresponding medical data; based on the label assignment results of all the text medical data and all the image medical data, obtaining a disease evaluation assignment initial model and a label assignment result corresponding to the amplification data; based on label assignment results of all the amplification data, a final model of disease evaluation assignment is obtained; and based on the disease evaluation assignment final model, obtaining a label assignment result of all text medical data and a label assignment result of all image medical data of each patient, and constructing a fusion judgment matrix of the corresponding patient. According to the invention, fusion analysis of the multi-source heterogeneous medical data of the patient is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and in particular to a multi-source heterogeneous data fusion method and system based on neural network. Background Art

[0002] At present, there are only 26,000 dermatologists in my country, and the number of patients visiting the hospital exceeds 200 million per year, which means that on average one dermatologist sees nearly 10,000 patients per year, and the service capacity is obviously insufficient. There are also great differences in the diagnostic ability and level of dermatologists, which will be affected by many factors (such as learning ability, information extraction speed, memory ability, physiological and psychological state, etc.). Moreover, it is not only the number of doctors and patients that is not proportional. What is more urgent is how to realize the fusion of multiple medical data of different structures of a single patient visiting the hospital, and realize the comprehensive judgment of the patient's condition. It has become a more important task. In this context, conducting research on auxiliary diagnosis and decision-making systems with artificial intelligence as the main line has very important social value and scientific significance for improving the work efficiency of doctors and improving the quality of life and medical experience of patients.

[0003] However, the existing multi-source heterogeneous data fusion method and system based on neural network only realizes the fusion storage of multi-source heterogeneous data of commodities obtained from different websites by preprocessing and text matching, and realizes the establishment of a knowledge base with strong integrity, accuracy and consistency, but cannot efficiently amplify and fuse text medical data and image medical data. For example, the publication number is "CN109308311A" and the patent name is "A Multi-Source Heterogeneous Data Fusion System", and its method includes the following steps: including a data acquisition module, a data preprocessing module, a matching module, a fusion module and an evaluation module, the data acquisition module is used to obtain multi-source heterogeneous data of commodities from different websites, the data preprocessing module is used to normalize the data and generate short texts, the normalization processing includes word segmentation and removal of stop words, the matching module is used to match the short texts, the fusion module is used to fuse the data based on the text matching results, and the evaluation module is used to evaluate the data fusion effect. The beneficial effects of the present invention are: providing a multi-source heterogeneous data fusion system, by preprocessing and text matching the data, realizing the fusion of multi-source heterogeneous data, and helping to establish a knowledge base with strong integrity, accuracy and consistency. However, this patent only achieves the fusion storage of multi-source heterogeneous data through data preprocessing and text matching, and establishes a knowledge base with strong integrity, accuracy and consistency, but it is unable to efficiently amplify and fuse the text medical data and image medical data.

[0004] Therefore, the present invention proposes a multi-source heterogeneous data fusion method and system based on neural network. Summary of the invention

[0005] The present invention provides a multi-source heterogeneous data fusion method and system based on a neural network, which are used to obtain the label assignment results of the amplified data of all text medical data and the label assignment results of all image medical data according to the label assignment results of all text medical data and the label assignment results of all image medical data, and label the amplified data to facilitate the subsequent retraining of the initial model for disease assessment assignment, and more accurately obtain the label assignment results of all text medical data and the label assignment results of all image medical data of the corresponding patient according to all text medical data, all image medical data and the final model for disease assessment assignment of the patient, and construct the fusion judgment matrix of the corresponding patient according to the label assignment results of all text medical data and the label assignment results of all image medical data of the patient, that is, to realize the amplification of text medical data and image medical data and the fusion of heterogeneous data, so as to facilitate the subsequent acquisition of the patient's diagnosis result.

[0006] The present invention provides a multi-source heterogeneous data fusion method based on a neural network, comprising:

[0007] S1: Perform data augmentation on all text medical data and all image medical data in the database to obtain augmented data of all text medical data and augmented data of all image medical data;

[0008] S2: Based on the label assignment results of all text medical data and all image medical data in the database, a neural network model is trained to obtain an initial model for disease assessment assignment;

[0009] S3: based on the label assignment results of all text medical data and the label assignment results of all image medical data, obtain the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data;

[0010] S4: Based on the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data, the initial model for disease assessment assignment is retrained to obtain a final model for disease assessment assignment;

[0011] S5: Based on all the text medical data, all the image medical data and the condition evaluation of each patient currently being diagnosed and treated, the final model is assigned to obtain the label assignment results of all the text medical data and all the image medical data of the corresponding patient;

[0012] S6: Based on the label assignment results of all text medical data and the label assignment results of all image medical data of each patient currently being diagnosed and treated, a fusion judgment matrix of the corresponding patient is constructed as the multi-source heterogeneous medical data fusion result of the corresponding patient.

[0013] Preferably, the multi-source heterogeneous data fusion method based on a neural network, S1: performing data amplification on all text medical data and all image medical data in the database to obtain amplified data of all text medical data and amplified data of all image medical data, including:

[0014] Obtain all text medical data and all image medical data in the hospital database, where the text medical data refers to medical record text and the image medical data refers to pathological examination images;

[0015] Interchanging each word in each text medical data with a corresponding replacement word in a preset word replacement table to obtain initial augmented data of each text medical data as first augmented data;

[0016] Divide the first augmented data corresponding to the text medical data into sentences based on the separation symbol in each text medical data to obtain all sentences of each first augmented data;

[0017] Counting the number of occurrences of each word in all the first amplified data and the total number of words in all the first amplified data, taking the number of occurrences of each word in all the first amplified data as a first vector element of the vector representation of the corresponding word, and determining a second vector element of the vector representation of the corresponding word based on the total number of words in all the first amplified data and the number of occurrences of each word in all the first amplified data;

[0018] The first vector element and the second vector element of each character are used as elements of the two-dimensional vector of the corresponding character to obtain the two-dimensional vector of each character, and the similarity between each sentence of each first amplified data and each sentence of each first amplified data except the corresponding first amplified data in all the first amplified data is obtained based on the two-dimensional vectors of all the characters and a preset first similarity algorithm, and all sentences in all the first amplified data except the current first amplified data whose similarity is greater than a preset text similarity threshold are used as replaceable sentences of the corresponding sentence, and all replaceable sentences of each sentence of each first amplified data are obtained;

[0019] Replace at least one sentence in each first amplification data with any corresponding replaceable sentence to obtain all amplification data of all first amplification data;

[0020] Data augmentation is performed on all medical image data to obtain augmented data of all medical image data.

[0021] Preferably, the multi-source heterogeneous data fusion method based on a neural network performs data amplification on all medical image data to obtain amplified data of all medical image data, including:

[0022] Randomly rotate and flip each image medical data left and right to obtain a plurality of initial amplified data of each image medical data as a plurality of second amplified data;

[0023] Taking the average of the three-channel pixel values ​​of each pixel point in each second amplified data as the grayscale value of the corresponding pixel point, obtaining the grayscale values ​​of all the pixels in each second amplified data;

[0024] Subtract the grayscale value of each pixel in each second amplified data from the grayscale values ​​of all adjacent pixels to obtain multiple difference values ​​for each pixel, and compare all the difference values ​​of each pixel obtained with the preset pixel difference value. If the difference of the pixel is greater than the preset pixel difference value, the corresponding pixel is taken as an edge pixel; otherwise, the corresponding pixel is taken as a tissue pixel, and all edge pixels and all tissue pixels of each second amplified data are obtained;

[0025] All amplification data of all second amplification data are obtained based on all edge pixels and all tissue pixels of each second amplification data.

[0026] Preferably, the multi-source heterogeneous data fusion method based on a neural network obtains all amplification data of all second amplification data based on all edge pixels and all tissue pixels of each second amplification data, including:

[0027] Connecting all edge pixels in each second amplified data that are in an adjacent relationship and whose number of continuously adjacent pixels is greater than the preset number of first pixels to obtain a texture connection graph corresponding to the second amplified data, and using an image composed of all tissue pixels closed and surrounded by the corresponding texture connection graph in each second amplified data as a pathological tissue image to obtain all pathological tissue images of each second amplified data;

[0028] Obtaining texture similarity between a texture connection graph of each second augmented data and a texture connection graph of each other second augmented data except the corresponding second augmented data based on a preset second similarity algorithm, and taking all second augmented data other than the corresponding second augmented data whose texture similarity is greater than a preset texture similarity threshold as all replaceable image medical data corresponding to the second augmented data;

[0029] Based on a preset third similarity algorithm, the similarity between each pathological tissue image of each second amplified data and each pathological tissue image of each replaceable image medical data corresponding to the second amplified data is obtained, and all pathological tissue images with a similarity greater than a preset tissue similarity threshold among all pathological tissue images of all replaceable image medical data corresponding to the second amplified data are used as replaceable pathological tissue images of the corresponding pathological tissue image, and at least one pathological tissue image in each second amplified data is replaced with any corresponding replaceable pathological tissue image to obtain all amplified data of all second amplified data.

[0030] Preferably, the multi-source heterogeneous data fusion method based on a neural network, S2: training a neural network model based on the label assignment results of all text medical data and the label assignment results of all image medical data in the database to obtain an initial model for disease assessment assignment, including:

[0031] Obtaining label assignment results of all pre-labeled text medical data and label assignment results of all image medical data;

[0032] All text medical data and the label assignment results of all text medical data and all image medical data and the label assignment results of all image medical data are used as training samples to train the neural network model to obtain an initial model for disease assessment assignment.

[0033] Preferably, the multi-source heterogeneous data fusion method based on a neural network, S3: based on the label assignment results of all text medical data and the label assignment results of all image medical data, obtaining the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data, including:

[0034] Acquire all text medical data or all image medical data related to each augmented data;

[0035] When the difference between the maximum and minimum values ​​of the label assignment results of all text medical data or all image medical data related to each augmented data is a number in the first preset value set, the label assignment result of the original text medical data or original image medical data related to each augmented data is used as the label assignment result of the corresponding augmented data;

[0036] When the difference between the maximum and minimum values ​​of the label assignment results of all text medical data or all image medical data related to each augmented data is a number in the second preset value set, the average value of the label assignment results of all text medical data or all image medical data related to each augmented data is used as the label assignment result of the corresponding augmented data;

[0037] The augmented data is augmented data of image medical data or augmented data of text medical data.

[0038] Preferably, the multi-source heterogeneous data fusion method based on a neural network, S5: based on all text medical data, all image medical data and condition evaluation of each patient currently diagnosed and treated, the final model is assigned to obtain the label assignment results of all text medical data and all image medical data of the corresponding patient, including:

[0039] Each text medical data or each image medical data received from each patient is input into the final model for condition assessment and assignment respectively to obtain the label assignment results of all text medical data and all image medical data of the corresponding patient.

[0040] Preferably, the multi-source heterogeneous data fusion method based on a neural network, S6: constructing a fusion judgment matrix corresponding to the patient based on the label assignment results of all text medical data and the label assignment results of all image medical data of each patient currently diagnosed and treated as the multi-source heterogeneous medical data fusion result of the patient, including:

[0041] Performing ordinal definition starting from 1 for all text medical data and all image medical data of each patient, and obtaining ordinal definition results of all text medical data and all image medical data;

[0042] Based on the label assignment results and ordinal definition results of all text medical data of each patient and the label assignment results and ordinal definition results of all image medical data, the fusion judgment matrix of the corresponding patient is constructed, which is:

[0043]

[0044] Among them, P is the patient's fusion judgment matrix, r1 is the label assignment result of the text medical data of the patient with ordinal number 1, r2 is the label assignment result of the text medical data of the patient with ordinal number 2, and r n is the label assignment result of the text medical data of the patient with ordinal number n, d1 is the label assignment result of the image medical data of the patient with ordinal number 1, d2 is the label assignment result of the image medical data of the patient with ordinal number 2, and d m is the label assignment result of the image medical data of the patient with ordinal number m, n is the total number of text medical data of the patient, and m is the total number of image medical data of the patient;

[0045] The patient's fusion judgment matrix is ​​regarded as the fusion result of the patient's multi-source heterogeneous medical data.

[0046] Preferably, the multi-source heterogeneous data fusion method based on a neural network, S6: constructing a fusion judgment matrix corresponding to the patient based on the label assignment results of all text medical data and the label assignment results of all image medical data of each patient currently diagnosed and treated, as the multi-source heterogeneous medical data fusion result of the patient, further comprising:

[0047] The rank of the patient's fusion judgment matrix is ​​numerically compared with the preset judgment threshold. If the rank of the fusion judgment matrix is ​​greater than the preset judgment threshold, the condition requiring intervention is taken as the fusion judgment result of the corresponding patient's condition. Otherwise, the condition requiring no intervention is taken as the fusion judgment result of the corresponding patient's condition.

[0048] The present invention provides a multi-source heterogeneous data fusion system based on a neural network, which is used to execute any one of the multi-source heterogeneous data fusion methods based on a neural network in Embodiments 1 to 9, including:

[0049] An amplification module, used to perform data amplification on all text medical data and all image medical data in the database to obtain amplified data of all text medical data and amplified data of all image medical data;

[0050] A training module is used to train a neural network model based on the label assignment results of all text medical data and all image medical data in the database to obtain an initial model for disease assessment assignment;

[0051] An assignment module, used for obtaining label assignment results of augmented data of all text medical data and label assignment results of augmented data of all image medical data based on label assignment results of all text medical data and label assignment results of all image medical data;

[0052] A retraining module, used to retrain the initial model for disease assessment assignment based on the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data, to obtain a final model for disease assessment assignment;

[0053] An acquisition module is used to assign a final model based on all text medical data, all image medical data and condition evaluation of each patient currently being diagnosed and treated, and obtain label assignment results of all text medical data and all image medical data of the corresponding patient;

[0054] The judgment module is used to construct a fusion judgment matrix of the corresponding patient based on the label assignment results of all text medical data and all image medical data of each patient currently being diagnosed and treated, as the multi-source heterogeneous medical data fusion result of the corresponding patient.

[0055] The beneficial effects of the present invention compared with the prior art are as follows: based on the label assignment results of all text medical data and the label assignment results of all image medical data, the label assignment results of the amplified data of all text medical data and the label assignment results of the amplified data of all image medical data are obtained, and the amplified data are labeled to facilitate the subsequent retraining of the initial model for disease assessment assignment, and based on all the patient's text medical data, all the image medical data and the final model for disease assessment assignment, the label assignment results of all the text medical data and the label assignment results of all the image medical data of the corresponding patient are more accurately obtained, and the fusion judgment matrix of the corresponding patient is constructed based on the label assignment results of all the text medical data and the label assignment results of all the image medical data of the patient, that is, the amplification of the text medical data and the image medical data and the fusion of heterogeneous data are realized, so as to facilitate the subsequent acquisition of the patient's diagnosis result.

[0056] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structure specifically pointed out in the written application document.

[0057] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0059] Figure 1 This is a flow chart of a multi-source heterogeneous data fusion method based on a neural network in an embodiment of the present invention;

[0060] Figure 2 Schematic diagram of a multi-source heterogeneous data fusion system based on a neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0062] Embodiment 1:

[0063] The present invention provides a multi-source heterogeneous data fusion method based on neural network. Figure 1 ,include:

[0064] S1: Perform data augmentation on all text medical data and all image medical data in the database to obtain augmented data of all text medical data and augmented data of all image medical data;

[0065] S2: Based on the label assignment results of all text medical data and all image medical data in the database, a neural network model is trained to obtain an initial model for disease assessment assignment;

[0066] S3: based on the label assignment results of all text medical data and the label assignment results of all image medical data, obtain the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data;

[0067] S4: Based on the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data, the initial model for disease assessment assignment is retrained to obtain a final model for disease assessment assignment;

[0068] S5: Based on all the text medical data, all the image medical data and the condition evaluation of each patient currently being diagnosed and treated, the final model is assigned to obtain the label assignment results of all the text medical data and all the image medical data of the corresponding patient;

[0069] S6: Based on the label assignment results of all text medical data and the label assignment results of all image medical data of each patient currently being diagnosed and treated, a fusion judgment matrix of the corresponding patient is constructed as the multi-source heterogeneous medical data fusion result of the corresponding patient.

[0070] In this embodiment, the database is a hospital database that provides data for multi-source heterogeneous data fusion.

[0071] In this embodiment, data augmentation is a process of performing sentence replacement or image composition replacement on all existing text medical data and all image medical data in the database to obtain new text medical data or image medical data.

[0072] In this embodiment, the augmented data of the text medical data is the expanded and increased text medical data obtained by performing sentence replacement on all existing text medical data in the database.

[0073] In this embodiment, the augmented data of the image medical data is the expanded and increased image medical data obtained by replacing the image components of all existing image medical data in the database.

[0074] In this embodiment, the label assignment result is the value assigned by the expert in advance to each text medical data or each image medical data in the database after evaluating the condition, wherein the label assignment results include: 1, 2, 3, 4, 5, which respectively represent the degree of the condition reflected by the corresponding text medical data or image medical data, and the label assignment result indicates that the more serious the condition is.

[0075] In this embodiment, the training of the neural network model is a process of obtaining an initial model for disease assessment and assignment based on existing conventional methods. Conventional methods include: early stopping parameter setting: by adjusting the early stopping parameter, the complexity and convergence speed of the model are controlled. A smaller early stopping parameter value can improve the model accuracy, but may cause overfitting; a larger early stopping parameter value can prevent overfitting, but may sacrifice some accuracy.

[0076] In this embodiment, the initial model for disease assessment assignment uses all text medical data and all image medical data of each patient as neural network model inputs, and the label assignment results of all text medical data and all image medical data of each patient are used as neural network model outputs. After model training is performed based on existing conventional methods, a trained model is obtained.

[0077] In this embodiment, the label assignment result of the amplification data is the label assignment result of the corresponding amplification data obtained based on all text medical data or all image medical data related to each amplification data.

[0078] In this embodiment, the final model for disease assessment assignment is a new training model obtained by secondary training (retraining) of the initial model for disease assessment assignment based on the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data.

[0079] In this embodiment, the patient is a patient who comes to the hospital for medical treatment at the current moment, and the hospital information system receives newly uploaded text medical data or image medical data.

[0080] In this embodiment, the label assignment result of all text medical data of the patient is the output value of the final model of condition assessment assignment obtained by taking all text medical data of the patient as the input of the final model of condition assessment assignment.

[0081] In this embodiment, the label assignment result of all the medical image data of the patient is the output value of the final model of condition assessment assignment obtained by taking all the medical image data of the patient as the input of the final model of condition assessment assignment.

[0082] In this embodiment, the fusion judgment matrix is ​​a matrix constructed by using the label assignment results of all text medical data and all image medical data of the patient as matrix elements, which generally reflects the results of the fusion of all text medical data and all image medical data of the patient.

[0083] In this embodiment, the multi-source heterogeneous medical data fusion result is a fusion judgment matrix obtained after fusion analysis of the multi-source heterogeneous medical data of the patient.

[0084] The beneficial effects of the above technology are: based on the label assignment results of all text medical data and the label assignment results of all image medical data, the label assignment results of the amplified data of all text medical data and the label assignment results of the amplified data of all image medical data are obtained, and the amplified data are labeled to facilitate the subsequent retraining of the initial model for condition assessment assignment, and based on all the patient's text medical data, all image medical data and the final model for condition assessment assignment, the label assignment results of all text medical data and the label assignment results of all image medical data of the corresponding patient are more accurately obtained, and the fusion judgment matrix of the corresponding patient is constructed based on the label assignment results of all text medical data and the label assignment results of all image medical data of the patient, that is, the amplification of text medical data and image medical data and the fusion of heterogeneous data are realized, so as to facilitate the subsequent acquisition of the patient's diagnosis result.

[0085] Embodiment 2:

[0086] Based on Example 1, a multi-source heterogeneous data fusion method based on a neural network, S1: performing data amplification on all text medical data and all image medical data in a database to obtain amplified data of all text medical data and amplified data of all image medical data, including:

[0087] Obtain all text medical data and all image medical data in the hospital database, where the text medical data refers to medical record text and the image medical data refers to pathological examination images;

[0088] Interchanging each word in each text medical data with a corresponding replacement word in a preset word replacement table to obtain initial augmented data of each text medical data as first augmented data;

[0089] Divide the first augmented data corresponding to the text medical data into sentences based on the separation symbol in each text medical data to obtain all sentences of each first augmented data;

[0090] Counting the number of occurrences of each word in all the first amplified data and the total number of words in all the first amplified data, taking the number of occurrences of each word in all the first amplified data as a first vector element of the vector representation of the corresponding word, and determining a second vector element of the vector representation of the corresponding word based on the total number of words in all the first amplified data and the number of occurrences of each word in all the first amplified data;

[0091] The first vector element and the second vector element of each character are used as elements of the two-dimensional vector of the corresponding character to obtain the two-dimensional vector of each character, and the similarity between each sentence of each first amplified data and each sentence of each first amplified data except the corresponding first amplified data in all the first amplified data is obtained based on the two-dimensional vectors of all the characters and a preset first similarity algorithm, and all sentences in all the first amplified data except the current first amplified data whose similarity is greater than a preset text similarity threshold are used as replaceable sentences of the corresponding sentence, and all replaceable sentences of each sentence of each first amplified data are obtained;

[0092] Replace at least one sentence in each first amplification data with any corresponding replaceable sentence to obtain all amplification data of all first amplification data;

[0093] Data augmentation is performed on all medical image data to obtain augmented data of all medical image data.

[0094] In this embodiment, the medical record text is the text portion of the medical records of all patients in the database, such as description of the patient's condition, doctor's treatment advice, etc.

[0095] In this embodiment, the pathological examination images are examination images of all patients in the database, such as CT images, ultrasound images, etc.

[0096] In this embodiment, the preset word replacement table is a pre-set word replacement table that contains all words that may be involved in the text medical data. For example, gastric cancer and gastric mucosal epithelial malignant tumor are a group of replacement words in the word replacement table.

[0097] In this embodiment, the first augmented data is new augmented data of each text medical data obtained by exchanging each word in each text medical data with a corresponding replacement word in a preset word replacement table.

[0098] In this embodiment, the separator is a separator in each text medical data, such as a comma, a period, etc.

[0099] In this embodiment, sentence division is a process of dividing the first augmented data corresponding to the text medical data into sentences based on the separation symbols in each text medical data.

[0100] In this embodiment, the first vector element is the first element in the two-dimensional vector corresponding to each word in the first amplified data.

[0101] In this embodiment, the second vector element is the second element in the two-dimensional vector corresponding to each word in the first amplified data.

[0102] In this embodiment, the preset first similarity algorithm is a preset cosine similarity algorithm, and when the number of words in two sentences is inconsistent, the sentence with longer words is deleted (the words with the smallest word weight are deleted first, and the word weight is determined according to a preset word weight list).

[0103] In this embodiment, the preset text similarity threshold is a preset value that the similarity between the replaceable sentence of each sentence of the first augmented data and the corresponding sentence needs to be greater than.

[0104] In this embodiment, the replaceable sentence is a sentence that can be exchanged with the corresponding sentence of the first amplification data.

[0105] The beneficial effects of the above technology are: data amplification is performed on all text medical data and all image medical data in the database to obtain amplified data of all text medical data and amplified data of all image medical data, and the text medical data and image medical data are amplified more efficiently to facilitate subsequent retraining of the initial model for disease assessment assignment and reduce the confidence interval of the initial model for disease assessment assignment. This embodiment provides a specific method for obtaining amplified data of all text medical data based on all text medical data in the database.

[0106] Embodiment 3:

[0107] Based on Example 2, a multi-source heterogeneous data fusion method based on a neural network is used to perform data amplification on all medical image data to obtain amplified data of all medical image data, including:

[0108] Randomly rotate and flip each image medical data left and right to obtain a plurality of initial amplified data of each image medical data as a plurality of second amplified data;

[0109] Taking the average of the three-channel pixel values ​​of each pixel point in each second amplified data as the grayscale value of the corresponding pixel point, obtaining the grayscale values ​​of all the pixels in each second amplified data;

[0110] Subtract the grayscale value of each pixel in each second amplified data from the grayscale values ​​of all adjacent pixels to obtain multiple difference values ​​for each pixel, and compare all the difference values ​​of each pixel obtained with the preset pixel difference value. If the difference of the pixel is greater than the preset pixel difference value, the corresponding pixel is taken as an edge pixel; otherwise, the corresponding pixel is taken as a tissue pixel, and all edge pixels and all tissue pixels of each second amplified data are obtained;

[0111] All amplification data of all second amplification data are obtained based on all edge pixels and all tissue pixels of each second amplification data.

[0112] In this embodiment, the random rotation is to rotate the image medical data at a random angle with the center point of the image as the origin.

[0113] In this embodiment, left-right flipping refers to horizontally flipping the image medical data.

[0114] In this embodiment, the second augmented data is new augmented data of each image medical data obtained by randomly rotating and flipping each image medical data left and right.

[0115] In this embodiment, the three-channel pixel values ​​are the pixel values ​​of the R channel, the G channel, and the B channel of each pixel point in the image medical data.

[0116] In this embodiment, the grayscale value is a value reflecting the brightness value of each pixel in the second amplified data.

[0117] In this embodiment, the preset pixel difference value is a pre-set pixel difference value used to determine edge pixel points and tissue pixel points.

[0118] In this embodiment, the edge pixel points are the pixel points that separate different pathological tissues in the second amplified data.

[0119] In this embodiment, the tissue pixels are all the remaining pixels in the second amplified data except the edge pixels.

[0120] The beneficial effects of the above technology are: all second amplification data are obtained based on all image medical data, and all edge pixels and all tissue pixels of the second amplification data are more accurately obtained based on the three-channel pixel values ​​of each pixel in the second amplification data, which facilitates the subsequent acquisition of all amplification data of all second amplification data.

[0121] Embodiment 4:

[0122] On the basis of Example 3, a multi-source heterogeneous data fusion method based on a neural network obtains all amplification data of all second amplification data based on all edge pixels and all tissue pixels of each second amplification data, including:

[0123] Connecting all edge pixels in each second amplified data that are in an adjacent relationship and whose number of continuously adjacent pixels is greater than the preset number of first pixels to obtain a texture connection graph corresponding to the second amplified data, and using an image composed of all tissue pixels closed and surrounded by the corresponding texture connection graph in each second amplified data as a pathological tissue image to obtain all pathological tissue images of each second amplified data;

[0124] Obtaining texture similarity between a texture connection graph of each second augmented data and a texture connection graph of each other second augmented data except the corresponding second augmented data based on a preset second similarity algorithm, and taking all second augmented data other than the corresponding second augmented data whose texture similarity is greater than a preset texture similarity threshold as all replaceable image medical data corresponding to the second augmented data;

[0125] Based on a preset third similarity algorithm, the similarity between each pathological tissue image of each second amplified data and each pathological tissue image of each replaceable image medical data corresponding to the second amplified data is obtained, and all pathological tissue images with a similarity greater than a preset tissue similarity threshold among all pathological tissue images of all replaceable image medical data corresponding to the second amplified data are used as replaceable pathological tissue images of the corresponding pathological tissue image, and at least one pathological tissue image in each second amplified data is replaced with any corresponding replaceable pathological tissue image to obtain all amplified data of all second amplified data.

[0126] In this embodiment, the preset first number of pixels is the number of pixels of the texture connection diagram that is preset and used to obtain the second augmented data.

[0127] In this embodiment, the texture line graph is an image graph containing line segments in the second amplified data for distinguishing different pathological tissues.

[0128] In this embodiment, the pathological tissue image is an image formed by all tissue pixels enclosed by the texture connection graph, and includes an image region where a single pathological tissue in the second amplified data is located.

[0129] In this embodiment, the preset second similarity algorithm is a preset algorithm used to calculate the texture similarity between two texture connection graphs, such as a correlation filter matching algorithm.

[0130] In this embodiment, the preset texture similarity threshold is a preset threshold for obtaining texture similarities of all replaceable image medical data of the second augmented data.

[0131] In this embodiment, the preset third similarity algorithm is a pre-set algorithm for calculating the similarity between each pathological tissue image of the second amplified data and each pathological tissue image of each replaceable image medical data corresponding to the second amplified data. In this embodiment, the preset third similarity algorithm is an algorithm for calculating the similarity of the external connection graphs of two pathological tissue images, such as a cosine similarity algorithm.

[0132] In this embodiment, the preset tissue similarity threshold is a preset similarity threshold of a replaceable pathological tissue image for each pathological tissue image used to obtain the second amplified data.

[0133] In this embodiment, the replaceable pathological tissue image is a pathological tissue image that can be exchanged with the corresponding pathological tissue image of the second amplified data.

[0134] The beneficial effects of the above technology are: obtaining the texture connection map of the second amplified data and all pathological tissue images based on all edge pixels and all tissue pixels of the second amplified data, and obtaining all amplified data of all second amplified data based on the texture connection map of the second amplified data and all pathological tissue images. This embodiment provides a specific method for obtaining the amplified data of all image medical data based on all image medical data in a database.

[0135] Embodiment 5:

[0136] Based on Example 1, a multi-source heterogeneous data fusion method based on a neural network, S2: training a neural network model based on the label assignment results of all text medical data and the label assignment results of all image medical data in the database to obtain an initial model for disease assessment assignment, including:

[0137] Obtaining label assignment results of all pre-labeled text medical data and label assignment results of all image medical data;

[0138] All text medical data and the label assignment results of all text medical data and all image medical data and the label assignment results of all image medical data are used as training samples to train the neural network model to obtain an initial model for disease assessment assignment.

[0139] In this embodiment, the pre-labeling refers to numerical values ​​that are labeled by experts after evaluating the condition of each text medical data or each image medical data in the database.

[0140] The beneficial effect of the above technology is: an initial model for disease assessment assignment is obtained based on all text medical data and the label assignment results of all text medical data and all image medical data, which facilitates the subsequent acquisition of the label assignment results of all text medical data and the label assignment results of all image medical data for each patient.

[0141] Embodiment 6:

[0142] Based on Example 1, a multi-source heterogeneous data fusion method based on a neural network, S3: based on the label assignment results of all text medical data and the label assignment results of all image medical data, obtain the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data, including:

[0143] Acquire all text medical data or all image medical data related to each augmented data;

[0144] When the difference between the maximum and minimum values ​​of the label assignment results of all text medical data or all image medical data related to each augmented data is a number in the first preset value set, the label assignment result of the original text medical data or original image medical data related to each augmented data is used as the label assignment result of the corresponding augmented data;

[0145] When the difference between the maximum and minimum values ​​of the label assignment results of all text medical data or all image medical data related to each augmented data is a number in the second preset value set, the average value of the label assignment results of all text medical data or all image medical data related to each augmented data is used as the label assignment result of the corresponding augmented data;

[0146] The augmented data is augmented data of image medical data or augmented data of text medical data.

[0147] In this embodiment, the first preset value set is a first set of differences between the maximum value and the minimum value of the preset label assignment results. In this embodiment, the first preset value set includes 3 and 4.

[0148] In this embodiment, the second preset value set is a second set of differences between the maximum value and the minimum value of the preset label assignment results. In this embodiment, the first preset value set includes 0, 1, and 2.

[0149] The beneficial effect of the above technology is: based on the label assignment results of all text medical data and the label assignment results of all image medical data, the label assignment results of the amplified data of all text medical data and the label assignment results of the amplified data of all image medical data are obtained, and the amplified data are labeled to facilitate the subsequent retraining of the initial model for disease assessment assignment.

[0150] Embodiment 7:

[0151] Based on Example 1, a multi-source heterogeneous data fusion method based on a neural network, S5: Based on all text medical data, all image medical data and condition evaluation of each patient currently diagnosed and treated, a final model is assigned to obtain label assignment results of all text medical data and all image medical data of the corresponding patient, including:

[0152] Each text medical data or each image medical data received from each patient is input into the final model for condition assessment and assignment respectively to obtain the label assignment results of all text medical data and all image medical data of the corresponding patient.

[0153] The beneficial effects of the above technology are: by assigning the final model based on all the patient's text medical data, all the image medical data and the condition assessment, the label assignment results of all the corresponding patient's text medical data and all the image medical data can be obtained more accurately, which is convenient for the subsequent construction of the fusion judgment matrix.

[0154] Embodiment 8:

[0155] Based on Example 1, the multi-source heterogeneous data fusion method based on a neural network, S6: constructing a fusion judgment matrix corresponding to the patient based on the label assignment results of all text medical data and the label assignment results of all image medical data of each patient currently diagnosed and treated as the multi-source heterogeneous medical data fusion result of the patient, including:

[0156] Performing ordinal definition starting from 1 for all text medical data and all image medical data of each patient, and obtaining ordinal definition results of all text medical data and all image medical data;

[0157] Based on the label assignment results and ordinal definition results of all text medical data of each patient and the label assignment results and ordinal definition results of all image medical data, the fusion judgment matrix of the corresponding patient is constructed, which is:

[0158]

[0159] Among them, P is the patient's fusion judgment matrix, r1 is the label assignment result of the text medical data of the patient with ordinal number 1, r2 is the label assignment result of the text medical data of the patient with ordinal number 2, and r n is the label assignment result of the text medical data of the patient with ordinal number n, d1 is the label assignment result of the image medical data of the patient with ordinal number 1, d2 is the label assignment result of the image medical data of the patient with ordinal number 2, and d m is the label assignment result of the image medical data of the patient with ordinal number m, n is the total number of text medical data of the patient, and m is the total number of image medical data of the patient;

[0160] The patient's fusion judgment matrix is ​​regarded as the fusion result of the patient's multi-source heterogeneous medical data.

[0161] The beneficial effect of the above technology is: constructing a fusion judgment matrix corresponding to the patient based on the label assignment results of all text medical data of the patient and the label assignment results of all image medical data, so as to facilitate more accurate subsequent fusion judgment results of the patient's condition.

[0162] Embodiment 9:

[0163] On the basis of Example 1, the multi-source heterogeneous data fusion method based on a neural network, S6: constructing a fusion judgment matrix corresponding to the patient based on the label assignment results of all text medical data and the label assignment results of all image medical data of each patient currently diagnosed and treated as the multi-source heterogeneous medical data fusion result of the patient, further comprising:

[0164] The rank of the patient's fusion judgment matrix is ​​numerically compared with the preset judgment threshold. If the rank of the fusion judgment matrix is ​​greater than the preset judgment threshold, the condition requiring intervention is taken as the fusion judgment result of the corresponding patient's condition. Otherwise, the condition requiring no intervention is taken as the fusion judgment result of the corresponding patient's condition.

[0165] In this embodiment, the preset judgment threshold is a threshold set in advance and used to obtain the fusion judgment result of the patient's condition.

[0166] The beneficial effect of the above technology is: it is clarified that the fusion judgment result of the patient's condition can be obtained according to the rank of the patient's fusion judgment matrix.

[0167] Embodiment 10:

[0168] The present invention provides a multi-source heterogeneous data fusion system based on a neural network, which is used to execute any one of the multi-source heterogeneous data fusion methods based on a neural network in embodiments 1 to 9, referring to Figure 2 ,include:

[0169] An amplification module, used to perform data amplification on all text medical data and all image medical data in the database to obtain amplified data of all text medical data and amplified data of all image medical data;

[0170] A training module is used to train a neural network model based on the label assignment results of all text medical data and all image medical data in the database to obtain an initial model for disease assessment assignment;

[0171] An assignment module, used for obtaining label assignment results of augmented data of all text medical data and label assignment results of augmented data of all image medical data based on label assignment results of all text medical data and label assignment results of all image medical data;

[0172] A retraining module, used to retrain the initial model for disease assessment assignment based on the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data, to obtain a final model for disease assessment assignment;

[0173] An acquisition module is used to assign a final model based on all text medical data, all image medical data and condition evaluation of each patient currently being diagnosed and treated, and obtain label assignment results of all text medical data and all image medical data of the corresponding patient;

[0174] The judgment module is used to construct a fusion judgment matrix of the corresponding patient based on the label assignment results of all text medical data and all image medical data of each patient currently being diagnosed and treated, as the multi-source heterogeneous medical data fusion result of the corresponding patient.

[0175] The beneficial effects of the above technology are: based on the label assignment results of all text medical data and the label assignment results of all image medical data, the label assignment results of the amplified data of all text medical data and the label assignment results of the amplified data of all image medical data are obtained, and the amplified data are labeled to facilitate the subsequent retraining of the initial model for condition assessment assignment, and based on all the patient's text medical data, all image medical data and the final model for condition assessment assignment, the label assignment results of all text medical data and the label assignment results of all image medical data of the corresponding patient are more accurately obtained, and the fusion judgment matrix of the corresponding patient is constructed based on the label assignment results of all text medical data and the label assignment results of all image medical data of the patient, that is, the amplification of text medical data and image medical data and the fusion of heterogeneous data are realized, so as to facilitate the subsequent acquisition of the patient's diagnosis result.

[0176] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-source heterogeneous data fusion method based on neural network, It is characterized in that include: S1: Perform data augmentation on all text medical data and all image medical data in the database to obtain augmented data of all text medical data and augmented data of all image medical data; S2: Based on the label assignment results of all text medical data and all image medical data in the database, a neural network model is trained to obtain an initial model for disease assessment assignment; S3: based on the label assignment results of all text medical data and the label assignment results of all image medical data, obtain the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data; S4: Based on the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data, the initial model for disease assessment assignment is retrained to obtain a final model for disease assessment assignment; S5: Based on all the text medical data, all the image medical data and the condition evaluation of each patient currently being diagnosed and treated, the final model is assigned to obtain the label assignment results of all the text medical data and all the image medical data of the corresponding patient; S6: Based on the label assignment results of all text medical data and the label assignment results of all image medical data of each patient currently being diagnosed and treated, a fusion judgment matrix of the corresponding patient is constructed as the multi-source heterogeneous medical data fusion result of the corresponding patient.

2. According to the multi-source heterogeneous data fusion method based on neural network in claim 1, It is characterized in that S1: Perform data augmentation on all text medical data and all image medical data in the database to obtain augmented data of all text medical data and augmented data of all image medical data, including: Obtain all text medical data and all image medical data in the hospital database, where the text medical data refers to medical record text and the image medical data refers to pathological examination images; Interchanging each word in each text medical data with a corresponding replacement word in a preset word replacement table to obtain initial augmented data of each text medical data as first augmented data; Divide the first augmented data corresponding to the text medical data into sentences based on the separation symbol in each text medical data to obtain all sentences of each first augmented data; Counting the number of occurrences of each word in all the first amplified data and the total number of words in all the first amplified data, taking the number of occurrences of each word in all the first amplified data as a first vector element of the vector representation of the corresponding word, and determining a second vector element of the vector representation of the corresponding word based on the total number of words in all the first amplified data and the number of occurrences of each word in all the first amplified data; The first vector element and the second vector element of each character are used as elements of the two-dimensional vector of the corresponding character to obtain the two-dimensional vector of each character, and the similarity between each sentence of each first amplified data and each sentence of each first amplified data except the corresponding first amplified data in all the first amplified data is obtained based on the two-dimensional vectors of all the characters and a preset first similarity algorithm, and all sentences in all the first amplified data except the current first amplified data whose similarity is greater than a preset text similarity threshold are used as replaceable sentences of the corresponding sentence, and all replaceable sentences of each sentence of each first amplified data are obtained; Replace at least one sentence in each first amplification data with any corresponding replaceable sentence to obtain all amplification data of all first amplification data; Data augmentation is performed on all medical image data to obtain augmented data of all medical image data.

3. According to the multi-source heterogeneous data fusion method based on neural network in claim 2, It is characterized in that Perform data augmentation on all medical image data to obtain augmented data of all medical image data, including: Randomly rotate and flip each image medical data left and right to obtain a plurality of initial amplified data of each image medical data as a plurality of second amplified data; Taking the average of the three-channel pixel values ​​of each pixel point in each second amplified data as the grayscale value of the corresponding pixel point, obtaining the grayscale values ​​of all the pixels in each second amplified data; Subtract the grayscale value of each pixel in each second amplified data from the grayscale values ​​of all adjacent pixels to obtain multiple difference values ​​for each pixel, and compare all the difference values ​​of each pixel obtained with the preset pixel difference value. If the difference of the pixel is greater than the preset pixel difference value, the corresponding pixel is taken as an edge pixel; otherwise, the corresponding pixel is taken as a tissue pixel, and all edge pixels and all tissue pixels of each second amplified data are obtained; All amplification data of all second amplification data are obtained based on all edge pixels and all tissue pixels of each second amplification data.

4. According to the multi-source heterogeneous data fusion method based on neural network in claim 3, It is characterized in that Obtaining all amplified data of all second amplified data based on all edge pixels and all tissue pixels of each second amplified data includes: Connecting all edge pixels in each second amplified data that are in an adjacent relationship and whose number of continuously adjacent pixels is greater than the preset number of first pixels to obtain a texture connection graph corresponding to the second amplified data, and using an image composed of all tissue pixels closed and surrounded by the corresponding texture connection graph in each second amplified data as a pathological tissue image to obtain all pathological tissue images of each second amplified data; Obtaining texture similarity between a texture connection graph of each second augmented data and a texture connection graph of each other second augmented data except the corresponding second augmented data based on a preset second similarity algorithm, and taking all second augmented data other than the corresponding second augmented data whose texture similarity is greater than a preset texture similarity threshold as all replaceable image medical data corresponding to the second augmented data; Based on a preset third similarity algorithm, the similarity between each pathological tissue image of each second amplified data and each pathological tissue image of each replaceable image medical data corresponding to the second amplified data is obtained, and all pathological tissue images with a similarity greater than a preset tissue similarity threshold among all pathological tissue images of all replaceable image medical data corresponding to the second amplified data are used as replaceable pathological tissue images of the corresponding pathological tissue image, and at least one pathological tissue image in each second amplified data is replaced with any corresponding replaceable pathological tissue image to obtain all amplified data of all second amplified data.

5. According to the multi-source heterogeneous data fusion method based on neural network in claim 1, It is characterized in that S2: Based on the label assignment results of all text medical data and all image medical data in the database, a neural network model is trained to obtain an initial model for disease assessment assignment, including: Obtaining label assignment results of all pre-labeled text medical data and label assignment results of all image medical data; All text medical data and the label assignment results of all text medical data and all image medical data and the label assignment results of all image medical data are used as training samples to train the neural network model to obtain an initial model for disease assessment assignment.

6. According to the multi-source heterogeneous data fusion method based on neural network in claim 1, It is characterized in that S3: Based on the label assignment results of all text medical data and the label assignment results of all image medical data, obtain the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data, including: Acquire all text medical data or all image medical data related to each augmented data; When the difference between the maximum and minimum values ​​of the label assignment results of all text medical data or all image medical data related to each augmented data is a number in the first preset value set, the label assignment result of the original text medical data or original image medical data related to each augmented data is used as the label assignment result of the corresponding augmented data; When the difference between the maximum and minimum values ​​of the label assignment results of all text medical data or all image medical data related to each augmented data is a number in the second preset value set, the average value of the label assignment results of all text medical data or all image medical data related to each augmented data is used as the label assignment result of the corresponding augmented data; The augmented data is augmented data of image medical data or augmented data of text medical data.

7. According to the multi-source heterogeneous data fusion method based on neural network in claim 1, It is characterized in that S5: Based on all the text medical data, all the image medical data and the final model of the condition evaluation of each patient currently being diagnosed and treated, the label assignment results of all the text medical data and all the image medical data of the corresponding patient are obtained, including: Each text medical data or each image medical data received from each patient is input into the final model for condition assessment and assignment respectively to obtain the label assignment results of all text medical data and all image medical data of the corresponding patient.

8. According to the multi-source heterogeneous data fusion method based on neural network in claim 1, It is characterized in that S6: Based on the label assignment results of all text medical data and the label assignment results of all image medical data of each patient currently being diagnosed and treated, a fusion judgment matrix corresponding to the patient is constructed as the multi-source heterogeneous medical data fusion result of the corresponding patient, including: Performing ordinal definition starting from 1 for all text medical data and all image medical data of each patient, and obtaining ordinal definition results of all text medical data and all image medical data; Based on the label assignment results and ordinal definition results of all text medical data of each patient and the label assignment results and ordinal definition results of all image medical data, the fusion judgment matrix of the corresponding patient is constructed, which is: Among them, P is the patient's fusion judgment matrix, r 1 The label assignment result for the text medical data with the patient's ordinal number 1, r 2 The label assignment result for the text medical data of the patient with ordinal number 2, r n The label assignment result for the text medical data of the patient with ordinal number n, d 1 The label assignment result for the image medical data of the patient with ordinal number 1, d 2 The label assignment result for the image medical data of the patient with ordinal number 2, d m is the label assignment result of the image medical data of the patient with ordinal number m, n is the total number of text medical data of the patient, and m is the total number of image medical data of the patient; The patient's fusion judgment matrix is ​​regarded as the fusion result of the patient's multi-source heterogeneous medical data.

9. According to the multi-source heterogeneous data fusion method based on neural network in claim 1, It is characterized in that S6: Based on the label assignment results of all text medical data and all image medical data of each patient currently being diagnosed and treated, a fusion judgment matrix corresponding to the patient is constructed as the multi-source heterogeneous medical data fusion result of the corresponding patient, and further includes: The rank of the patient's fusion judgment matrix is ​​numerically compared with the preset judgment threshold. If the rank of the fusion judgment matrix is ​​greater than the preset judgment threshold, the condition requiring intervention is taken as the fusion judgment result of the corresponding patient's condition. Otherwise, the condition requiring no intervention is taken as the fusion judgment result of the corresponding patient's condition.

10. A multi-source heterogeneous data fusion system based on neural network, It is characterized in that A method for fusion of multi-source heterogeneous data based on a neural network according to any one of claims 1 to 9, comprising: An amplification module, used to perform data amplification on all text medical data and all image medical data in the database to obtain amplified data of all text medical data and amplified data of all image medical data; A training module is used to train a neural network model based on the label assignment results of all text medical data and all image medical data in the database to obtain an initial model for disease assessment assignment; An assignment module, used for obtaining label assignment results of augmented data of all text medical data and label assignment results of augmented data of all image medical data based on label assignment results of all text medical data and label assignment results of all image medical data; A retraining module, used to retrain the initial model for disease assessment assignment based on the label assignment results of the augmented data of all text medical data and the label assignment results of the augmented data of all image medical data, to obtain a final model for disease assessment assignment; An acquisition module is used to assign a final model based on all text medical data, all image medical data and condition evaluation of each patient currently being diagnosed and treated, and obtain label assignment results of all text medical data and all image medical data of the corresponding patient; The judgment module is used to construct a fusion judgment matrix of the corresponding patient based on the label assignment results of all text medical data and all image medical data of each patient currently being diagnosed and treated, as the multi-source heterogeneous medical data fusion result of the corresponding patient.

Images