Key information extraction method and system based on multimodal large model

Abstract

The present invention relates to the technical field of multimodal large models, and a key information extraction method and system based on a multimodal large model, comprising: training an initial multimodal large model according to iterative training data, verifying the prediction error of the iterative multimodal large model, obtaining a maximum prediction error, judging whether the maximum prediction error is less than a prediction error threshold, if not, updating the training backup data, the initial multimodal large model and the initial sampling ratio, and returning to the training, if less than, using the iterative multimodal large model as a target multimodal large model, predicting disease names according to current multimodal medical data, and extracting key information from the current multimodal medical data according to the predicted disease names. The present invention can solve the problems of low training efficiency and poor training effect in the current training process of the multimodal large model.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models, and in particular to a key information extraction method and system based on a multimodal large model. Background Art

[0002] With the continuous advancement of artificial intelligence technology, the extraction of key information based on multimodal big models has begun to be applied in all walks of life. Multimodal big models can simultaneously process large amounts of data from different modalities (such as text, images, audio, etc.) and extract valuable information from them.

[0003] In the medical field, multimodal big models are mainly used in imaging diagnosis, medical record analysis, etc. For example, multimodal big models are used to analyze medical images such as X-rays, CT, and MRI, and multimodal big models are used to understand pathological images and text reports, thereby automatically generating pathology reports.

[0004] However, the current training process of large multimodal models is mainly based on training large models with large amounts of data. This training method only considers the use of prediction errors for feedback adjustment of large multimodal models, and does not adaptively adjust the training data based on the prediction errors. Therefore, the current training process of large multimodal models has the problems of low training efficiency and poor training effect. Summary of the invention

[0005] The present invention provides a key information extraction method and system based on a multimodal large model, the main purpose of which is to solve the problems of low training efficiency and poor training effect in the current training process of the multimodal large model.

[0006] To achieve the above object, the present invention provides a key information extraction method based on a multimodal large model, comprising:

[0007] Acquire multimodal medical training data, wherein the multimodal medical training data includes: medical text training data, medical image training data, medical voice training data and confirmed disease name training data;

[0008] Extracting prediction verification data from multimodal medical training data according to a preset unit verification data volume, obtaining training backup data, and acquiring an initial multimodal large model, wherein the initial multimodal large model is constructed based on a BP neural network;

[0009] Extracting iterative training data from the training backup data according to a preset unit training data amount and an initial sampling ratio to obtain iterative backup data;

[0010] Using the iterative training data to train the initial multimodal large model to obtain an iterative multimodal large model;

[0011] Using the prediction verification data to verify the prediction error of the iterative multimodal large model, and obtaining the prediction error ratio and the maximum prediction error;

[0012] Determining whether the maximum prediction error is less than a preset prediction error threshold;

[0013] If the maximum prediction error is not less than the prediction error threshold, the training backup data, the initial multimodal large model and the initial sampling ratio are updated using the iterative backup data, the iterative multimodal large model and the prediction error ratio, respectively, and the step of extracting iterative training data from the training backup data according to the preset unit training data amount and the initial sampling ratio is returned to the above-mentioned step;

[0014] If the maximum prediction error is less than the prediction error threshold, the iterative multimodal large model is used as the target multimodal large model;

[0015] Receiving multimodal medical current data, and predicting a disease name using the target multimodal large model according to the multimodal medical current data to obtain a predicted disease name;

[0016] Key information is extracted from the multimodal medical current data according to the predicted disease name.

[0017] Optionally, extracting prediction verification data from the multimodal medical training data according to a preset unit verification data amount to obtain training backup data includes:

[0018] randomly extracting multimodal medical case data in sequence from the multimodal medical training data;

[0019] Performing case statistics on the multimodal medical case data to obtain a random case number;

[0020] Determining whether the number of random cases is less than the unit verification data volume;

[0021] If the number of random cases is less than the unit verification data amount, returning to the above step of randomly extracting multimodal medical case data in the multimodal medical training data in sequence;

[0022] If the number of random cases is not less than the unit verification data volume, all multimodal medical case data are aggregated to obtain initial verification data;

[0023] Identify an initial verification disease name set of the initial verification data, and determine whether the initial verification disease name set is a true subset of a preset confirmed disease name set;

[0024] If the initial verification disease name set is a true subset of the confirmed disease name set, returning to the above step of randomly extracting multimodal medical case data from the multimodal medical training data in sequence;

[0025] If the initial verification disease name set is not a true subset of the confirmed disease name set, the initial verification data is used as the prediction verification data;

[0026] The prediction verification data is removed from the multimodal medical training data to obtain training backup data.

[0027] Optionally, the obtaining of the initial multimodal large model includes:

[0028] Construct the predicted disease name output node;

[0029] Setting a medical text input node, a medical image input node, and a medical voice input node to obtain a multimodal input layer node set;

[0030] According to the multimodal input layer node set and the predicted disease name output node, an initial multimodal large model is constructed using a pre-constructed BP neural network.

[0031] Optionally, extracting iterative training data from the training backup data according to a preset unit training data amount and an initial sampling ratio to obtain iterative backup data includes:

[0032] According to the unit training data volume and the initial sampling ratio, the number of text medical data cases, the number of image medical data cases and the number of voice medical data cases are calculated using the following formula:

[0033] ;

[0034] in, represents the number of medical data cases in the ith case, Represents the number of cases in text medical data, represents the number of cases of medical imaging data, Indicates the number of cases of voice medical data, represents the unit training data volume, Represents the i-th sampling ratio coefficient in the initial sampling ratio;

[0035] According to the number of text medical data cases, the number of image medical data cases and the number of voice medical data cases, text medical case data, image medical case data and voice medical case data are respectively extracted from the training backup data to obtain iterative training data;

[0036] The iterative training data is removed from the training backup data to obtain iterative backup data.

[0037] Optionally, the using the iterative training data to train the initial multimodal large model to obtain the iterative multimodal large model includes:

[0038] Performing word segmentation encoding processing on the text medical case data to obtain initial encoded text data;

[0039] Standardizing the initial encoded text data to obtain standard text data;

[0040] Performing vector conversion on the standard text data using a pre-built word embedding technology to obtain a text vector;

[0041] Resizing the image medical case data to obtain image data of the same size;

[0042] Normalizing the image data of the same size to obtain standard image data, and extracting features from the standard image data using a pre-built convolutional neural network to obtain an image vector;

[0043] Convert the vocal medical case data to obtain vocal digital signal data;

[0044] De-noising and segmenting the digital sound signal data to obtain standard sound data;

[0045] Extracting a vocalization vector from the standard vocalization data using a pre-built feature extraction algorithm;

[0046] Extracting a single text vector, a single image vector and a single voice vector from the text vector, the image vector and the voice vector respectively;

[0047] Inputting the single-instance text vector, the single-instance image vector and the single-instance voice vector from the medical text input node, the medical image input node and the medical voice input node into the initial multimodal large model respectively to obtain an iterative output value;

[0048] Obtaining a single confirmed disease name corresponding to the single text vector, the single image vector, and the single voice vector;

[0049] Extracting the predicted disease name value range of the single confirmed disease name from the pre-constructed disease name value range table, and extracting the value range center value from the predicted disease name value range;

[0050] Calculate the single-case prediction error between the iterative output value and the central value of the value range, and determine whether the single-case prediction error is less than the prediction error threshold;

[0051] If the single-case prediction error is not less than the prediction error threshold, feedback-adjusting the initial multimodal large model according to the single-case prediction error until the single-case prediction error is less than the prediction error threshold;

[0052] Determining whether the text vector, the image vector, and the voice vector have completed the extraction of a single text vector, a single image vector, and a single voice vector;

[0053] If the text vector, image vector and voice vector have not completed the extraction of a single text vector, a single image vector and a single voice vector, then return to the above steps of extracting a single text vector, a single image vector and a single voice vector from the text vector, image vector and voice vector respectively;

[0054] If the text vector, image vector and voice vector complete the extraction of a single text vector, a single image vector and a single voice vector, an iterative multimodal large model is obtained.

[0055] Optionally, the using the prediction verification data to perform prediction error verification on the iterative multimodal large model to obtain a prediction error ratio and a maximum prediction error includes:

[0056] Classifying the predicted verification data to obtain text verification case data, image verification case data, and voice verification case data;

[0057] Converting the text verification case data, image verification case data and voice verification case data into a text verification vector, an image verification vector and a voice verification vector respectively;

[0058] Sequentially extracting a single text verification vector, a single image verification vector, and a single voice verification vector from the text verification vector, the image verification vector, and the voice verification vector, respectively;

[0059] Inputting the single-instance text verification vector, the single-instance image verification vector, and the single-instance voice verification vector into the iterative multimodal large model respectively to obtain an iterative verification value;

[0060] Determining whether the text verification vector, the image verification vector, and the voice verification vector have completed the extraction of a single text verification vector, a single image verification vector, and a single voice verification vector;

[0061] If the text verification vector, the image verification vector, and the voice verification vector have not completed the extraction of a single text verification vector, a single image verification vector, and a single voice verification vector, then return to the above steps of extracting a single text verification vector, a single image verification vector, and a single voice verification vector from the text verification vector, the image verification vector, and the voice verification vector respectively;

[0062] If the text verification vector, the image verification vector, and the voice verification vector do not complete the extraction of a single text verification vector, a single image verification vector, and a single voice verification vector, then summarizing the iterative verification values ​​to obtain an iterative verification value set;

[0063] Classifying the iterative validation value set according to the confirmed disease name set to obtain multiple groups of validation prediction value sets;

[0064] Extracting validation prediction value sets in sequence from the multiple validation prediction value sets, and identifying the target center value of the target disease name value domain corresponding to the validation prediction value sets;

[0065] Calculate the disease name prediction error set between the verification prediction value set and the target central value, calculate the disease name prediction error mean according to the disease name prediction error set, and obtain the disease name prediction error mean set;

[0066] The disease name prediction error mean set is proportionally reduced to obtain the prediction error ratio, the maximum disease name prediction error mean is extracted from the disease name prediction error mean set, and the maximum disease name prediction error mean is used as the maximum prediction error.

[0067] Optionally, the predicting of disease names using the target multimodal large model based on the multimodal current medical data to obtain predicted disease names includes:

[0068] Classifying the multimodal medical current data to obtain text current case data, image current case data and voice current case data;

[0069] Converting the text current case data, the image current case data and the voice current case data into a text current vector, an image current vector and a voice current vector respectively;

[0070] Inputting the text current vector, the image current vector and the utterance current vector into the target multimodal large model respectively to obtain a target prediction value;

[0071] According to the target predicted numerical value, a target disease name value range is identified in the disease name value range table, and a predicted disease name corresponding to the target disease name value range is identified.

[0072] Optionally, extracting key information from the multimodal medical current data according to the predicted disease name includes:

[0073] Extracting text keywords, image key areas, and vocalization key indicators from a pre-constructed disease name-multimodal key data table according to the predicted disease name;

[0074] Extract key text data, key image areas and key vocalization index values ​​from the text current case data, the image current case data and the vocalization current case data respectively according to the text keywords, the image key areas and the vocalization key indicators;

[0075] The key text data, key image areas and key vocalization index values ​​are summarized to obtain key information.

[0076] Optionally, before extracting text keywords, image key areas, and phonation key indicators from a pre-constructed disease name-multimodal key data table according to the predicted disease name, the method further includes:

[0077] Extracting confirmed disease names in sequence from the confirmed disease name set;

[0078] Receive the disease name text keywords, disease name image key areas and disease name pronunciation key indicators set by the user for the confirmed disease name;

[0079] A disease name-multimodal key data table is constructed based on the confirmed disease name and the disease name text keywords, disease name image key areas and disease name pronunciation key indicators.

[0080] To achieve the above object, the present invention also provides a key information extraction system based on a multimodal large model, comprising:

[0081] An initial multimodal large model training module is used to obtain multimodal medical training data, wherein the multimodal medical training data includes: medical text training data, medical image training data, medical voice training data and confirmed disease name training data; extracting prediction verification data from the multimodal medical training data according to a preset unit verification data volume, obtaining training backup data, and obtaining an initial multimodal large model, wherein the initial multimodal large model is constructed based on a BP neural network; extracting iterative training data from the training backup data according to a preset unit training data volume and an initial sampling ratio, obtaining iterative backup data; using the iterative training data to train the initial multimodal large model to obtain an iterative multimodal large model;

[0082] An iterative multimodal large model error verification module is used to verify the prediction error of the iterative multimodal large model using the prediction verification data to obtain the prediction error ratio and the maximum prediction error;

[0083] a target multimodal large model acquisition module, for determining whether the maximum prediction error is less than a preset prediction error threshold; if the maximum prediction error is not less than the prediction error threshold, respectively updating the training backup data, the initial multimodal large model and the initial sampling ratio using the iterative backup data, the iterative multimodal large model and the prediction error ratio, and returning to the above step of extracting iterative training data from the training backup data according to the preset unit training data amount and the initial sampling ratio; if the maximum prediction error is less than the prediction error threshold, using the iterative multimodal large model as the target multimodal large model;

[0084] The key information extraction module is used to receive multimodal medical current data, predict disease names based on the multimodal medical current data using the target multimodal large model to obtain predicted disease names; and extract key information from the multimodal medical current data based on the predicted disease names.

[0085] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0086] a memory storing at least one instruction; and

[0087] A processor executes instructions stored in the memory to implement the above-mentioned key information extraction method based on the multimodal large model.

[0088] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the above-mentioned key information extraction method based on a multimodal large model.

[0089] In order to solve the problem described in the background technology, the present invention first trains the initial multimodal large model through iterative training data, and then uses the predicted verification data to verify the prediction error of the iterative multimodal large model. After passing the error verification, the target multimodal large model is used to predict the disease name according to the multimodal medical current data to obtain the predicted disease name, and then extracts key information from the multimodal medical current data according to the predicted disease name. Before training the initial multimodal large model, it is necessary to first obtain the multimodal medical training data, and then extract the predicted verification data from the multimodal medical training data according to the preset unit verification data amount to obtain the training backup data. At this time, the training backup data can be used to train the initial multimodal large model. During the training process, the initial multimodal large model is trained in multiple batches. First, it is necessary to extract the predicted verification data according to the preset unit training data amount and the initial sampling. The iterative training data is extracted from the training backup data, and the initial multimodal large model is trained using the iterative training data to obtain the iterative multimodal large model. In order to verify the effect of the training, the prediction verification data is used to verify the prediction error of the iterative multimodal large model to obtain the prediction error ratio and the maximum prediction error. When the maximum prediction error is not less than the prediction error threshold, the training backup data, the iterative multimodal large model and the prediction error ratio are respectively used to update the training backup data, the initial multimodal large model and the initial sampling ratio, and the next batch of iterative multimodal large models are retrained. When the maximum prediction error is less than the prediction error threshold, the iterative multimodal large model is used as the target multimodal large model. At this time, the target multimodal large model can be used to predict the disease name and extract key information according to the current multimodal medical data. Therefore, the present invention can solve the problems of low training efficiency and poor training effect in the current training process of the multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 A schematic diagram of a flow chart of a key information extraction method based on a multimodal large model provided by an embodiment of the present invention;

[0091] Figure 2 A functional module diagram of a key information extraction system based on a multimodal large model provided by an embodiment of the present invention;

[0092] Figure 3 A schematic diagram of the structure of an electronic device for implementing the key information extraction method based on a multimodal large model provided in one embodiment of the present invention.

[0093] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0094] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0095] The embodiment of the present application provides a key information extraction method based on a multimodal large model. The execution subject of the key information extraction method based on the multimodal large model includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the key information extraction method based on the multimodal large model can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0096] Reference Figure 1 FIG. 1 is a flow chart of a key information extraction method based on a multimodal large model provided by an embodiment of the present invention. In this embodiment, the key information extraction method based on a multimodal large model includes:

[0097] S1. Acquire multimodal medical training data, wherein the multimodal medical training data includes: medical text training data, medical image training data, medical voice training data and confirmed disease name training data.

[0098] It is understandable that the multimodal medical training data refers to a historical multimodal medical data set, the medical text training data refers to a historical patient case text data set, such as: the patient's chief complaint text, examination data text, etc., and the medical image training data refers to a historical patient case image or image data set, such as: the patient's X-ray image, CT image, MRI image, electrocardiogram, ultrasound image, etc. The medical voice training data refers to a historical patient voice examination data set, such as: laryngeal voice examination data, heartbeat voice examination data, respiratory voice examination data, etc. when the patient undergoes a voice examination.

[0099] Furthermore, the confirmed disease name training data refers to a set of confirmed disease names of historical patients, such as vocal cord polyps, pneumonia, tuberculosis, nervous system tumors, heart failure, pericardial effusion, etc. Among them, when the confirmed disease name is vocal cord polyps, the patient may need to undergo electronic laryngoscopy and fiberoptic laryngoscopy before diagnosis, so as to obtain relevant laryngoscopic images. For complex or latent laryngeal lesions, CT or MRI examinations may be required to obtain imaging examination data such as CT images and MRI image data. At the same time, voice examinations and case text records may be required before diagnosis, so as to obtain voice examination data and text examination data.

[0100] It should be understood that when the patient has not undergone a voice examination before diagnosis, the medical voice examination data can be left blank; when the patient has not undergone an imaging examination before diagnosis, the imaging examination data can be left blank.

[0101] S2. Extract prediction verification data from the multimodal medical training data according to a preset unit verification data volume, obtain training backup data, and acquire an initial multimodal large model.

[0102] It can be understood that the unit verification data volume refers to the number of cases used to perform error verification on the training results of the initial multimodal large model, which can be set according to the error verification accuracy of the initial multimodal large model. Specifically, the following formula can be referred to:

[0103] ;

[0104] in, Indicates the unit verification data volume, represents the adjustment coefficient, represents a natural constant, represents the error verification accuracy, Indicates the rounding symbol.

[0105] Furthermore, the unit verification data volume and the error verification accuracy should be in direct proportion, that is, the larger the error verification accuracy, the larger the unit verification data volume. The embodiment of the present invention uses the sigmoid function to construct a calculation formula for the unit verification data volume. Sigmoid better conforms to the positive proportional relationship between the unit verification data volume and the error verification accuracy, and will not increase indefinitely with the increase of the error verification accuracy. The error verification accuracy can take values ​​of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, etc. In order to make the unit verification data volume a suitable integer, it is necessary to set an adjustment coefficient and a rounding sign, wherein the adjustment coefficient can be set according to the actual situation. It can be known from the value range of the sigmoid function that The value range is Therefore, when the maximum value of the unit verification data volume is set to 1000, the value range of the unit verification data volume is .

[0106] Furthermore, the prediction verification data refers to a partial set of historical multimodal medical data for error verification of the training results of the initial multimodal large model, and the training backup data refers to a partial set of historical multimodal medical data after the prediction verification data is removed from the multimodal medical training data, which is used for batch training of the initial multimodal large model.

[0107] In detail, the initial multimodal large model is constructed based on the BP neural network, which refers to a multimodal large model that has not completed training. The BP neural network is described in detail in the following embodiment.

[0108] In an embodiment of the present invention, extracting prediction verification data from multimodal medical training data according to a preset unit verification data amount to obtain training backup data includes:

[0109] randomly extracting multimodal medical case data in sequence from the multimodal medical training data;

[0110] Performing case statistics on the multimodal medical case data to obtain a random case number;

[0111] Determining whether the number of random cases is less than the unit verification data volume;

[0112] If the number of random cases is less than the unit verification data amount, returning to the above step of randomly extracting multimodal medical case data in the multimodal medical training data in sequence;

[0113] If the number of random cases is not less than the unit verification data volume, all multimodal medical case data are aggregated to obtain initial verification data;

[0114] Identify an initial verification disease name set of the initial verification data, and determine whether the initial verification disease name set is a true subset of a preset confirmed disease name set;

[0115] If the initial verification disease name set is a true subset of the confirmed disease name set, returning to the above step of randomly extracting multimodal medical case data from the multimodal medical training data in sequence;

[0116] If the initial verification disease name set is not a true subset of the confirmed disease name set, the initial verification data is used as the prediction verification data;

[0117] The prediction verification data is removed from the multimodal medical training data to obtain training backup data.

[0118] It is understandable that the multimodal medical case data refers to the historical multimodal medical data of a single case, the random case number refers to the number of randomly extracted multimodal medical case data, and the initial verification disease name set refers to the set of confirmed disease names corresponding to each multimodal medical case data in the initial verification data. The confirmed disease name set refers to the set of all confirmed disease names preset by the user, such as: vocal cord polyps, pneumonia, tuberculosis, nervous system tumors, heart failure, pericardial effusion, etc. When the initial verification disease name set is a true subset of the confirmed disease name set, it indicates that the initial verification disease name set does not contain all the confirmed disease names in the confirmed disease name set. Therefore, it is necessary to continue to randomly extract multimodal medical case data in the multimodal medical training data in sequence until the initial verification disease name set is not a true subset of the confirmed disease name set (that is, the initial verification disease name set contains all the confirmed disease names in the confirmed disease name set). At this time, the initial verification data can be used as prediction verification data. Since some historical multimodal medical data sets in the prediction verification data are specifically used for error verification of the training results of the initial multimodal large model, they are not suitable for training the initial multimodal large model, so they need to be removed from the multimodal medical training data.

[0119] In the embodiment of the present invention, the step of obtaining the initial multimodal large model includes:

[0120] Construct the predicted disease name output node;

[0121] Setting a medical text input node, a medical image input node, and a medical voice input node to obtain a multimodal input layer node set;

[0122] According to the multimodal input layer node set and the predicted disease name output node, an initial multimodal large model is constructed using a pre-constructed BP neural network.

[0123] In detail, the BP neural network is a prior art and will not be described in detail here.

[0124] S3. Extracting iterative training data from the training backup data according to a preset unit training data amount and an initial sampling ratio to obtain iterative backup data.

[0125] It can be explained that the unit training data volume refers to the number of multimodal medical case data when a single batch training is performed on the initial multimodal large model. The initial sampling ratio refers to the quantitative ratio of each type of multimodal medical case data when the initial multimodal large model is trained for the first time. For example, the unit training data volume can be 1000, that is, the number of multimodal medical case data for a single batch training of the initial multimodal large model is 1000. When the number of confirmed disease types is 5, the initial sampling ratio can be 1:1:1:1:1. The iterative training data refers to the set of multimodal medical case data when a single batch training is performed on the initial multimodal large model. The iterative backup data refers to the set of multimodal medical case data used for subsequent training of the initial multimodal large model after the iterative training data is eliminated from the training backup data.

[0126] In the embodiment of the present invention, extracting iterative training data from the training backup data according to the preset unit training data amount and the initial sampling ratio to obtain iterative backup data includes:

[0127] According to the unit training data volume and the initial sampling ratio, the number of text medical data cases, the number of image medical data cases and the number of voice medical data cases are calculated using the following formula:

[0128] ;

[0129] in, represents the number of medical data cases in the ith case, Represents the number of cases in text medical data, represents the number of cases of medical imaging data, Indicates the number of cases of voice medical data, represents the unit training data volume, Represents the i-th sampling ratio coefficient in the initial sampling ratio;

[0130] According to the number of text medical data cases, the number of image medical data cases and the number of voice medical data cases, text medical case data, image medical case data and voice medical case data are respectively extracted from the training backup data to obtain iterative training data;

[0131] The iterative training data is removed from the training backup data to obtain iterative backup data.

[0132] It can be understood that the number of text medical data cases, the number of image medical data cases and the number of voice medical data cases refer to the number of text examination data, image examination data and voice examination data in the iterative training data respectively.

[0133] S4. Use the iterative training data to train the initial multimodal large model to obtain an iterative multimodal large model.

[0134] It can be understood that the iterative multimodal large model refers to a multimodal large model that is waiting for prediction error verification after being trained with the iterative training data.

[0135] In the embodiment of the present invention, the step of training the initial multimodal large model using the iterative training data to obtain the iterative multimodal large model includes:

[0136] Performing word segmentation encoding processing on the text medical case data to obtain initial encoded text data;

[0137] Standardizing the initial encoded text data to obtain standard text data;

[0138] Performing vector conversion on the standard text data using a pre-built word embedding technology to obtain a text vector;

[0139] Resizing the image medical case data to obtain image data of the same size;

[0140] Normalizing the image data of the same size to obtain standard image data, and extracting features from the standard image data using a pre-built convolutional neural network to obtain an image vector;

[0141] Convert the vocal medical case data to obtain vocal digital signal data;

[0142] De-noising and segmenting the digital sound signal data to obtain standard sound data;

[0143] Extracting a vocalization vector from the standard vocalization data using a pre-built feature extraction algorithm;

[0144] Extracting a single text vector, a single image vector and a single voice vector from the text vector, the image vector and the voice vector respectively;

[0145] Inputting the single-instance text vector, the single-instance image vector and the single-instance voice vector from the medical text input node, the medical image input node and the medical voice input node into the initial multimodal large model respectively to obtain an iterative output value;

[0146] Obtaining a single confirmed disease name corresponding to the single text vector, the single image vector, and the single voice vector;

[0147] Extracting the predicted disease name value range of the single confirmed disease name from the pre-constructed disease name value range table, and extracting the value range center value from the predicted disease name value range;

[0148] Calculate the single-case prediction error between the iterative output value and the central value of the value range, and determine whether the single-case prediction error is less than the prediction error threshold;

[0149] If the single-case prediction error is not less than the prediction error threshold, feedback-adjusting the initial multimodal large model according to the single-case prediction error until the single-case prediction error is less than the prediction error threshold;

[0150] Determining whether the text vector, the image vector, and the voice vector have completed the extraction of a single text vector, a single image vector, and a single voice vector;

[0151] If the text vector, image vector and voice vector have not completed the extraction of a single text vector, a single image vector and a single voice vector, then return to the above steps of extracting a single text vector, a single image vector and a single voice vector from the text vector, image vector and voice vector respectively;

[0152] If the text vector, image vector and voice vector complete the extraction of a single text vector, a single image vector and a single voice vector, an iterative multimodal large model is obtained.

[0153] It is understandable that the text medical case data can be segmented and encoded using technologies such as the bag of words model and TF-IDF, and the standardization refers to adjusting the initial encoded text data to a uniform length and format. The word embedding technology can be technologies such as Word2Vec, GloVe or BERT. The normalization refers to scaling the pixel values ​​in the same size image data to the interval of 0 to 1 to reduce the computational burden during the training process and improve the convergence speed of the initial multimodal large model. The data conversion refers to converting the analog voice signal of the vocal medical case data into a digital signal, and the segmentation processing refers to segmenting the long voice signal in the vocal digital signal data into shorter voice signal frames (generally 20-30 milliseconds). The feature extraction algorithm can be an algorithm such as MFCC and LPC. The single-case text vector, single-case image vector and single-case vocalization vector refer to the vectors corresponding to the multimodal medical case data in the text vector, image vector and vocalization vector, respectively, and the single-case text vector, single-case image vector and single-case vocalization vector correspond to the text examination data, image examination data and vocalization examination data of a historical patient. The single confirmed disease name refers to the confirmed disease name corresponding to a single text vector, a single image vector and a single voice vector. Since not every examination data corresponding to each confirmed disease name includes text examination data, image examination data and voice examination data, the single text vector, the single image vector and the single voice vector may not exist at the same time. For example, when there is no single voice vector, the single voice vector can be set to zero.

[0154] Furthermore, the disease name value range table refers to a correspondence table between confirmed disease names and predicted disease name value ranges. For example, when the confirmed disease names are vocal cord polyps, pneumonia, tuberculosis, and nervous system tumors, the predicted disease name value ranges can be (0-1], (1-2], (2-3], and (3-4], respectively. The value range center value refers to the middle value of the predicted disease name value range. For example, when the predicted disease name value range is (0-1], the predicted disease name value range is 0.5. The single-case prediction error refers to the difference between the iterative output value and the value range center value. The prediction error threshold can be set as needed, for example: 0.4. The smaller the prediction error threshold, the higher the training accuracy of the initial multimodal large model. Since a single batch of training needs to be completed, it is necessary to complete the extraction of all single-case text vectors, single-case image vectors, and single-case voice vectors in the text vector, image vector, and voice vector.

[0155] S5. Use the prediction verification data to verify the prediction error of the iterative multimodal large model to obtain the prediction error ratio and the maximum prediction error.

[0156] It can be understood that the prediction error ratio refers to the prediction error ratio of the iterative multimodal large model for various confirmed disease names. For example, the predicted disease name value ranges of vocal cord polyps, pneumonia, tuberculosis, and nervous system tumors can be (0-1], (1-2], (2-3], and (3-4], respectively, and the values ​​output by the predicted disease name output nodes corresponding to vocal cord polyps, pneumonia, tuberculosis, and nervous system tumors are 0.8, 1.1, 2.2, and 3.9, respectively. Then the prediction error ratio is (0.8-0.5): (1.5-1.1): (2.5-2.2): (3.9-3.5). The maximum prediction error refers to the maximum error in the prediction error ratio. When the prediction error ratio is (0.8-0.5): (1.5-1.1): (2.5-2.2): (3.9-3.5), the maximum prediction error is 0.4.

[0157] In the embodiment of the present invention, the prediction error verification of the iterative multimodal large model using the prediction verification data to obtain the prediction error ratio and the maximum prediction error includes:

[0158] Classifying the predicted verification data to obtain text verification case data, image verification case data, and voice verification case data;

[0159] Converting the text verification case data, image verification case data and voice verification case data into a text verification vector, an image verification vector and a voice verification vector respectively;

[0160] Sequentially extracting a single text verification vector, a single image verification vector, and a single voice verification vector from the text verification vector, the image verification vector, and the voice verification vector, respectively;

[0161] Inputting the single-instance text verification vector, the single-instance image verification vector, and the single-instance voice verification vector into the iterative multimodal large model respectively to obtain an iterative verification value;

[0162] Determining whether the text verification vector, the image verification vector, and the voice verification vector have completed the extraction of a single text verification vector, a single image verification vector, and a single voice verification vector;

[0163] If the text verification vector, the image verification vector, and the voice verification vector have not completed the extraction of a single text verification vector, a single image verification vector, and a single voice verification vector, then return to the above steps of extracting a single text verification vector, a single image verification vector, and a single voice verification vector from the text verification vector, the image verification vector, and the voice verification vector respectively;

[0164] If the text verification vector, the image verification vector, and the voice verification vector do not complete the extraction of a single text verification vector, a single image verification vector, and a single voice verification vector, then summarizing the iterative verification values ​​to obtain an iterative verification value set;

[0165] Classifying the iterative validation value set according to the confirmed disease name set to obtain multiple groups of validation prediction value sets;

[0166] Extracting validation prediction value sets in sequence from the multiple validation prediction value sets, and identifying the target center value of the target disease name value domain corresponding to the validation prediction value sets;

[0167] Calculate the disease name prediction error set between the verification prediction value set and the target central value, calculate the disease name prediction error mean according to the disease name prediction error set, and obtain the disease name prediction error mean set;

[0168] The disease name prediction error mean set is proportionally reduced to obtain the prediction error ratio, the maximum disease name prediction error mean is extracted from the disease name prediction error mean set, and the maximum disease name prediction error mean is used as the maximum prediction error.

[0169] It can be understood that the process of obtaining the iterative verification value is the same as the process of obtaining the iterative output value, which will not be repeated here.

[0170] Furthermore, since each iterative verification value in the iterative verification value set corresponds to a confirmed disease name, the iterative verification value set can be classified according to the confirmed disease name set. Each group of verification prediction value sets in the multiple groups of verification prediction value sets corresponds to a confirmed disease name. The target disease name value range refers to the predicted disease name value range of the confirmed disease name corresponding to the verification prediction value set. The target center value refers to the middle value of the target disease name value range. The disease name prediction error set refers to the difference set between each verification prediction error and the target center value in the verification prediction value set. The disease name prediction error mean refers to the mean of the disease name prediction error set. The proportional reduction refers to the proportional simplification of the disease name prediction error mean set. For example, when the confirmed disease name set is vocal cord polyps, pneumonia, pulmonary tuberculosis, and nervous system tumors, the disease name prediction error mean set can be 0.6, 0.2, 0.4, and 0.2. At this time, the prediction error ratio is 3:1:2:1, and the maximum prediction error is 0.6.

[0171] S6. Determine whether the maximum prediction error is less than a preset prediction error threshold.

[0172] If the maximum prediction error is not less than the prediction error threshold, execute S7, and respectively use the iterative backup data, the iterative multimodal large model and the prediction error ratio to update the training backup data, the initial multimodal large model and the initial sampling ratio.

[0173] It can be understood that when the maximum prediction error is not less than the prediction error threshold, it indicates that the iterative multimodal large model has not completed training. Therefore, it is necessary to reuse the iterative backup data, the iterative multimodal large model and the prediction error ratio to update the training backup data, the initial multimodal large model and the initial sampling ratio respectively, and return to the above step of extracting iterative training data from the training backup data according to the preset unit training data amount and the initial sampling ratio to continue training the iterative multimodal large model.

[0174] Furthermore, since the corresponding mean disease name prediction error in the prediction error ratio represents the prediction error of the iterative multimodal large model for the confirmed disease name, the larger the mean disease name prediction error, the lower the prediction accuracy of the confirmed disease name, and thus the more training data required. The embodiment of the present invention uses the prediction error ratio to update the initial sampling ratio, thereby achieving the purpose of adjusting the training data according to the prediction results, thereby making the training efficiency of the iterative multimodal large model higher.

[0175] If the maximum prediction error is less than the prediction error threshold, execute S8 and use the iterative multimodal large model as the target multimodal large model.

[0176] It should be understood that when the maximum prediction error is less than the prediction error threshold, it indicates that the iterative multimodal large model has completed training. Therefore, the iterative multimodal large model can be used as the target multimodal large model.

[0177] S9. Receive multimodal medical current data, and predict disease names using the target multimodal large model based on the multimodal medical current data to obtain predicted disease names.

[0178] It can be understood that the multimodal medical current data refers to the multimodal medical data that currently needs to be predicted for disease names.

[0179] In the embodiment of the present invention, the method of predicting a disease name by using the target multimodal large model based on the multimodal current medical data to obtain a predicted disease name includes:

[0180] Classifying the multimodal medical current data to obtain text current case data, image current case data and voice current case data;

[0181] Converting the text current case data, the image current case data and the voice current case data into a text current vector, an image current vector and a voice current vector respectively;

[0182] Inputting the text current vector, the image current vector and the utterance current vector into the target multimodal large model respectively to obtain a target prediction value;

[0183] According to the target predicted numerical value, a target disease name value range is identified in the disease name value range table, and a predicted disease name corresponding to the target disease name value range is identified.

[0184] It can be understood that the target predicted value refers to the output value of the target multimodal large model according to the multimodal medical current data. The target disease name value range refers to the disease name value range that contains the target predicted value. For example, when the target predicted value is 2.3, the predicted disease name value ranges of vocal cord polyps, pneumonia, tuberculosis, and nervous system tumors are (0-1], (1-2], (2-3], and (3-4] respectively, the target disease name value range is (2-3], and the corresponding predicted disease name is tuberculosis.

[0185] S10. Extract key information from the multimodal medical current data according to the predicted disease name.

[0186] It can be understood that the key information refers to the data information obtained by simplifying the data content of the multimodal medical current data by the predicted disease name.

[0187] In the embodiment of the present invention, extracting key information from the multimodal medical current data according to the predicted disease name includes:

[0188] Extracting text keywords, image key areas, and vocalization key indicators from a pre-constructed disease name-multimodal key data table according to the predicted disease name;

[0189] Extract key text data, key image areas and key vocalization index values ​​from the text current case data, the image current case data and the vocalization current case data respectively according to the text keywords, the image key areas and the vocalization key indicators;

[0190] The key text data, key image areas and key vocalization index values ​​are summarized to obtain key information.

[0191] It can be understood that the disease name-multimodal key data table refers to a data table that records the correspondence between various confirmed disease names and text keywords, image key areas and voice key indicators. For example, when the confirmed disease name is vocal cord polyp, the text keywords can be hoarseness, vocal cord morphology, vocal cord color, vocal cord polyp morphology, etc. The image key areas can be the vocal cord area, laryngeal vestibule area, laryngeal ventricle area in the laryngeal image, etc. The voice key indicators can be tone, volume, sound quality, speaking speed, speaking speed and other indicators.

[0192] Furthermore, the key text data, key image areas and key voice index values ​​respectively refer to phrases related to text keywords in the text current case data, images belonging to key image areas in the image current case data, and values ​​of key voice indicators in the voice current case data.

[0193] In the embodiment of the present invention, before extracting text keywords, image key areas and utterance key indicators from the pre-constructed disease name-multimodal key data table according to the predicted disease name, the method further includes:

[0194] Extracting confirmed disease names in sequence from the confirmed disease name set;

[0195] Receive the disease name text keywords, disease name image key areas and disease name pronunciation key indicators set by the user for the confirmed disease name;

[0196] A disease name-multimodal key data table is constructed based on the confirmed disease name and the disease name text keywords, disease name image key areas and disease name pronunciation key indicators.

[0197] It is understandable that the setting process of the disease name text keywords, disease name image key areas and disease name pronunciation key indicators should be set by professional medical staff or medical experts.

[0198] In order to solve the problem described in the background technology, the present invention first trains the initial multimodal large model through iterative training data, and then uses the predicted verification data to verify the prediction error of the iterative multimodal large model. After passing the error verification, the target multimodal large model is used to predict the disease name according to the multimodal medical current data to obtain the predicted disease name, and then extracts key information from the multimodal medical current data according to the predicted disease name. Before training the initial multimodal large model, it is necessary to first obtain the multimodal medical training data, and then extract the predicted verification data from the multimodal medical training data according to the preset unit verification data amount to obtain the training backup data. At this time, the training backup data can be used to train the initial multimodal large model. During the training process, the initial multimodal large model is trained in multiple batches. First, it is necessary to extract the predicted verification data according to the preset unit training data amount and the initial sampling. The iterative training data is extracted from the training backup data, and the initial multimodal large model is trained using the iterative training data to obtain the iterative multimodal large model. In order to verify the effect of the training, the prediction verification data is used to verify the prediction error of the iterative multimodal large model to obtain the prediction error ratio and the maximum prediction error. When the maximum prediction error is not less than the prediction error threshold, the training backup data, the iterative multimodal large model and the prediction error ratio are respectively used to update the training backup data, the initial multimodal large model and the initial sampling ratio, and the next batch of iterative multimodal large models are retrained. When the maximum prediction error is less than the prediction error threshold, the iterative multimodal large model is used as the target multimodal large model. At this time, the target multimodal large model can be used to predict the disease name and extract key information according to the current multimodal medical data. Therefore, the present invention can solve the problems of low training efficiency and poor training effect in the current training process of the multimodal large model.

[0199] like Figure 2 , which is a functional module diagram of a key information extraction system based on a multimodal large model provided by an embodiment of the present invention.

[0200] The key information extraction system 100 based on the multimodal large model of the present invention can be installed in an electronic device. According to the functions implemented, the key information extraction system 100 based on the multimodal large model can include an initial multimodal large model training module 101, an iterative multimodal large model error verification module 102, a target multimodal large model acquisition module 103 and a key information extraction module 104. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0201] The initial multimodal large model training module 101 is used to obtain multimodal medical training data, wherein the multimodal medical training data includes: medical text training data, medical image training data, medical voice training data and confirmed disease name training data; extracting prediction verification data from the multimodal medical training data according to a preset unit verification data volume, obtaining training backup data, and obtaining an initial multimodal large model, wherein the initial multimodal large model is constructed based on a BP neural network; extracting iterative training data from the training backup data according to a preset unit training data volume and an initial sampling ratio, obtaining iterative backup data; using the iterative training data to train the initial multimodal large model to obtain an iterative multimodal large model;

[0202] The iterative multimodal large model error verification module 102 is used to verify the prediction error of the iterative multimodal large model using the prediction verification data to obtain the prediction error ratio and the maximum prediction error;

[0203] The target multimodal large model acquisition module 103 is used to determine whether the maximum prediction error is less than a preset prediction error threshold; if the maximum prediction error is not less than the prediction error threshold, the training backup data, the initial multimodal large model and the initial sampling ratio are updated using the iterative backup data, the iterative multimodal large model and the prediction error ratio, and the step of extracting iterative training data from the training backup data according to the preset unit training data amount and the initial sampling ratio is returned; if the maximum prediction error is less than the prediction error threshold, the iterative multimodal large model is used as the target multimodal large model;

[0204] The key information extraction module 104 is used to receive multimodal medical current data, predict disease names using the target multimodal large model based on the multimodal medical current data, and obtain predicted disease names; and extract key information from the multimodal medical current data based on the predicted disease names.

[0205] In detail, each module in the key information extraction system 100 based on the multimodal large model in the embodiment of the present invention is used in the same manner as described above. Figure 1 The same technical means are used as the key information extraction method based on the multimodal large model described in , and can produce the same technical effects, so I will not go into details here.

[0206] like Figure 3 , is a schematic diagram of the structure of an electronic device for implementing a key information extraction method based on a multimodal large model provided by an embodiment of the present invention.

[0207] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a key information extraction method program based on a multimodal large model.

[0208] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 also includes an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the key information extraction method program based on the multimodal large model, but also can be used to temporarily store data that has been output or is to be output.

[0209] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (such as a key information extraction method program based on a multimodal large model, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

[0210] The bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 may be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0211] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0212] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0213] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0214] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0215] The key information extraction method program based on the multimodal large model stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0216] Acquire multimodal medical training data, wherein the multimodal medical training data includes: medical text training data, medical image training data, medical voice training data and confirmed disease name training data;

[0217] Extracting prediction verification data from multimodal medical training data according to a preset unit verification data volume, obtaining training backup data, and acquiring an initial multimodal large model, wherein the initial multimodal large model is constructed based on a BP neural network;

[0218] Extracting iterative training data from the training backup data according to a preset unit training data amount and an initial sampling ratio to obtain iterative backup data;

[0219] Using the iterative training data to train the initial multimodal large model to obtain an iterative multimodal large model;

[0220] Using the prediction verification data to verify the prediction error of the iterative multimodal large model, and obtaining the prediction error ratio and the maximum prediction error;

[0221] Determining whether the maximum prediction error is less than a preset prediction error threshold;

[0222] If the maximum prediction error is not less than the prediction error threshold, the training backup data, the initial multimodal large model and the initial sampling ratio are updated using the iterative backup data, the iterative multimodal large model and the prediction error ratio, respectively, and the step of extracting iterative training data from the training backup data according to the preset unit training data amount and the initial sampling ratio is returned to the above-mentioned step;

[0223] If the maximum prediction error is less than the prediction error threshold, the iterative multimodal large model is used as the target multimodal large model;

[0224] Receiving multimodal medical current data, and predicting a disease name using the target multimodal large model according to the multimodal medical current data to obtain a predicted disease name;

[0225] Key information is extracted from the multimodal medical current data according to the predicted disease name.

[0226] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figures 1 to 3 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0227] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0228] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0229] Acquire multimodal medical training data, wherein the multimodal medical training data includes: medical text training data, medical image training data, medical voice training data and confirmed disease name training data;

[0230] Extracting prediction verification data from multimodal medical training data according to a preset unit verification data volume, obtaining training backup data, and acquiring an initial multimodal large model, wherein the initial multimodal large model is constructed based on a BP neural network;

[0231] Extracting iterative training data from the training backup data according to a preset unit training data amount and an initial sampling ratio to obtain iterative backup data;

[0232] Using the iterative training data to train the initial multimodal large model to obtain an iterative multimodal large model;

[0233] Using the prediction verification data to verify the prediction error of the iterative multimodal large model, and obtaining the prediction error ratio and the maximum prediction error;

[0234] Determining whether the maximum prediction error is less than a preset prediction error threshold;

[0235] If the maximum prediction error is not less than the prediction error threshold, the training backup data, the initial multimodal large model and the initial sampling ratio are updated using the iterative backup data, the iterative multimodal large model and the prediction error ratio, respectively, and the step of extracting iterative training data from the training backup data according to the preset unit training data amount and the initial sampling ratio is returned to the above-mentioned step;

[0236] If the maximum prediction error is less than the prediction error threshold, the iterative multimodal large model is used as the target multimodal large model;

[0237] Receiving multimodal medical current data, and predicting a disease name using the target multimodal large model according to the multimodal medical current data to obtain a predicted disease name;

[0238] Key information is extracted from the multimodal medical current data according to the predicted disease name.

[0239] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative, and actual implementation may have other division methods.

[0240] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0241] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0242] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A key information extraction method based on a multimodal large model, characterized in that: The method comprises: Acquire multimodal medical training data, wherein the multimodal medical training data includes: medical text training data, medical image training data, medical voice training data and confirmed disease name training data; Extracting prediction verification data from multimodal medical training data according to a preset unit verification data volume, obtaining training backup data, and acquiring an initial multimodal large model, wherein the initial multimodal large model is constructed based on a BP neural network; Extracting iterative training data from the training backup data according to a preset unit training data amount and an initial sampling ratio to obtain iterative backup data; Using the iterative training data to train the initial multimodal large model to obtain an iterative multimodal large model; Using the prediction verification data to verify the prediction error of the iterative multimodal large model, and obtaining the prediction error ratio and the maximum prediction error; Determining whether the maximum prediction error is less than a preset prediction error threshold; If the maximum prediction error is not less than the prediction error threshold, the training backup data, the initial multimodal large model and the initial sampling ratio are updated using the iterative backup data, the iterative multimodal large model and the prediction error ratio, respectively, and the step of extracting iterative training data from the training backup data according to the preset unit training data amount and the initial sampling ratio is returned to the above-mentioned step; If the maximum prediction error is less than the prediction error threshold, the iterative multimodal large model is used as the target multimodal large model; Receiving multimodal medical current data, and predicting a disease name using the target multimodal large model according to the multimodal medical current data to obtain a predicted disease name; Extracting key information from the multimodal medical current data according to the predicted disease name; The step of extracting prediction verification data from the multimodal medical training data according to a preset unit verification data amount to obtain training backup data includes: randomly extracting multimodal medical case data in sequence from the multimodal medical training data; Performing case statistics on the multimodal medical case data to obtain a random case number; Determining whether the number of random cases is less than the unit verification data volume; If the number of random cases is less than the unit verification data amount, returning to the above step of randomly extracting multimodal medical case data in the multimodal medical training data in sequence; If the number of random cases is not less than the unit verification data volume, all multimodal medical case data are aggregated to obtain initial verification data; Identify an initial verification disease name set of the initial verification data, and determine whether the initial verification disease name set is a true subset of a preset confirmed disease name set; If the initial verification disease name set is a true subset of the confirmed disease name set, returning to the above step of randomly extracting multimodal medical case data from the multimodal medical training data in sequence; If the initial verification disease name set is not a true subset of the confirmed disease name set, the initial verification data is used as the prediction verification data; Eliminating the prediction verification data from the multimodal medical training data to obtain training backup data; The step of extracting iterative training data from the training backup data according to the preset unit training data amount and the initial sampling ratio to obtain iterative backup data includes: According to the unit training data volume and the initial sampling ratio, the number of text medical data cases, the number of image medical data cases and the number of voice medical data cases are calculated using the following formula: ； in, represents the number of medical data cases in the ith case, Represents the number of cases in text medical data, represents the number of cases of medical imaging data, Indicates the number of cases of voice medical data, represents the unit training data volume, Represents the i-th sampling ratio coefficient in the initial sampling ratio; According to the number of text medical data cases, the number of image medical data cases and the number of voice medical data cases, text medical case data, image medical case data and voice medical case data are respectively extracted from the training backup data to obtain iterative training data; Eliminating the iterative training data from the training backup data to obtain iterative backup data; The method of using the prediction verification data to verify the prediction error of the iterative multimodal large model to obtain the prediction error ratio and the maximum prediction error includes: Classifying the predicted verification data to obtain text verification case data, image verification case data, and voice verification case data; Converting the text verification case data, image verification case data and voice verification case data into a text verification vector, an image verification vector and a voice verification vector respectively; Sequentially extracting a single text verification vector, a single image verification vector, and a single voice verification vector from the text verification vector, the image verification vector, and the voice verification vector, respectively; Inputting the single-instance text verification vector, the single-instance image verification vector, and the single-instance voice verification vector into the iterative multimodal large model respectively to obtain an iterative verification value; Determine whether the text verification vector, the image verification vector, and the voice verification vector have completed the extraction of a single text verification vector, a single image verification vector, and a single voice verification vector; If the text verification vector, the image verification vector, and the voice verification vector have not completed the extraction of a single text verification vector, a single image verification vector, and a single voice verification vector, then return to the above steps of extracting a single text verification vector, a single image verification vector, and a single voice verification vector from the text verification vector, the image verification vector, and the voice verification vector respectively; If the text verification vector, the image verification vector and the voice verification vector complete the extraction of a single text verification vector, a single image verification vector and a single voice verification vector, then the iterative verification values ​​are summarized to obtain an iterative verification value set; Classifying the iterative validation value set according to the confirmed disease name set to obtain multiple groups of validation prediction value sets; Extracting validation prediction value sets in sequence from the multiple sets of validation prediction value sets, and identifying the target center value of the target disease name value domain corresponding to the validation prediction value sets; Calculate the disease name prediction error set between the verification prediction value set and the target central value, calculate the disease name prediction error mean according to the disease name prediction error set, and obtain the disease name prediction error mean set; The disease name prediction error mean set is proportionally reduced to obtain the prediction error ratio, the maximum disease name prediction error mean is extracted from the disease name prediction error mean set, and the maximum disease name prediction error mean is used as the maximum prediction error.

2. The key information extraction method based on multimodal large model as claimed in claim 1, characterized in that: The obtaining of the initial multimodal large model comprises: Construct the predicted disease name output node; Setting a medical text input node, a medical image input node, and a medical voice input node to obtain a multimodal input layer node set; According to the multimodal input layer node set and the predicted disease name output node, an initial multimodal large model is constructed using a pre-constructed BP neural network.

3. The key information extraction method based on multimodal large model as claimed in claim 2 is characterized in that: The using the iterative training data to train the initial multimodal large model to obtain the iterative multimodal large model includes: Performing word segmentation encoding processing on the text medical case data to obtain initial encoded text data; Standardizing the initial encoded text data to obtain standard text data; Performing vector conversion on the standard text data using a pre-built word embedding technology to obtain a text vector; Resizing the image medical case data to obtain image data of the same size; Normalizing the image data of the same size to obtain standard image data, and extracting features from the standard image data using a pre-built convolutional neural network to obtain an image vector; Convert the vocal medical case data to obtain vocal digital signal data; De-noising and segmenting the digital sound signal data to obtain standard sound data; Extracting a vocalization vector from the standard vocalization data using a pre-built feature extraction algorithm; Extracting a single text vector, a single image vector and a single voice vector from the text vector, the image vector and the voice vector respectively; Inputting the single-instance text vector, the single-instance image vector and the single-instance voice vector from the medical text input node, the medical image input node and the medical voice input node into the initial multimodal large model respectively to obtain an iterative output value; Obtaining a single confirmed disease name corresponding to the single text vector, the single image vector, and the single voice vector; Extracting the predicted disease name value range of the single confirmed disease name from the pre-constructed disease name value range table, and extracting the value range center value from the predicted disease name value range; Calculate the single-case prediction error between the iterative output value and the central value of the value range, and determine whether the single-case prediction error is less than the prediction error threshold; If the single-case prediction error is not less than the prediction error threshold, feedback-adjusting the initial multimodal large model according to the single-case prediction error until the single-case prediction error is less than the prediction error threshold; Determining whether the text vector, the image vector, and the voice vector have completed the extraction of a single text vector, a single image vector, and a single voice vector; If the text vector, image vector and voice vector have not completed the extraction of a single text vector, a single image vector and a single voice vector, then return to the above steps of extracting a single text vector, a single image vector and a single voice vector from the text vector, image vector and voice vector respectively; If the text vector, image vector and voice vector complete the extraction of a single text vector, a single image vector and a single voice vector, an iterative multimodal large model is obtained.

4. The key information extraction method based on multimodal large model as claimed in claim 3 is characterized in that: The method of predicting a disease name based on the multimodal medical current data and using the target multimodal large model to obtain a predicted disease name includes: Classifying the multimodal medical current data to obtain text current case data, image current case data and voice current case data; Converting the text current case data, the image current case data and the voice current case data into a text current vector, an image current vector and a voice current vector respectively; Inputting the text current vector, the image current vector and the utterance current vector into the target multimodal large model respectively to obtain a target prediction value; According to the target predicted numerical value, a target disease name value range is identified in the disease name value range table, and a predicted disease name corresponding to the target disease name value range is identified.

5. The key information extraction method based on multimodal large model as claimed in claim 4 is characterized in that: The extracting key information from the multimodal medical current data according to the predicted disease name includes: Extracting text keywords, image key areas, and vocalization key indicators from a pre-constructed disease name-multimodal key data table according to the predicted disease name; Extract key text data, key image areas and key vocalization index values ​​from the text current case data, the image current case data and the vocalization current case data respectively according to the text keywords, the image key areas and the vocalization key indicators; The key text data, key image areas and key vocalization index values ​​are summarized to obtain key information.

6. The key information extraction method based on multimodal large model as claimed in claim 5 is characterized in that: Before extracting text keywords, image key areas, and phonation key indicators from a pre-constructed disease name-multimodal key data table according to the predicted disease name, the method further includes: Extracting confirmed disease names in sequence from the confirmed disease name set; Receive the disease name text keywords, disease name image key areas and disease name pronunciation key indicators set by the user for the confirmed disease name; A disease name-multimodal key data table is constructed based on the confirmed disease name and the disease name text keywords, disease name image key areas and disease name pronunciation key indicators.

7. A key information extraction system based on a multimodal large model using the method as claimed in claim 1, characterized in that: The system comprises: An initial multimodal large model training module is used to obtain multimodal medical training data, wherein the multimodal medical training data includes: medical text training data, medical image training data, medical voice training data and confirmed disease name training data; extracting prediction verification data from the multimodal medical training data according to a preset unit verification data volume, obtaining training backup data, and obtaining an initial multimodal large model, wherein the initial multimodal large model is constructed based on a BP neural network; extracting iterative training data from the training backup data according to a preset unit training data volume and an initial sampling ratio, obtaining iterative backup data; using the iterative training data to train the initial multimodal large model to obtain an iterative multimodal large model; An iterative multimodal large model error verification module is used to verify the prediction error of the iterative multimodal large model using the prediction verification data to obtain the prediction error ratio and the maximum prediction error; a target multimodal large model acquisition module, used to determine whether the maximum prediction error is less than a preset prediction error threshold; if the maximum prediction error is not less than the prediction error threshold, then respectively using the iterative backup data, the iterative multimodal large model and the prediction error ratio to update the training backup data, the initial multimodal large model and the initial sampling ratio, and returning to the above step of extracting iterative training data from the training backup data according to the preset unit training data amount and the initial sampling ratio; if the maximum prediction error is less than the prediction error threshold, then using the iterative multimodal large model as the target multimodal large model; The key information extraction module is used to receive multimodal medical current data, predict disease names based on the multimodal medical current data using the target multimodal large model to obtain predicted disease names; and extract key information from the multimodal medical current data based on the predicted disease names.

Images