Multimodal disease assistance reasoning system, method, terminal and storage medium

Abstract

The application provides a multi-modal disease auxiliary reasoning system and method, a terminal and a storage medium, and particularly relates to the technical field of artificial intelligence, and comprises a multi-modal information acquisition module, a multi-modal information preprocessing module, an entity extraction module, a medical causal knowledge database and a multi-modal classification and discrimination module connected in sequence, realizes preprocessing, entity extraction and modal classification and discrimination of various modal information of a patient, matches the entity with medical knowledge data in the medical causal knowledge database, then extracts various modal features based on the matching result, splices the various modal features according to a set rule, inputs a classifier to obtain a reasoning result. The scheme fully utilizes various modal features, the extracted entity is more accurate and comprehensive, can improve the accuracy of matching the entity with the medical causal knowledge database, has better generalization, the reasoning result is more accurate, and is beneficial to improving the artificial diagnosis efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal disease-aided reasoning system, method, terminal, and storage medium. Background Technology

[0002] In recent years, due to advancements in artificial intelligence, especially natural language processing technology, deep learning-based disease-assisted diagnostic systems have garnered increasing attention from the research community and industrial companies. The integration of emerging AI technologies with medical data and knowledge empowers smart healthcare, significantly improving the quality and efficiency of medical care. By learning from the rich clinical experience contained in patients' electronic medical records, and building disease-assisted diagnostic systems based on deep learning neural networks to implement classification and prediction models, these systems can help doctors make rapid and accurate diagnostic decisions.

[0003] Currently, most deep learning-based disease-aided diagnostic models only use electronic medical records as input. They extract medical entities and use the patient's clinical data, such as physical signs, symptoms, and examination results, as input to the neural network to output the most probable inference result for the patient. Although this approach is simple, it only considers textual information and does not fully take into account the various forms of examination data of the patient, resulting in inaccurate inference results for the patient's disease. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a multimodal disease-assisted reasoning system, method, terminal and storage medium, which aims to solve the problem that the reasoning results of the existing deep learning-based disease-assisted diagnosis models are not accurate enough.

[0005] To achieve the above objectives, the first aspect of the present invention provides a multimodal disease-assisted reasoning system, comprising a multimodal information acquisition module, a multimodal information preprocessing module, an entity extraction module, a medical causal knowledge database, and a multimodal classification and discrimination module connected in sequence.

[0006] The multimodal information acquisition module is used to acquire several modal information of the patient and input the text modal information in the modal information into the multimodal information preprocessing module;

[0007] The multimodal information preprocessing module is used to preprocess the received text modal information to obtain preprocessed text modal information, and input the preprocessed text modal information into the entity extraction module and the multimodal classification and discrimination module;

[0008] The entity extraction module is used to extract entities from the received preprocessed text modal information using a named entity recognition model, and input the entities into the multimodal classification and discrimination module.

[0009] The medical causal knowledge database is used to store collected medical knowledge data in the target field;

[0010] The multimodal classification and discrimination module includes a multimodal classification model and a classifier, used to match the received entities with the medical knowledge data. If the match is successful, the first-order predicate logic rule of the match is output; otherwise, the multimodal classification model is invoked, the preprocessed text modal information is input into the multimodal classification model, various modal features are extracted, and the various modal features are concatenated to obtain concatenated features; the concatenated features are input into the classifier to obtain the inference result and output it.

[0011] Optionally, the multimodal information preprocessing module includes a text cleaning unit, a text segmentation unit, and a text splitting unit.

[0012] The text cleaning unit is used to clean the text modal information, obtain the cleaned text modal information, and input it into the text segmentation unit.

[0013] The text segmentation unit is used to perform word segmentation on the cleaned text modal information based on the medical word segmentation library, obtain the segmented text modal information, and input the segmented text modal information into the text segmentation unit;

[0014] The text segmentation unit is used to segment the word-segmented text modal information based on a preset window to obtain the preprocessed text modal information.

[0015] Optionally, the entity extraction module includes a text vector representation module, a label prediction module, and an entity filtering module.

[0016] The text vector representation module is used to input the preprocessed text modal information into a pre-trained language learning model to obtain the vector representation of the text, and then input the vector representation of the text into the label prediction module;

[0017] The tag prediction module is used to input the vector representation of the text into the named entity recognition model to obtain a tag sequence; and based on the probability distribution of each tag in the tag sequence, to establish the constraint relationship between the tags using the named entity recognition model to obtain the predicted tags, and to input the predicted tags into the entity filtering module.

[0018] The entity filtering module is used to filter text based on the predicted labels to obtain entities.

[0019] Optionally, constructing the medical causal knowledge database includes the following steps:

[0020] Collect medical knowledge data in the target field;

[0021] Based on first-order predicate logic, the collected medical knowledge data is transformed using first-order predicate logic to obtain the first-order predicate logic of the medical knowledge data.

[0022] A medical causal knowledge database is constructed using the medical knowledge data and the first-order predicate logic of the medical knowledge data.

[0023] Optionally, the multimodal classification and discrimination module includes an entity and predicate vector representation module, a similarity calculation module, and a matching processing module.

[0024] The entity and predicate vector representation module is used to use a pre-trained language learning model to perform vector representation on the entity and the first-order predicate logic in the medical causal knowledge database, output the vector representation of the entity and the vector representation of the first-order predicate logic, and input the vector representation of the entity and the vector representation of the first-order predicate logic into the similarity calculation module.

[0025] The similarity calculation module is used to calculate the similarity between the received vector representation of the entity and the vector representation of the first-order predicate logic, and input the similarity to the matching processing module;

[0026] The matching processing module is used to store the matched entities into the medical causal knowledge database and output the matching first-order predicate logic rules when the received similarity is greater than or equal to a preset threshold.

[0027] Optionally, the multimodal classification and discrimination module further includes a feature extraction unit and a feature concatenation unit.

[0028] The feature extraction unit is used to input the preprocessed text modal information into the multimodal classification model, and use a convolutional layer with filtering function to extract features from the image modal information of the patient's modal information to obtain image features; and use the pre-trained language learning model to obtain text features and special identifier features.

[0029] The feature splicing unit is used to splice the image features, the text features, and the special identifier features to obtain the spliced ​​features.

[0030] Optionally, concatenating the image features, the text features, and the special identifier features to obtain the concatenated features includes:

[0031] The special identifier features include a first vector representation for indicating the start position of the image features, a second vector representation for indicating the segmentation of the image and text features and the end position of the input, and a third vector representation for padding the input text to a preset length.

[0032] The first vector representation, the image feature, the second vector representation, the text feature, the third vector representation, and the second vector representation are sequentially concatenated to obtain the concatenated feature.

[0033] A second aspect of the present invention provides a multimodal disease-assisted reasoning method, comprising the following steps:

[0034] Acquire several modalities of patient information;

[0035] The text modal information in the modal information is preprocessed to obtain preprocessed text modal information;

[0036] Entities are obtained by extracting entities from the preprocessed text modal information using a named entity recognition model.

[0037] Construct a medical causal knowledge database;

[0038] The entity is matched with medical knowledge data in the medical causal knowledge database. If the match is successful, the first-order predicate logic rule of the match is output. Otherwise, the multimodal classification model is called, the preprocessed text modal information is input into the multimodal classification model, various modal features are extracted, and the various modal features are concatenated to obtain concatenated features. The concatenated features are input into the classifier to obtain the inference result and output it.

[0039] A third aspect of the present invention provides an intelligent terminal, the intelligent terminal including a memory, a processor, and a multimodal disease-aided reasoning program stored in the memory and executable on the processor, wherein the multimodal disease-aided reasoning program, when executed by the processor, performs any of the functions of the aforementioned multimodal disease-aided reasoning system.

[0040] A fourth aspect of the present invention provides a computer-readable storage medium storing a multimodal disease-aided reasoning program, wherein the multimodal disease-aided reasoning program, when executed by a processor, implements any of the functions of the aforementioned multimodal disease-aided reasoning system.

[0041] Compared with existing technologies, the beneficial effects of this solution are as follows:

[0042] This invention constructs a multimodal disease-assisted reasoning system, which is sequentially connected by a multimodal information acquisition module, a multimodal information preprocessing module, an entity extraction module, a medical causal knowledge database, and a multimodal classification and discrimination module. This system enables the preprocessing, entity extraction, and modal classification and discrimination of multiple modal information of patients. It matches entities with medical knowledge data in the medical causal knowledge database, extracts various modal features (such as image and text features) based on the matching results, and concatenates these features according to predefined rules before inputting them into a classifier to obtain the reasoning results.

[0043] This invention processes patient data across multiple modalities and matches it with expert knowledge data in a medical causal knowledge database. By fully utilizing various modal features, the extracted entities are more accurate and comprehensive, thereby improving the accuracy of matching entities with the medical causal knowledge database. This results in better generalization in new assisted diagnostic environments, more precise reasoning results, and improved efficiency of manual diagnosis. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the multimodal disease-assisted reasoning system of the present invention;

[0046] Figure 2 This is a flowchart of the multimodal disease-assisted reasoning method of the present invention;

[0047] Figure 3 This invention relates to a multimodal disease-aided reasoning intelligent terminal. Detailed Implementation

[0048] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0049] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0050] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0051] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0052] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0055] Most existing deep learning-based disease-aided diagnostic models use electronic medical records as input, extract medical entities, and input patient clinical data, such as physical signs, symptoms, and examination results, into a neural network to provide the most probable inference result for the patient. This process is purely black-box, the results lack interpretability, and its end-to-end nature prevents full utilization of expert knowledge. In clinical diagnosis, however, there is a general desire to demonstrate the inferential relationship between patient symptoms, test indicators, and their inference results. Furthermore, patient data includes not only textual modality information but also a large amount of image information. Therefore, how to integrate expert knowledge with deep learning models to build a disease-aided diagnostic system with diverse and multimodal functions has gradually become a research hotspot.

[0056] With the development of natural language processing and visual language, and addressing the aforementioned problems, this invention constructs a multimodal hepatobiliary and pancreatic disease auxiliary reasoning system using first-order predicate logic, named entity recognition methods, and a multimodal classification model. This system provides auxiliary diagnostic functions in multimodal scenarios, achieving higher diagnostic accuracy and expanding its application scenarios. We utilize existing first-order predicate logic construction methods, named entity recognition methods, and multimodal classification models to construct a multimodal hepatobiliary and pancreatic disease auxiliary reasoning system, which can improve the efficiency and quality of diagnosis and treatment in practical scenarios.

[0057] Exemplary System

[0058] This invention provides a multimodal disease-assisted reasoning system, which is deployed on electronic devices such as computers and servers. Its application fields include medical diagnosis, and its application scenarios include assisted reasoning for various internal and surgical diseases such as hepatobiliary and pancreatic diseases, pneumonia, and cardiovascular diseases.

[0059] Specifically, the structural diagram of the system in this embodiment is as follows: Figure 1As shown, the system includes a multimodal information acquisition module 110, a multimodal information preprocessing module 120, an entity extraction module 130, a medical causal knowledge database 140, and a multimodal classification and discrimination module 150 connected in sequence. The multimodal information acquisition module 110 acquires several modalities of the patient's information and inputs the text modal information from the modalities into the multimodal information preprocessing module 120. The multimodal information preprocessing module 120 preprocesses the received text modal information to obtain preprocessed text modal information and inputs the preprocessed text modal information into the entity extraction module 130. The entity extraction module 130 uses a named entity recognition model to extract the... The system receives entities from preprocessed text modal information, obtains the entities, and inputs them into the multimodal classification and discrimination module 150. The medical causal knowledge database 140 stores collected medical knowledge data corresponding to the patient's domain. The multimodal classification and discrimination module 150 matches the received entities with the medical knowledge data in the medical causal knowledge database 140. If a match is successful, the matching result is output; otherwise, the multimodal classification model is invoked. The preprocessed text modal information is input into the multimodal classification model to extract features of various modalities, and the features of various modalities are concatenated to obtain concatenated features. The concatenated features are input into the classifier to obtain the inference result and output it.

[0060] In this embodiment, the entity refers to the disease entity such as the patient's symptoms, signs, examination names, and examination index values ​​obtained during the auxiliary diagnosis and treatment stage.

[0061] The multimodal information acquisition module 110 acquires multiple modal information about the patient, converts the modal information into text modal information, and outputs the text modal information to the multimodal information preprocessing module 120. The patient's multiple modal information includes information such as examination reports, admission medical records, and images from imaging examinations.

[0062] The received text modal information is preprocessed by the multimodal information preprocessing module 120 to obtain preprocessed text modal information, and then the preprocessed text modal information is output to the entity extraction module 130 and the multimodal classification and discrimination module 150.

[0063] Specifically, the multimodal information preprocessing module 120 includes a text cleaning unit, a text segmentation unit, and a text splitting unit.

[0064] The text cleaning unit filters patient-input text using preset rules and specific keyword filtering methods, removing spaces, symbols, and private information to clean the text modal information, obtaining the cleaned text modal information, which is then input into the text segmentation unit. The text segmentation unit, based on a medical word segmentation library, utilizes a medical subject thesaurus and the Python word segmentation library jieba to perform word segmentation on the received cleaned text modal information, obtaining segmented text modal information that makes medical subject terms as complete as possible, and inputs the segmented text modal information into the text splitting unit. The text splitting unit, based on a preset window, segments the received segmented text modal information to obtain preprocessed text modal information, ensuring the length of the segmented text meets the input length. When the length of the segmented text is less than the preset input window size, it is padded with the special character [PAD], where [PAD] is a special identifier feature used to complete the input text to the specified length.

[0065] The entity extraction module 130 is used to extract entities from the received preprocessed text modal information using a named entity recognition model, obtain entities, and output the entities to the multimodal classification and discrimination module 150.

[0066] Specifically, the entity extraction module 130 includes a text vector representation module, a label prediction module, and an entity filtering module.

[0067] The text vector representation module is used to represent the information in the preprocessed patient text modality using a deep learning model based on BERT pretraining. It takes a text of a pre-set length as input and outputs a vector representation of the text of the corresponding length. The beginning of the text is marked with the special character [CLS], the insufficient part is marked with the special character [PAD], and the end of the text is marked with the special character [SEP].

[0068] The label prediction module is used to input the vector representation of the text into the named entity recognition model to obtain a label sequence; and based on the probability distribution of each label in the label sequence, it uses the named entity recognition model to establish the constraint relationship between the labels to obtain the predicted labels, and then inputs the predicted labels into the entity filtering module.

[0069] This embodiment uses BIO annotation, where B represents the start position of the entity, I represents the position inside the entity, and O represents the position outside the entity. Labels are added during the training process of the model, and the corresponding BIO labels are directly predicted during the prediction phase of the model. When the model predicts the label of a word, it returns the probability of all labels. The CRF layer is used to establish constraints between these probabilities. For example, if the previous word is labeled O (outside the entity), then the current word cannot be labeled I (inside the entity). The text embedding vector (i.e., the vector representation of the text) is input into the BiLSTM layer to obtain the latent vector representation. The BiLSTM layer is a bidirectional LSTM layer, including a forward LSTM layer and a backward LSTM layer. The latent vector representation is shown in formulas (1)-(3):

[0070]

[0071]

[0072]

[0073] in, This represents the output vector representation of the feedforward LSTM layer. x represents the output vector of the inverse LSTM layer. i h represents a vector in the input label sequence. i The final hidden state output (i.e., the latent vector representation) of the BiLSTM is formed by concatenating the forward hidden state and the backward hidden state of the LSTM.

[0074] The obtained potential vectors are represented as h i The label sequence matrix h is input into the CRF layer to establish the constraint relationship between the labels, and the final prediction result is obtained. Specifically, as shown in formula (4):

[0075]

[0076] Where γ(h) represents all possible label sequences, and W and b are two learnable weight matrices, representing the weights of a given label pair (yi, yj, ... i-1 ,y i The weight of y′ i The current possible label, y′ i-1 , is a possible tag of the previous text. It is a learnable weight matrix between two texts. The learned weights can be used to constrain the relationship between the two labels. Based on the final output label probability value, the label at the current position is predicted, where n represents the number of labels.

[0077] The entity filtering module is used to filter text based on predicted labels to obtain entities.

[0078] In this embodiment, the labeling is in the form of BIO, and the text between B and O is selected as the complete entity output, along with the entity category.

[0079] The Medical Causal Knowledge Database 140 is used to store collected medical knowledge data in the target field.

[0080] Specifically, the construction of a medical causal knowledge database 140 includes the following steps:

[0081] Collect medical knowledge data in the target domain. Medical knowledge data in the target domain refers to medical knowledge data specific to a particular medical diagnosis, such as expert consensus data and treatment guideline data.

[0082] Construct a medical causal knowledge database. Analyze collected medical knowledge, perform first-order predicate logic transformation on the medical knowledge, and construct the first-order predicate logic for the medical causal knowledge data; use the medical knowledge data and its first-order predicate logic to construct the medical causal knowledge database. An example of the first-order predicate logic expression is shown below:

[0083] ExCo(Biopsy(|LaboratoryExam|),unsure(|Examconclusion|))

[0084] →2-3months-VideoFollowUp(|Exam|)

[0085] The meaning of this first-order predicate logical expression is: if the result of the biopsy in the laboratory is indeterminate (ExCo), then 2-3 months of video follow-up is required, and further judgment should be made based on the results of the video follow-up.

[0086] The multimodal classification and discrimination module 150 includes a multimodal classification model and a classifier, which is used to match the received entities with the medical knowledge data in the medical causal knowledge database 140. If the match is successful, the matching result, i.e., the first-order predicate logic rule of the match, is output. Otherwise, the multimodal classification model is called, the preprocessed text modal information is input into the multimodal classification model, various modal features are extracted, and the various modal features are concatenated to obtain the concatenated features. The concatenated features are input into the classifier to obtain the inference result and output it.

[0087] Specifically, the multimodal classification and discrimination module 150 includes an entity and predicate vector representation module, a similarity calculation module, a matching processing module, a feature extraction unit, and a feature concatenation unit.

[0088] The entity and predicate vector representation module is used to use a deep learning model based on BERT pretraining to perform vector representation of entities and first-order predicate logic in the medical causal knowledge database 140, output the vector representation of entities and the vector representation of first-order predicate logic, and input the vector representation of entities and the vector representation of first-order predicate logic into the similarity calculation module.

[0089] The similarity calculation module is used to calculate the similarity between the vector representation of the received entity and the vector representation of the first-order predicate logic, and inputs the similarity into the matching processing module. The specific calculation of the similarity is shown in formula (5):

[0090]

[0091] Where, x i and y i These are the entities output by the model and the elements in the embedding vector of the first-order predicate logic, respectively.

[0092] The matching module is used to determine the cosine similarity with a preset threshold ρ. x The similarity is related to the magnitude of the similarity when it is greater than or equal to a preset threshold, i.e., cos(x,y)≥ρ. x The matched entities are stored in the medical causal knowledge database 140, and the matching first-order predicate logic rules are output; otherwise, the multimodal classification model is called for processing.

[0093] The feature extraction unit is used to input the preprocessed text modal information into the multimodal classification model, and to extract image features using a convolutional layer with filtering function to obtain image features; a pre-trained language learning model is used to obtain text features and special identifier features; the feature concatenation unit is used to concatenate image features, text features and special identifier features to obtain concatenated features.

[0094] In this embodiment, a convolutional layer with filtering function is used for image feature extraction. A shared-weight convolutional kernel is used to perform convolution operations on the image to extract local image features. Simultaneously, downsampling is used to merge neighboring features into a single feature to reduce the feature dimensionality, resulting in a high-dimensional mapping feature of the image. Here, neighboring features refer to features within a specified region of the image, and downsampling refers to selecting the point with the largest value or the average value within the region to represent that region, thereby reducing the image dimensionality. The specific calculation is shown in formula (6):

[0095] F = f cnn (I;θ cnn ), F∈R H×W×N (6)

[0096] Where, θ cnnThe parameters of the feature extraction module are: F represents the extracted features, N represents the number of feature channels, and I represents the given image with dimensions H×W.

[0097] A deep learning model based on BERT pre-training is used to input the pre-processed text modality information into the pre-trained language learning model to obtain the corresponding vector representations, including the vector representations of special identifier features ([CLS], [SEP], [PAD]). The vector at the position of the special identifier [SEP] is used as the semantic representation vector of the current text field.

[0098] In this diagram, the vector representation of [CLS] indicates the start of image features, the vector representation of [SEP] indicates the segmentation of image and text features and the end of the input, and the vector representation of [PAD] is used to pad the input text to the specified length. Image features, text features, and special identifier features are concatenated in the form of [CLS; image features; SEP; text features; PAD; SEP], and the concatenated features are used as the output.

[0099] Then, the concatenated features are input into the classifier, which outputs the final inference result.

[0100] Specifically, the concatenated features are input into a fully connected layer and an activation layer in the neural network for disease prediction. The activation layer uses the Sigmoid function to normalize the prediction score to between 0 and 1. The specific calculation is shown in formulas (7)-(8):

[0101] Y = W m X+b m (7)

[0102]

[0103] Where X represents the concatenated image modal features and text modal features, and W... m b m Let p be the learnable weight matrix, and p be the final predicted disease probability.

[0104] If the neural network output p is greater than the preset threshold ε (e.g., 0.5 or 0.7), the prediction result is considered positive, indicating that the prediction result has reference value; otherwise, it is negative, indicating that the prediction result has no reference value.

[0105] The pre-trained language learning model used in this embodiment is a deep learning model based on BERT pre-training. As other preferred embodiments, deep learning models based on BioBERT pre-training can also be selected according to actual application needs, and maximum positive matching models can also be selected, not limited to pre-trained language learning models.

[0106] This solution processes patient data across multiple modalities and matches it with expert knowledge data from a medical causal knowledge database. By fully utilizing data from various dimensions, it achieves better generalization in new assisted diagnostic environments, adapting to the needs of assisted diagnosis in different contexts and improving diagnostic effectiveness. Simultaneously, it assists in manual diagnosis, improving its efficiency and ultimately enhancing the efficiency and accuracy of diagnosis in real-world clinical scenarios.

[0107] Exemplary methods

[0108] like Figure 2 As shown, corresponding to the above-mentioned multimodal disease-assisted reasoning system, this embodiment of the invention also provides a multimodal disease-assisted reasoning method, including the following steps:

[0109] Step 210: Obtain several modalities of patient information;

[0110] Step 220: Preprocess the text modal information in the modal information to obtain the preprocessed text modal information;

[0111] Step 230: Use the named entity recognition model to extract entities from the preprocessed text modal information to obtain the entities;

[0112] Step 240: Construct a medical causal knowledge database;

[0113] Step 250: Match the received entity with the medical knowledge data in the medical causal knowledge database. If the match is successful, output the first-order predicate logic rule of the match. Otherwise, call the multimodal classification model, input the preprocessed text modal information into the multimodal classification model, extract the modal features of each type, and concatenate the modal features to obtain the concatenated features. Input the concatenated features into the classifier to obtain the inference result and output it.

[0114] Specifically, in this embodiment, the specific functions of the above-mentioned multimodal disease-assisted reasoning system can also be referred to the corresponding description in the above-mentioned multimodal disease-assisted reasoning method, and will not be repeated here.

[0115] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 3As shown. The aforementioned smart terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a multimodal disease-aided reasoning program. The internal memory provides an environment for the operation of the operating system and the multimodal disease-aided reasoning program stored in the non-volatile storage medium. The network interface of the smart terminal is used for communication with external terminals via a network connection. When executed by the processor, the multimodal disease-aided reasoning program implements the steps of any of the aforementioned multimodal disease-aided reasoning methods. The display screen of the smart terminal can be a liquid crystal display (LCD) or an e-ink display.

[0116] Those skilled in the art will understand that Figure 3 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0117] In one embodiment, a smart terminal is provided, the smart terminal including a memory, a processor, and a multimodal disease-aided reasoning program stored in the memory and executable on the processor. When the multimodal disease-aided reasoning program is executed by the processor, it implements the function of any of the multimodal disease-aided reasoning systems provided in the embodiments of the present invention.

[0118] This invention also provides a computer-readable storage medium storing a multimodal disease-aided reasoning program. When executed by a processor, the multimodal disease-aided reasoning program implements the functions of any of the multimodal disease-aided reasoning systems provided in this invention.

[0119] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0120] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0121] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0122] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0123] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of the above modules or units is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0124] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not mean that the essence of the corresponding technical solutions deviates from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multimodal disease-aided reasoning system, characterized in that, It includes a multimodal information acquisition module, a multimodal information preprocessing module, an entity extraction module, a medical causal knowledge database, and a multimodal classification and discrimination module, which are connected in sequence. The multimodal information acquisition module is used to acquire several modal information of the patient and input the text modal information in the modal information into the multimodal information preprocessing module; The multimodal information preprocessing module is used to preprocess the received text modal information to obtain preprocessed text modal information, and input the preprocessed text modal information into the entity extraction module and the multimodal classification and discrimination module; The entity extraction module is used to extract entities from the received preprocessed text modal information using a named entity recognition model, and input the entities into the multimodal classification and discrimination module. The medical causal knowledge database is used to store collected medical knowledge data in the target field; The multimodal classification and discrimination module includes a multimodal classification model and a classifier, used to match the received entities with the medical knowledge data. If the match is successful, the first-order predicate logic rule of the match is output; otherwise, the multimodal classification model is called, the preprocessed text modal information is input into the multimodal classification model, various modal features are extracted, and the various modal features are concatenated to obtain the concatenated features. The concatenated features are input into the classifier to obtain the inference result and output it. The multimodal classification and discrimination module includes a feature extraction unit and a feature concatenation unit; The feature extraction unit is used to input the preprocessed text modal information into the multimodal classification model, and use a convolutional layer with filtering function to extract features from the image modal information in the patient's modal information to obtain image features; and use a pre-trained language learning model to obtain text features and special identifier features. The feature splicing unit is used to splice the image features, the text features, and the special identifier features to obtain the spliced ​​features; The step of concatenating the image features, the text features, and the special identifier features to obtain the concatenated features includes: The special identifier features include a first vector representation for indicating the start position of the image features, a second vector representation for indicating the segmentation of the image and text features and the end position of the input, and a third vector representation for padding the input text to a preset length. The first vector representation, the image feature, the second vector representation, the text feature, the third vector representation, and the second vector representation are sequentially concatenated to obtain the concatenated feature.

2. The multimodal disease-assisted reasoning system according to claim 1, characterized in that, The multimodal information preprocessing module includes a text cleaning unit, a text segmentation unit, and a text splitting unit. The text cleaning unit is used to clean the text modal information, obtain the cleaned text modal information, and input it into the text segmentation unit. The text segmentation unit is used to perform word segmentation on the cleaned text modal information based on the medical word segmentation library, obtain the segmented text modal information, and input the segmented text modal information into the text segmentation unit; The text segmentation unit is used to segment the word-segmented text modal information based on a preset window to obtain the preprocessed text modal information.

3. The multimodal disease-assisted reasoning system according to claim 1, characterized in that, The entity extraction module includes a text vector representation module, a label prediction module, and an entity filtering module. The text vector representation module is used to input the preprocessed text modal information into a pre-trained language learning model to obtain the vector representation of the text, and then input the vector representation of the text into the label prediction module; The tag prediction module is used to input the vector representation of the text into the named entity recognition model to obtain a tag sequence; and based on the probability distribution of each tag in the tag sequence, to establish the constraint relationship between the tags using the named entity recognition model to obtain the predicted tags, and to input the predicted tags into the entity filtering module. The entity filtering module is used to filter text based on the predicted labels to obtain entities.

4. The multimodal disease-aided reasoning system according to claim 1, characterized in that, Constructing the medical causal knowledge database includes the following steps: Collect medical knowledge data in the target field; Based on first-order predicate logic, the collected medical knowledge data is transformed using first-order predicate logic to obtain the first-order predicate logic of the medical knowledge data. A medical causal knowledge database is constructed using the medical knowledge data and the first-order predicate logic of the medical knowledge data.

5. The multimodal disease-aided reasoning system according to claim 1, characterized in that, The multimodal classification and discrimination module includes an entity and predicate vector representation module, a similarity calculation module, and a matching processing module. The entity and predicate vector representation module is used to use a pre-trained language learning model to perform vector representation on the entity and the first-order predicate logic in the medical causal knowledge database, output the vector representation of the entity and the vector representation of the first-order predicate logic, and input the vector representation of the entity and the vector representation of the first-order predicate logic into the similarity calculation module. The similarity calculation module is used to calculate the similarity between the received vector representation of the entity and the vector representation of the first-order predicate logic, and input the similarity to the matching processing module; The matching processing module is used to store the matched entities into the medical causal knowledge database and output the matching first-order predicate logic rules when the received similarity is greater than or equal to a preset threshold.

6. A multimodal disease-assisted reasoning method, characterized in that, The multimodal disease-assisted reasoning method is implemented based on the multimodal disease-assisted reasoning system according to any one of claims 1-5, and the method includes the following steps: Acquire several modalities of patient information; The text modal information in the modal information is preprocessed to obtain preprocessed text modal information; Entities are obtained by extracting entities from the preprocessed text modal information using a named entity recognition model. Construct a medical causal knowledge database; The entity is matched with medical knowledge data in the medical causal knowledge database. If the match is successful, the first-order predicate logic rule of the match is output. Otherwise, the multimodal classification model is called, the preprocessed text modal information is input into the multimodal classification model, various modal features are extracted, and the various modal features are concatenated to obtain concatenated features. The concatenated features are input into the classifier to obtain the inference result and output it.

7. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a multimodal disease-aided reasoning program stored in the memory and executable on the processor. When the multimodal disease-aided reasoning program is executed by the processor, it performs the functions of the multimodal disease-aided reasoning system as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multimodal disease-aided reasoning program, which, when executed by a processor, implements the functions of the multimodal disease-aided reasoning system as described in any one of claims 1-5.

Images