Inner ear disease diagnosis model construction method, system, medium, product and terminal

Abstract

The present application provides a method, system, medium, product and terminal for constructing an inner ear disease diagnosis model, constructs a multimodal entity based on a deep learning model, associates entity objects of various modalities, unifies heterogeneous multimodal underlying representations into the same high-level semantic feature space, and mines the intrinsic correlation of disease-related factors in massive multimodal data sources to construct a multimodal knowledge graph. Using the prompt learning analogy reasoning algorithm and the transfer learning algorithm, the multimodal pre-trained model is subjected to knowledge enhancement operations, task transfer operations and self-supervised model training operations, adding more professional knowledge guidance and rich context perception capabilities to the model to generate a multimodal consultation model. The multimodal consultation model is subjected to multiple rounds of iterative training to generate an inner ear disease diagnosis model, accelerate the learning and application of the inner ear disease diagnosis model in the medical field, realize a new type of intelligent decision-making assistance and intervention system, and ultimately realize early diagnosis and treatment of the disease.

Description

Technical Field

[0001] The present application relates to the technical field of medical device data transmission, and in particular to a method, system, medium, product and terminal for constructing an inner ear disease diagnosis model. Background Art

[0002] In the field of medicine and bioinformatics, disease knowledge mining is a crucial task that helps us gain a deeper understanding of the causes, development processes, and treatment strategies of diseases. Disease-related knowledge is often distributed in a variety of data formats, including text, pathological information, biological information, clinical indications, historical disease archives, and diagnostic images. Therefore, it is very necessary to make full use of various disease-related information, unify various types of information into high-level feature spaces, and mine the intrinsic correlation of disease-related factors in massive multimodal data sources for early diagnosis and treatment of diseases.

[0003] However, traditional disease knowledge mining methods mainly rely on analyzing entities and relationships in text databases, which usually contain rich medical knowledge and disease-related information, such as the name of the disease, symptoms, treatment methods, etc. However, this method can only mine the knowledge explicitly mentioned in the text, and cannot fully utilize data from other modalities, such as medical images, pathological sections, etc., which often contain richer and deeper disease-related factors and are crucial to understanding the nature and internal connections of the disease.

[0004] Since traditional methods mainly focus on entities and relationships in text data, they can often only mine superficial and direct knowledge associations. However, the occurrence and development of diseases often involve complex biological processes and networks, and there are deep and indirect associations between the factors in these processes and networks. It is difficult for traditional methods to discover these deep intrinsic associations, making it difficult to strengthen and iteratively optimize knowledge in professional fields, resulting in misunderstandings and deviations when mining knowledge. In addition, with the continuous deepening of medical research and the continuous emergence of new knowledge, it is difficult for traditional methods to update and optimize existing knowledge bases and models in a timely manner, thus limiting their effectiveness in practical applications. Summary of the invention

[0005] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a method, system, medium, product and terminal for constructing an inner ear disease diagnosis model, which is used to solve the problems that the existing disease knowledge mining methods cannot fully utilize data of various modalities and cannot mine the deeper intrinsic correlations of disease-related factors. It is difficult to strengthen and iteratively optimize knowledge in professional fields, which leads to technical problems such as misunderstandings and deviations when mining knowledge.

[0006] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a method for constructing an inner ear disease diagnosis model, comprising: obtaining multimodal data related to inner ear diseases from different databases; based on a deep learning model, performing entity information extraction operations, high-level semantic unification operations, and intrinsic association mining operations on each modal data to construct a multimodal knowledge graph; based on the multimodal knowledge graph, using the deep learning model to perform endogenous correlation learning between multimodal entities to generate a multimodal pre-trained model; using a prompt learning analogy reasoning algorithm to perform knowledge enhancement operations, task transfer operations, and self-supervised model training operations on the multimodal pre-trained model to generate a multimodal consultation model; based on an inner ear disease cycle database, using a transfer learning algorithm and a Prompt tuning algorithm to perform multiple rounds of iterative training on the multimodal consultation model to generate an inner ear disease diagnosis model.

[0007] In some embodiments of the first aspect of the present application, based on a deep learning model, entity information extraction operations, high-level semantic unification operations, and intrinsic association mining operations are performed on each modal data to construct a multimodal knowledge graph, including: based on a deep learning model, each modal data is encoded to extract the initial features of each modal entity data; the initial features of each modal entity data are embedded into a unified semantic space, and based on a heterogeneous graph attention network model, entity association operations are performed on the initial features of each modal entity data to construct a heterogeneous knowledge graph; the nodes in the heterogeneous knowledge graph represent the initial features of entity data of different modalities, the edges in the heterogeneous knowledge graph represent the similarity between the initial features of entity data of different modalities, and the meta-paths in the heterogeneous knowledge graph represent the association relationship between the initial features of entity data of different modalities; triple relationships and entity association modal features related to inner ear diseases are extracted from the heterogeneous knowledge graph, and multimodal entity feature fusion operations and intrinsic association mining operations are performed through the structure aggregation module in the deep learning model to construct a multimodal knowledge graph.

[0008] In some embodiments of the first aspect of the present application, based on the multimodal knowledge graph, the deep learning model is used to perform endogenous correlation learning between multimodal entities to generate a multimodal pre-trained model, including: integrating the image encoder, text encoder and knowledge encoder in the deep learning model to form a multimodal encoder; using a collaborative training strategy to update the multimodal encoder by sharing parameters or jointly optimizing the objective function; combining different modal entity features generated by the multimodal encoder, and based on a preset scoring function, calculating the probability value between the combined different modal entity features to generate a condensed summary of the semantics, attributes and biomedical knowledge of each modal entity; based on the condensed summary of the semantics, attributes and biomedical knowledge of each modal entity, using association learning to perform endogenous correlation learning between multimodal entities to generate a multimodal pre-trained model.

[0009] In some embodiments of the first aspect of the present application, the hint learning analogy reasoning algorithm is an implicit structure mapping analogy reasoning algorithm.

[0010] In some embodiments of the first aspect of the present application, the formula of the implicit structure mapping analogy reasoning algorithm includes:

[0011]

[0012] in, Indicates the conversion of the input analogy examples and questions into an objective function or template; ‖ indicates the connection operation in the template input, that is, connecting different parts in series to form a complete input; e h Represents the first entity object; e t Represents the second entity object; e q Represents the third entity object; I h Represents the first entity object e h The corresponding image information; T t Represents the second entity object e t The corresponding text information; q Represents the third entity object e q The corresponding question and answer information; T e T stands for implicit abduction; a represents implicit induction; [R] represents I h 、T t [CLS] indicates the start of a category sequence; [SEP] indicates a separator marker used to distinguish different parts in a sequence; and [MASK] indicates an occluded area.

[0013] In some embodiments of the first aspect of the present application, the deep learning model includes a Transformer model.

[0014] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a system for constructing an inner ear disease diagnosis model, including: a data acquisition module for acquiring multimodal data related to inner ear diseases from different databases; a multimodal knowledge graph construction module for performing entity information extraction operations, high-level semantic unification operations, and intrinsic association mining operations on each modal data based on a deep learning model to construct a multimodal knowledge graph; a multimodal pre-trained model generation module for performing endogenous correlation learning between multimodal entities based on the multimodal knowledge graph using the deep learning model to generate a multimodal pre-trained model; a multimodal consultation model generation module for performing knowledge enhancement operations, task transfer operations, and self-supervised model training operations on the multimodal pre-trained model using a prompt learning analogy reasoning algorithm to generate a multimodal consultation model; an inner ear disease diagnosis model generation module for performing multiple rounds of iterative training on the multimodal consultation model based on an inner ear disease cycle database using a transfer learning algorithm and a prompt tuning algorithm to generate an inner ear disease diagnosis model.

[0015] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing an inner ear disease diagnosis model as described above.

[0016] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the method for constructing an inner ear disease diagnosis model as described above.

[0017] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides an electronic terminal, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the method for constructing an inner ear disease diagnostic model as described above.

[0018] As described above, the present application has the following beneficial effects: the present application provides a method, system, medium, product and terminal for constructing an inner ear disease diagnosis model, constructs a multimodal entity based on a deep learning model, associates entity objects of various modalities, unifies heterogeneous multimodal underlying representations into the same high-level semantic feature space, and mines the intrinsic associations of disease-related factors in massive multimodal data sources to construct a multimodal knowledge graph. The multimodal knowledge graph is combined with transformer-based multimodal pre-training, and technologies such as transfer learning, prompt learning, multi-head attention and relationship mapping are used to realize knowledge-enhanced model pre-training and task migration, adding more professional knowledge guidance and rich context perception capabilities to the model, realizing knowledge-enhanced model pre-training and task migration to generate a multimodal consultation model. Based on the inner ear disease cycle database, the multimodal consultation model is iteratively trained for multiple rounds using the transfer learning algorithm and the prompt tuning algorithm to generate an inner ear disease diagnosis model, accelerate the learning and application of the inner ear disease diagnosis model in the medical field, realize a new intelligent decision-making and intervention system, and ultimately realize early diagnosis and treatment of diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a flow chart of a method for constructing an inner ear disease diagnosis model in one embodiment of the present application.

[0020] Figure 2 Shown is a schematic diagram of the process of constructing a multimodal knowledge graph in one embodiment of the present application.

[0021] Figure 3 Shown is a schematic diagram of the process of generating a multimodal pre-training model in one embodiment of the present application.

[0022] Figure 4 Shown is a schematic block diagram of a system for constructing an inner ear disease diagnosis model in one embodiment of the present application.

[0023] Figure 5 Shown is an architecture diagram of an inner ear disease diagnosis model construction system in one embodiment of the present application.

[0024] Figure 6 Shown is a schematic diagram of the structure of an electronic terminal in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0026] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0027] <1> Prompt: It is an interactive method based on natural language processing, which enables communication between users and machines through machine parsing of natural language.

[0028] <2> Self-supervised training model: A training method that does not require manually labeled data, and learns representations by generating labels or supervisory signals through the model itself.

[0029] <3> Graph embedding technology: It is a technology that maps the nodes in a graph or the entire graph structure into a low-dimensional vector space. The main purpose of this technology is to preserve the similarity or relationship between the nodes in the graph and convert these nodes or graph structures into low-dimensional vector representations to facilitate various machine learning tasks or applications.

[0030] <4> Backpropagation algorithm (BP algorithm for short): is an algorithm used to train artificial neural networks (ANN). This algorithm is based on the gradient descent method. By using the gradient descent method to train artificial neural networks, the weights and biases in the neural network are continuously adjusted to make the output of the network as close to the expected output as possible.

[0031] <5> Co-attention Transformer: is a variant based on the Transformer architecture that introduces a new attention mechanism that allows the model to focus on information in both sequences when processing a pair of input sequences. This mechanism is particularly suitable for tasks that require understanding the relationship between two input sequences at the same time, such as machine translation, question answering systems, or dialogue systems.

[0032] <5> HAN (Heterogeneous Graph Attention Network): Heterogeneous Graph Attention Network model is a special graph neural network used to process heterogeneous graphs containing different types of nodes and links. This network model can effectively capture and represent the importance of different nodes and links in heterogeneous graphs by introducing the attention mechanism.

[0033] <6> Collaborative Training: is a machine learning strategy that involves the joint training of multiple models or multiple tasks so that they can share knowledge and improve their respective performance.

[0034] <7> MRI (Magnetic Resonance Imaging): is a medical imaging technique that uses powerful magnetic fields, radio waves, and computer technology to produce detailed images of the body's internal structures.

[0035] <8> CT (Computed Tomography): is a medical imaging technique that produces cross-sectional images of the inside of the body by combining special X-ray equipment and computer processing.

[0036] <9> Cross-Entropy Loss Function: It is one of the commonly used loss functions in machine learning and deep learning. It measures the difference between the probability distribution predicted by the model and the probability distribution of the true label. The cross-entropy loss function is not only used for optimization during training, but also for evaluating the performance of the model.

[0037] The medical field of inner ear diseases is a highly specialized field that covers a variety of complex and interrelated diseases, such as Meniere's disease, otosclerosis, benign paroxysmal positional vertigo, etc. These diseases not only have unique causes, pathological characteristics and clinical manifestations, but also involve expertise in multiple disciplines such as medicine, biology, and physics. In this field, knowledge mining technology is of great significance for the diagnosis and treatment of diseases and the development of new drugs.

[0038] Traditional inner ear disease knowledge mining methods mainly rely on analyzing entities and relationships in text databases, and can only mine knowledge explicitly mentioned in the text, but cannot fully utilize data from other modalities (such as medical images, pathological sections, etc.) and cannot mine the deeper intrinsic connections between disease-related factors. In addition, the occurrence and development of diseases often involve complex biological processes and networks. There are deep and indirect connections between these processes and factors in the network. It is difficult for traditional methods to discover these deep intrinsic connections, making it difficult to strengthen and iteratively optimize knowledge in professional fields, which leads to misunderstandings and deviations when mining knowledge. In addition, with the continuous deepening of medical research and the continuous emergence of new knowledge, it is difficult for traditional methods to update and optimize existing knowledge bases and models in a timely manner, thus limiting their effectiveness in practical applications.

[0039] In order to solve the problems in the above-mentioned background technology, the present invention provides a method, system, medium, product and terminal for constructing an inner ear disease diagnosis model, aiming to solve the problem that the existing disease knowledge mining methods cannot fully utilize various modalities of data to mine the deeper intrinsic correlations of disease-related factors, and it is difficult to strengthen and iteratively optimize knowledge in professional fields, which leads to technical problems such as misunderstandings and deviations when mining knowledge.

[0040] To facilitate understanding of the embodiments of the present application, first Figure 1 Detailed description. Figure 1 The flowchart of a method for constructing an inner ear disease diagnosis model in an embodiment of the present invention is shown. The method for constructing an inner ear disease diagnosis model in this embodiment mainly includes the following steps:

[0041] S101: Acquire multimodal data related to inner ear diseases from different databases.

[0042] In this embodiment, the database includes but is not limited to a tinnitus disease database, a deafness disease database, a general database, a refractory inner ear disease circulation database, etc. The multimodal data includes multiple modalities such as text modal data, image modal data, and knowledge modal data. The knowledge modal data includes entities such as diseases, symptoms, treatment methods, drugs, side effects, and the relationships between them.

[0043] In this embodiment, obtaining multimodal data includes: collecting existing deafness and tinnitus disease-related guidelines, 10,000-level patient disease data, literature data, diagnosis and treatment plans, medical images, etc. Among them, the 10,000-level patient disease data covers a large number of (up to 10,000) patient clinical cases, medical history, clinical texts and medical images in physical examinations and auxiliary examinations, imaging examination data, three-dimensional nystagmus data, hearing examination data, electronic information entry of pre-treatment information, etc. The data in the general database includes, but is not limited to: deafness guideline consensus, tinnitus guideline consensus, vertigo guideline consensus, deafness literature data, tinnitus literature data, vertigo literature data, related subject guidelines, related subject literature, etc.

[0044] S102: Based on the deep learning model, perform entity information extraction operations, high-level semantic unification operations, and intrinsic association mining operations on each modal data to construct a multimodal knowledge graph.

[0045] In this embodiment, if Figure 2 As shown, a schematic diagram of the process of constructing a multimodal knowledge graph in an embodiment of the present invention is shown. Based on a deep learning model, entity information extraction operations, high-level semantic unification operations, and intrinsic association mining operations are performed on each modal data to construct a multimodal knowledge graph, including:

[0046] S1021: Based on the deep learning model, encode each modal data to extract the initial features of each modal entity data.

[0047] In this embodiment, the deep learning model includes a Transformer model. The Transformer model is a deep learning model based on the self-attention mechanism. The core idea is to capture the dependencies in the input sequence through the self-attention mechanism, thereby generating a more global sequence representation. The Transformer model is mainly composed of two core components, an encoder and a decoder. Through the interaction between the encoder and the decoder, a deep understanding of the input sequence and the ability to generate an output sequence are achieved. The core is the self-attention mechanism, which captures the dependencies between any two positions in the input sequence, so that the Transformer model has an advantage in processing long text sequences.

[0048] In this embodiment, the Transformer model is composed of multiple stacked Transformer encoder layers. Each Transformer encoder layer contains two main sublayers: a multi-head self-attention mechanism and a fully connected feed-forward neural network. In addition, each sublayer is followed by a residual connection and layer normalization. This stacked structure enables the Transformer model to capture dependencies at different levels in the input sequence. Each encoder layer is further encoded based on the output of the previous encoder layer to generate a higher-level representation. By stacking multiple encoder layers, the Transformer model is able to process complex input sequences and generate high-quality output representations.

[0049] In this embodiment, additional knowledge enhancement layers are embedded in the encoder layer of the Transformer model. These layers are specially designed to process and embed the structural information of the traditional knowledge graph to integrate the knowledge encoder into the Transformer model. Based on the cross-modal attention mechanism, the output of the text encoder is allowed to interact with the output of the knowledge encoder to achieve information fusion between modalities. Design pre-training tasks, such as predicting missing entities or relationships in the traditional knowledge graph, or completing the task of linking entities mentioned in the text to the traditional knowledge graph.

[0050] In this embodiment, when encoding each modal data, for multiple modal data such as text modal data, image modal data, and knowledge modal data, the initial features of the surface forms of images, texts, and triples can be extracted based on the encoder of the Transformer model. The encoder parameters of the shared text modal data and knowledge modal data can transfer the knowledge learned by the model in one modality to another modality, thereby improving the model's ability to understand and represent different types of knowledge, so as to extract the initial features of each modal entity data.

[0051] Further, when encoding image modality data, the image encoder in the Transformer model is used to extract the initial features of the image. Exemplarily, the image encoder can be Vision Transformer (ViT), which is a Transformer structure specifically used for computer vision tasks. The pre-training task for image modality data can be a masked object model, in which some areas in the image are randomly masked and the model is asked to predict these masked areas. When encoding text modality data, the text encoder of the Transformer model is used to extract the semantic features of the text. The pre-training task for text modality data is a masked language model (MLM), in which some words in the text are randomly masked and the model is asked to predict these masked words. When encoding knowledge modality data, the knowledge encoder of the Transformer model is used to process knowledge triples, usually mapping entities and relationships into vectors and learning the interactions between them. The pre-training task of knowledge modal data is the link prediction model, which needs to predict the missing links between entities in the traditional knowledge graph or determine whether a given triple is valid. In this way, mask pattern encoding is performed on different modal data, and by randomly masking part of the input data and letting the model predict the masked content, the model's understanding of the input data is enhanced, overcoming the problems of modal missing and modal noise in traditional knowledge graphs. This enables the model to still have strong data completion and fault tolerance capabilities when facing incomplete or noisy data. In addition, mask pattern encoding can also be combined with subsequent training tasks. The model can learn how to compensate for missing information from other modalities and perform robust processing of noise, so that the diagnostic system has strong data missing completion and data fault tolerance capabilities.

[0052] S1022: Embed the initial features of each modal entity data into a unified semantic space, and based on the heterogeneous graph attention network model, perform entity association operations on the initial features of each modal entity data to construct a heterogeneous knowledge graph; the nodes in the heterogeneous knowledge graph represent the initial features of entity data of different modalities, the edges in the heterogeneous knowledge graph represent the similarities between the initial features of entity data of different modalities, and the meta-paths in the heterogeneous knowledge graph represent the association relationship between the initial features of entity data of different modalities.

[0053] In this embodiment, based on the cross-modal embedding algorithm, the initial features of each modality entity data are embedded into a unified semantic space, so that the entity data from different modalities have similar representations in the unified semantic space for subsequent association operations. The cross-modal embedding algorithm is a technology that involves mapping entity data of different modalities (such as text, images, audio, etc.) into a common embedding space. In the cross-modal embedding algorithm, entity data of different modalities are first converted into feature vectors, which usually contain key information of the data. Then, these feature vectors are mapped to a common embedding space, and the distance between entity data of different modalities in the embedding space can reflect the semantic similarity between them. This process usually requires the use of some machine learning or deep learning techniques, such as neural networks, autoencoders, projection matrices, etc.

[0054] In this embodiment, after the initial features of each modality entity data are embedded into a unified semantic space, the heterogeneous graph attention network model (HAN) is used to perform association operations on multimodal entity data features to construct a heterogeneous knowledge graph. The following processes are included:

[0055] (1) Based on the association relationship between entity data of different modalities, a heterogeneous graph is constructed.

[0056] In this embodiment, in the heterogeneous graph, nodes represent entity data, and edges represent the relationship between entity data. Since entity data of multiple modalities are involved, the nodes and edges in the heterogeneous graph have different types.

[0057] (2) Determine the edge connection mode by analyzing the similarity, correlation or other relationships between entity data of different modalities, so as to connect the nodes in the heterogeneous graph, thereby ensuring the connectivity and rationality of the heterogeneous graph.

[0058] (3) Inputting the connected heterogeneous graph into the heterogeneous graph attention network model to perform entity association operation.

[0059] In this embodiment, the heterogeneous graph attention network model uses the attention mechanism to focus on important nodes and edges in the graph and learn the dependencies between them. The node-level attention module in the heterogeneous graph attention network model aims to learn the attention weight between a node and its neighbors, while the semantic-level attention module aims to learn the importance of different meta-paths (i.e., the composite relationship connecting two nodes). By propagating node information on the heterogeneous graph and aggregating the features of neighboring nodes, the heterogeneous graph attention network model can generate a final representation of each node, which contains rich semantic information and association relationships.

[0060] (4) Construct a heterogeneous knowledge graph based on the output of the heterogeneous graph attention network model.

[0061] In this embodiment, the heterogeneous knowledge graph is a graph structure containing multiple types of entities and relationships, which can fully reflect the associations and dependencies between entity data of different modalities. In the heterogeneous knowledge graph, nodes represent the initial features of entity data of different modalities, edges represent the similarities between the initial features of entity data of different modalities, and meta-paths are a path composed of multiple nodes and edges, which are used to represent the associations between the initial features of entity data of different modalities. Through the meta-path pattern, a series of entity nodes and relationship paths are constructed, such as "Disease" → "May Cause" → "Disease", "Disease" → "Differential Diagnosis" → "Disease" pattern. Each knowledge node has its own encoding and is associated with other knowledge nodes through multiple meta-paths. The heterogeneous knowledge graph can capture the complex associations and dependencies between entity data of different modalities and provide rich semantic information for subsequent applications.

[0062] S1023: Extracting triple relationships and entity association modal features related to inner ear diseases from the heterogeneous knowledge graph, and performing multimodal entity feature fusion operations and intrinsic association mining operations through the structure aggregation module in the deep learning model to construct a multimodal knowledge graph.

[0063] In this embodiment, the triple relationship related to inner ear diseases is generally used to describe the association between entities such as diseases, symptoms, and treatment methods. Exemplarily, a triple relationship may be (inner ear disease A, symptoms, tinnitus), where "inner ear disease A" is an entity, "symptoms" is a relationship, and "tinnitus" is an attribute value. The entity-associated modal features are modal features associated with the entities and attribute values ​​in the triple relationship. For example, in the triple relationship of inner ear disease A, "inner ear disease A" may be associated with specific image modal features (such as the shape and size of the lesion area, etc.), while "tinnitus" may be associated with text modal features (such as a text description describing tinnitus).

[0064] In this embodiment, the process of performing the multimodal entity feature fusion operation and the intrinsic association mining operation through the structure aggregation module in the deep learning model includes:

[0065] (1) The extracted entity association modal features are mapped and aligned with the entities in the triple relationship to ensure consistency.

[0066] In this embodiment, for each entity in the triple relationship (such as inner ear disease A), its corresponding entity-associated modal features need to be determined. Mapping alignment: Map the entity-associated modal features to the corresponding entity to ensure the consistency between the entity and its entity-associated modal features, thereby ensuring the correct association between the entity and its related features.

[0067] (2) The mapped entity-related modal features are subjected to feature fusion operations and intrinsic association mining operations to construct a multimodal knowledge graph.

[0068] In this embodiment, the multimodal knowledge graph is a structured data model for representing and storing inner ear medical knowledge, including entities such as diseases, symptoms, drugs, genes, proteins, and the relationships between them.

[0069] In this embodiment, the feature fusion operation adopts methods such as co-attention Transformer, feature concatenation, feature weighting, and attention mechanism. The image modal features and text modal features are fused to obtain the initial representation of the target inner ear disease entity. By utilizing the triple relationship, the entity-associated modal features are represented as the attribute value of the corresponding attribute. Exemplarily, for the triple relationship (inner ear disease A, symptoms, x), the entity-associated modal features of the symptoms of "inner ear disease A" extracted from the text or image can be used as the attribute value x of the attribute "symptoms". Specifically, the inner ear disease entity (such as inner ear disease A) can be used as the central node, and it can be connected to the attribute nodes such as related symptoms and treatment methods through relationship edges. Each attribute node is associated with a set of entity-associated modal features, and the entity-associated modal features can be used as the attribute value of the attribute. In this way, a structured multimodal knowledge graph is constructed to describe the attributes and attribute values ​​of inner ear diseases, as well as the relationships between them.

[0070] In this embodiment, the structural aggregation module in the deep learning model includes a self-attention mechanism module and a multi-head attention module in the Transformer model. The self-attention mechanism module aggregates the attribute information associated with the target inner ear disease entity, and the model can learn which attributes are more important for understanding the entity and performing specific tasks. Exemplarily, the self-attention mechanism module allows the model to independently calculate the attention weight at each position of the sequence to help the model identify the importance between different modal features in the process of multimodal feature fusion, and weight the modal features accordingly, so as to map or project different modal features into a common feature space so that they can be processed and compared under the same framework. The multi-head attention module is used to enable the model to focus on the features of different modalities in different subspaces in parallel in the common feature space to aggregate information from different modalities, thereby generating a comprehensive representation of the fusion of different modal features for subsequent model training or downstream tasks. The fused attribute values ​​are represented as embedded vectors, which capture the semantic information of different modalities and can be used for link prediction or attribute completion tasks in knowledge graphs. In this way, inner ear disease attributes and attribute association information are propagated and aggregated on the target inner ear disease entity through the structure aggregation module.

[0071] In this embodiment, after completing the multimodal entity feature fusion operation, the intrinsic associations between these fused features are further mined, and the structural aggregation module in the deep learning model can learn the interactions and dependencies between different modal features using the attention mechanism, thereby realizing the mining of the intrinsic associations of multimodal entities. For example, by dynamically adjusting the importance of different features, the attention mechanism can help the model pay more attention to the features most relevant to the current task, thereby improving the performance of the model.

[0072] It is worth noting that the present invention fuses image modal features and text modal features to obtain the initial representation of the target inner ear disease entity. The attributes and attribute values ​​of the target inner ear disease entity are represented in a structured form using triple relationship and entity association modal features to obtain a richer and more comprehensive entity description, and reasoning is performed using the fused multimodal features and knowledge graph structure to discover new knowledge and relationships, or to verify existing knowledge. The new relationships and attribute values ​​obtained by reasoning are updated to the knowledge graph to enrich and expand the knowledge representation, thereby providing strong support for subsequent tasks such as disease diagnosis and treatment.

[0073] Furthermore, the Transformer model is selected to process the inner ear disease multimodal data to construct a multimodal knowledge graph for the following reasons:

[0074] First, the Transformer model has significant advantages in processing sequence data. Its self-attention mechanism allows the model to take into account other elements in the sequence while processing an element (such as a word), thereby better understanding the context. This global attention mechanism enables the Transformer to make full use of the global information of the input sequence, which is particularly important when processing multimodal data of inner ear diseases, because multimodal data usually contains information from different sources and forms, and the model needs to be able to fully understand and integrate this information. The self-attention mechanism in the Transformer model can directly calculate the dependency between any two positions, which enables the model to better capture long-distance dependencies. In the field of inner ear diseases, this ability helps to discover the long-term associations between various factors in the development of the disease, providing a deeper understanding of the diagnosis and treatment of the disease.

[0075] Secondly, the parallel processing capability of the Transformer model makes it more efficient when processing long sequences. Compared with traditional recurrent neural networks (RNN) and long short-term memory networks (LSTM), Transformer can process the entire sequence at the same time without having to process elements one by one in order, thereby better capturing global information. When processing multimodal data of inner ear diseases, this helps to discover potential associations and interactions between different modal data, providing more comprehensive information for the construction of knowledge graphs.

[0076] Furthermore, the multi-head attention mechanism of the Transformer model can further improve its ability to process multimodal data. The multi-head attention mechanism applies the self-attention mechanism to multiple subspaces, allowing each subspace to focus on different attention contents. This helps the model better model the complex relationship between information from different sources and forms when processing multimodal data, thereby improving the accuracy and richness of knowledge graph construction.

[0077] In addition, the structure of the Transformer model is flexible and scalable, supporting a variety of pre-trained models and customized model structures. This means that when building a multimodal knowledge graph for inner ear diseases, you can choose a suitable pre-trained model according to specific needs and fine-tune the model structure to suit specific application scenarios. The Transformer model has a powerful representation learning capability and can automatically learn deep feature representations of data. When processing multimodal data on inner ear diseases, this helps discover potential patterns and associations in the data, providing a deeper understanding of the construction of the knowledge graph.

[0078] Finally, the construction of a multimodal knowledge graph requires processing data from different fields and forms, including text, images, audio, knowledge, etc. As a model that can process sequence data, the Transformer model can easily adapt to these different forms of data and extract and integrate information from these data through its powerful self-attention mechanism and parallel processing capabilities. The Transformer model can process different types of data and achieve cross-modal interaction. This makes it possible to fully utilize the complementarity between different modal data when constructing a multimodal knowledge graph for inner ear diseases, thereby improving the accuracy and completeness of the knowledge graph.

[0079] Therefore, the Transformer model has significant advantages in processing inner ear disease multimodal data to construct a multimodal knowledge graph, including its self-attention mechanism, parallel processing capabilities, multi-head attention mechanism, and adaptability to data in different forms. These features enable the Transformer model to better understand and integrate information in multimodal data, thereby constructing a more accurate and richer multimodal knowledge graph.

[0080] S103: Based on the multimodal knowledge graph, use the deep learning model to perform endogenous correlation learning between multimodal entities to generate a multimodal pre-training model.

[0081] In this embodiment, if Figure 3 As shown, a schematic diagram of a process for generating a multimodal pre-training model in an embodiment of the present invention is shown. Based on the multimodal knowledge graph, the deep learning model is used to perform endogenous correlation learning between multimodal entities to generate a multimodal pre-training model, including:

[0082] S1031: Integrate the image encoder, text encoder, and knowledge encoder in the deep learning model to form a multimodal encoder.

[0083] In this embodiment, the image encoder maps the original image data to a low-dimensional, information-rich feature vector. The text encoder converts a text sequence (such as a word, sentence, or paragraph) into a fixed-length vector representation. The knowledge encoder converts knowledge data into a vector representation. The output feature vectors of the image encoder, text encoder, and knowledge encoder are concatenated together or input into a common fusion layer to construct a multimodal encoder.

[0084] S1032: Use a collaborative training strategy to update the multimodal encoder by sharing parameters or jointly optimizing the objective function.

[0085] In this embodiment, in certain layers or components of the multimodal encoder, shared parameters are designed, which can play a key role in processing data of different modalities because they can learn common features across modalities. Exemplarily, the image encoder and the text encoder can share the same convolution kernel or weight matrix in the first few layers, and when the encoder processes data of different modalities, the same vocabulary or concept space can be mapped.

[0086] In this embodiment, in order to collaboratively train the multimodal encoder, one or more objective functions are defined, and the objective function simultaneously considers the encoding effects of images, texts, and knowledge. Exemplarily, a multi-task learning framework is preset, in which each task corresponds to a specific modality, and a shared loss function is used to optimize the entire multimodal encoder. During the training process, the image, text, and knowledge data are respectively input into the corresponding encoders. Then, the outputs of these encoders are passed to the fusion layer (for example, a fully connected layer or an attention mechanism) to generate a multimodal representation. Finally, this multimodal representation is used to complete downstream tasks. The loss function is calculated based on the output of the downstream task and the true label. Through the back propagation algorithm, the gradient of the loss function with respect to the model parameters is calculated to update the parameters of the multimodal encoder.

[0087] It is worth noting that inner ear diseases may manifest as a variety of different symptoms, which may involve different modalities, such as auditory test data, medical images (such as MRI or CT scans), patient-reported medical history and symptoms, etc. Using a collaborative training strategy to optimize the multimodal encoder can enable it to process information from different modalities and provide doctors with a more comprehensive diagnosis basis. By combining information from different modalities, the collaboratively trained multimodal encoder can capture important features or patterns that may be ignored by a single modality, which helps to reduce misdiagnosis and missed diagnosis and improve the accuracy of diagnosis. In addition, in some cases, the data of some modalities may be incomplete or contaminated by noise. The collaboratively trained multimodal encoder can use the data of other modalities to compensate for this missing or contaminated information, thereby improving the robustness and fault tolerance of the model. Finally, the collaboratively trained multimodal encoder can automatically process and analyze data from different modalities, thereby accelerating the diagnosis process. This can not only reduce the workload of doctors, but also improve the efficiency and accuracy of diagnosis.

[0088] S1033: Combine the different modal entity features generated by the multimodal encoder, and calculate the probability values ​​between the combined different modal entity features based on a preset scoring function to generate a condensed summary of the semantics, attributes, and biomedical knowledge of each modal entity.

[0089] In this embodiment, the multimodal encoder receives input data from different modalities (such as text, images, biological signals, etc.) and generates entity features of the corresponding modalities to capture key information related to a specific entity in the original data. The entity features from different modalities are combined, and the combination method includes but is not limited to: feature splicing, weighted fusion, attention mechanism, etc. The combined features can comprehensively reflect the overall information of the multimodal data, which covers the semantics, attributes and condensed summary of biomedical knowledge of inner ear diseases. In order to evaluate the correlation or importance between the combined features, one or more scoring functions are preset to score the combined features and convert them into probability values ​​to represent the degree of association, importance or confidence between entity features of different modalities. The features are sorted according to the probability value, and the most important features are selected for display. The selected features are converted into understandable semantic representations, such as generating descriptive text using natural language processing technology. Attribute information related to the entity is extracted from the selected features and displayed in a structured manner. The extracted attribute information is compared and integrated with the existing biomedical knowledge base to provide a more comprehensive and accurate explanation.

[0090] S1034: Based on the condensed summary of the semantics, attributes, and biomedical knowledge of each modal entity, association learning is used to perform endogenous correlation learning among multimodal entities to generate a multimodal pre-trained model.

[0091] In this embodiment, the association learning of the Transformer model mainly relies on its attention mechanism, which allows the model to focus on information at different positions when processing the input sequence and perform accurate association learning. The purpose of association learning is to discover endogenous correlations between entities of different modalities. Based on the multimodal knowledge graph, and according to the condensed summary of the semantics, attributes and biomedical knowledge of each modal entity, various medical applications such as semantic search, knowledge question and answer, disease diagnosis and treatment plan recommendation are constructed in combination with graph embedding technology to generate a multimodal pre-training model. Exemplarily, using association learning, historical medical records similar to the current patient's characteristics are searched in the multimodal knowledge graph to help doctors understand the treatment methods and treatment effects of similar conditions, and provide references for formulating treatment plans. According to the patient's diagnosis and the severity of the disease, recommendations are made based on the clinical pathways and guidelines in the multimodal knowledge graph to provide doctors with standardized treatment processes and operating guidelines. According to the hierarchical structure of the multimodal knowledge graph, the patient's multimodal data (such as vital signs, laboratory test results, etc.) are monitored in real time for abnormality detection. When it is detected that certain indicators are out of the normal range or there is a potential critical situation, an early warning can be automatically issued to remind doctors to pay attention and deal with it in time. Similarly, when a doctor prescribes a medicine, the rationality of the prescription is checked based on the multimodal knowledge graph. If abnormal conditions such as drug interactions and contraindications are found in the prescription, the system will issue an early warning and give suggestions in time. Alternatively, when a doctor enters a disease name, reasoning is performed based on the medical ontology to recommend other possible diseases, complications, related drugs, etc. related to the disease, thereby enriching the recommendation results and helping doctors to have a more comprehensive understanding of the patient's condition and treatment needs.

[0092] S104: Using the prompt learning analogy reasoning algorithm, perform knowledge enhancement operation, task transfer operation and self-supervised model training operation on the multimodal pre-trained model to generate a multimodal consultation model.

[0093] In this embodiment, the prompt learning analogy reasoning algorithm is an implicit structure mapping analogy reasoning algorithm. The implicit structure mapping analogy reasoning algorithm is a technology used in artificial intelligence and computational linguistics. It is used to process and analyze analogy relationships, especially when these relationships are not obvious or "implicit" on the surface. The implicit structure mapping analogy reasoning algorithm attempts to discover deep and imperceptible similarities from complex semantic structures. Through analogy reasoning, new knowledge points, new relationships or new patterns can be discovered, thereby enhancing existing knowledge. The knowledge enhancement operation not only increases the breadth and depth of knowledge, but also improves the quality and application value of knowledge. Exemplarily, the implicit structure mapping analogy reasoning algorithm can be used to identify implicit relationships or structural patterns between nodes in a multimodal knowledge graph. For example, the similarities between the two diseases "deafness" and "tinnitus" in symptoms, treatment methods, etc. can be found, so that they are classified into the same category or an analogy relationship is constructed. Based on the extracted entities and relationships, as well as the analogy relationships obtained through implicit structure mapping, a question-answer pair pattern is constructed to represent the expertise of each node and to enhance the existing knowledge. These question-answer pairs can cover aspects such as the definition, symptoms, diagnostic methods, and treatment methods of the disease. For example, for the node "otitis media", we can construct the following question-answer pairs: Question: What are the main symptoms of otitis media? Answer: The main symptoms of otitis media include hearing loss, ear pain, tinnitus, etc. Question: What are the treatments for otitis media? Answer: Treatments for otitis media include drug therapy (such as antibiotics, painkillers, etc.), surgical treatment (such as tympanotomy, etc.), and conservative treatment (such as rest, keeping the ears clean, etc.).

[0094] In this embodiment, during the model training process, a mask and restore task mode is used to mask and automatically restore the multimodal information and questions and answers in the task. The formula of the implicit structure mapping analogy reasoning algorithm includes:

[0095]

[0096] in, Indicates the conversion of the input analogy examples and questions into an objective function or template; ‖ indicates the connection operation in the template input, that is, connecting different parts in series to form a complete input; e h Represents the first entity object; e t Represents the second entity object; e q Represents the third entity object; I h Represents the first entity object e h The corresponding image information; T t Represents the second entity object e t The corresponding text information; q Represents the third entity object e qThe corresponding question and answer information; T e T stands for implicit abduction; a represents implicit induction; [R] represents I h 、T t [ClS] indicates the start of a category sequence; [SEP] indicates a separator marker used to distinguish different parts in a sequence; and [MASK] indicates an occluded region.

[0097] In this embodiment, learning is expressed based on the implicit structure mapping analogy reasoning algorithm and unified in an end-to-end learning manner. Subsequently, targeted tuning can be performed on downstream diagnostic tasks. Analogy reasoning is restated as predicting [MASK] on multimodal analogy entity embeddings to enhance domain knowledge. At the same time, the model can use the learned or labeled contextual information in the multimodal knowledge graph to predict missing content, thereby solving problems such as modal missing and modal noise in existing knowledge graphs, and enabling the system to have powerful data missing completion and data fault tolerance capabilities.

[0098] In this embodiment, a self-supervised model training task mode is used in the training of the multimodal consultation model, and part of the information and Answer of the text, image or other modalities are blocked, and the cross entropy loss function is used to guide the model to learn how to automatically restore this information, so as to train the model's ability to understand the context. The cross entropy loss function is used to calculate the difference between the answer predicted by the model and the true answer. The cross entropy loss function can measure the degree of mismatch between the probability distribution predicted by the model and the probability distribution of the true answer. According to the calculated loss value, the parameters of the model are updated by the back propagation algorithm to minimize the loss value and optimize the performance of the model. This process will be continuously iterated until the performance of the model on the validation set reaches the preset standard, thereby obtaining a multimodal consultation model. In this way, through the steps of recognition enhancement operation, task migration operation and self-supervised model training operation, the multimodal consultation model can predict and infer unknown information based on existing knowledge and experience, thereby providing more accurate and useful diagnostic suggestions.

[0099] S105: Based on the inner ear disease cycle database, the multimodal consultation model is trained for multiple rounds of iterations using a transfer learning algorithm and a Prompt tuning algorithm to generate an inner ear disease diagnosis model.

[0100] In this embodiment, the inner ear disease circulation database is used to store, analyze and share data related to inner ear diseases and their circulatory systems (such as blood supply). During the training process, the inner ear disease circulation database is continuously updated for the continuous expansion of the data set, and the features in the inner ear disease circulation database are optimized, and the multimodal knowledge graph, multimodal consultation model and inner ear disease diagnosis model are continuously iterated and trained using a self-feedback update algorithm. The self-feedback update algorithm is an algorithm that can automatically adjust and optimize model parameters or structures based on the output or prediction results of the model itself, and can automatically detect the differences or conflicts between new data and multimodal knowledge graphs, and evaluate the importance of new data based on factors such as the source, accuracy, and consistency of the data, so as to integrate new data or updated data into the multimodal knowledge graph, and update related entities, relationships and attributes at the same time to achieve automated data integration and interdisciplinary data integration, and build a real-time updated multimodal knowledge graph. Data related to the multimodal knowledge graph is extracted from the inner ear disease circulation database, including text, images, audio, etc. The extracted data is annotated, including the type, symptoms, and treatment methods of the disease. Use the labeled data to iteratively train the multimodal interview model and the inner ear disease diagnosis model. Evaluate the performance of the model through the validation set or test set, including indicators such as accuracy, recall, F1 score, and statistical significance. According to the evaluation results, adjust the parameters, structure, or algorithm of the multimodal interview model and the inner ear disease diagnosis model to improve the performance of the model. Repeat the above iterative training process, continuously collect feedback, analyze feedback, adjust the model, and train again until the performance of the model reaches the preset standard or can no longer be significantly improved through iterative training.

[0101] It should be understood that accuracy refers to the proportion of samples that are truly positive among the samples that are judged as positive by the classifier. It measures the proportion of samples that are actually positive among the samples predicted to be positive. Recall refers to the proportion of positive samples that are correctly judged by the classifier to the total positive samples. It measures the proportion of samples that are predicted to be positive among the samples that are actually positive. The F1 score refers to the harmonic mean of accuracy and recall, which is used to comprehensively evaluate the performance of the classifier and can be used to find a balance between accuracy and recall. Statistical significance is mainly used to compare the performance of different models or models under different settings.

[0102] In this embodiment, the transfer learning algorithm uses the knowledge learned on the source task to enhance the performance on the target task. Combined with the implicit structure mapping analogy reasoning algorithm, transfer learning can help the model transfer the knowledge learned on one data structure to another data structure, achieving more comprehensive and professional clinical diagnosis and treatment reasoning.

[0103] In this embodiment, the prompt tuning algorithm refers to fine-tuning the prompt text (Prompt) in the generative model to improve the quality and accuracy of the model generation results. This technology enables the model to understand the user's intention more accurately and generate output that better meets user needs by optimizing the question input or prompt text. In pre-trained language models, prompt tuning technology is widely used. The pre-trained model is first trained in a large corpus to learn a general language representation, and then the pre-trained model is fine-tuned for different downstream tasks. By optimizing the prompt, the model can perform better on specific tasks and improve the accuracy and efficiency of the model.

[0104] It is worth noting that based on the self-feedback update algorithm, automatic data integration and cross-disciplinary data integration are realized, a real-time updated multimodal knowledge graph is constructed, and the method of automatic knowledge extraction and structured knowledge features is used to efficiently and accurately expand the relevant data. Through algorithms such as prompt learning, transfer learning, and prompt tuning, the rapidity and robustness of multimodal knowledge graph and model optimization iteration can be guaranteed. Finally, based on the construction of a circular database of auditory-related refractory inner ear diseases and self-feedback optimization technology, the intelligent diagnosis and treatment system can be continuously verified, optimized, and iteratively upgraded.

[0105] Further, the inner ear disease diagnosis model construction method of the present invention develops a multimodal knowledge graph system in the professional field of hearing-related inner ear diseases, integrates existing deafness and tinnitus disease-related guidelines, 10,000-level patient disease data, literature data, diagnosis and treatment plans and medical images and other multimodal data, and is used to connect, organize and mine various disease-related factors, including text, pathological information, biological information, clinical indications, historical disease archives and diagnosis and treatment images, etc., and builds multimodal entities based on deep learning models, and associates entity objects of various modalities, unifies heterogeneous multimodal underlying representations into the same high-level semantic feature space, and mines the intrinsic associations of disease-related factors in massive multimodal data sources to construct a multimodal knowledge graph. And combine the multimodal knowledge graph with transformer-based multimodal pre-training, use transfer learning, prompt learning, multi-head attention and relationship mapping and other technologies to realize knowledge-enhanced model pre-training and task migration, add more professional knowledge guidance and rich context perception capabilities to the model, and realize knowledge-enhanced model pre-training and task migration to generate a multimodal consultation model. The multimodal consultation model is iteratively trained for multiple rounds to generate an inner ear disease diagnosis model, accelerate the learning and application of the inner ear disease diagnosis model in the medical field, realize a new type of intelligent decision-making support and intervention system, and ultimately achieve early diagnosis and treatment of the disease.

[0106] Furthermore, it is difficult for traditional disease diagnosis reasoning models to perform knowledge enhancement and iterative optimization for professional fields. The present invention realizes the analysis of inner ear disease-related factors based on multimodal knowledge representation, association learning and analogical reasoning through the construction of a large database of professional fields, a multimodal knowledge graph, a multimodal pre-training model and a knowledge-enhanced consultation model, providing a deeper level of modal knowledge complementarity and disease reasoning capabilities for clinical diagnosis, and improving the ability to identify subgroups of patients with refractory inner ear diseases. According to the characteristics of the patients, efficient subdivision and diversion of patients with auditory-related refractory inner ear diseases and risk factor identification and reasoning are performed to provide the best auxiliary diagnosis and treatment plan. The present invention uses the Co-attentionTransformer interactive feature fusion module to learn the initial features of multimodal entities, and uses the structural aggregation module to perform endogenous correlation learning and disease diagnosis reasoning on multimodal entities. At the same time, for subgroups of patients with refractory inner ear diseases, based on the professional knowledge and guidelines of refractory inner ear diseases, prompt learning analogical reasoning and other technologies are used to achieve knowledge-enhanced model pre-training and task transfer. In addition, based on the self-feedback update algorithm, automatic data integration, non-standard data alignment and cross-disciplinary data integration are realized, and a real-time updated multimodal knowledge graph is constructed. The method of automatic knowledge extraction and structured knowledge features can efficiently and accurately expand the relevant data. Through prompt learning, transfer learning and prompt tuning and other technologies, the rapidity and robustness of multimodal knowledge graph and model optimization iteration can be guaranteed. Finally, based on the construction of a circular database of hearing-related refractory inner ear diseases and self-feedback optimization technology, the intelligent diagnosis and treatment system can be continuously verified, optimized and iteratively upgraded.

[0107] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first entity object and the second entity object are only used to distinguish different entity objects, and do not limit their order. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0108] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0109] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0110] Figure 4 is a schematic block diagram of the inner ear disease diagnosis model construction system provided in the embodiment of the present application. Figure 4 As shown, the inner ear disease diagnosis model construction system 400 includes:

[0111] The data acquisition module 401 is used to acquire multimodal data related to inner ear diseases from different databases.

[0112] The multimodal knowledge graph construction module 402 is used to perform entity information extraction operations, high-level semantic unification operations, and intrinsic association mining operations on each modal data based on a deep learning model to construct a multimodal knowledge graph.

[0113] The multimodal pre-trained model generation module 403 is used to perform endogenous correlation learning between multimodal entities based on the multimodal knowledge graph using the deep learning model to generate a multimodal pre-trained model.

[0114] A multimodal consultation model generation module 404 is used to perform a knowledge enhancement operation, a task transfer operation and a self-supervised model training operation on the multimodal pre-trained model using a prompt learning analogy reasoning algorithm to generate a multimodal consultation model;

[0115] The inner ear disease diagnosis model generation module 405 is used to perform multiple rounds of iterative training on the multimodal consultation model based on the inner ear disease cycle database using a transfer learning algorithm and a Prompt tuning algorithm to generate an inner ear disease diagnosis model.

[0116] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0117] It should also be understood that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0118] Figure 5 Schematic diagram of the inner ear disease diagnosis model construction system provided in the embodiment of the present application. Figure 5 As shown, multimodal disease data are collected, including but not limited to: patient clinical cases, imaging examination data, three-dimensional nystagmus data, hearing examination data, consensus on deafness guidelines, consensus on tinnitus guidelines, consensus on vertigo guidelines, deafness literature data, tinnitus literature data, vertigo literature data, relevant subject guidelines, relevant subject literature, patient medical history data, expert judgment records, multimodal omics information, etc. Based on the above multimodal data, a tinnitus disease database, a deafness disease database, a general database, and a refractory inner ear disease circulation database are constructed respectively. After the multimodal disease data in the above databases are structured, entity association operations, unified high-level semantic operations, and intrinsic association mining operations are performed on each modality data to construct a multimodal knowledge graph. The multimodal knowledge graph is embedded in the deep learning model, and endogenous correlation learning, causation, matching, and induction are performed between multimodal entities to generate a multimodal pre-training model. Using the prompt learning analogy reasoning algorithm, the multimodal pre-trained model is subjected to knowledge enhancement operations, task transfer operations, and self-supervised model training operations to generate a multimodal consultation model. Based on the inner ear disease cycle database, the multimodal consultation model is subjected to multiple rounds of iterative training using the transfer learning algorithm and the prompt tuning algorithm to generate an inner ear disease diagnosis model.

[0119] Figure 6 is a schematic block diagram of an electronic terminal provided in an embodiment of the present application. Figure 6 As shown, the electronic terminal includes: at least one processor 601, a memory 602, at least one network interface 603 and a user interface 605. The various components in the device are coupled together through a bus system 604. It can be understood that the bus system 604 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 604 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 6 In the specification, various buses are labeled as bus systems.

[0120] The user interface 605 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0121] It is understood that the memory 602 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0122] The memory 602 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 600. Examples of these data include: any executable program for operating on the electronic terminal 600, such as an operating system 6021 and an application 6022; the operating system 6021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 6022 may include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The inner ear disease diagnosis model construction method provided in the embodiment of the present invention may be included in the application 6022.

[0123] The method disclosed in the above embodiment of the present invention can be applied to the processor 601, or implemented by the processor 601. The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 601 or the instruction in the form of software. The above processor 601 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 601 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 601 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0124] In an exemplary embodiment, the electronic terminal 600 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0125] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, the computer executes Figures 1 to 3 A method according to any one of the embodiments shown.

[0126] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code, and when the program code is run on a computer, the computer executes Figures 1 to 3 A method according to any one of the embodiments shown.

[0127] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0128] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0130] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0133] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When loading and executing computer program instructions (programs) on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs), etc.).

[0134] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0135] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0136] In summary, the present application provides a method, system, medium, product and terminal for constructing an inner ear disease diagnosis model, constructs a multimodal entity based on a deep learning model, associates entity objects of various modalities, unifies heterogeneous multimodal underlying representations into the same high-level semantic feature space, and mines the intrinsic associations of disease-related factors in massive multimodal data sources to construct a multimodal knowledge graph. And the multimodal knowledge graph is combined with transformer-based multimodal pre-training, and technologies such as transfer learning, prompt learning, multi-head self-attention and relationship mapping alignment are used to realize knowledge-enhanced model pre-training and task migration, add more professional knowledge guidance and rich context perception capabilities to the model, realize knowledge-enhanced model pre-training and task migration, and generate a multimodal consultation model. The multimodal consultation model is trained for multiple rounds of iterations to generate an inner ear disease diagnosis model, accelerate the learning and application of the inner ear disease diagnosis model in the medical field, realize a new intelligent decision-making and intervention system, and finally realize early diagnosis and treatment of the disease. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0137] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for constructing an inner ear disease diagnosis model, characterized in that: include: Obtain multimodal data related to inner ear diseases from different databases; Based on the deep learning model, entity information extraction, high-level semantic unification, and intrinsic association mining are performed on each modal data to build a multimodal knowledge graph. Based on the multimodal knowledge graph, the deep learning model is used to perform endogenous correlation learning between multimodal entities to generate a multimodal pre-training model; comprising the following steps: Integrate the image encoder, text encoder, and knowledge encoder in the deep learning model to form a multimodal encoder; Use collaborative training strategies to update multimodal encoders by sharing parameters or jointly optimizing objective functions; Combining the different modal entity features generated by the multimodal encoder, and calculating the probability values ​​between the combined different modal entity features based on a preset scoring function to generate a condensed summary of the semantics, attributes, and biomedical knowledge of each modal entity; Based on the condensed summary of semantics, attributes and biomedical knowledge of each modality entity, association learning is used to learn endogenous correlations between multimodal entities to generate a multimodal pre-trained model. Using a prompt learning analogy reasoning algorithm, the multimodal pre-trained model is subjected to a knowledge enhancement operation, a task transfer operation and a self-supervised model training operation to generate a multimodal consultation model; the prompt learning analogy reasoning algorithm is an implicit structure mapping analogy reasoning algorithm; Based on the inner ear disease circulation database, the multimodal consultation model is iteratively trained for multiple rounds using a transfer learning algorithm and a Prompt tuning algorithm to generate an inner ear disease diagnosis model.

2. The method for constructing an inner ear disease diagnosis model according to claim 1, characterized in that: Based on the deep learning model, entity information extraction, high-level semantic unification, and intrinsic association mining are performed on each modal data to construct a multimodal knowledge graph in the following ways: Based on the deep learning model, each modal data is encoded to extract the initial features of each modal entity data; The initial features of each modality entity data are embedded into a unified semantic space, and based on a heterogeneous graph attention network model, entity association operations are performed on the initial features of each modality entity data to construct a heterogeneous knowledge graph; the nodes in the heterogeneous knowledge graph represent the initial features of entity data of different modalities, the edges in the heterogeneous knowledge graph represent the similarities between the initial features of entity data of different modalities, and the meta-paths in the heterogeneous knowledge graph represent the association relationship between the initial features of entity data of different modalities; The triple relationships and entity association modal features related to inner ear diseases are extracted from the heterogeneous knowledge graph, and multimodal entity feature fusion operations and intrinsic association mining operations are performed through the structure aggregation module in the deep learning model to construct a multimodal knowledge graph.

3. The method for constructing an inner ear disease diagnosis model according to claim 1, characterized in that: The formula of the implicit structure mapping analogy reasoning algorithm includes: in, Indicates the conversion of the input analogy examples and questions into an objective function or template; ‖‖ indicates the connection operation in the template input, that is, connecting different parts in series to form a complete input; e h Represents the first entity object; e t Represents the second entity object; e q Represents the third entity object; I h Represents the first entity object e h The corresponding image information; T t Represents the second entity object e t The corresponding text information; q Represents the third entity object e q The corresponding question and answer information; T e T stands for implicit abduction; a represents implicit induction; [R] represents I h 、T t [CLS] indicates the start of a category sequence; [SEP] indicates a separator marker used to distinguish different parts in a sequence; and [MASK] indicates an occluded area.

4. The method for constructing an inner ear disease diagnosis model according to claim 1, characterized in that: The deep learning model includes a Transformer model.

5. A system for constructing an inner ear disease diagnosis model, characterized in that: include: A data acquisition module, used to acquire multimodal data related to inner ear diseases from different databases; A multimodal knowledge graph construction module is used to perform entity information extraction operations, high-level semantic unification operations, and intrinsic association mining operations on each modality data based on a deep learning model to construct a multimodal knowledge graph; A multimodal pre-training model generation module is used to perform endogenous correlation learning between multimodal entities based on the multimodal knowledge graph using the deep learning model to generate a multimodal pre-training model; comprising the following steps: Integrate the image encoder, text encoder, and knowledge encoder in the deep learning model to form a multimodal encoder; Use collaborative training strategies to update multimodal encoders by sharing parameters or jointly optimizing objective functions; Combining the different modal entity features generated by the multimodal encoder, and calculating the probability values ​​between the combined different modal entity features based on a preset scoring function to generate a condensed summary of the semantics, attributes, and biomedical knowledge of each modal entity; Based on the condensed summary of semantics, attributes and biomedical knowledge of each modality entity, association learning is used to learn endogenous correlations between multimodal entities to generate a multimodal pre-trained model. A multimodal consultation model generation module is used to use a prompt learning analogy reasoning algorithm to perform knowledge enhancement operations, task transfer operations and self-supervised model training operations on the multimodal pre-trained model to generate a multimodal consultation model; the prompt learning analogy reasoning algorithm is an implicit structure mapping analogy reasoning algorithm; The inner ear disease diagnosis model generation module is used to perform multiple rounds of iterative training on the multimodal consultation model based on the inner ear disease circulation database using a transfer learning algorithm and a Prompt tuning algorithm to generate an inner ear disease diagnosis model.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing an inner ear disease diagnosis model according to any one of claims 1 to 4 is implemented.

7. A computer program product, characterized in that The computer program product includes computer program codes, and when the computer program codes are executed on a computer, the computer is enabled to implement the method for constructing an inner ear disease diagnosis model according to any one of claims 1 to 4.

8. An electronic terminal comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the inner ear disease diagnosis model construction method according to any one of claims 1 to 4.

Images