Medical ultrasound knowledge base construction method, related equipment and program products

Abstract

The present application discloses a method for constructing a knowledge base in the field of medical ultrasound, related equipment, and program products. The present application obtains multimodal medical ultrasound data, performs entity extraction on the medical text data therein to obtain target entities, and extracts the relationships between target entity pairs in the medical text data. Furthermore, a multimodal unsupervised pre-training model is used to extract features from the medical text data and its associated other modal data to obtain aligned text features and other modal features. The text features and other modal features are fused to obtain fused features, and an association relationship is established between the fused features and the target entities contained in the medical text data. The target entities, the relationships between target entity pairs, and the fused features associated with the target entities are then added to the knowledge base in the field of medical ultrasound, thereby improving the knowledge coverage of the knowledge base and providing a more comprehensive knowledge expression. In addition, the method of the present application does not rely on manually formulated rules, thereby improving the efficiency of knowledge base construction.

Description

[0001] This application claims priority to a domestic application filed with the Patent Office of China on May 7, 2025, with application number 202510582273.X and invention name “Method for constructing a knowledge base in the field of medical ultrasound, related equipment and program products”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the technical field of knowledge base construction, and more specifically, to a method for constructing a knowledge base in the field of medical ultrasound, related equipment, and program products. Background Art

[0003] In healthcare, contrast-enhanced ultrasound (CEUS) is a diagnostic imaging technique used for the liver and other organs. In the field of medical ultrasound, there is a vast library of medical guidelines, medical records, medical images, physiological data, and test results, all describing the different processes or rationales for diagnosis and treatment. Building a specialized knowledge base for medical ultrasound can not only support clinical decision-making but also serve as a valuable resource for education and research.

[0004] Traditional knowledge base construction methods generally use rule-based approaches, where experts define rules and logical relationships to convert domain textual information into structured knowledge to build the knowledge base. However, this approach relies on manually defined rules and logic and can only organize information in the textual modality, failing to integrate knowledge from other modalities. Consequently, the knowledge coverage of the constructed domain knowledge base is limited. Summary of the Invention

[0005] In view of the above problems, this application is proposed to provide a method for constructing a knowledge base in the field of medical ultrasound, related equipment, and program products to improve the efficiency of knowledge base construction and knowledge coverage, and enhance the quality of the knowledge base. The specific solution is as follows:

[0006] In a first aspect, a method for constructing a knowledge base in the field of medical ultrasound is provided, comprising:

[0007] Acquiring multimodal medical ultrasound data, where the multimodal medical ultrasound data includes medical text data and other associated modality data;

[0008] Performing entity extraction on the medical text data to obtain target entities, and extracting relationships between target entity pairs in the medical text data;

[0009] Calling a multimodal unsupervised pre-trained model to perform feature extraction on the medical text data and its associated other modal data to obtain text features and other modal features, wherein the multimodal unsupervised pre-trained model is configured to extract features of the multimodal data and perform modality alignment;

[0010] fusing the text features and other modality features to obtain fused features, and establishing an association relationship between the fused features and the target entity contained in the medical text data;

[0011] The target entities, the relationships between the target entity pairs, and the fusion features associated with the target entities are added to a medical ultrasound domain knowledge base.

[0012] In a second aspect, a device for constructing a knowledge base in the field of medical ultrasound is provided, comprising:

[0013] A data acquisition unit, configured to acquire multimodal medical ultrasound data, wherein the multimodal medical ultrasound data includes medical text data and other associated modality data;

[0014] An entity and relationship extraction unit, configured to perform entity extraction on the medical text data to obtain target entities, and extract relationships between pairs of target entities in the medical text data;

[0015] a multimodal feature extraction unit, configured to call a multimodal unsupervised pre-trained model to perform feature extraction on the medical text data and its associated other modal data to obtain text features and other modal features, wherein the multimodal unsupervised pre-trained model is configured to extract features of the multimodal data and perform modality alignment;

[0016] a multimodal feature fusion unit, configured to fuse the text feature and other modal features to obtain a fused feature, and establish an association relationship between the fused feature and the target entity contained in the medical text data;

[0017] The knowledge base updating unit is configured to add the target entity, the relationship between the target entity pairs, and the fusion features associated with the target entities to a medical ultrasound field knowledge base.

[0018] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0019] The memory is used to store programs;

[0020] The processor is used to execute the program to implement the various steps of the method for constructing a medical ultrasound field knowledge base provided in the first aspect of the embodiment of the present application.

[0021] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the method for constructing a medical ultrasound field knowledge base provided in the first aspect of the embodiment of the present application are implemented.

[0022] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the various steps of the method for constructing a medical ultrasound field knowledge base provided in the first aspect of the embodiment of the present application.

[0023] By means of the above technical solution, the present application obtains multimodal medical ultrasound data, including medical text data and data from other modalities associated with the medical text data, such as image data (e.g., ultrasound examination reports), audio and video data, and the like. Based on this, entity extraction is performed on the medical text data to obtain target entities, and relationships between pairs of target entities in the medical text data are extracted, thereby obtaining information about the extracted target entities and relationships between pairs of target entities. Furthermore, the present application also includes a multimodal unsupervised pre-training model that is configured to extract features from the multimodal data and perform modal alignment. This model is used to extract features from the medical text data and its associated other modal data, obtaining aligned text features and other modal features. The text features and other modal features are then fused to obtain fused features, which are then associated with the target entities contained in the medical text data. The target entities, relationships between pairs of target entities, and fused features associated with the target entities can then be added to a knowledge base in the medical ultrasound field. The knowledge base contains not only text modal information but also other modal information, thereby improving the knowledge coverage of the knowledge base and providing a more comprehensive knowledge representation. Furthermore, the method of the present application does not rely on manual rule-making, thereby improving the efficiency of knowledge base construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0025] Figure 1 A schematic diagram of an implementation system architecture of the method for constructing a medical ultrasound knowledge base provided in an embodiment of the present application;

[0026] Figure 2 A schematic flow chart of a method for constructing a medical ultrasound knowledge base provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the training process of a multi-view semi-supervised training model based on an auxiliary objective function is shown;

[0028] Figure 4 This example illustrates a schematic diagram of the relationship extraction model structure based on remote supervision;

[0029] Figure 5 This paper demonstrates a process of aligning text and image modality features;

[0030] Figure 6 An example of a process diagram for entity linking is shown;

[0031] Figure 7 A schematic diagram illustrating the processing of an entity link prediction model;

[0032] Figure 8 A schematic diagram of the structure of a device for constructing a medical ultrasound knowledge base provided in an embodiment of the present application;

[0033] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0036] The data involved in the technical solution of this application (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0037] This application provides a method for constructing a knowledge base in the field of medical ultrasound, which can be applied to Figure 1 The system architecture shown in FIG. 1 may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 (This section includes a server as an example).

[0038] The terminal 100 or the server 200 can be used alone to execute the method for constructing a medical ultrasound knowledge base provided in the embodiment of the present application. In addition, the terminal 100 and the server 200 can also be used together to execute the method for constructing a medical ultrasound knowledge base provided in the embodiment of the present application.

[0039] The terminal 100 in the embodiment of the present application may be a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0040] The medical ultrasound knowledge base construction method of this application can be applied to knowledge base construction scenarios in various sub-fields within the medical ultrasound field, including but not limited to: the construction of a liver ultrasound professional knowledge base, etc. The following text uses the liver ultrasound professional knowledge base construction process as an example to illustrate.

[0041] The embodiment of the present application provides a method for constructing a knowledge base in the field of medical ultrasound. The method is applied to a computer device as an example. The computer device can be Figure 1 The terminal 100 or the system consisting of the terminal 100 and the server 200. Figure 2 The method for constructing a knowledge base in the field of medical ultrasound specifically includes the following steps:

[0042] Step S100: Acquire multimodal medical ultrasound data, where the multimodal medical ultrasound data includes medical text data and other associated modality data.

[0043] Specifically, multimodal medical ultrasound data includes text modalities and data from other modalities, such as image modalities, audio and video modalities, etc. Multimodal medical ultrasound data can be multi-source heterogeneous data, including but not limited to the following five types of data sources:

[0044] Anatomy and physiology knowledge: This mainly comes from medical textbooks. For example, knowledge of liver anatomy and physiology mainly covers the normal anatomical structure of the liver and the physiological mechanisms of its normal function.

[0045] Ultrasound examination techniques and equipment parameters: This includes the basic principles of ultrasound imaging, Doppler ultrasound, and parameter settings and optimization methods for various imaging modes. These parameters are primarily obtained from the equipment's technical manual.

[0046] Disease diagnosis information: This includes common diseases and their ultrasound manifestations, typical ultrasound features of various cases, and graded diagnostic criteria. This information is primarily based on large-scale clinical case records and diagnostic criteria guidelines.

[0047] Image data and annotation information: This includes a large amount of annotated ultrasound image data, as well as image datasets of various cases used for training AI models or reference analysis. These can be obtained from imaging examination and inspection reports of medical institutions.

[0048] Clinical case data: mainly ultrasound examination reports and diagnostic results, as well as case follow-up data required to establish disease progression or treatment effect prediction models.

[0049] Based on the multimodal medical ultrasound data in the example above, a knowledge base for medical ultrasound is constructed. This knowledge base includes fused feature representations of multimodal data, including entities, relationships between entities, and entity associations. Entity types include, but are not limited to, patient information, medical terminology, measurement indicators, imaging data, and text descriptions. Relationship patterns encompass causal relationships, affiliation, spatiotemporal relationships, and operational relationships. Furthermore, necessary attributes can be defined for each entity. For example, for a disease entity, related attributes include incidence, risk factors, and treatment methods.

[0050] By defining the knowledge structure, a clear goal and framework are provided for subsequent information extraction, ensuring that the extracted information is organized, usable, and in line with the professional field.

[0051] Step S110: Perform entity extraction on the medical text data to obtain target entities, and extract relationships between target entity pairs in the medical text data.

[0052] Specifically, the entity extraction model is used to extract the textual data (i.e., medical text data) from the multimodal medical ultrasound data acquired in the previous step, obtaining target entities contained in the medical text data. Furthermore, the relationships between pairs of target entities in the medical text data are extracted.

[0053] Step S120: Call a multimodal unsupervised pre-training model to perform feature extraction on the medical text data and its associated other modal data to obtain text features and other modal features. The multimodal unsupervised pre-training model is configured to extract features of the multimodal data and perform modality alignment.

[0054] In order to make full use of the complementarity between different modalities and achieve more comprehensive and accurate knowledge expression, this application also pre-configures a multimodal unsupervised pre-training model, which can extract features from the input multimodal data and achieve cross-modal feature alignment, that is, to associate data of different modalities (such as text, images, structured data, etc.) into the same semantic space.

[0055] In this step, by calling the multimodal unsupervised pre-training model, feature extraction is performed on the medical text data and its associated other modal data to obtain text features corresponding to the medical text data and other modal features corresponding to the other modal data.

[0056] Taking an ultrasound examination report as an example, it contains both the ultrasound image and the corresponding examination report text. Using a multimodal unsupervised pre-trained model, we can extract the image features of the ultrasound image and the text features of the examination report. Furthermore, because the multimodal unsupervised pre-trained model has the ability to align features across modalities, it ensures that the extracted image features and text features are aligned, facilitating feature fusion in subsequent steps.

[0057] Step S130: Fusing the text features and other modality features to obtain fused features, and establishing an association relationship between the fused features and the target entity contained in the medical text data.

[0058] Specifically, the text features extracted in the previous step and features from other modalities can be fused to produce fused features. Through this feature extraction, alignment, and fusion, the complementarity between different modalities can be fully utilized to achieve more comprehensive and accurate knowledge representation. For example, semantic associations can be established between lesion features in images and text descriptions and numerical parameters, improving the accuracy and interpretability of diagnostic results.

[0059] The fused features obtained in this step correspond to the medical text data and its associated other modal data. Since the knowledge base stores knowledge information in the form of entity objects, in order to facilitate the construction of the knowledge base, in this step, the fused features can be associated with the target entities contained in the medical text data.

[0060] Step S140: adding the target entity, the relationship between the target entity pairs, and the fusion features associated with the target entities to a medical ultrasound domain knowledge base.

[0061] Through the processing in the aforementioned steps, the target entity, the relationship between the target entities, and the fusion features associated with the target entity (the fusion features serve as the feature representation of the medical text data where the target entity is located and its associated other modal data) have been obtained from the multimodal medical ultrasound data. The above types of information can be added to the medical ultrasound field knowledge base.

[0062] The method provided in an embodiment of the present application obtains multimodal medical ultrasound data, including medical text data and data of other modalities associated with the medical text data, such as image data (such as ultrasound examination reports) and audio and video data associated with the medical text. Based on this, entity extraction is performed on the medical text data to obtain target entities, and the relationships between target entity pairs in the medical text data are extracted, thereby obtaining the extracted target entities and the relationship information between target entity pairs. Furthermore, the present application also provides a multimodal unsupervised pre-training model, which is configured to extract features from the multimodal data and perform modal alignment. With the help of this model, feature extraction is performed on the medical text data and its associated other modal data to obtain aligned text features and other modal features. The text features and other modal features are fused to obtain fused features, and the fused features are associated with the target entities contained in the medical text data. The target entities, the relationships between target entity pairs, and the fused features associated with the target entities can then be added to the medical ultrasound domain knowledge base. The knowledge base contains not only text modal information but also other modal information, thereby improving the knowledge coverage of the knowledge base and providing a more comprehensive knowledge representation. Furthermore, the method of the present application does not rely on manual rule-making, thereby improving the efficiency of knowledge base construction.

[0063] In some embodiments of the present application, the process of performing entity extraction on medical text data to obtain target entities in the aforementioned step S110 is described.

[0064] Specifically, an entity extraction model can be used to extract entities from medical text data to obtain target entities in the medical text data.

[0065] Medical text data can be pre-divided into basic units, for example, by sentences, and then entities are extracted from each sentence to obtain the target entity corresponding to each sentence.

[0066] Among them, the entity extraction model can be trained using sample data labeled with entity extraction results.

[0067] Conventional entity extraction model training requires extensive manual annotation, which is inefficient and costly. This embodiment introduces a weakly supervised learning paradigm that doesn't rely on large amounts of precisely labeled data for training. The entity extraction model is trained using a multi-view semi-supervised learning approach.

[0068] Multi-view semi-supervised learning (MSL) is a machine learning method that combines multi-view learning and semi-supervised learning. It leverages data from different perspectives and limited labeled data to improve model learning. Multi-view learning refers to learning from different perspectives (i.e., different feature sets or data representations) of the same object or instance. Each perspective can be considered a different way of describing the same thing. For example, in medical diagnosis, different perspectives may include a patient's symptom description or laboratory test results. By integrating these different perspectives, the model can obtain more comprehensive information, thereby improving performance. Semi-supervised learning refers to a training method that uses a small amount of labeled data and a large amount of unlabeled data. Its core concept is that even without labels, unlabeled data still contains valuable information, such as the structural characteristics of the data distribution.

[0069] The role of multi-view semi-supervised training is to improve the "representation" learning of the model. Figure 3 , which illustrates a multi-view semi-supervised training model based on an auxiliary objective function, which includes a primary prediction module and several auxiliary prediction modules. The auxiliary prediction modules can learn from the predictions of the primary prediction module because the primary prediction module has better input with unrestricted view. Although the auxiliary prediction modules are fed with restricted input samples, they can still learn to make the predictions of the primary prediction module after training, thus improving the quality of "representation". This in turn improves the entire model because they share the encoding module Encoder. Figure 3 BERT is used as an encoding module for illustrative purposes. The BERT pre-trained language model generates word vectors representing contextual semantic information and automatically extracts a large number of word-level and semantic-level features from the text. These are decoded in the main and auxiliary prediction modules to generate entity label sequences, which are then used for entity recognition and extraction. Figure 3 The auxiliary prediction module shown can use a small neural network, for example, a conditional random field (CRF) model, which explicitly considers contextual associations at the output.

[0070] Figure 3 The paper illustrates a cross-view training (CVT) strategy, which is a semi-supervised algorithm that uses unlabeled and labeled data to train word representations in a distributed manner. When learning unlabeled data, CVT uses several auxiliary prediction modules, each of which receives an intermediate representation generated by the model. As input, output a label distribution . Each is selected to use only a portion of the input The specific choice may depend on the task and model architecture. The auxiliary prediction module is only used during training; the predictions during the model inference phase are generated by the main prediction module. .

[0071] Figure 3 In the model structure shown, the main prediction module and the auxiliary prediction module share the same encoding module, which adopts the BERT architecture. The main prediction module makes predictions based on labeled training data and is trained to output semantically appropriate labels. The auxiliary prediction module generates training data based on multi-view random noise addition (i.e., for image data, random noise is added by adding Gaussian noise or randomly occluding regions; for text data, random word replacement or deletion is performed). Unlabeled training data is processed separately by the main prediction module and the auxiliary prediction module. The auxiliary prediction module takes input from a partially restricted view (for example, only a portion of a sentence) to train and fit the predictions obtained from the full view input in the main prediction module. Because the main and auxiliary prediction modules share the encoding module, the cross-view training method can improve the model's representation learning.

[0072] After the entity extraction model is trained, the entity extraction task can be performed through the main prediction module.

[0073] Figure 3 The middle right side shows a specific example of using a trained entity extraction model to perform an entity extraction task.

[0074] In this embodiment, by introducing a multi-view semi-supervised learning method to train the entity extraction model, the purpose of reducing manual labeling costs can be achieved and the ability to process large-scale data can be improved.

[0075] In some embodiments of the present application, the process of extracting the relationship between target entity pairs in medical text data in the aforementioned step S110 is described.

[0076] This embodiment provides an end-to-end relationship extraction technology based on remote supervision. Specifically, a trained remote supervision relationship extraction model can be used to extract the relationship between target entity pairs in medical text data.

[0077] Traditional supervised approaches require manual annotation of large amounts of data, making it difficult to achieve fast and effective knowledge construction. In order to quickly obtain rich entity relationships, a remotely supervised end-to-end relationship extraction technique is proposed.

[0078] Distantly supervised relation extraction is a technique that uses an external knowledge base to automatically annotate training data and learn relationships between entities. This method generates labels for large amounts of unlabeled text by assuming that known entity relationships in the knowledge base also exist in the text, which can then be used to train machine learning models.

[0079] Traditional distant supervision methods assume that all sentences containing the same pair of entities express predefined relationships in external knowledge bases (such as the FreeBase knowledge base). However, in reality, a large number of sentences are noise (do not express predefined relationships), resulting in low quality of labeled data.

[0080] To this end, this embodiment provides a remotely supervised relationship extraction model based on a sentence-level attention mechanism. In this model, dynamic weighting is performed using a sentence-level attention mechanism. This involves calculating attention weights for multiple sentences (i.e., a set of sentences) corresponding to each entity pair, highlighting key sentences that express predefined relationships and suppressing noisy sentences.

[0081] This embodiment introduces a training method for a remotely supervised relationship extraction model, which may specifically include the following steps:

[0082] S1. For sample entities A and B, extract sentences containing both sample entities A and B from medical text data to form a sentence set, and identify the relationship r between sample entities A and B from an external knowledge base, and use the relationship r as the relationship type label of the sentence set.

[0083] S2. Send the sentence set and the syntactic tree representation of each sentence therein into the remote supervision relationship extraction model, calculate the relationship vector representation of each sentence and the attention weight of each sentence, perform weighted addition on the relationship vector representation of each sentence according to the attention weight of the sentence, and obtain a weighted relationship vector representation. Based on the weighted relationship vector representation, predict the relationship type corresponding to the sentence set.

[0084] S3. Calculate the value of the loss function based on the predicted relationship type of the sentence set and the relationship type label of the sentence set, and update the model parameters according to the value of the loss function until the training end condition is met.

[0085] The model training method described in this embodiment includes not only a sentence set as input, but also a syntactic tree representation of each sentence. The syntactic tree representation can cover information at the semantic space level of the sample entity pairs, which is conducive to more accurate calculation of the relationship vector representation of each sentence. Within the model, by calculating sentence-level attention weights, the model will learn to give higher weights to sentences that truly express the predefined relationship r, thereby reducing the influence of noisy sentences in the weighted aggregation and reducing the impact of noisy sentences in the sentence set on model training.

[0086] After obtaining the trained remote supervision relationship extraction model, the medical text data containing the target entity pairs and the syntactic tree representation of the medical text data can be sent to the remote supervision relationship extraction model to obtain the relationship between the target entity pairs output by the model, thereby completing the relationship extraction task.

[0087] Reference Figure 4 , which illustrates a structural diagram of a remote supervision relationship extraction model.

[0088] like Figure 4 As shown in Figure 2, the remote supervision relationship extraction model can include the following network structures:

[0089] The input layer is used to input a sentence collection and the syntax tree representation of each sentence.

[0090] Figure 4 In sentence 1 of the example, "[DIS]" indicates that entity A "Krabbe disease" belongs to the disease type, and "SYM" indicates that entity B "lower limb spasms" belongs to the symptom type.

[0091] Sentences containing both sample entities A and B are collected to form a sentence set. A corresponding syntax tree representation is generated for each sentence. The sentence set and syntax tree representation are fed into the model through the input layer.

[0092] The encoding layer includes a text encoding module and a graph neural network module. The text encoding module is used to encode each sentence to obtain a sentence-level vector representation of sample entities A and B. The graph neural network module is used to encode the syntactic tree representation of each sentence to obtain a semantic space-level vector representation of sample entities A and B.

[0093] Figure 4 The encoding layer uses the BERT network as the text encoding module, but other text encoding structures can also be used. The graph neural network module can encode the syntactic tree representation of each sentence through graph convolution to obtain the semantic space level vector representation of sample entities A and B.

[0094] The relation representation layer is used to calculate the relation vector representation of each sentence based on the sentence-level vector representation and semantic space-level vector representation of entities A and B.

[0095] Specifically, we can calculate the sentence-level relationship vector representation based on the sentence-level vector representation of sample entities A and B, for example, v1 relation =v A -v B , where v A 、v B Represent the sentence-level vector representations of sample entities A and B respectively.

[0096] Based on the semantic space vector representation of sample entities A and B, calculate the relationship vector representation at the semantic space level. For example, through aggregation methods such as max-pool, the semantic space vector representation of sample entities A and B is aggregated to obtain the relationship vector representation v2 at the semantic space level. relation .

[0097] For each sentence, the sentence-level relationship vector v1 can be represented relation and semantic space level relation vector representation v2 relation Merge and get the relation vector representation of the sentence.

[0098] The vector fusion layer is used to calculate the attention weight of each sentence and perform weighted addition on the relationship vector representations of each sentence according to the attention weight of the sentence to obtain the weighted relationship vector representation.

[0099] Figure 4 In the definition, the sentence set contains m sentences, then the attention weight a=[a1,a2,…,a m ], that is, there is a corresponding attention weight for each sentence. By weighted addition of the relationship vector representation of the sentence according to the attention weight, a weighted relationship vector representation can be obtained.

[0100] The output layer is used to predict the relationship type corresponding to the sentence set based on the weighted relationship vector representation.

[0101] Specifically, the output layer can judge the relationship type through a softmax classifier.

[0102] The remotely supervised relationship extraction model provided in the embodiments of the present application can reduce the cost of manually annotating data and, through a sentence-level attention mechanism, highlight the importance of sentences expressing the predefined relationship r, thereby reducing the impact of noise sentences in the sentence set on model training. In addition, by simultaneously inputting sentences and their syntactic tree representations into the input layer, the relationship vector representation of the sentence can be calculated at both the sentence level and the semantic space level, improving the correctness of the relationship vector representation and, in turn, helping to improve the accuracy of the final relationship classification results.

[0103] In some embodiments of the present application, the training process of the multimodal unsupervised pre-training model used in the aforementioned step S120 is described.

[0104] Medical ultrasound data exists in multiple modalities, such as text, images, and videos, and is also an important medium for knowledge storage. To effectively utilize data from different modalities, this embodiment proposes a unified representation technique for multimodal information based on mask prediction. By modeling the inter-modal correlations, a shared embedding space is learned that can simultaneously represent multimodal features. This enables the association of semantic spaces such as text and images, thereby improving the model's performance in downstream tasks.

[0105] In this embodiment, the initial model is unsupervisedly trained using a mask prediction task to obtain a trained multimodal model, wherein the mask prediction task includes at least two of the following:

[0106] Predict the masked text, predict whether the image area corresponding to the text is masked, predict whether the text corresponding to the image area is masked, and whether the predicted text and image match.

[0107] Mask generation can be achieved by randomly selecting data fragments from some modalities and replacing them with special tags (such as [MASK]). For images, this can be done by masking parts of the image, and for text, by masking some words or sentences.

[0108] The initial model may be a model capable of processing multimodal data.

[0109] By adopting the multimodal model trained by the above-mentioned mask prediction task, it has the ability to uniformly represent multimodal information.

[0110] Furthermore, in order to achieve alignment between cross-modal features, based on the above-mentioned mask prediction task, the contrastive learning method can be further used to train the multimodal model to train the cross-modal semantic alignment ability of the multimodal model and obtain the trained multimodal unsupervised pre-training model.

[0111] Combine Figure 5 , which illustrates a process of aligning text and image modality features.

[0112] Positive training samples and negative training samples are selected. The text and image in the positive training samples are related, for example, the text and image in the same ultrasound examination report are used as positive training samples. The text and image in the negative training samples are unrelated.

[0113] The multimodal model consists of an input layer and an encoder layer. The encoder layer includes encoding modules for different modalities. Figure 5 As shown in the example, the BERT network can be used to encode text, and the image can be projected through the Linear Projection of Flattened Patches layer and then encoded through the Transformer Encoder.

[0114] During the training phase, the training goal is to narrow the distance between the encoding features (vector representations) of different modalities in positive training samples and to widen the distance between the features of different modalities in negative training samples. After training, a multimodal unsupervised pre-training model with cross-modal semantic alignment capabilities is obtained.

[0115] Further optionally, the multimodal unsupervised pre-training model can also include a position encoding module for modeling the spatial position of the input image data. By adding position encoding to the image data, position features and layout features can be extracted to better characterize the anatomical structure and its associated diagnostic information. The position features can clarify the spatial position of each pixel or partition in the image (such as two-dimensional coordinates), helping the model to distinguish different anatomical regions. The layout features capture the overall anatomical layout by modeling the spatial distribution relationship between anatomical structures (such as adjacency and shape features).

[0116] In some embodiments of the present application, the process of fusing text features and other modality features to obtain fused features in the aforementioned step S130 is described.

[0117] An optional feature fusion strategy is to directly splice text features and other modal features and use the spliced ​​features as fusion features.

[0118] The above feature fusion strategy can quickly fuse multimodal features, but there is a lack of information exchange and complementarity between different modal features. To this end, this application provides another feature fusion strategy:

[0119] First, the text features and other modal features are concatenated to obtain concatenated features.

[0120] Furthermore, the concatenated features are fed into the Transformer model to achieve cross-modal feature fusion using the Transformer's attention mechanism, and the hidden features extracted by the Transformer decoder are obtained as fused features.

[0121] In this embodiment, the attention mechanism of the Transformer model can be used to achieve cross-modal feature fusion processing, so that information exchange and complementation can be carried out between features of different modalities in the spliced ​​features, thereby improving the expressive power of the fused features.

[0122] In a possible implementation, the decoder of the above Transformer model may adopt an autoregressive tree decoder (TD).

[0123] The existing autoregressive tree decoder is used in scenarios where mathematical expressions or characters are parsed. This application innovatively applies the decoder to the process of cross-modal feature fusion to optimize the information complementarity between different modalities.

[0124] The autoregressive tree-structured decoder consists of four modules: the Parent Module, the Child Module, the Memory Module, and the Relation Module. They simultaneously achieve the self-supervised pre-training goal of the mask prediction task by fusing and transferring information. The Parent State represents the state of the parent node, which contains information passed from the encoder and the parent node's own state. Through an attention mechanism, a portion of this state is transferred to the Child State of the Child Module, depending on the task objective. A portion forms the Memory Module, which stores and manages intermediate states and information generated during the decoding process. The Child State, through the attention mechanism, performs mask prediction for downstream tasks and, through the Relation Module, ensures correlation with the information stored in the Memory Module at the current level. The detailed structure of the autoregressive tree-structured decoder can be found in the relevant technical specifications and will not be elaborated here.

[0125] The multimodal feature fusion technology provided in this embodiment uses a splicing method to combine features of different modalities, and further uses the Transformer architecture for cross-modal feature fusion. In essence, the spliced ​​features are not a whole and do not have the semantic continuity of the context. Therefore, simply using the attention of the Transformer architecture cannot well obtain the hidden layer representation of different modal features from a higher dimension. Therefore, combined with the properties of the attention mechanism and the structural characteristics of the current input, an autoregressive tree decoder is introduced. By constructing four modules with a hierarchical structure and logical order, it captures the different dimensional information of the data, while meeting the transmission and fusion between different modal information, thereby improving the expressive power of the final fusion feature.

[0126] In some embodiments of the present application, the process of adding the target entity, the relationship between the target entity pair, and the fusion features associated with the target entity to the medical ultrasound field knowledge base in the aforementioned step S140 is described.

[0127] The multimodal medical ultrasound data acquired in this application can come from multiple sources, potentially leading to knowledge duplication and unclear relationships between knowledge. The same entity may be represented differently in different data sources, making entity linking a crucial step in knowledge fusion. Entity linking is the process of determining whether entities in heterogeneous, multi-source data refer to the same real-world object.

[0128] Entity extraction technology can identify target entities within text. These target entities may be ambiguous or even unknown. In the face of ambiguity, entity linking technology is needed to link the entity to a unique entity in the objective world. Entity linking technology can, on the one hand, eliminate conceptual ambiguity within knowledge elements and eliminate redundant and erroneous knowledge elements, thereby ensuring the quality of knowledge base construction; on the other hand, it can proactively discover new entities, thereby maintaining the real-time and completeness of the knowledge base.

[0129] Combine Figure 6 , which illustrates a process of entity linking.

[0130] For the target entity extracted from medical text data, the confidence of the entity link is calculated for the target entity and each entity object in the medical ultrasound knowledge base. If there is a candidate entity object with a confidence greater than a threshold, the target entity is linked to the candidate entity object as a synonym of the candidate entity object.

[0131] Among them, the construction process of the knowledge base in the field of medical ultrasound can also be understood as an updating process, that is, the process of adding the latest extracted target entities, the relationship between target entities, and the fusion features of the target entity associations to the knowledge base. The target entity storage process involves entity linking. At this time, it is necessary to calculate the confidence between the target entity and the existing entity objects in the knowledge base, and select the storage method of the target entity according to the confidence level.

[0132] Combine Figure 6 As shown, if there is a candidate entity object in the knowledge base whose confidence with the target entity is greater than a threshold, the target entity can be used as a synonym of the candidate entity object and linked to the candidate entity object.

[0133] If there is no candidate entity in the knowledge base with a confidence score greater than a threshold for the target entity, an expert can be arranged to verify whether the target entity is legal. If it is illegal, it will be discarded directly. If it is legal, the target entity can be added to the database, that is, added to the knowledge base as a new entity object.

[0134] After the target entities are stored in the database, the relationships between the target entities can be further updated in the knowledge base, and the fusion features associated with the target entities can be added to the knowledge base.

[0135] This embodiment introduces an optional confidence calculation process for entity linking between a target entity and each entity object in a medical ultrasound field knowledge base.

[0136] Combine Figure 7 As shown, Figure 7 This example illustrates the processing of an entity link prediction model.

[0137] The input of the input layer includes: the target entity, the syntactic tree representation of the medical text data where the target entity is located, and the entity object to be linked in the knowledge base.

[0138] The target entity, the entity object to be linked and the syntactic tree representation of the medical text data where the target entity is located are encoded respectively through the encoding layer to obtain the target entity feature representation, the entity object feature representation to be linked and the syntactic tree feature representation.

[0139] The encoding process of the target entity and the entity object to be linked can be performed using a text encoder, such as the BERT model. The encoding process of the syntactic tree representation of the medical text data containing the target entity can be performed using a graph convolutional network (GCN).

[0140] The attention layer performs cross attention calculation on the feature representation of the linked entity object and the target entity feature representation to obtain two sub-attention context vectors ( Figure 7The difference between the two sub-attention context vectors is calculated by the differential operator, and the result is used as the first differential attention context vector.

[0141] Similarly, the attention layer performs cross-attention calculation on the feature representation of the linked entity object and the feature representation of the syntactic tree to obtain two sub-attention context vectors ( Figure 7 The difference between the two sub-attention context vectors is calculated by the differential operator, and the result is used as the second differential attention context vector.

[0142] This embodiment proposes a strategy that combines a cross-attention mechanism with a differential operator, also known as a differential attention mechanism. The cross-attention mechanism enables information interaction and fusion, while the differential operator eliminates attention noise, further enhancing the ability to model context between entities and improving the accuracy of entity link confidence calculation results.

[0143] The confidence level of entity linking between the target entity and the entity object to be linked (also known as the similarity between the target entity and the entity object to be linked) is determined through the output layer based on the first differential attention context vector and the second differential attention context vector.

[0144] Since the first differential attention context vector and the second differential attention context vector represent the similarity between the target entity and the entity object to be linked from two dimensions, namely the target entity and the syntactic tree representation of the medical text data where the target entity is located, the first differential attention context vector and the second differential attention context vector can be weighted averaged, and based on the weighted average vector, the similarity between the target entity and the entity object to be linked is determined as the confidence level of the entity link between the two.

[0145] In summary:

[0146] In terms of the overall solution process, this application adopts an entity extraction method based on multi-perspective semi-supervision and a relationship extraction technology based on remote supervision to reduce the cost of manual labeling and improve the ability to process large-scale data. It proposes a self-supervised multimodal information representation technology based on mask prediction, uses a multi-task learning strategy to achieve a unified representation of the entity semantic space, and introduces the attention mechanism of the differential operator in the entity link prediction model to achieve highly accurate knowledge links, thereby ensuring the alignment of multimodal information as a whole.

[0147] In the entity extraction process, a multi-view semi-supervised training model is adopted. Through cross-view training, accurate deep information extraction from multi-source data is achieved. In addition, through the shared encoder architecture, the entity extraction task is completed during the entity recognition process, which also indirectly demonstrates the idea of ​​transfer learning.

[0148] In the relationship extraction process, a remotely supervised relationship extraction model is adopted. It assumes that if two entities have a certain relationship in the knowledge base, then the text instances containing these two entities also express the same relationship. This enables learning complex relationship patterns from the original text without the need for explicit feature engineering, thereby greatly reducing the need for manual labeling.

[0149] This paper proposes a multimodal unsupervised pre-training model based on mask prediction. By designing a specific masking task, it learns the intrinsic connections between data from different modalities (such as text and images), thereby constructing a universal feature space that can capture cross-modal information. Using a multi-task joint learning approach and an autoregressive tree-like decoder design, it hierarchically addresses the problem of integrating and aligning data from different modalities, ensuring that they have similar semantic meaning in a common space.

[0150] Based on the results of entity extraction and relationship extraction, by constructing an entity link prediction model that includes cross-attention and differential operators, while eliminating knowledge ambiguity, it ensures the dynamic update of the knowledge base and meets the needs of incremental knowledge base construction.

[0151] The following describes a device for constructing a medical ultrasound field knowledge base provided in an embodiment of the present application. The device for constructing a medical ultrasound field knowledge base described below and the method for constructing a medical ultrasound field knowledge base described above can refer to each other.

[0152] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a medical ultrasound field knowledge base construction device disclosed in an embodiment of the present application.

[0153] like Figure 8 As shown, the device may include:

[0154] A data acquisition unit 11 is configured to acquire multimodal medical ultrasound data, wherein the multimodal medical ultrasound data includes medical text data and other associated modality data;

[0155] An entity and relationship extraction unit 12 is configured to perform entity extraction on the medical text data to obtain target entities, and to extract relationships between pairs of target entities in the medical text data;

[0156] A multimodal feature extraction unit 13 is configured to call a multimodal unsupervised pre-trained model to perform feature extraction on the medical text data and its associated other modal data to obtain text features and other modal features, wherein the multimodal unsupervised pre-trained model is configured to extract features of the multimodal data and perform modality alignment;

[0157] A multimodal feature fusion unit 14 is configured to fuse the text feature and other modal features to obtain a fused feature, and establish an association relationship between the fused feature and the target entity contained in the medical text data;

[0158] The knowledge base updating unit 15 is configured to add the target entity, the relationship between the target entity pairs, and the fusion features associated with the target entities to a medical ultrasound field knowledge base.

[0159] In one possible implementation, the apparatus of the present application may further include:

[0160] The multimodal model training unit is used to train the multimodal unsupervised pre-training model. The training process includes:

[0161] The initial model is unsupervisedly trained using a mask prediction task to obtain a trained multimodal model, wherein the mask prediction task includes at least two of the following:

[0162] Predict the masked text, predict whether the image area corresponding to the text is masked, predict whether the text corresponding to the image area is masked, and whether the predicted text and image match;

[0163] The multimodal model is trained using a contrastive learning method to train the cross-modal semantic alignment capability of the multimodal model, thereby obtaining a trained multimodal unsupervised pre-training model.

[0164] In one possible implementation, the process of the multimodal feature fusion unit fusing the text feature and other modal features to obtain a fused feature includes:

[0165] Splicing the text features and other modal features to obtain splicing features;

[0166] The concatenated features are fed into the Transformer model to achieve cross-modal feature fusion using the Transformer's attention mechanism, and the hidden layer features extracted by the Transformer decoder are obtained as the fused features.

[0167] In one possible implementation, the decoder of the Transformer model adopts an autoregressive tree decoder.

[0168] In one possible implementation, the entity and relationship extraction unit extracts entities from the medical text data to obtain target entities, including:

[0169] An entity extraction model is used to extract entities from the medical text data to obtain target entities, and the entity extraction model is trained using a multi-view semi-supervised learning method.

[0170] In one possible implementation, the process of extracting the relationship between the target entity pairs in the medical text data by the entity and relationship extraction unit includes:

[0171] Sending medical text data containing target entity pairs and a syntactic tree representation of the medical text data into a remotely supervised relationship extraction model to obtain the relationship between the target entity pairs output by the model;

[0172] The training process of the remote supervision relationship extraction model includes:

[0173] For sample entities A and B, extract sentences containing both sample entities A and B from medical text data to form a sentence set, and identify the relationship r between sample entities A and B from an external knowledge base, and use the relationship r as the relationship type label of the sentence set;

[0174] The sentence set and the syntactic tree representation of each sentence therein are fed into a remotely supervised relation extraction model, the relation vector representation of each sentence and the attention weight of each sentence are calculated, the relation vector representations of each sentence are weightedly added according to the attention weight of the sentence to obtain a weighted relation vector representation, and the relation category corresponding to the sentence set is predicted based on the weighted relation vector representation;

[0175] The loss function is calculated based on the predicted relationship types of the sentence set and the relationship type labels of the sentence set, and the model parameters are updated according to the value of the loss function until the training end condition is met.

[0176] In one possible implementation, the remote supervision relationship extraction model includes:

[0177] An input layer, for inputting the sentence set and the syntax tree representation of each sentence therein;

[0178] The encoding layer includes a text encoding module and a graph neural network module. The text encoding module is used to encode each sentence to obtain a sentence-level vector representation of sample entities A and B. The graph neural network module is used to encode the syntactic tree representation of each sentence to obtain a semantic space-level vector representation of sample entities A and B.

[0179] The relation representation layer is used to calculate the relation vector representation of each sentence based on the sentence-level vector representation and semantic space-level vector representation of sample entities A and B;

[0180] The vector fusion layer is used to calculate the attention weight of each sentence and perform weighted addition of the relationship vector representations of each sentence according to the sentence's attention weight to obtain a weighted relationship vector representation;

[0181] The output layer is used to predict the relationship type corresponding to the sentence set based on the weighted relationship vector representation.

[0182] In one possible implementation, the process of the knowledge base updating unit adding the target entity to the medical ultrasound field knowledge base includes:

[0183] The confidence of entity linking is calculated for the target entity and each entity object in the medical ultrasound field knowledge base. If there is a candidate entity object with a confidence greater than a threshold, the target entity is linked to the candidate entity object as a synonym of the candidate entity object.

[0184] In one possible implementation, the process of calculating the confidence of entity links between the target entity and each entity object in the medical ultrasound field knowledge base by the knowledge base updating unit includes:

[0185] Encoding the target entity, the entity object to be linked, and the syntax tree representation of the medical text data where the target entity is located respectively to obtain a target entity feature representation, a to-be-linked entity object feature representation, and a syntax tree feature representation;

[0186] Performing cross attention calculation on the feature representation of the entity object to be linked and the feature representation of the target entity to obtain two sub-attention context vectors, and calculating the difference between the two sub-attention context vectors by a difference operator, and using the result as the first differential attention context vector;

[0187] Performing cross attention calculation on the feature representation of the entity object to be linked and the feature representation of the syntactic tree to obtain two sub-attention context vectors, and calculating the difference between the two sub-attention context vectors by a differential operator, and using the result as a second differential attention context vector;

[0188] Based on the first differential attention context vector and the second differential attention context vector, determine the confidence of entity linking between the target entity and the entity object to be linked.

[0189] In one possible implementation, the process of determining the confidence level of entity linking between the target entity and the to-be-linked entity object based on the first differential attention context vector and the second differential attention context vector by the knowledge base updating unit includes:

[0190] A weighted average is performed on the first differential attention context vector and the second differential attention context vector, and based on the weighted average vector, the confidence level of the entity linking between the target entity and the entity object to be linked is determined.

[0191] An electronic device is also provided in an embodiment of the present application. Figure 9 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, etc. Figure 9 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0192] like Figure 9 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 into the random access memory (RAM) 603 to implement the medical ultrasound field knowledge base construction method of the aforementioned embodiment of the present application. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0193] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 9 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0194] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the medical ultrasound field knowledge base construction methods provided in the embodiments of the present application.

[0195] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the medical ultrasound field knowledge base construction methods provided in the embodiment of the present application.

[0196] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0197] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0198] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0199] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0200] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referenced to each other.

Claims

1. A method for constructing a knowledge base in the field of medical ultrasound, characterized in that: include: Acquiring multimodal medical ultrasound data, where the multimodal medical ultrasound data includes medical text data and other associated modality data; Performing entity extraction on the medical text data to obtain target entities, and extracting relationships between target entity pairs in the medical text data; Calling a multimodal unsupervised pre-trained model to perform feature extraction on the medical text data and its associated other modal data to obtain text features and other modal features, wherein the multimodal unsupervised pre-trained model is configured to extract features of the multimodal data and perform modality alignment; fusing the text features and other modality features to obtain fused features, and establishing an association relationship between the fused features and the target entity contained in the medical text data; adding the target entity, the relationship between the target entity pairs, and the fusion features associated with the target entities to a medical ultrasound domain knowledge base; The process of adding the target entity to the medical ultrasound domain knowledge base includes: Calculating the confidence of entity linking for the target entity and each entity object in the medical ultrasound field knowledge base; if there is a candidate entity object with a confidence greater than a threshold, linking the target entity to the candidate entity object as a synonym of the candidate entity object; The process of calculating the confidence of entity linking of the target entity and each entity object in the medical ultrasound field knowledge base includes: Encoding the target entity, the entity object to be linked, and the syntax tree representation of the medical text data where the target entity is located respectively to obtain a target entity feature representation, a to-be-linked entity object feature representation, and a syntax tree feature representation; Performing cross attention calculation on the feature representation of the entity object to be linked and the feature representation of the target entity to obtain two sub-attention context vectors, and calculating the difference between the two sub-attention context vectors by a difference operator, and using the result as the first differential attention context vector; Performing cross attention calculation on the feature representation of the entity object to be linked and the feature representation of the syntactic tree to obtain two sub-attention context vectors, and calculating the difference between the two sub-attention context vectors by a differential operator, and using the result as a second differential attention context vector; Based on the first differential attention context vector and the second differential attention context vector, determine the confidence of entity linking between the target entity and the entity object to be linked.

2. The method according to claim 1, characterized in that The training process of the multimodal unsupervised pre-training model includes: The initial model is unsupervisedly trained using a mask prediction task to obtain a trained multimodal model, wherein the mask prediction task includes at least two of the following: Predict the masked text, predict whether the image area corresponding to the text is masked, predict whether the text corresponding to the image area is masked, and whether the predicted text and image match; The multimodal model is trained using a contrastive learning method to train the cross-modal semantic alignment capability of the multimodal model, thereby obtaining a trained multimodal unsupervised pre-training model.

3. The method according to claim 1, characterized in that The process of fusing the text features and other modal features to obtain fused features includes: Splicing the text features and other modal features to obtain splicing features; The concatenated features are fed into the Transformer model to achieve cross-modal feature fusion using the Transformer's attention mechanism, and the hidden layer features extracted by the Transformer decoder are obtained as the fused features.

4. The method according to claim 3, characterized in that The decoder of the Transformer model adopts an autoregressive tree decoder.

5. The method according to claim 1, wherein The process of extracting entities from the medical text data to obtain target entities includes: An entity extraction model is used to extract entities from the medical text data to obtain target entities, and the entity extraction model is trained using a multi-view semi-supervised learning method.

6. The method according to claim 1, wherein The process of extracting the relationship between target entity pairs in the medical text data includes: Sending medical text data containing target entity pairs and a syntactic tree representation of the medical text data into a remotely supervised relationship extraction model to obtain the relationship between the target entity pairs output by the model; The training process of the remote supervision relationship extraction model includes: For sample entities A and B, extract sentences containing both sample entities A and B from medical text data to form a sentence set, and identify the relationship r between sample entities A and B from an external knowledge base, and use the relationship r as the relationship type label of the sentence set; The sentence set and the syntactic tree representation of each sentence therein are fed into a remotely supervised relation extraction model, the relation vector representation of each sentence and the attention weight of each sentence are calculated, the relation vector representations of each sentence are weightedly added according to the attention weight of the sentence to obtain a weighted relation vector representation, and the relation category corresponding to the sentence set is predicted based on the weighted relation vector representation; The loss function is calculated based on the predicted relationship types of the sentence set and the relationship type labels of the sentence set, and the model parameters are updated according to the value of the loss function until the training end condition is met.

7. The method according to claim 6, characterized in that The remote supervision relationship extraction model includes: An input layer, for inputting the sentence set and the syntax tree representation of each sentence therein; The encoding layer includes a text encoding module and a graph neural network module. The text encoding module is used to encode each sentence to obtain a sentence-level vector representation of sample entities A and B. The graph neural network module is used to encode the syntactic tree representation of each sentence to obtain a semantic space-level vector representation of sample entities A and B. The relation representation layer is used to calculate the relation vector representation of each sentence based on the sentence-level vector representation and semantic space-level vector representation of sample entities A and B; The vector fusion layer is used to calculate the attention weight of each sentence and perform weighted addition of the relationship vector representations of each sentence according to the sentence's attention weight to obtain a weighted relationship vector representation; The output layer is used to predict the relationship type corresponding to the sentence set based on the weighted relationship vector representation.

8. The method according to claim 1, characterized in that The process of determining, based on the first differential attention context vector and the second differential attention context vector, a confidence level of entity linking between the target entity and the to-be-linked entity object, comprising: A weighted average is performed on the first differential attention context vector and the second differential attention context vector, and based on the weighted average vector, the confidence level of the entity linking between the target entity and the entity object to be linked is determined.

9. A device for constructing a knowledge base in the field of medical ultrasound, characterized in that: include: A data acquisition unit, configured to acquire multimodal medical ultrasound data, wherein the multimodal medical ultrasound data includes medical text data and other associated modality data; An entity and relationship extraction unit, configured to perform entity extraction on the medical text data to obtain target entities, and extract relationships between pairs of target entities in the medical text data; a multimodal feature extraction unit, configured to call a multimodal unsupervised pre-trained model to perform feature extraction on the medical text data and its associated other modal data to obtain text features and other modal features, wherein the multimodal unsupervised pre-trained model is configured to extract features of the multimodal data and perform modality alignment; a multimodal feature fusion unit, configured to fuse the text feature and other modal features to obtain a fused feature, and establish an association relationship between the fused feature and the target entity contained in the medical text data; a knowledge base updating unit, configured to add the target entity, the relationship between the target entity pair, and the fusion feature associated with the target entity to a medical ultrasound field knowledge base; The process of the knowledge base updating unit adding the target entity to the medical ultrasound field knowledge base includes: Calculating the confidence of entity linking for the target entity and each entity object in the medical ultrasound field knowledge base; if there is a candidate entity object with a confidence greater than a threshold, linking the target entity to the candidate entity object as a synonym of the candidate entity object; The process of calculating the confidence of entity links of the target entity and each entity object in the medical ultrasound field knowledge base by the knowledge base updating unit includes: Encoding the target entity, the entity object to be linked, and the syntax tree representation of the medical text data where the target entity is located respectively to obtain a target entity feature representation, a to-be-linked entity object feature representation, and a syntax tree feature representation; Performing cross attention calculation on the feature representation of the entity object to be linked and the feature representation of the target entity to obtain two sub-attention context vectors, and calculating the difference between the two sub-attention context vectors by a difference operator, and using the result as the first differential attention context vector; Performing cross attention calculation on the feature representation of the entity object to be linked and the feature representation of the syntactic tree to obtain two sub-attention context vectors, and calculating the difference between the two sub-attention context vectors by a differential operator, and using the result as a second differential attention context vector; Based on the first differential attention context vector and the second differential attention context vector, determine the confidence of entity linking between the target entity and the entity object to be linked.

10. An electronic device, characterized in that: include: memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the method for constructing a medical ultrasound field knowledge base according to any one of claims 1 to 8.

11. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method for constructing a medical ultrasound field knowledge base according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the method for constructing a medical ultrasound field knowledge base as claimed in any one of claims 1 to 8 is implemented.

Images