Medical named entity identification method and device, equipment, storage medium and product
By integrating multimodal representations of text and visual information and using multiple encoders for information fusion, the problem of insufficient recognition accuracy and adaptability in existing technologies is solved, achieving higher precision medical named entity recognition and medical knowledge adaptability.
Patent Information
- Application Number
- CN202510595348.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2026-02-10
AI Technical Summary
Existing medical named entity recognition technologies rely on a single text modality, making it difficult to adapt to the rapid updates in medical knowledge, resulting in insufficient recognition accuracy and adaptability.
Multimodal representations that integrate textual and visual information are encoded using different types of encoders, and information is fused between the encoders to generate medical named entity recognition results.
It improves the recognition accuracy of named entities in medical texts, enhances the model's adaptability to medical knowledge, and enables it to better adapt to the continuous updating of medical knowledge.
Smart Images
Figure CN121503480A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a medical named entity recognition method, apparatus, device, storage medium, and product. Background Technology
[0002] Medical Named Entity Recognition (NER) primarily refers to identifying entities with specific meanings from medical texts, such as disease names and drug names. In the medical field, this type of NER provides a foundation for in-depth data analysis and mining, helps improve the efficiency of medical data processing, and provides strong support for clinical decision-making, medical research, and drug development. To accurately extract key information from unstructured medical texts, researchers have developed various technical solutions for this field, such as rule-based and dictionary-based methods, machine learning methods, and pre-trained language models.
[0003] However, regardless of the specific technical solution mentioned above, the models used in their implementation primarily rely on a single text modality, which limits the model's comprehensive understanding of medical named entities. Furthermore, these models typically use pre-trained clinical language models or outdated knowledge bases, resulting in limitations in their coverage and difficulty in adapting to the rapid updates in medical knowledge. For example, when distinguishing between "myocardial infarction" and "infarcted cardiomyopathy," relying solely on a single text modality may fail to fully capture the specific context and meaning behind the terms. Moreover, when medical knowledge is updated or ambiguous, over-reliance on text-based methods may lead to recognition errors. Therefore, existing technical solutions have significant shortcomings in handling semantic ambiguity and adapting to knowledge updates, affecting both their recognition accuracy and limiting their application in the dynamically changing medical field. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to provide a medical named entity recognition method, device, equipment, storage medium and product, which inputs a multimodal representation that integrates textual and visual information into multiple encoders of different types for processing, realizing information fusion between different modalities and between different encoders. This not only improves the model's recognition accuracy of named entities in medical texts, but also helps the model better adapt to the continuous updating of medical knowledge.
[0005] To achieve the above objectives, embodiments of the present invention provide a medical named entity recognition method, comprising:
[0006] Obtain the medical text to be identified, and generate visual information based on the text information of the medical text to be identified;
[0007] Based on the text information and the visual information, multimodal input information is generated;
[0008] The multimodal input information is input into multiple encoders of different types for encoding processing.
[0009] The outputs of different encoders are fused to obtain the medical named entity recognition result in the medical text to be identified.
[0010] As an improvement to the above solution, the step of generating multimodal input information based on the text information and the visual information includes:
[0011] The text information is preprocessed;
[0012] The preprocessed text information is encoded to generate a high-dimensional vector representation of the text;
[0013] The visual information is preprocessed;
[0014] Feature extraction is performed on the preprocessed visual information to generate a high-dimensional visual vector representation;
[0015] The multimodal input information is obtained based on the high-dimensional vector representation of the text and the high-dimensional vector representation of the vision.
[0016] As an improvement to the above scheme, the encoder includes a syntax encoder, a semantic encoder, and a domain encoder;
[0017] The syntax encoder is used to analyze the syntactic structure of the text and determine the syntactic position and relationship of each word;
[0018] The semantic encoder is used to understand the deeper meaning of text and to identify and classify medical named entities;
[0019] The domain encoder is used to extract medical domain knowledge from text.
[0020] As an improvement to the above scheme, the step of inputting the multimodal input information into multiple encoders of different types for encoding processing includes:
[0021] The multimodal input information is respectively input to the syntax encoder, the semantic encoder, and the domain encoder;
[0022] Within each encoder, the high-dimensional vector representation of the text is used as the initial key matrix and value matrix of the encoder, and the high-dimensional vector representation of the vision is used as the query matrix to generate a first context representation;
[0023] Independent contextual understanding is performed on the high-dimensional text vector and the high-dimensional visual vector representation between different encoders to generate a second contextual representation.
[0024] As an improvement to the above scheme, the step of performing independent contextual understanding on the high-dimensional text vector and the high-dimensional visual vector representation between different encoders to generate a second contextual representation includes:
[0025] Perform independent contextual understanding on the high-dimensional vector of the text to generate a text context representation;
[0026] Perform independent contextual understanding on the aforementioned high-dimensional visual vectors to generate visual contextual representations;
[0027] The text context representation and the visual context representation are aligned and integrated to generate a second context representation.
[0028] As an improvement to the above scheme, the step of fusing the outputs of different encoders to obtain the medical named entity recognition result in the medical text to be recognized includes:
[0029] The initial key and value matrices of any two encoders with shared attention mechanisms are concatenated and updated with the initial key and value matrices of a third encoder, respectively.
[0030] Concatenate the multi-head attention outputs of any two encoders and update the attention score of the third encoder;
[0031] The FFN outputs of the syntax encoder and the domain encoder are concatenated with the FFN output of the semantic encoder to obtain the medical named entity recognition result in the medical text to be recognized.
[0032] This invention also provides a medical named entity recognition device, comprising:
[0033] The acquisition module is used to acquire the medical text to be identified and generate visual information based on the text information of the medical text to be identified;
[0034] A generation module is used to generate multimodal input information based on the text information and the visual information;
[0035] The encoding module is used to input the multimodal input information into multiple encoders of different types for encoding processing;
[0036] The recognition module is used to fuse the outputs of different encoders to obtain the medical named entity recognition result in the medical text to be recognized.
[0037] This invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the medical named entity recognition method described above.
[0038] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the medical named entity recognition method described above.
[0039] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the medical named entity recognition method described above.
[0040] Compared to existing technologies, the beneficial effects of the medical named entity recognition method, apparatus, device, storage medium, and product provided by this invention are as follows: By acquiring the medical text to be recognized and generating visual information based on the textual information of the medical text; generating multimodal input information based on the textual and visual information; inputting the multimodal input information into multiple encoders of different types for encoding processing; and fusing the outputs of different encoders to obtain the medical named entity recognition result in the medical text to be recognized. This invention inputs a multimodal representation that integrates textual and visual information into multiple encoders of different types for processing, achieving information fusion between different modalities and between different encoders. This not only improves the model's recognition accuracy of named entities in medical text but also helps the model better adapt to the continuous updating of medical knowledge. Attached Figure Description
[0041] Figure 1 This is a flowchart illustrating a preferred embodiment of a medical named entity recognition method provided by the present invention;
[0042] Figure 2 This is a schematic diagram of a preferred embodiment of a medical named entity recognition device provided by the present invention;
[0043] Figure 3 This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. Detailed Implementation
[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0045] Please see Figure 1 , Figure 1 This is a flowchart illustrating a preferred embodiment of a medical named entity recognition method provided by the present invention. The medical named entity recognition method includes:
[0046] S1, acquire the medical text to be identified, and generate visual information based on the text information of the medical text to be identified;
[0047] S2, Generate multimodal input information based on the text information and the visual information;
[0048] S3, the multimodal input information is input into multiple encoders of different types for encoding processing;
[0049] S4. The outputs of different encoders are fused to obtain the medical named entity recognition result in the medical text to be recognized.
[0050] Specifically, this embodiment of the invention first acquires the medical text to be identified and generates corresponding visual information based on the textual information. The textual information may include sentences, paragraphs, or any form of natural language text. The textual information is cleaned and normalized through preprocessing steps (such as word segmentation, stop word removal, stemming, etc.) to facilitate the subsequent generation of corresponding image information. The visual information is preprocessed using image processing techniques (such as scaling, cropping, normalization, etc.) to extract useful features or reduce data dimensionality. The processed textual and visual information are then fused to generate multimodal input information. It should be noted that this can be achieved through simple concatenation, feature mapping, or more complex fusion strategies (such as attention mechanisms). Then, the multimodal input information is fed into multiple encoders of different types, each encoder encoding the received input information. This typically involves multi-layer neural networks, attention mechanisms, or other deep learning techniques to extract deep features from the input data. Finally, the encoded outputs are extracted from each encoder; these outputs may include feature vectors, hidden states, or any form of encoded representation. The outputs of different encoders are fused, which can include simple concatenation, weighted averaging, attention-weighted fusion, or more complex fusion networks (such as multilayer perceptrons, Transformers, etc.). The fused output is then fed into a classifier, regressor, or any other type of decision module to generate the final medical named entity recognition result.
[0051] The embodiments of the present invention input multimodal representations that integrate textual and visual information into multiple encoders of different types for processing, realizing information fusion between different modalities and between different encoders. This not only improves the model's recognition accuracy of named entities in medical texts, but also helps the model better adapt to the continuous updating of medical knowledge.
[0052] In another preferred embodiment, step S2, generating multimodal input information based on the text information and the visual information, includes:
[0053] S201, Preprocess the text information;
[0054] S202, Encode the preprocessed text information to generate a high-dimensional vector representation of the text;
[0055] S203, preprocess the visual information;
[0056] S204, extract features from the preprocessed visual information to generate a high-dimensional visual vector representation;
[0057] S205, the multimodal input information is obtained based on the high-dimensional vector representation of the text and the high-dimensional vector representation of the vision.
[0058] Specifically, this embodiment of the invention integrates textual and visual information to generate an initial representation. First, the textual data of the medical text to be identified undergoes conventional preprocessing, such as word segmentation and stop word removal. Then, it is encoded using a pre-trained PubMedBERT model to generate a high-dimensional vector representation of the text rich in medical semantics and linguistic information. Simultaneously, based on descriptive prompts derived from the text content, images closely related to the text semantics are generated using the OpenAIDALL-E model. These images undergo preprocessing, such as resizing and normalization, to adapt to the pre-trained CLIP model, thereby extracting a high-dimensional visual vector representation that integrates visual and linguistic features.
[0059] It should be noted that after the representations are generated, they need to be standardized to ensure that textual and visual information can be processed uniformly in the model. For example, in this embodiment of the invention, the maximum length of both textual and visual representations is uniformly set to 50. If the length of a text sequence is less than 50, it will be extended to the maximum length using zero-padding; for visual representations, if their length is less than 50, the maximum length is matched by repeating visual tokens. This standardization process not only ensures the consistency of data from different modalities but also facilitates subsequent fusion and analysis.
[0060] This invention provides a multimodal representation for the model that integrates textual and visual information. These representations contain rich linguistic, semantic, and visual information, providing a solid foundation for subsequent medical named entity recognition tasks. This meticulous processing flow demonstrates the depth and rigor of this proposal in multimodal data processing, laying an important foundation for improving the accuracy and robustness of medical named entity recognition.
[0061] In yet another preferred embodiment, the encoder includes a syntax encoder, a semantic encoder, and a domain encoder;
[0062] The syntax encoder is used to analyze the syntactic structure of the text and determine the syntactic position and relationship of each word;
[0063] The semantic encoder is used to understand the deeper meaning of text and to identify and classify medical named entities;
[0064] The domain encoder is used to extract medical domain knowledge from text.
[0065] Specifically, the encoders in this embodiment of the invention include three types: a syntax encoder, a semantic encoder, and a domain encoder, each modeling different linguistic features. The syntax encoder analyzes the syntactic structure of the text, using a Transformer architecture. The input is the word latent data of the text, and the output is a representation of the syntactic position and structure of each word. The semantic encoder understands the deeper meaning of the text, identifying and classifying medical named entities. It is also based on the Transformer architecture but focuses on identifying and classifying named entities such as drugs and diseases. The domain encoder extracts medical domain knowledge from the text, such as specific medical terms and concepts like disease, treatment, DNA, and RNA. It enhances the model's ability to understand and interpret specific medical language. It should be noted that each encoder is based on the Transformer architecture, capable of handling multimodal inputs and internally implementing information fusion between modalities.
[0066] This invention integrates grammar, semantics, and domain knowledge. This comprehensive approach can better understand the complexity of medical texts, contributing to improved accuracy and reliability of recognition. Furthermore, this invention allows for the further integration of other modalities, such as temporal data or structured information, exhibiting excellent scalability and adaptability to future technological developments.
[0067] In another preferred embodiment, step S3 involves inputting the multimodal input information to multiple encoders of different types for encoding processing, including:
[0068] S301, the multimodal input information is input to the syntax encoder, the semantic encoder and the domain encoder respectively;
[0069] S302, within each encoder, the high-dimensional vector representation of the text is used as the initial key matrix and value matrix of the encoder, and the high-dimensional vector representation of the vision is used as the query matrix to generate a first context representation;
[0070] S303, between different encoders, independent contextual understanding is performed on the text high-dimensional vector and the visual high-dimensional vector representation to generate a second contextual representation.
[0071] Specifically, in this embodiment of the invention, multimodal input information is input to the syntax encoder, semantic encoder, and domain encoder respectively, so that textual and visual information are effectively integrated within each encoder. This embodiment employs an input integration method within each encoder, directly fusing textual and visual information within each encoder. For example, text embeddings generated by PubMedBERT are used as the initial key and value matrices for all encoders (syntax, semantic, and domain), while image embeddings generated by CLIP are used as the query matrix. This design allows the model to associate textual and visual information at the first moment, assess the correlation between them, and generate a unified first contextual representation containing multimodal information within each encoder. This architecture supports direct intermodal interaction, thereby enhancing the model's comprehensive understanding of the input data. A multimodal Transformer architecture is adopted between different encoders. In the multimodal Transformer architecture, the context of each single modality is first understood, and then the multimodal information is aligned and integrated through multi-view cross-integration to generate a second contextual representation. This architecture first processes textual and image information separately, and then integrates the information from the two modalities through a specific cross-integration strategy. This approach allows the model to achieve intermodal information exchange and enhancement through carefully designed cross-integration steps while maintaining modal independence.
[0072] It should be noted that, in this embodiment of the invention, for cases where many input medical texts do not contain medical named entities, the domain encoder, originally focused on a domain-specific task, can be updated to an image-to-text module, and then integrated with the outputs of the other two encoders. This architecture improves the model's ability to recognize medical named entities by combining the image-to-text generation task with the tasks of other encoders. This strategy not only enhances the model's understanding of text but also strengthens the model's utilization of visual information through the image generation task.
[0073] In yet another preferred embodiment, the step of performing independent contextual understanding on the text high-dimensional vector and the visual high-dimensional vector representation between different encoders to generate a second contextual representation includes:
[0074] Perform independent contextual understanding on the high-dimensional vector of the text to generate a text context representation;
[0075] Perform independent contextual understanding on the aforementioned high-dimensional visual vectors to generate visual contextual representations;
[0076] The text context representation and the visual context representation are aligned and integrated to generate a second context representation.
[0077] Specifically, this embodiment of the invention employs a multimodal Transformer architecture across different encoders. In this architecture, the context of each single modality is first understood independently: the high-dimensional text vector is understood independently to generate a text context representation; and the high-dimensional visual vector is understood independently to generate a visual context representation. Then, the text and visual context representations are aligned and integrated through multi-view cross-integration to generate a second context representation.
[0078] In another preferred embodiment, step S4 involves fusing the outputs of different encoders to obtain the medical named entity recognition result in the medical text to be recognized, including:
[0079] S401, the initial key matrix and value matrix of any two encoders with shared attention mechanism are concatenated and updated with the initial key matrix and value matrix of the third encoder respectively;
[0080] S402, concatenates the multi-head attention outputs of any two encoders and updates the attention score of the third encoder;
[0081] S403, the FFN outputs of the syntax encoder and the domain encoder are concatenated with the FFN output of the semantic encoder to obtain the medical named entity recognition result in the medical text to be recognized.
[0082] Specifically, this embodiment of the invention employs a multi-perspective cross-integration method to further fuse key parts of information from different encoders, aiming to promote knowledge exchange and complementarity between different modalities and perspectives. The key-value cross-integration method is used to concatenate and update the initial key and value matrices of any two encoders sharing an attention mechanism with the initial key and value matrices of a third encoder. It should be noted that the key-value cross-integration method involves concatenating and updating the key(k) and value(v) matrices of the shared attention mechanism between any two encoders with the k and v matrices of the third encoder.
[0083] Taking the semantic encoder as an example, its original k and v matrices are updated by concatenating the k and v matrices of the syntax encoder and the domain encoder. Then, the updated output of the attention mechanism is fed into a dense layer to adjust the tensor dimension, thereby achieving information fusion and feature enhancement between different encoders.
[0084]
[0085] Here, sem represents the semantic encoder; ksem represents the key of the semantic encoder; vsem represents the value of the semantic encoder; syn represents the syntax encoder; and dom represents the domain encoder.
[0086] An attention crossover method is employed to concatenate the multi-head attention outputs of any two encoders and update the attention score of a third encoder. The core of this method is sharing the multi-head attention outputs across different encoders and updating the current encoder's knowledge based on the attention scores of the other two encoders. For example, the attention score of the semantic encoder is updated by concatenating the multi-head attention outputs of the grammar encoder and the domain encoder. This approach allows the model to consider the contextual information from other encoders when updating information within a single encoder, thus achieving a more comprehensive contextual understanding.
[0087] MultiHead semnew =concat(MultiGead) syn MultiHead dom ).
[0088] A feedforward crossover method is used to concatenate the outputs of the FFN sublayers of the syntax encoder and domain encoder with the FFN output of the semantic encoder. After the feedforward neural network (FFN) layer in the Transformer encoder, information is shared among the three encoders. The feedforward crossover method involves sharing the outputs of the feedforward network layer among different encoders. In the Transformer encoder, the layer normalized formula can be expressed as:
[0089] LayerNorm(x+FFN(x));
[0090] Where x represents the input of the FFN layer.
[0091] In feedforward cross-integration, the FFN sublayer outputs of the syntax encoder and the domain encoder are shared and concatenated with the FFN output of the semantic encoder to compute the layer normalization of the semantic encoder:
[0092] LayerNorm(x sen +concat(FFN(x syn ;FFN(x dom ))));
[0093] Here, concat represents the concatenation operation.
[0094] In this way, the outputs of different encoders can be effectively fused before the layer normalization step, which enhances the model's ability to comprehensively process information from different modes.
[0095] The model evaluation metrics used in this embodiment of the invention are accuracy and Macro-F1. Accuracy represents the ratio of the number of samples correctly classified by the model to the total number of samples for a given dataset. The formula for F1 is as follows:
[0096] F1 = 2PR / (P+R);
[0097] Wherein, precision P represents the proportion of correct samples among all predicted positive samples; recall R represents the proportion of all positive samples that were predicted as positive.
[0098] Macro-F1 is the mean F1 score across all classes. Therefore, Macro-F1 is used to comprehensively evaluate the model; a higher Macro-F1 score indicates better performance.
[0099] This invention, through a multimodal and multi-perspective approach, effectively improves the accuracy and robustness of medical named entity recognition and provides a new direction for future research in the field of medical informatics. This invention enhances the capabilities of medical named entity recognition by integrating not only textual information but also data from other modalities such as images, enriching the machine's understanding of the context. This multimodal approach mimics how humans integrate multiple sensory information when understanding complex situations, helping to resolve the inherent semantic ambiguity in textual data. Furthermore, this invention explores how to seamlessly integrate information from multiple modalities to overcome the limitations of single-modal methods. By introducing key technologies such as multimodal input generation, multimodal encoder construction, and multi-faceted integration, this invention aims to promote the development of medical informatics and provide new perspectives and methods for building more effective and accurate medical named entity recognition systems. This not only improves the model's accuracy in recognizing named entities in medical texts but also helps the model better adapt to the continuous updating of medical knowledge, thus playing a greater role in practical applications.
[0100] Accordingly, the present invention also provides a medical named entity recognition device, which can implement all the processes of the medical named entity recognition method in the above embodiments.
[0101] Please see Figure 2 , Figure 2 This is a schematic diagram of a preferred embodiment of a medical named entity recognition device provided by the present invention. The medical named entity recognition device includes:
[0102] The acquisition module 201 is used to acquire the medical text to be identified and generate visual information based on the text information of the medical text to be identified;
[0103] The generation module 202 is used to generate multimodal input information based on the text information and the visual information;
[0104] Encoding module 203 is used to input the multimodal input information into multiple encoders of different types for encoding processing;
[0105] The recognition module 204 is used to fuse the outputs of different encoders to obtain the medical named entity recognition result in the medical text to be recognized.
[0106] Preferably, the generation module 202 is specifically used for:
[0107] The text information is preprocessed;
[0108] The preprocessed text information is encoded to generate a high-dimensional vector representation of the text;
[0109] The visual information is preprocessed;
[0110] Feature extraction is performed on the preprocessed visual information to generate a high-dimensional visual vector representation;
[0111] The multimodal input information is obtained based on the high-dimensional vector representation of the text and the high-dimensional vector representation of the vision.
[0112] Preferably, the encoder includes a syntax encoder, a semantic encoder, and a domain encoder;
[0113] The syntax encoder is used to analyze the syntactic structure of the text and determine the syntactic position and relationship of each word;
[0114] The semantic encoder is used to understand the deeper meaning of text and to identify and classify medical named entities;
[0115] The domain encoder is used to extract medical domain knowledge from text.
[0116] Preferably, the encoding module 203 is specifically used for:
[0117] The multimodal input information is respectively input to the syntax encoder, the semantic encoder, and the domain encoder;
[0118] Within each encoder, the high-dimensional vector representation of the text is used as the initial key matrix and value matrix of the encoder, and the high-dimensional vector representation of the vision is used as the query matrix to generate a first context representation;
[0119] Independent contextual understanding is performed on the high-dimensional text vector and the high-dimensional visual vector representation between different encoders to generate a second contextual representation.
[0120] Preferably, the step of performing independent contextual understanding on the high-dimensional text vector and the high-dimensional visual vector representation between different encoders to generate a second contextual representation includes:
[0121] Perform independent contextual understanding on the high-dimensional vector of the text to generate a text context representation;
[0122] Perform independent contextual understanding on the aforementioned high-dimensional visual vectors to generate visual contextual representations;
[0123] The text context representation and the visual context representation are aligned and integrated to generate a second context representation.
[0124] Preferably, the identification module 204 is specifically used for:
[0125] The initial key and value matrices of any two encoders with shared attention mechanisms are concatenated and updated with the initial key and value matrices of a third encoder, respectively.
[0126] Concatenate the multi-head attention outputs of any two encoders and update the attention score of the third encoder;
[0127] The FFN outputs of the syntax encoder and the domain encoder are concatenated with the FFN output of the semantic encoder to obtain the medical named entity recognition result in the medical text to be identified.
[0128] In specific implementation, the working principle, control process and technical effects of the medical named entity recognition device provided in the embodiments of the present invention are the same as those of the medical named entity recognition method in the above embodiments, and will not be repeated here.
[0129] Please see Figure 3 , Figure 3 This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 301, a memory 302, and a computer program stored in the memory 302 and configured to be executed by the processor 301. When the processor 301 executes the computer program, it implements the medical named entity recognition method described in any of the above embodiments.
[0130] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 302 and executed by the processor 301 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.
[0131] The processor 301 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 301 can be any conventional processor. The processor 301 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.
[0132] The memory 302 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., while the data storage area can store related data, etc. Furthermore, the memory 302 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital card (SD), and a flash card, or it can be other volatile solid-state storage devices.
[0133] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 3 The structural diagram is merely an example of the terminal device described above and does not constitute a limitation on the terminal device described above. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components.
[0134] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the medical named entity recognition method described in any of the above embodiments.
[0135] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the medical named entity recognition method described in any of the above embodiments.
[0136] This invention provides a method, apparatus, device, storage medium, and product for medical named entity recognition. The method involves acquiring medical text to be recognized and generating visual information based on the textual information of the medical text; generating multimodal input information based on the textual and visual information; inputting the multimodal input information into multiple encoders of different types for encoding processing; and fusing the outputs of the different encoders to obtain the medical named entity recognition result in the medical text to be recognized. This invention integrates textual and visual information into a multimodal representation, inputting it into multiple encoders of different types for processing. This achieves information fusion between different modalities and between different encoders, which not only improves the model's recognition accuracy of named entities in medical text but also helps the model better adapt to the continuous updating of medical knowledge.
[0137] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0138] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A medical named entity recognition method, characterized in that, include: Obtain the medical text to be identified, and generate visual information based on the text information of the medical text to be identified; Based on the text information and the visual information, multimodal input information is generated; The multimodal input information is input into multiple encoders of different types for encoding processing. The outputs of different encoders are fused to obtain the medical named entity recognition result in the medical text to be identified.
2. The medical named entity recognition method as described in claim 1, characterized in that, The step of generating multimodal input information based on the text information and the visual information includes: The text information is preprocessed; The preprocessed text information is encoded to generate a high-dimensional vector representation of the text; The visual information is preprocessed; Feature extraction is performed on the preprocessed visual information to generate a high-dimensional visual vector representation; The multimodal input information is obtained based on the high-dimensional vector representation of the text and the high-dimensional vector representation of the vision.
3. The medical named entity recognition method as described in claim 2, characterized in that, The encoders include a syntax encoder, a semantic encoder, and a domain encoder; The syntax encoder is used to analyze the syntactic structure of the text and determine the syntactic position and relationship of each word; The semantic encoder is used to understand the deeper meaning of text and to identify and classify medical named entities; The domain encoder is used to extract medical domain knowledge from text.
4. The medical named entity recognition method as described in claim 3, characterized in that, The step of inputting the multimodal input information into multiple encoders of different types for encoding processing includes: The multimodal input information is respectively input to the syntax encoder, the semantic encoder, and the domain encoder; Within each encoder, the high-dimensional vector representation of the text is used as the initial key matrix and value matrix of the encoder, and the high-dimensional vector representation of the vision is used as the query matrix to generate a first context representation; Independent contextual understanding is performed on the high-dimensional text vector and the high-dimensional visual vector representation between different encoders to generate a second contextual representation.
5. The medical named entity recognition method as described in claim 4, characterized in that, The process of performing independent contextual understanding on the high-dimensional text vector and the high-dimensional visual vector representation across different encoders to generate a second contextual representation includes: Perform independent contextual understanding on the high-dimensional vector of the text to generate a text context representation; Perform independent contextual understanding on the aforementioned high-dimensional visual vectors to generate visual contextual representations; The text context representation and the visual context representation are aligned and integrated to generate a second context representation.
6. The medical named entity recognition method as described in claim 5, characterized in that, The process of fusing the outputs of different encoders to obtain the medical named entity recognition result in the medical text to be recognized includes: The initial key and value matrices of any two encoders with shared attention mechanisms are concatenated and updated with the initial key and value matrices of a third encoder, respectively. Concatenate the multi-head attention outputs of any two encoders and update the attention score of the third encoder; The outputs of the feedforward neural networks (FFN) of the syntax encoder and the domain encoder are concatenated with the output of the semantic encoder to obtain the medical named entity recognition result in the medical text to be recognized.
7. A medical named entity recognition device, characterized in that, include: The acquisition module is used to acquire the medical text to be identified and generate visual information based on the text information of the medical text to be identified; A generation module is used to generate multimodal input information based on the text information and the visual information; The encoding module is used to input the multimodal input information into multiple encoders of different types for encoding processing; The recognition module is used to fuse the outputs of different encoders to obtain the medical named entity recognition result in the medical text to be recognized.
8. A terminal device, characterized in that, The device includes a processor and a memory, wherein the memory stores a computer program and the computer program is configured to be executed by the processor, wherein the processor, when executing the computer program, implements the medical named entity recognition method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the medical named entity recognition method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the medical named entity recognition method as described in any one of claims 1 to 6.