A method, device and equipment for identifying attributes of an entity tag, and a storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SICHUAN JIUYUAN YINHAI SOFTWARE
- Filing Date
- 2023-02-09
- Publication Date
- 2026-07-21
Smart Images
Figure CN116069932B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, and in particular to a method, apparatus, device and storage medium for entity tag attribute recognition. Background Technology
[0002] Entity labeling and attribute recognition is an important task in text mining, primarily providing crucial support for tasks such as entity linking, relation extraction, and event extraction in information extraction techniques. It also assists downstream applications like question-answering systems and recommendation systems. Specifically, entity labeling and attribute recognition is defined as: given candidate entities and their context, predicting the possible label categories of the candidate entities. Taking the medical field as an example, medical texts such as doctor-patient dialogues, present medical history, past medical history, and discharge summaries contain many entities describing body parts and patient conditions, such as: left breast, right lung, L3 vertebra, vomiting, diarrhea, and fever. To provide a more comprehensive and accurate description of the patient's condition, it is necessary to identify the labels of these entities and determine their attributes.
[0003] Currently, in the field of entity label attribute recognition, most existing technologies rely on rule-based methods. These methods involve manually designing rule features to obtain sample representations, and then using classic classification algorithms to determine the entity label type. However, these hand-designed features are often deeply coupled with the dataset or obtained through external general-purpose tools, resulting in significant errors when applied to specific text domains. Alternatively, the model structure of a masked language model (e.g., the BERT model) can be modified to enable it to perform entity label attribute recognition tasks. However, such models are difficult to build, and since entity label attribute recognition is a downstream task, its incompatibility with the original pre-training structure of the masked language model leads to poor training results, which in turn affects the performance of entity label attribute recognition. Summary of the Invention
[0004] In view of this, the purpose of this application is to provide a method, apparatus, device and storage medium for entity label attribute recognition. By constructing an attention masking matrix, the entity label attribute recognition task is transformed into a masking word prediction task. In this way, the downstream task can be integrated with the upstream pre-training task, so that a better entity label attribute recognition effect can be achieved based on a simple masking language model.
[0005] This application provides a method for entity tag attribute recognition, the method comprising:
[0006] Construct the text vector and attention masking matrix of the text containing the target entity; wherein, the values of the matrix elements in the attention masking matrix are used to characterize the mutual visibility between the context of the text containing the target entity and the mask of the target entity;
[0007] The text vector and the attention masking matrix are input into the masking language model trained by the pre-training task to obtain the masking vector corresponding to the target entity output by the masking language model.
[0008] The label attribute category of the target entity is determined based on the mask vector.
[0009] Furthermore, the text vector includes: a text segmentation vector, a positional encoding vector, and a semantic encoding vector; constructing the text vector of the text containing the target entity includes:
[0010] The text containing the target entity is split into multiple split units to form a split text sequence;
[0011] Add the mask identifier corresponding to the mask of the target entity to the split text sequence to obtain the text split vector;
[0012] Based on the position of each splitting unit in the text splitting vector corresponding to the target entity in the text and the position of the target entity corresponding to the mask identifier in the text, a position encoding vector is constructed;
[0013] The semantic encoding corresponding to each splitting unit in the text splitting vector and the mask identifier is set to a preset value to obtain a semantic encoding vector.
[0014] Furthermore, an attention masking matrix is constructed for the text containing the target entity, including:
[0015] An initial attention masking matrix is constructed based on each splitting unit and mask identifier in the text splitting vector; wherein each position in the row and column of the initial attention masking matrix represents each splitting unit and mask identifier;
[0016] Based on the visibility between each segmentation unit corresponding to the text containing the target entity, and the visibility between each segmentation unit corresponding to the text containing the target entity and the mask identifier, the value of each matrix element in the initial attention masking matrix is determined to obtain the attention masking matrix; wherein, each segmentation unit corresponding to the text containing the target entity is visible to each other; the mask identifier is invisible to each segmentation unit corresponding to the text containing the target entity; except for the target entity, each segmentation unit corresponding to the text containing the target entity is invisible to the mask identifier; the target entity is visible to the mask identifier.
[0017] Furthermore, the step of inputting the text vector and the attention masking matrix into a masking language model trained through a pre-training task to obtain the masking vector corresponding to the target entity output by the masking language model includes:
[0018] The text vector and the attention masking matrix are input into a masking language model trained by a pre-training task. The masking language model encodes each splitting unit and mask identifier in the text splitting vector.
[0019] The vector obtained by encoding the mask identifier using the mask language model is used as the mask vector corresponding to the target entity.
[0020] Furthermore, when the method is applied to entity label attribute recognition in the medical field, determining the label attribute category of the target entity based on the mask vector includes:
[0021] The mask vector is input into the linear output layer to obtain the label attribute categories output by the linear output layer; wherein, the label attribute categories include negative, positive, unlabeled, and others; negative indicates that it is related to symptoms and diseases that the patient does not have; positive indicates that it is related to symptoms and diseases that the patient has or may have in the future; unlabeled indicates medical entities that are independent of the patient's current state and have no actual diagnostic significance; others indicates other medical entities that cannot be clearly inferred to be negative, positive, or unlabeled.
[0022] Furthermore, the mask language model is obtained through the following steps via a pre-training task:
[0023] A predetermined proportion of words are selected from the training text and masked to obtain the training text vector;
[0024] Input the training text vector into the initial mask language model to obtain the training mask vector corresponding to each mask in the training text;
[0025] Based on the training mask vector corresponding to each mask, determine the probability distribution of the mask with respect to the vocabulary;
[0026] Based on the probability distribution of the mask with respect to the vocabulary and the words corresponding to the mask in the training text, the initial mask language model is trained to obtain the mask language model.
[0027] This application also provides an apparatus for entity tag attribute recognition, the apparatus comprising:
[0028] A construction module is used to construct the text vector and attention masking matrix of the text containing the target entity; wherein, the values of the matrix elements in the attention masking matrix are used to characterize the mutual visibility between the context of the text containing the target entity and the mask of the target entity;
[0029] The input module is used to input the text vector and the attention masking matrix into the masking language model trained by the pre-training task, and obtain the mask vector corresponding to the target entity output by the masking language model;
[0030] The determination module is used to determine the label attribute category of the target entity based on the mask vector.
[0031] Furthermore, the text vector includes: a text splitting vector, a positional encoding vector, and a semantic encoding vector; when constructing the text vector of the text containing the target entity, the construction module is used to:
[0032] The text containing the target entity is split into multiple split units to form a split text sequence;
[0033] Add the mask identifier corresponding to the mask of the target entity to the split text sequence to obtain the text split vector;
[0034] Based on the position of each splitting unit in the text splitting vector corresponding to the target entity in the text and the position of the target entity corresponding to the mask identifier in the text, a position encoding vector is constructed;
[0035] The semantic encoding corresponding to each splitting unit in the text splitting vector and the mask identifier is set to a preset value to obtain a semantic encoding vector.
[0036] This application embodiment also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the entity tag attribute recognition method described above are performed.
[0037] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the entity tag attribute recognition method described above.
[0038] This application provides a method, apparatus, device, and storage medium for entity tag attribute recognition, comprising: constructing a text vector of the text containing the target entity and an attention masking matrix; wherein, the values of the matrix elements in the attention masking matrix are used to characterize the context of the text containing the target entity and the mutual visibility between the masks of the target entity; inputting the text vector and the attention masking matrix into a masking language model trained by a pre-training task to obtain a masking vector corresponding to the target entity output by the masking language model; and determining the tag attribute category of the target entity based on the masking vector.
[0039] In this way, by constructing an attention masking matrix, the entity label attribute recognition task is transformed into a masking word prediction task. This allows the downstream task to be integrated with the upstream pre-training task, enabling a simple masking language model to achieve better entity label attribute recognition results.
[0040] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0041] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 A flowchart illustrating a method for entity tag attribute recognition provided in an embodiment of this application is shown;
[0043] Figure 2 A schematic diagram of a method for entity tag attribute recognition provided in an embodiment of this application is shown;
[0044] Figure 3 This illustration shows a schematic diagram of an attention masking matrix provided in an embodiment of this application;
[0045] Figure 4 This illustration shows the effect of an entity label attribute annotation provided in an embodiment of this application;
[0046] Figure 5 A schematic diagram of the structure of an entity tag attribute recognition device provided in an embodiment of this application is shown;
[0047] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.
[0049] Research has shown that entity labeling and attribute recognition is a crucial task in text mining, primarily providing essential support for entity linking, relation extraction, and event extraction in information extraction techniques. It also assists downstream applications such as question-answering systems and recommendation systems. Specifically, entity labeling and attribute recognition is defined as: given candidate entities and their context, predicting the possible label categories of the candidate entities. Taking the medical field as an example, medical texts such as doctor-patient dialogues, present medical history, past medical history, and discharge summaries contain many entities describing body parts and patient conditions, such as: left breast, right lung, L3 vertebra, vomiting, diarrhea, and fever. To provide a more comprehensive and accurate description of the patient's condition, it is necessary to identify the labels of these entities and determine their attributes.
[0050] Currently, in the field of entity label attribute recognition, most existing technologies rely on rule-based methods. These methods involve manually designing rule features to obtain sample representations, and then using classic classification algorithms to determine the entity label type. However, these hand-designed features are often deeply coupled with the dataset or obtained through external general-purpose tools, resulting in significant errors when applied to specific text domains. Alternatively, the model structure of a masked language model (e.g., the BERT model) can be modified to enable it to perform entity label attribute recognition tasks. However, such models are difficult to build, and since entity label attribute recognition is a downstream task, its incompatibility with the original pre-training structure of the masked language model leads to poor training results, which in turn affects the performance of entity label attribute recognition.
[0051] Based on this, this application provides a method for entity label attribute recognition, which transforms the entity label attribute recognition task into a masking word prediction task by constructing an attention masking matrix. This allows the downstream task to be integrated with the upstream pre-training task, enabling better entity label attribute recognition performance based on a simple masking language model.
[0052] Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating a method for recognizing entity tag attributes provided in an embodiment of this application. Figure 2 This is a schematic diagram illustrating a method for recognizing entity label attributes provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the method includes:
[0053] S101. Construct the text vector and attention masking matrix of the text containing the target entity.
[0054] In one possible implementation, the text vector includes: a text splitting vector, a positional encoding vector, and a semantic encoding vector; the construction of the text vector of the target entity in step S101 may include:
[0055] The text containing the target entity is split into multiple split units to form a split text sequence; a mask identifier corresponding to the mask of the target entity is added to the split text sequence to obtain the text split vector; a position encoding vector is constructed based on the position of each split unit in the text split vector corresponding to the target entity in the text containing the target entity and the position of the target entity in the text containing the mask identifier; the semantic encoding corresponding to each split unit in the text split vector and the mask identifier is set to a preset value to obtain a semantic encoding vector.
[0056] Please see Figure 2 Taking text 1 "My thigh hurts" as an example, with "thigh" as the target entity, the text 1 containing the target entity is first split according to the model's dictionary / vocabulary to obtain multiple split units to form a split text sequence. Here, [CLS] represents the task identifier and [SEP] represents the separator or end character. The mask identifier [MASK] corresponding to the mask of the target entity is added to the split text sequence to obtain the text split vector tokens. For example, the same number of special characters [MASK] as the number of target entities to be identified can be added to the end of the split text sequence obtained by standard splitting.
[0057] Secondly, based on the position of each segmentation unit in the text segmentation vector corresponding to the text 1 and the position of the target entity "thigh" of the mask identifier in the text 1, a position encoding vector positionids is constructed. For example, if the position ID of the mask identifier [MASK] is the same as the position ID of the first character of the target entity it represents, then the position encoding vector positionids is [0 1 2 3 4 5 2] (the position of [MASK] is only related to the position ID, and is not related to whether it is placed before, after, or in the middle of the text. For ease of processing, it is placed at the end of the text for example).
[0058] Finally, the semantic encoding vector token type ids has no obvious meaning in the entity tag attribute recognition task of this application embodiment. Therefore, it is only necessary to set the semantic encoding corresponding to each segmentation unit and mask identifier in the text segmentation vector to a preset value to obtain the semantic encoding vector to cooperate with the pre-training task of the mask language model. For example, Figure 2 Each semantic code can be 0.
[0059] The values of the matrix elements in the attention mask matrix are used to characterize the mutual visibility between the context of the text containing the target entity and the mask of the target entity.
[0060] Please see Figure 3 , Figure 3 This is a schematic diagram of an attention masking matrix provided in an embodiment of this application. In another possible implementation, the construction of the attention masking matrix of the text containing the target entity in step S101 may include:
[0061] Based on each splitting unit and mask identifier in the text splitting vector, an initial attention masking matrix is constructed; wherein each position in the row and column of the initial attention masking matrix represents each splitting unit and mask identifier in turn; based on the visibility between each splitting unit corresponding to the text where the target entity is located, and the visibility between each splitting unit corresponding to the text where the target entity is located and the mask identifier, the value of each matrix element in the initial attention masking matrix is determined, and the attention masking matrix is obtained.
[0062] In this context, each segmentation unit corresponding to the text containing the target entity is visible to each other; the mask identifier is invisible to each segmentation unit corresponding to the text containing the target entity; except for the target entity, each segmentation unit corresponding to the text containing the target entity is invisible to the mask identifier; and the target entity is visible to the mask identifier.
[0063] like Figure 3As shown, each position on the rows and columns of the attention masking matrix represents each splitting unit {t k …t k+m} obtained by splitting the text where the target entity is located and the mask identifier [E_MASK]. The white squares represent visible, and the black squares represent invisible. Correspondingly, it can be represented that when the value of the matrix element is 1, it is visible, and when the value of the matrix element is 0, it is invisible. Corresponding to the above example, when the input text is "[CLS] My thigh hurts [SEP] [MASK]", the characters in "[CLS] My thigh hurts [SEP]" can be visible to each other, but [MASK] is invisible, while [MASK] can see the entity "thigh" it represents. That is, for the two matrix elements in the row where [MASK] is located and the columns where "big" and "thigh" are located in the attention masking matrix, both are represented as visible.
[0064] S102. Input the text vector and the attention masking matrix into a masked language model trained by a pre-trained task, and obtain the masked vector corresponding to the target entity output by the masked language model.
[0065] In specific implementation, step S102 may include: inputting the text vector and the attention masking matrix into a masked language model trained by a pre-trained task, and the masked language model encodes each splitting unit and the mask identifier in the text splitting vector respectively; using the vector obtained by the masked language model encoding the mask identifier as the masked vector corresponding to the target entity.
[0066] In this step, the text splitting vector, the position encoding vector, the semantic encoding vector, and the attention masking matrix can be input into the masked language model to obtain the encoded vector and extract the vector representing [MASK], that is, the masked vector h corresponding to the target entity [MASK] .
[0067] Here, the masked language model can be a bert model; the masked language model is trained by a pre-trained task through the following steps:
[0068] Select a predetermined proportion of words in the training text for masking to obtain a training text vector; input the training text vector into an initial masked language model to obtain the training masked vector corresponding to each mask in the training text; based on the training masked vector corresponding to each mask, determine the probability distribution of the mask with respect to the vocabulary; based on the probability distribution of the mask with respect to the vocabulary and the word corresponding to the mask in the training text, train the initial masked language model to obtain the masked language model.
[0069] Here, the pre-training task of the masked language model is a "cloze test" task, that is, masking some words in the text and then predicting the words through the model. Specifically, a certain number of words (usually 15% of the sentence length) are randomly selected from the input text and replaced with the [MASK] label to obtain the model input. After the input is passed through an encoder such as BERT, a mask vector is obtained. Finally, the output layer obtains the probability distribution of the mask with respect to the vocabulary. The word with the highest probability in the vocabulary is the word predicted by the masked language model. Then, based on the word predicted by the masked language model and the actual words corresponding to the mask in the training text, a loss function can be designed and the initial masked language model can be iteratively trained to obtain the masked language model.
[0070] S103. Determine the tag attribute category of the target entity based on the mask vector.
[0071] In a specific implementation, step S103 may include: inputting the mask vector into the linear output layer to obtain the label attribute category output by the linear output layer.
[0072] When the method is applied to entity label attribute recognition in the medical field, the label attribute categories include negative, positive, unlabeled, and others; negative indicates that it is related to symptoms and diseases that the patient does not have; positive indicates that it is related to symptoms and diseases that the patient has or may have in the future, such as a doctor's diagnosis; unlabeled indicates that medical entities that are independent of the patient's current state and have no actual diagnostic significance generally appear in the general knowledge explained by the doctor, and also include disease names that appear in examination items and drug names; others indicate other medical entities that cannot be clearly inferred to be negative, positive, or unlabeled, generally referring to situations where the patient does not know or has not answered, including unclear answers and other situations that are difficult to infer.
[0073] Specifically, when the method is applied to entity label attribute recognition in the medical field, this embodiment uses online consultation data as a training set for model fine-tuning. Given relevant medical named entities, medical personnel annotate the attributes of the entity labels. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This diagram illustrates the effect of entity label attribute annotation provided in an embodiment of this application. In one experiment, using the average Macro-F1 score as the evaluation metric, and employing the pre-trained, non-medical-related model Lawformer, an average F1 score of 0.771 was achieved for a single model. This demonstrates that the method provided in this embodiment has good entity label attribute recognition performance and strong versatility, allowing for simultaneous transfer to entity label attribute recognition tasks in other fields.
[0074] This application provides a method for entity label attribute recognition, comprising: constructing a text vector of the text containing the target entity and an attention masking matrix; wherein, the values of the matrix elements in the attention masking matrix are used to characterize the context of the text containing the target entity and the mutual visibility between the mask of the target entity; inputting the text vector and the attention masking matrix into a masking language model trained by a pre-training task to obtain a masking vector corresponding to the target entity output by the masking language model; and determining the label attribute category of the target entity based on the masking vector.
[0075] In this way, by constructing an attention masking matrix, the entity label attribute recognition task is transformed into a masked word prediction task. This allows the downstream task to be integrated with the upstream pre-training task, enabling a simple masked language model to achieve better entity label attribute recognition performance. Specifically, it is compatible with the pre-training task structure of masked language models, such as most BERT-like models, offering a simpler model structure and better training results. Compared to manually identifying entity labels and attributes on massive amounts of medical text, this method saves significant manual work and improves efficiency. It provides crucial support for tasks such as entity linking, relation extraction, and event extraction in medical information extraction technology.
[0076] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an entity tag attribute recognition device provided in an embodiment of this application. Figure 5 As shown, the device 500 includes:
[0077] The construction module 510 is used to construct the text vector and attention masking matrix of the text containing the target entity; wherein, the values of the matrix elements in the attention masking matrix are used to characterize the mutual visibility between the context of the text containing the target entity and the mask of the target entity.
[0078] The input module 520 is used to input the text vector and the attention masking matrix into the masking language model trained by the pre-training task, so as to obtain the masking vector corresponding to the target entity output by the masking language model.
[0079] The determination module 530 is used to determine the tag attribute category of the target entity based on the mask vector.
[0080] Furthermore, the text vector includes: a text splitting vector, a positional encoding vector, and a semantic encoding vector; when constructing the text vector of the text containing the target entity, the construction module 510 is used to:
[0081] The text containing the target entity is split into multiple split units to form a split text sequence;
[0082] Add the mask identifier corresponding to the mask of the target entity to the split text sequence to obtain the text split vector;
[0083] Based on the position of each splitting unit in the text splitting vector corresponding to the target entity in the text and the position of the target entity corresponding to the mask identifier in the text, a position encoding vector is constructed;
[0084] The semantic encoding corresponding to each splitting unit in the text splitting vector and the mask identifier is set to a preset value to obtain a semantic encoding vector.
[0085] Furthermore, when constructing the attention masking matrix of the text containing the target entity, the construction module 510 is used to:
[0086] An initial attention masking matrix is constructed based on each splitting unit and mask identifier in the text splitting vector; wherein each position in the row and column of the initial attention masking matrix represents each splitting unit and mask identifier;
[0087] Based on the visibility between each segmentation unit corresponding to the text containing the target entity, and the visibility between each segmentation unit corresponding to the text containing the target entity and the mask identifier, the value of each matrix element in the initial attention masking matrix is determined to obtain the attention masking matrix; wherein, each segmentation unit corresponding to the text containing the target entity is visible to each other; the mask identifier is invisible to each segmentation unit corresponding to the text containing the target entity; except for the target entity, each segmentation unit corresponding to the text containing the target entity is invisible to the mask identifier; the target entity is visible to the mask identifier.
[0088] Furthermore, when the input module 520 is used to input the text vector and the attention masking matrix into the masking language model trained by the pre-training task to obtain the masking vector corresponding to the target entity output by the masking language model, the input module 520 is used to:
[0089] The text vector and the attention masking matrix are input into a masking language model trained by a pre-training task. The masking language model encodes each splitting unit and mask identifier in the text splitting vector.
[0090] The vector obtained by encoding the mask identifier using the mask language model is used as the mask vector corresponding to the target entity.
[0091] Furthermore, when the method is applied to entity label attribute recognition in the medical field, the determining module 530, when determining the label attribute category of the target entity based on the mask vector, is used to:
[0092] The mask vector is input into the linear output layer to obtain the label attribute categories output by the linear output layer; wherein, the label attribute categories include negative, positive, unlabeled, and others; negative indicates that it is related to symptoms and diseases that the patient does not have; positive indicates that it is related to symptoms and diseases that the patient has or may have in the future; unlabeled indicates medical entities that are independent of the patient's current state and have no actual diagnostic significance; others indicates other medical entities that cannot be clearly inferred to be negative, positive, or unlabeled.
[0093] Furthermore, the device 500 also includes a pre-training module; the pre-training module is used to train the mask language model through a pre-training task via the following steps:
[0094] A predetermined proportion of words are selected from the training text and masked to obtain the training text vector;
[0095] Input the training text vector into the initial mask language model to obtain the training mask vector corresponding to each mask in the training text;
[0096] Based on the training mask vector corresponding to each mask, determine the probability distribution of the mask with respect to the vocabulary;
[0097] Based on the probability distribution of the mask with respect to the vocabulary and the words corresponding to the mask in the training text, the initial mask language model is trained to obtain the mask language model.
[0098] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 600 includes a processor 610, a memory 620, and a bus 630.
[0099] The memory 620 stores machine-readable instructions executable by the processor 610. When the electronic device 600 is running, the processor 610 and the memory 620 communicate via the bus 630. When the machine-readable instructions are executed by the processor 610, they can perform the operations described above. Figures 1 to 4 The steps of an entity label attribute recognition method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.
[0100] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figures 1 to 4 The steps of an entity label attribute recognition method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.
[0101] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0102] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0104] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0105] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0106] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for recognizing entity label attributes, characterized in that, The method includes: Construct the text vector and attention masking matrix of the text containing the target entity; wherein, the values of the matrix elements in the attention masking matrix are used to characterize the mutual visibility between the context of the text containing the target entity and the mask of the target entity; The text vector and the attention masking matrix are input into the masking language model trained by the pre-training task to obtain the masking vector corresponding to the target entity output by the masking language model; wherein, the pre-training task is a cloze test task that predicts the partial words by masking some words in the text. The label attribute category of the target entity is determined based on the mask vector; The text vector includes: text segmentation vector, positional encoding vector, and semantic encoding vector; Construct the attention masking matrix of the text containing the target entity, including: An initial attention masking matrix is constructed based on each splitting unit and mask identifier in the text splitting vector; wherein each position in the row and column of the initial attention masking matrix represents each splitting unit and mask identifier; Based on the visibility between each segmentation unit corresponding to the text containing the target entity, and the visibility between each segmentation unit corresponding to the text containing the target entity and the mask identifier, the value of each matrix element in the initial attention masking matrix is determined to obtain the attention masking matrix; wherein, each segmentation unit corresponding to the text containing the target entity is visible to each other; the mask identifier is invisible to each segmentation unit corresponding to the text containing the target entity; except for the target entity, each segmentation unit corresponding to the text containing the target entity is invisible to the mask identifier; the target entity is visible to the mask identifier.
2. The method according to claim 1, characterized in that, Construct the text vector of the text containing the target entity, including: The text containing the target entity is split into multiple split units to form a split text sequence; Add the mask identifier corresponding to the mask of the target entity to the split text sequence to obtain the text split vector; Based on the position of each splitting unit in the text splitting vector corresponding to the target entity in the text and the position of the target entity corresponding to the mask identifier in the text, a position encoding vector is constructed; The semantic encoding corresponding to each splitting unit in the text splitting vector and the mask identifier is set to a preset value to obtain a semantic encoding vector.
3. The method according to claim 2, characterized in that, The step of inputting the text vector and the attention masking matrix into a masking language model trained through a pre-training task to obtain the masking vector corresponding to the target entity output by the masking language model includes: The text vector and the attention masking matrix are input into a masking language model trained by a pre-training task. The masking language model encodes each splitting unit and mask identifier in the text splitting vector. The vector obtained by encoding the mask identifier using the mask language model is used as the mask vector corresponding to the target entity.
4. The method according to claim 1, characterized in that, When the method is applied to entity label attribute recognition in the medical field, determining the label attribute category of the target entity based on the mask vector includes: The mask vector is input into the linear output layer to obtain the label attribute categories output by the linear output layer; wherein, the label attribute categories include negative, positive, unlabeled, and others; negative indicates that it is related to symptoms and diseases that the patient does not have; positive indicates that it is related to symptoms and diseases that the patient has or may have in the future; unlabeled indicates medical entities that are independent of the patient's current state and have no actual diagnostic significance; others indicates other medical entities that cannot be clearly inferred to be negative, positive, or unlabeled.
5. The method according to claim 1, characterized in that, The mask language model is obtained through the following steps via a pre-training task: A predetermined proportion of words are selected from the training text and masked to obtain the training text vector; Input the training text vector into the initial mask language model to obtain the training mask vector corresponding to each mask in the training text; Based on the training mask vector corresponding to each mask, determine the probability distribution of the mask with respect to the vocabulary; Based on the probability distribution of the mask with respect to the vocabulary and the words corresponding to the mask in the training text, the initial mask language model is trained to obtain the mask language model.
6. A device for recognizing entity label attributes, characterized in that, The device includes: A construction module is used to construct the text vector and attention masking matrix of the text containing the target entity; wherein, the values of the matrix elements in the attention masking matrix are used to characterize the mutual visibility between the context of the text containing the target entity and the mask of the target entity; The input module is used to input the text vector and the attention masking matrix into the masking language model trained by the pre-training task to obtain the masking vector corresponding to the target entity output by the masking language model; wherein, the pre-training task is a cloze test task that predicts the partial words by masking some words in the text. The determination module is used to determine the label attribute category of the target entity based on the mask vector; The text vector includes: a text splitting vector, a positional encoding vector, and a semantic encoding vector; when constructing the attention masking matrix of the text containing the target entity, the construction module is used for: An initial attention masking matrix is constructed based on each splitting unit and mask identifier in the text splitting vector; wherein each position in the row and column of the initial attention masking matrix represents each splitting unit and mask identifier; Based on the visibility between each segmentation unit corresponding to the text containing the target entity, and the visibility between each segmentation unit corresponding to the text containing the target entity and the mask identifier, the value of each matrix element in the initial attention masking matrix is determined to obtain the attention masking matrix; wherein, each segmentation unit corresponding to the text containing the target entity is visible to each other; the mask identifier is invisible to each segmentation unit corresponding to the text containing the target entity; except for the target entity, each segmentation unit corresponding to the text containing the target entity is invisible to the mask identifier; the target entity is visible to the mask identifier.
7. The apparatus according to claim 6, characterized in that, The text vector includes: a text splitting vector, a positional encoding vector, and a semantic encoding vector; when constructing the text vector of the text containing the target entity, the construction module is used to: The text containing the target entity is split into multiple split units to form a split text sequence; Add the mask identifier corresponding to the mask of the target entity to the split text sequence to obtain the text split vector; Based on the position of each splitting unit in the text splitting vector corresponding to the target entity in the text and the position of the target entity corresponding to the mask identifier in the text, a position encoding vector is constructed; The semantic encoding corresponding to each splitting unit in the text splitting vector and the mask identifier is set to a preset value to obtain a semantic encoding vector.
8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of a method for identifying entity tag attributes as described in any one of claims 1 to 5.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of a method for identifying entity tag attributes as described in any one of claims 1 to 5.