Feature representation method applied to text fine-grained entity classification and related device

Abstract

The application provides a feature representation method applied to text fine-grained entity classification and related equipment. The entity mention in the character of the target text can be determined, the entity mention is put back to the context character, and then the context character representation is obtained. The context representation model is used for tensor determination processing, the context tensor information is obtained, and the context tensor information is calculated and processed through the attention mechanism. The context feature obtained in this way can have the characteristics of the context, and the entity mention and the context can be contained in the context feature. When fine-grained entity classification is performed subsequently, the classification result is more accurate, and the effect of fine-grained entity classification is further improved.

Description

Technical Field

[0001] This application relates to the field of semantic processing technology, and in particular to a feature representation method and related equipment for fine-grained entity classification of text. Background Technology

[0002] Fine-grained entity classification tasks assign fine-grained categories to entities in text, providing rich semantic information through category information and playing an important role in downstream tasks such as relation extraction, entity linking, and question answering systems.

[0003] Since the length and position of entities in a sentence are not uniform, it is impossible to directly calculate the representation of an entity in the context. Existing fine-grained entity classification models process and represent entity mentions and their contexts separately, thus severing the semantic relationship between entities and their contexts. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a feature representation method and related equipment for fine-grained entity classification of text, which should solve or partially solve the above-mentioned technical problems.

[0005] To achieve the above objectives, the first aspect of this application provides a feature representation method for fine-grained entity classification of text, comprising:

[0006] Represent the target text using characters;

[0007] Determine the entity reference in the character and the context character corresponding to the entity reference, and put the entity reference back into the context character to obtain the context character representation;

[0008] The context character representation is input into the context representation model, and tensor determination processing is performed to obtain context tensor information. The context tensor information is then processed to obtain the context features of the target text. The context representation model is obtained by training a neural network using training samples.

[0009] Based on the same inventive concept, a second aspect of this application proposes a feature representation device for fine-grained entity classification of text, comprising:

[0010] The character representation module is configured to represent the target text as characters;

[0011] The context representation module is configured to determine the entity reference in the character and the context character corresponding to the entity reference, and put the entity reference back into the context character to obtain the context character representation;

[0012] The model processing module is configured to input the context character representation into the context representation model, perform tensor determination processing to obtain context tensor information, and perform calculation processing on the context tensor information to obtain the context features of the target text. The context representation model is obtained by training a neural network using training samples.

[0013] Based on the same inventive concept, a third aspect of this application proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0014] As can be seen from the above, the feature representation method and related device for fine-grained entity classification of text provided in this application can determine entity references in the characters of the target text, put the entity references back into the context characters to obtain the context character representation, and use the context representation model to perform tensor determination processing to obtain context tensor information. Then, the context tensor information is processed by the attention mechanism. The resulting context features have the characteristics of context and can include the relationship between entity references and context in the context features, making the classification results more accurate when performing fine-grained entity classification, and further improving the effect of fine-grained entity classification. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram illustrating the processing of a traditional fine-grained entity classification model.

[0017] Figure 2A This is a flowchart illustrating a feature representation method applied to fine-grained entity classification of text, as described in an embodiment of this application.

[0018] Figure 2B This is a schematic diagram of a traditional context representation;

[0019] Figure 2C This is a schematic diagram illustrating the feature representation of an entity with context returned according to an embodiment of this application;

[0020] Figure 2D This is a schematic diagram comparing the context representation models based on FastText+BILSTM and BERT in embodiments of this application;

[0021] Figure 2E This is a schematic diagram illustrating the processing of the context representation model with attention mechanism in an embodiment of this application;

[0022] Figure 2F This is a schematic diagram illustrating the performance of models trained on different crowdsourced data ratios in this application on a test set.

[0023] Figure 2G This is a schematic diagram illustrating the performance of different module combinations of models in this application on a test set;

[0024] Figure 2H This is a table comparing the solutions of this application embodiment with CFET results;

[0025] Figure 2I This is a table comparing the solutions of this application embodiment with the decomposition results of CFET for different granularity entity categories;

[0026] Figure 3 This is a schematic diagram of the feature representation device applied to fine-grained entity classification of text according to an embodiment of this application;

[0027] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0029] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0030] Based on the background technology, the relevant technologies of this application are further described below:

[0031] An entity refers to an object with specific meaning in text. Based on different granularities of entity categories, the academic community has two tasks: Named Entity Recognition (NER) and Fine-grained Entity Typing (FGET). NER aims to find the location of entities in text and assign their categories, targeting a small number of coarse-grained categories, such as people, places, and organizations. FGET, on the other hand, given an entity and its location, assigns one or more entity categories based on the context. Its target category granularity is usually more refined, and the categories form a category tree (e.g., / person / politician, / person / celebrity), thus providing a more accurate characterization of entities. Both NER and FGET require predicting entity categories from context. The difference is that in NER, the entity and its location need to be predicted and output, while in FGET, the entity and its location are already given. Table 1 uses a sentence from the People's Daily as an example to compare the similarities and differences between NER and FGET tasks in terms of both input and output.

[0032] Table 1

[0033]

[0034] In 2018, Choi et al. proposed the English ultra-fine entity classification task UFET (Ultra-Fine Entity Typing) and its dataset, containing 9 major categories, 121 minor categories, and 10,201 ultra-fine granular categories. Prior to this, the FIGER dataset, which had the most categories in entity classification, only had 113 categories. Compared to English, the Chinese FGET task started later. In 2020, Lee et al., following the construction method of UFET, proposed the first Chinese fine-grained entity classification task, the CFET (Chinese Fine-grained Entity Typing) dataset, containing 10 major categories and over 7,182 minor categories. In both UFET and CFET datasets, the coverage of entity mentions is no longer limited to named entities, but also includes pronouns and noun phrases such as 'you,' 'I,' and 'he,' increasing the coverage of entity categories and entity mentions, thus posing a greater challenge to fine-grained entity classification tasks.

[0035] This application addresses the fine-grained entity classification task in Chinese, using the CFET dataset as the research object. It tackles the problem that existing fine-grained entity classification methods process and represent entity mentions and contexts separately, thus severing the connection between entities and their contexts. This application proposes a feature representation method that integrates entities back into their context, feeding the model with complete sentences and resolving the feature computation problem when entity positions and lengths are inconsistent after context integration. In experiments, this application compares character representations based on fasttext+BILSTM and BERT, and adds an attention layer to the context representation of the fasttext+BILSTM model. Regarding training strategies, this application explores the impact of using distant supervised data on training results and compares the impact of using a general category mapping on entity categories of different granularities.

[0036] Fine-grained entity classification (FGET) is essentially a special type of text classification task. Unlike sentiment classification, which categorizes entire sentences, and named entity recognition, which categorizes (or labels) individual characters, entity classification focuses on entities of indefinite length within a sentence, defined by their context. In FGET, an entity mention refers to a segment of text that refers to an entity within a specific sentence; the context refers to the text preceding and following the entity mention; and the lexical representation refers to the specific text that constitutes the entity mention, independent of the context. The lexical representation of an entity mention contains information about the entity itself, but entity mentions with the same lexical representation can have different categories in different contexts. For example, the same person's name or the noun pronoun "he" can have different professions or identities in different sentences, belonging to different fine-grained categories. Therefore, both the lexical representation and its context are crucial for determining entity categories. The core of text classification tasks is text feature representation and classification models. Thus, a classic fine-grained entity classification model consists of three parts: entity mention (lexical) representation, context representation, and a classification inference layer. Its model structure is as follows: Figure 1 As shown.

[0037] Characters are the basic units that constitute entity mentions and their context. To represent entity mentions and their context using features, we first need to obtain the distributed feature representations of their constituent characters. Similar to word vectors in English, we can obtain character vectors for Chinese characters by pre-training on a large amount of Chinese text using unsupervised algorithms. Depending on whether the character vectors are fixed, pre-trained character representation models can be divided into context-independent static character representations and context-dependent dynamic character representations.

[0038] Context-independent static character representations are obtained through a lookup table. The vector representation of each character is obtained through the lookup table. The vector of each character is fixed in the table and does not change with different contexts. Commonly used static pre-trained models include Word2vec (2013), GloVe (2014), and FastText (2016).

[0039] Dynamic character representations are typically generated by pre-trained language models that can capture contextual relationships. The character representations generated by these models change with the context; that is, the same character will have different vector representations in different sentences. The most classic pre-trained language model is BERT. Pre-trained language models such as BERT can obtain rich prior information from large-scale corpora, so each character token can generate an informative context vector.

[0040] Entity mention representation in the FGET task involves representing the features of the word face that constitutes an entity mention. This is typically achieved by averaging the static character vectors of the entity mention word face, i.e.:

[0041] Where u(m) i ) represents the vector representation of the i-th character in entity mention m.

[0042] In the FGET task, context representation, besides averaging the character vectors of the context (similar to entity mentions), typically employs an encoder such as a Long Short-Term Memory (LSTM) network to encode the context when using static vector representations. The encoder's role is to model the semantics of characters within a sentence, analyzing the position of characters and the dependencies between them to obtain a representation for each character in the current sentence. Building upon this, Shimaoka et al. introduced an attention mechanism to weight the output of the LSTM.

[0043] Based on the above, the feature representation method proposed in this embodiment for fine-grained entity classification of text, such as... Figure 2A As shown, it includes:

[0044] Step 201: Represent the target text as a character.

[0045] In practice, since the machine cannot recognize the text content, it is necessary to convert the text into corresponding characters for representation.

[0046] Step 202: Determine the entity reference in the character and the context character corresponding to the entity reference, and put the entity reference back into the context character to obtain the context character representation.

[0047] In practice, the obtained context characters include: preceding characters, entity references, and following characters. This allows us to determine the relationship between entity references and the context.

[0048] Step 203: Input the context character representation into the context representation model, perform tensor determination processing to obtain context tensor information, and calculate the context tensor information to obtain the context features of the target text. The context representation model is obtained by training a neural network using training samples.

[0049] In specific implementation, such as Figure 2B As shown, traditional entity feature representation methods process entity mentions and their contexts separately, severing the natural connection between entities and their contexts. To better represent the contextual features of entities, referencing feature representation methods in named entity recognition, entity mentions are placed back into their context, such as... Figure 2C As shown, the entire text is fed into a neural network model for training. The resulting context representation model can perform feature processing on the mentions of various entities in the text and the context character representations composed of context characters, thereby obtaining context features with contextual relationships. Entities are placed back into the context, and context features are generated based on the character representations of the entity mention positions.

[0050] The above scheme can identify entity references in the target text, place these entity references back into the context characters to obtain context character representations, and then use a context representation model to perform tensor determination processing to obtain context tensor information. An attention mechanism is then used to process the context tensor information, resulting in context features that possess the characteristics of context, including the relationship between entity references and the context. This makes subsequent fine-grained entity classification more accurate and further improves the performance of fine-grained entity classification.

[0051] In some embodiments, the context representation model includes: a Fast Text text classification model and a BILSTM (Bidirectional Long Short-Term Memory) model, with an attention mechanism added after the BILSTM model. For example... Figure 2D As shown, taking a section of corpus from CFET as an example, the context representation process of the two types of models is compared.

[0052] Step 201 includes: using FastText to perform static character representation of the target text.

[0053] In specific implementation, the context characters obtained in step 202 include: preceding static characters, entity mention static characters, and following static characters.

[0054] Step 203 includes:

[0055] Step 2031: The context character representation is a static character representation. The context character representation is input into BILSTM for context encoding processing to obtain the dynamic representation of the entity mention in the context, and thus obtain the context tensor information.

[0056] Step 2032: The context tensor information is processed using an attention mechanism to obtain the context features of the target text.

[0057] In practical implementation, the FastText+BILSTM-based model uses BILSTM to perform context encoding on the character representations since the static characters generated in FastText are context-independent, thereby obtaining the dynamic representation of entities in the context and thus the context tensor information. This embodiment uses the Chinese pre-trained vectors (1) from the FastText official website as static character representations, used for both the static representation of entity mentions and the static context representation in the BILSTM-based model.

[0058] When calculating the average vector representation of an entity's context-replaced feature representation, it's impossible to directly average its static representation as in calculating the static representation. This is because during training, the context is processed in batches as tensors, and the position of an entity in the context for each sample is not fixed, and the entity length is inconsistent. Therefore, only the vector representation of entity mentions can be directly calculated. To address this issue, existing methods (such as CFET and Dai) use a fixed symbol to replace entity mentions of arbitrary length to circumvent the problem of inconsistent entity lengths. For example, in CFET, a fixed [MASK] symbol is used in the context, and the vector at the [MASK] position is used as the entity's context feature representation. However, different entity literals should have different representations in the context. Using [MASK] ignores the differences in literals between entities, thus losing entity literal information and affecting the entity's representation in the context. This embodiment uses entity context-replaced feature representation, encoding the entity itself back into the context to obtain the dynamic representation of entity mentions in the context, thus obtaining the context tensor information. Then, an attention mechanism is used to process the context tensor information to obtain the context features of the target text.

[0059] In some embodiments, step 2032 includes:

[0060] Step 20321: Use the torch.cumsum function to sum the context tensor information along the dimensions of the target text's context to obtain a tensor S of the same size as the context tensor information. i The formula is: Among them, S i For the sum of the first i elements in the context tensor information C, C k Let i be the k-th value in the context tensor information C, where i and k are both positive integers, and k∈i.

[0061] Step 20322, determine S i The cumulative tensor value S of the number of tails T in the middle T , and S i The cumulative tensor value S of the number of heads H in the middle H And the cumulative tensor value C of the number of heads H in C. H Calculate the contextual features V of the target text. C The formula is:

[0062] In practice, tensor operations are used to calculate the average vector representation V of entity mentions. C As a contextual representation of entities, this solves the computational problem when entity positions and lengths are inconsistent. In addition to averaging the tensors of entity mentions as described above, this embodiment uses an attention-weighted contextual processing method to further utilize the information of contextual characters, rather than being limited to using the tensor representation of entity mentions. Figure 2E It demonstrates an attention-weighted context representation model. When using attention-weighting, the context representation of entity mentions changes from averaging the word meanings of entity mentions to attention-weighting the entire sentence.

[0063] In some embodiments, the context representation model includes the BERT model.

[0064] Step 203 includes:

[0065] The context character representation is input into the BERT model for context encoding to obtain the dynamic representation of the entity mention in the context, thus obtaining the context tensor information. The context tensor information is then processed to obtain the context features of the target text.

[0066] In practice, the dynamic character representations generated by the BERT-based context representation model are context-dependent, and BERT itself has context encoding capabilities and can be trained through fine-tuning. Therefore, no additional context encoder is added to the BERT model. The BERT-base-Chinese model pre-trained on Chinese corpora from the Huggingface website is used as the dynamic character representation in the BERT model.

[0067] In some embodiments, the BERT model includes a 6-layer encoder and a 6-layer decoder, with the first 9 layers of the BERT model frozen and the last 3 layers of the BERT model trained and adjusted during training.

[0068] The above approach can accelerate the training process of the BERT model while ensuring its processing accuracy.

[0069] In some embodiments, prior to step 201, the training process of the context representation model includes:

[0070] Step 20A: Obtain crowdsourced data of manually labeled tensor information, as well as remote supervision data of machine-labeled tensor information.

[0071] In its implementation, CFET includes crowdsourced data and remote supervision data. The crowdsourced data consists of manually labeled samples, while the remote supervision data consists of machine-labeled pseudo-samples. Remote supervision is a method that uses weakly labeled training sets to train classifiers, addressing the problem of insufficient training samples. To overcome the limitations of manual data labeling in supervised learning, Mintz et al. proposed a remote supervision algorithm. The core idea of ​​this algorithm is to align text with entities in a large-scale knowledge graph, using the existing entity relationships in the knowledge graph to label the text, thereby automatically generating pseudo-samples for training. The CFET remote supervision dataset contains 1,913,197 samples, while the crowdsourced dataset only has 4,798 samples (the original description states 4,800, but the actual number is only 4,798), making the remote supervision data approximately 400 times larger than the crowdsourced data.

[0072] Step 20B: Divide the crowdsourced data into a training set, a validation set, and a test set.

[0073] In specific implementation, this embodiment follows the approach in the original CFET paper and divides the 4798 crowdsourced samples into a training set, a validation set, and a test set, with 1598 samples in the training set, and 1600 samples each in the validation set and the test set.

[0074] Step 20C: The crowdsourced data and the remote supervision data are combined into the training sample according to a predetermined ratio. The training sample includes: a training set, a validation set, a test set, and remote supervision data.

[0075] Step 20D: Set the corresponding learning rate for the pre-built neural network model.

[0076] In some embodiments, the Adam optimizer is used to perform convergence optimization on the neural network model during training. The learning rate of the Adam optimizer is set to 0.001, and when using BERT, the learning rate for BERT fine-tuning is set to 3e. -5 The learning rate for modules other than BERT in the model is still set to 0.001.

[0077] Step 20E: Train the neural network model using the training set in the training samples and the remote supervision data; validate the trained neural network model using the validation set; and test the validated neural network using the test set after validation.

[0078] Step 20F: In response to the determination that the test has passed, the final neural network model is used as the context representation model.

[0079] CFET maps each entity mention's fine-grained type (FGT) to a higher-level, more general type (GT). All FGTs are defined as 10 more general GTs: Person, Organization, Place, Product, Group, Creation, Event, Concept, Organism, and Other. This application compares the impact of incorporating general type mapping in experiments. When considering general type mapping, fine-grained types are mapped to their respective general types according to the type mapping, and the corresponding general types are used as labels for entity mentions during training.

[0080] For ease of comparison, in subsequent experiments of this application, models that consider general category mapping will be represented by "+General", models that use entity-back-to-context feature representations will be represented by "+Context", models without "+Context" will be represented by [MASK] instead of entity-back-to-context, and models with added attention layers will be represented by "+Attention".

[0081] In some embodiments, the method further includes, prior to step 20E:

[0082] Step 20E': Truncate the training samples that are longer than a predetermined length.

[0083] Because the BERT model consumes a lot of resources, the batch size is set to 64 when using BERT, and 128 when using FastText+BILSTM. For longer samples, this embodiment truncates the samples according to the maximum sequence length (max seq length), which is set to 128 in this embodiment.

[0084] Based on the description of the above embodiments, the experimental results of the context representation model obtained by the scheme of this application are analyzed below:

[0085] I. The impact of remote supervised training on the model

[0086] In CFET, a training set is formed by mixing remotely supervised datasets and crowdsourced data training sets. Specifically, for each batch of training samples, a portion are randomly selected with replacement from the crowdsourced training set, while the remaining samples are randomly selected without replacement from the remotely supervised data. To investigate the impact of crowdsourced and remotely supervised data on the results, this application sets up multiple control experiments with crowdsourced data ratios (CR) of 1, 2 / 3, 1 / 2, 1 / 3, 1 / 6, and 0, respectively, and corresponding remotely supervised data ratios of 1-CR, with values ​​of 0, 1 / 3, 1 / 2, 2 / 3, 5 / 6, and 1.

[0087] When the proportion of crowdsourced data is 1%, the proportion of remote supervision data is 0%, meaning training is done using only crowdsourced data. One epoch of training represents traversing the crowdsourced dataset once. Since crowdsourced data is scarce, the model validation period is set to 20 epochs. When the proportion of crowdsourced data is less than 1%, the proportion of remote supervision data is not 0, and remote supervision data is added for training. One epoch of training represents traversing the remote supervision dataset once. Because the amount of data in the remote supervision training set is large (approximately 400 times that of the crowdsourced data), the model validation period is set to 1 / 4 epoch. If the macro-average F1 score (F-Measure) on the validation set does not improve within four consecutive validation epochs, training is prematurely interrupted, and the model with the highest macro-average F1 score on the validation set is retained.

[0088] like Figure 2F As shown, the performance of BERT-based and BILSTM-based models trained with different proportions of crowdsourced data on the test set is illustrated. The experiments consistently used the character representation of the regression context (+Context) and the general category mapping (+General), providing a more comprehensive comparison than simply considering the impact of fine-grained categories on model performance.

[0089] like Figure 2FIt can be seen that when the proportion of crowdsourced data is 0 and the proportion of remotely supervised data is 1, the F1 scores of both models trained using only remotely supervised data are significantly lower than those trained using crowdsourced data. Therefore, when using remotely supervised data, adding crowdsourced data for training significantly improves the model performance. This result indicates that the quality of remotely supervised data is significantly worse than that of crowdsourced data because the sources of remotely supervised data and the test set are different; however, it also demonstrates the effectiveness of remotely supervised training. The model trained using only machine-labeled remotely supervised data can still achieve certain results on the human-labeled crowdsourced data test set. The current mainstream view is that adding remotely supervised data on top of crowdsourced data can improve experimental results, but the experimental results of this application show that when using crowdsourced data (CR / =0), the effect of adding remotely supervised data on the model is inconsistent between the BILSTM-based model and the BERT-based model. When using the BILSTM model, the F1 score when adding remotely supervised data (CR=1 / 6, 1 / 3, 1 / 2, 2 / 3) is greater than the result when using only crowdsourced data (CR=1). However, when using the BERT model, the F1 score on the test set is highest when using only crowdsourced data. Adding remote supervision data actually degrades the model's performance. This analysis suggests that the improvement of the model by remote supervision data depends on the high quality of the data. Since the encoder structure of the BILSTM model is not pre-trained, it is prone to overfitting when trained on crowdsourced datasets due to insufficient data volume. Adding remote supervision data can introduce more information to train the BILSTM structure. On the other hand, the BERT model itself has an encoding structure and is pre-trained on a large-scale corpus, resulting in high-quality pre-training information. Its effect on the model improvement is greater than that of remote supervision data. In this case, adding remote supervision data will actually weaken the pre-training effect of the BERT model.

[0090] When using both crowdsourced and remotely supervised data simultaneously, the different proportions of crowdsourced data have little impact on the F1 score. During CFET training, each batch was configured with 1 / 3 crowdsourced data and 2 / 3 remotely supervised data. For easier comparison, subsequent experiments in this application will set the crowdsourced data proportion (CR) to 1 or 1 / 3, corresponding to using only crowdsourced data or crowdsourced data comprising 1 / 3 of each batch, to further observe the impact of remotely supervised data on the model results.

[0091] II. The impact of each module on the experimental results

[0092] This application addresses the problem of feature computation after an entity is placed back into its context. In the experiments, the feature representation model after placing the entity back into its context is represented by "Context=1", while the feature representation model that replaces the entity with [MASK] is represented by "Context=0". Based on placing the entity or [MASK] back into its context, this application introduces an attention mechanism to weight the character representation of the entire context to obtain more comprehensive contextual information. In the experiments, the model with the attention mechanism is represented by "Attention=1", while the model with "Attention=0" represents using only the average character vector of the entity word literal or the representation vector of [MASK] as the context representation. Figure 2G The results of different model settings on the test set are presented in the form of bar charts.

[0093] (1) Feature representation of an entity placed back into context. Figure 2G As can be seen, regardless of whether the crowdsourced data ratio is 1% or 1 / 3, whether the model is based on BERT or BILSTM, and whether an attention layer is used, the F1 score of the representation model that puts entities back into context is significantly higher than that of the representation model that uses [MASK] to put entities back into context. According to calculations, the average improvement in the F1 score of the feature representation method that puts entities back into context compared to the [MASK] representation method reaches 5.85% on the test set, proving the effectiveness of the entity-back-to-context method in this application.

[0094] (2) Attention mechanism. Figure 2G As can be seen, under the same data ratio, regardless of the feature representation model used, adding an attention layer ("Attention=1") to the BILSTM-based model improves performance, while it leads to worse results for the BERT-based model. In the BERT-based model, adding an attention layer actually reduces performance, indicating that using the average character representation of entity word literals in the BERT model is superior to attention representation of the entire context. This analysis suggests that BERT itself possesses an attention structure sufficient to capture contextual information. Adding an attention layer requires training the added layer, and this fine-tuning weakens BERT's context encoding ability when trained on large-scale corpora.

[0095] III. Comparison with the original CFET results

[0096] like Figure 2HAs shown in the table, the results of this application and the original CFET paper are listed on BERT-based and BILSTM-based models. For the BERT-based model, no attention is used, and training is performed using only crowdsourced data (CR=1); for the BILSTM-based model, an attention layer is added, and training is performed using both crowdsourced data and remote supervised data (CR=1 / 3). In this application's BILSTM-based model, after introducing character representations of regression context (+Context) and an attention mechanism (+Attention), the F1 score without using general class mapping increased from 19.8% to 32.8%, and the F1 score with general class mapping (+General) increased by 20% to 45.5%. In this application's BERT-based model, after introducing character representations of regression context (+Context), using only crowdsourced data and no remote supervised data, the F1 score without using general class mapping increased from 35.7% to 46.9%, and the F1 score with general class mapping (+General) increased from 47.9% to 58.5%.

[0097] Regarding the decomposition results for entity categories at different granularities, the original CFET paper only lists the results based on the BERT model, such as... Figure 2I As shown in the table, the results of this application based on BERT are also listed and compared with those of the original paper. It can be seen that this application has achieved better results than the original paper in entity categories at all granularities. In particular, the F1 score of the model in this application in fine-grained categories has increased from 24.9% in the original paper to 35.8%, an increase of about 44%.

[0098] In summary, this application addresses the problems of existing FGET methods that process entity mentions and context separately, proposing a feature representation method that integrates entity context. The average character representation of the entity word literal is obtained through tensor computation as the entity's context representation. Based on this, an attention mechanism is introduced, using attention to weight all contexts to obtain the entity's context representation. The results show that the attention mechanism is effective for BILSTM models but has a negative impact on BERT-based models, indicating that using the average character representation of the entity word literal is superior to attention-based representation of the entire context in BERT models. This application also explores the impact of remote supervised training on the model, finding that BERT-based models trained without remote supervised data outperform those trained with it. Furthermore, this application compares the decomposition results of entity categories at different granularities using and without a general category mapping, finding that using a general category mapping yields better results for general categories, while not using it achieves better results for fine-grained categories. Finally, the method in this application is compared with the original CFET paper, showing that the proposed solution significantly improves performance.

[0099] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0100] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0101] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a feature representation device for fine-grained entity classification of text, such as... Figure 3 As shown, it includes:

[0102] Character representation module 31 is configured to represent target text as characters;

[0103] The context representation module 32 is configured to determine the entity reference in the character and the context character corresponding to the entity reference, and put the entity reference back into the context character to obtain the context character representation;

[0104] The model processing module 33 is configured to input the context character representation into the context representation model, perform tensor determination processing to obtain context tensor information, and perform calculation processing on the context tensor information to obtain the context features of the target text. The context representation model is obtained by training a neural network using training samples.

[0105] In some embodiments, the context representation model includes: a Fast Text text classification model and a BILSTM bidirectional long short-term memory model, with an attention mechanism added after the BILSTM bidirectional long short-term memory model;

[0106] Character representation module 31 is also configured as follows:

[0107] The target text is represented statically using Fast Text.

[0108] Model processing module 33 is also configured as follows:

[0109] The context character representation is a static character representation. The context character representation is input into BILSTM for context encoding processing to obtain the dynamic representation of the entity mention in the context, thus obtaining the context tensor information. The context tensor information is then processed using an attention mechanism to obtain the context features of the target text.

[0110] In some embodiments, the model processing module 33 is further configured to:

[0111] The `torch.cumsum` function is used to sum the context tensor information along the dimensions of the target text's context, resulting in a tensor S of the same size as the context tensor information. i The formula is:

[0112] Among them, S i For the sum of the first i elements in the context tensor information C, C k Let i be the k-th value in the context tensor information C, where i and k are both positive integers, and k∈i;

[0113] Determine S i The cumulative tensor value S of the number of tails T in the middle T , and S i The cumulative tensor value S of the number of heads H in the middle H And the cumulative tensor value C of the number of heads H in C. H Calculate the contextual features V of the target text. C The formula is:

[0114]

[0115] In some embodiments, the context representation model includes: the BERT model;

[0116] Model processing module 33 is also configured as follows:

[0117] The context character representation is input into the BERT model for context encoding to obtain the dynamic representation of the entity mention in the context, thus obtaining the context tensor information. The context tensor information is then processed to obtain the context features of the target text.

[0118] In some embodiments, the BERT model includes a 6-layer encoder and a 6-layer decoder, with the first 9 layers of the BERT model frozen and the last 3 layers of the BERT model trained and adjusted during training.

[0119] In some embodiments, the apparatus further includes a training module configured to:

[0120] Crowdsourced data for obtaining manually labeled tensor information, and remote supervision data for machine-labeled tensor information;

[0121] The crowdsourced data is divided into a training set, a validation set, and a test set;

[0122] The crowdsourced data and the remote supervision data are combined in a predetermined ratio to form the training sample, which includes: a training set, a validation set, a test set, and remote supervision data.

[0123] Set the corresponding learning rate for the pre-built neural network model;

[0124] The neural network model is trained using the training set in the training samples and the remote supervision data, the trained neural network model is validated using the validation set, and the validated neural network is tested using the test set after validation.

[0125] Upon confirmation that the test has passed, the final neural network model is used as the context representation model.

[0126] In some embodiments, the training module is further configured to: use the Adam optimizer to perform convergence optimization processing on the neural network model during the training process.

[0127] In some embodiments, the training module is further configured to:

[0128] Training samples that are longer than a predetermined length are truncated.

[0129] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0130] The apparatus of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0131] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the above embodiments.

[0132] Figure 4This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0133] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0134] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0135] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0136] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0137] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0138] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0139] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0140] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0141] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0142] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0143] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0144] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0145] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0146] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A feature representation method applied to text fine-grained entity classification, characterized in that, The method comprises: performing character representation on a target text; determining an entity mention in the characters and context characters corresponding to the entity mention, and putting the entity mention back into the context characters to obtain context character representation; inputting the context character representation into a context representation model, performing tensor determination processing to obtain context tensor information, and performing calculation processing on the context tensor information to obtain context features of the target text, wherein the context representation model is obtained by training a neural network using training samples; the character representation on the target text comprises: performing static character representation on the target text using a Fast Text text classification model; the inputting of the context character representation into the context representation model, the performing of tensor determination processing, the obtaining of context tensor information, and the performing of calculation processing on the context tensor information to obtain context features of the target text comprise: the context character representation is static character representation, and the context character representation is inputted into a BILSTM bidirectional long short-term memory model to perform context encoding processing, to obtain dynamic representation of the entity mention in the context, and to obtain the context tensor information; The context tensor information is accumulated and summed along the dimension of the context of the target text by using a torch. cumsum function, to obtain a tensor S with the same size as the context tensor information i The formula is: where S i is the sum of the first i of the context tensor information C, C k is the kth value of the context tensor information C, i and k are both positive integers, k e i; Determine S i The cumulative tensor value S of the number of tails T in the middle T , and S i The cumulative tensor value S of the number of heads H in the middle H And the cumulative tensor value C of the number of heads H in C. H Calculate the contextual features V of the target text. C The formula is:

2. The method of claim 1, wherein, the training process of the context representation model comprises: obtaining crowd-sourced data with manually labeled tensor information and remote supervision data with machine automatically labeled tensor information; dividing the crowd-sourced data into a training set, a validation set and a test set; the crowd-sourced data and the remote supervision data are combined into the training samples according to a predetermined proportion, and the training samples comprise the training set, the validation set, the test set and the remote supervision data; setting a corresponding learning rate for a pre-constructed neural network model; training the neural network model using the training set in the training samples and the remote supervision data, verifying the trained neural network model using the validation set, and testing the verified neural network using the test set after the verification is completed; in response to determining that the test is passed, the finally obtained neural network model is used as the context representation model.

3. The method of claim 2, wherein, In the process of training the neural network model, an Adam optimizer is used to perform convergence optimization processing on the neural network model.

4. The method of claim 2, wherein, After the crowd-sourced data and the remote supervision data are combined into the training samples according to the predetermined proportion, the method further comprises: truncating training samples greater than a predetermined length in the training samples.

5. A feature representation apparatus applied to text fine-grained entity classification, characterized in that, The method comprises: a character representation module configured to perform character representation on a target text; a context representation module configured to determine an entity mention in the characters and context characters corresponding to the entity mention, and put the entity mention back into the context characters to obtain context character representation; a model processing module configured to input the context character representation into a context representation model, perform tensor determination processing to obtain context tensor information, and perform calculation processing on the context tensor information to obtain context features of the target text, wherein the context representation model is obtained by training a neural network using training samples. The character representation module is specifically configured to perform static character representation on the target text by using a Fast Text text classification model. The model processing module is specifically configured to: The context character representation is a static character representation, and the context character representation is input into a BILSTM bidirectional long short-term memory model for context encoding processing to obtain a dynamic representation of the entity mention in the context, and obtain the context tensor information. The context tensor information is accumulated and summed along the dimension of the context of the target text by using a torch. cumsum function, to obtain a tensor S with the same size as the context tensor information i The formula is: where S i is the sum of the first i of the context tensor information C, C k is the kth value of the context tensor information C, i and k are both positive integers, k e i; determining S i the accumulated tensor value S of the tail number T in S T , and the accumulated tensor value S of the head number H in S i , and the accumulated tensor value C of the head number H in C H , and the accumulated tensor value C of the head number H in C H , calculating the context feature V of the target text C , the formula is:

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method in any one of claims 1 to 4 when executing the program.

Images