A hyperbolic space-based timestamp-enhanced event temporal relationship classification method

Abstract

The application discloses a timestamp-enhanced event time sequence relation classification method based on hyperbolic space, realizes extraction of semantic representation of events in a sentence in a sentence embedding manner based on a Robert-base pre-training model, considers semantic information and syntax information of the sentence, maps vector representation of a Euclidean space into a hyperbolic space by using a Poincare ball model, inputs the obtained vector representation into a hyperbolic space neural network for training, learns and obtains distance features and timestamp prediction features of event pair vectors from the characteristic vectors obtained through training, fuses the two types of features with common sense knowledge base features, and classifies event time sequence relations. The method comprises the following steps: a pre-training layer is responsible for converting a text sentence into a real value vector; a hyperbolic space layer is responsible for mapping vector representation of a Euclidean space into a hyperbolic space and extracting features by using a hyperbolic space neural network; a feature fusion layer is responsible for further processing and utilizing the characteristic vectors obtained from the hyperbolic space, calculating distance similarity between event pairs and predicting relative timestamps between event pairs, and fusing the obtained feature attributes with common sense knowledge base features; and a logistic regression layer is responsible for performing multinomial logistic regression calculation on the obtained fusion features, and finally classifying and predicting event time sequence relations.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, and particularly relates to relation classification, hyperbolic space neural networks, and other related fields. Background Technology

[0002] In the era of big data, network data is experiencing explosive growth, and people are playing a greater role in its application. However, the application of big data also presents numerous challenges. Traditional analysis methods are clearly insufficient to handle the massive amounts of network data. Improvements in hardware and the emergence of artificial intelligence algorithms have made it possible to effectively utilize massive amounts of data.

[0003] Event temporal relationship classification is an important natural language understanding task. Events are often described as changes in state, possessing attributes such as trigger words, subjects, objects, and types. The evolution from one event to another follows inherent patterns and connections; event temporal relationships are connections along the time dimension. Event temporal relationship classification plays a crucial role in understanding the development trends and logic of events, with significant applications in information extraction, question-answering systems, and decision-making. For example, event graphs incorporating temporal relationships have a more rigorous structure, clearer relationship networks, and a more accurate and clear representation of event development trends and evolutionary patterns, enabling question-answering or decision-making systems to provide higher-quality answers. In the medical field, automatically constructing timelines using temporal relationship technology to develop treatment plans for patients can, to some extent, reduce the workload of medical staff, allowing them to focus their energy on more important matters.

[0004] In recent years, deep learning methods have gained increasing popularity among researchers and have gradually become mainstream. Current deep learning methods, such as CNNs and RNNs, are mostly based on Euclidean space. While these offer significant advantages over traditional methods, they still struggle to extract more hierarchical relationships from text. Hyperbolic space provided excellent inspiration for this invention, possessing superior hierarchical spatial representation capabilities, particularly adept at handling textual data with hierarchical relationships, such as entity relationships and genus-species relationships. Later, researchers attempted to leverage prior knowledge from commonsense knowledge bases to improve model performance. While this can enhance the model's predictive ability to some extent, it also introduces noise from external data.

[0005] To address these issues, this invention proposes an event temporal relationship classification method based on hyperbolic space-enhanced timestamps. Given the excellent performance of the Roberta pre-trained model in representation learning, this invention uses the Roberta pre-trained model to train text data, thereby obtaining the vector representation of sentences. Hyperbolic space possesses excellent hierarchical relationship representation capabilities and can better optimize the event representation distortion problem in Euclidean space. This invention employs a Poincaré sphere model to complete the mapping from Euclidean space to hyperbolic space. To address the statistical probability imbalance problem caused by the commonsense knowledge base, this invention designs a hyperbolic space-based time anchor prediction module. By obtaining contextual representation information from the hyperbolic space layer, it predicts the relative timestamps of event pairs in the time series. Each timestamp is represented by a real number, reflecting the relative order of the two events on the timeline. Summary of the Invention

[0006] This invention proposes a hyperbolic time-series relationship classification method based on timestamps in hyperbolic space. This method utilizes a Roberta pre-trained model to obtain a learned representation in Euclidean space. A PoincaréBall model is then used to map word vector representations from Euclidean space to hyperbolic space. After computation by a hyperbolic gated recurrent unit, a hyperbolic vector representation is obtained. Subsequently, this invention proposes a timestamp prediction module that uses the hyperbolic space representation of event pairs to predict the relative time of the event pairs. This time is not represented by real time or time format, but by a real number. By comparing the magnitudes of these real numbers, the order of occurrence of the event pairs is determined, which to some extent resists noise problems from the commonsense knowledge base and enhances the model's prediction performance.

[0007] The model of this invention is mainly divided into the following four layers: (1) Pre-training layer: The Roberta-base pre-training model is used to learn the vector representation of text in Euclidean space; (2) Hyperbolic space layer: The output of the pre-training layer is put into the Poincaréball and hyperbolic gated loop unit for processing to obtain the vector representation of event pairs in hyperbolic space; (3) Feature fusion layer: The vector representation of event pairs is input into the timestamp module and the distance feature module respectively. After obtaining the output of the two modules, it is fused with the event pair vector and the common sense knowledge base vector; (4) Logistic regression layer: Hyperbolic polynomial logistic regression layer is used for prediction to obtain four-class labels.

[0008] 1) Pre-training layer

[0009] This invention directly inputs the dataset text into Roberta for pre-training and fine-tuning. First, it gives a sentence T = {T1, T2, ..., T...} of length n containing two events. nUsing a Roberta-based pre-trained model, 15% of the words in the input sentence are randomly masked. After fine-tuning and training, the context embedding E = {E1, E2, ..., E} for each word is obtained. n}

[0010] 2) Hyperbolic Space Layer

[0011] The pre-trained layer outputs a sentence vector representation based on Euclidean space, which is then mapped to a hyperbolic space-based vector representation in a hyperbolic space layer. This invention employs a Poincaré ball model for the mapping operation, which is an n-dimensional open unit sphere. Its associated Riemannian metric tensor is

[0012]

[0013] in g E It is the Euclidean metric tensor. For two points x and y in hyperbolic space, the distance between them can be expressed by the following formula:

[0014]

[0015] An exponential mapping can be defined for each point, which will map any point... Mapped to Therefore, the present invention can obtain the sentence representation in hyperbolic space through Formula 3.

[0016]

[0017] This invention represents the context of a sentence as a matrix. Where l represents the sentence length and d represents the dimension of word embeddings. After completing the mapping from Euclidean space to hyperbolic space, this invention obtains the sentence representation s in hyperbolic space. t This yields the embedding representation of each word in the sentence. The output is fed into a hyperbolic gated recurrent unit (HGRU) to derive the hidden state of each word. The positional masking vector m is then used... i or m j Its value is 1 at the event location and 0 at non-event locations. The event pair (H) is calculated using Formula 4. i H j ).

[0018] H v =HGRU(exp0(B))·m v (4)

[0019] 3) Feature fusion layer

[0020] For c≥0, it is represented as If c = 0, then If c > 0, then For a radius of If c = 1, then a normal ball is obtained.

[0021]

[0022] The external knowledge feature CSE, distance feature DSP and timestamp feature TSM are fused using formula (6) and then passed to the logistic regression layer for final prediction.

[0023]

[0024] Time is a crucial feature for classifying the temporal relationships of events; however, text contains very little temporal information, with only 20% of sentences in TimeBank containing such information. Time information directly given in a sentence is called explicit time, while time information obtained indirectly or through semantic logic analysis is called implicit time. For explicit time, this invention can directly construct a timeline to predict the temporal relationships of events. Following this approach, this invention can transform implicit time into the relative position information of events on the timeline through a timestamp prediction module by fusing semantic and syntactic structural information. The timestamp prediction module uses the event pair representation vectors obtained from the hyperbolic space layer as input. In CNNs, Dropout randomly discards some neural network units with a certain probability during each batch of training, effectively improving the training speed of the neural network while preventing overfitting. In natural language processing tasks, models often have high dimensionality, resulting in even larger-dimensional feature vectors after mapping to hyperbolic space. After receiving the vector representation from the hyperbolic space layer, the timestamp prediction module uses the Dropout algorithm to randomly discard features, ignoring nodes with weak representation capabilities. The network calculation formula using Dropout is shown below.

[0025]

[0026]

[0027]

[0028] The Bernoulli function in the above formula is used to generate a probability vector r, thus obtaining a randomly generated 0-1 vector. Multiplying the probability vector r with the feature vector y performs the random discarding of features.

[0029] 4) Logistic Regression Layer

[0030] The logistic regression layer directly uses a multinomial logistic regression algorithm based on hyperbolic space to obtain the classification results, where k represents the number of categories.

[0031]

[0032] In this invention, k is set to 4, and the probability of each category is predicted. The one with the highest probability is the temporal relationship of the event pair.

[0033] The method of this invention can effectively perform event temporal relationship classification, and has the following advantages compared with the prior art:

[0034] 1. This invention proposes a hyperbolic space-based representation method that maps word vectors from Euclidean space to hyperbolic space for feature extraction. This method can fully explore the hierarchical relationships in the text and provide a deeper understanding of the semantics.

[0035] 2. This invention solves the problem of statistical imbalance in simple counting models caused by introducing a common sense knowledge base in the task of classifying event time sequence relationships. It has a stronger understanding of local data, strengthens the weight of the local dataset to resist the introduced external noise, and improves the classification performance of the model. Attached Figure Description

[0036] Figure 1 This is a diagram of the overall framework of the hyperbolic space-based timestamp-enhanced event temporal relationship classification method of the present invention, which is mainly divided into four parts: pre-training layer, hyperbolic space layer, feature fusion layer and logistic regression layer.

[0037] Figure 2 This is a schematic diagram of the timestamp prediction module of the present invention, which describes the processes of timestamp prediction, such as dimensionality reduction, activation, and prediction.

[0038] Figure 3 The results obtained by this invention using different dropout rates on the MATRES dataset show that the best results are achieved when the dropout rate is 0.5.

[0039] Figure 4 The results obtained by this invention using different dropout rates on the TCR dataset show that the best results are achieved when the dropout rate is 0.5.

[0040] Figure 5 The experimental results of this invention on the MATRES and TCR datasets were compared with other baseline models, showing that the invention has better performance in the task of classifying event temporal relationships.

[0041] Figure 6To illustrate the obfuscation experiment results of this invention, experiments were conducted by removing the timestamp prediction module, the distance feature module, the common sense knowledge base module, and replacing the hyperbolic space timestamp prediction module with the Euclidean space timestamp prediction module. Detailed Implementation

[0042] To make the above-mentioned features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings. The specific training process is as follows: Figure 1 As shown, its main steps include:

[0043] Step 101: The corpus in this paper is fine-tuned using a Roberta-base pre-trained model to obtain a Euclidean space vector representation.

[0044] Step 102: Map the vector representation of Euclidean space to hyperbolic space using the Poincaré sphere model operator.

[0045] Step 201: Train the hyperbolic space neural network to learn text features and deeply mine semantic and syntactic information.

[0046] Step 301: Input the feature vectors trained in hyperbolic space into the distance feature module and the timestamp prediction module.

[0047] Step 302: After fusing the features obtained from the two modules in step 301 with the features from the common sense knowledge base, the mixture is input into the logistic regression layer for prediction.

[0048] Step 401: Perform multinomial logistic regression calculation on the fused features to obtain the prediction results, and classify the results.

[0049] This invention uses the publicly available MATRES and TCR datasets for training and validation. Experimental results show that the invention uses hyperbolic space for sentence embedding training, considering the semantic and syntactic relationships of event words, which facilitates understanding the text context. Event timestamp prediction provides the chronological order of events, which to some extent strengthens the weight of the local dataset. Combined with a commonsense knowledge base, this improves the performance of the event temporal relationship classification method proposed in this invention.

[0050] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for classifying event temporal relationships based on timestamp enhancement in hyperbolic space, characterized in that, include: A. For text data in the corpus, a Roberta-base pre-trained model is used to vectorize the text and learn the vector representation of the text in Euclidean space. A Poincaré sphere model is used to perform a mapping operation to map the vector representation in Euclidean space to hyperbolic space. The obtained vector representation is then input into a gated recurrent convolutional unit for convolution operation. B. For the word vector representation in the obtained hyperbolic space, feature learning is performed using a distance module and a timestamp prediction module. The distance module is responsible for calculating the similarity distance between two word vectors in the hyperbolic space, and the timestamp prediction module predicts the relative timestamps of two event word vectors based on the context information in the word vectors and represents them with real values. C. The feature vectors obtained from the distance module and the timestamp prediction module are fused with common sense knowledge features. The fusion algorithm is as follows: Where CSE represents common sense knowledge base features, TSM represents timestamp features, and DSP represents distance features; Denotes the Möbius striping in hyperbolic space, for c≥0, as If c = 0, then If c > 0, then For a radius of The hyperbolic space; if c = 1, then a normal sphere is obtained. ; D. The fused features are used as input to the logistic regression layer, and the classification result is obtained through the multinomial logistic regression algorithm.

2. The event temporal relationship classification method based on hyperbolic space timestamp enhancement according to claim 1, characterized in that, Step A further includes the following steps: A1. Preprocess and perform lexical and syntactic analysis on unstructured data to obtain the dataset required for the model; A2. Input the data into the Roberta-base pre-trained model and fine-tune the training to obtain a vector representation based on Euclidean space; A3. Using the Poincaré sphere model, vector representation is used to map from Euclidean space to hyperbolic space; for an n-dimensional open unit sphere... Its associated Riemannian metric tensor is （1） in , , It is a Euclidean metric tensor; for two points in hyperbolic space and The distance between them is expressed by the following formula: (2) Define an exponential mapping for each point, which will map any point... Mapped to Above, the sentence representation in hyperbolic space is obtained using Formula 3: (3) Represent the context of a sentence as a matrix ,in Represents the length of the sentence. This represents the dimension of word embeddings; after completing the mapping from Euclidean space to hyperbolic space, we obtain the representation of the sentence in hyperbolic space. This yields the embedding representation of each word in the sentence; the output is then passed to a hyperbolic gated loop unit. This is used to derive the hidden state of each word; the positional masking vector is then used. or Its value is 1 at the event location and 0 at non-event locations, using formula 4: （4） Calculate event pairs .

3. The event temporal relationship classification method based on hyperbolic space timestamp enhancement according to claim 1, characterized in that, Step D further includes the following steps: D1. Use the feature vector W obtained after fusion in C as the input to the logistic regression layer; D2. The logistic regression layer directly uses a multinomial logistic regression algorithm based on hyperbolic space to obtain classification results, given... Indicates the number of categories. , , (5) D3. Predict the probability of each category, and the one with the highest probability is the temporal relationship of that event pair.

Images