Event extraction method and system based on multi-granularity fusion and scene graph enhancement

Abstract

The invention belongs to the technical field of event extraction, and provides an event extraction method based on multi-granularity fusion and scene graph enhancement, and the method comprises the steps: obtaining the text information and image information of a to-be-extracted event; obtaining a text embedding representation and an image embedding representation based on the obtained text information and image information; performing unified embedding representation on the obtained image embedding representation and the text embedding representation to obtain a text embedding vector and an image embedding vector; performing multi-granularity fusion on the obtained text embedding representation and image embedding representation to obtain multi-granularity fusion embedding representation comprising coarse granularity embedding representation and fine granularity embedding representation; splicing the multi-granularity fusion embedding representation with the text embedding representation and the image embedding representation to obtain a text fusion vector and an image fusion vector, and combining a classifier to obtain a text event type and an image event type; and calculating a similarity score of the image embedding vector and the text embedding vector, and determining an extraction type and argument information of the event according to the obtained similarity score to complete event extraction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of event extraction, and particularly relates to an event extraction method and system based on multi-granularity fusion and scene graph enhancement. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Traditional event extraction methods are limited to processing single modal data such as images, texts or videos, aiming to identify, classify and extract event information from unstructured or semi-structured texts and represent them in a structured form. Nowadays, events tend to be presented in multimedia forms, providing richer event descriptions. To cope with this challenge, the multi-modal event extraction task emerges, aiming to jointly extract text and image event information from multimedia posts, with event types identified by triggers such as verbs and event arguments including time and location. The multi-modal event extraction task can more comprehensively capture event elements and provide structured event representations for downstream tasks such as event analysis and event graph construction.

[0004] Compared with the single modal event extraction task, the biggest challenge of the multi-modal event extraction task is how to fuse different modal data, utilize their complementary information and improve the accuracy and completeness of event extraction. Existing different modal data fusion methods mainly rely on explicit co-occurrence information of images and texts, i.e. the information co-occurring in different modal data is fused to obtain cross-modal fusion features by using simple splicing or attention mechanism, without effectively modeling the implicit context background association, so that the performance of the existing methods significantly decreases in complex scenarios with sparse co-occurrence information of images and texts.

[0005] Although there are many multi-modal event extraction methods based on images and texts, there are still the following problems in utilizing image information: existing methods only extract isolated objects in images such as "people", "ships" and other discrete entities, ignoring the spatial and semantic interaction relationship between objects. The interaction between objects can dynamically describe the occurrence of related events, which is crucial for a complete description of an event. In addition, the arguments of objects and events correspond directly, and the interaction information between objects provides a key clue for determining the event type, which is crucial for extracting comprehensive and accurate event argument information. SUMMARY

[0006] To solve the above problems, the application provides an event extraction method and system based on multi-granularity fusion and scene graph enhancement, which adopts multi-granularity fusion to fuse image and text data, in coarse-granularity fusion, dynamically balances the contribution of modalities according to the correlation between modalities, and focuses on the co-occurrence information between image and text; in fine-granularity fusion, text words and image regions are uniformly represented as nodes by constructing a multi-modal graph, a multi-layer graph convolutional neural network is used to mine the correlation of background information of the graph and text, and the model is helped to further determine the event type and argument role by using the background information; the context representation of object interaction is obtained by constructing a visual scene graph using a graph convolutional neural network, a structured representation of visual events is generated, the event representation is enriched, and the accuracy of the event extraction task is improved.

[0007] According to some embodiments, the first aspect of the application provides an event extraction method based on multi-granularity fusion and scene graph enhancement, which adopts the following technical scheme: An event extraction method based on multi-granularity fusion and scene graph enhancement, comprising: obtaining text information and image information of an event to be extracted; based on the obtained text information and image information, obtaining text embedding representation and image embedding representation, wherein the obtained image information is encoded by a visual encoder, a visual scene graph is constructed, and the image embedding representation is obtained according to the visual scene graph and a graph convolutional neural network; unifying the obtained image embedding representation and text embedding representation to obtain text embedding vectors and image embedding vectors; performing multi-granularity fusion on the obtained text embedding representation and image embedding representation to obtain multi-granularity fusion embedding representation including coarse-granularity embedding representation and fine-granularity embedding representation; splicing the multi-granularity fusion embedding representation with the text embedding representation and the image embedding representation respectively to obtain text fusion vectors and image fusion vectors, and obtaining text event types and image event types in combination with a classifier; calculating the similarity score of the image embedding vectors and the text embedding vectors, determining the type and argument information of the event according to the obtained similarity score, and completing event extraction.

[0008] It should be noted that after the multi-granularity fusion embedding representation is spliced with the text embedding representation and the image embedding representation respectively, the text fusion vectors and the image fusion vectors are obtained, and the obtained text fusion vectors and image fusion vectors are input into the corresponding classifier for classification, and the obtained classification results are the text event types and the image event types.

[0009] As a further technical limitation, in the process of obtaining the unified embedding representation, a multi-modal training large model is adopted, a cosine similarity between the text embedding representation and the image embedding representation is calculated to learn an alignment relationship between the image and the text, and the text embedding representation and the image embedding representation are mapped to the same vector space to obtain the image embedding vector and the text embedding vector of the same dimension vector representation.

[0010] As a further technical limitation, the multi-granularity fusion includes coarse-grained fusion and fine-grained fusion; the coarse-grained fusion focuses on co-occurrence information in image information and text information, and adopts a cross-attention mechanism and a filtering mechanism to obtain a cross-modal fusion representation; and the fine-grained fusion focuses on background information in the image information and the text information, and excavates an implicit correlation between the image and the text.

[0011] As a further technical limitation, when a similarity score of the obtained text embedding vector and the image embedding vector exceeds a score threshold, the text and the image are of the same event type, i.e., a multi-modal event, and the argument information is merged information of the image argument information and the text argument information; otherwise, the event is single-modal information, and the image argument information and the text argument information are not merged.

[0012] As a further technical limitation, the similarity score of the text embedding vector and the image embedding vector is an inner product of an interaction vector of the text embedding vector and the image embedding vector.

[0013] As a further technical limitation, in the process of obtaining the image embedding representation, object information in the image is extracted by an object detector, an interaction relationship between objects is extracted by an object relationship detector, an image scene graph is generated by using the objects and the relationship, a structured representation between the objects and the objects is obtained, and the obtained structured representation is spliced to obtain the image embedding representation.

[0014] According to some embodiments, a second aspect of the present application provides an event extraction system based on multi-granularity fusion and scene graph enhancement, which adopts the following technical solution: An event extraction system based on multi-granularity fusion and scene graph enhancement, comprising: An acquisition module configured to acquire text information and image information of an event to be extracted; A processing module configured to obtain a text embedding representation and an image embedding representation based on the acquired text information and image information, wherein the acquired image information is encoded by a visual encoder, a visual scene graph is constructed, an image embedding representation is obtained according to the visual scene graph and a graph convolutional neural network, and the obtained image embedding representation and text embedding representation are unified to obtain a text embedding vector and an image embedding vector; The fusion module is configured to perform multi-granularity fusion on the obtained text embedding representation and image embedding representation to obtain a multi-granularity fused embedding representation including coarse-grained embedding representation and fine-grained embedding representation; The concatenation module is configured to concatenate the multi-granularity fusion embedding representation with the text embedding representation and the image embedding representation respectively to obtain the text fusion vector and the image fusion vector, and combine them with the classifier to obtain the text event type and the image event type; The extraction module is configured to calculate the similarity score between the image embedding vector and the text embedding vector, determine the extraction type and argument information of the event based on the obtained similarity score, and complete the event extraction.

[0015] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium, employing the following technical solution: A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the event extraction method based on multi-granularity fusion and scene graph enhancement as described in the first aspect of the present invention.

[0016] According to some embodiments, the fourth aspect of the present invention provides an electronic device, which adopts the following technical solution: An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps in the event extraction method based on multi-granularity fusion and scene graph enhancement as described in the first aspect of the present invention.

[0017] According to some embodiments, the fifth aspect of the present invention provides a computer program product, which adopts the following technical solution: A computer program product includes software code, wherein the program in the software code performs the steps of the event extraction method based on multi-granularity fusion and scene graph enhancement as described in the first aspect of the present invention.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention employs a multi-granularity fusion approach to enhance the ability to process complex event instances. Coarse-grained fusion focuses on co-occurrence information between images and text, while fine-grained fusion focuses on the correlation between image and text background information. Scene graphs are introduced into the event extraction task to model spatial relationships and semantic interactions between objects, generating structured image event representations and enriching event representations. This improves the performance of multimodal event extraction tasks in terms of both comprehensiveness and accuracy. Attached Figure Description

[0019] The accompanying drawings, which form part of this embodiment, are used to provide a further understanding of this embodiment. The illustrative embodiments and their descriptions are used to explain this embodiment and do not constitute an improper limitation of this embodiment.

[0020] Figure 1 This is a flowchart of an event extraction method based on multi-granularity fusion and scene graph enhancement according to an embodiment of the present invention; Figure 2 This is an architecture diagram of the event extraction method based on multi-granularity fusion and scene graph enhancement in Embodiment 1 of the present invention; Figure 3 This is a structural block diagram of the event extraction system based on multi-granularity fusion and scene graph enhancement in Embodiment 2 of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] Example 1 Embodiment 1 of this invention introduces an event extraction method based on multi-granularity fusion and scene graph enhancement.

[0026] Existing fusion methods primarily rely on displayed image-text co-occurrence information for alignment and fusion to obtain cross-modal fusion features. While these methods perform well for instances with abundant image-text co-occurrence information, in complex scenarios with sparse image-text co-occurrence information and consistent background information, this embodiment adds a fine-grained fusion method to obtain comprehensive event information, building upon existing cross-attention fusion methods. This allows direct interaction between image and text vectors and utilizes graph convolutional neural networks to mine the correlation between image and text background information, helping the model further determine event types and argument roles. Furthermore, based on cross-attention, a filtering mechanism is added to further filter out other modal features with low contribution to text or image understanding, enabling the model to accurately capture the contribution information between images and text and improving robustness to noisy data.

[0027] To address the problem of neglecting the spatial and semantic interactions between objects in an image and failing to adequately describe dynamic event information, this embodiment adds a scene graph enhancement module to the traditional approach of only globally encoding the image. In constructing the scene graph, an object detector is used to extract objects from the image, and an object relationship detector is used to extract the relationships between objects, thereby constructing a visual scene graph. Furthermore, a graph convolutional network (GCN) is used to learn the spatial associations and semantic interactions between objects, and the contextual representation of object interactions is obtained using GCN, helping the model to accurately and comprehensively extract event arguments.

[0028] like Figure 1 The event extraction method shown includes multi-granularity fusion and scene graph enhancement. Obtain the text and image information of the event to be extracted; Based on the acquired text and image information, text embedding representation and image embedding representation are obtained. Specifically, the acquired image information is encoded by a visual encoder to construct a visual scene graph. Based on the visual scene graph and the graph convolutional neural network, the image embedding representation is obtained. The obtained image embedding representation and text embedding representation are unified into a single embedding representation to obtain text embedding vector and image embedding vector; The obtained text embedding representation and image embedding representation are fused at multiple granularities to obtain a multi-granularity fused embedding representation that includes coarse-grained embedding representation and fine-grained embedding representation; The multi-granularity fusion embedding representation is concatenated with the text embedding representation and the image embedding representation respectively to obtain the text fusion vector and the image fusion vector. The text event type and the image event type are obtained by combining them with the classifier. Calculate the similarity score between the image embedding vector and the text embedding vector, determine the event extraction type and argument information based on the obtained similarity score, and complete the event extraction.

[0029] This embodiment employs a multi-granularity fusion and scene graph enhancement strategy, enabling the model to adapt to both instances with significant contributing information and complex event instances where text and image co-occurrence information is not obvious. This allows for the acquisition of rich visual image information, improving the accuracy and comprehensiveness of event extraction. Specifically, the event extraction method based on multi-granularity fusion and scene graph enhancement in this embodiment includes the following steps: Step 1: Receive text and image information for event extraction; The multimodal event extraction task aims to enable computers to jointly extract text and image event information from multimedia posts. Event types are identified by triggers such as verbs, and event arguments include time and location. Multimodal event extraction can capture event elements more comprehensively, providing structured event representations for downstream tasks such as event analysis and event graph construction.

[0030] The input to the multimodal event extraction task is a multimodal post, consisting of a set of images and a set of sentences. After a series of system processing steps such as feature extraction and fusion, a structured event type and event argument information is provided as the output.

[0031] Step 2: Based on the text information of the event, obtain the implicit text representation using a text encoder; based on the image information of the event, obtain the global implicit image representation using a multimodal large model; and simultaneously, based on the object detector and the scene graph generation model, obtain the structured representation of the interaction between objects. The two are then concatenated to obtain the final image embedding representation. For obtaining the implicit text representation, a pre-trained text encoder, BERT, is used to process the input text. Encoding BERT gives it a significant advantage in learning dependencies between words. This involves obtaining the hidden vector representation for each word. : ; in, , indicating that it contains n d-dimensional word vectors. This represents each word.

[0032] For the acquisition of implicit image representation, for images The image is encoded using the visual encoder CLIP as a global visual representation.

[0033] ; Faster R-CNN is used to identify a set of object regions and their bounding box coordinates in an image. An object relationship detection model is used to identify the relationship between each pair of objects, and the above information is used to construct a visual scene graph.

[0034] For image I, the set of objects is represented as The set of relations is represented as ,in Representation Object and The relationships between them are defined in the scene graph representation as follows: Where N represents the set of objects, This represents a set of directed edges. For example, if... and If a relationship exists, a [relationship] will be generated. point to The edge.

[0035] Next, GCN is used to aggregate neighborhood information to learn a context-aware representation for each object, the first... Each node in the layered GCN is continuously updated based on the representations of its neighbors. Formally, ; in, Indicates that object i has passed the first time. Embedded representation of layered GCN, Represents a normalized matrix. , Indicates the number of objects. Indicates the GCN network's first The training weights of the layers are used. LeakyReLU is used as the activation function. In this module, to capture deeper interactions between objects and maintain the stability of the GCN model, the output of two GCN layers is used as the feature representation of object i, i.e. .

[0036] To help the model understand the spatial relationships between objects, a fully connected (FC) layer is used to apply a graph embedding representation to each object and its corresponding bounding box coordinates, generating the final embedding representation of object i. .formal, ; in, , These are fully connected layers that process objects and their bounding boxes, where `box` represents the bounding box coordinates of the object.

[0037] Obtain the representation of each local object in the image Then, with the global embedding representation of the image. The final image representation is obtained by concatenating the images using the concat method. The format is as follows:

[0038] Step 3: Feed the unified text and image embedding representations into the multi-granularity fusion module to obtain coarse and fine granularity embedding representations respectively; In the coarse-grained fusion module, a filtering mechanism is introduced on the basis of cross-attention fusion to filter information in the image that has a positive contribution to text understanding, so that the model can accurately capture the co-occurrence information between the image and the text and improve the robustness to noisy data.

[0039] In the cross-attention module, the visual features obtained from the alignment module are used. The text representation obtained by the alignment module as a key-value vector Used as a query vector, the text representation is used to focus on the most relevant visual features, generating a text-aware visual representation c.

[0040] ; in, Represents the query vector. These are the key vector and the value vector, respectively. This represents the dimension of the K vector.

[0041] A filtering mechanism is introduced to further filter visual features that contribute little to text understanding. Specifically, text and image features are first further processed to generate complex text and image representations, respectively. , The calculation is as follows: ; ; in, This indicates a global average pooling operation. The Multilayer Perceptron (MLP) uses ReLU as its activation function.

[0042] Next, the global correlation between the image and the text is calculated to obtain the correlation matrix m, which is used to control the weights of the fused feature c and filter out visual features that contribute less to text understanding. m is calculated by the following formula: ; Here, matmul represents the product operation.

[0043] Filtered text cross-modal features It is expressed as follows: ; Similarly, the coarse-grained fusion module is used to obtain the filtered visual cross-modal features. .

[0044] To address the problem that coarse-grained fusion methods struggle to capture implicit relationships between images and text when co-occurrence information is not readily apparent, this embodiment employs a fine-grained fusion module. This module constructs a multimodal graph and uses GCN to capture implicit background semantic relationships between images and text. Specifically, constructing a multimodal graph Vertex set ,in This represents the vector representation of the i-th image patch after the alignment module. Represents the edge set. To achieve deep interaction between images and text, in... During edge construction, each word vector representation is associated with all patch blocks in the image, and adjacent nodes within each modality are also connected. Each edge in the graph is assigned a weight, denoted as M. The relevance representation of words to image patches calculated by CLIP.

[0045] GCN is used to generate the graph context representation of each node. To alleviate the information loss problem in deep networks, residual connections are added to the output of each GCN layer and fused with the historical features of the previous layer, as follows: ; in, , These represent the feature representations of the i-th word and the j-th patch after passing through two layers of GCN, respectively.

[0046] Ultimately, Average pooling is performed on all node features to obtain fine-grained fused features. .

[0047] ; in, This indicates the average pooling operation.

[0048] Step 4: Concatenate the coarse and fine granularity embedding representations with the text embedding representation and feed them into the text event type and argument classifier to obtain text event type and related argument information; Specifically, event trigger identification is performed by predicting event types using a classification approach. The classifier predicts the event type of word i in the text based on the combined features of the word i. Among them, the comprehensive features of words It is based on the characteristics of the original word coarse-grained text fusion features and fine-grained fusion features It is pieced together. In terms of form, ; ; in, It contains a hidden layer and a Softmax layer.

[0049] For each entity in the text, we first average all its token embeddings to obtain the entity representation. , Predict the role type of each entity in the event. .

[0050]

[0051] in, .

[0052] Given event type and parameter types The cross-entropy loss function is used to calculate the loss of the text event classifier and the parameter classifier. and ,formal: ; ; Where n and k represent the number of words and entities, respectively.

[0053] Step 5: Concatenate the coarse and fine granularity embedding representations with the image embedding representation and feed them into the text event type and argument classifier to obtain image event type and related argument information; Specifically, integrating image features Send to another event classifier Classify events in the image, where From original features Image coarse-grained fusion features Fine-grained fusion features It is pieced together, and its form is as follows: ; ; in, This indicates the prediction results for visual event types. It contains a hidden layer and a Softmax layer. Argument recognition is performed on an object-by-object basis. Object features are synthesized. Send in The classifier predicts the role type of object i. Wherein, From the object's original characteristics , and It was pieced together.

[0054] ; in, This represents the prediction result for the role type of object i in the image.

[0055] Given event type and parameter types The loss is evaluated using the cross-entropy loss function. and ,formal: ; ; Where n represents the number of objects in the image.

[0056] Therefore, the overall objective of MEE task optimization is formally expressed as: : ; Step 6: Calculate the similarity score of the embedded representations of the image and the text. Based on the score, determine whether the event is a multimodal event. If so, the event type and related argument information of the text event information acquisition unit and the image event information acquisition unit are merged; otherwise, they are not merged.

[0057] For multimedia events, the coreference problem between text events and visual events needs to be addressed. Given a multimedia post, the CLIP model is used to calculate the cosine similarity of each sentence-image pair. Text events and visual events are considered the same event if they have the same event type and the similarity between the sentence and the image is greater than a threshold. Multimedia events inherit all text and visual event parameters as multimedia event parameters.

[0058] This embodiment employs a multi-granularity fusion approach to enhance the ability to process complex event instances. Coarse-grained fusion focuses on co-occurrence information between images and text, while fine-grained fusion focuses on the correlation between image and text background information. Scene graphs are introduced into the event extraction task to model spatial relationships and semantic interactions between objects, generating structured image event representations and enriching event representations. This improves the performance of multimodal event extraction tasks in terms of both comprehensiveness and accuracy.

[0059] Example 2 Embodiment 2 of the present invention introduces an event extraction system based on multi-granularity fusion and scene graph enhancement.

[0060] like Figure 3 The event extraction system shown includes: multi-granularity fusion and scene graph enhancement. The acquisition module is configured to acquire text and image information of the event to be extracted; The processing module is configured to obtain text embedding representations and image embedding representations based on the acquired text and image information. Specifically, the acquired image information is encoded by a visual encoder to construct a visual scene graph. Based on the visual scene graph and a graph convolutional neural network, an image embedding representation is obtained. The obtained image embedding representations and text embedding representations are then unified into a single embedding representation to obtain text embedding vectors and image embedding vectors. The fusion module is configured to perform multi-granularity fusion on the obtained text embedding representation and image embedding representation to obtain a multi-granularity fused embedding representation including coarse-grained embedding representation and fine-grained embedding representation; The concatenation module is configured to concatenate the multi-granularity fusion embedding representation with the text embedding representation and the image embedding representation respectively to obtain the text fusion vector and the image fusion vector, and combine them with the classifier to obtain the text event type and the image event type; The extraction module is configured to calculate the similarity score between the image embedding vector and the text embedding vector, determine the extraction type and argument information of the event based on the obtained similarity score, and complete the event extraction.

[0061] The detailed steps are the same as those of the event extraction method based on multi-granularity fusion and scene graph enhancement provided in Example 1, and will not be repeated here.

[0062] Example 3 Embodiment 3 of the present invention provides a computer-readable storage medium.

[0063] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the event extraction method based on multi-granularity fusion and scene graph enhancement as described in Embodiment 1 of the present invention.

[0064] The detailed steps are the same as those of the event extraction method based on multi-granularity fusion and scene graph enhancement provided in Example 1, and will not be repeated here.

[0065] Example 4 Embodiment 4 of the present invention provides an electronic device.

[0066] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, it implements the steps in the event extraction method based on multi-granularity fusion and scene graph enhancement as described in Embodiment 1 of the present invention.

[0067] The detailed steps are the same as those of the event extraction method based on multi-granularity fusion and scene graph enhancement provided in Example 1, and will not be repeated here.

[0068] Example 5 Embodiment 5 of the present invention provides a computer program product.

[0069] A computer program product includes software code, wherein the program in the software code executes the steps of the event extraction method based on multi-granularity fusion and scene graph enhancement as described in Embodiment 1 of the present invention.

[0070] The detailed steps are the same as those of the event extraction method based on multi-granularity fusion and scene graph enhancement provided in Example 1, and will not be repeated here.

[0071] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0072] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0076] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0077] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. An event extraction method based on multi-granularity fusion and scene graph enhancement, characterized in that, include: Obtain the text and image information of the event to be extracted; Based on the acquired text and image information, text embedding representation and image embedding representation are obtained. Specifically, the acquired image information is encoded by a visual encoder to construct a visual scene graph. Based on the visual scene graph and the graph convolutional neural network, the image embedding representation is obtained. The obtained image embedding representation and text embedding representation are unified into a single embedding representation to obtain text embedding vector and image embedding vector; The obtained text embedding representation and image embedding representation are fused at multiple granularities to obtain a multi-granularity fused embedding representation that includes coarse-grained embedding representation and fine-grained embedding representation; The multi-granularity fusion embedding representation is concatenated with the text embedding representation and the image embedding representation respectively to obtain the text fusion vector and the image fusion vector. The text event type and the image event type are obtained by combining them with the classifier. Calculate the similarity score between the image embedding vector and the text embedding vector, determine the event extraction type and argument information based on the obtained similarity score, and complete the event extraction.

2. The event extraction method based on multi-granularity fusion and scene graph enhancement as described in claim 1, characterized in that, In the process of unified embedding representation, a large model is trained using multimodal methods. By calculating the cosine similarity between the text embedding representation and the image embedding representation, the alignment relationship between the image and the text is learned. The text embedding representation and the image embedding representation are mapped to the same vector space, resulting in image embedding vectors and text embedding vectors with the same dimension.

3. The event extraction method based on multi-granularity fusion and scene graph enhancement as described in claim 1, characterized in that, The multi-granularity fusion includes coarse-grained fusion and fine-grained fusion; the coarse-grained fusion focuses on co-occurrence information in image information and text information, and uses cross-attention mechanism and filtering mechanism to obtain cross-modal fusion representation; the fine-grained fusion focuses on background information in image information and text information, and mines the implicit correlation between images and text.

4. The event extraction method based on multi-granularity fusion and scene graph enhancement as described in claim 1, characterized in that, When the similarity score between the obtained text embedding vector and the image embedding vector exceeds the score threshold, the text and the image are of the same event type, i.e., a multimodal event, and the argument information is the combined information of image argument information and text argument information; otherwise, the event is single-modal information, and the image argument information and text argument information are not combined.

5. The event extraction method based on multi-granularity fusion and scene graph enhancement as described in claim 1, characterized in that, The similarity score between the text embedding vector and the image embedding vector is the inner product of the interaction vectors of the text embedding vector and the image embedding vector.

6. The event extraction method based on multi-granularity fusion and scene graph enhancement as described in claim 1, characterized in that, In the process of obtaining the image embedding representation, object information in the image is extracted by an object detector, interaction relationships between objects are extracted by an object relationship detector, and an image scene graph is generated using objects and their relationships to obtain a structured representation between objects. The obtained structured representations are then concatenated to obtain the image embedding representation.

7. An event extraction system based on multi-granularity fusion and scene graph enhancement, characterized in that, include: The acquisition module is configured to acquire text and image information of the event to be extracted; The processing module is configured to obtain text embedding representations and image embedding representations based on the acquired text and image information. Specifically, the acquired image information is encoded by a visual encoder to construct a visual scene graph. Based on the visual scene graph and a graph convolutional neural network, an image embedding representation is obtained. The obtained image embedding representations and text embedding representations are then unified into a single embedding representation to obtain text embedding vectors and image embedding vectors. The fusion module is configured to perform multi-granularity fusion on the obtained text embedding representation and image embedding representation to obtain a multi-granularity fused embedding representation including coarse-grained embedding representation and fine-grained embedding representation; The concatenation module is configured to concatenate the multi-granularity fusion embedding representation with the text embedding representation and the image embedding representation respectively to obtain the text fusion vector and the image fusion vector, and combine them with the classifier to obtain the text event type and the image event type; The extraction module is configured to calculate the similarity score between the image embedding vector and the text embedding vector, determine the extraction type and argument information of the event based on the obtained similarity score, and complete the event extraction.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the event extraction method based on multi-granularity fusion and scene graph enhancement as described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the event extraction method based on multi-granularity fusion and scene graph enhancement as described in any one of claims 1-6.

10. A computer program product, comprising software code, characterized in that, The program in the software code executes the steps of the event extraction method based on multi-granularity fusion and scene graph enhancement as described in any one of claims 1-6.

Images