A social platform multi-modal unified information extraction method

Abstract

This invention implements a unified multimodal information extraction method for social media platforms. Inputting text and image information from the social media platform, the method performs entity and relation extraction for both text and images through a multimodal feature fusion module and a multimodal information intelligent extraction module. The unified information extraction module uses intelligent decision-making to identify and classify entities in the data, employs reinforcement learning-based intelligent decision-making to determine the execution order of extraction tasks, and uses Q-learning to perform multimodal information extraction and intelligent task selection to find the optimal task execution order. This invention fully utilizes the diversity of information modalities on social media, proposing to construct a heterogeneous graph from information of different modalities. Graph structure learning and causal intervention are used to optimize the graph structure and graph neural network, respectively, thereby obtaining text and image representations with richer semantics.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a method for extracting multimodal unified information from social platforms. Background Technology

[0002] Automatic information extraction from multimodal data is a challenging problem. When processing multimodal data, it's necessary to consider how to handle different data types, such as images, text, and audio, and transform them into a unified representation. Most existing methods perform cross-modal interaction representations of data from various modalities, then input them into downstream models to extract various types of structured information. However, this approach ignores the inherent connections and dependencies between multimodal data, making it difficult to learn multimodal features with strong correlations and failing to leverage the inherent connections and dependencies between multimodal extraction tasks. Furthermore, modeling relationships between different modalities has inherent flaws, such as error accumulation (errors in entity extraction affect subsequent relationship extraction), entity redundancy (candidate entities without relationships bring redundant information, increasing error rates and computational complexity), and missing interactions (ignoring the inherent connections and dependencies between tasks), thus degrading the performance of multimodal multi-task information extraction models. Summary of the Invention

[0003] To this end, this invention first proposes a multimodal unified information extraction method for social platforms. The method inputs text and image information from the social platform, and through a multimodal feature fusion module and a multimodal information intelligent extraction module, it achieves entity extraction and relation extraction from the text and images, thereby obtaining entity and relation information from the text and images. This helps users extract specific and useful information from the unstructured social media field more quickly, accelerates the speed and process of information filtering and processing, and enables rapid analysis and blocking of social data containing negative information.

[0004] The multimodal feature fusion module takes as input the original text and image information of the social platform. It preprocesses the data of each modality, uses the text and image information in the database to construct a heterogeneous multimodal graph of text and image representation, and then performs graph structure learning and causal feature perturbation to finally output causal text representation and causal image representation.

[0005] The unified information extraction module for intelligent decision-making takes the causal text representation and causal image representation as input, identifies and classifies entities in the data through entity extraction, and determines the relationship types between different entities through relation extraction. Based on historical extraction results, it uses the category set for the current extraction task. By restricting the extraction categories, it enhances the connection between various extraction tasks and reduces redundant entities in multimodal extraction tasks. It employs reinforcement learning-based intelligent decision-making to determine the execution order of extraction tasks. The Q-learning algorithm is used for reinforcement learning to extract multimodal information and intelligently select tasks to find the optimal task execution order. Specifically, the steps for determining the execution order are as follows:

[0006] Step 1: The reinforcement learning agent selects a task that has not been executed from the extracted task set;

[0007] Step 2: Perform the extraction task using an extraction model and save the extraction results. Use the F1 score of the extraction results as the reward for the reinforcement learning agent, and use the class results of all extraction tasks performed as the class set for subsequent extraction tasks;

[0008] Step 3: Repeat steps 1-2 until all extraction tasks in the set are completed;

[0009] Step 4: The agent selects the optimal task execution order and feeds back the loss of each task under the optimal execution order to the feature extractor and each task model.

[0010] The method for constructing the heterogeneous multimodal graph is as follows: A preliminary heterogeneous graph G, centered on text representation and image representation, is constructed. TI ;

[0011] Among them, the word representation E of the text modality word object representation E of image modality object They are considered as nodes, and the relationships between them are considered as edges;

[0012] Subsequently, regarding G TI Graph structure learning is performed to obtain a heterogeneous graph representation G with optimized structure. TI_pef ;

[0013] During the training process, with G TI_pef Based on this, a heterogeneous graph G representing environmental changes is obtained by altering the feature information of different nodes using a diffusion model. TI_env The graph G containing information about different nodes is constrained using the regularization term V_zz. TI_pef and G TI_env Thus, the causal text word representation E is obtained. word_casual and causal image object representation E object_casual .

[0014] The construction of the preliminary graphic representation heterogeneous graph G TI The specific process is as follows:

[0015] First, representation extraction is performed on the preprocessed information. For the text modality, an existing large-scale pre-trained model is used to obtain the representation of each text word. For the image modality, Faster R-CNN is used to obtain the representation of each object in the image. Then, based on the above two types of representations, a heterogeneous graph G of image and text representations is constructed. TI = (A, X), where A represents the adjacency matrix of the heterogeneous graph, which is a two-dimensional matrix whose each element A ij X represents whether there is an edge between node i and node j (1: exists, 0: does not exist); X is the feature vector of the node, which is a two-dimensional matrix, where each row Xi represents the feature vector of node i.

[0016] There are two types of nodes in a heterogeneous graph representation: text (word) representing information (E). word Image object representation information E object There are three edge types: image object representation - image object, image object representation - text word representation, and text word representation - text word representation.

[0017] Aside from the relationships inherent in the data, the rules for constructing other relationships are as follows:

[0018] d. Image-Image: When the similarity between two different objects in the same image reaches a set threshold, the two image objects are connected by an edge;

[0019] e. Image-Text: If the similarity between the representations of an image object and a text word reaches a set threshold, then connect the image object representation and the text word representation with an edge.

[0020] f. Text-Text: When the similarity between two different word representations in the same text reaches a set threshold, the two text word representations are connected by an edge.

[0021] The graph structure learning and causal feature perturbation process is based on a text-image modality alignment and fusion model framework that combines causal intervention and graph structure learning. Specifically, it first uses the existing IDGL graph structure learning method to perform graph structure learning on G... TI The structure is further optimized and learned to obtain G. TI_pef During training, based on the structure-optimized graph G TI_pef By using a diffusion model to modify the feature information of different nodes in a heterogeneous graph representing a graph, we obtain G. TI_envTo mimic graphs in different environments, a regularization term V_zz is then used to constrain the graph G containing information about different nodes. TI_pef and G TI_env , will G TI_pef Obtaining the causal text word representation E using a single-layer graph neural network. word_casual and causal image object representation E object_casual The specific calculation process is as follows:

[0022] V_zz=Var(G TI_pef G TI_env )

[0023] E word_casual E object_casual =GNN(G TI_pef )

[0024] Among them, GNN(G TI_pef ) represents G TI_pef Through a single layer of graph convolutional neural network, this module ultimately outputs the text word representation E. word Image object representation E object Causal text word representation E i,word_casual Causal image object representation E j,object_casual , where i represents the i-th morpheme and j represents the j-th target in the image.

[0025] The entity extraction is divided into multiple tasks, including entity extraction for text and object detection for images. The entity extraction model for text labels the type of each word in the text sequence by superimposing a fully connected layer and a CRF layer on the representation acquisition module, using cross-entropy as the loss function and the F1 score of multi-class classification as the evaluation metric. The entity classification for images superimposes N SVM classifiers on the representation acquisition module, where N is the number of classes, using cross-entropy as the loss function and the F1 score of multi-class classification as the evaluation metric.

[0026] The relation extraction is the determination of the relationship type between different entities. Depending on the different modalities from which the entities originate, relation extraction can also be divided into multiple tasks, such as relation extraction between text entities, relation extraction between image entities, and relation extraction between text entities and image entities. The relation extraction model superimposes a fully connected layer and a softmax layer on the representation acquisition module. After obtaining the representations of the two entities, they are concatenated and input into the fully connected layer and the softmax layer for classification. Cross-entropy is used as the loss function, and F1 multi-class classification is used as the evaluation metric.

[0027] The reinforcement learning agent is modeled using value function reinforcement learning, specifically through the following parameters:

[0028] State representation:

[0029] Consider the tasks in the entity extraction set as the states of the agent. Each task can be uniquely represented by its position in the set. Suppose there are n tasks in the set, then the state space is S = {s1, s2, ..., sn}. n};

[0030] Action representation:

[0031] An agent can choose tasks from the set that have not yet been executed, that is, select an action from the current state S. Let the current state be S, and the action chosen by the agent be a, then the action space is A(S) = {s}. i |s i ∈S};

[0032] Reward function:

[0033] The extraction result for each task is compared with the true label, and the F1 score of the extraction result is calculated as the agent's immediate reward. If the extraction result for task i is... The actual label is y i Then the immediate reward for task i is:

[0034]

[0035] Where Precision(·,·) and Recall(·,·) represent precision and recall, respectively.

[0036] State transition function:

[0037] After the agent performs action a, it enters the next state S′. Since each task can only be executed once, after performing action a, the task s in state S will change. i The state will be removed, and the next state S′ is Sa. Therefore, the state transition function is:

[0038]

[0039] Policy function:

[0040] The policy function is a function used in reinforcement learning to guide an agent to choose an action in a specific state. The policy function is usually represented by π(S,a), which is the probability of choosing action a in state S. The policy function uses a greedy policy.

[0041] Value functions:

[0042] Let the value function be represented as Q(S,a), which represents the long-term reward obtained by performing action a in state S. Since each task can only be performed once, the long-term reward that the agent can obtain after completing all tasks in the set is the sum of the immediate rewards of all tasks. The Q-learning algorithm learns the value function, and its update formula is: Q(S,a)=Q(S,a)+α[r+γmax] a′ [Q(S′,a′)-Q(S,a)]; where α is the learning rate and γ is the discount factor, used to control the importance of long-term rewards. The value function is updated after each task is completed, until all extraction tasks in the set are finished.

[0043] The technical effects to be achieved by this invention are as follows:

[0044] 1. This invention fully utilizes the diversity of information modalities on social media, and proposes to construct a heterogeneous graph by combining information from different modalities. The graph structure and graph neural network are optimized by graph structure learning and causal intervention, respectively, thereby obtaining text representations and image representations with richer semantics.

[0045] 2. This invention addresses the entity redundancy problem in multimodal information extraction by designing a constraint scheme for extraction task categories, thereby enhancing the connection between various extraction tasks. Simultaneously, it uses an intelligent decision-making method based on reinforcement learning to determine the optimal task execution order, improving the performance of the multimodal information extraction device. Attached Figure Description

[0046] Figure 1 Architecture of a unified multimodal information extraction method for social platforms;

[0047] Figure 2 The original input image in the example;

[0048] Figure 3 Image after region extraction;

[0049] Figure 4 The resulting graph-text representation heterogeneous graph;

[0050] Figure 5 Entity map extracted from the target image;

[0051] Figure 6 A relationship diagram extracted from the model. Detailed Implementation

[0052] The following are preferred embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0053] This invention proposes a method for extracting multimodal unified information from social platforms.

[0054] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments, so that those skilled in the art can implement it based on the description.

[0055] The example chosen in the implementation plan is "Entrepreneur Person A, his wife Person B, and entrepreneur Person C plant trees on the lawn in front of a big house" to illustrate the entire implementation process.

[0056] like Figure 1 As shown, this invention provides a method and apparatus for extracting multimodal unified information from social platforms based on graph causal reasoning and reinforcement decision-making. The input is the original multimodal information from the social platform, including all posted text and image information, and the output is the extracted structured data. This invention includes two modules:

[0057] I. Text-Image Modality Alignment Fusion and Representation Acquisition Module: First, the acquired raw multimodal information from social media platforms is preprocessed. Data preprocessing includes simple data cleaning, converting emojis to English words (using API calls), etc. Representation extraction is then performed using the preprocessed information. For the text modality, existing large-scale pre-trained models such as BERT are used to obtain the representation of each text word, transforming the input text into a feature vector: w = {w1, ..., w2}, where T is the number of words in the text. Furthermore, a [CLS] label is added before each sentence and a [SEP] label is added between every two sentences. The final word composition is obtained by adding the feature vector and the position vector, and then normalized through an LN layer. For... Figure 2 The sentence "Entrepreneur Person A, his wife Person B, and entrepreneur Person C plant trees on the lawn in front of a big house" can be broken down into 22 vectors according to the rule that each word corresponds to a vector. Taking "Person A" as an example, if the vector dimension is specified as 8, then after BERT text preprocessing, "Person A" will be transformed into:

[0058] w3={0.03213,0.02312,0.34349,-0.09734,-0.12932,0.78547,0.13214,0.25312}

[0059] Similarly, other words will also be converted into vectors of the same dimension. Thus, the final vector of this sentence is the w vector corresponding to the formula above.

[0060] For image modalities, Faster R-CNN is used to obtain the representation of each object in the image. Faster R-CNN feeds the image into a convolutional neural network to generate regions of interest (ROIs). Pooling layers are then applied to all ROIs, and the region sizes are adjusted to ensure each region has the same size. Each region is then fed into a fully connected network. Finally, existing SoftMax layers and linear regression layers are used simultaneously for classification, and bounding boxes are returned. This yields the visual features of each ROI and its corresponding feature vector.

[0061] In addition, a five-dimensional vector is used to encode the location features of each region, and the expression for this vector is:

[0062]

[0063] Where (x1, y1) and (x2, y2) are the coordinates of the lower left and upper right corners of the region, respectively, and W and H are the width and height of the region, respectively. Visual and positional features, after training through fully connected layers (FC layers), are projected into the same embedding space. The final vector representation of each region is obtained by adding the outputs of two FC layers and then passing the result through another LN layer. The final image region can then be represented as:

[0064] v = {v1, ..., v} I}

[0065] Where I is the number of regions extracted from this image. For Figure 2 The image in the image, the final extracted image region is as follows Figure 3 As shown.

[0066] Subsequently, based on the two types of representations mentioned above, a heterogeneous graph representation diagram is constructed, such as... Figure 4 As shown in the figure, the edges are connected according to the aforementioned rules as follows: if the similarity between two different object representations in the same image reaches 0.5, then the two image objects are connected; if the similarity between the image object and the text word representation reaches 0.35, then the image object representation and the text word representation are connected; if the similarity between two different word representations in the same text reaches 0.3, then the two text word representations are connected, such as “Person A” and “entrepreneur”, and “Person C” and “entrepreneur”.

[0067] against Figure 4The graph representation of heterogeneous graph G TI This invention proposes a text-image modality alignment and fusion model framework based on causal intervention and graph structure learning. Specifically, it first uses the existing IDGL graph structure learning method to perform graph alignment and fusion on G... TI The structure is further optimized and learned to obtain G. TI_pef During training, based on the structure-optimized graph G... TI_pef By using a diffusion model to modify the feature information of different nodes in a heterogeneous graph representing a graph, we obtain G. TI_env This is used to simulate graphs under different environments. Subsequently, a regularization term V_zz is used to constrain the graph G containing information about different nodes. TI_pef and G TI_env , will G TI_pef Obtaining the causal text word representation E using a single-layer graph neural network. word_casual and causal image object representation E object_casual The specific calculation process is as follows:

[0068] V_zz=Var(G TI_pef G TI_env )

[0069] E word_casual E object_casual =GNN(G TI_pef )

[0070] Among them, GNN(G TI_pef ) represents G TI_pef Through a single layer of graph convolutional neural network

[0071] Finally, this module also outputs the text word representation E. word Image object representation E object Causal text word representation E i,word_casual Causal image object representation E j,object_casual , where i represents the i-th morpheme and j represents the j-th target in the image.

[0072] II. Unified Information Extraction Module for Intelligent Decision Making: The input to this module is the multimodal feature vector obtained from the previous module. After the text and image data are processed by the multimodal alignment, fusion, and representation module, the causal text word representation E is obtained. i,word_casual and causal image object representation E j,object_casual The output of this module is the extracted structured data, which depends on the specific extraction task, including entity extraction and relation extraction.

[0073] The objective of entity extraction is to identify and classify entities present in data. Depending on the modality, entity extraction can be divided into several tasks, such as entity extraction for text and object detection for images. The specific implementation method for entity extraction for text is as follows:

[0074] 1. Overlay a fully connected layer and a CRF layer on the causal text word representation to label the type of each representation in the representation sequence. The labeling method is BIO labeling.

[0075] 2. During the training phase, cross-entropy is used as the loss function for optimization, and the F1 score for multi-class classification is used as the evaluation metric for model training and validation. During the testing phase, the text representation is input into the model for inference, and the final output is the labeled text sequence.

[0076] In the example above, the text entity extraction model can extract the following entities: People: Person A, Person B, Person C; Job title: entrepreneur; Location: big house.

[0077] The specific implementation method of the image target detection task is as follows:

[0078] 1. Stack N SVM classifiers on the causal image object representation, where N is the number of categories. Each SVM classifier is used to determine whether the current target object belongs to the corresponding category.

[0079] 2. During the training phase, cross-entropy is used as the loss function for optimization, and the F1 score for multi-class classification is used as the evaluation metric for model training and validation. During the testing phase, for each target object, its feature vector is input into the model for inference, and the final output is the category label of each target object.

[0080] In the example above, image object detection can extract entities such as Figure 5 As shown.

[0081] Relation extraction is the process of determining the types of relationships between different entities. Depending on the modality from which the entities originate, relation extraction can be divided into several tasks, such as relationship extraction between text entities and relationship extraction between image entities. The specific implementation methods for relation extraction tasks are as follows:

[0082] 1. Overlay a fully connected layer and a softmax layer on the representation acquisition model. After obtaining two entity representations, concatenate them and input them into the fully connected layer and softmax layer for classification.

[0083] 2. During training, cross-entropy is used as the loss function, and F1 multi-class classification is used as the evaluation metric. During testing, the representations of text or image entities are input into the relation extraction model, and the final classification result is output.

[0084] In the above example, the relationships that the model can extract from the text include: spousal relationship: Person A & Person B; job relationship: entrepreneur & Person A; entrepreneur & Person C; location relationship: Person A & big house, Person B & big house, Person C & big house. The relationships that the model can extract from the image are as follows: Figure 6 As shown:

[0085] This invention enhances the connection between various extraction tasks and reduces redundant entity problems in multimodal extraction tasks by restricting the extraction categories. In specific implementation, to achieve the restriction on entity types, this invention uses historical extraction results as the category set for the current extraction task. For example, if the execution order of the extraction tasks is: text entity extraction, image object detection, and the extraction results of text entity extraction include: people, locations, and job titles, then during the execution of the image object detection task, the model only considers classifying image entities into people, locations, and job titles, thereby achieving the restriction on entity types and improving the performance of the extraction model.

[0086] Determining the execution order of extraction tasks is a key problem addressed in this invention. This invention employs reinforcement learning-based intelligent decision-making to determine the execution order of extraction tasks, using the Q-learning algorithm to find the optimal task execution order. The specific implementation steps of this invention for multimodal information extraction and intelligent selection of task execution order using reinforcement learning are as follows:

[0087] Step 1: The reinforcement learning agent selects a task that has not been executed from the extracted task set.

[0088] Step 2: Perform the extraction task using an extraction model and save the extraction results. Use the F1 score of the extraction results as the reward for the reinforcement learning agent, and use the class results of all extraction tasks performed as the class set for subsequent extraction tasks.

[0089] Step 3: Repeat steps 1-2 until all extraction tasks in the set are completed.

[0090] Step 4: The agent selects the optimal task execution order and feeds back the loss of each task under the optimal execution order to the feature extractor and each task model.

[0091] The overall model for multimodal intelligent information extraction on social platforms is divided into two stages: a training stage and a prediction stage. In the training stage, a causal-based multimodal feature fusion module outputs fused image and text representations. These representations are then input into the multimodal intelligent information extraction module, where an agent selects and executes different information extraction tasks. As the model trains, the agent learns the optimal execution order for each extraction task and continuously optimizes the extraction model parameters, ultimately resulting in the trained overall model.

[0092] In the prediction phase: Input text and image information from a social platform, use the trained model to calculate the fused text and image representations, and output multimodal extraction results through each extraction model according to the optimal execution order selected by the agent.

[0093] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A method for extracting multimodal unified information from social platforms, characterized in that: Input text and image information from social media platforms, and through the multimodal feature fusion module and the multimodal information intelligent extraction module, entity and relationship extraction of text and images are achieved, thereby obtaining entity and relationship information in text and images. This helps users extract specific and useful information from unstructured social media fields more quickly, accelerating the speed and process of information filtering and processing. The multimodal feature fusion module takes as input the original text and image information of the social platform. It preprocesses the data of each modality, uses the text and image information in the database to construct a heterogeneous multimodal graph of text and image representation, and then performs graph structure learning and causal feature perturbation to finally output causal text representation and causal image representation. The unified information extraction module for intelligent decision-making takes the causal text representation and causal image representation as input, identifies and classifies entities in the data through entity extraction, and determines the relationship types between different entities through relation extraction. Based on historical extraction results, it uses the category set for the current extraction task. By restricting the extraction categories, it enhances the connection between various extraction tasks and reduces redundant entities in multimodal extraction tasks. It employs reinforcement learning-based intelligent decision-making to determine the execution order of extraction tasks. The Q-learning algorithm is used for reinforcement learning to extract multimodal information and intelligently select tasks to find the optimal task execution order. Specifically, the steps for determining the execution order are as follows: Step 1: The reinforcement learning agent selects a task that has not been executed from the extracted task set; Step 2: Perform the extraction task using an extraction model and save the extraction results; use the F1 score of the extraction results as the reward for the reinforcement learning agent, and use the class results of all extraction tasks performed as the class set for subsequent extraction tasks. Step 3: Repeat steps 1-2 until all extraction tasks in the set are completed; Step 4: The agent selects the optimal task execution order and backfeeds the loss of each task under the optimal execution order to the feature extractor and each task model; The method for constructing the heterogeneous multimodal graph is as follows: A preliminary heterogeneous graph of text and image representations is constructed, centered on text representation and image representation. ; Among them, the word representation of the text modality and object representation of image modalities They are considered as nodes, and the relationships between them are considered as edges; Subsequently, Graph structure learning is performed to obtain a heterogeneous graph representation with optimized structure. ; During the training process, Based on this, a heterogeneous graph representing environmental change is obtained by altering the feature information of different nodes using a diffusion model. Use regularization terms Graphs with constraints containing information about different nodes and Thus, we can obtain the causal text word representation. and causal image object representation .

2. The method for extracting multimodal unified information from a social platform as described in claim 1, characterized in that: The construction of preliminary graphic representation heterogeneous graphs The specific process is as follows: First, representation extraction is performed on the preprocessed information. For the text modality, an existing large-scale pre-trained model is used to obtain the representation of each text word. For the image modality, Faster R-CNN is used to obtain the representation of each object in the image. Then, based on the above two types of representations, a heterogeneous graph of image and text representations is constructed. Where A represents the adjacency matrix of the heterogeneous graph, which is a two-dimensional matrix whose each element A ij X represents whether there is an edge between node i and node j, where 1 indicates that there is an edge and 0 indicates that there is no edge; X is the feature vector of the node, which is a two-dimensional matrix, and each row of Xi represents the feature vector of node i. There are two types of nodes in a heterogeneous graph representation: text (word) representing information. Image object representation information ; There are three edge types: image object representation - image object, image object representation - text word representation, and text word representation - text word representation; Aside from the relationships inherent in the data, the rules for constructing other relationships are as follows: a. Image-Image: When the similarity between two different objects in the same image reaches a set threshold, the two image objects are connected by an edge; b. Image-Text: If the similarity between the representations of an image object and a text word reaches a set threshold, then connect the image object representation and the text word representation with an edge; c. Text-Text: When the similarity between two different word representations in the same text reaches a set threshold, the word representations of these two texts are connected by an edge.

3. The method for extracting multimodal unified information from a social platform as described in claim 2, characterized in that: The graph structure learning and causal feature perturbation process is based on a text-image modality alignment and fusion model framework that combines causal intervention and graph structure learning. Specifically, it first uses the existing IDGL graph structure learning method to... The structure is further optimized and learned to obtain... During training, based on the optimized graph... By using a diffusion model to change the feature information of different nodes in a heterogeneous graph representing a graph, we obtain... To mimic graphs in different environments, regularization terms are then applied. Graphs with constraints containing information about different nodes and ,Will Obtaining causal text word representations using a single-layer graph neural network and causal image object representation The specific calculation process is as follows: in, express Through a single layer of graph convolutional neural network, this module simultaneously outputs the text word representation. Image object representation Causal text word representation Causal image object representation , where i represents the i-th morpheme and j represents the j-th target in the image.

4. The method for extracting multimodal unified information from a social platform as described in claim 3, characterized in that: The entity extraction is divided into multiple tasks, including entity extraction for text and object detection for images. The entity extraction model for text labels the type of each word in the text sequence by superimposing a fully connected layer and a CRF layer on the representation acquisition module, using cross-entropy as the loss function and the F1 score of multi-class classification as the evaluation metric. The entity classification for images superimposes N SVM classifiers on the representation acquisition module, where N is the number of classes, using cross-entropy as the loss function and the F1 score of multi-class classification as the evaluation metric.

5. The method for extracting multimodal unified information from a social platform as described in claim 4, characterized in that: The relation extraction is the determination of the relationship type between different entities. Based on the different modalities from which the entities originate, relation extraction is divided into multiple tasks, including relation extraction between text entities, relation extraction between image entities, and relation extraction between text entities and image entities. The relation extraction model superimposes a fully connected layer and a softmax layer on the representation acquisition module. After obtaining the representations of the two entities, they are concatenated and input into the fully connected layer and the softmax layer for classification. Cross-entropy is used as the loss function, and F1 multi-class classification is used as the evaluation metric.

6. The method for extracting multimodal unified information from a social platform as described in claim 5, characterized in that: The reinforcement learning agent is modeled using value function reinforcement learning, specifically through the following parameters: State representation: Treating the tasks in the entity extraction set as the states of agents, each task is uniquely represented by its position in the set. Let there be a total of [number missing] tasks in the set. If there are 1 task, then the state space is: ; Action representation: The agent selects a task from the set that has not yet been executed, that is, a task related to the current state. Perform action selection; assuming the current state is... The action chosen by the agent is The action space is ; Reward function: The extraction results for each task are compared with the true labels, and the F1 score of the extraction results is calculated as the agent's immediate reward. If the task... The extraction results are The actual label is Then the task The instant reward is: in and These represent precision and recall, respectively. State transition function: The agent performs actions After that, it will enter the next state. Since each task can only be executed once, the action is executed. After, state Tasks in Will be removed, next status for Thus, the state transition function is obtained: ； Policy function: The policy function is a function used in reinforcement learning to guide an agent in choosing actions under specific states. Indicates that, in the state Select action The probability is given, and the policy function uses a greedy policy; Value functions: The value function is expressed as , indicating the state Next action The long-term reward that can be obtained; since each task can only be executed once, the long-term reward that the agent can obtain after completing all tasks in the set is the sum of the immediate rewards of all tasks; The Q-learning algorithm learns the value function, and its update formula is: ;in For learning rate, This is a discount factor used to control the importance of long-term rewards; the value function is updated after each task is completed, until all extraction tasks in the set are completed.

Images