A method for relation extraction in specific industries based on multimodal large language models

Abstract

This invention discloses a method for extracting relationships from specific industries based on a multimodal large language model. The method includes inputting sample information into a Transformer encoder, using a cross-modal attention module to perform multimodal information fusion in a hierarchical manner to capture semantic relationships in the text, introducing a knowledge graph constructed from crawled industry-specific data using knowledge fusion technology as supplementary information during modal fusion, enabling the model to compare the content in the knowledge graph when processing text, images, and audio to better understand the potential relationships between different modalities, and inputting the fused features into a reinforcement learning-based reward function module. Based on the reward function setting and the degree of matching between the predicted results and the real entity relationships, the method outputs the relationship between two entities in the original text with high accuracy.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for extracting relations for specific industries based on a multimodal large language model. Background Technology

[0002] With the rapid advancements in artificial intelligence and natural language processing technologies, large language models have demonstrated outstanding performance in a wide range of application scenarios. Multimodal large models, in particular, aim to fuse data from different modalities. However, in practical applications, effectively fusing information from text, images, audio, and other modalities remains a challenge. Taking the marketing industry as an example, the marketing field involves numerous sub-sectors, each with its unique terminology and expressions. Current fusion methods cannot fully uncover the deep relationships between different modalities, resulting in an incomplete understanding of the complex multimodal relationships within marketing scenarios. For instance, when analyzing advertising videos, it is difficult to accurately and effectively correlate and integrate the visual content in the video with the key information in the advertising copy. Summary of the Invention

[0003] This application provides a method for extracting relationships from specific industries based on a multimodal large language model. In the modality fusion process, a knowledge fusion module is introduced. By utilizing information from the knowledge graph, the model's semantic understanding ability is enhanced by dynamically reasoning about relevant entities and relationships according to task requirements, enabling the model to use prior knowledge for deeper reasoning and analysis.

[0004] This application provides a method for extracting relations from specific industries based on a multimodal large language model, including the following process:

[0005] S1: Input the sample information (text content, image content, or video content) into the Transformer encoder of the multimodal large language model. The Transformer encoder extracts deep features from the text.

[0006] S2: Utilize the cross-modal attention module to fuse text modality with image modality and audio modality in a multi-level fusion manner;

[0007] S3: The knowledge fusion module is introduced to construct a knowledge graph from the crawled industry-specific data. When performing modal fusion, the knowledge graph is used as supplementary information. When the model processes text, images, and audio, it compares the features of these modalities with the content in the knowledge graph.

[0008] S4: The fused features are input into the reinforcement learning-based reward function module. Based on the setting of the reward function, the model parameters are adjusted according to the degree of matching between the prediction result and the real entity relationship. By continuously adjusting the model based on the feedback of the reward function, the model can eventually output the relationship between the two entities in the original sample with high accuracy.

[0009] The further cross-modal attention module, following the basic layer fusion, mid-layer feature fusion, and high-layer semantic fusion approach, incorporates a knowledge fusion module to construct a knowledge graph as supplementary information when fusing text modality with image and audio modality. The specific operational process includes:

[0010] First, the base layer performs feature fusion, segmenting the text into words, and then using a word embedding model to convert each word into a vector to form a text feature matrix.

[0011] Image frames are extracted from the advertisement video, and image features are extracted using a convolutional neural network to obtain feature maps. Where n i It is the number of image regions, d i These are the feature dimensions of each region;

[0012] The audio in the advertisement video is converted into a spectrogram, and then a convolutional neural network is used to extract audio features to obtain an audio feature vector. Where n a It is the number of audio segments, d a It is the feature dimension of each segment;

[0013] Data from a specific industry is crawled to construct a knowledge graph. Graph embedding methods are then used to extract entity and relation features from the knowledge graph, representing the knowledge graph as a vector form. Where n k It is the number of knowledge entities, d k It is the feature dimension of each entity;

[0014] The base layer features F after fusion using the fusion function base for:

[0015] F base =Concat(T, I, A, K);

[0016] Concat represents a vector concatenation operation, which joins the feature vectors of four modalities end-to-end to form a longer fused feature vector. The resulting base layer feature vector is as follows:

[0017]

[0018] In the mid-level feature fusion, the focus is on the interaction and fusion between modalities. The correlation between text and image modalities, and audio modalities, is calculated to generate attention weights. The attention weights for text-image modalities, text-audio modalities, and text-knowledge graph modalities are obtained using the softmax function.

[0019] For the image modality, attention weights are obtained by calculating the similarity between text features and features of each image region.

[0020] For the audio modality, attention weights are obtained by calculating the similarity between text features and features of each audio segment.

[0021] For knowledge graph modalities, attention weights are obtained by calculating the similarity between text features and the features of each knowledge entity.

[0022] Among them W t W is the weight matrix between text features and image features. a W is the weight matrix between text features and audio features. k It is the weight matrix between text features and knowledge graph features;

[0023] The features of the modalities are weighted and fused to obtain the fused mid-level feature vector F. mid :

[0024]

[0025] Based on the task requirements, the knowledge graph fusion module obtains the entities and relations inferred from the knowledge graph, resulting in the inference result R. F-layer fusion feature in splicing mid Calculate the weights:

[0026] γ = sigmoid(W g ×Concat(F mid ,R)+b g )

[0027] Finally, the high-level semantic feature vector F is obtained by dynamically balancing the mid-level features and knowledge graph inference results using weight γ. high :

[0028] F high =γ⊙F mid +(1-γ)⊙R

[0029] The high-level semantic feature vector output after fusion

[0030] The fused features are input into the reinforcement learning-based reward function module. A weight factor α is introduced based on immediate and long-term rewards to dynamically adjust their importance, enabling the model to focus on different objectives at different stages, thereby improving learning efficiency. The specific operation process of the reinforcement learning-based reward function module includes:

[0031] First, the high-level semantic feature vector F is output. high As a state representation of the reinforcement learning environment, the immediate reward, reflecting the degree of matching between the currently predicted entity relationship and the real relationship, is used as a weighted component of the total reward. The calculation of the immediate reward is as follows:

[0032]

[0033] in, r represents the predicted entity relationship, and r represents the actual entity relationship. It is some kind of distance metric between the two (such as edit distance or semantic distance), d max This is the maximum possible value of the distance metric;

[0034] The long-term reward is weighted based on the model's overall performance across a series of actions to encourage the model to learn a more stable policy. The calculation of the long-term reward is as follows:

[0035]

[0036] Where γ' is the discount factor, Q(S, c) is the state-action value function of taking action c in state S, S' is the new state after taking action c, A is the action space defined as all possible entity relation labels, and S is the output high-level semantic feature vector F. high As a state representation of the reinforcement learning environment;

[0037] By introducing a weighting factor α, immediate and long-term rewards are weighted. This α factor dynamically adjusts the importance of immediate and long-term rewards, allowing the model to focus on different objectives at different stages, thereby improving learning efficiency. The weighted sum of immediate and long-term rewards is defined as the total reward.

[0038] R 总奖励 =α·R 即时 +(1-α)·R 长期

[0039] According to R 总奖励 The reinforcement learning algorithm Q-learning continuously updates the state-action value function Q(S, c) during the learning process to maximize the cumulative reward:

[0040]

[0041] After training, for a given input state S, select Q(S, c) that maximizes the action c as the predicted entity relation label, and output the label as the relation between the two entities.

[0042] The relationship between the two output entities is fed back to the knowledge fusion module to update and improve the entities and relationships in the knowledge graph. The updated knowledge graph contains the latest and most accurate information, reducing the impact of outdated or erroneous data. This helps the model obtain more accurate information support during modality fusion, thereby improving the overall analysis and prediction accuracy.

[0043] In the knowledge graph update process, in addition to relying on the matching degree between the prediction results and the real entity relationships, the semantic similarity between the newly extracted entities / relationships and the existing corresponding items in the knowledge graph is calculated using cosine similarity. The newly extracted entity is defined as E. new In a knowledge graph, existing entities are defined as E. old First, entity E new and E old The vector representations are obtained by inputting the data into the cross-modal attention module. Then, the cosine similarity between the new relation and the existing relations in the knowledge graph is calculated using the cosine similarity formula SSS (Semantic Similarity Score):

[0044]

[0045] The knowledge graph fusion module obtains the entities and relations inferred from the knowledge graph, resulting in the inference result R. By designing specific semantic reasoning to uncover potential entity relationships;

[0046] First, collect professional knowledge in the marketing field, such as industry reports and market research papers, to build a semantic knowledge base, which contains semantic relationships between various marketing concepts, such as the relationship between "brand positioning" and "target audience characteristics", and the relationship between "promotional activity type" and "sales growth expectation".

[0047] Define each entity pair in the knowledge graph as (e i ,e j Based on cosine similarity, a correction term based on a marketing semantic knowledge base is introduced to calculate the semantic similarity of each entity pair in the embedding space: Definition and Entity e i and e j The similarity of the embedding vectors is calculated based on the constructed marketing semantic knowledge base:

[0048]

[0049] Where β is a weighting parameter used to balance the contributions of the two similarities; K(e i ,e j ) represents entity e calculated based on the marketing semantic knowledge base. i and e j The semantic similarity between them. If the semantic knowledge base explicitly indicates a strong correlation between "high-end brands" and "high-income target audience," then when e i As a "high-end brand", e j When targeting a "high-income audience", K(e) i ,e j ) will have a higher value;

[0050] Simultaneously, graph neural networks are introduced to effectively capture the complex relationships and features in graph-structured data. Entities in the knowledge graph are treated as nodes, and the relationships between entities are treated as edges. For example, "brand positioning" and "target audience characteristics" are two nodes, and the relationship between them is the edge connecting these two nodes. A feature vector is initialized for each node. This feature vector can contain some basic information about the entity, such as the entity type and frequency of occurrence.

[0051] In each iteration, a message-passing mechanism is used. Each node receives information from its neighboring nodes and updates its own feature vector based on this information. Let node e... i The feature vector in round t is Its neighbor node set is N(e i Then node e i The formula for updating the feature vector in the (t+1)th round can be:

[0052]

[0053] Where W' is the learnable weight matrix and σ is the activation function;

[0054] After several iterations, the feature vector of each node has fully incorporated information from its neighboring nodes. At this point, the similarity between nodes is calculated to uncover potential entity relationships. For two nodes e... i and e j Calculate the cosine similarity of their final feature vectors as the strength R of the relationship between them. ij :

[0055]

[0056] Among them, T ij This is the total number of iterations;

[0057] According to the set threshold θ', when R ijWhen >θ', the entity e is inferred. i and e j There is a potential relationship between them. At the same time, the relationship types in the marketing semantic knowledge base can be referenced to assign specific semantic types to the inferred relationship, such as "association" or "influence".

[0058] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: A knowledge fusion module is introduced during the modal fusion process. Utilizing information from the knowledge graph, relevant entities and relationships are dynamically inferred according to task requirements to enhance the model's semantic understanding ability, enabling the model to use prior knowledge for deeper reasoning and analysis. The fused features are dynamically adjusted and optimized through a reinforcement learning-based reward function module to predict entity results. The model is continuously optimized through trial and error and feedback mechanisms to improve the accuracy and robustness of entity relationship prediction. The relationship between the two output entities is fed back to the knowledge fusion module to update and improve the entities and relationships in the knowledge graph. The updated knowledge graph contains the latest and most accurate information, reducing the impact of outdated or erroneous data and helping the model obtain more accurate information support during modal fusion, thereby improving the overall analysis and prediction accuracy. Attached Figure Description

[0059] Figure 1 This is a model diagram for the relationship extraction in this application;

[0060] Figure 2 This is a flowchart of the relationship extraction process for this application;

[0061] Figure 3 This is a schematic diagram of multi-level feature fusion in the cross-modal attention module of this application. Detailed Implementation

[0062] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0064] Example 1

[0065] Please see Figure 1-3A method for extracting relationships in specific industries based on a multimodal large language model includes: inputting text, image, and video content into a Transformer encoder for deep feature extraction; using a cross-modal attention module to fuse text modality with image and audio modality at multiple levels, introducing the knowledge fusion module to construct a knowledge graph, and using it as supplementary information during modality fusion; and using a reinforcement learning-based reward function module to adjust model parameters to improve the accuracy of relationship extraction.

[0066] The cross-modal attention module performs multi-level feature fusion, including basic layer feature fusion, mid-level feature fusion, and high-level semantic fusion. During basic layer feature fusion...

[0067] After text segmentation, a text feature matrix T is formed using a word embedding model. Image frames are extracted from the video, and image features are extracted using a convolutional neural network to obtain feature map I. The audio in the video is converted into a spectrogram, and audio features are extracted to obtain audio feature vector A. Specific industry data is crawled to construct a knowledge graph, and entity and relation features are extracted and represented as vector K. T, I, A, and K are concatenated using a fusion function to obtain the base layer feature vector F. base ;

[0068] When fusing mid-level features,

[0069] The similarity between text and image, audio, and knowledge graph modalities is calculated. Attention weights are generated using the softmax function. The features of each modality are then weighted and fused to obtain the mid-level feature vector F. mid ;

[0070] During high-level semantic fusion,

[0071] The reasoning results R of entities and relations are obtained from the knowledge graph fusion module, and then concatenated with the mid-level features F. mid R and R use a gating mechanism to calculate weights, and use the gating weight γ to dynamically balance the mid-level features and knowledge graph inference results to obtain the high-level semantic feature vector F. high .

[0072] The fused features are input into the reinforcement learning-based reward function module, and F... high The state representation of the reinforcement learning environment is defined as S, the action space is defined as A, and the entity relation labels are defined as all possible entities. Immediate and long-term rewards are calculated and weighted by a weight factor α. The reinforcement learning algorithm Q-learning is used to continuously update the state-action value function Q(S, c) to maximize the cumulative reward during the learning process. After training, for a given input state S, the action c that maximizes Q(S, c) is selected as the predicted entity relation label.

[0073] By leveraging the matching degree between the output results and the real entity relationships, and combining cosine similarity to calculate the semantic similarity between the newly extracted entities / relationships and the existing corresponding items in the knowledge graph, the entities and relationships in the original knowledge graph are updated and improved. This reduces the impact of outdated or erroneous data, helps the model obtain more accurate information support during modality fusion, and improves the overall analysis and prediction accuracy.

[0074] The knowledge graph fusion module obtains the entities and relations inferred from the knowledge graph, and produces the inference result R. By designing specific semantic reasoning to uncover potential entity relationships;

[0075] First, we extensively collect professional knowledge in the marketing field, including but not limited to industry reports, market research papers, classic marketing case studies, and professional marketing books, to build a well-structured and rich marketing semantic knowledge base. This base contains semantic relationships between various marketing concepts, such as the relationship between "brand positioning" and "target audience characteristics," and the connection between "promotional activity types" and "sales growth expectations."

[0076] The semantic similarity of each entity pair in the knowledge graph is calculated in the embedding space by incorporating cosine similarity into a correction term based on a marketing semantic knowledge base. A weight parameter is introduced to balance the contributions of the two similarities.

[0077] Based on semantic similarity S, a threshold θ is set to infer whether there is a potential relationship between entity pairs; at the same time, referring to the relationship types in the marketing semantic knowledge base, specific semantic types are assigned to the inferred relationships, such as "association" and "influence".

[0078] Experimental Example 1

[0079] In the construction of knowledge graphs in the marketing industry, accurately identifying entity relationships in advertising videos is crucial. Multimodal data fusion and reinforcement learning methods can be used to improve the accuracy of entity relationship reasoning.

[0080] Experimental data

[0081] At least 1000 marketing industry advertising videos were selected as experimental data. Each data point includes the video, the corresponding advertising copy, and labeled entity relationships. Data preprocessing was performed as follows:

[0082] Frame extraction is performed on the video to obtain image information for each frame, so that visual features can be extracted later.

[0083] The audio signal is converted from a time-domain signal to a frequency-domain signal to prepare for audio feature extraction.

[0084] The advertising copy is segmented and embedded into words, transforming the text into a vector form that can be processed by computers, thus providing a foundation for subsequent feature extraction.

[0085] The sample content (an advertisement video for a certain brand of mobile phones, including the text "XX mobile phone, clearer photos, longer battery life", the image content "mobile phone camera interface and battery icon", and the audio content "background music and narration") was input into the Transformer encoder of a multimodal large model for feature extraction.

[0086] The text content is converted into a text feature matrix T (512-dimensional) using word embedding technology. Image frames are processed by convolutional neural networks to obtain image feature maps (1024-dimensional). Audio is converted into a spectrogram and obtained by convolutional neural networks to obtain audio feature vector A (512-dimensional). Knowledge feature vector K (512-dimensional) is obtained by crawling mobile phone industry data to construct a knowledge graph.

[0087] The text feature matrix T, image feature map I, audio feature vector A, and knowledge feature vector K are concatenated to obtain a basic fusion feature with a dimension of 2560, providing rich data for subsequent in-depth fusion.

[0088] The attention weights of text, image, audio, and knowledge graph are calculated using the softmax function, and then weighted and fused to obtain the mid-level fusion feature F. mid (Dimensionality is 512); Concatenate the middle layer features F mid The reasoning result R of entities and relations obtained by the knowledge graph fusion module is used to calculate the weights using a gating mechanism to obtain the high-level semantic feature vector F. high (512 dimensions);

[0089] Using high-level semantic feature vector F high As a state representation of the reinforcement learning environment, α is defined as 0.8, and R... 即时 R is 1 长期 R is 0.5 总奖励 The calculation is as follows:

[0090] R 总奖励 =0.8·1+(1-0.8)·0.5=0.9

[0091] The reinforcement learning algorithm Q-learning is based on R 总奖励 The update state-action value function outputs the relationship "XX phone - photo effect", while the actual labeled relationship is "XX phone - photo effect".

[0092] The model output was evaluated using accuracy, recall, and F1 score. The number of relationships correctly predicted by the model was 184, the number of negative examples incorrectly predicted as positive was 16, and the number of positive examples incorrectly predicted as negative was 20.

[0093]

[0094] Wherein, TP represents true positives (positive examples correctly predicted by the model), FP represents false positives (negative examples incorrectly predicted as positive by the model), and FN represents false negatives (positive examples incorrectly predicted as negative by the model).

[0095] Most traditional multimodal models simply concatenate multimodal features without fully considering the interactions between different modalities. For example, when processing advertising videos, they may simply connect text, image, and audio features directly, ignoring the varying importance of each modality in expressing entity relationships across different scenarios. This experiment uses the softmax function to calculate attention weights for weighted fusion, which can dynamically adjust the contributions of different modal features and more accurately capture entity relationships.

[0096] Some models lack further optimization mechanisms after feature fusion and inference. This experiment introduces reinforcement learning by setting a reasonable reward mechanism, enabling the model to continuously adjust its strategy based on environmental feedback, thereby improving the accuracy and stability of predictions. Models without reinforcement learning optimization may get stuck in local optima and cannot dynamically adjust according to the actual situation.

[0097] Experimental results

[0098]

[0099]

[0100] Based on the experimental results, our experimental model outperforms existing traditional multimodal models and models without reinforcement learning optimization in terms of accuracy, recall, and F1 score. By introducing a knowledge fusion module when using a cross-modal attention module for multi-level feature fusion, we obtain entities and relationships inferred from the knowledge graph as supplementary information for feature fusion. This significantly improves the accuracy of relation extraction in multimodal large language models, enabling more accurate identification of entity relationships in advertising videos and providing more reliable data support for the construction of knowledge graphs in the marketing industry.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for relation extraction in a specific industry based on a multimodal large language model, characterized in that, Includes the following processes, S1: Input the sample information into the Transformer encoder of the multimodal large language model. The Transformer encoder extracts deep features from the text. S2: Utilize the cross-modal attention module to fuse text modality with image modality and audio modality in a multi-level fusion manner; S3: Introducing a knowledge fusion module to construct a knowledge graph from crawled industry-specific data, and using the knowledge graph as supplementary information when performing modal fusion; S4: The fused features are input into the reinforcement learning-based reward function module. By continuously adjusting the model based on the feedback of the reward function, the model can eventually output the relationship between two entities in the original sample with high accuracy. The cross-modal attention module performs multi-level feature fusion according to the basic layer fusion, mid-level feature fusion, and high-level semantic fusion methods. The operation of introducing a knowledge fusion module to construct a knowledge graph as supplementary information includes... The base layer performs feature fusion and forms a text feature matrix T∈ through a word embedding model. , Representing the text feature dimension; using a convolutional neural network to extract image features to obtain the feature map I∈ , Indicates the number of image regions. The image region feature dimension is represented; then, a convolutional neural network is used to extract audio features, resulting in an audio feature vector A∈ , Indicates the number of audio segments. Representing the feature dimension of audio segments; crawling specific industry data to construct a knowledge graph, using graph embedding methods to extract entity and relation features from the knowledge graph, and representing the knowledge graph as a vector form K∈ , Indicates the number of knowledge entities. Represents the dimension of knowledge entity features; The base layer features are fused using the fusion function. ∈ ; Mid-level feature fusion obtains the attention weights for the text-image modality separately using the softmax function. =softmax(T) T represents the text feature matrix. Represents the text-image feature weight matrix. This represents the feature vector of the i-th image region. Represents the transpose of the feature vector of the i-th image region; attention weights for the text-audio modality. =softmax(T) ), This represents the text-audio feature weight matrix. Indicates the first Feature vectors of audio segments Indicates the first Transpose of the feature vectors of each audio segment; attention weights for the text-knowledge graph modality. =softmax(T) ), Represents the text-knowledge graph feature weight matrix. This represents the feature vector of the k-th knowledge entity. This represents the transpose of the feature vector of the k-th knowledge entity. The fused mid-level feature vector is obtained by weighted fusion of the modal features. : ， The knowledge graph fusion module obtains the entities and relations inferred from the knowledge graph, and the inference result R∈ , Representing the dimension of entity and relation feature vectors, and concatenating them into a middle-layer fusion feature. Calculate the weights: sigmoid( ), Indicates the gating weight, Represents the gate weight matrix. Knowledge graph reasoning results Indicates the gating bias term; Finally, the weights are used. The high-level semantic feature vector is obtained by dynamically balancing the mid-level features and the reasoning results from the knowledge graph. ∈ , High-level semantic feature dimension Feature dimensions: = ⊙ （ ）⊙R， The knowledge graph fusion module obtains the entities and relations inferred from the knowledge graph, resulting in the inference result R. Each entity pair in the knowledge graph is defined as... Based on cosine similarity, a correction term based on a marketing semantic knowledge base is introduced to calculate the semantic similarity of each entity pair in the embedding space. At the same time, a graph neural network is introduced to effectively capture the complex relationships and features in graph structure data.

2. The method for relation extraction from a specific industry based on a multimodal large language model as described in claim 1, characterized in that: Output high-level semantic feature vector The state representation of the reinforcement learning environment is defined as S, the action space is defined as all possible entity relation labels as A, and the weighted composition of the total reward is based on the degree of matching between the currently predicted entity relation and the real relation, using immediate reward as a weighted component. The calculation of immediate reward is as follows: ， in, It is a predicted entity relationship. It is a real entity relationship. It is a kind of distance metric between the two. It is the maximum possible value of this distance metric.

3. The method for relation extraction from a specific industry based on a multimodal large language model as described in claim 2, characterized in that: To incorporate long-term rewards, consider the model's overall performance across a series of actions. The calculation of long-term rewards is as follows: ( , ) Q(S,c), in, It is a discount factor, and Q(S, c) is the state-action value function of taking action c in state S. It is the new state after taking action c.

4. The method for relation extraction from a specific industry based on a multimodal large language model as described in claim 2, characterized in that: Immediate rewards and long-term rewards are weighted using a weighting factor. The importance of immediate and long-term rewards is dynamically adjusted, and the weighted sum of immediate and long-term rewards is defined as the total reward: （1- ） ， according to The reinforcement learning algorithm Q-learning continuously updates the state-action value function Q(S,c) during the learning process to maximize the cumulative reward: Q(S,c)= Q(S,c) , After training, for a given input state S, select Q(S, c) that maximizes the action c as the predicted entity relation label, and output the label as the relation between the two entities.

5. The method for relation extraction from a specific industry based on a multimodal large language model as described in claim 4, characterized in that: The relationship between the two output entities is fed back to the knowledge fusion module to update and improve the entities and relationships in the knowledge graph. Cosine similarity is used to calculate the semantic similarity between the newly extracted entity / relation and its existing counterpart in the knowledge graph. The newly extracted entity is defined as... Existing entities in a knowledge graph are defined as follows: First, the entity and The vector representations are obtained by inputting the data into the cross-modal attention module. , Then, the cosine similarity formula SSS is used to calculate the cosine similarity between the new relation and the existing relations in the knowledge graph: SSS( , )= 。 6. The method for relation extraction from a specific industry based on a multimodal large language model as described in claim 1, characterized in that: The knowledge graph fusion module uses semantic reasoning to uncover potential entity relationships. The specific process is as follows: First, we collect professional knowledge in the marketing field to build a semantic knowledge base, which contains semantic relationships between various marketing concepts; Define each entity pair in the knowledge graph as Based on cosine similarity, a correction term based on a marketing semantic knowledge base is introduced to calculate the semantic similarity of each entity pair in the embedding space, defining... and Entities and The similarity of the embedding vectors is calculated based on the constructed marketing semantic knowledge base: S K , in, K is a weighting parameter used to balance the contributions of the two similarities; This refers to entities calculated based on the marketing semantic knowledge base. and Semantic similarity between them; A threshold is set based on semantic similarity S. When S Inferring entities and There is a certain potential relationship between them; at the same time, referring to the relationship types in the marketing semantic knowledge base, the inferred relationship is assigned a specific semantic type.

7. The method for relation extraction from a specific industry based on a multimodal large language model as described in claim 1, characterized in that: Graph neural networks use entities in a knowledge graph as nodes and relationships between entities as edges. In each iteration, a message-passing mechanism is used. Each node receives information from its neighboring nodes and updates its own feature vector based on this information. Let the nodes... The feature vector in round t is Its neighbor node set is Then the node In the The formula for updating the eigenvector of the wheel is: ， in, It is a learnable weight matrix. It is an activation function; After several iterations, the feature vector of each node has fully incorporated information from its neighboring nodes. At this point, the similarity between nodes is calculated to uncover potential entity relationships. For two nodes... and Calculate the cosine similarity of their final feature vectors as the strength of the relationship between them. : ， in, This is the total number of iterations; Based on the set threshold ,when When, infer the entity and There are potential relationships between them, and the relationship types in the marketing semantic knowledge base are referenced to assign specific semantic types to the inferred relationships.

Images