Multi-modal relation extraction method based on goal-oriented description enhancement and dynamic weight
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV AT QINHUANGDAO
- Filing Date
- 2026-05-12
- Publication Date
- 2026-06-09
AI Technical Summary
Existing multimodal relation extraction methods struggle to accurately focus on target relations in social media data processing, suffering from noise sensitivity and semantic drift, leading to misleading and redundant interference.
By acquiring target entity pairs from the original text and images, a large language model is used to generate target-guided descriptive text, and a target diffusion model is combined to generate semantically enhanced images. These images are then input into a pre-trained relation extraction network, and feature fusion and classification are performed using dynamic weights and class bias weights.
It significantly improves the accuracy and robustness of multimodal relation extraction, effectively filters noise interference, and enhances the accuracy of relation extraction in complex contexts and the recall capability of long-tailed distributions.
Smart Images
Figure CN122174138A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal natural language processing technology, and in particular relates to a multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights. Background Technology
[0002] With the explosive growth of multimedia content, Multimodal Relation Extraction (MRE) has become a core technology in fields such as knowledge graph construction, intelligent public opinion monitoring, and privacy risk analysis. In real-world scenarios such as social networks, cross-border e-commerce, and anti-fraud monitoring, target information is typically composed of text and accompanying images. Compared to plain text, images can provide intuitive semantic completion, helping to identify the complex relationships implicit between entities.
[0003] However, in the processing of social media data, multimodal information alignment faces serious challenges of "noise sensitivity" and "semantic drift." First, the original images often contain a large amount of background objects or environmental noise unrelated to the core entity pairs, causing the neural network model to be misled during feature fusion. Second, while introducing generative artificial intelligence can enhance visual semantics, the generated content carries the risk of "illusion"—poor image quality or deviation from the original text semantics can introduce secondary interference. When there is conflict or redundancy between text semantics and visual features, existing models often struggle to accurately focus on the target relationship. Therefore, there is an urgent need for a multimodal relationship extraction method that can achieve target-oriented guidance and adaptive noise suppression capabilities. Summary of the Invention
[0004] To address the aforementioned problems in the existing technology, this invention provides a multimodal relationship extraction method based on goal-oriented description enhancement and dynamic weights.
[0005] The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights, comprising: Obtain the original text, original image, and corresponding target entity pairs; Based on the original text, target entity pairs, and preset targeted guidance prompts, a large language model is used to generate target guidance description text. The original image is input into the target diffusion model, and the output is a semantically enhanced image; The original text, target-guided descriptive text, original image, and semantically enhanced image are input into a pre-trained target relation extraction network, which outputs relation classification results for target entity pairs.
[0006] This invention provides a multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights. It constructs directional guidance prompts based on the original text, target entity pairs, and preset directional guidance prompt templates. These prompts guide a large language model to generate target-oriented descriptive text. The original text, target-oriented descriptive text, original image, and semantically enhanced image are then input into a target relation extraction network, which outputs relation classification results. By utilizing the target-oriented descriptive text constructed for target entity pairs, the method effectively drives the target relation extraction network to accurately extract interaction features related to target entity pairs in complex social media contexts. This filters out semantic interference from irrelevant and redundant objects at the source, significantly improving the accuracy and robustness of the multimodal relation extraction results.
[0007] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description
[0008] Figure 1 This is a flowchart illustrating a multimodal relationship extraction method based on goal-oriented description enhancement and dynamic weighting provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the processing procedure of a multimodal relationship extraction method based on goal-oriented description enhancement and dynamic weighting provided in an embodiment of the present invention. Detailed Implementation
[0009] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.
[0010] This invention provides a multimodal relation extraction method based on goal-oriented description enhancement and dynamic weighting. See also... Figure 1 and Figure 2 The method includes the following steps: S10. Obtain the original text, original image, and corresponding target entity pairs.
[0011] For example, the original text can be an unstructured string containing social media context, and the target entity pair can contain a header entity. With tail entity The target entity pairs can be obtained by extracting entities from the original text using the Named Entity Recognition (NER) model.
[0012] S20. Based on the original text, target entity pairs, and preset directional guidance prompts, generate target guidance description text using a large language model.
[0013] For example, extracting pairs of the target entity from the original text. Relevant contextual fragments, combined with preset targeted guidance prompt templates, are used to construct targeted guidance prompts containing entity role information. A large language model (such as MiniCPM-Llama3-V-2.5, which has visual feature extraction and understanding capabilities) is then invoked to generate targeted guidance description text for that specific entity relationship. In a preferred embodiment, the preset targeted guidance prompt template is as follows: Given the original text: {sentence}; in the original image, what are the visual associations and interactions between the subject entity {h_name} and the object entity {t_name}? Please describe in detail the action characteristics of the subject and object entities and their relative spatial positions.
[0014] In the preset targeted guidance prompt template, {sentence} represents the original text, and the subject entity {h_name} and object entity {t_name} represent the names of the head entity and tail entity of the target entity pair, respectively.
[0015] The final generated target guidance description text is shown in the following example: The original image depicts a man (the subject entity "Man A") and a woman (the object entity "Woman B") sharing an intimate moment. At the visual level, Man A is kissing Woman B, and their relative spatial positions suggest close physical contact, implying a deep emotional connection. The man faces the woman, and their bodies are turned towards each other, presenting a typical romantic interaction. The woman's hand is gently placed on the man's face, further emphasizing the intimacy of this emotional interaction through the details of the action. This visual scene, through specific actions and spatial connections, precisely corresponds to the semantic description in the original text about "Man A getting engaged to his girlfriend, Woman B, after 14 years of dating," constituting a clear semantic connection and feature enhancement.
[0016] It will be understood by those skilled in the art that the preset targeted guidance prompt template and the target guidance description text can be in Chinese, English, or other natural languages, and this embodiment does not limit this. In specific calculation implementation, the target guidance description text can be converted into a high-dimensional feature vector through the corresponding tokenizer, or converted into an equivalent English semantic representation for calculation in the preprocessing stage. The choice of language does not affect the subsequent technical implementation of the dynamic weighting mechanism and multimodal fusion framework of this invention.
[0017] This embodiment reconstructs the interaction logic between entities from a visual perspective by leveraging the prior knowledge of a large language model, thus eliminating redundant interference in the original image.
[0018] S30. Input the original image into the target diffusion model and output a semantically enhanced image.
[0019] For example, the original image Input a pre-trained target diffusion model, and generate a semantically enhanced image that is semantically consistent with the original image but with enhanced details through diffusion inversion and reconstruction. The target diffusion model can be Stable Diffusion v1.5.
[0020] S40. Input the original text, target-guided descriptive text, original image, and semantically enhanced image into a pre-trained target relation extraction network, and output the relation classification results for the target entity pairs.
[0021] Optionally, the target relation extraction network includes a text encoder, an image encoder, a feature fusion network, and a linear classification layer. Step S40 may specifically include: S401. The original text and the target guiding descriptive text are feature-mapped by a text encoder to obtain the feature vector of the original text and the feature vector of the descriptive text.
[0022] For example, a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model can be used as a text encoder to process the original text. With target-guided descriptive text Perform feature mapping to obtain the original text feature vector. With description text feature vector .
[0023] S402. Spatial features are extracted from the original image and the semantically enhanced image using an image encoder to obtain the feature vectors of the original image and the enhanced image.
[0024] For example, a pre-trained ResNet50 (Residual Network 50 layers) network can be used as an image encoder to process the original image. With semantically enhanced images Spatial feature extraction is performed to obtain the original image feature vector. With enhanced image feature vectors .
[0025] S403. The original text feature vector, the descriptive text feature vector, the original image feature vector, and the enhanced image feature vector are fused using a feature fusion network to obtain fused features.
[0026] Optionally, step S403 may specifically include: The feature fusion network calculates the similarity between the original image feature vector and the enhanced image feature vector to obtain semantic similarity. The semantic similarity is then linearly mapped to obtain a dynamic weighting factor. The enhanced image feature vector is weighted based on the dynamic weighting factor to obtain a weighted enhanced image feature vector. The original text feature vector, the descriptive text feature vector, the original image feature vector, and the weighted enhanced image feature vector are then concatenated to obtain the fused features.
[0027] For example, the feature vector of the original image is calculated in the feature space. With enhanced image feature vectors The semantic similarity is obtained by considering the cosine similarity between the two pairs of strings. To quantitatively evaluate the semantic fidelity of semantically enhanced images relative to the original images.
[0028] Among them, semantic similarity Represented as: ; In the formula, This represents the L2 norm.
[0029] Optionally, the dynamic weighting factor can be expressed as: ; in, Represents dynamic weighting factors. Indicates semantic similarity. Represents the feature vector of the original image. Represents the enhanced image feature vector. This represents the preset proportional mapping coefficient.
[0030] For example, in this embodiment, the preset scaling factor is preferably 0.2.
[0031] This embodiment uses similarity feedback to adaptively suppress low-quality or semantically drifted semantically enhanced images generated by the diffusion model; that is, the lower the similarity, the more weight is assigned to the feature vector of the enhanced image. The smaller.
[0032] Alternatively, the weighted enhanced image feature vector can be represented as: ; in, This represents the weighted enhanced image feature vector.
[0033] For example, using the calculated dynamic weighting factor Enhanced image feature vectors Reweight the vectors to convert them into weighted enhanced image feature vectors. .
[0034] Next, the original text feature vector , describing text feature vectors Original image feature vector and weighted enhanced image feature vector By splicing the data, the fused features are obtained. , represented as: ; in, This indicates a splicing operation.
[0035] S404. The fused features are mapped to the relation category space through a linear classification layer, and the relation classification results for the target entity pair are output.
[0036] For example, fusion features Input a linear classification layer (composed of fully connected layers) and base it on the prior distribution of each relation category contained in the training dataset. Inject class bias weights into the output of the linear classification layer. This is achieved by compensating for the activation threshold to improve the model's ability to capture rare relation categories in long-tailed distributions.
[0037] Optionally, the relationship classification result can be represented as: ; in, This represents the relation classification result output by the linear classification layer. This represents the Softmax activation function. This represents the weight matrix of the linear classification layer. Indicates fusion characteristics, This represents the inherent bias term of the linear classification layer. This represents the class bias weights obtained in advance based on the prior distribution of each relation category contained in the training dataset.
[0038] For example, the weight matrix of the linear classification layer is used to map high-dimensional fused features to the category space, and the inherent bias term is used to perform a basic translation shift on the linear transformation result.
[0039] Here, class bias weight The calculation process is as follows: A1. Traverse the training dataset (total number of training samples) ), count each relation category Corresponding number of training samples ; A2. Calculate the probability of each relation category occurring. : ; in, An index representing the relation category.
[0040] A3. Calculate each relation category using negative logarithms or reciprocals. Corresponding bias weights , represented as: ; A4, comprised of all relation categories Corresponding bias weights Constitutes class bias weights , represented as ,in, Indicates the total number of relation categories, superscript This indicates transpose.
[0041] Optionally, the target loss function used by the target relation extraction network during training can be expressed as: ; in, Describes the target loss function. This represents the cross-entropy loss function based on class bias weights. Indicates the preset weighting factor. This represents a weighted contrastive loss function based on dynamic weighting factors.
[0042] For example, the weighted contrastive learning loss function Cross-entropy loss function for classification tasks We perform weighted summation to construct the target loss function. Finally, we use the backpropagation algorithm to calculate the gradient and update the parameters of the target relation extraction network, completing the end-to-end model optimization training.
[0043] Alternatively, the cross-entropy loss function can be expressed as: ; in, This represents the total number of training samples in the training dataset. Indicates the index of the training sample. This represents the class bias weights obtained based on the prior distribution of each relation category contained in the training dataset. Indicates the first The relation labels corresponding to each training sample Indicates the first The predicted relationship classification results corresponding to each training sample.
[0044] For example, to address the long-tail distribution problem in multimodal relationship data, a class bias weight is introduced into the cross-entropy loss function. This enhances the model's ability to identify relationships in small samples by increasing the proportion of loss for rare relationship categories. Relationship Labels It can include ( (a positive integer greater than 1) are known semantic relation categories.
[0045] Alternatively, the weighted contrastive loss function can be expressed as: ; in, Represents dynamic weighting factors. This represents the contrastive loss function. This represents the original text feature vector sample. This represents a sample of text feature vectors.
[0046] For example, in a contrastive learning alignment task, dynamic weight factors are used. Introduced as a loss adjustment factor at the sample level into the contrastive loss function This yields the weighted contrastive loss function.
[0047] In this embodiment, dynamic weighting factors are used. The contrastive loss is adjusted to ensure that the model automatically increases the gradient contribution of high-quality aligned samples during training, while reducing the interference of low-quality noisy samples on model convergence. This loss function design incorporates... The core logic lies in achieving sample-level adaptive gradient adjustment when semantically enhanced images... When semantic shifts occur, due to their corresponding similarity and A lower value significantly reduces the gradient contribution of the noisy sample in the target loss function. This mechanism effectively prevents the model from forcibly fitting incorrect cross-modal alignment relationships during training, thus ensuring that the model maintains robustness of the feature space and purity of the representation when facing high-noise data such as social media.
[0048] The present invention provides a multimodal relation extraction method based on goal-oriented description enhancement and dynamic weighting, which has the following significant advantages compared with the prior art: 1. Strong noise suppression capability: through dynamic weighting factors Mapping can automatically identify and downweight semantically enhanced images of poor quality, effectively solving the "semantic drift" problem caused by generative models.
[0049] 2. Precise positioning: Targeted guidance prompts force the large language model to focus on the logical relationships between entities rather than the global background, reducing interference from irrelevant objects in the image and improving the accuracy of relationship extraction.
[0050] 3. More stable training process: The introduction of a similarity-weighted loss function makes the model more tolerant to noisy samples in the early stage of training, the loss descent curve is smoother, and the convergence efficiency is improved.
[0051] 4. Long-tail distribution optimization: By adjusting the class bias weights, the recall ability of the model in rare and small sample relationship categories is significantly improved.
[0052] The following simulation experiment further illustrates the multimodal relation extraction method based on goal-oriented description enhancement and dynamic weighting provided by this invention.
[0053] This invention is implemented based on the PyTorch deep learning framework. The hardware environment utilizes a high-performance computing cluster configured with vGPU-48G, and the software environment includes Python 3.9 and above, with pre-installed necessary dependency libraries such as Transformers, Accelerate, and ModelScope. In a preferred embodiment, the Multimodal Large Language Model (MLLM) is used as the core for semantic extraction, specifically the MiniCPM-Llama3-V-2.5 model. This model, through a hybrid expert architecture and a high-resolution visual encoder, can accurately parse fine-grained entity features in complex social media images. A sampling mode is enabled to improve semantic generalization ability in complex contexts. The sampling temperature is set to 0.7, a value verified through multiple rounds of experiments, balancing the creativity and logical stability of the descriptive text. Furthermore, to ensure that the generated descriptive text focuses on the core interactions between entities and filters redundant information, the maximum generated length is limited to 256. The experimental dataset uses the Multimodal Relation Extraction Standard Benchmark Dataset MNRE (containing 9201 pairs of image-text samples, covering 23 preset relation labels). To verify the directional guidance prompts and dynamic weighting factors proposed in this invention The effectiveness of this invention was demonstrated through end-to-end training on the MNRE dataset. The proposed method was then compared with existing technologies, including the CAMIM network (based on maximizing mutual information) for description-aware multimodal relation extraction and three ablation experiments. The ablation experiments removed the dynamic weight factor feedback mechanism, the directional guidance prompt word module, and the category bias weights to verify the independent contribution of each innovation. The experimental comparison results are shown in Table 1. Table 1. Performance Comparison of Multimodal Relation Extraction
[0054] In Table 1, Accuracy represents accuracy, Precision represents precision, Recall represents recall, and F1Score represents F1 score.
[0055] As can be observed from the experimental data in Table 1, the method of this invention achieves excellent overall performance on the MNRE dataset. Its core advantage stems from the deep synergy of visual information purification, generation noise suppression, and long-tail distribution optimization. First, the directional guidance mechanism achieves precise semantic focusing. Experimental comparisons show that by calling the MiniCPM-Llama3-V-2.5 model in conjunction with directional guidance prompts, the F1 score is improved by 0.47% compared to the scheme that removes directional guidance prompts. This improvement proves that guidance instructions constructed for specific entity pairs can effectively drive the multimodal large model to accurately extract interaction features related to the target entity in complex social media contexts, thereby filtering out semantic interference from irrelevant and redundant objects at the source. Second, the dynamic weight factor feedback mechanism ensures the reliability of the enhanced modality. Compared with the scheme that removes dynamic weight factors, it can be seen that after introducing dynamic weight factors based on the absolute value mapping of cosine similarity, the model exhibits stronger robustness. The core value of this mechanism lies in establishing an adaptive quality evaluation system that can automatically identify and reduce the weighting of possible hallucination noise generated by the model in real time based on the semantic consistency between the original image and the semantically enhanced image, ensuring the purity of the fused features. Furthermore, adjusting the class bias weights improved the accuracy of class discrimination. A comparison with schemes that removed the class bias weights revealed that injecting a class bias designed for long-tailed distributions further improved the F1 score. This confirms that by compensating for the activation threshold, the model can more effectively capture rare relational classes in long-tailed distributions.
[0056] In summary, this invention outperforms traditional CAMIM methods and various ablation experimental schemes. Experimental results strongly demonstrate that this fusion framework, combining goal-oriented enhancement, quality-aware feedback, and long-tail bias adjustment, can systematically solve industry challenges such as modal alignment defocusing, inconsistent sample quality, and uneven category distribution in social media scenarios, demonstrating significant technological innovation and application prospects.
[0057] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.
[0058] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.
[0059] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.
[0060] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights, characterized in that, include: Obtain the original text, original image, and corresponding target entity pairs; Based on the original text, the target entity pairs, and the preset targeted guidance prompt word template, a target guidance description text is generated using a large language model; The original image is input into the target diffusion model, which outputs a semantically enhanced image. The original text, the target guiding description text, the original image, and the semantically enhanced image are input into a pre-trained target relation extraction network, which outputs relation classification results for the target entity pairs.
2. The multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights according to claim 1, characterized in that, The target relation extraction network includes a text encoder, an image encoder, a feature fusion network, and a linear classification layer; The step of inputting the original text, the target guiding descriptive text, the original image, and the semantically enhanced image into a pre-trained target relation extraction network, and outputting relation classification results for the target entity pairs, includes: The text encoder performs feature mapping on the original text and the target guiding descriptive text to obtain the original text feature vector and the descriptive text feature vector; The image encoder extracts spatial features from the original image and the semantically enhanced image to obtain feature vectors for the original image and the enhanced image. The feature fusion network is used to fuse the original text feature vector, the descriptive text feature vector, the original image feature vector, and the enhanced image feature vector to obtain fused features. The fused features are mapped to the relation category space through the linear classification layer, and the relation classification result for the target entity pair is output.
3. The multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights according to claim 2, characterized in that, The step of fusing the original text feature vector, the descriptive text feature vector, the original image feature vector, and the enhanced image feature vector through the feature fusion network to obtain fused features includes: The feature fusion network is used to calculate the similarity between the original image feature vector and the enhanced image feature vector to obtain semantic similarity; and the semantic similarity is linearly mapped to obtain a dynamic weight factor. The enhanced image feature vector is weighted based on the dynamic weighting factor to obtain a weighted enhanced image feature vector. The original text feature vector, the descriptive text feature vector, the original image feature vector, and the weighted enhanced image feature vector are concatenated to obtain the fused feature.
4. The multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights according to claim 3, characterized in that, The dynamic weighting factor is expressed as: ； in, Represents dynamic weighting factors. Indicates semantic similarity. Represents the feature vector of the original image. Represents the enhanced image feature vector. This represents the preset proportional mapping coefficient.
5. The multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights according to claim 4, characterized in that, The weighted enhanced image feature vector is represented as: ； in, This represents the weighted enhanced image feature vector.
6. The multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights according to claim 5, characterized in that, The relationship classification result is expressed as follows: ； in, This represents the relation classification result output by the linear classification layer. This represents the Softmax activation function. This represents the weight matrix of the linear classification layer. Indicates fusion characteristics, This represents the inherent bias term of the linear classification layer. This represents the class bias weights obtained in advance based on the prior distribution of each relation category contained in the training dataset.
7. The multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights according to claim 6, characterized in that, The target loss function used by the target relation extraction network during training is expressed as: ； in, Describes the target loss function. This represents the cross-entropy loss function based on the class bias weights. Indicates the preset weighting factor. This represents the weighted contrastive loss function based on the dynamic weighting factor.
8. The multimodal relation extraction method based on goal-oriented description enhancement and dynamic weights according to claim 7, characterized in that, The cross-entropy loss function is expressed as: ； in, This represents the total number of training samples in the training dataset. Indicates the index of the training sample. This represents the class bias weights obtained based on the prior distribution of each relation category contained in the training dataset. Indicates the first The relation labels corresponding to each training sample Indicates the first The predicted relation classification results corresponding to each training sample; The weighted contrast loss function is expressed as: ； in, This represents the dynamic weighting factor. This represents the contrastive loss function. This represents the original text feature vector sample. This represents a sample of text feature vectors.