Construction method, extraction model and extraction method of multimodal information extraction instruction dataset

Abstract

This invention relates to the field of computer vision technology, and discloses a method for constructing a multimodal information extraction instruction dataset, an extraction model, and an extraction method. The construction method includes: designing different prompt word templates for different modal data according to the extraction task; sending an information extraction request and calling an open-source multimodal large model to obtain entity, relationship, attribute, and event information of the multimodal data; running the execution program of the prompt word templates and outputting initial JSON results; filtering out JSON results that meet the criteria as preliminary JSON results; filtering out JSON results that meet the criteria from the preliminary JSON results as verified JSON results; and constructing a multimodal information extraction instruction dataset from multiple verified JSON results corresponding to the multimodal data. The multimodal information extraction model is obtained by applying the multimodal information extraction instruction dataset and by supervised instruction fine-tuning training. This invention can alleviate the problems of insufficient multimodal information extraction instruction datasets and the illusion of large multimodal models, thereby improving the accuracy of multimodal information extraction.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for constructing a multimodal information extraction instruction dataset, an extraction model, and an extraction method. Background Technology

[0002] Multimodal information extraction technology refers to extracting information such as entities, relationships, attributes, and events from modalities such as text, images, and videos generated in the industrial field. Multimodal information extraction typically employs different extraction techniques for different modalities, which can easily lead to problems such as long model training time, high development costs, large resource consumption, long development cycles, and poor extraction results. With the development of large language models using Decoder-Only Transformer architectures such as GPT4, significant progress has been made in the research and application of large multimodal models, and techniques for information extraction using large multimodal models have gained widespread attention, research, and application.

[0003] Traditional techniques for extracting entity, relation, attribute, and event information from multimodal data require the development of a corresponding extraction algorithm for each modality and extraction task. Combining different modalities and tasks necessitates dozens of different models to complete the multimodal information extraction task, resulting in high resource and time costs. Furthermore, pipeline-based information extraction methods can propagate and accumulate errors, leading to poor extraction performance. In the framework of large-scale multimodal models such as LLaVA, the integration of feature vectors from different modalities is achieved through simple concatenation during the image and text feature fusion stage. This fusion does not fully utilize textual and image information. Additionally, the lack of instruction-based fine-tuning datasets further contributes to the poor information extraction performance of large-scale multimodal models.

[0004] In summary, extracting multimodal information based on large multimodal models faces two main challenges. First, there is a lack of instruction datasets for multimodal information extraction. Due to the specific nature of the domain, collecting training data is difficult, and the absence of instruction datasets for text, images, and videos results in poor multimodal information extraction performance. Second, because the Transformer architecture involves sequential prediction of single words, and pre-training data contains errors, the extracted multimodal data suffers from issues such as non-standardization, non-compliance with instructions, errors, and omissions. Summary of the Invention

[0005] The purpose of this invention is to provide a method for constructing a multimodal information extraction instruction dataset, an extraction model, and an extraction method. The multimodal large-scale model extraction technique based on a stacked attention network fusion strategy first constructs real-time and historical multimodal instruction datasets (text, images, videos, etc.) for fine-tuning the information extraction instructions of the multimodal large-scale model. Secondly, it employs a stacked attention network to enhance the modality fusion strategy, reducing illusion problems such as the multimodal large-scale model not following instructions, creating unnecessary data, errors, and omissions.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for constructing a multimodal information extraction instruction dataset, comprising the following steps:

[0008] S1. Design different prompt word templates for different modal data according to the extraction task. Each prompt word template can meet the extraction function of entity, relation, attribute and event information of the corresponding modal data.

[0009] S2. Send an information extraction request, call the open-source multimodal large model to obtain entity, relationship, attribute and event information of multimodal data, that is, insert the corresponding modal data into the preset position of the prompt word template;

[0010] S3. Run the program that executes the prompt word template and output the initial JSON result;

[0011] S4. Based on preset initial screening criteria, select JSON results that meet the criteria from the initial JSON results to obtain the initial screening JSON results;

[0012] As one possible implementation, S4 includes:

[0013] S40. Determine whether the initial JSON result meets the standard JSON format. If yes, execute S41; otherwise, execute S3.

[0014] As one possible implementation, S40 includes the following steps:

[0015] S400. Determine if the string generated by the open-source multimodal large model is a number. If yes, return False; otherwise, execute S401.

[0016] S401. Use Python's built-in Json library to convert a string to JSON. Return True if the conversion is successful, otherwise return False.

[0017] S41. Determine whether all JSON results output by S40 that meet the standard format conform to the JSON format defined in the prompt word template; at the same time, for entity, relation, and attribute extraction results, determine whether the JSON key contains "entities", "relations", and "properties", and for event extraction results, determine whether the JSON key contains "events"; in addition, determine whether the value corresponding to the key conforms to the specified data type.

[0018] S42. Save all matching JSON results output by S41, and define them as the initial JSON results.

[0019] S5. Based on preset verification criteria, select JSON results that meet the criteria from the initial JSON results to obtain verified JSON results. The multimodal information extraction instruction dataset is composed of multiple verified JSON results corresponding to the multimodal data.

[0020] Secondly, the present invention provides a multimodal information extraction model, which is obtained by applying the multimodal information extraction instruction dataset and by supervised instruction fine-tuning training.

[0021] As one possible implementation, multimodal information extraction models include:

[0022] A visual encoder is used to receive raw images or video keyframe sequences and output visual block features. The visual encoder can embed each visual block and extract image features to obtain a feature vector.

[0023] The text encoder embeds the prompt word template into a vector space with the same dimension as the feature vector to obtain a query vector with the same dimension as the feature vector, and then inputs it into a stacked attention layer.

[0024] The visual-language connector embeds visual block features from the visual encoder input into the vector space of a large speech model to align visual and textual modalities.

[0025] The stacked attention layer receives visual block features aligned with the text modality from the visual language connector input and query vectors from the text encoder input, and calculates attention scores for the query vectors and visual blocks to search for regions strongly correlated with cue words.

[0026] And the large language model, which is the base layer for multimodal information extraction models, uses tokenizer and embedding layers to segment text and map it to the input vector space of the large language model.

[0027] As one possible implementation, the attention score is calculated as follows:

[0028] Visual block features are mapped to... through a fully connected layer. 3D vector space, in In a dimensional vector space, the query vector and visual block features are added together to calculate the hidden layer feature vector;

[0029] The hidden layer feature vectors are mapped to the distributed space through a fully connected layer, and the attention probability of the visual blocks is calculated.

[0030] Multiple attention layers are used, and the attention score of each attention layer is calculated and updated. The last attention layer only needs to assign weights to the feature vector of the visual block.

[0031] As one possible implementation, the hidden layer feature vector can be calculated using the following method:

[0032]

[0033] in, For hidden layer feature vectors, It is a visual block feature vector It is a dimension of visual features. It is the number of visual blocks. It is a query vector; It is a mapping matrix. It is the bias value. yes A 3D vector representing each visual block. The probability of attention.

[0034] As one possible implementation, the attention probability of a visual block is calculated as follows: :

[0035]

[0036] in, It is a mapping matrix. It is the bias.

[0037] As one possible implementation, the calculation formula for the k-th attention layer is as follows:

[0038]

[0039]

[0040] in Initialize to in the first layer , It refers to the number of attention layers. The initialization method is shown in the following formula:

[0041]

[0042]

[0043] Update using the above iterative formula at each layer. .

[0044] Thirdly, the present invention provides an information extraction method using the multimodal information extraction model provided in the second aspect, comprising the following steps:

[0045] Send an information extraction request and call the multimodal information extraction model provided by the second aspect to obtain entity, relationship, attribute and event information of multimodal data, that is, insert the corresponding modal data into the preset position of the prompt word template;

[0046] Run the executable program of the prompt word template and output the initial JSON result;

[0047] Based on preset initial screening criteria, JSON results that meet the criteria are selected from the initial JSON results to obtain the initial screening JSON results;

[0048] Based on preset verification criteria, JSON results that meet the criteria are selected from the initial JSON results to obtain verified JSON results, which are then used as information extraction results.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] 1. The method for constructing a multimodal information extraction instruction dataset proposed in this invention constructs a relatively accurate and complete multimodal information extraction instruction dataset through steps such as designing prompt word templates for extracting information entities, relationships, attributes, and events from multimodal data such as text, images, and videos; generating open-source multimodal large model results; automated initial screening of results; and verification. This provides data support for training a dedicated large model for multimodal information extraction.

[0051] 2. The multimodal information extraction model proposed in this invention includes the construction of a multimodal information extraction instruction dataset and an improvement of the multimodal large model architecture based on a stacked attention network fusion strategy, which can alleviate problems such as insufficient multimodal information extraction instruction dataset and the illusion of multimodal large models.

[0052] 3. The multimodal information extraction model architecture proposed in this invention uses stacked attention modules to replace simple feature vector concatenation operations, which enhances the fusion of image and text information and improves the accuracy of multimodal information extraction. Attached Figure Description

[0053] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0054] Figure 1 This is a flowchart of the method for constructing a multimodal information extraction instruction dataset proposed in an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of a multimodal information extraction model using a stacked attention network strategy in an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram of the multimodal information extraction model structure in an embodiment of the present invention. Detailed Implementation

[0057] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0058] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component.

[0059] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0060] In the description of this invention, it should be understood that the terms "upper" and "lower" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0061] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0062] Traditional techniques for extracting entity, relation, attribute, and event information from multimodal data require the development of a corresponding extraction algorithm for each modality and extraction task. Combining different modalities and tasks necessitates dozens of different models to complete the multimodal information extraction task, resulting in high resource and time costs. Furthermore, pipeline-based information extraction methods can propagate and accumulate errors, leading to poor extraction performance. In the framework of large-scale multimodal models such as LLaVA, the feature vectors from different modalities are simply concatenated during the image and text feature fusion stage, failing to fully utilize textual and image information. Additionally, the lack of instruction fine-tuning datasets further contributes to poor information extraction performance in industrial applications based on large-scale multimodal models. This embodiment, using the industrial field as an example, proposes a method for constructing a multimodal information extraction instruction dataset, an extraction model, and an extraction method. The multimodal large model extraction technique based on the stacked attention network fusion strategy firstly constructs a multimodal instruction dataset including real-time and historical text, images, and videos for fine-tuning of information extraction instructions for the multimodal large model. Secondly, it enhances the modality fusion strategy by employing a stacked attention network to reduce illusion problems such as the multimodal large model not following instructions, creating new instructions, errors, and omissions.

[0063] Firstly, embodiments of the present invention propose a method for constructing a multimodal information extraction instruction dataset, see [link to relevant documentation]. Figure 1 It includes the following steps:

[0064] S1. Design different prompt word templates for different modal data according to the extraction task. Each prompt word template can meet the extraction function of entity, relation, attribute and event information of the corresponding modal data.

[0065] As an example, S1 includes the following steps:

[0066] S10. Construct instruction datasets for document data extraction tasks of different modalities, including: constructing instruction datasets for entities, relations, and attributes, as well as constructing instruction datasets for event information extraction.

[0067] As an example, the templates for entity, relation, and attribute instruction datasets in different modal document data extraction tasks are as follows:

[0068]

[0069]

[0070] As an example, the template for the event information extraction instruction dataset in different modal document data extraction tasks is as follows:

[0071]

[0072]

[0073] S11. Construct instruction datasets for image and video data extraction tasks of different modalities, including: constructing instruction datasets for entities, relationships, and attributes, as well as constructing instruction datasets for event information extraction.

[0074] As an example, the templates for entity, relation, and attribute instruction datasets in different modal image and video data extraction tasks are as follows:

[0075]

[0076]

[0077] As an example, the template for the event information extraction instruction dataset in different modal image and video data extraction tasks is as follows:

[0078]

[0079]

[0080] S2. Send an information extraction request, call the open-source multimodal large model to obtain entity, relationship, attribute and event information of multimodal data, that is, insert the corresponding modal data into the preset position of the prompt word template;

[0081] As an example, using the instruction dataset built in S1, we can obtain entity, relationship, attribute, and event information of multimodal data by calling a local open-source multimodal large model via API. For example, the document data extraction request method is as follows:

[0082]

[0083] The image and video data extraction request methods are as follows:

[0084]

[0085] S3. Run the program that executes the prompt word template and output the initial JSON result;

[0086] For example, for each extraction request, the document or image / video keyframe sequence to be extracted from the sample is first filled into a preset...<begin_of_doc><end_of_doc> or<begin_of_images><end_of_images> In the location, after calling the interface, the result returned by the multimodal large model is obtained through completion.choices[0].message.

[0087] S4. Based on preset initial screening criteria, select JSON results that meet the criteria from the initial JSON results to obtain the initial screening JSON results;

[0088] As one possible implementation, S4 includes:

[0089] S40. Determine if the initial JSON result meets the standard JSON format. If it does, execute S41; otherwise, execute S3.

[0090] As one possible implementation, S40 includes the following steps:

[0091] S400. Determine whether the string generated by the open-source multimodal large model is a number. If yes, return False; otherwise, execute S401.

[0092] S401. Use Python's built-in Json library to convert a string to JSON. Return True if the conversion is successful, otherwise return False.

[0093] Specifically, the following code is used to determine whether the initial JSON result meets the standard JSON format:

[0094]

[0095] S41. Determine whether all JSON results output from S40 that meet the standard format conform to the JSON format defined in the prompt word template. It should be explained that the standard format JSON refers to JavaScript Object Notation, as defined in the computer science field. The JSON format defined in the prompt word template refers to JSON that conforms to JavaScript Object Notation and the keys and values ​​defined in our designed prompt word template. Simultaneously, for entity, relation, and attribute extraction results, check if the JSON keys "entities", "relations", and "properties" exist; for event extraction results, check if the JSON key "events" exists. Furthermore, determine if the values ​​corresponding to the keys, i.e., the data following the key like "events", conform to the specified data type, i.e., the type of value defined in the prompt word template. For example, if the data type of the value corresponding to the key "entities" is "list", then check if the value of "entities" in the generated JSON result is a list.

[0096] S42. Save all matching JSON results output by S41, and define them as the initial JSON results.

[0097] S5. Based on preset verification criteria, select JSON results that meet the criteria from the initial JSON screening results to obtain verified JSON results. Multiple verified JSON results corresponding to the multimodal data constitute a multimodal information extraction instruction dataset. For example, the preset verification criteria are used to determine whether the content is correct, ensuring the accuracy of the final extraction instruction dataset. This mainly examines the accuracy and completeness of the generated content, i.e., determining whether the generated content is correct and whether the extraction results are complete or redundant.

[0098] Secondly, this invention provides a multimodal information extraction model. This model applies the multimodal information extraction instruction dataset and is obtained through supervised instruction fine-tuning training. Existing large-scale multimodal models generally use simple concatenation methods to fuse image and text features. However, the generation process does not fully utilize image and text information, leading to illusions in the model's generated results, such as errors and omissions. To avoid this, this embodiment proposes a new fusion strategy: using a stacked attention network to fuse image and text features.

[0099] See Figures 2 to 3 As one possible implementation method, multimodal information extraction models include:

[0100] A visual encoder is used to receive raw image or video keyframe sequences and output visual block features. The visual encoder can embed each visual block and extract image features to obtain a feature vector. For example, CLIP-ViT-L / 336px is selected as the visual encoder. Its image processor scales images of different sizes to [336,336] pixels and selects the last latent feature of CLIP as the feature matrix of the image or video keyframe sequence.

[0101] A text encoder embeds prompt word templates into a vector space with the same dimensions as the feature vectors to obtain a query vector with the same dimensions as the feature vectors. This is a term specific to deep learning. To avoid ambiguity, "Query Vector" can be used as a gaze vector and input into a stacked attention layer. For example, an LSTM is chosen as the text encoder. After extracting and segmenting the prompt words, the feature vectors are obtained through a lookup table. Then, a fixed-dimensional feature vector is obtained through the LSTM network and used as input into the stacked attention layer.

[0102] The visual-language connector embeds the visual block features input from the visual encoder into the vector space of a large speech model to align visual and textual modalities. For example, a two-layer perceptron is used as the visual-language connector, and the dimensions become [576, 4096] after the visual features are mapped to the vector space of the large language model.

[0103] A stacked attention layer receives visual block features aligned with the text modality from the visual-language connector and a query vector from the text encoder. It then calculates attention scores for the query vector and the visual blocks, searching for regions strongly correlated with the cue words. As one possible implementation, the attention scores are calculated as follows:

[0104] Visual block features are mapped to... through a fully connected layer. 3D vector space, in In a dimensional vector space, the query vector and visual block features are added together to obtain the hidden layer feature vector. As one possible implementation, the hidden layer feature vector is calculated using the following method:

[0105]

[0106] in, For hidden layer feature vectors, It is a visual block feature vector It is a dimension of visual features. It is the number of visual blocks. It is a query vector; It is a mapping matrix. It is the bias value. yes A 3D vector representing each visual block. The probability of attention.

[0107] The hidden layer feature vectors are mapped to the distributed space through a fully connected layer, and the attention probability of the visual blocks is calculated.

[0108] As one possible implementation, the attention probability of a visual block is calculated as follows: :

[0109]

[0110] in, It is a mapping matrix. It is the bias.

[0111] Compared to simply concatenating text and image feature vectors, stacked attention modules can better represent and assign visual blocks related to text instructions. For complex instructions, multiple attention layers are used, and the attention score of each attention layer is calculated and updated. The last attention layer only needs to assign weights to the feature vector of the visual block.

[0112] As one possible implementation, the calculation formula for the k-th attention layer is as follows:

[0113]

[0114]

[0115] in Initialize to in the first layer , It refers to the number of attention layers. The initialization method is shown in the following formula:

[0116]

[0117]

[0118] Update using the above iterative formula at each layer. The last layer only needs to assign weights to the feature vectors of the image patches.

[0119] Thirdly, embodiments of the present invention provide an information extraction method using the multimodal information extraction model provided in the second aspect, comprising the following steps:

[0120] Send an information extraction request and call the multimodal information extraction model provided by the second aspect to obtain entity, relationship, attribute and event information of multimodal data, that is, insert the corresponding modal data into the preset position of the prompt word template;

[0121] Run the executable program of the prompt word template and output the initial JSON result;

[0122] Based on preset initial screening criteria, JSON results that meet the criteria are selected from the initial JSON results to obtain the initial screening JSON results;

[0123] Based on preset verification criteria, JSON results that meet the criteria are selected from the initial JSON results to obtain verified JSON results, which are then used as information extraction results.

[0124] And the large language model, as the foundational layer of the multimodal information extraction model, uses tokenizer and embedding layers to segment text and map it to the input vector space of the large language model. At each step, it is based on the previous... Prediction of individual words The probability of each word is calculated until a cutoff is generated.

[0125] In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0126] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal information extraction system, characterized in that, The multimodal information extraction system includes a multimodal information extraction model, which is obtained by applying a multimodal information extraction instruction dataset and by supervised instruction fine-tuning training. Multimodal information extraction models include: A visual encoder is used to receive raw images or video keyframe sequences and output visual block features. The visual encoder can embed each visual block and extract image features to obtain a feature vector. The text encoder embeds the prompt word template into a vector space with the same dimension as the feature vector to obtain a query vector with the same dimension as the feature vector, and then inputs it into a stacked attention layer. The visual-language connector embeds visual block features from the visual encoder into the vector space of a large language model to align visual and textual modalities. A stacked attention layer receives visual block features aligned with the text modality from the visual-language connector and a query vector from the text encoder. It then calculates attention scores for the query vector and the visual blocks, searching for regions strongly correlated with the prompt words. The attention scores are calculated by mapping the visual block features through a fully connected layer to... 3D vector space, in In a dimensional vector space, the query vector and visual block features are added together to calculate the hidden layer feature vector. This hidden layer feature vector is mapped to a distributed space through a fully connected layer, and the attention probability of the visual block is calculated. Multiple attention layers are used, and the attention score of each layer is calculated and updated. The last attention layer only needs to assign weights to the visual block feature vector. The hidden layer feature vector is calculated using the following method: ;in, For hidden layer feature vectors, It is a visual block feature vector It is a dimension of visual features. It is the number of visual blocks. It is a query vector; It is a mapping matrix. It is the bias value. yes A dimensional vector represents the attention probability of each visual patch; the attention probability of a visual patch is calculated as follows: : ,in, It is a mapping matrix. This is the bias; the calculation formula for the k-th attention layer is as follows: in Initialize to in the first layer , It refers to the number of attention layers. The initialization method is shown in the following formula: Update using the above iterative formula at each layer. ; In addition, a large language model is used as the base layer for multimodal information extraction models. The tokenizer and embedding layers are used to segment the text and map it to the input vector space of the large language model. The modality fusion strategy is enhanced by a stacked attention network.

2. The multimodal information extraction system according to claim 1, characterized in that, The multimodal information extraction instruction dataset is obtained in the following way: S1. Design different prompt word templates for different modal data according to the extraction task. Each prompt word template can meet the extraction function of entity, relation, attribute and event information of the corresponding modal data. S2. Send an information extraction request, call the open-source multimodal large model to obtain entity, relationship, attribute and event information of multimodal data, that is, insert the corresponding modal data into the preset position of the prompt word template; S3. Run the program that executes the prompt word template and output the initial JSON result; S4. Based on preset initial screening criteria, select JSON results that meet the criteria from the initial JSON results to obtain the initial screening JSON results; S5. Based on preset verification criteria, select JSON results that meet the criteria from the initial JSON results to obtain verified JSON results. The verified JSON results corresponding to the multimodal data constitute a multimodal information extraction instruction dataset.

3. The multimodal information extraction system according to claim 2, characterized in that, S4 includes: S40. Determine whether the initial JSON result meets the standard JSON format. If yes, execute S41; otherwise, execute S3. S41. Determine whether all JSON results output by S40 that meet the standard format conform to the JSON format defined in the prompt word template; at the same time, for entity, relation, and attribute extraction results, check whether the JSON keys "entities", "relations", and "properties" exist, and for event extraction results, check whether the JSON key "events" exists; in addition, determine whether the value corresponding to the key conforms to the specified data type. S42. Save all matching JSON results output by S41, and define them as the initial JSON results.

4. The multimodal information extraction system according to claim 3, characterized in that, S40 includes the following steps: S400. Determine whether the string generated by the open-source multimodal large model is a number. If yes, return False; otherwise, execute S401. S401. Use Python's built-in Json library to convert a string to JSON. Return True if the conversion is successful, otherwise return False.

5. An information extraction method using the multimodal information extraction system according to any one of claims 1 to 4, characterized in that, Includes the following steps: Send an information extraction request and invoke the multimodal information extraction system according to any one of claims 1 to 4 to obtain entity, relation, attribute and event information of multimodal data, that is, insert the corresponding modal data into the preset position of the prompt word template; Run the executable program of the prompt word template and output the initial JSON result; Based on preset initial screening criteria, JSON results that meet the criteria are selected from the initial JSON results to obtain the initial screening JSON results; Based on preset verification criteria, JSON results that meet the criteria are selected from the initial JSON results to obtain verified JSON results, which are used as information extraction results.

Images