A method for information detection based on multimodal deep learning

Abstract

This invention relates to the field of information detection technology, and more particularly to an information detection method based on multimodal deep learning. The invention inputs text, invokes a set of associations associated with the text, extracts feature text information from each document and feature image information from each image, sequentially obtains the association values ​​between each document and each document and image within the association set, classifies the association set according to the association values, and selects a detection strategy based on the association set category. This includes identifying entries with differing field values ​​and marking those entries; or, based on the document association degree between a document and each document within the association set, and the image association degree between a document and each image within the association set, extracting feature text information from the document and feature image information from the image. This invention classifies the association set category by calculating the association between documents and images and adopts corresponding methods to detect information, thereby improving the accuracy and completeness of information detection.

Description

Technical Field

[0001] This invention relates to the field of information detection technology, and in particular to an information detection method based on multimodal deep learning. Background Technology

[0002] In user interaction scenarios of artificial intelligence systems, text and images may each carry real information, but also conceal forgery or noise: text can be fabricated, misidentified by OCR, or numerically tampered with, while images may be deepfake, photoshopped, or deceived by metadata. Jointly modeling semantic consistency, temporal rationality, and physical feasibility in cross-modal representation space, and achieving authenticity verification through adversarial detection, cross-modal alignment, and credibility weighting, has become a research problem that urgently needs to be solved in the current multimodal information fusion.

[0003] For example, Chinese Patent Publication No. CN113239926A discloses an adversarial multimodal fake information detection model, relating to the field of fake information detection technology. This model solves the problem of frequent errors in existing fake information spoofing detectors. The device includes a visual feature extraction module, a multimodal pre-training module, a collaborative visual-text multi-head attention module, and an adversarial training module. The multimodal pre-training module is used to obtain multi-fused text sequence features, visual sequence features, and image-text content features. The collaborative visual-text multi-head attention module is used to obtain text content features and visual content features representing different levels of relationships between multimodal cues. The adversarial training module is used to train the model adversarially from text, visual, and image-text perspectives to improve robustness. This invention can capture the element-level and modal-level relationships between multimodal cues in information data, fully exploring the hidden connections between cues; enabling the multimodal pre-training module to be directly applied to fake information detection.

[0004] However, the following problems still exist in the existing technology.

[0005] The text and image information obtained by searching by inputting text may be repetitive or complementary. Existing technologies do not classify text and images, making information integration too complex. In practice, it is difficult to identify false information. Existing technologies do not identify differences based on the correlation between text and images, resulting in a low success rate in identifying false information. Summary of the Invention

[0006] To address this, the present invention provides an information detection method based on multimodal deep learning, which overcomes the problems in existing technologies where text and images are not classified, leading to overly complex information integration and a low success rate in identifying false information due to the lack of identification of discrepancies based on the correlation between text and images.

[0007] To achieve the above objectives, the present invention provides an information detection method based on multimodal deep learning, comprising:

[0008] Input text, and retrieve the associated set of documents and images based on the text.

[0009] Extract the feature text information from each document and the feature image information from each image.

[0010] Based on the feature text information and the feature image information, the association values ​​between each document and each document and each image in the association set are obtained in sequence, and the association set categories are divided according to the association values.

[0011] A detection strategy is selected based on the categories of the association set, including:

[0012] Based on the input text, a fact element template is generated. The feature text information of each document and the feature image information of each image in the associated set are traversed and mapped to the corresponding fields of the fact element template. Entries with different field values ​​are identified and marked.

[0013] Alternatively, based on the input text, a fact element template is generated. Based on the document correlation degree between the document and each document in the association set, and the image correlation degree between the document and each image in the association set, feature text information in the document and feature image information in the image are extracted, and the feature text information and feature image information are mapped to the corresponding fields of the fact element template.

[0014] Furthermore, the process of calling the associated set of text based on the text includes,

[0015] The text is converted into a text semantic vector through a pre-trained language model;

[0016] The matching degree between the text semantic vector and the document semantic vector of each document and the visual feature vector of each image is calculated using the cosine similarity formula.

[0017] Documents and images with a matching degree higher than a preset matching degree threshold are selected to form an association set.

[0018] Furthermore, if the number of documents and images included in the associated set is less than a preset threshold, the preset matching degree threshold is adjusted.

[0019] Furthermore, the process of extracting feature text information from each document includes,

[0020] A named entity recognition model is used to scan each document sentence by sentence to extract feature text containing people, places, organizations, and times;

[0021] The feature text is mapped into a fixed-dimensional document semantic vector using a pre-trained language model;

[0022] The document semantic vector is the feature text information.

[0023] Furthermore, the process of extracting feature image information from each image includes,

[0024] A convolutional neural network is used to extract features from each image sequentially, generating a visual feature vector that includes image texture, contour, and color distribution.

[0025] OCR technology is used to identify the text information embedded in images, and a pre-trained language model is used to map the embedded text information into a fixed-dimensional image text semantic vector.

[0026] The feature image information includes visual feature vectors and image text semantic vectors.

[0027] Furthermore, the process of sequentially obtaining the association values ​​between each document and each document and image within the associated set includes,

[0028] Documents are obtained sequentially, and the similarity between the feature text information of the document and the feature text information of each document in the association set is calculated using the cosine similarity formula and recorded as the document association degree.

[0029] The cross-modal similarity between the feature text information of the document and the visual feature vectors and semantic vectors of each image in the associated set is calculated using the cosine similarity formula. The cross-modal similarities are then weighted and summed to obtain the image association degree.

[0030] The document relevance and the image relevance are weighted and summed to obtain the relevance value of each document;

[0031] The average of several associated values ​​is denoted as the associated value of the associated set.

[0032] Furthermore, the process of classifying association sets into categories based on association values ​​includes,

[0033] The association values ​​of the association set are compared with a preset association value comparison threshold.

[0034] If the association value of the association set is less than the preset association value comparison threshold, the association set is classified as a complementary category.

[0035] If the association value of an association set is greater than or equal to a preset association value comparison threshold, then the association set is classified as a similarity category.

[0036] Furthermore, a fact element template is generated, which includes: event type, time, location, subject, object, and value.

[0037] Furthermore, the process of identifying and marking entries with differing field values ​​includes,

[0038] Retrieve the document and image containing each sub-segment value;

[0039] The average value of the image association degree of each image is recorded as the image association value of each image;

[0040] Sort the association values ​​of each document and each image in descending order, and mark the document and image serial numbers in the corresponding field values.

[0041] Furthermore, based on the document association degree between a document and each document in the association set, and the image association degree between a document and each image in the association set, feature text information from the document and feature image information from the image are extracted.

[0042] If a document has a document correlation degree with a document in the associated set that is less than a preset document correlation degree threshold, then that document is extracted;

[0043] If there exists a document and an image within the associated set whose image association degree is less than a preset image association degree threshold, then the image is extracted;

[0044] Extract the feature text information from the document and the feature image information from the image.

[0045] Compared with existing technologies, this invention, by inputting text, invokes an association set associated with the text, extracts feature text information from each document and feature image information from each image, sequentially obtains the association values ​​between each document and each document and image within the association set, classifies the association set according to the association values, and selects a detection strategy based on the association set category. This includes identifying entries with different field values ​​and marking the entries; or, based on the document association degree between the document and each document within the association set and the image association degree between the document and each image within the association set, extracting feature text information from the document and feature image information from the image. This invention classifies the association set category by calculating the association between documents and images and adopts corresponding methods to detect information, thereby improving the accuracy and completeness of information detection.

[0046] In particular, this invention obtains the association values ​​between each document and each document and each image in the association set based on feature text information and feature image information. In practice, the documents and images obtained through input text may be largely repetitive or contain different information. By obtaining the association values ​​between each document and each document and each image in the association set, the association within the association set can be characterized to a certain extent. This effectively distinguishes the repetition and complementarity of information obtained from the input text search, and then adopts corresponding detection strategies for different types of information. This avoids the redundant interference of a large amount of repetitive information and fully integrates complementary information, thereby improving the accuracy of information detection.

[0047] In particular, the present invention identifies entries with differing field values. In practice, false information may exist in the association set obtained by inputting text. Identifying entries with differing field values ​​can increase the likelihood of discovering false information, thereby improving the accuracy of information detection.

[0048] In particular, this invention arranges the association values ​​of each document and each image in descending order and identifies the document and image serial numbers in the corresponding field values. In reality, documents and images may contain real and false information, and the association values ​​between documents and images can characterize the authenticity of the information within them to a certain extent. The higher the association value between a document and an image, the higher the likelihood of the information being authentic. By arranging the association values ​​of each document and each image in descending order and identifying the document and image serial numbers in the corresponding field values, the authenticity of the field values ​​can be characterized to a certain extent based on the serial numbers, thereby increasing the accuracy of information detection.

[0049] In particular, this invention extracts feature text information from documents and feature image information from images based on the document correlation degree between documents and each document in the association set and the image correlation degree between documents and each image in the association set. The smaller the correlation degree between documents and images, the higher the complementarity of information between documents and images may be. By extracting information from documents and images with low correlation and making them complementary to each other, the efficiency can be improved while ensuring the integrity of information detection. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the steps of an information detection method based on multimodal deep learning, as described in an embodiment of the invention.

[0051] Figure 2 A logical decision diagram for classifying related set categories in an embodiment of the invention;

[0052] Figure 3 This is a selection diagram for selecting a detection strategy based on the association set category in an embodiment of the invention. Detailed Implementation

[0053] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0054] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0055] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0056] Please see Figure 1 - Figure 3 As shown, an information detection method based on multimodal deep learning according to the present invention includes:

[0057] Step S1: Input text, and call the association set associated with the text based on the text. The association set includes several documents and several images.

[0058] Step S2: Extract the feature text information from each document and the feature image information from each image.

[0059] Step S3: Based on the feature text information and the feature image information, sequentially obtain the association values ​​between each document and each document and each image in the association set, and classify the association set categories according to the association values;

[0060] Step S4, select a detection strategy based on the association set category, including,

[0061] Based on the input text, a fact element template is generated. The feature text information of each document and the feature image information of each image in the associated set are traversed and mapped to the corresponding fields of the fact element template. Entries with different field values ​​are identified and marked.

[0062] Alternatively, based on the input text, a fact element template is generated. Based on the document correlation degree between the document and each document in the association set, and the image correlation degree between the document and each image in the association set, feature text information in the document and feature image information in the image are extracted, and the feature text information and feature image information are mapped to the corresponding fields of the fact element template.

[0063] In this embodiment, there is a database containing several documents and images. By entering text into the database, one can search for the documents and images in the database.

[0064] Specifically, the process of calling the associated set of texts based on the text includes,

[0065] The text is converted into a text semantic vector through a pre-trained language model;

[0066] The matching degree between the text semantic vector and the document semantic vector of each document and the visual feature vector of each image is calculated using the cosine similarity formula.

[0067] Documents and images with a matching degree higher than a preset matching degree threshold are selected to form an association set.

[0068] In this embodiment, the pre-trained language model is the BERT model.

[0069] In this embodiment, the preset matching degree threshold is selected within the range of [0.6, 0.8].

[0070] Specifically, if the number of documents and images included in the associated set is less than the preset number threshold, the preset matching degree threshold will be adjusted.

[0071] In this embodiment, the preset quantity threshold is selected within the range of [3,5].

[0072] In this embodiment, if the association set contains one document and one image, the preset matching threshold is reduced, thereby increasing the number of documents and images contained in the association set.

[0073] Specifically, the process of extracting feature text information from each document includes,

[0074] A named entity recognition model is used to scan each document sentence by sentence to extract feature text containing people, places, organizations, and times;

[0075] The feature text is mapped into a fixed-dimensional document semantic vector using a pre-trained language model;

[0076] The document semantic vector is the feature text information.

[0077] In this embodiment, the named entity recognition model is the BERT-CRF named entity recognition model.

[0078] Specifically, the process of extracting feature image information from each image includes,

[0079] A convolutional neural network is used to extract features from each image sequentially, generating a visual feature vector that includes image texture, contour, and color distribution.

[0080] OCR technology is used to identify the text information embedded in images, and a pre-trained language model is used to map the embedded text information into a fixed-dimensional image text semantic vector.

[0081] The feature image information includes visual feature vectors and image text semantic vectors.

[0082] In this embodiment, a ResNet50 convolutional neural network is used.

[0083] Specifically, the process of sequentially obtaining the association values ​​of each document with each document and each image within the associated set includes,

[0084] Documents are obtained sequentially, and the similarity between the feature text information of the document and the feature text information of each document in the association set is calculated using the cosine similarity formula and recorded as the document association degree.

[0085] The cross-modal similarity between the feature text information of the document and the visual feature vectors and semantic vectors of each image in the associated set is calculated using the cosine similarity formula. The cross-modal similarities are then weighted and summed to obtain the image association degree.

[0086] The document relevance and the image relevance are weighted and summed to obtain the relevance value of each document;

[0087] The average of several associated values ​​is denoted as the associated value of the associated set.

[0088] In this embodiment, the weighting coefficient of the cross-modal similarity between the feature text information of the document and the visual feature vector of the image in the associated set is 0.6, and the weighting coefficient of the cross-modal similarity between the feature text information of the document and the image text semantic vector of the image in the associated set is 0.4.

[0089] In this embodiment, the weighting coefficient for document relevance is 0.5, and the weighting coefficient for image relevance is 0.5.

[0090] Specifically, this invention obtains the association values ​​between each document and each document and each image in the associated set based on feature text information and feature image information. In practice, the documents and images obtained through input text may be largely repetitive or contain different information. By obtaining the association values ​​between each document and each document and each image in the associated set, the association within the associated set can be characterized to a certain extent. This effectively distinguishes between the repetition and complementarity of information obtained from the input text search, and then adopts corresponding detection strategies for different types of information. This avoids the redundant interference of a large amount of repetitive information and fully integrates complementary information, thereby improving the accuracy of information detection.

[0091] Specifically, the process of classifying association sets into categories based on association values ​​includes,

[0092] The association values ​​of the association set are compared with a preset association value comparison threshold.

[0093] If the association value of the association set is less than the preset association value comparison threshold, the association set is classified as a complementary category.

[0094] If the association value of an association set is greater than or equal to a preset association value comparison threshold, then the association set is classified as a similarity category.

[0095] In this embodiment, the preset correlation value comparison threshold is selected within the range of [0.6, 0.7].

[0096] Specifically, a detection strategy is selected based on the category of the association set.

[0097] If the association set category is a similarity category, then based on the input text, a fact element template is generated, the feature text information of each document and the feature image information of each image in the association set are traversed, and mapped to the corresponding fields of the fact element template. Entries with different field values ​​are identified and marked.

[0098] If the association set category is a complementary category, then based on the input text, a fact element template is generated. Based on the document association degree between the document and each document in the association set, and the image association degree between the document and each image in the association set, feature text information in the document and feature image information in the image are extracted, and the feature text information and feature image information are mapped to the corresponding fields of the fact element template.

[0099] Specifically, this invention identifies entries with differing field values ​​and marks them accordingly. In practice, false information may exist within the association set obtained through input text. Identifying entries with differing field values ​​can increase the likelihood of discovering false information, thereby improving the accuracy of information detection.

[0100] Specifically, a fact element template is generated, which includes: event type, time, location, subject, object, and value.

[0101] In this embodiment, the input text is input into a pre-trained event extraction model. In this embodiment, a BERT-based event detection model is selected. Based on the identified event type, the corresponding basic template framework is called from the preset event element ontology library. The semantic matching model maps the feature text information of each document and the feature image information of the image to the fields of the basic template framework.

[0102] Specifically, the process of identifying entries with differing field values ​​and marking those entries includes,

[0103] Retrieve the document and image containing each sub-segment value;

[0104] The average value of the image association degree of each image is recorded as the image association value of each image;

[0105] Sort the association values ​​of each document and each image in descending order, and mark the document and image serial numbers in the corresponding field values.

[0106] Specifically, this invention arranges the association values ​​of each document and each image in descending order, and identifies the document and image serial numbers in the corresponding field values. In reality, documents and images may contain real and false information, and the association values ​​between documents and images can characterize the authenticity of the information within them to a certain extent. The higher the association value between a document and an image, the more likely the information contained within is to be authentic. By arranging the association values ​​of each document and each image in descending order and identifying the document and image serial numbers in the corresponding field values, the authenticity of the field values ​​can be characterized to a certain extent based on the serial numbers, thereby increasing the accuracy of information detection.

[0107] Specifically, based on the document association degree between a document and each document in the association set, and the image association degree between a document and each image in the association set, feature text information from the document and feature image information from the image are extracted.

[0108] If a document has a document correlation degree with a document in the associated set that is less than a preset document correlation degree threshold, then that document is extracted;

[0109] If there exists a document and an image within the associated set whose image association degree is less than a preset image association degree threshold, then the image is extracted;

[0110] Extract the feature text information from the document and the feature image information from the image.

[0111] In this embodiment, the preset document relevance threshold is selected within the range of [0.5, 0.7].

[0112] In this embodiment, the preset image correlation threshold is selected within the range of [0.5, 0.7].

[0113] Specifically, this invention extracts feature text information from documents and feature image information from images based on the document correlation degree between documents and each document in the association set, and the image correlation degree between documents and each image in the association set. The smaller the correlation degree between documents and images, the higher the complementarity of information between documents and images may be. By extracting information from documents and images with low correlation, they can complement each other, thereby improving efficiency while ensuring the integrity of information detection.

[0114] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. An information detection method based on multimodal deep learning, characterized in that, include: Input text, and retrieve the associated set of documents and images based on the text. Extract the feature text information from each document and the feature image information from each image. Based on the feature text information and the feature image information, the association values ​​between each document and each document and each image in the association set are obtained in sequence, and the association set categories are divided according to the association values. The process of sequentially obtaining the association values ​​of each document with each document and each image in the associated set includes, Documents are obtained sequentially, and the similarity between the feature text information of the document and the feature text information of each document in the association set is calculated using the cosine similarity formula and recorded as the document association degree. The cross-modal similarity between the feature text information of the document and the visual feature vectors and semantic vectors of each image in the associated set is calculated using the cosine similarity formula. The cross-modal similarities are then weighted and summed to obtain the image association degree. The document relevance and the image relevance are weighted and summed to obtain the relevance value of each document; The average of several associated values ​​is denoted as the associated value of the associated set; The process of classifying association sets according to association values ​​includes, The association values ​​of the association set are compared with a preset association value comparison threshold. If the association value of the association set is less than the preset association value comparison threshold, the association set is classified as a complementary category. If the association value of the association set is greater than or equal to the preset association value comparison threshold, then the association set is classified as a similarity category. A detection strategy is selected based on the categories of the association set, including: Based on the input text, a fact element template is generated. The feature text information of each document and the feature image information of each image in the associated set are traversed and mapped to the corresponding fields of the fact element template. Entries with different field values ​​are identified and marked. Alternatively, based on the input text, a fact element template is generated. Based on the document correlation degree between the document and each document in the association set, and the image correlation degree between the document and each image in the association set, feature text information in the document and feature image information in the image are extracted, and the feature text information and feature image information are mapped to the corresponding fields of the fact element template.

2. The information detection method based on multimodal deep learning according to claim 1, characterized in that, The process of calling the associated set of text based on the text includes, The text is converted into a text semantic vector through a pre-trained language model; The matching degree between the text semantic vector and the document semantic vector of each document and the visual feature vector of each image is calculated using the cosine similarity formula. Documents and images with a matching degree higher than a preset matching degree threshold are selected to form an association set.

3. The information detection method based on multimodal deep learning according to claim 2, characterized in that, If the number of documents and images included in the associated set is less than the preset threshold, then the preset matching degree threshold will be adjusted.

4. The information detection method based on multimodal deep learning according to claim 1, characterized in that, The process of extracting feature text information from each document includes, A named entity recognition model is used to scan each document sentence by sentence to extract feature text containing people, places, organizations, and times; The feature text is mapped into a fixed-dimensional document semantic vector using a pre-trained language model; The document semantic vector is the feature text information.

5. The information detection method based on multimodal deep learning according to claim 1, characterized in that, The process of extracting feature image information from each image includes, A convolutional neural network is used to extract features from each image sequentially, generating a visual feature vector that includes image texture, contour, and color distribution. OCR technology is used to identify the text information embedded in images, and a pre-trained language model is used to map the embedded text information into a fixed-dimensional image text semantic vector. The feature image information includes visual feature vectors and image text semantic vectors.

6. The information detection method based on multimodal deep learning according to claim 1, characterized in that, Generate a fact element template, which includes: event type, time, location, subject, object, and value.

7. The information detection method based on multimodal deep learning according to claim 1, characterized in that, The process of identifying entries with differing field values ​​and marking those entries includes, Retrieve the document and image containing each sub-segment value; The average value of the image association degree of each image is recorded as the image association value of each image; Sort the association values ​​of each document and each image in descending order, and mark the document and image serial numbers in the corresponding field values.

8. The information detection method based on multimodal deep learning according to claim 1, characterized in that, Based on the document association degree between a document and each document in the association set, and the image association degree between a document and each image in the association set, feature text information from the document and feature image information from the image are extracted. If a document has a document correlation degree with a document in the associated set that is less than a preset document correlation degree threshold, then that document is extracted; If there exists a document and an image within the associated set whose image association degree is less than a preset image association degree threshold, then the image is extracted; Extract the feature text information from the document and the feature image information from the image.

Images