A multimodal knowledge representation method for information compliance governance

Abstract

The present invention relates to a multimodal knowledge representation method for information compliance governance. First, the visual modal features are extracted, and the efficiency problem encountered in extracting pictures by convolution is considered. The image is standardized and segmented by a non-convolutional image segmentation method, and then the image data features are obtained by projection; then the key words are extracted from the long text by the KeyBert model, the text is encoded by MacBert, and the original text is semantically replaced by the position information of the key words contained in the text, and the text modal features are obtained by combining the type features of the original text; finally, the BiLSTM-Attention model is used to perform multimodal fusion of the two modal features to achieve semantic representation of government affairs information. The present invention can improve the efficiency of visual modal feature extraction, realize text modal feature extraction for long text data, and complete the effective representation of government affairs information through the fusion model, so as to further identify government affairs information.

Description

Technical Field

[0001] The present invention relates to the technical field of government information processing, and in particular to a multimodal knowledge representation method for information compliance governance. Background Art

[0002] At present, data in the government field is gradually shifting from traditional text to multimodal data, and the integration and application of massive government data has become a problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a multimodal knowledge representation method for information compliance governance, which can bypass the image detection task, alleviate the problem of disparity in the time overhead distribution of feature extraction of different modalities, improve the overall efficiency of the model, and effectively solve the representation problem of multimodal government information with "long text, multiple images" characteristics, and provide technical support for government services for the convenience of the people, integration of multimodal information, etc.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal knowledge representation method for information compliance governance, comprising the following steps:

[0005] Step S1: standardize the image data in the government affairs information and divide it into Patches of the same size, and map them through a linear projection layer to obtain visual modality features;

[0006] Step S2, using the KeyBert pre-trained model to extract keywords from the long text of government affairs information, and then using the MacBert model to obtain the feature representation corresponding to the keywords, and then fusing the modal type feature and the position feature to obtain the text modal feature;

[0007] Step S3: Input the text modal features and visual modal features into the trained BiLSTM-Attention network for feature fusion encoding, and output multimodal feature representation.

[0008] In a preferred embodiment: in step S1, obtaining visual modality features in government affairs information specifically includes the following steps:

[0009] Step S11: Define an image set of government affairs data , the image collection The image in is segmented to obtain a 224×224×3 image. The formula is as follows:

[0010]

[0011] in, Represents the original image data P i To perform conversion segmentation,P i Represents the first i samples, and represents the standardized sample;

[0012] Step S12: traverse the image set , divided into 16×16×3 blocks, and the image block set is obtained Patches , the formula is as follows:

[0013]

[0014] in, Represents the standardized image data Split into 16×16×3 sizes. represents the image data after being standardized in step S11, and Patches i Representative i A subset of segmented image blocks;

[0015] Step S13: Use a linear projection function to project each segmented image block subset to obtain its projection features, then traverse the projection features and corresponding position vector features of each segmented image block subset, and fuse the projection features and corresponding position vector features of each segmented image block subset. The projection and fusion functions are:

[0016]

[0017] in, represents the linear projection function, Patches i Representative i A subset of the image blocks, and Represents the projection result, that is Patches i Image features obtained by linear projection function; represents the feature fusion function, represent Patches i The original position information in the image before segmentation; Representative i The feature representation of the final visual modality is obtained by fusing the position features and projection features of a subset of image patches.

[0018] In a preferred embodiment: in step S2, extracting text modal features from the government affairs information text specifically includes the following steps:

[0019] Step S21, using the basic word segmentation capability of the jieba library and the secondary word segmentation capability of the KeyBert model based on N-gram, calculate and select keywords by comparing the cosine similarity between the original sentence encoding and the word segmentation result encoding; the calculation formula is as follows:

[0020]

[0021] in, t i Represents the original text dataset T The i data samples, Indicates that the jieba library is used t i Perform basic word segmentation. Indicates that after the basic participle Model to calculate and select keywords; Indicates the pairing between the Jieba library and the KeyBert model t i The extracted keyword result set, for The position vector of the corresponding keyword in the original text;

[0022] Step S22: Use the MacBert model to encode the text data sample, use the first layer of the MacBert hidden layer as the word-level feature, and the last layer as the text semantic information, and weight the two to obtain the semantic information of the text data sample, and use the topic word position information to map it to the embedded representation of the topic word. The calculation formula is as follows:

[0023]

[0024] in, V t refers to the text modality feature vector, V t and They are text data samples t i After inputting the MacBert pre-trained model sequence t i Output at the first and last hidden layer; Representative text data samples t i The weighted result of the first and last hidden layers after MacBert encoding, i.e. the text data sample t i Embedding features of Indicates i Keywords Length, Indicatesi Keywords In the text sequence t i The starting position in Represents a sample of text data t i After encoding, it is located at After k The feature vector at the position, Represents the semantic feature results of the subject word;

[0025] Step S23: Add the text keyword vector to the modality type feature information , get the final text modality feature vector V t .

[0026] In a preferred embodiment: In step S3, the obtained text modality feature vector V t , visual modality features V p Input the feature fusion module based on the BiLSTM-Attention architecture to obtain the final semantic representation of government data. The calculation formula is as follows:

[0027]

[0028] In the formula, V p , V t Represent visual modality features and text modality features respectively, V It represents the semantic representation of the final government affairs data.

[0029] Compared with the prior art, the present invention has the following beneficial effects: the non-convolution visual feature extraction method based on image segmentation and linear projection proposed in the present invention completes the standardization and segmentation of image data, and obtains visual modal features through projection, thereby simplifying the extraction process, balancing the computational overhead of text and visual modalities, and improving the overall efficiency. The feature extraction of long texts is realized by replacing the original text semantics with the semantics of the subject words based on the KeyBert and MacBert models. Finally, multimodal fusion of different modal features is performed based on the BiLSTM-Attention model. The effectiveness of the model in this paper is confirmed by comparing it with the existing multimodal representation model of government data; and a more effective representation is obtained in the context of long text data to improve the integration and utilization efficiency of information data in the government field. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1Schematic diagram of a multimodal fusion model of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0034] like Figure 1 As shown in the figure, a multimodal knowledge representation method for information compliance governance is proposed. The method first extracts visual modal features, considers the efficiency problem encountered in extracting images by convolution, uses a convolution-free image segmentation method to standardize and segment the image, and then obtains the image data features by projection; then, the KeyBert model is used to extract keywords from long texts, the text is encoded by MacBert, and the original text is semantically replaced by the position information of the keywords contained in the text, and the text modal features are obtained by combining the type features of the original text; finally, the BiLSTM-Attention model is used to perform multimodal fusion of the two modal features to achieve semantic representation of information in the government field.

[0035] The detection method comprises the following steps:

[0036] Step S1: standardize the image data in the government affairs information and divide it into Patches of the same size, and map them through a linear projection layer to obtain visual modality features;

[0037] Step S2, using the KeyBert pre-trained model to extract keywords from the long text of government affairs information, and then using the MacBert model to obtain the feature representation corresponding to the keywords, and then fusing the modal type feature and the position feature to obtain the text modal feature;

[0038] Step S3: Input the text modal features and visual modal features into the trained BiLSTM-Attention network for feature fusion encoding, and output multimodal feature representation.

[0039] In step S1, the visual modality features in the government affairs information are obtained, which specifically includes the following steps:

[0040] Step S11: Define an image set of government affairs data , the image collection The image in is segmented to obtain a 224×224×3 image. The formula is as follows:

[0041]

[0042] in, Represents the original image data P i To perform conversion segmentation, P i Represents the first i samples, and represents the standardized sample;

[0043] Step S12: traverse the image set , divided into 16×16×3 blocks, and the image block set is obtained Patches , the formula is as follows:

[0044]

[0045] in, Represents the standardized image data Split into 16×16×3 sizes. represents the image data after being standardized in step S11, and Patches i Representative i A subset of segmented image patches.

[0046] Step S13: Use a linear projection function to project each segmented image block subset to obtain its projection features, then traverse the projection features and corresponding position vector features of each segmented image block subset, and fuse the projection features and corresponding position vector features of each segmented image block subset. The projection and fusion functions are:

[0047]

[0048] in, represents the linear projection function, Patches i Representative i A subset of the image blocks, and Represents the projection result, that is Patches i Image features obtained by linear projection function; represents the feature fusion function, represent Patches i The original position information in the image before segmentation; Representative i The feature representation of the final visual modality is obtained by fusing the position features and projection features of a subset of image patches.

[0049] In step S2, text modal features are extracted from the information text in the government field, which specifically includes the following steps:

[0050] Step S21, using the basic word segmentation capability of the jieba library and the secondary word segmentation capability of the KeyBert model based on N-gram, calculate and select keywords by comparing the cosine similarity between the original sentence encoding and the word segmentation result encoding; the calculation formula is as follows:

[0051]

[0052] in, t i Represents the original text dataset T The i data samples, Indicates that the jieba library is used t i Perform basic word segmentation. Indicates that after the basic participle Model to calculate and select keywords; Indicates the pairing between the jieba library and the KeyBert model t i The extracted keyword result set, for The position vector of the corresponding keyword in the original text;

[0053] Step S22: Use the MacBert model to encode the text data sample, use the first layer of the MacBert hidden layer as the word-level feature, and the last layer as the text semantic information, and weight the two to obtain the semantic information of the text data sample, and use the topic word position information to map it to the embedded representation of the topic word. The calculation formula is as follows:

[0054]

[0055] in, V t refers to the text modality feature vector, V t and They are text data samples t i After inputting the MacBert pre-trained model sequence ti Output at the first and last hidden layer; Representative text data samples t i The weighted result of the first and last hidden layers after MacBert encoding, i.e. the text data sample t i Embedding features of Indicates i Keywords Length, Indicates i Keywords In the text sequence t i The starting position in Represents a sample of text data t i After encoding, it is located at After k The feature vector at the position, Represents the semantic feature results of the subject word;

[0056] Step S23: Add the text keyword vector to the modality type feature information , get the final text modality feature vector V t .

[0057] In step S3, the obtained text modality feature vector V t , visual modality features V p Input the feature fusion module based on the BiLSTM-Attention architecture to obtain the final semantic representation of government data. The calculation formula is as follows:

[0058]

[0059] In the formula, V p , V t Represent visual modality features and text modality features respectively, V It represents the semantic representation of the final government affairs data.

[0060] This embodiment involves a multimodal knowledge representation method for information compliance governance, which aims to propose a design for the multimodal feature knowledge representation and fusion in government affairs information. First, the visual modal features are extracted. Considering the efficiency problems encountered in extracting images by convolution, the image segmentation method without convolution is used to standardize and segment the image, and then the image data features are obtained by projection; then the keywords are extracted from the long text by the KeyBert model, the text is encoded by MacBert, and the original text is semantically replaced by the position information of the keywords contained in the text, and the text modal features are obtained by combining the type features of the original text; finally, the BiLSTM-Attention model is used to perform multimodal fusion of the two modal features to achieve the semantic representation of government affairs information.

[0061] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0062] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0063] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0065] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A multimodal knowledge representation method for information compliance governance, characterized by: The following steps are involved: Step S1: standardize the image data in the government affairs information and divide it into Patches of the same size, and map them through a linear projection layer to obtain visual modality features; Step S2, using the KeyBert pre-trained model to extract keywords from the long text of government affairs information, and then using the MacBert model to obtain the feature representation corresponding to the keywords, and then fusing the modal type feature and the position feature to obtain the text modal feature; Step S3: input the text modality features and visual modality features into the trained BiLSTM-Attention network for feature fusion encoding, and output multimodal feature representation; In step S1, the visual modality features in the government affairs information are obtained, which specifically includes the following steps: Step S11: define the original image data set of government affairs data , the original image data set The image in is segmented to obtain a 224×224×3 image. The formula is as follows: in, Represents the original image data P i To perform conversion segmentation, P i Represents the original image data set Middle i samples, and Represents standardized graphic data; Step S12: traverse the image data set , divided into 16×16×3 blocks, and the image block set is obtained Patches , the formula is as follows: in, Represents the standardized image data Split into 16×16×3 sizes. represents the image data set after standardization in step S11 Middle i samples, and Patches i Representative i A subset of segmented image blocks; Step S13: Use a linear projection function to project each segmented image block subset to obtain its projection features, then traverse the projection features and corresponding position vector features of each segmented image block subset, and fuse the projection features and corresponding position vector features of each segmented image block subset. The projection and fusion functions are: in, represents the linear projection function, Patches i Representative i A subset of the image blocks, and Represents the projection result, that is Patches i Image features obtained by linear projection function; represents the feature fusion function, represent Patches i The original position information in the image before segmentation; Representative i The feature representation of the final visual modality is obtained by fusing the position features and projection features of a subset of image patches.

2. A multimodal knowledge representation method for information compliance governance according to claim 1, characterized in that: In step S2, text modal features are extracted from the information text in the government field, which specifically includes the following steps: Step S21, using the basic word segmentation capability of the jieba library and the secondary word segmentation capability of the KeyBert model based on N-gram, calculate and select keywords by comparing the cosine similarity between the original sentence encoding and the word segmentation result encoding; The calculation formula is as follows: in, t i Represents the original text dataset T The i data samples, Indicates that the jieba library is used t i Perform basic word segmentation. Indicates that after the basic participle Model to calculate and select keywords; Indicates the pairing between the jieba library and the KeyBert model t i The extracted keyword result set, for The position vector of the corresponding keyword in the original text; Step S22: Use the MacBert model to encode the text data sample, use the first layer of the MacBert hidden layer as the word-level feature, and the last layer as the text semantic information, and weight the two to obtain the semantic information of the text data sample, and use the topic word position information to map it to the embedded representation of the topic word. The calculation formula is as follows: in, V t and They are text data samples t i After inputting the MacBert pre-trained model sequence t i Output at the first and last hidden layer; Representative text data samples t i The weighted result of the first and last hidden layers after MacBert encoding, i.e. the text data sample t i Embedding features of Indicates i Keywords Length, Indicates i Keywords In the text sequence t i The starting position in Represents a sample of text data t i After encoding, it is located at After k The feature vector at the position, Represents the semantic feature results of the subject word; Step S23: Add the text keyword vector to the modality type feature information , and obtain the final text modality feature vector V t .

3. A multimodal knowledge representation method for information compliance governance according to claim 1, characterized in that: In step S3, the obtained text modality feature vector V t , visual modality features V p Input the feature fusion module based on the BiLSTM-Attention architecture to obtain the final semantic representation of government data. The calculation formula is as follows: In the formula, V p , V t Represent visual modality features and text modality features respectively, V It represents the semantic representation of the final government affairs data.

Images