Multimodal information comparison method based on semantic guidance

Abstract

The invention relates to the technical field of AI information, in particular to a multi-modal information comparison method based on semantic guidance, which comprises the following steps: S1, data acquisition and preprocessing; s2, constructing a model; s3, model training; s4, evaluating and optimizing the model; and S5, carrying out online prediction and feedback. Through the designed and constructed algorithm architecture, heterogeneous image information comparison of specific information is realized through image processing by means of the text semantic guide model. Specific information searching and comparison of the heterogeneous images are realized, and the complexity of searching and comparison of the specific information in the heterogeneous images is greatly reduced. According to the method, the same semantic information of different images is searched and compared by analyzing the semantic information of the characters, so that the usability is improved. The layout distribution, the information format and the like of the to-be-compared image are allowed to be inconsistent, and the application scene of the model is widened. The influence of human factors is reduced, the production efficiency is improved, and the probability of information error comparison is reduced.

Description

Technical Field

[0001] This invention relates to the field of AI information technology, specifically to a semantically guided multimodal information comparison method. It utilizes algorithms such as multilayer perceptron, word embedding, and transformer networks to achieve the function of analyzing and comparing specified content of two images with similar information through semantic analysis of text. Background Technology

[0002] Currently, there is a significant need to manually search and compare specific information across different documents. However, manual comparison is not only time-consuming and labor-intensive but also prone to errors. With the continuous development of deep learning technology, it has become possible to use deep learning algorithms to replace manual methods for extracting and analyzing visual information. Therefore, there is an urgent need for an intelligent method that can leverage the powerful computing and analytical capabilities of computers to replace the current reliance on manual methods for finding and comparing specific information.

[0003] Although there is already a great deal of research and engineering practice in the field of machine vision, there are still many shortcomings. For example, common methods require a combination of many technologies such as optical character recognition (OCR), layout analysis, and natural language processing (NLP), as well as a great deal of engineering expertise, to meet the needs of some specific scenarios.

[0004] This patent combines the most advanced deep learning algorithms to design and develop a method for searching and comparing specific visual information that is not limited by the specific structure of visual information, which greatly improves information processing efficiency. Summary of the Invention

[0005] The purpose of this invention is to solve the problems of time-consuming and error-prone manual searching and comparison of specific information between different materials. It proposes a multimodal information comparison method based on a semantically guided image attention module, a visual information relationship extraction module, and an image information cross-comparison module. This invention enables the search and comparison of specific semantic objects within different visual information by analyzing textual semantic information.

[0006] The technical solution adopted to achieve the purpose of this invention is as follows: A semantically guided multimodal information comparison method includes the following steps: S1 Data Acquisition and Preprocessing: Collect image samples to be compared from different sources, clean, label and format the image samples to obtain standardized data; S2 Model Construction: Construct a semantically guided multimodal information comparison model, which includes a semantically guided image attention module, a visual information relationship extraction module, and an image information cross-comparison module; S3 Model Training: The constructed model is trained using a preset training method, which includes optimizer selection, loss function adjustment, and learning rate descent setting. S4 Model Evaluation and Optimization: Evaluate the performance of the trained model using specific metrics, and adjust the model parameters or structure based on the evaluation results; S5 Online Prediction and Feedback: Deploy the optimized model to the production environment, receive the text description of the semantics to be compared and the image pairs to be compared, output the comparison results through the model, and establish a feedback system to continuously optimize the model based on actual business performance. The semantically guided image attention module performs semantic analysis on the input text information, matches regions in the image information that conform to the same semantics and can explain the semantics, and performs feature enhancement; the visual information relationship extraction module performs feature matching and recognition on the feature-enhanced image in different sequence directions, and completes key-value pair matching of semantic information and semantic explanation information; the image information cross-comparison module performs semantic information comparison on the feature map after preprocessing, and outputs the final comparison result.

[0007] Furthermore, in step S2, the image attention module consists of a semantic encoder, a feature extraction layer, and a feature decoding layer. The semantic encoder, feature extraction layer, and feature decoding layer transmit signals through feature superposition or feature concatenation, where a plus sign represents feature superposition and a cross sign represents feature concatenation. The semantic encoder includes a word embedding unit, a position encoding unit, an encoder group, and a multi-scale feature encoder. The encoder group consists of a multi-head attention unit and a multi-layer perceptron. The encoder group is a serial structure and has N (1≤N≤10) units. The input of the multi-scale feature encoder is equal to the output size of the encoder group, and the number of output features is n (1≤n≤10). The size of each feature is consistent with the size of the output of the corresponding feature extraction layer. The internal structure of the feature decoding layer consists of an input feature interface, a convolutional layer, a convolutional layer, a deconvolutional layer, and an output feature interface. The input features are processed by two convolutional layers and then transformed by the deconvolutional layer to obtain the output features.

[0008] Furthermore, the image information cross-comparison module in step S2 mainly includes: a block location embedding unit, an information cross-fusion module, and an information analysis module; The block-based location embedding unit performs block-based and location embedding on the input feature map to achieve location labeling of different blocks in the feature map; The information cross-fusion module consists of an image encoder, an image decoder, and a cross-connection structure. It achieves information fusion of different feature maps through the cross-connection structure and the multi-head attention mechanism in the image decoder. The information analysis module consists of three identical feature analyzers and a feature connection structure. Two feature analyzers independently analyze the semantic information of the cross-fused feature map and output the comparison features. After being connected by the feature connection structure, the third separate feature analyzer performs information comparison analysis and outputs the final analysis result.

[0009] Furthermore, the information cross-fusion module includes an image encoder and an image decoder; The image encoder is internally configured with a normalization layer, a multi-head attention layer, a normalization layer, and a multilayer perceptron in sequence. The image decoder is internally configured with a masking multi-head attention layer, a fusion layer, another multi-head attention layer, a fusion layer, a multi-layer perceptron, and a fusion layer in sequence.

[0010] Furthermore, the internal structure of the feature analyzer within the information analysis module includes an image encoder, an image decoder, a super buffer, and an output layer; the output layer utilizes the Softmax function, and the above functional units achieve feature transfer and integration through a fusion layer, while the super buffer is used for temporary feature storage and retrieval.

[0011] Furthermore, the internal processing flow of the visual information relationship extraction module in step S2 is as follows: After inputting features, the processing is divided into three paths. The first path of input features is processed sequentially through feature transposition. One path involves the features entering the transformer network model, undergoing feature transposition again, and then outputting to the segmentation pooling. The other path involves the features being reversed, entering the transformer network model, undergoing reversed processing again, undergoing feature transposition again, and then outputting to the segmentation pooling. The second path of input features enters the transformer network model and outputs to the segmentation pooling. The third path of input features is reversed, enters the transformer network model, undergoes reversed processing again, and then outputs to the segmentation pooling. Finally, after the segmentation pooling operation, the target features are output.

[0012] Furthermore, the multi-scale encoder is internally configured with an input layer, a multi-scale feature extraction layer, and an output layer, each layer consisting of artificial neurons. The neurons in the multi-scale feature extraction layer are divided into n groups, and each group of neurons is fully connected to the neurons in the input layer. The neurons in the output layer are also divided into n groups, and each group of neurons is fully connected to the neurons in the corresponding group of the multi-scale feature extraction layer.

[0013] Furthermore, when the data to be processed is a multi-page document, the multi-page document is sequentially concatenated and stored through a transcendental buffer. The transcendental buffer is initially empty, and then it will continuously increment to store the output of each page of the document output layer until the last page. The multi-page document is then merged into a single image through the transcendental buffer, and the data format is unified for output.

[0014] Furthermore, the specific steps of the segmentation pooling operation are as follows: S1 input has four features A, B, C, and D, with pooling kernel sizes n respectively. Characteristics of n; S2 segments features A, B, C, and D element by element and then recombines them; S3 designs pooling methods (including max pooling and mean pooling) as well as pooling kernel size and step size.

[0015] For example, the first layer, A, B, C, and D, has a size of 2. Features 2; Features A, B, C, and D are segmented and recombined to obtain a size of 4. New features of 4; design step size of 2, dimension of 2 2. The pooling kernel is obtained using the max pooling method.

[0016] Word embedding is a natural language processing technique that transforms words into dense vector representations. This technique is based on neural network models such as Word2Vec, GloVe, or FastText, and aims to capture the semantic and syntactic relationships between words. Each word is mapped to a fixed-dimensional space, and similar words are closer together in the vector space; for example, the vectors of "king" and "queen" will be closer than those of "king" and "apple".

[0017] Multilayer Perceptron (MPP): A feedforward neural network model containing input, output, and at least one hidden layer. It learns complex function mappings through weight adjustment and is widely used in classification and regression tasks.

[0018] Multi-HeaAttention is a variant of self-attention, where the attention mechanism is divided into multiple independent "heads," each responsible for focusing on a different part or feature of the input sequence. Each head computes its own weight distribution, and then the results of all heads are concatenated together. This design allows the model to focus on the input from multiple perspectives simultaneously, improving its ability to understand complex patterns.

[0019] Transformer Networks (TNNs) are neural network architectures based on self-attention mechanisms, primarily used for processing sequential data, and particularly adept at capturing long-range dependencies. TNNs overcome the limitations of traditional sequence models (such as RNNs and LSTMs) in parallel computation and long-range dependency modeling, demonstrating excellent performance in various natural language processing tasks. The core idea of ​​Transformers is to introduce multi-head self-attention and positional encoding. Self-attention allows the model to directly compute the association between any two positions in a sequence, regardless of distance; positional encoding provides the model with positional information of elements in the sequence. Transformers also include two main components: an encoder and a decoder, which process input and generate output by stacking multiple layers of attention and feedforward neural networks. This structure enables Transformer Networks to perform efficient parallel computation while maintaining sensitivity to long-range context.

[0020] The CBA module consists of convolutional layers, batch normalization layers, and activation function layers, so the module name is derived from the first letter of each layer's English name.

[0021] 1. This invention utilizes a designed and constructed algorithm architecture to achieve heterogeneous image information comparison for specific information through image processing, guided by a textual semantic model. Compared to existing technologies, this integrated approach to finding and comparing specific information in heterogeneous images significantly reduces the complexity of such searches.

[0022] 2. This invention, based on a semantically guided image attention module, achieves semantic detection of targets by fusing text semantic encoders at different scales within a traditional image encoder-decoder architecture, and by designing and constructing a multi-scale feature encoder within the semantic encoder. This is accomplished through the fusion analysis of text semantic features and image features.

[0023] 3. The visual information relationship extraction module of this invention introduces a Transformer Network with sequential feature processing and analysis capabilities in parallel, and performs transpose and reverse order processing on the input features in four channels respectively. Finally, it selectively filters the feature data by performing segmentation and pooling operations on the output features of the four channels. Through multi-directional analysis and fusion processing of input features, the module's robustness to erroneous features and adaptability to complex spatial distributions of target features are improved, and visual information relationship matching is completed.

[0024] 4. The image information cross-comparison module of this invention introduces image features from different channels at cross-channels in the image processing module, and introduces an independent-joint dual analysis architecture in the information analysis module. By cross-introducing and fusing image features from opposing channels, information comparison can be pre-compared separately in different feature analyzers, and then a secondary comparison analysis can be performed in the final feature analyzer, greatly improving the reliability of the analysis.

[0025] 5. This invention improves usability by analyzing textual semantic information to find and compare identical semantic information between different images. It allows for inconsistencies in the layout and information format of the images to be compared, broadening the application scenarios of the model. It reduces the impact of human factors, improves production efficiency, and reduces the probability of erroneous information comparison. Attached Figure Description

[0026] Figure 1 This is a system block diagram of the semantically guided multimodal information comparison method of the present invention; Figure 2 This is a diagram showing the internal structure of the semantically guided image attention module of the present invention. Figure 3 This is a structural diagram of the semantic encoder of the present invention; Figure 4 This is a detailed internal structure diagram of the multi-scale encoder of the present invention; Figure 5 This is a diagram showing the internal structure of the feature extraction layer of the present invention; Figure 6 This is a diagram of the internal structure of the feature coding layer of the present invention; Figure 7 This is a diagram showing the internal structure of the visual information relationship extraction module of the present invention. Figure 8 This is a diagram showing the internal structure of the image information cross-comparison module of the present invention; Figure 9 This is a diagram showing the internal structure of the image encoder and decoder within the information cross-fusion module of the present invention. Figure 10 This is a diagram showing the internal structure of the feature analyzer within the information analysis module of the present invention. Figure 11 This is a diagram of the segmentation pooling process of the present invention. Detailed Implementation

[0027] The following is in conjunction with the instruction manual appendix. Figure 1-11 The present invention will be described in further detail below.

[0028] A semantically guided multimodal information comparison method includes the following steps: S1 Data Acquisition and Preprocessing: Collect image samples to be compared from different sources, clean, label and format the image samples to obtain standardized data; S2 Model Construction: Construct a semantically guided multimodal information comparison model, which includes a semantically guided image attention module, a visual information relationship extraction module, and an image information cross-comparison module; S3 Model Training: The constructed model is trained using a preset training method, which includes optimizer selection, loss function adjustment, and learning rate descent setting. S4 Model Evaluation and Optimization: Evaluate the performance of the trained model using specific metrics, and adjust the model parameters or structure based on the evaluation results; S5 Online Prediction and Feedback: Deploy the optimized model to the production environment, receive the text description of the semantics to be compared and the image pairs to be compared, output the comparison results through the model, and establish a feedback system to continuously optimize the model based on actual business performance. The semantically guided image attention module performs semantic analysis on the input text information, matches regions in the image information that conform to the same semantics and can explain the semantics, and performs feature enhancement; the visual information relationship extraction module performs feature matching and recognition on the feature-enhanced image in different sequence directions, and completes key-value pair matching of semantic information and semantic explanation information; the image information cross-comparison module performs semantic information comparison on the feature map after preprocessing, and outputs the final comparison result.

[0029] Based on the semantically guided multimodal information comparison model, the model requires input textual descriptions of the semantics to be compared and image pairs to be compared. The model consists of a semantically guided image attention module, a visual information relationship extraction module, and an image information cross-comparison module. The specific structure is shown in the attached figure.

[0030] The semantically guided image attention module performs semantic analysis on the input text information. Based on the semantic encoding, the module matches regions in the input image information that have the same semantic information and regions that can interpret the semantic information, and then performs feature enhancement on the image.

[0031] The visual information relationship extraction module performs feature matching and recognition on the input feature-enhanced image in different sequence directions, thereby completing the key-value pair matching of semantic information and semantic interpretation information in the feature image.

[0032] The image information cross-comparison module compares the semantic information of feature regions in the input feature map after preprocessing and outputs the comparison results. This module consists of three main parts: block location embedding, information cross-fusion, and information analysis. The block location embedding module divides the input feature map into blocks and embeds their locations, enabling the labeling of different blocks and enhancing the spatial relationship representation of the feature map. The information cross-fusion module comprises an image encoder, an image decoder, and a cross-connection structure. Due to the cross-connection structure and the multi-head attention mechanism within the image decoder, information fusion between different feature maps is achieved. The information analysis module consists of three identical feature analyzers and a feature connection layer. Its main function is that before feature connection, each feature analyzer independently analyzes the semantic information contained in the cross-fused feature map and outputs information comparison features that conform to the feature semantics. After feature connection, the connected features are analyzed by a separate feature analyzer, outputting the final information comparison result. Example

[0033] Scenario Description: A company uses this invention to build an intelligent contract and invoice information matching audit system to improve the efficiency of its audit work and management effectiveness. The implementation steps are as follows: Step 1: Data Acquisition and Preprocessing: Collect and prepare image pairs of contract and invoice information to be processed in the intelligent auditing system. During the data preprocessing stage, multi-page documents are merged into a single image through image processing, data formats are standardized, and data annotations are performed on the information to be compared within the image pairs.

[0034] Step 2: Model Building and Training. Build the model according to the illustrated model architecture and set appropriate model hyperparameters and training parameters. For example, use ReLU as the activation function, use the Adam optimizer and mean squared error loss function for model training, and set the training set to test set ratio to 8:2 for model training and performance optimization.

[0035] Step 3: Online Prediction and Feedback. After the system goes live, it automatically processes newly collected data monthly. Users confirm and annotate the system's comparison results. Every six months, the system fine-tunes the model parameters based on the latest annotated data to ensure the model's accuracy and adaptability.

[0036] Building upon the existing implementation, to further enhance the model's adaptability to various scenarios, this can be achieved by modifying the model's internal structure and processing its input information. For example, changing the standard convolution method in the model's convolutional layers to deformable convolution, dilated convolution, or similar methods can compress model parameters and improve performance. Furthermore, the model's video information comparison function can be implemented by performing image frame segmentation on the video.

Claims

1. A semantically guided multimodal information comparison method, characterized in that, Includes the following steps: S1 Data Acquisition and Preprocessing: Collect image samples to be compared from different sources, clean, label and format the image samples to obtain standardized data; S2 Model Construction: Construct a semantically guided multimodal information comparison model, which includes a semantically guided image attention module, a visual information relationship extraction module, and an image information cross-comparison module; S3 Model Training: The constructed model is trained using a preset training method, which includes optimizer selection, loss function adjustment, and learning rate descent setting. S4 Model Evaluation and Optimization: Evaluate the performance of the trained model using specific metrics, and adjust the model parameters or structure based on the evaluation results; S5 Online Prediction and Feedback: Deploy the optimized model to the production environment, receive the text description of the semantics to be compared and the image pairs to be compared, output the comparison results through the model, and establish a feedback system to continuously optimize the model based on actual business performance. The semantically guided image attention module performs semantic analysis on the input text information, matches regions in the image information that conform to the same semantics and can explain the semantics, and performs feature enhancement; the visual information relationship extraction module performs feature matching and recognition on the feature-enhanced image in different sequence directions, and completes key-value pair matching of semantic information and semantic explanation information; the image information cross-comparison module performs semantic information comparison on the feature map after preprocessing, and outputs the final comparison result.

2. The semantically guided multimodal information comparison method according to claim 1, characterized in that, In step S2, the image attention module consists of a semantic encoder, a feature extraction layer, and a feature decoding layer. The semantic encoder, feature extraction layer, and feature decoding layer transmit signals to each other through feature superposition or feature concatenation. The plus sign represents feature superposition, and the cross sign represents feature concatenation. The semantic encoder includes a word embedding unit, a position encoding unit, an encoder group, and a multi-scale feature encoder; The encoder group consists of a multi-head attention unit and a multi-layer sensing mechanism. The encoder group is a series structure and the number of units is N (1≤N≤10). The input of the multi-scale feature encoder is equal to the output size of the encoder group, and the number of output features is n (1≤n≤10). The size of each feature is consistent with the size of the output of the corresponding feature extraction layer. The internal structure of the feature decoding layer consists of an input feature interface, a convolutional layer, a convolutional layer, a deconvolutional layer, and an output feature interface. The input features are processed by two convolutional layers and then transformed by the deconvolutional layer to obtain the output features.

3. The semantically guided multimodal information comparison method according to claim 1, characterized in that, The image information cross-comparison module in step S2 mainly includes: a block location embedding unit, an information cross-fusion module, and an information analysis module; The block-based location embedding unit performs block-based and location embedding on the input feature map to achieve location labeling of different blocks in the feature map; The information cross-fusion module consists of an image encoder, an image decoder, and a cross-connection structure. It achieves information fusion of different feature maps through the cross-connection structure and the multi-head attention mechanism in the image decoder. The information analysis module consists of three identical feature analyzers and a feature connection structure. Two feature analyzers independently analyze the semantic information of the cross-fused feature map and output the comparison features. After being connected by the feature connection structure, the third separate feature analyzer performs information comparison analysis and outputs the final analysis result.

4. The semantically guided multimodal information comparison method according to claim 3, characterized in that, The information cross-fusion module includes an image encoder and an image decoder; The image encoder is internally configured with a normalization layer, a multi-head attention layer, a normalization layer, and a multilayer perceptron in sequence. The image decoder is internally configured with a masking multi-head attention layer, a fusion layer, another multi-head attention layer, a fusion layer, a multi-layer perceptron, and a fusion layer in sequence.

5. The semantically guided multimodal information comparison method according to claim 3, characterized in that, The internal structure of the feature analyzer in the information analysis module includes an image encoder, an image decoder, a super buffer, and an output layer. The output layer utilizes the Softmax function. The above functional units achieve feature transfer and integration through a fusion layer. The super buffer is used for temporary feature storage and retrieval.

6. The semantically guided multimodal information comparison method according to claim 1, characterized in that, The internal processing flow of the visual information relationship extraction module in step S2 is as follows: After inputting features, the processing is divided into three paths. The first path of input features is processed sequentially through feature transposition. One path involves the features entering the transformer network model, undergoing feature transposition again, and then being output to the segmentation pooling. The other path involves the features being reversed, entering the transformer network model, undergoing reversed processing again, undergoing feature transposition again, and then being output to the segmentation pooling. The second path of input features enters the transformer network model and is output to the segmentation pooling. The third path of input features is reversed, enters the transformer network model, undergoes reversed processing again, and then is output to the segmentation pooling. Finally, after the segmentation pooling operation, the target features are output.

7. The semantically guided multimodal information comparison method according to claim 2, characterized in that, The multi-scale encoder is internally configured with an input layer, a multi-scale feature extraction layer, and an output layer, each composed of artificial neurons. The neurons in the multi-scale feature extraction layer are divided into n groups, and each group of neurons is fully connected to the neurons in the input layer. The neurons in the output layer are also divided into n groups, and each group of neurons is fully connected to the neurons in the corresponding group of the multi-scale feature extraction layer.

8. The semantically guided multimodal information comparison method according to claim 5, characterized in that, When the data to be processed is a multi-page document, the multi-page document is sequentially concatenated and stored by using a transcendental cache. The transcendental cache is initially empty, and then the features output by the output layer of each page of the document are stored incrementally. By bypassing the cache, features from multiple pages of documents are concatenated into continuous features, and the data format is output in a unified manner.

9. The semantically guided multimodal information comparison method according to claim 6, characterized in that, The specific steps of the segmented pooling operation are as follows: S1 input has four features A, B, C, and D, with pooling kernel sizes n respectively. Characteristics of n; S2 segments features A, B, C, and D element by element and then recombines them; S3 designs pooling methods (including max pooling and mean pooling) as well as pooling kernel size and step size.

Images