A Smart Contract Analysis Method and System Based on Multimodal Feature Fusion Algorithm

Abstract

This application provides a smart contract analysis method and system based on a multimodal feature fusion algorithm, relating to the technical field of smart contract analysis. The method includes: acquiring a PDF contract document and segmenting it into multiple image patches; constructing a multimodal collaborative feature extraction model to obtain visual features, topological features, and textual features of each image patch; using an attention mechanism to weightedly fuse the three features of the same image patch; inputting the fused features into a pre-trained classification model and outputting the classification result. This application can improve the accuracy of contract analysis results.

Description

Technical Field

[0001] This application relates to the technical field of smart contract analysis, and in particular to a smart contract analysis method and system based on a multimodal feature fusion algorithm. Background Technology

[0002] In today's society, where business activities and legal affairs are highly intertwined, contracts, as crucial legal documents that clarify the rights and obligations of all parties and ensure transaction security and standardization, are experiencing explosive growth in number. Whether in business cooperation between enterprises, credit operations of financial institutions, or administrative approvals by government departments, contracts play an indispensable role. With the advancement of economic globalization and the widespread application of digital technologies, the complexity and diversity of contracts are constantly increasing, encompassing various information carriers such as text, images, and tables. Therefore, how to efficiently and accurately analyze contracts, extract key information, identify potential risks, and determine contract types has become an urgent need for various industries. Intelligent contract analysis technology has emerged to address this need, aiming to automate and intelligently process contracts using advanced artificial intelligence algorithms and computer technology, improving the efficiency and quality of contract management while reducing labor costs and error rates.

[0003] The current field of intelligent contract document analysis has formed a technical system dominated by rule engines, single-modal learning, and general-purpose large models. However, the performance of different solutions varies significantly in practical applications. OCR-based rule matching methods have advantages in standardized form parsing scenarios, with their preset template mechanism ensuring a stable accuracy rate of 82%-87% for key field extraction. However, when faced with unstructured legal clauses, the lack of semantic understanding results in an accuracy rate of less than 41% for complex rights and obligations clauses. While single-modal deep learning models have achieved breakthroughs in specific dimensions, text models suffer from a 29% deviation rate in clause location due to neglecting layout features. Visual models, on the other hand, exhibit a particularly prominent problem in their ability to resolve ambiguities in legal terminology.

[0004] While multimodal large-scale modeling technology, which has emerged in recent years, theoretically possesses the potential for cross-modal fusion, for example, Chinese invention patent application CN118734032A provides a method and device for extracting contract information based on a multimodal large-scale language model. This patent first acquires contract data in different formats to construct a model training dataset; then, it constructs a latent space mapping model between contract image and text data; it trains this mapping model using the training dataset, and obtains a general text and image data encoder based on the verification results; then, it constructs a large-scale language model for contract extraction based on these two encoders; finally, it trains the large-scale model using the training dataset and verifies its information extraction accuracy.

[0005] However, this patent processes contract image data through patching, encoding, and location encoding. Although it can capture local semantics and basic spatial relationships, it still has significant limitations in extracting topological features (such as complex layouts, structured information, and nonlinear spatial dependencies). If the contract contains cross-page tables, patching may cause the rows and columns of the same table to be encoded independently on different pages, thus destroying the topological consistency of cross-page tables. Summary of the Invention

[0006] To improve the accuracy of contract analysis results, this application provides an intelligent contract analysis method and system based on a multimodal feature fusion algorithm.

[0007] Firstly, this application provides a smart contract analysis method based on a multimodal feature fusion algorithm, employing the following technical solution:

[0008] The smart contract analysis method based on multimodal feature fusion algorithm includes the following steps:

[0009] Data acquisition and processing: Obtain PDF contract documents, segment the PDF contract documents, and obtain multiple image blocks;

[0010] Feature extraction: Construct a multimodal collaborative feature extraction model, input each image patch into the multimodal collaborative feature extraction model, and obtain the visual features, topological features and text features of each image patch respectively;

[0011] Classification: For the same image patch, visual features, topological features, and text features are weighted and fused using an attention mechanism to obtain fused features. The fused features are then input into a pre-trained classification model, which outputs the classification result of the fused features.

[0012] This application constructs a multimodal collaborative feature extraction model to obtain visual, topological, and textual features of contract image blocks. Then, different weights are assigned to each feature based on its importance, highlighting the role of key features. The fused features are then input into a pre-trained classification model. These fused features contain more comprehensive and effective information, helping the classification model to more accurately classify contract content. PDF contract documents often contain complex hierarchical relationships, and traditional visual features are easily affected by layout formatting. By analyzing the topological relationships of PDF documents to restore the logical hierarchy, layout biases are reduced, thereby improving the accuracy of contract analysis results.

[0013] Optionally, the multimodal collaborative feature extraction model includes a visual feature extraction sub-model for extracting visual features of image patches, a topological structure feature extraction sub-model for extracting topological features between image patches, and a text semantic understanding sub-model for extracting text features of image patches.

[0014] The multimodal collaborative feature extraction model employs three sub-models—visual feature extraction, topological structure feature extraction, and text semantic understanding—to work together to deconstruct image patch information from different dimensions. This not only solves the problem of the one-sidedness of single-modal feature information but also provides input data for the attention mechanism to perform weighted fusion processing of features from different modalities.

[0015] Optionally, the visual feature extraction sub-model includes a CSPDarknet53 network with CBAM layers and a bidirectional feature pyramid network. The CSPDarknet53 network is used to perform five-level downsampling on image patches to obtain multi-scale feature maps. The bidirectional feature pyramid network is used to enhance the target region in the multi-scale feature maps and output the enhanced multi-scale feature maps. The enhanced multi-scale feature maps are then weighted by CBAM layers to output visual features.

[0016] This application employs the CSPDarknet53 network to perform five-level downsampling processing on image patches, gradually expanding the receptive field and generating multi-scale feature maps ranging from small-scale high resolution to large-scale low resolution. This approach can capture microscopic details while preserving macroscopic structural information, which helps to comprehensively understand image content and improve the detection and recognition capabilities of targets of different sizes.

[0017] Because the multi-scale feature maps generated by the CSPDarknet53 network suffer from hierarchical fragmentation—low-level feature maps are rich in detail but lack global semantics, while high-level feature maps have clear structures but lose details—this application employs a bidirectional feature pyramid network. By leveraging a bidirectional information flow mechanism, features at different scales are fused, allowing information about the target region to be more fully transmitted and enhanced across features at each scale. This further highlights the target region and improves its localization and classification accuracy. The CBAM layer automatically identifies and enhances key visual features through dual weighting of channel attention and spatial attention, reducing the risk of useful information being obscured by redundant information during feature fusion.

[0018] By adopting the above scheme, this application transforms the unstructured visual information of contract image blocks into structured visual vectors that meet the business needs of contracts, highlight key signals, and are multi-dimensionally complementary. This not only solves the problems of traditional visual extraction where details and structure cannot be obtained simultaneously and redundant information interferes, but also allows visual features to form a precise match with topological features and text features, further improving the accuracy of contract analysis results.

[0019] Optionally, the data acquisition and processing steps include:

[0020] The PDF contract document is parsed to obtain the page pixel map. The image blocks in the page pixel map are separated by the OpenCV contour detection algorithm. The bounding boxes of all image blocks are projected onto the same coordinate system to obtain the coordinate values ​​of each vertex of each image block.

[0021] Based on the coordinate values ​​of each vertex of each image patch, a coordinate index for each image patch is established.

[0022] This application parses PDF contract documents to generate page pixel images, which not only transforms the logical content of the PDF into image data with a unified visual form, but also completely preserves the visual details in the contract. Directly extracting text and performing structural analysis on PDF contract documents would result in the loss of non-textual visual information such as color, annotations, and signatures, leading to a lack of key dimensions in multimodal feature extraction. Subsequently, this application uses the OpenCV contour detection algorithm to separate regions with continuous edges in the PDF contract document, thereby splitting each page into multiple independent image blocks. The bounding boxes of all image blocks are projected onto the same coordinate system to obtain the coordinate values ​​of each vertex of each image block. A coordinate index is established based on the coordinate values ​​of each vertex of each image block, providing a mechanism for rapid lookup and location of image blocks. By establishing a coordinate index, this application can quickly find the corresponding image block based on its coordinate information, improving the management efficiency and ease of operation of image blocks in the document.

[0023] Optionally, the text semantic understanding sub-model includes a CRNN sub-model and an MLP sub-model. The process of extracting text features by the text semantic understanding sub-model is as follows:

[0024] Image blocks of text type are located and identified using a CRNN sub-model, denoted as text blocks. Geometric parameters of the text blocks are obtained, including center coordinates and width and height. The tilt angle of the text blocks is calculated based on the geometric parameters.

[0025] Determine if the tilt angle is zero. If it is, do nothing. If not, perform an affine transformation on the text block based on the tilt angle to obtain a corrected text block, and update the text block with the corrected text block.

[0026] The geometric parameters of the text block are normalized, and the MLP sub-model is used to map the normalized geometric parameters into position vectors to obtain the coordinate index of the text block. The coordinate index is encoded to obtain the original position encoding vector. The position vector is added to the original position encoding vector to obtain the text features.

[0027] This application utilizes a CRNN sub-model to locate and identify text-type image blocks (i.e., text blocks), effectively locating text-containing regions within image blocks and accurately identifying their text content. Subsequently, this application calculates the tilt angle based on the center coordinates and dimensions of the text block to quantify its tilt degree within the image. By determining whether the tilt angle is zero, this application performs an affine transformation on text blocks with non-zero tilt angles to obtain corrected text blocks. These corrected text blocks better align with normal reading directions, facilitating more accurate subsequent text feature extraction and reducing recognition errors and feature extraction difficulties caused by text tilt. Next, this application normalizes the geometric parameters (center coordinates and dimensions) of the text blocks, eliminating differences caused by factors such as size and position. Then, an MLP sub-model maps the normalized geometric parameters into a 768-dimensional position vector, transforming the geometric information of the text block into a high-dimensional vector representation, adding position-related semantic information to the text features.

[0028] Subsequently, this application obtains the coordinate index of the text block and performs encoding processing to obtain the original position encoding vector. The 768-dimensional position vector is added to the original position encoding vector to finally obtain the text features. The above scheme can combine positional information and geometric information, so that the text features not only contain the semantic information of the text itself, but also incorporate the position and geometric features of the text in the image, enriching the connotation of the text features and improving the expressive power and semantic understanding ability of the text features.

[0029] Optionally, the classification steps may also include:

[0030] Based on the coordinate index of each image patch, image patches belonging to the same page are integrated into an image patch set. The dimension of the attention matrix in the attention mechanism is obtained, and a binary mask matrix with the same dimension as the attention matrix is ​​constructed. The element in the i-th row and j-th column of the binary mask matrix indicates whether the i-th image patch and the j-th image patch belong to the same page. If yes, the element has a value of 1; if no, the element has a value of 0.

[0031] Multiply the binary mask matrix and the attention matrix element by element to obtain the multiplied matrix, and use the multiplied matrix as the new attention matrix.

[0032] This application first obtains the dimension of the attention matrix in the attention mechanism, and then constructs a binary mask matrix with the same dimension as the attention matrix. By setting the value rule of the element in the i-th row and j-th column of the matrix, this application can clearly mark which image patches are related within the same page. Subsequently, this application multiplies the constructed binary mask matrix element-wise with the attention matrix. After multiplication, only the attention matrix elements corresponding to image patches within the same page retain their original values, while the attention matrix elements corresponding to image patches on different pages become 0. The resulting new attention matrix focuses more on the relationships between image patches within the same page, highlighting the local structure and related information within the page.

[0033] This application constructs a binary mask matrix and adjusts the attention matrix, enabling the multimodal collaborative feature extraction model to focus more on local information within the same page when processing image patches. In tasks such as document classification, elements within the same page usually have closer semantic and structural relationships. The above approach helps the multimodal collaborative feature extraction model to better capture these local features and improve the accuracy of understanding page content.

[0034] In multi-page document processing, image patches across different pages may share some similar features, but their semantic connections are usually weak. The original attention matrix may be affected by these cross-page similarities, leading to biases in the multimodal collaborative feature extraction model's understanding of page content. However, by using a binary mask matrix, the new attention matrix eliminates the correlation between cross-page image patches, reducing this interference and enabling the multimodal collaborative feature extraction model to more accurately extract key information within the same page.

[0035] Optionally, the classification steps may also include:

[0036] Obtain the center coordinates of the page corresponding to the i-th image patch set, calculate the Manhattan distance between the image patch containing the center coordinates of the page corresponding to the i-th image patch set and the remaining image patches in the i-th image patch set, construct an attention bias term based on the Manhattan distance, and use the sum of the attention bias term and the attention score in the attention mechanism as the new attention score.

[0037] Based on the coordinate index of each image block, this application integrates image blocks on the same page into an image block set, thereby grouping image blocks belonging to the same visual level into a group. Then, by obtaining the center coordinates of the i-th image block set and calculating the Manhattan distance between the image block at the center coordinates and the remaining image blocks in the set, this application can measure the relative positional relationship of image blocks on the page and quantify the degree of spatial association between different image blocks within the same page.

[0038] Subsequently, this application constructs an attention bias term based on Manhattan distance. The introduction of this attention bias term allows the attention mechanism to consider the spatial relationships between image patches, rather than just the similarity of the features themselves. This enables the model to pay more attention to image patches spatially close to the central image patch when processing them, which is more in line with human cognitive habits of spatial layout when observing documents. Finally, this application adds the attention bias term to the attention score in the attention mechanism to obtain a new attention score. The new attention score comprehensively considers feature similarity and spatial relationships, allowing the fused features to more comprehensively reflect the information of the image patches, thus helping to improve classification accuracy.

[0039] Optionally, the topology feature extraction sub-model includes an edge enhancement graph convolutional network, and the process of extracting topology features by the topology feature extraction sub-model is as follows:

[0040] Using image patches as nodes, the features of each node include: the center coordinates, width and height of the image patch, and visual features. The Euclidean distance between image patches is calculated based on the center coordinates. Edges are constructed between image patches whose Euclidean distance is less than a preset distance threshold to obtain a dynamic graph structure. An edge-enhanced graph convolutional network is used to capture the topological features in the dynamic graph structure.

[0041] This application treats image patches as nodes, and the node features encompass the center coordinates, width and height, and visual features of the image patch. The center coordinates and width and height provide information about the position and size of the image patch in the image, which helps the multimodal collaborative feature extraction model understand the spatial relationships between image patches; the visual features contain semantic and texture information of the image patch itself. The above approach enables the multimodal collaborative feature extraction model to describe image patches from multiple dimensions.

[0042] Subsequently, this application calculates the Euclidean distance based on the center coordinates of the image patches and constructs edges between image patches whose Euclidean distance is less than a preset distance threshold, thereby forming a dynamic graph structure. In an image, image patches that are closer together are often more likely to belong to the same target or have related semantic information. Constructing edges in this way can connect these related image patches.

[0043] This application employs edge-enhanced graph convolutional networks to capture topological features in dynamic graph structures. Graph convolutional networks can update node features by aggregating information from nodes and their neighbors, thereby learning local and global information within the graph structure. Edge enhancement mechanisms further emphasize the importance of edges in the graph structure, making the multimodal collaborative feature extraction model focus more on the connections between nodes, thus capturing topological features more accurately. In image classification tasks, combining visual and topological features enables the model to more accurately distinguish between different image categories, especially in image pairs with similar visual features but different topological structures. Topological features can complement visual features, jointly enhancing the model's ability to understand images.

[0044] Optionally, the classification steps may also include:

[0045] The visual features, topological features, and text features belonging to the same image patch are aligned and concatenated to obtain a concatenated vector. The concatenated vector is then input into a pre-built MLP model to obtain the gating weights of the visual features, topological features, and text features, respectively.

[0046] The product of visual features and their corresponding gating weights is used as the new visual features; the product of topological features and their corresponding gating weights is used as the new topological features; and the product of text features and their corresponding gating weights is used as the new text features.

[0047] This application first aligns the dimensions of visual features, topological features, and textual features belonging to the same image patch. Then, it concatenates the aligned features end-to-end to obtain a concatenated vector, integrating features from different sources into a unified vector space. This concatenated vector is then input into a pre-constructed MLP model. The MLP model learns the importance of different features in the classification task and outputs gating weights for visual, topological, and textual features respectively. Subsequently, this application multiplies the visual, topological, and textual features by their corresponding gating weights to obtain new visual, topological, and textual features. Through this weighting operation, this application can dynamically adjust the contribution of each feature according to its importance in the current classification task. This application uses a gating mechanism to dynamically adjust the weights of different features, achieving dynamic feature fusion. Compared with traditional fixed-weight fusion methods, the dynamic fusion method adopted in this application can better adapt to the needs of different image patches and different classification tasks. Through weighting operations, this application can highlight features more valuable to the classification task and suppress the influence of irrelevant or redundant features, thereby improving the overall expressive power of the features.

[0048] Secondly, this application provides an intelligent contract analysis system based on a multimodal feature fusion algorithm, which adopts the following technical solution:

[0049] A smart contract analysis system based on a multimodal feature fusion algorithm includes:

[0050] The data acquisition and processing module is used to acquire PDF contract documents, segment the PDF contract documents, and obtain multiple image blocks.

[0051] The feature extraction module is used to build a multimodal collaborative feature extraction model. Each image patch is input into the multimodal collaborative feature extraction model to obtain the visual features, topological features and text features of each image patch.

[0052] The classification module is used to perform weighted fusion of visual features, topological features, and text features of the same image patch using an attention mechanism to obtain fused features. The fused features are then input into the pre-trained classification model, which outputs the classification result of the fused features.

[0053] In summary, this application includes at least one of the following beneficial technical effects:

[0054] 1. This application constructs a multimodal collaborative feature extraction model to obtain visual, topological, and textual features of contract image blocks. Then, different weights are assigned according to the importance of each feature to highlight the role of key features. The fused features are then input into a pre-trained classification model. The fused features contain more comprehensive and effective information, helping the classification model to more accurately classify contract content. PDF contract documents contain complex hierarchical relationships, and traditional visual features are easily affected by layout formatting. By analyzing the topological relationships of PDF documents to restore the logical hierarchy, layout bias is reduced, thereby improving the accuracy of contract analysis results.

[0055] 2. The multimodal collaborative feature extraction model uses three sub-models—visual feature extraction, topological structure feature extraction, and text semantic understanding—to work together to deconstruct the information of contract image patches from different dimensions. This not only solves the problem of the one-sidedness of single-modal feature information, but also provides input data for the attention mechanism to perform weighted fusion processing of features from different modalities.

[0056] 3. This application employs an edge-enhanced graph convolutional network to capture topological features in dynamic graph structures. Graph convolutional networks can update node features by aggregating information from nodes and their neighbors, thereby learning local and global information within the graph structure. The edge enhancement mechanism further emphasizes the importance of edges in the graph structure, making the multimodal collaborative feature extraction model focus more on the connections between nodes, thus capturing topological features more accurately. In image classification tasks, combining visual and topological features enables the model to more accurately distinguish between different image categories, especially in image pairs with similar visual features but different topological structures. Topological features can complement visual features, jointly enhancing the model's ability to understand images. Attached Figure Description

[0057] Figure 1 This is a flowchart of Embodiment 1 of this application;

[0058] Figure 2 This is a flowchart of Embodiment 2 of this application. Detailed Implementation

[0059] The following combination Figure 1 and Figure 2 This application will be described in further detail.

[0060] Example 1: This example discloses a smart contract analysis method based on a multimodal feature fusion algorithm, referring to... Figure 1 The method includes: S11 data acquisition and processing, S12 feature extraction, and S13 classification, acquiring a PDF contract document and segmenting it into multiple image patches; then constructing a multimodal collaborative feature extraction model to obtain the visual features, topological features, and text features of each image patch; then using an attention mechanism to weightedly fuse the three features of the same image patch, inputting the fused features into a pre-trained classification model and outputting the classification result. The execution process of each step in this embodiment is as follows:

[0061] The S11 data acquisition and processing module obtains PDF contract documents. Based on specific elements in the PDF contract documents (such as titles, paragraphs, tables, etc.), it uses non-uniform grid division technology to initially divide the document into 80×80 pixel units. It then further subdivides areas containing high-frequency details (such as dense text or complex charts) to obtain multiple image blocks.

[0062] In other embodiments, the S11 data acquisition and processing further includes:

[0063] Adaptive binarization is applied to the PDF contract document, employing a tilt correction algorithm based on minimum bounding rectangle to eliminate deformation caused by document rotation or wrinkles. Simultaneously, a non-local means denoising algorithm is used to reduce image noise, resulting in a pre-processed PDF contract document.

[0064] Tools such as PyMuPDF are used to parse the pre-processed PDF contract document to obtain page pixel maps. Based on the page pixel maps, the vector graphics regions and bitmap regions, i.e. image blocks, in the pre-processed PDF document are separated using the OpenCV contour detection algorithm.

[0065] In document analysis tasks, the consistency of the coordinate system is fundamental to subsequent modeling. If pixel coordinates are used directly, different resolutions or devices will lead to inconsistent geometric descriptions of the same element. Therefore, this embodiment converts the pixel coordinates of the bounding box of an image patch into physical coordinates by setting a scaling factor. The calculation model is as follows:

[0066] ;

[0067] in, This refers to the scan resolution, i.e., dots per inch.

[0068] The pixel coordinates of the bounding box of the image patch are adjusted using a scaling factor. The conversion to physical coordinates is as follows:

[0069] ;

[0070] These are the physical coordinates of the image patch; is the center coordinate of the image patch; w represents the width of the image patch; h represents the height of the image patch.

[0071] Project the bounding boxes of all image patches onto the same coordinate system to obtain the coordinate values ​​of each vertex of each image patch. Based on the coordinate values ​​of each vertex of each image patch, establish the coordinate index of each image patch, including:

[0072] Create a coordinate index for each image block (including page number, x and y coordinates of the top left corner, and x and y coordinates of the bottom right corner), associate the coordinate index with the original page layout information, and obtain the metadata of each image block.

[0073] Metadata is stored in JSON-LD format, and the type of image patch is marked by semantic tags (such as signature area, clause table), which supports the rapid retrieval and association of subsequent multimodal features.

[0074] S12 Feature extraction: Construct a multimodal collaborative feature extraction model, which includes a visual feature extraction sub-model for extracting visual features of image patches, a topological structure feature extraction sub-model for extracting topological features between image patches, and a text semantic understanding sub-model for extracting text features of image patches.

[0075] First, the visual feature extraction sub-model gradually extracts visual features from low to high levels from image patches. This sub-model includes a CSPDarknet53 network with CBAM layers and a bidirectional feature pyramid network (BiFPN). The process of extracting visual features using this sub-model is as follows:

[0076] The visual feature extraction sub-model uses the CSPDarknet53 network to perform five levels of downsampling on image patches. Each level reduces computational redundancy through cross-stage partial connections (CSP) and outputs processing results at different scales (gradually reducing the resolution of image patches from 640×640 to 20×20).

[0077] The visual feature extraction sub-model uses a bidirectional feature pyramid network (BiFPN) to fuse features at different scales, and enhances the perception of small targets (such as stamps) and large-scale structures (such as tables) through a weighted summation mechanism.

[0078] The visual feature extraction sub-model embeds a spatial-channel dual attention module (CBAM) into the feature map, dynamically focusing on salient regions (such as handwritten annotations or bold text) to output visual features.

[0079] Secondly, a topological structure is constructed based on the spatial relationship of image patches, and the topological structure feature extraction sub-model is used to learn the topological features of the image patches. The topological features can reflect the layout and connection method of the image patches.

[0080] The topology feature extraction sub-model includes an edge enhancement graph convolutional network. The process of extracting topology features by the topology feature extraction sub-model is as follows:

[0081] Using image patches as nodes, the features of each node include: the center coordinates, width, height, and visual features of the image patch. The Euclidean distance between image patches is calculated based on the center coordinates. Edges are constructed between image patches whose Euclidean distance is less than a preset distance threshold to obtain a dynamic graph structure. Edge-enhanced graph convolution (EdgeConv) is used to learn the local geometric relationships (such as horizontal parallelism or vertical hierarchy) between image patches. Edge-enhanced graph convolution introduces edge feature calculation on the basis of traditional GCN. In this embodiment, two layers of graph convolution and global mean pooling in EdgeConv are used to process the image patches and output 128-dimensional topological features.

[0082] In this embodiment, the preset distance threshold is set to 15% of the average value of all Euclidean distances.

[0083] In other embodiments, a preset distance threshold value can be set as needed.

[0084] Finally, as one possible implementation method in this embodiment, if the image block contains text, the text content is first identified using OCR technology, and then the text content is input into the text semantic understanding sub-model to extract text features. The text features can capture the semantic information in the image block, and this semantic information can supplement the visual features and topological features.

[0085] As another possible implementation method in this embodiment, the text semantic understanding sub-model includes a CRNN sub-model and an MLP sub-model. The process of the text semantic understanding sub-model extracting text features is as follows:

[0086] The image blocks of text type are located and identified by the CRNN sub-model and denoted as text blocks. The geometric parameters of the text blocks are obtained according to their coordinate indices. Then, the tilt angle is calculated based on the center coordinates and width and height of the text blocks.

[0087] Determine if the tilt angle is zero. If it is, do nothing. If not, perform an affine transformation on the text block based on the tilt angle to obtain a corrected text block, and update the corrected text block as the text block.

[0088] The geometric parameters of the text block are normalized, including center coordinates, width, and height.

[0089] An MLP sub-model is used to map the normalized geometric parameters into position vectors to obtain the coordinate indices of text blocks. A pre-defined BERT model is then used to encode these coordinate indices, yielding the original position encoded vectors. These original position encoded vectors are then added to the original position encoded vectors to obtain the text features. In addition to global [CLS] features, these text features also retain the hidden state of each token for subsequent fine-grained alignment with visual features.

[0090] After processing by the multimodal collaborative feature extraction model, each image patch will obtain three feature vectors: visual features, topological features, and textual features.

[0091] S13 classification maps visual features (512-dimensional), text features (768-dimensional), and topological features (128-dimensional) to a unified space (512-dimensional) through deformable convolutional layers, obtaining the mapped visual features, text features, and topological features.

[0092] In other embodiments, the deformable convolutional layer introduces a learnable spatial offset to adaptively align the feature distributions of different modalities.

[0093] Visual features, topological features, and textual features belonging to the same image patch after mapping are concatenated to obtain a concatenated vector.

[0094] The concatenated vectors are input into a pre-built MLP model. After nonlinear activation and Softmax normalization, the gating weights for visual features, topological features, and text features are obtained. The calculation model for the gating weights is as follows:

[0095] ;

[0096] in, Let c be the gating weight; and This is the weight matrix; The temperature parameter has a range of values. The closer the temperature parameter value is to 0, the closer the Softmax is to a one-hot distribution; the closer the temperature parameter value is to 10, the closer the Softmax is to a uniform distribution. The function is used to ensure that the sum of the three-dimensional feature weights is 1; These are the visual features after mapping processing; These are the topological features after mapping. The text features are the mapped features, i.e., BERT's global [CLS] features; || is the feature concatenation operator; ReLU is the activation function. The concatenated vector has a dimension that is the sum of the dimensions of the three feature vectors; For the activation function, this embodiment selects the ReLU function.

[0097] The product of visual features and their corresponding gating weights is used as the new visual features; the product of topological features and their corresponding gating weights is used as the new topological features; and the product of text features and their corresponding gating weights is used as the new text features.

[0098] Using new visual features as the query (Q), the new text features and new topological features are concatenated as key-value pairs (K / V). Cross-modal interaction is achieved through an 8-head attention mechanism. Each attention head focuses on a specific semantic relationship (such as "title-body text" or "chart-description text"). Relative position bias is added during calculation to strengthen the association between spatially adjacent elements and output cross-modal attention features.

[0099] The new visual features, new topological features, and new textual features are weighted and fused to obtain a fused feature (512 dimensions). The calculation model of the fused feature is as follows:

[0100] ;

[0101] ;

[0102] ;

[0103] As a feature of fusion, For new visual features, For cross-modal attention features, This indicates that the outputs of the 8 heads will be concatenated. This represents the output of the i-th head. This represents the vector after projecting the query and key-value pairs onto the i-th subspace. Let be the relative position offset matrix of the i-th head, representing the offset between the query position and the key-value pair position in the i-th head. The key information for text features refers to the maximum value of the text features (i.e., the BERT output) along the sequence dimension; The dimension of the key and query vector.

[0104] The fused features are input into the pre-trained classification model, and the classification result of the fused features is output.

[0105] Example 2: Refer to Figure 2 The difference between this embodiment and Embodiment 1 is that the S13 classification also includes:

[0106] S21 divides the set, iterates through the coordinate index of each image patch, and integrates image patches belonging to the same page into a single image patch set.

[0107] S22 constructs a mask matrix, obtains the dimension of the attention matrix in the attention mechanism, and constructs a binary mask matrix with the same dimension as the attention matrix. The binary mask matrix can prohibit attention interactions between different page elements, making the inference process conform to the physical structure of the document.

[0108] The element in the i-th row and j-th column of the binary mask matrix indicates whether the i-th image block and the j-th image block belong to the same page. If yes, the value of the element is 1; if no, the value of the element is 0.

[0109] S23 calculates the new weights, obtains the center coordinates of the i-th image patch set, and calculates the Manhattan distance between the m-th image patch containing the center coordinates of the i-th image patch set and the remaining image patches in the i-th image patch set. The formula for calculating the Manhattan distance is as follows:

[0110] ;

[0111] in, The Manhattan distance is the distance between the m-th image patch containing the center coordinates of the i-th image patch set and the remaining n-th image patch in the i-th image patch set; The coordinates of the center of the m-th image patch; The coordinates of the center of the nth image patch.

[0112] An attention bias term is constructed based on Manhattan distance. The sum of the attention bias term and the attention score in the attention mechanism is used as the new attention score. The calculation model for the new attention score is as follows:

[0113] ;

[0114] in, This is a learnable parameter, with a value range of 0.3-0.7; This represents the maximum Manhattan distance corresponding to the i-th image patch set; For the new attention score; This is the attention bias term.

[0115] Update the attention matrix with the new attention score to obtain the updated attention matrix.

[0116] S24 calculates the new attention matrix by multiplying the binary mask matrix element-wise with the updated attention matrix to obtain the multiplied matrix, and uses the multiplied matrix as the new attention matrix.

[0117] In S13 classification, when using the attention mechanism to perform weighted fusion of visual features, topological features and text features of the same image patch, the new attention matrix in S24 is used to calculate the new attention matrix to fuse the three feature vectors.

[0118] Preferably, the original loss function (cross-entropy) of the classification model is adjusted using the bounding box regression loss function and the contrastive loss function. The classification model is then retrained using the adjusted loss function. The calculation model for the adjusted loss function is as follows:

[0119] ;

[0120] in, The adjusted loss function; For the original loss function The weights; For bounding box regression loss function The weights; To compare loss functions The weight.

[0121] The original loss function calculation model is as follows:

[0122] ;

[0123] Where N is the number of training samples; E is the number of classification categories; The one-hot encoded value corresponding to the true label of training sample i; Predict the probability that training sample i belongs to category e for the classification model.

[0124] The calculation model for the bounding box regression loss function is as follows:

[0125] ;

[0126] in, These are the predicted bounding box coordinates; is the true boundary coordinates; f is the coordinate component; (x,y) is the center point coordinates; w is the width; h is the height.

[0127] The contrastive loss function forces the multimodal features of the same sample to align in the latent space. The calculation model is as follows:

[0128] ;

[0129] in, The preset boundary threshold, ranging from 0.1 to 2.0, is used to enhance intermodal consistency.

[0130] Example 3: This example discloses an intelligent contract analysis system based on a multimodal feature fusion algorithm. The system includes:

[0131] The data acquisition and processing module first extracts the layout information of the page (such as text box positions, table areas, and signature areas) through a PDF parsing library (such as PyPDF2 or PDFMiner), and then divides the PDF contract document into image blocks based on the layout.

[0132] The feature extraction module, connected to the data acquisition and processing module, is used to construct a multimodal collaborative feature extraction model. Each image patch is input into the multimodal collaborative feature extraction model to obtain the visual features, topological features, and text features of each image patch.

[0133] The classification module communicates with the feature extraction module. It inputs visual features, topological features, and text features belonging to the same image patch into the attention weight network, outputs the weights of the three types of features, performs weighted fusion processing on the three types of features based on their respective weights, obtains fused features, inputs the fused features into the pre-trained classification model, and outputs the classification result of the fused features.

[0134] The classification model can adopt structures such as CNN and RNN. As long as the classification model is trained with training data, it can classify the fused features. The training data refers to the fused features with training labels corresponding to historical image patches after the above processing. The training labels include resident ID cards, household registration books, calculation sheets, etc.

[0135] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A smart contract analysis method based on a multimodal feature fusion algorithm, characterized in that, include: Data acquisition and processing: Obtain PDF contract documents, segment the PDF contract documents, and obtain multiple image blocks; Feature extraction: Construct a multimodal collaborative feature extraction model, input each image patch into the multimodal collaborative feature extraction model, and obtain the visual features, topological features and text features of each image patch respectively; The multimodal collaborative feature extraction model includes a visual feature extraction sub-model for extracting visual features of image patches, a topological structure feature extraction sub-model for extracting topological features between image patches, and a text semantic understanding sub-model for extracting text features of image patches. The visual feature extraction sub-model includes a CSPDarknet53 network with CBAM layers and a bidirectional feature pyramid network. The CSPDarknet53 network is used to perform five-level downsampling on image patches to obtain multi-scale feature maps. The bidirectional feature pyramid network is used to enhance the target region in the multi-scale feature map and output the enhanced multi-scale feature map. The enhanced multi-scale feature map is then weighted by the CBAM layer to output the visual features. The text semantic understanding sub-model includes a CRNN sub-model and an MLP sub-model. The process of extracting text features by the text semantic understanding sub-model is as follows: Image blocks of text type are located and identified using a CRNN sub-model, denoted as text blocks. Geometric parameters of the text blocks are obtained, including center coordinates and width and height. The tilt angle of the text blocks is calculated based on the geometric parameters. Determine if the tilt angle is zero. If it is, do nothing. If not, perform an affine transformation on the text block based on the tilt angle to obtain a corrected text block, and update the text block with the corrected text block. The geometric parameters of the text block are normalized, and the MLP sub-model is used to map the normalized geometric parameters into position vectors to obtain the coordinate index of the text block. The coordinate index is encoded to obtain the original position encoding vector. The position vector is added to the original position encoding vector to obtain the text features. The topology feature extraction sub-model includes an edge enhancement graph convolutional network. The process of extracting topology features by the topology feature extraction sub-model is as follows: Using image blocks as nodes, the features of the nodes include: the center coordinates, width and height of the image blocks and visual features. The Euclidean distance between image blocks is calculated based on the center coordinates. Edges are constructed between image blocks whose Euclidean distance is less than a preset distance threshold to obtain a dynamic graph structure. An edge-enhanced graph convolutional network is used to capture the topological features in the dynamic graph structure. Classification: For the same image patch, visual features, topological features, and text features are weighted and fused using an attention mechanism to obtain fused features. The fused features are then input into a pre-trained classification model, which outputs the classification result of the fused features.

2. The smart contract analysis method based on multimodal feature fusion algorithm according to claim 1, characterized in that, The steps of data acquisition and processing include: The PDF contract document is parsed to obtain the page pixel map. The image blocks in the page pixel map are separated by the OpenCV contour detection algorithm. The bounding boxes of all image blocks are projected onto the same coordinate system to obtain the coordinate values ​​of each vertex of each image block. Based on the coordinate values ​​of each vertex of each image patch, a coordinate index for each image patch is established.

3. The smart contract analysis method based on multimodal feature fusion algorithm according to claim 2, characterized in that, The classification process also includes: Based on the coordinate index of each image patch, image patches belonging to the same page are integrated into an image patch set. The dimension of the attention matrix in the attention mechanism is obtained, and a binary mask matrix with the same dimension as the attention matrix is ​​constructed. The element in the i-th row and j-th column of the binary mask matrix indicates whether the i-th image patch and the j-th image patch belong to the same page. If yes, the element has a value of 1; if no, the element has a value of 0. Multiply the binary mask matrix and the attention matrix element by element to obtain the multiplied matrix, and use the multiplied matrix as the new attention matrix.

4. The smart contract analysis method based on multimodal feature fusion algorithm according to claim 3, characterized in that, The classification process also includes: Obtain the center coordinates of the page corresponding to the i-th image patch set, calculate the Manhattan distance between the image patch containing the center coordinates of the page corresponding to the i-th image patch set and the remaining image patches in the i-th image patch set, construct an attention bias term based on the Manhattan distance, and use the sum of the attention bias term and the attention score in the attention mechanism as the new attention score.

5. The smart contract analysis method based on multimodal feature fusion algorithm according to claim 1, characterized in that, The classification process also includes: The visual features, topological features, and text features belonging to the same image patch are aligned and concatenated to obtain a concatenated vector. The concatenated vector is then input into a pre-built MLP model to obtain the gating weights of the visual features, topological features, and text features, respectively. The product of visual features and their corresponding gating weights is used as the new visual features; the product of topological features and their corresponding gating weights is used as the new topological features; and the product of text features and their corresponding gating weights is used as the new text features.

6. An intelligent contract analysis system based on a multimodal feature fusion algorithm, wherein the system is used to implement the method as described in any one of claims 1-5, characterized in that, include: The data acquisition and processing module is used to acquire PDF contract documents, segment the PDF contract documents, and obtain multiple image blocks. The feature extraction module is used to build a multimodal collaborative feature extraction model. Each image patch is input into the multimodal collaborative feature extraction model to obtain the visual features, topological features and text features of each image patch. The classification module is used to perform weighted fusion of visual features, topological features, and text features of the same image patch using an attention mechanism to obtain fused features. The fused features are then input into the pre-trained classification model, which outputs the classification result of the fused features.

Images