Unstructured data extraction method and device based on texture enhancement and feature fusion

Abstract

The invention relates to the technical field of target detection, in particular to an unstructured data extraction method and device based on texture enhancement and feature fusion, and the method comprises the steps: extracting the multi-scale features of an input image through a feature extraction network, carrying out the adaptive enhancement of the image texture through a parallel multi-branch differential convolution structure, and carrying out the extraction of the multi-scale features of the input image; obtaining an enhanced multi-scale feature map; inputting the enhanced multi-scale feature map into a feature fusion network, adaptively determining a reference layer of a feature pyramid based on target size distribution statistical information, and aligning feature maps of other levels with the reference layer; fusing the local features and the global context features on the aligned feature map through an attention-driven mechanism to obtain fusion features containing multi-scale information; and performing target detection through the detection head, and outputting the structured information of the unstructured data elements. And the accuracy of fuzzy target identification and the adaptability to detection of targets with different sizes are improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of target detection, in particular to a non-structured data extraction method and device based on texture enhancement and feature fusion. BACKGROUND

[0002] In the process of enterprise digital transformation, the intelligent transformation of the fields of finance, tax and the like is a key link, and the core challenge lies in efficient and accurate automatic processing of massive non-structured data such as invoices and statements. The traditional processing method is inefficient and has a high error rate, and it is difficult to meet the needs of medium and large enterprises to process tens of thousands of documents per month, which has become a bottleneck for improving operational efficiency and reducing costs. Deep integration of non-structured data and ERP, BI and other systems is crucial for automatic accounting matching, tax declaration, supply chain data analysis and intelligent decision-making.

[0003] Currently, the automatic extraction method based on one-stage target detection is the mainstream technology, but in complex actual business scenarios, its detection accuracy and robustness still have significant shortcomings. The specific technical defects are as follows: first, in the face of moire caused by scanning, reflection caused by mobile device shooting, and fuzzy text and noise caused by multiple copying, the traditional image preprocessing method has single enhancement effect and cannot adapt to different image degradation characteristics. Second, if a complex deep learning model is used for image pre-enhancement, significant computational overhead will be introduced, causing detection process delay and failing to meet real-time requirements. These technical limitations directly lead to serious problems at the business level: recognition errors occur frequently, making ERP system data inconsistent, requiring a large amount of manual intervention for checking, and the automatic process is just a name; misidentification of key fields such as amount and tax number may cause business bias and compliance risk; and distorted basic data input into the BI system will make the basis for business decision-making unreliable.

[0004] Therefore, there is an urgent need in the art for a non-structured data extraction method that can balance high accuracy and high efficiency in complex scenarios, adaptively enhance fuzzy text texture, and fully integrate multi-scale features to improve the accuracy of positioning and classification, while ensuring processing speed, so as to truly support the automation and intelligentization of enterprise financial processes. SUMMARY

[0005] To solve the above problems, the application provides a non-structured data extraction method and device based on texture enhancement and feature fusion, which is implemented based on a one-stage target detection framework, improves the accuracy of fuzzy target recognition and the adaptability to different size targets, thereby improving the positioning accuracy and detection accuracy.

[0006] In a first aspect, the present invention provides a method for extracting unstructured data based on texture enhancement and feature fusion, executed by a trained neural network model, the method comprising: S1. Multi-scale features of the input image are extracted through the feature extraction network of the model. In the feature extraction process, a parallel multi-branch differential convolution structure is used to adaptively enhance the image texture in order to obtain the enhanced multi-scale feature map. S2. Input the enhanced multi-scale feature map into the feature fusion network of the model, adaptively determine the baseline layer of the feature pyramid based on the target size distribution statistics, and align the feature maps of other levels with the baseline layer; S3. On the aligned feature map, local features and global context features are fused through the attention-driven mechanism of the model to obtain fused features containing multi-scale information. S4. Based on the fusion features, target detection is performed using the detection head of the model, and the structured information of the unstructured data elements in the input image is output.

[0007] As a further limitation of the technical solution of the present invention, in S1, a parallel multi-branch differential convolution structure is used to adaptively enhance the image texture, including: S11. Deploy at least two convolutional branches with different receptive fields to process input features in parallel; wherein each of the convolutional branches is constructed by integrating a regular convolutional kernel with one or more differential convolutional kernels; S12. Reparameterize the parallel multi-branch differential convolutional structure: During the model training phase, the ordinary convolutional kernel and each differential convolutional kernel participate in the forward computation as independent convolutional layers; during the model inference phase, the weight parameters at the same position in each convolutional branch and each differential convolutional kernel are linearly added together and merged into an equivalent ordinary convolutional kernel.

[0008] Employing a parallel multi-branch differential convolutional structure, input features are processed in parallel through convolutional branches with different receptive fields, enabling the capture of image texture information at multiple scales and enhancing the detection capability for targets at different scales. During model training, ordinary convolutional kernels and differential convolutional kernels participate in forward computation as independent convolutional layers, fully utilizing the characteristics of each kernel for feature extraction. During model inference, a reparameterization operation is performed, merging multiple convolutional kernels into an equivalent ordinary convolutional kernel. This reduces the number of model parameters and computational complexity, improving inference speed and efficiency. The model maintains high detection accuracy while enabling faster extraction of unstructured data, enhancing its practicality in real-world applications.

[0009] As a further limitation of the technical solution of the present invention, at least two convolutional branches with different receptive fields include a first convolutional branch and a second convolutional branch. Differential convolution kernels include at least two of the following types: central difference convolution CDConv, angular difference convolution ADConv, horizontal difference convolution HDConv, and vertical difference convolution VDConv.

[0010] Convolutional branches with different receptive fields can extract features at different scales, while multiple differential convolutional kernels can enhance image texture from different directions, making the model more expressive and adaptable to the texture features of images, thereby further improving the detection accuracy and robustness of complex texture targets in unstructured data.

[0011] As a further limitation of the technical solution of the present invention, the reparameterization process is to linearly add the weight parameters at the same position in all convolution kernels within each convolution branch and merge them into an equivalent ordinary convolution kernel. This can be achieved through the following formula: For the first convolutional branch, the equivalent weights , ; For the second convolutional branch, the equivalent weights , ; in, Represents a specific type of convolution kernel in the first... Weights at each position This represents the weight of a specific type of convolutional kernel at position j.

[0012] By linearly summing the weights at the same positions in all convolutional kernels within each convolutional branch, merging them into an equivalent ordinary convolutional kernel, not only is the number of model parameters reduced, lowering the model's storage and computational costs, but the model can also perform convolution operations more efficiently during the inference phase, improving the model's runtime efficiency. This optimization method enhances the model's lightweight nature without compromising its performance.

[0013] As a further limitation of the technical solution of the present invention, in S2, the adaptive determination of the reference layer of the feature pyramid based on the statistical information of the target size distribution includes: S21. Determine the number of clusters based on the number of layers in the feature pyramid of the feature fusion network; S22. During the model training phase, cluster analysis is performed based on the center coordinates of the true bounding boxes in the training dataset. S23. Based on the clustering results, select the feature layer containing the largest number of targets as the baseline layer.

[0014] Adaptively determining the baseline layer of the feature pyramid based on target size distribution statistics allows for automatic selection of the most suitable feature layer as the baseline, avoiding potential biases caused by manually setting the baseline. By clustering the center coordinates of the ground truth bounding boxes in the training dataset, the feature layer containing the most targets is selected as the baseline layer. This makes the feature alignment process more reasonable and effective, reducing the loss of key information due to improper feature layer alignment, improving the quality and accuracy of feature fusion, and ultimately enhancing the model's detection performance for targets of different sizes. It also strengthens the model's adaptability and detection accuracy for diverse targets in unstructured data.

[0015] As a further limitation of the technical solution of the present invention, in S3, a transpose injection module is used to fuse local features and global context features. The transpose injection module performs the following operations: S31. Combine the local features with the transformed global features to form a combined feature; S32. Based on the splicing features, generate two attention weight maps; S33. The local features and global context features are weighted using the two attention weight maps respectively to obtain the enhanced local representation and the enhanced global representation; S34. Add the enhanced local representation to the local features, and add the enhanced global representation to the global context features to obtain the fused features.

[0016] The introduction of the transpose injection module enables the effective fusion of local features and global contextual features. By concatenating local features with the transformed global features, a concatenated feature is formed. Two attention weight maps are then generated based on this concatenated feature. These attention weight maps are used to weight the local and global contextual features, resulting in enhanced local and global representations. Finally, the enhanced representation is added to the original features to obtain the fused feature. This process not only fuses feature information from different ranges but also highlights important feature information through the attention mechanism, enhancing the model's ability to perceive global feature information and express features. This allows the model to better understand the contextual information of the target, thereby improving the accuracy and robustness of target detection and further enhancing the quality and reliability of unstructured data extraction.

[0017] As a further limitation of the technical solution of the present invention, step S32 includes: The splicing features are processed using a dual-branch structure to generate a first proxy token and a second proxy token; each branch of the dual-branch structure contains a 1×1 convolution and a depthwise separable 3×3 convolution. Projecting the spliced ​​features generates the original query and the original key; Multiply the second proxy token by the original key and multiply the first proxy token by the original query to obtain two reduced-dimensional intermediate representations; Multiply the two reduced intermediate representations to reconstruct two complete attention weight maps of size C×C.

[0018] This approach not only reduces the number of parameters and improves computational efficiency, but also enables more accurate calculation of the attention weight map, allowing the model to focus more effectively on important feature information. This further improves the accuracy of feature fusion and the model's ability to perceive global information, thereby enhancing the model's detection performance for complex targets in unstructured data.

[0019] As a further limitation of the technical solution of the present invention, step S33 includes: Local features and global context features are used as values ​​and projected onto the same feature space; The first attention weight map is multiplied by the projected local feature values ​​to obtain the enhanced local representation; The second attention weight map is multiplied by the projected global feature values ​​to obtain the enhanced global representation.

[0020] The ability to dynamically adjust feature weights based on their importance allows the model to focus more on key feature information, further enhancing the expressive power of features and the model's ability to perceive global information. The fused features obtained in this way are more accurate and effective, better guiding the target detection process and improving the accuracy and reliability of extracting structured information from unstructured data elements, thus further enhancing the overall performance of the model.

[0021] As a further limitation of the technical solution of the present invention, in S4, the target detection based on the fused features is performed by the detection head network. The detection head network adopts an adaptive anchor box mechanism to dynamically adjust the aspect ratio of the preset anchor box according to the shape of the detected target. The output structured information includes the category label of the target element and its bounding box coordinates in the image.

[0022] The detection head network employs an adaptive anchor box mechanism, dynamically adjusting the aspect ratio of preset anchor boxes based on the shape of the detected target. This allows for better matching of targets with different shapes, improving the accuracy and adaptability of target detection. The output structured information includes the category label of the target element and its bounding box coordinates in the image, providing accurate structured information for further processing and analysis of unstructured data. This makes the model's output more consistent with practical application needs and better meets the requirements for unstructured data extraction and processing in different scenarios.

[0023] Secondly, the present invention also provides an unstructured data extraction device based on texture enhancement and feature fusion, comprising: The feature extraction network is used to extract multi-scale features from the input image. During the feature extraction process, a parallel multi-branch differential convolutional structure is used to adaptively enhance the image texture to obtain the enhanced multi-scale feature map. A feature fusion network is used to receive the enhanced multi-scale feature map and adaptively determine the reference layer of the feature pyramid based on the target size distribution statistics, and align the feature maps of other levels with the reference layer; The attention-driven module is used to fuse local features and global contextual features on the aligned feature map to obtain fused features containing multi-scale information. The detection head is used to perform target detection based on the fused features and output structured information of unstructured data elements in the input image.

[0024] As can be seen from the above technical solutions, this application has the following advantages: Adaptive enhancement of image texture through parallel multi-branch differential convolutional structures effectively improves the detection capability for complex textures such as blur. The enhanced multi-scale feature maps help extract target information more accurately. Adaptively determining the baseline layer of the feature pyramid based on the statistical information of target size distribution and aligning the feature maps avoids information loss due to feature layer alignment, improving the accuracy of feature fusion. The attention-driven mechanism fuses local features and global contextual features to obtain fused features containing multi-scale information, further enhancing the model's ability to perceive global information and express features, thereby improving the accuracy of target detection. This provides a more accurate basis for extracting structured information from unstructured data elements, improving the overall accuracy and efficiency of unstructured data extraction, and better meeting the needs of unstructured data processing in different scenarios. Attached Figure Description

[0025] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the method provided in an embodiment of the present invention.

[0027] Figure 2 This is the reparameterization process for the 3×3 convolution kernel.

[0028] Figure 3 This is the reparameterization process for the 5×5 convolution kernel. Detailed Implementation

[0029] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0031] This invention provides a method for extracting unstructured data based on texture enhancement and feature fusion, executed by a trained neural network model. The model includes a feature extraction network, a feature fusion network, and a detection head. The feature extraction network integrates C2F, SPPCSPC modules, and a texture enhancement module based on the YOLOv8 backbone network. The feature fusion network is based on GoldYOLO and improves the retention of key and global feature information through improved integration and distribution injection methods. Figure 1 As shown, the method includes: S1. Multi-scale features of the input image are extracted through a feature extraction network. In the feature extraction process, a parallel multi-branch differential convolution structure is used to adaptively enhance the image texture in order to obtain the enhanced multi-scale feature map. S2. Input the enhanced multi-scale feature map into the feature fusion network, adaptively determine the reference layer of the feature pyramid based on the target size distribution statistics, and align the feature maps of other levels with the reference layer; S3. On the aligned feature map, local features and global context features are fused through an attention-driven mechanism to obtain fused features containing multi-scale information. S4. Target detection is performed based on the fused features, and the structured information of the unstructured data elements in the input image is output.

[0032] In some embodiments, the step of adaptively enhancing image texture using a parallel multi-branch differential convolution structure in S1 includes: S11. Deploy at least two convolutional branches with different receptive fields to process input features in parallel; wherein each of the convolutional branches is constructed by integrating a regular convolutional kernel with one or more differential convolutional kernels to enhance the ability to extract image gradient information. S12. Reparameterize the parallel multi-branch differential convolution structure, specifically including: During the model training phase, the ordinary convolutional kernel and each differential convolutional kernel participate in the forward computation as independent convolutional layers; During the model inference stage, the ordinary convolutional kernel in each convolutional branch is linearly added to the weight parameters at the same position in each differential convolutional kernel, and then merged into an equivalent ordinary convolutional kernel for inference acceleration.

[0033] This step is implemented using the Difference Convolution-based Texture Enhancement Module (DTEM). Because some unstructured data has small font sizes and low pixel variation, unstructured data extraction requires locating the target using a series of receptive fields of different sizes and capturing information changes by enhancing pixel differences and local texture. Compared to traditional convolution, differential convolution is particularly sensitive to gradient information and can capture subtle texture changes. Specifically, as the convolution kernel slides across the image, differential convolution uses the difference between adjacent pixels to replace the pixel value at the original position for convolution operations, thereby enhancing the edge and detail information of the texture. DTEM deploys two reparameterized parallel convolution kernels (i.e., 3×3 and 5×5 kernels) to extract gradient features with different receptive fields. A 3×3 convolution consists of a regular convolution (VConv), a central difference convolution (CDConv), an angular difference convolution (ADConv), a horizontal difference convolution (HDConv), and a vertical difference convolution (VDconv); a 5×5 difference convolution consists of a regular convolution (VConv), a central difference convolution (CDConv), and an angular difference convolution (ADConv). Differential convolution differs from traditional convolution in that it uses the weight difference between the two ends of the kernel arrow to replace the weights at the kernel position in a traditional convolution. Figure 2 As shown, taking 3×3CDConv as an example, compared to ordinary convolution weights, w1 is replaced with w1-w5, w2 is replaced with w2-w5, etc., and the parameter value in the blank part of the graph is 0. Figure 3 This describes the reparameterization process for the 5×5 convolution kernel. Furthermore, to reduce the number of parameters and computational complexity, a reparameterization technique is used, summing the weights at the same positions in multiple convolution kernels. The reparameterization process for the 3×3 convolution is expressed by the formula: Where (i∈(0,9]∩i∈Z); the reparameterization calculation process of the 5×5 convolution is expressed by the formula: Where j∈(0,25]∩j∈Z). The final result is obtained from the parameterized... (i∈(0,9]∩i∈Z) is equivalent to a 3×3 ordinary convolution kernel with weights and (j∈(0,25]∩j∈Z) is a 5×5 ordinary convolution kernel with weights.

[0034] in, Represents a specific type of convolution kernel in the first... Weights at each position This represents the weight of a specific type of convolutional kernel at position j.

[0035] In some embodiments, the step of adaptively determining the baseline layer of the feature pyramid based on the target size distribution statistics in S2 includes: S21. Determine the number of clusters based on the number of layers in the feature pyramid of the feature fusion network; S22. Perform cluster analysis based on the center coordinates of the true bounding boxes in the training dataset to determine the distribution of target size; S23. Based on the clustering results, select the feature layer containing the largest number of targets as the baseline layer.

[0036] The original fusion network uses the intermediate layer B4 as the baseline layer for the shallow feature alignment module, while the other feature layers B2, B3, and B5 are aligned using bilinear interpolation or downsampling. This alignment method preserves most of the defect features in the image, but it lacks the ability to adaptively preserve important features for different datasets. Since the network depth for feature information required for defects of different sizes varies, shallow features contain more location information and are more suitable for detecting smaller targets, while deep features contain more semantic information and are more suitable for detecting larger targets. If statistical data on defect sizes can be used to select the feature layer containing the largest defect target size and use that layer as the target size for feature alignment, the problem of feature information loss can be alleviated. This application employs an adaptive feature alignment method based on the K-means clustering algorithm. First, the number of cluster categories is determined based on the number of feature pyramid layers. Then, the k-means algorithm is used to cluster the center coordinates (x, y) of all GTboxes, and the number of each category is counted in descending order, corresponding to feature layers from shallow to deep. Furthermore, the category containing the largest number of ground truth annotations is selected, and its corresponding feature layer is used as the baseline layer. Finally, the remaining feature layers are aligned using bilinear interpolation or downsampling. This method can adaptively adjust the size of the baseline feature layer according to different datasets, minimizing the loss of defective features during the feature alignment process.

[0037] In some embodiments, in S3, the transpose injection module fuses local features and global contextual features through an attention-driven mechanism. The transpose injection module specifically performs the following operations: S31. The local features are concatenated with the transformed global features to form concatenated features, wherein the transformed global features are obtained by transforming the global context features; S32. Based on the splicing features, generate an attention weight map; S33. Use the attention weight map to weight the local features and the global context features respectively to obtain the enhanced local representation and the enhanced global representation; S34. Add the enhanced local representation to the local features, and add the enhanced global representation to the global context features to obtain the fused features containing multi-scale information.

[0038] It should be further explained that the steps in S32 include: The steps for generating the attention weight map based on the splicing features include: The splicing features are processed using a dual-branch structure to generate a first proxy token and a second proxy token; each branch of the dual-branch structure contains a 1×1 convolution and a depthwise separable 3×3 convolution. Projecting the spliced ​​features generates the original query and the original key; Multiply the second proxy token by the original key and multiply the first proxy token by the original query to obtain two reduced-dimensional intermediate representations; Multiply the two reduced intermediate representations to reconstruct two complete attention weight maps of size C×C.

[0039] The steps in S33 include: Local features and global context features are used as values ​​and projected onto the same feature space; The corresponding attention weight map is multiplied by the projected local and global eigenvalues ​​to obtain the enhanced local and global representations.

[0040] The local features and global context features are projected onto the same feature space as values, which is achieved through independent 1×1 convolutional layers.

[0041] Before matrix multiplication, local features and global context features are reshaped from spatial dimensions into token sequences, and the feature tensor is transformed from the shape of [batch size, number of channels, height, width] to the shape of [batch size, sequence length, number of channels], where the sequence length = height × width.

[0042] The transposed-injection module first concatenates local features with the transformed global features to fuse information from different ranges. Then, the concatenated features are processed by a two-branch attention structure, ultimately generating two attention maps of size C×C. The computation process for each attention head is as follows: A 1×1 convolution and a depthwise separable 3×3 convolution are applied to the normalized features to generate two sets of surrogate tokens, A1 and A2. To reduce the number of parameters, A1 and A2 are further mapped through two independent depthwise separable 3×3 convolutions to obtain intermediate surrogate tokens. In the attention computation, A acts as a surrogate for the key K, used to aggregate global information, while A2 acts as a surrogate for the query Q, used to capture relevance. Multiplying A2 with the original key K and A with the original query Q yields two dimensionality-reduced attention weight maps (dimension C×C / r). Multiplying these two attention weight maps reconstructs two complete C×C attention maps. These two attention maps are then multiplied by the reshaped value V features to generate the enhanced local representation Xla and the global representation Xlg. Finally, the transpose injection module adds the original local output Ol and global output Og to the attention enhancement results Xla and Xlg to form the final output. The transpose injection module calculates self-attention through a surrogate mechanism, which enhances the ability to extract global feature information while avoiding excessive parameter increases.

[0043] In some embodiments, in S4, target detection based on the fused features is performed by a detection head network, which adopts an adaptive anchor frame mechanism to dynamically adjust the aspect ratio of the preset anchor frame according to the shape of the detected target. The adaptive anchor box mechanism is achieved by clustering the aspect ratios of all real target boxes in the dataset during the training phase and using the cluster centers as the aspect ratios of the prior anchor boxes. The output structured information includes the category label of the target element and the bounding box coordinates of the target element in the image.

[0044] This invention employs a pre-trained neural network model as the texture enhancement and ensemble distribution fusion network model. The training process is based on an end-to-end deep learning training paradigm, encompassing four core stages: data preparation, model construction, multi-stage training optimization, and performance verification. Through a combination of hierarchical training and joint optimization, it ensures that all components of the model (feature extraction network, feature fusion network, attention-driven module, and detection head) work collaboratively to achieve optimal performance. It should be noted that the model training process is as follows: S01. Data Preparation: Obtain a training dataset containing unstructured data images (such as invoices and reports), and label the target elements (such as text boxes and table lines) in the images to generate true annotation boxes and category labels.

[0045] S02, Model Initialization: Construct a texture enhancement and ensemble distribution fusion network model, including a feature extraction network, a feature fusion network, and a detection head that integrate the texture enhancement module. Initialize the network parameters using pre-trained weights or a random initialization method.

[0046] S03, Forward Propagation: Input the training image into the network and execute steps S1 to S4 to obtain the predicted bounding boxes and class probabilities.

[0047] S04. Loss Calculation: Calculate the total loss between the model's prediction results and the actual annotations. The total loss is a weighted average of the localization loss, confidence loss, and classification loss.

[0048] S05. Backpropagation and parameter update: Using gradient descent algorithms (such as Adam, SGD), calculate the gradient based on the total loss value and update all trainable parameters in the network (including the parameters of the feature extraction network, fusion network, and detection head).

[0049] S06. Iterative Optimization: Repeat steps S03 to S05 until the model loss converges or the predetermined number of training epochs is reached, resulting in a trained unstructured data extraction model. The training process described above is further explained below: S01. Data Preparation and Preprocessing 1. Dataset Construction Collect diverse unstructured data images, including invoices of different formats (VAT invoices, general invoices), various reports (financial statements, statistical reports), documents (reimbursement forms, delivery forms), and mixed documents. The image resolution covers 300dpi to 600dpi and includes scenes with normal lighting, backlighting, shadows, and tilt.

[0050] The target elements are precisely annotated using a polygon annotation tool: text boxes (accurate to the character level boundary), table lines (distinguishing between horizontal / vertical / diagonal lines), QR codes / barcodes, etc., generating a JSON format annotation file containing coordinate information (x1, y1, x2, y2, x3, y3, x4, y4) and category labels.

[0051] The training set, validation set, and test set are divided in a 7:2:1 ratio to ensure that the data categories are distributed consistently in each set.

[0052] 2. Data Augmentation Processing Basic enhancements: Random rotation (-15° to +15°), horizontal flip, scaling, cropping.

[0053] Texture enhancement: Randomly add Gaussian noise (σ=0.01-0.05), blur (Gaussian blur kernel 3×3-7×7), and adjust contrast (0.5-1.5 times) to simulate the effect of low-quality scanned parts.

[0054] Color enhancement: For color documents, random color gamut conversion (RGB-HSV-LAB) and brightness adjustment (0.3-1.7 times) are performed to maintain the brightness consistency of grayscale documents.

[0055] S02, Model Component Initialization and Pre-training 1. Feature extraction network initialization Backbone network (based on YOLOv8): The C2F and SPPCSPC modules are initialized with weights pre-trained on the COCO dataset, and the parameters of the first 50% of the convolutional layers are frozen to preserve the low-level feature extraction capability.

[0056] Texture Enhancement Module (DTEM): 3×3 Convolution Branch: Ordinary convolution (VConv) is initialized using Kaiming normality, and the weights of differential convolution (CDConv / ADConv / HDConv / VDconv) are initialized by calculating the gradient sensitivity coefficients of neighboring pixels.

[0057] 5×5 convolutional branch: The initial weights are generated based on the parameters of the 3×3 convolutional kernel through a symmetric expansion method to ensure that the receptive field matches the feature scale.

[0058] Reparameterization: Initialize the fusion weight matrix as an identity matrix to ensure balanced contributions from each convolutional branch.

[0059] 2. Feature fusion network initialization Ensemble-Distribution Network: The feature ensemble layers for high and low stages are initialized uniformly using Xavier, and the distribution convolutional layers are initialized using a normal distribution with a bias of 0.

[0060] Transpose injection module: The query / key / value matrix of the attention mechanism is initialized with a random orthogonal matrix to ensure uniform attention distribution; 1×1 convolution and depthwise separable convolution are initialized with unit mapping.

[0061] 3. Initialize the detection head Classification branch: The output layer weights are initialized with a scaling factor related to the number of categories (scaling factor = 1 / √number of categories), and the bias is initialized with log (prior probability).

[0062] Localization branch: The weights and biases of the bounding box regression layer are initialized to 0 to ensure that the initial predicted values ​​are close to the center of the anchor box.

[0063] S03, Phased Training Process Phase 1: Texture enhancement module training (10-15 epochs) 1. Training Configuration With fixed parameters for the backbone network and feature fusion network, only the texture enhancement module is trained.

[0064] Learning rate: The initial learning rate is set to 1e-4, and a cosine annealing strategy is used to decay the learning rate.

[0065] Batch size: Based on GPU memory settings (16-32 images / batch).

[0066] 2. Training tasks Input samples with rich texture (such as densely packed table lines), and optimize the differential convolution parameters through contrastive loss to make the cosine distance between texture features and non-texture features greater than a preset threshold (0.7).

[0067] Monitoring metrics: Texture feature extraction accuracy (the proportion of correctly identified texture edges), gradient response intensity (the average gradient value of the texture region).

[0068] Phase 2: Feature fusion network training (20-30 epochs) 1. Training Configuration Unfreeze the top convolutional layer (30% parameters) of the feature extraction network and jointly train the texture enhancement module and the feature fusion network.

[0069] Learning rate: The learning rate for the backbone network is 5e-5, and the learning rate for the fusion network is 2e-4.

[0070] 2. Training tasks The Feature Consistency Loss method is used to make the MSE error between the fused feature map and the high-resolution feature map less than 1e-3.

[0071] The attention weights of the transpose injection module are optimized, and the correlation between local and global features is maximized through mutual information loss.

[0072] Monitoring metrics: Feature fusion IoU (Intersection over Union ratio of fused features to target region), attention map entropy (the lower the entropy value, the more concentrated the attention).

[0073] Phase 3: End-to-end joint training (50-80 epochs) 1. Training Configuration Unfreeze all network parameters and perform global optimization.

[0074] Optimizer: The AdamW optimizer is used (weight decay of 1e-4), with an initial learning rate of 1e-4 and a decay of 10 times every 20 epochs.

[0075] Regularization: Add a stochastic depth strategy (survival probability 0.8-0.9) to prevent overfitting.

[0076] 2. Loss Function Design

[0077] Location loss CIoU loss is used to accurately measure the accuracy of bounding box regression; Confidence loss Focal Loss (γ=2, α=0.25) solves the imbalance between positive and negative samples; Classification loss Improved cross-entropy loss, adding weight coefficients (1.5-2.0) to small sample classes; texture enhancement loss. Contrast loss based on texture gradient enhances the discriminative power of texture features. Weighting coefficients: (Can be dynamically adjusted based on validation set performance).

[0078] 3. Training Strategies Mixed precision training (FP16) is used to accelerate convergence, and gradient accumulation (4-step accumulation) is used to simulate the effect of large-scale training.

[0079] The top 30% of samples with the highest loss values ​​in each batch are selected for focused training to improve the model's ability to handle difficult samples.

[0080] S04, Model Optimization and Validation 1. Early Stop Strategy Using the average accuracy of the validation set as the monitoring metric, training is stopped and the optimal model weights are saved when there is no improvement for 10 consecutive epochs.

[0081] 2. Model pruning and quantization Perform channel pruning on the trained model: remove convolutional channels whose contribution (measured by L1 norm) is below the threshold (0.01), reducing the number of parameters by 30-40%.

[0082] By employing INT8 quantization technology, weights and activation values ​​are quantized from 32-bit floating-point to 8-bit integers, improving inference speed by 2-3 times while maintaining less than 2% precision loss.

[0083] 3. Performance Verification Evaluate on the test set: object detection accuracy, inference speed (FPS), and robustness under different data quality (detection accuracy for blurred / noisy images). Validate the model's accuracy improvement (≥5%) and speed advantage (≥10 FPS) on unstructured data extraction tasks.

[0084] This invention also provides an unstructured data extraction device based on texture enhancement and feature fusion, comprising: The feature extraction network is used to extract multi-scale features from the input image. During the feature extraction process, a parallel multi-branch differential convolutional structure is used to adaptively enhance the image texture to obtain the enhanced multi-scale feature map. A feature fusion network is used to receive the enhanced multi-scale feature map and adaptively determine the reference layer of the feature pyramid based on the target size distribution statistics, and align the feature maps of other levels with the reference layer; The attention-driven module is used to fuse local features and global contextual features on the aligned feature map to obtain fused features containing multi-scale information. The detection head is used to perform target detection based on the fused features and output structured information of unstructured data elements in the input image.

[0085] In some embodiments, the feature extraction network includes: At least two convolutional branches with different receptive fields, each of which is constructed by integrating a regular convolutional kernel with one or more differential convolutional kernels; The reparameterization module is used to enable the ordinary convolutional kernel and each differential convolutional kernel to participate in forward computation as independent convolutional layers during the model training phase; and to linearly add the weight parameters at the same position in each convolutional branch of the ordinary convolutional kernel and each differential convolutional kernel into an equivalent ordinary convolutional kernel during the model inference phase.

[0086] In some embodiments, the at least two convolutional branches with different receptive fields include a 3×3 convolutional branch and a 5×5 convolutional branch; the differential convolutional kernel includes at least two of the following types: central differential convolution CDConv, angular differential convolution ADConv, horizontal differential convolution HDConv, and vertical differential convolution VDConv.

[0087] The reparameterization module linearly adds the weight parameters at the same position in all convolutional kernels within each convolutional branch, merging them into an equivalent ordinary convolutional kernel; this is achieved through the following formula: For the first convolutional branch, the equivalent weights , ; For the second convolutional branch, the equivalent weights , ; in, Represents a specific type of convolution kernel in the first... Weights at each position This represents the weight of a specific type of convolutional kernel at position j.

[0088] In some embodiments, the feature fusion network includes: The clustering module is used to determine the number of clusters based on the number of layers in the feature pyramid of the feature fusion network; and to perform cluster analysis based on the center coordinates of the ground truth bounding boxes in the training dataset during the model training phase. The baseline layer selection module is used to select the feature layer containing the most targets as the baseline layer based on the clustering results.

[0089] In some embodiments, the attention-driven module includes a transpose injection module, which is used to perform the following operations: The local features are concatenated with the transformed global features to form concatenated features; Based on the splicing features, two attention weight maps are generated; The local features and global context features are weighted using the two attention weight maps respectively to obtain the enhanced local representation and the enhanced global representation; The enhanced local representation is added to the local features, and the enhanced global representation is added to the global context features to obtain the fused features.

[0090] In some embodiments, the transpose injection module includes: A dual-branch structure is used to process the splicing features and generate a first proxy token and a second proxy token; each branch of the dual-branch structure contains a 1×1 convolution and a depthwise separable 3×3 convolution. The projection module is used to project the splicing features to generate the original query and the original key; The attention calculation module is used to multiply the second proxy token with the original key and multiply the first proxy token with the original query to obtain two dimensionality-reduced intermediate representations; multiply the two dimensionality-reduced intermediate representations to reconstruct two complete attention weight graphs of size C×C.

[0091] In some embodiments, the transpose injection module further includes: The value projection module is used to project local features and global context features as values ​​onto the same feature space. The weighting module is used to perform matrix multiplication between the first attention weight map and the projected local feature values ​​to obtain the enhanced local representation; and to perform matrix multiplication between the second attention weight map and the projected global feature values ​​to obtain the enhanced global representation.

[0092] In some embodiments, the detection head employs an adaptive anchor frame mechanism, dynamically adjusting the aspect ratio of a preset anchor frame according to the shape of the detected target; the structured information output by the detection head includes the category label of the target element and its bounding box coordinates in the image.

[0093] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting unstructured data based on texture enhancement and feature fusion, characterized in that, The method, executed by a trained neural network model, includes: S1. Multi-scale features of the input image are extracted through the feature extraction network of the model. In the feature extraction process, a parallel multi-branch differential convolution structure is used to adaptively enhance the image texture in order to obtain the enhanced multi-scale feature map. S2. Input the enhanced multi-scale feature map into the feature fusion network of the model, adaptively determine the baseline layer of the feature pyramid based on the target size distribution statistics, and align the feature maps of other levels with the baseline layer; S3. On the aligned feature map, local features and global context features are fused through the attention-driven mechanism of the model to obtain fused features containing multi-scale information. S4. Based on the fusion features, target detection is performed using the detection head of the model, and the structured information of the unstructured data elements in the input image is output.

2. The method for extracting unstructured data based on texture enhancement and feature fusion according to claim 1, characterized in that, In S1, a parallel multi-branch differential convolution structure is used to adaptively enhance the image texture, including: S11. Deploy at least two convolutional branches with different receptive fields to process input features in parallel; wherein each of the convolutional branches is constructed by integrating a regular convolutional kernel with one or more differential convolutional kernels; S12. Reparameterize the parallel multi-branch differential convolutional structure: During the model training phase, the ordinary convolutional kernel and each differential convolutional kernel participate in the forward computation as independent convolutional layers; during the model inference phase, the weight parameters at the same position in each convolutional branch and each differential convolutional kernel are linearly added together and merged into an equivalent ordinary convolutional kernel.

3. The method for extracting unstructured data based on texture enhancement and feature fusion according to claim 2, characterized in that, At least two convolutional branches with different receptive fields include a first convolutional branch and a second convolutional branch; Differential convolution kernels include at least two of the following types: central difference convolution CDConv, angular difference convolution ADConv, horizontal difference convolution HDConv, and vertical difference convolution VDConv.

4. The unstructured data extraction method based on texture enhancement and feature fusion according to claim 3, characterized in that, The reparameterization process involves linearly adding the weights at the same position in all convolutional kernels within each convolutional branch, merging them into an equivalent ordinary convolutional kernel. This can be achieved through the following formula: For the first convolutional branch, the equivalent weights , ; For the second convolutional branch, the equivalent weights , ; in, Represents a specific type of convolution kernel in the first... Weights at each position This represents the weight of a specific type of convolutional kernel at position j.

5. The unstructured data extraction method based on texture enhancement and feature fusion according to claim 4, characterized in that, In S2, the adaptive determination of the baseline layer of the feature pyramid based on the statistical information of the target size distribution includes: S21. Determine the number of clusters based on the number of layers in the feature pyramid of the feature fusion network; S22. During the model training phase, cluster analysis is performed based on the center coordinates of the true bounding boxes in the training dataset. S23. Based on the clustering results, select the feature layer containing the largest number of targets as the baseline layer.

6. The method for extracting unstructured data based on texture enhancement and feature fusion according to claim 5, characterized in that, In S3, a transpose injection module fuses local features and global context features. The transpose injection module performs the following operations: S31. Combine the local features with the transformed global features to form a combined feature; S32. Based on the splicing features, generate two attention weight maps; S33. The local features and global context features are weighted using the two attention weight maps respectively to obtain the enhanced local representation and the enhanced global representation; S34. Add the enhanced local representation to the local features, and add the enhanced global representation to the global context features to obtain the fused features.

7. The unstructured data extraction method based on texture enhancement and feature fusion according to claim 6, characterized in that, The steps in S32 include: The splicing features are processed using a dual-branch structure to generate a first proxy token and a second proxy token; each branch of the dual-branch structure contains a 1×1 convolution and a depthwise separable 3×3 convolution. Projecting the spliced ​​features generates the original query and the original key; Multiply the second proxy token by the original key and multiply the first proxy token by the original query to obtain two reduced-dimensional intermediate representations; Multiply the two reduced intermediate representations to reconstruct two complete attention weight maps of size C×C.

8. The method for extracting unstructured data based on texture enhancement and feature fusion according to claim 7, characterized in that, The steps in S33 include: Local features and global context features are used as values ​​and projected onto the same feature space; The first attention weight map is multiplied by the projected local feature values ​​to obtain the enhanced local representation; The second attention weight map is multiplied by the projected global feature values ​​to obtain the enhanced global representation.

9. The method for extracting unstructured data based on texture enhancement and feature fusion according to claim 8, characterized in that, In S4, target detection based on the fused features is performed by a detection head network. This detection head network adopts an adaptive anchor box mechanism, which dynamically adjusts the aspect ratio of the preset anchor box according to the shape of the detected target. The output structured information includes the category label of the target element and its bounding box coordinates in the image.

10. A device for extracting unstructured data based on texture enhancement and feature fusion, characterized in that, include: The feature extraction network is used to extract multi-scale features from the input image. During the feature extraction process, a parallel multi-branch differential convolutional structure is used to adaptively enhance the image texture to obtain the enhanced multi-scale feature map. A feature fusion network is used to receive the enhanced multi-scale feature map, adaptively determine the reference layer of the feature pyramid based on the target size distribution statistics, and align the feature maps of other levels with the reference layer. The attention-driven module is used to fuse local features and global contextual features on the aligned feature map to obtain fused features containing multi-scale information. The detection head is used to perform target detection based on the fused features and output structured information of unstructured data elements in the input image.

Images