Ocr recognition method based on large model enhancement

Abstract

This invention relates to the fields of artificial intelligence and computer vision, and particularly to an OCR recognition method based on large model enhancement. This method generates a directional texture feature map by performing multi-directional difference operations and smoothing on the input image; extracts multi-scale features from the map and adaptively weights and fuses them to generate a multi-scale aggregated feature map; calculates energy statistics to generate spatial gating weights, and weights and enhances the aggregated feature map to obtain a spatially enhanced feature map; performs feature transformation on the spatially enhanced feature map and the directional texture feature map, and fuses deep and shallow features to generate a unified feature representation; based on this representation, reconstructs character morphology in the visual branch and performs semantic reasoning in the language branch, fusing the results of the two branches to obtain the target recognition result. This invention significantly improves the accuracy and robustness of OCR recognition in complex scenes through directional texture modeling, adaptive multi-scale fusion, and visual-language dual-branch collaboration.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to an OCR recognition method based on large model enhancement. Background Technology

[0002] Current OCR solutions generally employ two main approaches: one is a pipeline centered on convolutional features and sequence modeling (such as CNN / CRNN+CTC or attention-based decoding), often supplemented by fixed or lightweight super-resolution and post-processing dictionaries; the other is a Transformer-style OCR / document understanding model centered on self-attention, combining multi-scale features and language models for contextual correction. To improve clarity, some methods overlay image enhancement or super-resolution networks at the front end; to improve robustness, some methods introduce channel / spatial attention, layout priors, and dictionary constraints. However, these solutions generally focus on general features to unified decoding, lacking a refined coupling design oriented towards the collaboration between text morphology and semantics.

[0003] Problems with existing technology:

[0004] 1. The direction and fine strokes are not sensitive, and the edges and the direction of the strokes are easily lost in low resolution.

[0005] 2. Insufficient scale adaptation; over- or under-segmentation is prone to occur when fonts of different thicknesses are mixed or overlapped.

[0006] 3. In low light, complex textures, and noisy environments, high-frequency interference in the background is amplified, leading to false edges and false detections.

[0007] 4. Statistical mismatch between deep and shallow layers can easily lead to numerical conflicts when directly added together.

[0008] 5. Relying solely on linguistic a priori evidence or solely on visual evidence is prone to error: the former may be semantically correct but morphologically incorrect, while the latter may be morphologically similar but incomprehensible.

[0009] 6. The lack of an auditable and controllable error correction chain makes it difficult to trace the source and iteratively optimize industrial processes. Summary of the Invention

[0010] Therefore, the present invention provides an OCR recognition method based on large model enhancement to solve the aforementioned problems existing in the prior art.

[0011] To achieve the above objectives, this invention provides an OCR recognition method based on large model enhancement, comprising:

[0012] Step S1: Perform multi-directional difference operations and smoothing on the input image to generate a directional texture feature map;

[0013] Step S2: Multi-scale feature extraction is performed on the directional texture feature map, and adaptive weights are used to weight and fuse features at different scales to generate a multi-scale aggregated feature map.

[0014] Step S3: Calculate the energy statistics of the multi-scale aggregated feature map, generate spatial gating weights based on the energy statistics, and use the spatial gating weights to perform weighted enhancement on the multi-scale aggregated feature map to obtain a spatially enhanced feature map.

[0015] Step S4: Perform feature transformation on the spatial enhancement feature map and the directional texture feature map respectively, and generate a unified feature representation by fusing parameters to combine shallow and deep features;

[0016] Step S5: Based on the unified feature representation, reconstruct the character shape in the visual branch, perform semantic reasoning using the language model in the language branch, and obtain the target recognition result by fusing the visual reconstruction result and the language reasoning result.

[0017] Furthermore, the process of step S1 includes:

[0018] ,

[0019] in, Represents directional texture feature map, This represents the fundamental feature value output at coordinates (x, y); This represents the pixel intensity of the input image at coordinates (x, y); Represents the absolute value response of the horizontal difference; Represents the absolute value response of the vertical difference; It represents the absolute value response of the difference in the main diagonal direction; mean(•) represents the local mean operator, which takes the average of the input scalar field in a finite neighborhood centered at (x,y); , and Three directional weighting coefficients are used to adjust the contribution of horizontal, vertical, and diagonal responses to the final texture energy.

[0020] Furthermore, the directional weight coefficients are learnable parameters, and by applying normalization constraints to each directional weight coefficient, the sum of each directional weight coefficient is made equal to 1, so as to avoid directional response bias and maintain numerical stability.

[0021] Furthermore, the process of step S2 includes:

[0022] The directional texture feature maps are input in parallel to multiple scale branches, and each branch performs a convolution operation using a receptive kernel of a different size to obtain response features at the corresponding scale.

[0023] The response features of each branch are extended and aligned in the channel dimension to obtain channel-aligned features;

[0024] The confidence level is selected from the scale of the corresponding features through a weight prediction mechanism, and dynamic weights related to each spatial location are generated.

[0025] The channel alignment features and the dynamic weights are weighted and summed to generate the multi-scale aggregated feature map.

[0026] Furthermore, the process of weighted summing of the channel alignment features and the dynamic weights to generate the multi-scale aggregated feature map includes:

[0027] ,

[0028] in, This represents the scale response obtained by convolving the directional texture feature map with the region receptive kernel Ki at the i-th scale. Represents the receptive nucleus of the i-th layer region; This indicates an operation that scales or aligns the tensor X along the channel dimension. This indicates the channel expansion ratio or target channel alignment factor, and is usually a positive integer or the target number of channels; This represents the dynamic weight of the i-th scale branch; F1 represents the weighted fusion of multi-scale results; F2 represents the multi-scale aggregated feature map.

[0029] Furthermore, the process of step S3 includes:

[0030] Calculate the average energy statistic and fluctuation energy statistic for the multi-scale aggregated feature map;

[0031] The average energy statistic and the fluctuating energy statistic are linearly weighted and fused according to a learnable energy ratio coefficient to obtain an energy indicator map.

[0032] Apply sigmoid nonlinear activation to the energy indicator map to generate a spatially gated weight map with values ​​ranging from [0,1].

[0033] The spatial gating weight map is multiplied element-wise with the multi-scale aggregated feature map to obtain the spatial enhanced feature map.

[0034] Furthermore, the process of performing element-wise multiplication of the spatial gating weight map and the multi-scale aggregated feature map to obtain the spatially enhanced feature map includes:

[0035] ,

[0036] in, This represents the vector or scalar value of the input feature map output by the region interaction aggregation unit at coordinates (x, y). Indicates to The result of performing average pooling; Indicates to The result of performing standard deviation pooling; β represents the scaling factor of the global average energy term, used to adjust the influence of avgpool(F2) on the gating; β represents the scaling factor of the local fluctuation energy term, used to adjust the influence of stdpool(F2) on the gating; sigmoid(•) represents the sigmoid nonlinear mapping; ⊙ represents element-wise multiplication; This represents the value of the spatially augmented feature map at coordinates (x, y).

[0037] Furthermore, the process of step S4 includes:

[0038] The spatial enhancement feature map is convolved and mapped, and the convolution result is subjected to tanh nonlinear suppression to obtain deep structural features;

[0039] The directional texture feature map is convolved and the convolution result is normalized to obtain shallow detail features.

[0040] The deep structural features and the shallow detail features are convexly combined using learnable fusion parameters to generate the unified feature representation.

[0041] Furthermore, the process of generating the unified feature representation by convexly combining the deep structural features and the shallow detail features using learnable fusion parameters includes:

[0042] ,

[0043] in, Indicates to Apply convolution weights The linear mapping; Wd is a learnable parameter; tanh(•) denotes hyperbolic tangent nonlinearity, and the output range is [-1, 1]; Indicates to Apply convolution weights The linear mapping; Ws is a learnable parameter; norm(•) represents the hierarchical normalization operation; λ represents the fusion ratio parameter, with a value range of 0≤λ≤1; (1 - λ) represents the complementary weights of the detail branches; This represents a unified feature representation.

[0044] Furthermore, the process of step S5 includes:

[0045] In the visual branch, the unified feature representation is deconvolutionally mapped to obtain high-resolution reconstructed features, and morphological correction processing is applied to the high-resolution reconstructed features to generate visual candidate results.

[0046] In the language branch, the unified feature representation is embedded and mapped to be converted into a sequence representation. The sequence representation is then input into a large language model for contextual semantic reasoning to generate semantic candidate results.

[0047] The visual candidate results and the semantic candidate results are fused together using a confidence-weighted mechanism to generate the target recognition result.

[0048] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention extracts directional features of character texture through multi-directional difference operations, combines smoothing processing to suppress background noise, and enhances the distinguishability of character structure from natural interference; it adopts adaptive weight fusion of multi-scale features to achieve information complementarity between local details and global context; it generates spatial gating weights based on energy statistics to focus feature enhancement on highly saliency character regions; it constructs a unified representation with both discriminative and robust properties through deep and shallow feature transformation and dynamic parameter fusion; it reconstructs the geometric shape of characters using a visual branch and performs semantic disambiguation using a language branch, with the two branches working together to simultaneously improve character accuracy and word rationality, significantly enhancing OCR recognition accuracy and anti-interference ability in complex scenes. Attached Figure Description

[0049] Figure 1 A schematic diagram of the structure of the OCR recognition method based on large model enhancement provided by the present invention;

[0050] Figure 2 This is a schematic diagram of step S2 in the OCR recognition method based on large model enhancement provided by the present invention;

[0051] Figure 3 This is a schematic diagram of step S3 in the OCR recognition method based on large model enhancement provided by the present invention;

[0052] Figure 4 This is a schematic diagram of step S4 in the OCR recognition method based on large model enhancement provided by the present invention; Detailed Implementation

[0053] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0054] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0055] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0056] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0057] Please see Figure 1 As shown, this invention provides an OCR recognition method based on large model enhancement, comprising:

[0058] Step S1: Perform multi-directional difference operations and smoothing on the input image to generate a directional texture feature map;

[0059] Specifically, step S1 includes the following process:

[0060] ,

[0061] in, Represents directional texture feature map, This represents the fundamental feature value output at coordinates (x, y); This represents the pixel intensity of the input image at coordinates (x, y); Represents the absolute value response of the horizontal difference; Represents the absolute value response of the vertical difference; It represents the absolute value response of the difference in the main diagonal direction; mean(•) represents the local mean operator, which takes the average of the input scalar field in a finite neighborhood centered at (x,y); , and Three directional weighting coefficients are used to adjust the contribution of horizontal, vertical, and diagonal responses to the final texture energy.

[0062] Specifically, a single-channel luminance input is used as I to reduce the interference of irrelevant color changes on gradient estimation. When the input is a multi-channel image, a linear weighted transformation can be performed first to obtain the luminance component before participating in the difference. Regarding boundary handling, x±1 or y±1 may exceed the limits; mirror filling or edge duplication are recommended to ensure the integrity of the difference and avoid black edge artifacts. In high-resolution or extremely noisy scenes, reflection filling can also be used to improve stability. The local window of the mean operator is generally set to 3×3 or 5×5. Smaller values ​​can preserve fine transitions, while larger values ​​can improve robustness but risk over-smoothing. It is recommended to adaptively adjust according to the target resolution and character size, or dynamically sample the window size from an optional set during the training phase to enhance generalization. To obtain higher numerical stability, the absolute difference can be directly calculated using standard absolute value operations in the forward pass, while a smooth approximation (e.g., approximating the derivative with a small threshold) is used in the backpropagation to avoid training instability caused by zero gradient points. If integer or fixed-point arithmetic is used at the inference end, the absolute value and difference operations can be implemented in a SIMD or vectorized manner to improve throughput. To suppress noise, lightweight normalization and truncation operations are added, such as local contrast normalization of F1 and threshold pruning at extremely high response points to avoid a few strong noise points dominating subsequent features. In engineering, the three-directional differences can be implemented as three depthwise separable 1D convolutions: the horizontal direction uses a kernel [-1,0,1] sliding along the x-axis, the vertical direction slides along the y-axis, and the diagonal direction can obtain the corresponding pixel differences through pre-offset before entering the mean window, thus accelerating the process with very few additional parameters. For batch inference, the three responses can be merged into a tensor for vectorized operations, reducing memory access. To adapt to multiple OCR input scenarios (tickets, nameplates, scanned documents, street scenes), adaptive histogram equalization or light denoising can be combined in the preprocessing stage to make TGIU's difference response more focused on realistic stroke transitions rather than illumination fluctuations.

[0063] Specifically, the directional weight coefficients are learnable parameters, and by applying normalization constraints to each directional weight coefficient, the sum of each directional weight coefficient is made equal to 1, so as to avoid directional response bias and maintain numerical stability.

[0064] Specifically, α1, α2, and α3 can be set using two strategies: one is a fixed strategy, which gives initial values ​​based on the prior features of the target language's font (e.g., horizontal Latin script emphasizes vertical strokes, so the weight of the horizontal difference can be appropriately increased) and freezes them before deployment; the other is a learnable strategy, which treats α1, α2, and α3 as trainable parameters at the channel or sub-block level and optimizes them together with the loss function during end-to-end training.

[0065] Step S2: Multi-scale feature extraction is performed on the directional texture feature map, and adaptive weights are used to weight and fuse features at different scales to generate a multi-scale aggregated feature map.

[0066] Specifically, step S2 includes:

[0067] Step S21: The directional texture feature map is input in parallel to multiple scale branches, and each branch uses a receptive kernel of a different size to perform a convolution operation to obtain the response features of the corresponding scale.

[0068] Specifically, the directional texture feature maps are input in parallel to multiple scale processing branches, each configured with a region receptive kernel of a different size. A typical configuration includes three scale branches with receptive kernel sizes set to 3×3, 5×5, and 7×7, respectively, to capture fine strokes, medium strokes, and block-level contextual information. Each branch independently performs convolution operations to generate response feature maps at the corresponding scale. Each response feature map retains the original spatial resolution and extracts stroke structure information at a specific scale.

[0069] Step S22: Perform extended alignment processing on the response features of each branch in the channel dimension to obtain channel aligned features;

[0070] Specifically, since the number of channels in the response feature maps generated by convolutions at different scales varies, channel expansion and alignment operations need to be performed on the outputs of each branch. The expand function maps the response features at each scale to a unified channel dimension. This can be achieved by channel duplication, 1×1 convolutional linear projection, or depthwise separable convolution, ensuring that all scale branches have the same number of channels, thus providing a numerically compatible basis for subsequent weighted fusion.

[0071] Step S23: Select confidence level from the scale of the corresponding feature through the weight prediction mechanism, and generate dynamic weights related to each spatial location;

[0072] Specifically, a lightweight weighted prediction subnetwork is constructed, using the directional texture feature map or the scale response features of each branch as input, to independently predict the confidence level at each spatial location. This subnetwork generates the original confidence score through channel compression and spatial aggregation operations, and then performs softmax normalization to output a dynamic weight map that satisfies the constraint Σw_i=1. The dynamic weight map achieves adaptive adjustment in space: smaller receptive kernels are given higher weights in fine stroke regions, while larger receptive kernels are preferentially activated in bold text regions.

[0073] Step S24: The channel alignment features and the dynamic weights are weighted and summed to generate the multi-scale aggregated feature map.

[0074] Specifically, the multi-scale response features aligned to the channel are summed element-wise with the dynamic weight map, and the resulting multi-scale aggregated feature map is generated using a convex combination strategy. This weighted fusion process dynamically selects the optimal scale information at each spatial location, effectively integrating fine-grained details with a broad context.

[0075] Specifically, the process of weighted summation of the channel alignment features and the dynamic weights to generate the multi-scale aggregated feature map includes:

[0076] ,

[0077] in, This represents the scale response obtained by convolving the directional texture feature map with the region receptive kernel Ki at the i-th scale. Represents the receptive nucleus of the i-th layer region; This indicates an operation that scales or aligns the tensor X along the channel dimension. This indicates the channel expansion ratio or target channel alignment factor, and is usually a positive integer or the target number of channels; This represents the dynamic weight of the i-th scale branch; F1 represents the weighted fusion of multi-scale results; F2 represents the multi-scale aggregated feature map.

[0078] Specifically, the F2 dimension is typically [h, w, c'] (c' depends on the channel alignment strategy), which carries contextual and structural information that adapts to the scale in space. Ordinary convolutional kernels or depthwise separable kernels can be used to reduce computational cost, and different dilation rates and grouping methods can be set according to task requirements. si is the channel expansion ratio specified for the i-th scale (e.g., mapping the scale response of lower channels to the target number of channels through channel replication, linear projection, or 1×1 convolution), used to ensure that all scale branches are consistent in the number of channels, facilitating subsequent weighting and summation.

[0079] Specifically, Ki is set as a set of lightweight convolutional kernels with increasing receptive fields (e.g., depthwise separable convolutions or point convolutions + channel-wise convolutions) to reduce parameters and MACs. If necessary, small dilatation rates can be introduced for 5×5 and 7×7 kernels to expand the effective receptive field while avoiding excessive sparsity and meshing effects. The three outputs of conv(F1, Ki) can be computed in parallel and axially concatenated within the same tensor for easy caching and SIMD optimization. Subsequently, expand(•, si) is applied to each output. It is recommended to use 1×1 convolutions for channel alignment rather than simple duplication to obtain stronger cross-channel linear combination capabilities with almost no increase in latency. The generation of dynamic weights wi can be achieved using a very lightweight weight prediction head: first, perform a global or local statistical analysis (such as channel averaging or 3×3 average pooling) on ​​the F1 or concatenated scale response, then follow with a 1×1 convolution and nonlinear mapping, and finally use normalization (e.g., soft normalization on the channel or scale dimension) to obtain wi. Note that the entire process only serves as a weight generation path and does not change the distribution of the main features. To ensure numerical stability and physical rationality, entropy regularization or a minimum lower bound (clip) can be added to wi during training to avoid the phenomenon of single-scale "monopolization". When performing element-wise weighted summation after channel alignment, it is necessary to ensure that the [h, w] of the tensors of each scale are strictly consistent. The boundaries can be filled with mirror or reflection to maintain the continuity of the response. In mixed-precision inference, the weights wi and weighted summation operations should be kept at least fp16 precision to prevent the scale allocation error from being amplified. To further enhance robustness, a slight normalization (such as channel-wise normalization or range truncation) can be applied to the responses at each scale before weighted aggregation. A "warm start" strategy should be adopted in the early stages of training to initialize the wi to a uniform distribution, followed by data-driven learning of the optimal scale preference. If deployed on GPUs / NPUs, the three convolutions can be merged into group convolution batch processing or kernel fusion to reduce memory round trips. For edge devices, 7×7 branches can be pruned and replaced with 5×5 dilated convolutions to maintain an approximately unchanged receptive field, thereby reducing latency. In multi-language or multi-font scenarios, it is recommended to incorporate controllable geometric scaling and stroke thickness perturbations into data augmentation, enabling the weight prediction head to learn a wi allocation pattern that matches the actual character width distribution, thus significantly improving generalization ability across fonts and resolutions.

[0080] Specifically, the warm-start strategy refers to initializing the multi-scale dynamic weights w_i to a uniform distribution at the beginning of training (e.g., if three scale branches are used, then w_i = [1 / 3, 1 / 3, 1 / 3]), rather than randomly initializing or pre-training the weights. During training, the weight prediction network starts from this uniform distribution and learns and adjusts the weight allocation at each scale gradually, driven by data, eventually converging to a non-uniform optimal distribution that matches the local morphology of the character (thin strokes, thick strokes, and connected structures).

[0081] Step S3: Calculate the energy statistics of the multi-scale aggregated feature map, generate spatial gating weights based on the energy statistics, and use the spatial gating weights to perform weighted enhancement on the multi-scale aggregated feature map to obtain a spatially enhanced feature map.

[0082] Specifically, such as Figure 3 As shown, the process of step S3 includes:

[0083] Step S31: Calculate the average energy statistic and fluctuation energy statistic for the multi-scale aggregated feature map;

[0084] Specifically, two types of pooling operations are performed in parallel on the multi-scale aggregated feature map: the first is the average energy statistic, which calculates the mean in the local or global neighborhood using the avgpool operator to form an energy base representing the overall brightness of the background and the main character; the second is the fluctuation energy statistic, which calculates the local standard deviation using the stdpool operator to capture high-order details such as stroke edges and texture abrupt changes. Both types of statistics use a sliding window mechanism (window size k×k, where k is an odd number) to generate an energy map aligned with the input spatial resolution at each position in the feature map space dimension, ensuring that subsequent gating has pixel-level fine granularity.

[0085] Step S32: The average energy statistic and the fluctuation energy statistic are linearly weighted and fused according to a learnable energy ratio coefficient to obtain an energy indicator map;

[0086] Specifically, learnable energy proportion coefficients γ and β are introduced to adjust the contributions of average energy and fluctuation energy, respectively. The two types of energy statistics are linearly weighted element-wise according to γ ​​and β to obtain a fused energy indicator map. This indicator map shows a high response value in the character stroke area (due to the simultaneous presence of high average energy and high fluctuation energy), while the response value in the background area is low (only average energy and weak fluctuation), thus constructing a separable energy distribution for characters and background.

[0087] Step S33: Apply sigmoid nonlinear activation to the energy indicator map to generate a spatially gated weight map with values ​​ranging from [0,1].

[0088] Specifically, the spatial gating weight map physically represents the feature retention ratio at each spatial location: regions with weights approaching 1 are identified as the main character, and their features are fully preserved and enhanced; regions with weights approaching 0 are identified as background or noise, and their features are significantly suppressed. This design achieves adaptive spatial selection through a differentiable gating mechanism, and the gating weights themselves are interpretable.

[0089] Step S34: Multiply the spatial gating weight map and the multi-scale aggregated feature map element-wise to obtain the spatial enhancement feature map.

[0090] Specifically, the process of multiplying the spatial gating weight map element-wise with the multi-scale aggregated feature map to obtain the spatially enhanced feature map includes:

[0091] ,

[0092] in, This represents the vector or scalar value of the input feature map output by the region interaction aggregation unit at coordinates (x, y). Indicates to The result of performing average pooling; Indicates to The result of performing standard deviation pooling; β represents the scaling factor of the global average energy term, used to adjust the influence of avgpool(F2) on the gating; β represents the scaling factor of the local fluctuation energy term, used to adjust the influence of stdpool(F2) on the gating; sigmoid(•) represents the sigmoid nonlinear mapping; ⊙ represents element-wise multiplication; This represents the value of the spatially augmented feature map at coordinates (x, y).

[0093] Specifically, the pooling strategy should be determined first: if the exposure and contrast of the input scene change drastically (e.g., street scene, natural scene OCR), local average pooling and local standard deviation pooling (e.g., window size k×k, k is 3 or 5) are recommended to obtain more refined spatial adaptation; if the deployment platform has limited resources or the input domain is relatively stable (tickets, scanned documents), global statistics can be used to reduce overhead. To avoid numerical instability in the standard deviation calculation in the extremely low variance region, it is recommended to add a very small positive number epsilon (e.g., 1e-6) to the variance term in the stdpool implementation and then take the square root, or use robust estimation (e.g., absolute deviation average) as an alternative. In the processing of the channel dimension, the gating weights can be generated channel by channel (shape [h, w, c]) or shared by channel (shape [h, w, 1] and then broadcast to c). The former is suitable for scenarios where different channels represent different semantic subspaces, while the latter has lower parameters and stronger spatial consistency priors. γ and β can be set as learnable parameters and can be implemented in a channel-wise scalar form, initialized to γ=1.0, β=1.0, or biased based on data statistics. To ensure stability in the early stages of training, scaling and offset can be added to the gating input (e.g., linear normalization before entering sigmoid), or the sigmoid can be replaced with a smooth hard-limited approximation and gradually switched back to sigmoid. During inference, avgpool and stdpool can be implemented using highly optimized sliding window kernels or through im2col+ vectorization. On GPUs / NPUs, kernel fusion with adjacent convolutional layers can be performed to reduce memory access. To further improve robustness, it is recommended to add a slight normalization or residual path after the SEGU output: for example, the output can be set as a convex combination of F2 and gating enhancement, F3 = η * (gating enhancement) + (1 - η) * F2, where η is a learnable or fixed small weight, which can prevent information flow blockage caused by excessive gating in the early stages of training. For ultra-low light or strong noise scenarios, a mild bilateral filter or guided filter can be added before the stdpool branch to suppress the misleading effect of high-frequency noise on the fluctuation energy; at the same time, regularization or range clipping can be added to γ ​​and β to prevent a certain branch from being over-amplified and causing instability.

[0094] Step S4: Perform feature transformation on the spatial enhancement feature map and the directional texture feature map respectively, and generate a unified feature representation by fusing parameters to combine shallow and deep features;

[0095] Specifically, step S4 includes the following process:

[0096] Step S41: Perform convolution mapping on the spatial enhancement feature map and apply tanh nonlinear suppression to the convolution result to obtain deep structure features;

[0097] Specifically, a convolutional mapping operation is performed on the spatially enhanced feature map, using a large-size receptive kernel (such as 5×5 or 7×7) or a depthwise separable convolution with a dilation rate to capture the global context structure at the character line block and character block level. The convolutional output is then subjected to tanh hyperbolic tangent nonlinear suppression, constraining the feature range to the [-1,1] interval, suppressing anomalous activation peaks, making the deep structure representation insensitive to the absolute strength of strokes, and focusing on a robust representation of "presence and weakness", ultimately generating a deep structure feature map.

[0098] Step S42: Perform convolution mapping on the directional texture feature map and apply normalization processing to the convolution result to obtain shallow detail features;

[0099] Specifically, convolutional mapping is performed on the directional texture feature map, using a small receptive kernel (e.g., 3×3) to extract local high-frequency details such as edges, corners, and fine textures. The convolutional output is normalized by using LayerNorm or GroupNorm to align the statistical distribution along the channel or group dimension, standardizing the shallow features to zero mean and unit variance, eliminating numerical scale differences with deeper branches, and generating a shallow detail feature map with the same channel dimension as the deep structural feature map.

[0100] Step S43: Perform a convex combination of the deep structural features and the shallow detail features using learnable fusion parameters to generate the unified feature representation.

[0101] Specifically, the process of generating the unified feature representation by convexly combining the deep structural features and the shallow detail features using learnable fusion parameters includes:

[0102] ,

[0103] in, Indicates to Apply convolution weights The linear mapping; Wd is a learnable parameter; tanh(•) denotes hyperbolic tangent nonlinearity, and the output range is [-1, 1]; Indicates to Apply convolution weights The linear mapping; Ws is a learnable parameter; norm(•) represents the hierarchical normalization operation; λ represents the fusion ratio parameter, with a value range of 0≤λ≤1; (1 - λ) represents the complementary weights of the detail branches; This represents a unified feature representation.

[0104] Specifically, first, ensure that F1 and F3 are aligned in spatial resolution [h, w]: if the upstream-generated F3 resolution is low, subpixel upsampling or lightweight deconvolution can be used to align it to match F1; if the number of channels is different, a 1×1 convolution can be used to map the two branches to the same number of channels c. For the convolution mappings conv(F3, Wd) and conv(F1, Ws), a lightweight structure of "1×1 point convolution aligning channels + 3×3 depthwise separable convolution for refinement" is recommended, preserving representational power while controlling latency. When using tanh for structural branches, it often makes the values ​​more concentrated, facilitating subsequent addition with normalized detail branches; if the deployment side is sensitive to the cost of tanh, a piecewise approximation of hard tanh can be used, gradually switching to standard tanh during training to maintain accuracy. For norm, it is recommended to use normalization that does not require batch statistics (such as LayerNorm or GroupNorm) to avoid variance jitter in small batches or online inference; if LayerNorm is used, learnable scaling and bias should be applied in the channel dimension to restore expressive power. There are three strategies for setting the fusion coefficient λ: a fixed scalar (e.g., 0.5, simple and stable), a learnable scalar (globally adaptive), and a learnable channel-wise or position-wise graph (spatially adaptive). In practice, the initial value of λ is often biased towards structural branches (e.g., 0.6 to 0.7), allowing the network to adjust itself during training. To avoid overfitting, range clipping or regularization can be applied to λ. To improve stability, it is recommended to add residual bypass: for example, the output can be short-circuited with F3 or F1 with small weights to prevent information blocking caused by improper weights in the early stages of training. Regarding quantization and deployment, tanh is sensitive to fixed-point implementation and should be symmetrically fixed-pointed after normalization; convolution and normalization can reduce memory round trips through kernel fusion. If the target scene has a large font range (from light to bold) and a large resolution range (from mobile phone photography to high-resolution scanning), weak supervision can be applied to λ based on character width and contrast during training (e.g., increasing λ on wide stroke samples and decreasing λ on thin stroke samples), or λ can be dynamically predicted from F1 and F3 statistics through gating little head to improve cross-domain generalization.

[0105] Step S5: Based on the unified feature representation, reconstruct the character shape in the visual branch, perform semantic reasoning using the language model in the language branch, and obtain the target recognition result by fusing the visual reconstruction result and the language reasoning result.

[0106] Specifically, step S5 includes the following process:

[0107] In the visual branch, the unified feature representation is deconvolutionally mapped to obtain high-resolution reconstructed features, and morphological correction processing is applied to the high-resolution reconstructed features to generate visual candidate results.

[0108] Specifically, a deconvolution mapping (i.e., transposed convolution) is performed on the unified feature representation, and a learnable upsampling kernel is used to restore the low-resolution feature map to the original image size or higher resolution, generating high-resolution reconstructed features containing stroke geometry. Subsequently, morphological correction processing is applied, using edge sharpening operators to enhance character outline clarity, employing morphological restoration algorithms to repair broken strokes, and performing background suppression and binarization thinning operations to finally generate visual candidate results. These candidate results are presented in the form of character probability maps or structured stroke maps, accurately reflecting the morphological features and spatial layout of the characters.

[0109] In the language branch, the unified feature representation is embedded and mapped to be converted into a sequence representation. The sequence representation is then input into a large language model for contextual semantic reasoning to generate semantic candidate results.

[0110] Specifically, the unified feature representation is embedded and mapped using a two-dimensional to one-dimensional transformation strategy (such as line-first scanning, ROI region segmentation and splicing, or learnable attention convergence) to convert spatial features into a sequence representation that preserves the reading order, with additional positional and layout encodings. This sequence representation is then input into a large language model, leveraging its deep context modeling capabilities to impose prior language constraints (including spelling rules, lexical and grammatical structures, and domain lexicon knowledge) across characters and words, generating distributed semantic candidate results. These semantic candidate results can be character-level, sub-word-level, or word-level probability distribution sequences.

[0111] The visual candidate results and the semantic candidate results are fused together using a confidence-weighted mechanism to generate the target recognition result.

[0112] Specifically, it can be expressed by the formula: O = φ( deconv(F4, Wr) ) + ψ( LM(embed(F4)) ) where deconv(F4, Wr) represents the deconvolution upsampling operator, mapping F4 to a higher spatial resolution; Wr is the set of learnable weights for deconvolution, including parameters such as kernel size, stride, and padding, used to recover stroke width, endpoints, and gaps. The output of deconv can be understood as high-resolution features or an approximate reconstructed pattern. φ(•) represents the visual reconstruction function, which enhances the readability of the deconvolution result in the image domain and performs post-processing. Its functions include edge sharpening, fine stroke shape correction, stroke interruption repair, background suppression, and binarization thinning. The output of φ can be a character probability map, a structured stroke map, or a direct character mask, depending on the implementation strategy. `embed(F4)` represents the embedding mapping function, which converts the two-dimensional spatial features F4 into a one-dimensional or hybrid sequence representation that can be processed by the language model. Spatial information can be projected onto the sequence channels through row-first or column-first scanning, ROI segmentation followed by concatenation, or learnable attention convergence, with the addition of positional or layout encoding to preserve reading order and layout. `LM(•)` represents the large model language module, used for contextual modeling and sequence prediction on the embedded sequences. `LM` can output the distribution of subwords, characters, or word-level units, and apply language and knowledge priors (e.g., spelling, lexical, grammatical, domain lexicon) across characters, words, and even lines. `ψ(•)` represents the semantic correction function, which performs visual consistency-based reconstraints and confidence integration on the `LM` output. Typical operations include alignment with candidates in the visual branch, priority matching of pocket dictionaries or domain lexicons, rescoring based on edit distance or pronunciation similarity, and lightweight validation of syntactic consistency. The output of `ψ` is a semantic candidate sequence or its score mapping. O represents the target recognition result, which can be an aligned character sequence, a sub-word sequence, or a structured result containing positional information and confidence level. O can simultaneously carry text content, page coordinates, block-level confidence level, and error correction tags, facilitating downstream retrieval or auditing.

[0113] Specifically, firstly, `deconv(F4, Wr)` is executed in the vision branch: for common 2× or 4× upsampling, "deconvolution + learnable interpolation" or "equivalent deconvolution with pixel-shuffle" is used to reduce checkerboard artifacts; if the deployment environment is latency-sensitive, deconvolution can be replaced with a lightweight combination of "nearest neighbor / bilinear upsampling + 3×3 convolution". Then, hierarchical post-processing is added within φ: first, adaptive thresholding or guided binarization is performed to improve sharpness, then thinning and break repair (such as stroke completion based on skeletonization and local curvature constraints) are done, and finally, morphological little-kernel opening and closing operations are used to eliminate isolated noise and adhesion. In the language branch, the key to embed(F4) is the lossy but structurally preserved mapping from space to sequence: for single-line text, line-first sliding window pooling and positional encoding are sufficient; for multi-line or complex layouts, it is recommended to first score the character block regions using a lightweight layout header and sort them according to the reading order, and then perform splicing and learnable aggregation. LM can use a large model with instruction fine-tuning and domain vocabulary expansion to better handle proper nouns, units of measurement, and mixed languages; if necessary, introduce a domain dictionary or regularization rules to reduce illusions and misclassifications of homographs. The implementation of ψ can adopt a two-stage alignment + re-scoring process: the first stage uses CTC style or edit distance to establish the shortest path alignment between visual and language candidates; the second stage uses joint confidence (such as the balance between visual pixel confidence and language conditional probability) to re-scorrate and select the optimal sequence; for noisy samples, a mild error recovery strategy can be added (e.g., allowing low-weight character insertion or deletion). For training, multi-task optimization is recommended: use a hybrid loss strategy combining reconstruction and segmentation (e.g., pixel-level cross-entropy and contour consistency terms) on the vision side, and sequence cross-entropy or CTC loss on the language side, with collaborative consistency as a constraint (e.g., encouraging high consistency between visual and language candidates at the character level). During inference, a lightweight beam search or temperature-controlled greedy strategy can be employed to balance accuracy and latency. In edge devices, distillation and vocabulary pruning can be performed on the LM, and fixed-point friendly approximation operators can be used at φ and ψ. To improve industrial usability, it is recommended to output character-level and word-level confidence scores, traceable error correction tags, and replayable candidate paths in O, facilitating human review and closed-loop iteration.

[0114] Specifically, this invention extracts directional features of character texture through multi-directional difference operations, combines smoothing processing to suppress background noise, and enhances the distinguishability of character structure from natural interference; it employs adaptive weight fusion of multi-scale features to achieve information complementarity between local details and global context; it generates spatial gating weights based on energy statistics to focus feature enhancement on highly saliency character regions; it constructs a unified representation with both discriminative and robust properties through deep and shallow feature transformation and dynamic parameter fusion; it reconstructs the geometric shape of characters using a visual branch and performs semantic disambiguation using a language branch, with the two branches working together to simultaneously improve character accuracy and word rationality, significantly enhancing OCR recognition accuracy and anti-interference capability in complex scenarios.

[0115] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0116] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An OCR recognition method based on large model enhancement, characterized in that, include: Step S1: Perform multi-directional difference operations and smoothing on the input image to generate a directional texture feature map; Step S2: Multi-scale feature extraction is performed on the directional texture feature map, and adaptive weights are used to weight and fuse features at different scales to generate a multi-scale aggregated feature map. Step S3: Calculate the energy statistics of the multi-scale aggregated feature map, generate spatial gating weights based on the energy statistics, and use the spatial gating weights to perform weighted enhancement on the multi-scale aggregated feature map to obtain a spatially enhanced feature map. Step S4: Perform feature transformation on the spatial enhancement feature map and the directional texture feature map respectively, and generate a unified feature representation by fusing parameters to combine shallow and deep features; Step S5: Based on the unified feature representation, reconstruct the character shape in the visual branch, perform semantic reasoning using the language model in the language branch, and obtain the target recognition result by fusing the visual reconstruction result and the language reasoning result.

2. The OCR recognition method based on large model enhancement according to claim 1, characterized in that, The process of step S1 includes: ， in, Represents directional texture feature map, This represents the fundamental feature value output at coordinates (x, y); This represents the pixel intensity of the input image at coordinates (x, y); Represents the absolute value response of the horizontal difference; Represents the absolute value response of the vertical difference; It represents the absolute value response of the difference in the main diagonal direction; mean(•) represents the local mean operator, which takes the average of the input scalar field in a finite neighborhood centered at (x,y); , and Three directional weighting coefficients are used to adjust the contribution of the horizontal, vertical, and diagonal responses to the final texture energy.

3. The OCR recognition method based on large model enhancement according to claim 2, characterized in that, The directional weight coefficients are learnable parameters, and by applying normalization constraints to each directional weight coefficient, the sum of each directional weight coefficient is made equal to 1, so as to avoid directional response bias and maintain numerical stability.

4. The OCR recognition method based on large model enhancement according to claim 3, characterized in that, The process of step S2 includes: The directional texture feature maps are input in parallel to multiple scale branches, and each branch performs a convolution operation using a receptive kernel of a different size to obtain response features at the corresponding scale. The response features of each branch are extended and aligned in the channel dimension to obtain channel-aligned features; The confidence level is selected by scale from the response features through a weight prediction mechanism, and dynamic weights related to each spatial location are generated. The channel alignment features and the dynamic weights are weighted and summed to generate the multi-scale aggregated feature map.

5. The OCR recognition method based on large model enhancement according to claim 4, characterized in that, The process of weighted summing of the channel alignment features and the dynamic weights to generate the multi-scale aggregated feature map includes: ， in, This represents the scale response obtained by convolving the directional texture feature map with the region receptive kernel Ki at the i-th scale. Represents the receptive nucleus of the i-th layer region; This indicates an operation that scales or aligns the tensor X along the channel dimension. This indicates the channel expansion ratio or target channel alignment factor, and is usually a positive integer or the target number of channels; This represents the dynamic weight of the i-th scale branch; F1 represents the weighted fusion of multi-scale results; F2 represents the multi-scale aggregated feature map.

6. The OCR recognition method based on large model enhancement according to claim 5, characterized in that, The process of step S3 includes: Calculate the average energy statistic and fluctuation energy statistic for the multi-scale aggregated feature map; The average energy statistic and the fluctuating energy statistic are linearly weighted and fused according to a learnable energy ratio coefficient to obtain an energy indicator map. Apply sigmoid nonlinear activation to the energy indicator map to generate a spatially gated weight map with values ​​ranging from [0,1]. The spatial gating weight map is multiplied element-wise with the multi-scale aggregated feature map to obtain the spatial enhanced feature map.

7. The OCR recognition method based on large model enhancement according to claim 6, characterized in that, The process of multiplying the spatial gated weight map element-wise with the multi-scale aggregated feature map to obtain the spatially enhanced feature map includes: ， in, This represents the vector or scalar value of the input feature map output by the region interaction aggregation unit at coordinates (x, y). Indicates to The result of performing average pooling; Indicates to The result of performing standard deviation pooling; β represents the scaling factor of the global average energy term, used to adjust the influence of avgpool(F2) on the gating; β represents the scaling factor of the local fluctuation energy term, used to adjust the influence of stdpool(F2) on the gating; sigmoid(•) represents the sigmoid nonlinear mapping; ⊙ represents element-wise multiplication; This represents the value of the spatially augmented feature map at coordinates (x, y).

8. The OCR recognition method based on large model enhancement according to claim 7, characterized in that, The process of step S4 includes: The spatial enhancement feature map is convolved and mapped, and the convolution result is subjected to tanh nonlinear suppression to obtain deep structural features; The directional texture feature map is convolved and the convolution result is normalized to obtain shallow detail features. The deep structural features and the shallow detail features are convexly combined using learnable fusion parameters to generate the unified feature representation.

9. The OCR recognition method based on large model enhancement according to claim 8, characterized in that, The process of generating the unified feature representation by performing a convex combination of the deep structural features and the shallow detail features using learnable fusion parameters includes: ， in, Indicates to Apply convolution weights The linear mapping; Wd is a learnable parameter; tanh(•) denotes hyperbolic tangent nonlinearity, and the output range is [-1, 1]; Indicates to Apply convolution weights The linear mapping; Ws is a learnable parameter; norm(•) represents the hierarchical normalization operation; λ represents the fusion ratio parameter, with a value range of 0≤λ≤1; (1 - λ) represents the complementary weights of the detail branches; This represents a unified feature representation.

10. The OCR recognition method based on large model enhancement according to claim 9, characterized in that, The process of step S5 includes: In the visual branch, the unified feature representation is deconvolutionally mapped to obtain high-resolution reconstructed features, and morphological correction processing is applied to the high-resolution reconstructed features to generate visual candidate results. In the language branch, the unified feature representation is embedded and mapped to be converted into a sequence representation. The sequence representation is then input into a large language model for contextual semantic reasoning to generate semantic candidate results. The visual candidate results and the semantic candidate results are fused together using a confidence-weighted mechanism to generate the target recognition result.

Images