Document image character recognition method based on key point detection
By constructing a dynamic energy function to iteratively update the set of key points in the document image, the problem of key point detection failure caused by deformation in the existing technology is solved, and efficient recognition of flexible document images is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAN UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-03-23
- Publication Date
- 2026-06-26
Smart Images
Figure CN121904794B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision and image processing technology, specifically to a method for recognizing characters in document images based on key point detection. Background Technology
[0002] Document image character recognition technology is a core means of realizing automated financial processing. It usually involves geometric correction of the acquired image and subsequent text content extraction. It has high accuracy and robustness when processing flat or slightly tilted document images.
[0003] With the development of deep learning technology, character recognition methods have gradually evolved from traditional template matching to end-to-end recognition modes based on deep neural networks. However, when dealing with flexible documents that are severely curled or undergo non-rigid deformation, existing geometric correction techniques still rely excessively on the initial accuracy of key point detection. Large deformations can cause key point feature extraction to fail, thus falling into a logical deadlock where correction depends on key points but deformation renders key points unusable, resulting in a significant drop in the final character recognition rate. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention provides a method for character recognition in document images based on key point detection. Specifically, the technical solution of this invention includes:
[0005] The acquired document images are encoded using a feature extraction network to generate feature maps containing semantic information of the images.
[0006] Based on the response intensity distribution of the feature map, an initial set of candidate key points is generated, and each key point in the set of candidate key points is defined as a virtual particle characterized by coordinate parameters.
[0007] A dynamic energy function based on the feature map is constructed, wherein the dynamic energy function includes a data gravity term, a sequence elasticity term, and a topological repulsion term acting on the virtual particle;
[0008] During the optimization process, the coordinate parameters of the candidate key point set are iteratively updated by minimizing the total energy function containing the data attraction term, the sequence elasticity term, and the topological repulsion term, so as to generate a target key point sequence in a stable equilibrium state.
[0009] Based on the target key point sequence, the feature map is spatially transformed and corrected to generate aligned text features;
[0010] The text features are decoded using a sequence recognition network, and the character recognition result corresponding to the document image is output.
[0011] Preferably, constructing the dynamic energy function based on the feature map includes:
[0012] Based on the pixel gradient information of the feature map, the data gravity term is determined, which is used to characterize the numerical attraction of the high-response region in the feature map to the virtual particle.
[0013] Based on the relative distance between adjacent key points in the candidate key point set and the preset character spacing constraint, the sequence elasticity term is determined. The sequence elasticity term is used to characterize the geometric constraint force that constrains adjacent virtual particles to maintain a preset spacing and be collinear.
[0014] Based on the Euclidean distance between any two key points in the candidate key point set, the topological repulsion term is determined. The topological repulsion term is used to characterize the numerical repulsion force that prevents the virtual particles from coinciding with each other.
[0015] Preferably, the iterative update of the coordinate parameters of the candidate keypoint set by minimizing the total energy function containing the data gravity term, the sequence elasticity term, and the topological repulsion term includes:
[0016] The evolution process of the dynamic energy function is embedded in the forward propagation process of the neural network;
[0017] If the current iteration step count has not reached the preset evolution step count threshold, calculate the negative gradient of the total energy function with respect to the virtual particle coordinate parameters to determine the resultant force vector of the virtual particle under the combined action of the data gravity term, the sequence elasticity term, and the topological repulsion term.
[0018] Based on the resultant force vector, the coordinate parameters of the virtual particle are updated using the Euler integral algorithm until the current iteration step reaches the evolution step threshold, and the target key point sequence is output.
[0019] Preferably, the method further includes:
[0020] In the process of calculating the resultant force vector, the sequential elastic term and the topological repulsive term experienced by the virtual particle are aggregated through sparse matrix multiplication.
[0021] The sparse matrix multiplication operation is accelerated by utilizing the parallel computing core of the graphics processing unit.
[0022] Preferably, the step of performing spatial transformation correction on the feature map based on the target key point sequence includes:
[0023] The target key point sequence is used as the control points for thin-plate spline interpolation transformation;
[0024] Calculate the transformation parameter matrix based on the control points;
[0025] The text features are generated by mapping the non-rigidly deformed text regions in the feature map to flat rectangular regions using the transformation parameter matrix.
[0026] Preferably, the method further includes:
[0027] Based on the spatial distribution of the target key point sequence, a coordinate mapping relationship is constructed;
[0028] The original pixels of the document image are reverse-resampled using the coordinate mapping relationship to generate an enhanced document image that eliminates wrinkles and deformation.
[0029] Preferably, the decoding of the text features through a sequence recognition network includes:
[0030] The text features are input into an attention-based decoder;
[0031] The decoder converts the text features into a character probability sequence;
[0032] The final text content is determined based on the character probability sequence.
[0033] Preferably, the document image is a flexible media image, which includes thermal paper receipt images, fabric label images, or flexible packaging instruction manual images;
[0034] The character recognition result includes the text content and its corresponding confidence score.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] 1. This method introduces a dynamic evolution mechanism from physics to construct an energy function that includes data attraction, sequence elasticity, and topological repulsion terms. It treats key points of characters as virtual particles in the energy field, enabling automatic deconstruction and reconstruction of text lines without explicit 3D reconstruction. This mechanism utilizes a self-organizing process that minimizes energy, effectively solving the logical deadlock problem in existing technologies when dealing with severely curled or non-rigidly deformed flexible documents, where key point detection fails due to deformation, thus preventing geometric correction. This significantly improves the robustness of recognition for flexible media such as thermal paper receipts or soft packaging instructions.
[0037] 2. This method achieves end-to-end differentiable training of the physical model by embedding the dynamic evolution process into the forward propagation of the neural network and combining differentiable bilinear interpolation sampling with Euler integral algorithm. This design enables the network to automatically learn the optimal energy weights and update step size through backpropagation without the need for manual adjustment of complex physical parameters. At the same time, the physical dimension mapping mechanism eliminates the dimensional difference between pixel coordinates and energy values, ensuring the model's adaptability and numerical stability on data with different deformation degrees.
[0038] 3. This method aggregates nonlinear sequential elastic terms and topological repulsion terms by employing a sparse matrix multiplication strategy, and combines the construction of dynamic stiffness matrix and sparse Laplacian matrix to transform complex particle interactions into standard tensor operations adapted to graphics processor hardware acceleration. This scheme utilizes a parallel computing core to process the force gradients of all particles simultaneously, which significantly reduces computational complexity while ensuring the accuracy of the physical model, thus ensuring low latency and real-time performance of single-file image processing.
[0039] 4. This method uses inverse thin-plate spline interpolation transformation based on physical step size locking and dense pixel inverse resampling technology to force a fixed physical width to be assigned to each character during the correction process. This effectively avoids the flattening or elongation distortion that occurs when texts of different lengths are stretched to a fixed size, and eliminates sampling holes. Combined with a confidence evaluation mechanism that integrates energy convergence state and character prediction probability, it can not only generate visually flat enhanced images to assist manual review, but also accurately quantify the quality of geometric correction, thereby providing highly reliable recognition results in scenarios such as financial reimbursement audits. Attached Figure Description
[0040] The present invention will be further explained below with reference to the accompanying drawings and embodiments:
[0041] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.
[0043] Example 1:
[0044] Please see Figure 1 A method for character recognition in document images based on key point detection includes:
[0045] The acquired document images are encoded using a feature extraction network to generate feature maps containing semantic information of the images.
[0046] Based on the response intensity distribution of the feature map, an initial set of candidate key points is generated, and each key point in the set of candidate key points is defined as a virtual particle characterized by coordinate parameters.
[0047] A dynamic energy function based on feature maps is constructed, which includes a data attraction term, a sequence elasticity term, and a topological repulsion term acting on the virtual particle.
[0048] During the optimization process, the coordinate parameters of the candidate key point set are iteratively updated by minimizing the total energy function, which includes the data attraction term, the sequence elasticity term, and the topological repulsion term, in order to generate a target key point sequence in a stable equilibrium state.
[0049] Based on the target key point sequence, spatial transformation correction is performed on the feature map to generate aligned text features;
[0050] The text features are decoded using a sequence recognition network, and the character recognition results corresponding to the document image are output.
[0051] This embodiment details the character recognition logic based on a dynamic evolution mechanism. This method aims to solve the logical deadlock problem in existing technologies when processing flexible documents with severe curling or non-rigid deformation. This is because geometric correction relies on key point detection, but deformation causes key point detection to fail. The core idea of this embodiment is to introduce a dynamic evolution mechanism from physics, treating character key points as virtual particles in an energy field. Through a self-organizing process that minimizes energy, automatic deconstruction and reconstruction of text lines can be achieved without explicit 3D reconstruction.
[0052] Specifically, the system encodes the acquired document image I through a feature extraction network. In this embodiment, the feature extraction network uses a deep residual network ResNet-50 with fully connected layers removed as the backbone, and connects a feature pyramid structure FPN at its end. The input is the document image after size normalization, and the output is a feature map F containing semantic information of the image. This feature map not only contains the texture information of the characters, but the response intensity of its pixel values also corresponds to the confidence level of the character's existence.
[0053] The system generates an initial set of candidate keypoints based on the response intensity distribution of the feature map F. During this process, non-maximum suppression is applied to the feature map F, extracting local response extrema as candidate keypoints. To ensure the robustness of subsequent dynamic evolution and filter out spurious extrema generated by background noise, the system applies a preset response threshold at this point. For example, 0.3; the specific method for determining this threshold is: statistically analyze the mean feature response of non-character regions (i.e., the background) in the training set. with standard deviation ,set up This is to effectively suppress background noise; only responses with a strength greater than [a certain value] are retained. The extreme points; if the number of retained points exceeds the preset computational load limit. For example, 100. This upper limit is determined through stress testing based on the target inference hardware, such as the GPU's memory capacity and the number of parallel threads, to ensure that the processing latency of a single image meets real-time requirements, such as being less than 100ms. Then, the images are truncated in descending order of response intensity. The system extracts N key points; the system defines the extracted N key points as a set P, where each key point is represented by coordinate parameters; the initial set may be unordered and contain noise.
[0054] To ensure the validity of the geometric constraints of the sequential elastic terms in the dynamic model and to prevent topological entanglement caused by random indexing (i.e., virtual springs crossing and knotting in space), the system performs topological initialization preprocessing here. Specifically, to eliminate randomness in the sorting process and establish a unique initial topological structure, the system executes a deterministic lexicographical coordinate sorting algorithm here.
[0055] The algorithm defines a strict comparison operator: for sets Any two key points in and ,determination Ranked The previous necessary and sufficient condition was or ,in, For floating-point comparison tolerance, for example, 1e-5; this value is based on the image coordinates being normalized to... The assumed range needs to be adjusted accordingly if unnormalized absolute pixel coordinates are used; for example, it could be set to a value equal to the image width. The system multiplies the sorting by a factor of 1 to ensure that the comparison accuracy matches the coordinate scale; based on this uniquely determined sorting result, the system sorts the set... Sort in ascending order and reallocate the index. to This step initially organizes the discrete point set into a chain structure that conforms to the direction of text writing, such as from left to right, providing a correct adjacency relationship basis for subsequent elastic force calculation.
[0056] The system constructs a dynamic energy function E based on the feature map F. This function is a scalar field used to quantify the instability of the current keypoint distribution state, including a data attraction term derived from image features, a sequence elasticity term derived from text line priors, and a topological repulsion term derived from physical exclusivity. Based on this, the system iteratively updates the coordinate parameters of the candidate keypoint set by minimizing the total energy function E during the optimization process. This process simulates the evolution of a physical system from a high-energy state to a low-energy state. After K iterations, the virtual particles move to a stable position under the action of the force field, outputting a target keypoint sequence P in a stable equilibrium state. Then, based on the target keypoint sequence P, the system performs spatial transformation correction on the feature map F. Using a thin plate spline interpolation algorithm, with P as the control point, the curved and wrinkled text regions are straightened and mapped into a standard rectangular feature map, generating aligned text features. The system decodes the text features through a sequence recognition network and outputs the character recognition results corresponding to the document image.
[0057] Constructing a feature map-based dynamic energy function includes:
[0058] Based on the pixel gradient information of the feature map, the data gravity term is determined. The data gravity term is used to characterize the numerical attraction of high-response regions in the feature map to virtual particles.
[0059] Based on the relative distance between adjacent key points in the candidate key point set and the preset character spacing constraint, the sequence elasticity term is determined. The sequence elasticity term is used to characterize the geometric constraint force that constrains adjacent virtual particles to maintain a preset spacing and be collinear.
[0060] Based on the Euclidean distance between any two key points in the candidate key point set, a topological repulsion term is determined. The topological repulsion term is used to characterize the numerical repulsive force that prevents virtual particles from coinciding with each other.
[0061] This embodiment provides a detailed explanation of the specific implementation of constructing a dynamic energy function based on feature maps; to quantify the force situation of virtual particles, this embodiment defines the total energy function E as follows:
[0062]
[0063] in, , , These are the data attraction term, the sequence elasticity term, and the topological repulsion term, respectively.
[0064] : These are the corresponding weighting coefficients; to ensure the consistency of the energy dimensions in the physical model, i.e., to unify them as dimensionless scalars, this embodiment explicitly sets: The coefficient is dimensionless. Includes spatial scale normalization factor, physical unit is ; Includes spatial scale compensation factor, physical unit is In practical implementation, these coefficients can be used as fixed hyperparameters or learnable parameters of the network, with typical initial values satisfying... For example, set to The aforementioned weighting coefficients can be either fixed hyperparameters or learnable parameters of the network, which are automatically adjusted during training using gradient descent. The logic for setting their initial values is to ensure that the gravity term of the data dominates in the early stages of evolution, preventing excessive geometric constraints from leading to local minima.
[0065] The system determines the data attraction term G based on the pixel gradient information of the feature map. This term characterizes the numerical attraction of high-response regions in the feature map to virtual particles, aiming to ensure that keypoints do not detach from the real character stroke regions in the image. Let the keypoint set P contain N virtual particles, and the coordinates of the i-th particle be... The calculation formula is as follows:
[0066]
[0067] in, : The index of the key point, with a value range from 1 to N; : Coordinate parameters of the i-th virtual particle ; The response intensity value of the feature map at a specific coordinate is derived from the feature extraction network. To ensure the validity of the logarithmic operation and to give it a clear physical meaning, the feature map F is the result of a specific channel output by the feature extraction network being processed by the Sigmoid activation function. Its value range is strictly normalized to between 0 and 1, representing the posterior probability that the pixel position is the center of the character. A tiny constant derived from a preset value, for example, a value of [value to be filled in]. The physical meaning is to prevent the logarithm from being negative infinity;
[0068] By taking the negative logarithm of the feature response, the region with the larger the feature value (i.e., the character center) has the lower energy value, i.e., the potential energy trough, causing virtual particles to naturally slide towards the lowest energy point. The system determines the sequence elasticity term S based on the relative distance between adjacent keypoints in the candidate keypoint set and the preset character spacing constraint. This term characterizes the geometric constraint force that keeps adjacent virtual particles at a preset spacing and collinear, aiming to simulate the semantic connection between characters in a text line, similar to connecting adjacent characters with virtual springs. The calculation formula is as follows:
[0069]
[0070] in, : These are the first and second key points in the sequence. The and the first The coordinate vectors of a virtual particle;
[0071] The preset character spacing constraint is set to the average character width statistical value of the training set, in units of... ;
[0072] : Curvature penalty coefficient, with length squared The physical dimensions of , and its typical range of values is to Between them, to eliminate the dimensional difference with the squared distance value of the first term;
[0073] : No. The angle between local vectors at each key point. Specifically, Calculated based on the vector dot product formula:
[0074]
[0075] Here To prevent numerical stability constants with a denominator of zero, for example This characterizes The degree of collinearity of three adjacent points;
[0076] This formula calculates the preorder vector. with the subsequent vector The cosine of the angle between them; introduced here. ,For example This is to prevent numerical calculations from being halted due to overlapping coordinates of adjacent points, resulting in a denominator of 0. When three points are collinear and in the same direction, the included angle is 0, the cosine value is 1, and the energy term... The first part is a variant of Hooke's Law, constraining the distance between adjacent points to be close to the standard spacing d; the second part is a smoothing constraint, forcing three adjacent points to tend to be collinear; these two forces work together to straighten the curled text lines topologically; the system determines the topological repulsion term R based on the Euclidean distance between any two key points in the candidate key point set; this term is used to characterize the numerical repulsion force that prevents virtual particles from coinciding with each other, and its purpose is to avoid multiple key points collapsing to the same strong feature point in severely wrinkled areas, such as crumpled paper; in order to solve the energy divergence problem of the classical Coulomb force field when dealing with variable-length sequences, this embodiment introduces a sequence length normalization factor. The calculation formula is revised as follows:
[0077]
[0078] in, Normalization coefficient, derived from the total number of key points. Its physical significance lies in eliminating the cumulative effect of sequence length on the total potential energy; since the order of magnitude of the repulsive pairs is... Gravity and elasticity are Without normalization, the repulsive term of long texts will dominate the gradient, causing the system to diverge; introducing this term ensures the scale invariance of the force field across texts of different lengths. : Repulsion strength coefficient, derived from a preset value; applicable to coordinate parameters In image processing, this property is often considered a dimensionless quantity. This embodiment introduces a physical mapping mechanism when constructing the dynamic model, transforming the numerical distance of the pixel coordinate system... It is explicitly defined as a physical quantity with virtual characteristic length dimensions;
[0079] Based on this definition, in order to ensure the dimensional consistency of physical formulas, that is, to guarantee the topological repulsion term... Dimensions and total energy function The scalar energy dimensions are matched, and the coefficients Defined as a dimensionless intensity adjustment coefficient, its physical meaning is to adjust the relative amplitude of the repulsive potential energy; in conjunction with the aforementioned definition of a dimensionless coefficient of length... Weighting coefficients And the reciprocal of the distance term in the formula ,dimension Together, they ensure that the topological repulsion term is transformed into a dimensionless pure scalar when included in the total energy function, thus achieving dimensional unification with the data gravity term and the sequence elasticity term; : No. The and the first The Euclidean distance between key points, in physical terms, represents the spatial interval between particles; in practical implementation... The value of is usually set according to the normalization scale of the feature map, for example, a value of to The selection of this range is based on balancing topological separation and character compactness: too large... A value that is too high will cause the character spacing to increase abnormally, while a value that is too low will not effectively eliminate overlap. This coefficient can also be designed as an adaptive parameter that decays with the number of iterations.
[0080] This ensures that repulsive potential energy only becomes dominant when particles are extremely close; the addition here... ,For example This is to solve the singularity problem in the mathematical model and prevent division by zero errors caused by two particles randomly falling into the same point during the random initialization or high-energy evolution phase, which could lead to program crashes or gradient explosions; the summation range ensures that repulsion force calculations are performed on all non-repeating particle pairs in the set.
[0081] This simulation demonstrates the Coulomb force between charged particles; in response to two particles being too close, their energy increases sharply, generating a huge repulsive force that pushes them apart, thus ensuring that the output key point sequence is spatially separated and ordered.
[0082] The coordinate parameters of the candidate keypoint set are iteratively updated by minimizing the total energy function, which includes data attraction, sequence elasticity, and topological repulsion terms.
[0083] The evolution process of the dynamic energy function is embedded in the forward propagation process of the neural network;
[0084] If the current iteration step count has not reached the preset evolution step count threshold, calculate the negative gradient of the total energy function with respect to the virtual particle coordinate parameters to determine the resultant force vector of the virtual particle under the combined action of the data gravity term, the sequence elasticity term, and the topological repulsion term.
[0085] Based on the resultant force vector, the coordinate parameters of the virtual particles are updated using the Euler integral algorithm until the current iteration step reaches the evolution step threshold, and the target key point sequence is output.
[0086] This embodiment details the process of updating coordinate parameters by minimizing the total energy function. Instead of using traditional offline optimization algorithms, this embodiment embeds the dynamic evolution process into the forward propagation of the neural network, making it a differentiable layer. The system constructs a loop module within the neural network, setting an evolution step threshold T. This module receives the initial keypoint set P and outputs the final keypoint set. Before the current iteration step k reaches the threshold T, the system calculates the negative gradient of the total energy function with respect to the virtual particle's coordinate parameters. The resultant force vector f refers to the vector in the opposite direction of the gradient experienced by the virtual particle in the potential energy field, representing the trend of the particle's motion. The calculation formula is as follows:
[0087]
[0088] in, The net force vector acting on a particle is derived from the gradient calculation of the energy function; The partial derivative operator, in physical terms, is the rate of change of the energy field along the coordinate parameter direction;
[0089] Regarding the formula The specific calculations depend on the feature map. It is a discrete pixel grid, and conventional indexing operations are not differentiable; this embodiment uses a differentiable bilinear interpolation sampling mechanism; specifically, for arbitrary floating-point coordinates... The system finds it in the feature map The feature value at the four nearest integer pixels is calculated using the bilinear interpolation formula. Since the interpolation formula only includes weighted summation and multiplication operations, this process is sensitive to coordinates. It is continuously differentiable, thus enabling the accurate calculation of the gradient of the feature intensity with respect to the coordinate position through an automatic differentiation engine such as PyTorchAutograd, thereby achieving the backpropagation of the data gravitational field.
[0090] The system updates the coordinate parameters of the virtual particles based on the resultant force vector using the Euler integral algorithm; the calculation formula is as follows:
[0091]
[0092] in, The coordinate parameters of the current iteration step are derived from the superposition of the state and update amount of the previous time step; The update step size, or learning rate, is derived from a preset value and its physical meaning is the displacement amplitude in a single iteration; due to the resultant force vector... The dimensionless energy gradient with respect to coordinates has the following physical dimensions: Therefore, step size It should have The dimensions are set to maintain consistency between the units on both sides of the formula; to ensure the stability of the dynamic evolution and avoid virtual particles flying out of the effective region due to excessively large step sizes, The typical range of values is usually set as follows: to Between these values, the specific values depend on the degree of coordinate system normalization; when the keypoint coordinates are normalized to... When the interval is, Usually, a smaller value is chosen, such as If absolute pixel coordinates are used, then The image needs to be enlarged proportionally according to its resolution to ensure an equivalent physical displacement step size. The momentum coefficient, derived from a preset value, physically represents the inertial term used to accelerate convergence and overcome local minima; its value range is typically set to... Within the range, for example, the preferred value is This preferred value is based on the empirical value of the Nesterov momentum method, which can effectively suppress optimization oscillations while maintaining sufficient inertia to accelerate convergence;
[0093] This parameter simulates the damping characteristics in a physical environment; a larger value indicates better damping. A value signifies smaller environmental damping, which helps particles maintain their inertia and escape shallow local optima; particularly, in the first iteration of the algorithm, since there is no... The historical state at any given moment, the system sets the momentum term at that moment. It is a zero vector, meaning that the virtual particle is assumed to have an initial velocity of zero and begins to move only under the influence of the current net force;
[0094] The system repeats the above steps of calculating the resultant force vector and updating the Euler integral until the current iteration number reaches the evolution step number threshold T, and then outputs the target key point sequence.
[0095] This embodiment models the physical evolution process as a neural network layer with a fixed number of steps, making the entire system end-to-end differentiable. This means that the network can automatically learn the optimal energy weights through backpropagation without manual adjustment of physical parameters, greatly improving the model's adaptability to data with different degrees of deformation. Specifically, to ensure the numerical validity of physical parameters during the learning process and avoid negative energy weights that could cause the system to diverge, the system adjusts the energy weights before the parameters participate in the calculation. and update step size Applying the Softplus activation function Its value range is strictly constrained to be ;
[0096] Meanwhile, during the training phase, the system uses the Connection Temporal Classification (CTC) loss function as a supervision signal. This loss function calculates the alignment probability between the predicted character probability sequence and the real text label. Since the Euler integral and the interpolation process are differentiable, the gradient of the CTC loss with respect to the coordinate parameters can flow through the entire dynamic evolution layer through the backpropagation time-backpropagation (BPTT) algorithm, thereby dynamically adjusting the above physical parameters and optimizing the trajectory of the dynamic evolution in a direction that is conducive to the final character recognition.
[0097] Example 2:
[0098] The method also includes:
[0099] In the process of calculating the resultant force vector, the sequential elastic term and topological repulsive term of the virtual particle are aggregated through sparse matrix multiplication.
[0100] The parallel computing core of the graphics processing unit is used to accelerate sparse matrix multiplication operations.
[0101] This embodiment optimizes computational efficiency. Since the topological repulsion term involves interactions between pairs of particles, its computational complexity is theoretically high. To meet real-time requirements, this embodiment employs a sparse matrix acceleration strategy and specifically discloses how to map nonlinear physical forces into matrix multiplication operations. When calculating the resultant force vector f, the system does not directly perform a double loop traversal but instead constructs an adjacency matrix A. Here, the adjacency matrix A serves as a general computational container concept, instantiated into different sparse structures when processing different physical forces: when processing sequential elastic terms, matrix A is specifically represented as the dynamic stiffness matrix A_dyn, while when processing topological repulsion terms, matrix A is specifically represented as the sparse Laplace matrix L_rep, thus distinguishing the connection relationships under different force fields.
[0102] Specifically, for the sequential elasticity term, since it includes nonlinear distance constraints and angle constraints involving three-point coupling, directly using a common tridiagonal Laplacian matrix containing only first-order neighborhoods would lead to distortion of the physical model. Therefore, this embodiment adopts a dynamic stiffness matrix strategy based on Hessian local linearization: the system predefines a fixed banded sparse structure, specifically a pentagonal matrix with a bandwidth of 2, to accommodate the angular constraints introduced. to Second-order neighborhood dependency;
[0103] To address the black-box problem of transforming nonlinear angular forces into linear matrix multiplication, this embodiment clarifies the calculation path for the stiffness coefficient: In each iteration, for the angular potential energy term... The system analyzes and calculates its relation to the involved nodes. The Hessian matrix block is the second-order partial derivative matrix; specifically, for the angular potential term... Define vector and The formula for calculating the second-order partial derivative matrix block is as follows:
[0104]
[0105] For ease of engineering implementation, this embodiment uses a simplified first-order approximate stiffness formula:
[0106]
[0107] This approximation formula ignores the first derivative of the angle with respect to distance, retaining only the principal term of the second derivative of the angle with respect to the orthogonal components of the direction vector, in order to reduce computational complexity; where, Defined as dynamic stiffness matrix The Middle line, number The off-diagonal elements of a column represent nodes. With nodes The coupling strength between them;
[0108] Numerical stability constant, typically taking the value of Used to prevent angle Approaching or This leads to the disappearance of stiffness coefficients, resulting in matrix singularity; The curvature penalty coefficient is the same as defined above.
[0109] This approximation formula extracts the main contributing components on the main diagonal of the Hessian matrix, where the denominator term... The scale factor originates from the angle differential; to adapt to the scalar storage format of sparse matrices, the system uses the trace projection method to extract the equivalent stiffness: for example, for nodes... and Calculate the corresponding Hessian blocks for the coupling terms between them. negative mean of the trace As the instantaneous spring constant; similarly, the angular potential energy... and The indirect coupling produces the second-order neighborhood stiffness coefficient. These calculated dynamic coefficients are then filled into the sparse matrix. and and Index position; boundary conditions for the beginning and end of the sequence, i.e., when... At that time, since there is no complete second-order neighborhood to calculate the angular constraints, the system will involve the stiffness coefficient of the out-of-bounds index. Forced to Only first-order distance constraints are retained to prevent index overflow errors during sparse matrix construction.
[0110] In this process, to ensure sparse matrix operations It can accurately represent the physical restoring force, that is, the magnitude of the force is proportional to the relative displacement, and the direction is towards the equilibrium position. The construction follows the principle of constructing the Laplace matrix: the calculated positive stiffness coefficients are used... Fill to the corresponding off-diagonal element position, while forcing the main diagonal element to satisfy the zero-sum constraint: This mathematically guarantees that matrix multiplication is equivalent to calculating the resultant force vector. This step ensures that bending forces involving second-order neighborhood dependencies can be uniformly encapsulated within a sparse matrix multiplication framework; at this point, the aggregation of elastic forces is transformed into sparse matrix-matrix multiplication (SpMM). This allows us to leverage the hardware acceleration of sparse tensor operations using the GPU while maintaining physical accuracy.
[0111] For topological repulsion terms, their adjacency relationships are dynamic; the system utilizes spatial hashing or a KD-tree to only consider adjacency relationships within the cutoff radius. Calculate the repulsive force between particle pairs within the area; cutoff radius is used here. The value is determined based on the character spacing constraint. , usually set ,For example The selection criteria are related to the maximum expected curvature of the text lines and the text density. Experiments show that... Double-spaced characters typically cover the potential topological conflict regions of most adjacent characters and characters in both upstream and downstream lines; this numerical range encompasses local neighborhoods where topological entanglement may occur, effectively pruning long-distance ineffective interactions, and constructing a dynamic sparse Laplace matrix. The construction of this matrix follows the principles of graph signal processing: for particle pairs whose distances satisfy the conditions... The system calculates the equivalent repulsion stiffness coefficient based on the modified energy formula:
[0112]
[0113] in, For example, the numerical stability constant. Here, the negative power of distance is used to conform to the physical property that repulsive force decreases with increasing distance. To reuse the normalization term of the energy function and ensure consistency between the sparse approximation and the global energy definition, the discrete off-diagonal elements of the sparse matrix are set to... Simultaneously, perform an accumulation operation on the diagonal elements. At this point, the process of repulsive aggregation is also transformed into standard sparse matrix-matrix multiplication (SpMM): This formula is mathematically equivalent to calculating That is, all neighboring particles to the current particle The system calculates the repulsive resultant force vector and is fully compatible with the underlying optimization interface of existing deep learning frameworks such as cuSPARSE. The system uses sparse matrix multiplication (SpMM) to aggregate the forces acting on all particles at once. Based on this, the system uses the parallel computing cores of the graphics processing unit (GPU), such as CUDACores, to map the above sparse matrix operations onto the GPU's TensorCores for execution. Since the force calculation for each particle is independent, the GPU starts N threads in parallel to calculate the gradients of N particles simultaneously.
[0114] Based on the target key point sequence, spatial transformation correction is performed on the feature map, including:
[0115] The sequence of key target points is used as the control points for thin-plate spline interpolation transformation.
[0116] Calculate the transformation parameter matrix based on the control points;
[0117] By using a transformation parameter matrix, text regions in the feature map that undergo non-rigid deformation are mapped into flat rectangular regions, thus generating text features.
[0118] This embodiment details the spatial transformation correction process based on the target keypoint sequence. The system uses the target keypoint sequence P after dynamic evolution as the control point set for thin-plate spline interpolation transformation, i.e., the aforementioned related point set. Here, the term "control point set" is used uniformly to conform to the standard definition of the TPS algorithm. At the same time, a set of corresponding standard target control points P' is defined. To ensure that the deformed text features present a standard horizontal linear structure without destroying the original aspect ratio of the characters, the system adopts a physical step size locking strategy to construct the point set. ;
[0119] Specifically, the system sets a mapping step size. The step size The determination logic is as follows: set it to be consistent with the downsampling step size of the feature extraction network, such as 16 pixels, or adaptively set according to the average width of the characters in the feature map, for example... To maintain the aspect ratio of the characters; standard dot set In the diagram, the ordinates of all points The horizontal coordinate is fixed at half the height of the output feature map to ensure horizontal alignment, while the horizontal coordinate... According to the formula Linear arrangement is performed; correspondingly, the target region width of the TPS transform is implicitly defined as This definition ensures that each character is assigned a fixed physical width regardless of the length of the text sequence, thus avoiding non-affine distortions such as flattening or elongation that occur when texts of different lengths are forcibly stretched to a fixed-width feature map.
[0120] The system solves for the TPS transform parameter matrix based on the source control points and the target control points. In this step, to solve the logical closed-loop problem of inverse mapping sampling using the parameter matrix, the transform direction must be clearly defined; if calculated directly... The forward transformation will cause sampling holes in the output image; therefore, this embodiment adopts an inverse TPS solution strategy: the system uses the standard point set in the flat space Treat them as input control points and fill them into the matrix. and The target key points of the curled space Treat it as the target value and fill it into the right side of the equation;
[0121] The system constructs the following block-based linear equation system:
[0122]
[0123] in, This represents a 3x3 zero matrix used to satisfy the constraints of affine transformations.
[0124] For standard point set The calculated radial basis function matrix, its 6th... Line 1 Column elements Defined as:
[0125]
[0126] use As a kernel function;
[0127] in, : These are the standard target control point sets. The first in The and the first One control point;
[0128] : The preset numerical stability constant, with typical values. The lower limit of floating-point precision is used to prevent the number from being in the range of 10 ... An error occurred during logarithmic calculation;
[0129] Augmented coordinate matrix of standard point set ;
[0130] The inverse weight matrix can be obtained by performing LU decomposition or inversion on the symmetric indeterminate matrix. and affine coefficients The matrix This describes how to map coordinates from a flat space back to a curved space; using the transformation parameter matrix. Inverse mapping sampling is performed on the feature map F; specifically, for each integer pixel coordinate on the output feature map... Determine if its x-coordinate is less than If it is less than 1, substitute it into the inverse transformation formula to calculate its corresponding floating-point coordinates in the original graph. And use bilinear interpolation to obtain the feature value at that position; if it is greater than If the zero vector is filled, non-rigid deformation will occur, such as fan-shaped bending or wavy text regions, which will be forcibly mapped to flat rectangular regions to generate text features.
[0131] The method also includes:
[0132] Based on the spatial distribution of the target key point sequence, construct a coordinate mapping relationship;
[0133] By using coordinate mapping relationships, the original pixels of the document image are reverse-sampled to generate an enhanced document image that eliminates wrinkles and deformation.
[0134] This embodiment further proposes an image enhancement method for generating enhanced document images with wrinkles removed. The system constructs a dense pixel mapping field M from the original distorted image space to the corrected planar space based on the spatial distribution of the target keypoint sequence P. Specifically, constructing the coordinate mapping relationship does not directly reuse the forward transformation parameters, but requires solving the inverse mapping. The system uses the target control points in the standard flat space... Using this as the source point, the optimized target key point sequence As the target point, the transformation weight matrix of the inverse thin-plate spline interpolation is resolved. The system defines a pixel grid for the target flat image, such as a standard rectangular canvas. For each pixel coordinate in this grid... Substitute into the inverse transformation formula
[0135]
[0136] in, For radial basis functions, Inverse transformation weight matrix The row element, Affine coefficient matrix The amount, The target control point is used; its corresponding floating-point coordinates in the original curled image are calculated using this formula, thereby generating a dense mapping field M covering the entire image;
[0137] The system uses the mapping field M to reverse resample the original pixels, i.e., the RGB values, of the document image. Reverse resampling refers to the process of finding the corresponding pixel value in the source image and filling it in, starting from the pixel grid of the target image. The system outputs a visually flat enhanced document image.
[0138] This embodiment produces a virtual smoothing effect; even if the physical document is crumpled up, the system can generate a flat, scanned image. This not only assists machine recognition but also provides excellent visualization for human review, and has significant application value in scenarios requiring human-computer interaction, such as financial reimbursement audits.
[0139] Decoding text features using a sequence recognition network includes:
[0140] Text features are input into an attention-based decoder;
[0141] The decoder converts text features into a sequence of character probabilities.
[0142] The final text content is determined based on the character probability sequence.
[0143] This embodiment describes the specific decoding process of the sequence recognition network; the system inputs the corrected text features into the decoder; the decoder adopts an attention-based recurrent neural network (RNN) or Transformer structure; at each time step, the decoder calculates the attention weight between the current hidden state and the text features, focusing on the feature region of the current character to be recognized; the decoder outputs a character probability sequence S, where each element represents the probability distribution of the corresponding position belonging to each character category; the system determines the final text content from the probability sequence through a greedy search or bundle search algorithm;
[0144] This embodiment combines front-end dynamic correction, so that the attention decoder no longer needs to deal with complex spatial transformations, but only needs to focus on character shape discrimination. This decoupling design of correction and recognition greatly improves the overall recognition rate in complex scenarios and effectively avoids the attention drift problem caused by deformation.
[0145] The document image is a flexible media image, which includes thermal paper receipt images, fabric label images, or flexible packaging instruction manual images.
[0146] The character recognition results include the text content and its corresponding confidence score.
[0147] This embodiment defines the specific application scenario and output format of the method; the document images processed by this method specifically refer to flexible media images; flexible media refers to materials that lack rigid support and are easily subjected to external forces to undergo non-affine deformation, specifically including easily curled thermal paper receipts, easily folded and stretched fabric labels, or easily crumpled soft packaging instructions; at the same time, the system's output results not only include text content but also the corresponding confidence score value; this confidence score value is jointly determined by the final convergence value of the energy function and the output probability of the decoder; specifically, in order to comprehensively evaluate the quality of geometric correction and the determinism of character recognition, the confidence score... The calculation formula is defined as follows:
[0148]
[0149] in, The decoder output of the first The predicted probability of each character;
[0150] : The normalized average energy value at the end of the dynamic evolution. This represents the total number of key points; it should be noted that, in order to eliminate the influence of image resolution differences on the energy value dimensions, and to ensure the preset threshold is met... The universality here It is calculated based on a normalized coordinate system, or calculated in pixel coordinates and then explicitly divided by the square factor of the image scale. ;
[0151] : Preset energy tolerance threshold, typical values are as follows Based on the convergence energy of the positive samples in the training set Quantile setting; when the system average energy is higher than this value, it indicates a serious topological conflict or failure to converge to the character center;
[0152] Sensitivity coefficient, typical values are as follows: This is used to control the sensitivity of the confidence function to energy changes. The larger the value, the more severe the penalty for high-energy states; this formula ensures that the system outputs a high-confidence result only when the text recognition probability is high and the physical system is in a low-energy stable state, i.e., when geometric correction is successful.
[0153] This embodiment clearly defines the areas of advantage of this technology; for rigid cards, traditional methods are sufficient, but for the flexible media defined in this embodiment, the dynamic evolution method of this invention has irreplaceable advantages and can solve the pain point of traditional OCR failing to recognize due to media deformation in such scenarios.
[0154] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A method for character recognition in document images based on key point detection, characterized in that, include: The acquired document images are encoded using a feature extraction network to generate feature maps containing semantic information of the images. Based on the response intensity distribution of the feature map, an initial set of candidate key points is generated, and each key point in the set of candidate key points is defined as a virtual particle characterized by coordinate parameters. A dynamic energy function based on the feature map is constructed, wherein the dynamic energy function includes a data gravity term, a sequence elasticity term, and a topological repulsion term acting on the virtual particle; During the optimization process, the coordinate parameters of the candidate key point set are iteratively updated by minimizing the total energy function containing the data attraction term, the sequence elasticity term, and the topological repulsion term, so as to generate a target key point sequence in a stable equilibrium state. Based on the target key point sequence, the feature map is spatially transformed and corrected to generate aligned text features; The text features are decoded using a sequence recognition network, and the character recognition result corresponding to the document image is output. The construction of the dynamic energy function based on the feature map includes: Based on the pixel gradient information of the feature map, the data gravity term is determined, which is used to characterize the numerical attraction of the high-response region in the feature map to the virtual particle. Based on the relative distance between adjacent key points in the candidate key point set and the preset character spacing constraint, the sequence elasticity term is determined. The sequence elasticity term is used to characterize the geometric constraint force that constrains adjacent virtual particles to maintain a preset spacing and be collinear. Based on the Euclidean distance between any two key points in the candidate key point set, the topological repulsion term is determined. The topological repulsion term is used to characterize the numerical repulsion force that prevents the virtual particles from coinciding with each other.
2. The document image character recognition method based on key point detection according to claim 1, characterized in that, The iterative update of the coordinate parameters of the candidate keypoint set by minimizing the total energy function containing the data gravity term, the sequence elasticity term, and the topological repulsion term includes: The evolution process of the dynamic energy function is embedded in the forward propagation process of the neural network; If the current iteration step count has not reached the preset evolution step count threshold, calculate the negative gradient of the total energy function with respect to the virtual particle coordinate parameters to determine the resultant force vector of the virtual particle under the combined action of the data gravity term, the sequence elasticity term, and the topological repulsion term. Based on the resultant force vector, the coordinate parameters of the virtual particle are updated using the Euler integral algorithm until the current iteration step reaches the evolution step threshold, and the target key point sequence is output.
3. The document image character recognition method based on key point detection according to claim 2, characterized in that, The method further includes: In the process of calculating the resultant force vector, the sequential elastic term and the topological repulsive term experienced by the virtual particle are aggregated through sparse matrix multiplication. The sparse matrix multiplication operation is accelerated by utilizing the parallel computing core of the graphics processing unit.
4. The document image character recognition method based on key point detection according to claim 1, characterized in that, The spatial transformation correction of the feature map based on the target key point sequence includes: The target key point sequence is used as the control points for thin-plate spline interpolation transformation; Calculate the transformation parameter matrix based on the control points; The text features are generated by mapping the non-rigidly deformed text regions in the feature map to flat rectangular regions using the transformation parameter matrix.
5. The document image character recognition method based on key point detection according to claim 1, characterized in that, The method further includes: Based on the spatial distribution of the target key point sequence, a coordinate mapping relationship is constructed; The original pixels of the document image are reverse-resampled using the coordinate mapping relationship to generate an enhanced document image that eliminates wrinkles and deformation.
6. The document image character recognition method based on key point detection according to claim 1, characterized in that, Decoding the text features using a sequence recognition network includes: The text features are input into an attention-based decoder; The decoder converts the text features into a character probability sequence; The final text content is determined based on the character probability sequence.
7. The document image character recognition method based on key point detection according to claim 1, characterized in that, The document image is a flexible media image, which includes thermal paper receipt images, fabric label images, or flexible packaging instruction manual images. The character recognition result includes the text content and its corresponding confidence score.