Image-text recognition method and device based on deep learning and medium

Abstract

The invention discloses an image-text recognition method and device based on deep learning, and a medium, and relates to the technical field of image-text recognition, and the method comprises the steps: carrying out the scale normalization of a to-be-recognized image, calculating a gradient magnitude, obtaining a skeleton diagram, and carrying out the projection convergence in a width direction, and generating a time sequence stroke energy sequence; inputting the normalized image into a shared coding network to obtain a time sequence feature sequence, constructing a weight in a height dimension according to the skeleton graph, performing weighted aggregation on the feature graph, and obtaining an aggregation feature aligned with a width time sequence; constructing a CTC branch and an autoregression branch based on the aggregation features, and outputting an emission probability sequence, a character probability and attention distribution; determining an alignment path according to the emission probability sequence and acquiring a character time interval, executing constrained beam search by taking the character time interval and the time sequence stroke energy sequence as constraints, and outputting an identification text sequence by combining CTC sequence scores and autoregression scores. Timing alignment is enhanced and alignment drift is suppressed.

Description

Technical Field

[0001] This invention relates to the field of image and text recognition technology, and in particular to an image and text recognition method, device and medium based on deep learning. Background Technology

[0002] Image and text recognition technology is an important research direction in the fields of computer vision and pattern recognition. It is widely used in scenarios such as document digitization, invoice processing, natural scene text understanding, and information retrieval. With the development of deep learning, existing image and text recognition methods usually extract image features through convolutional neural networks and combine them with temporal modeling structures to map two-dimensional images into one-dimensional feature sequences. On this basis, connection-based temporal classification or autoregressive decoding is used to complete character recognition. Among them, the end-to-end recognition framework based on temporal features has become the mainstream research route in this field.

[0003] In complex graphic scenarios, the aforementioned conventional methods still face certain technical challenges in practical applications. On the one hand, when there are connected characters, overlapping characters, or characters that are broken across lines, it is difficult to stably model the correspondence between temporal features and the actual character positions, affecting the temporal alignment consistency during the decoding process. On the other hand, the recognition process based on a single decoding strategy lacks a constraint mechanism on stroke distribution and temporal structure during the generation process, leading to increased uncertainty in the temporal selection and character generation of candidate sequences. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a deep learning-based image and text recognition method to solve the problems of insufficient character temporal alignment stability and lack of effective constraints in the decoding process in existing technologies.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a deep learning-based image-text recognition method, comprising: acquiring an image to be recognized and performing scale normalization; calculating gradient magnitude and performing closure operation and thinning to obtain a skeleton map; projecting and converging along the width direction to generate a temporal stroke energy sequence; inputting the normalized image into a shared coding network to obtain a temporal feature sequence; constructing weights in the height dimension based on the skeleton map and performing weighted aggregation on the feature map to obtain aggregated features aligned with the width temporal sequence; constructing a CTC branch and an autoregressive branch based on the aggregated features, wherein the CTC branch outputs an emission probability sequence and the autoregressive branch outputs character probabilities and forms an attention distribution; determining the alignment path from the emission probability sequence and obtaining the character time interval; using the character time interval and the temporal stroke energy sequence as constraints to perform a constrained bundle search on the autoregressive branch; and combining the CTC sequence score and the autoregressive score to output a recognized text sequence.

[0007] As a preferred embodiment of the deep learning-based image recognition method of the present invention, the scale normalization includes scaling the image to be recognized according to the target height while keeping the aspect ratio of the original image unchanged, and performing scale cropping processing on the width direction of the image according to the relationship between the width of the scaled image and the fixed width to obtain the scale-normalized image.

[0008] As a preferred embodiment of the deep learning-based image and text recognition method of the present invention, the step of obtaining the skeleton image includes: converting the scale-normalized image into a grayscale image; calculating the horizontal and vertical gradients of the grayscale image respectively, and calculating the gradient magnitude based on the horizontal and vertical gradients; performing morphological closing operation on the gradient magnitude after binarization to obtain the binary image after closing operation; and obtaining the skeleton image by thinning the binary image after closing operation.

[0009] As a preferred embodiment of the deep learning-based image and text recognition method of the present invention, the step of generating a temporal stroke energy sequence by projection and convergence along the width direction includes: projecting the skeleton image in the height direction to obtain a skeleton density sequence in the width direction; and based on the skeleton density sequence, segmenting and converging the skeleton density sequence according to the interval length corresponding to the downsampling factor of the shared coding network in the width direction to generate a temporal stroke energy sequence aligned with the width temporal sequence.

[0010] As a preferred embodiment of the deep learning-based image and text recognition method of the present invention, the shared coding network includes a convolutional feature extraction sub-network that downsamples along the width direction and a temporal context modeling sub-network that establishes contextual dependencies in the width temporal dimension; the convolutional feature extraction sub-network outputs a two-dimensional feature map; the temporal context modeling sub-network performs temporal modeling on the two-dimensional feature map in the width direction to obtain a temporal feature sequence.

[0011] As a preferred embodiment of the deep learning-based image and text recognition method of the present invention, the step of constructing weights based on the skeleton map in the height dimension includes: mapping the skeleton map to a mapped skeleton map corresponding to the spatial scale of the encoded feature map; constructing height weights based on the response intensity of the mapped skeleton map in the height direction; and performing a weighted summation of the encoded feature map in the height dimension based on the height weights to obtain aggregated features that are temporally aligned with the width.

[0012] As a preferred embodiment of the deep learning-based image and text recognition method of the present invention, wherein: the CTC branch outputs the emission probability distribution containing the whitespace symbol at each width temporal position based on the aggregated features; and the autoregressive branch, under the condition that the character prefix has been generated, gradually outputs the conditional probability distribution of the corresponding character by performing attention selection on the aggregated features.

[0013] As a preferred embodiment of the deep learning-based image and text recognition method of the present invention, the constrained bundle search includes: determining the time interval of the character in the width temporal sequence based on the emission probability sequence output by the CTC branch; based on the time interval of the character in the width temporal sequence, during the bundle search process, according to the attention distribution formed by the autoregressive branch, combined with the character time interval and the temporal stroke energy sequence, applying a consistency penalty to the candidate sequence; based on the consistency penalty, and on the basis of the combined CTC score and the autoregressive score, selecting the candidate sequence with the highest comprehensive score as the recognized text sequence.

[0014] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the deep learning-based image and text recognition method described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the deep learning-based image and text recognition method described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By combining a high-dimensional weight construction mechanism based on skeleton graphs in a shared coding network and performing high-dimensional weighted aggregation on feature maps, it is possible to effectively convert two-dimensional image features into aggregated feature sequences that are temporally aligned with width without relying on character boundary segmentation, thereby enhancing the correspondence between temporal features and real text structures; by determining the character time interval based on the CTC emission probability sequence and combining the character time interval with the temporal stroke energy sequence as a constraint term in the autoregressive bundle search process, explicit constraints on autoregressive attention alignment behavior are achieved, reducing the probability of temporal offset and alignment drift during character generation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a deep learning-based image and text recognition method.

[0019] Figure 2 The flowchart for generating the skeleton diagram.

[0020] Figure 3A flowchart for generating aggregated features for shared coding networks and temporal alignment.

[0021] Figure 4 This is a flowchart of a constrained search.

[0022] Figure 5 This is a CER curve.

[0023] Figure 6 The CDF plot shows the length of the character time interval. Detailed Implementation

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0025] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0026] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0027] Reference Figures 1-6 This is one embodiment of the present invention, which provides a deep learning-based image and text recognition method, including the following steps: S1. Obtain the image to be recognized, perform scale normalization, calculate the gradient magnitude and perform closing operation and thinning to obtain the skeleton map, and project and converge along the width direction to generate a temporal stroke energy sequence.

[0028] Further, the image to be identified is obtained, denoted as: ; in, Indicates the original height. Indicates the original width. Indicates the number of channels.

[0029] Preferably, It can be the number of RGB channels.

[0030] Set height and width The height and width are positive integers used to standardize the network input size, ensuring the aspect ratio of the image to be recognized is maintained. Scale the image so that the height after scaling is... .

[0031] The scaling factor is the height. and original height The ratio of .

[0032] The scaled width is the product of the scaling ratio and the original width, rounded down.

[0033] The scaled image is normalized to a fixed width to obtain a scale-normalized image. .

[0034] Specifically, the normalization process involves cropping the width direction when the scaled width is not less than the fixed width, retaining the first fixed width column of pixels; and padding the width direction to the fixed width when the scaled width is less than the fixed width, with the padding pixels taking a fixed fill value, which can generally be zero or the average value of the image background.

[0035] It should be noted that the fixed width is determined by the maximum possible width of the image to be recognized after scale normalization and the downsampling structure of the subsequent shared coding network in the width direction, and is set to a width that meets the requirement of being an integer multiple of the downsampling structure.

[0036] Furthermore, the normalized image is converted into a grayscale image through a weighted linear combination. .

[0037] Using convolution kernel , Calculate the horizontal and vertical gradients of a grayscale image. The convolution kernel can be a Sobel kernel or an equivalent difference kernel. The resulting horizontal and vertical gradient maps of the grayscale image are represented as follows: ; ; in, Represents grayscale image The gradient response plot in the horizontal direction, Represents grayscale image The gradient response map in the vertical direction, This represents the convolution kernel used to calculate the gradient in the horizontal direction. This represents the convolution kernel used to calculate the gradient in the vertical direction. This represents a two-dimensional convolution operator.

[0038] Based on the acquired horizontal and vertical gradient maps, the gradient magnitude map is calculated. Then, normalize the gradient magnitude map to obtain the normalized gradient magnitude map. .

[0039] Furthermore, the normalized gradient magnitude map is binarized to form an edge binary map. .

[0040] Perform morphological closing operations on the edge binary image to obtain the closed binary image. , represented as: ; in, Represents the dilation operator, Represents the erosion operator. Represents a structural element.

[0041] The structural element can be cross-shaped, square, or circular, and its size is determined by the scale normalization process of the image to be recognized.

[0042] Thinning is performed on the binary image after the closing operation to obtain the skeleton image. The thinning algorithm employs an iterative method that removes boundary points while maintaining connectivity, such as the Zhang-Suen thinning.

[0043] Furthermore, the skeleton map is projected along the height direction to obtain the skeleton density sequence in the width direction. , represented as: ; in, The width-direction skeleton density sequence is represented in the first... The value of each width position, Indicates the length direction index. Indicates the location of the skeleton diagram The pixel value at that location.

[0044] Set the downsampling factor to The downsampling factor is consistent with the total downsampling factor in the width direction of the subsequent shared coding network to ensure timing alignment.

[0045] Based on the downsampling factor, the timing length is defined as the ratio of the width to the downsampling factor.

[0046] The skeleton density sequence is aggregated into non-overlapping intervals of a length equal to a downsampling factor to obtain the temporal stroke energy sequence. , represented as: ; in, Indicates the first The stroke energy value corresponding to each temporal position. Indicates the downsampling factor. Indicates timing length, The width-direction skeleton density sequence is represented in the first... The value of each width position.

[0047] It should be noted that the width direction refers to the horizontal direction of the image, that is, the direction from left to right along the column coordinates of the image; the length direction refers to the vertical direction of the image, that is, the direction from top to bottom along the row coordinates of the image.

[0048] It should also be noted that by performing scale normalization on the image to be recognized and extracting the skeleton map, a temporal stroke energy sequence consistent with the encoding temporal sequence is generated along the width direction. This achieves a stable characterization of the stroke distribution of characters when there are connected characters, overlapping characters, or overlapping strokes, providing reliable structural prior information for subsequent temporal modeling and decoding.

[0049] S2. Input the normalized image into the shared coding network to obtain the temporal feature sequence. Construct weights in the height dimension based on the skeleton map and perform weighted aggregation on the feature map to obtain aggregated features aligned with the width temporal sequence.

[0050] Furthermore, a shared coding network is constructed, which includes a convolutional feature extraction subnetwork and a temporal context modeling subnetwork. The convolutional feature extraction subnetwork is used to extract two-dimensional feature maps from the image and perform downsampling in the width direction, while the temporal context modeling subnetwork is used to establish long-range dependencies in the width direction.

[0051] Based on the constructed shared coding network, the normalized image is... Input the convolutional feature extraction subnetwork to obtain a two-dimensional feature map. To ensure that the total downsampling factor of the convolutional subnetwork in the width direction is consistent with the downsampling factor of the generated temporal stroke energy sequence.

[0052] The two-dimensional feature map is unfolded into a sequence in the width dimension and input into the temporal context modeling subnetwork to obtain the enhanced two-dimensional feature map. .

[0053] Furthermore, to enable the skeleton information to be used to constrain the high-level aggregation of encoded features, the skeleton map is... Mapping to the spatial scale of encoded features to obtain the mapping skeleton map Specifically, the skeleton diagram is scaled according to the height scaling factor. and width scaling factor Divide into blocks, each block corresponding to an output position. The average value of the pixels within the block is represented as: ; in, This indicates that the mapping skeleton diagram is at a certain height. Timing position The value at that location, This indicates the mapping to height in the skeleton diagram. Timing position A set of pixels.

[0054] Based on the mapping skeleton graph, the mapping skeleton graph is normalized in the height dimension to construct normalized weights and obtain the height weight vector for each time position.

[0055] Specifically, normalization involves suppressing the influence of non-skeleton regions on aggregation through skeleton gating. The gating factor is defined as follows: ; in, Indicates the gating factor at high Timing position The value at that location, This represents the Sigmoid function. The slope coefficient, This is the bias threshold.

[0056] Subsequently, the gating factor and softmax weights are multiplied and then normalized to obtain the final height weight vector.

[0057] It should be noted that the slope coefficient is determined by setting the slope coefficient as a gating function. A positive constant that rapidly transitions from 0 to 1 within the transition range of the skeleton response from low to high, with a value range of [5,20]; the bias threshold, which is set to a fixed quantile threshold of the mapped skeleton map, so that height positions above the bias threshold are preferentially gated and opened. Generally, the bias threshold is the 70th quantile of all pixel values ​​in the mapped skeleton map, and its value range is usually [0,1].

[0058] Furthermore, the two-dimensional feature map is subjected to height-dimensional weighted aggregation based on the height weight vector to obtain the aggregated feature sequence.

[0059] It should be noted that the shared coding network consists of a convolutional feature extraction subnetwork and a temporal context modeling subnetwork connected in series. The input is a normalized image. First, the convolutional subnetwork obtains a two-dimensional feature map. Then, the two-dimensional feature map is organized into a temporal sequence along the width direction and modeled by the temporal subnetwork to obtain another two-dimensional feature map. Subsequently, a height weight is constructed in the height dimension by combining the mapped skeleton map. Based on the height weight, the two-dimensional feature map is weighted and aggregated in the height dimension to obtain aggregated features aligned with the width temporal sequence. The total downsampling factor in the width direction is consistent with the downsampling factor of the generated temporal stroke energy sequence.

[0060] It should be noted that the convolutional feature extraction subnetwork adopts a staged two-dimensional convolutional structure. Each stage includes a convolution operator, a normalization operator, and a nonlinear activation operator. Between stages, downsampling is performed in the width direction and compression is performed in the height direction through stride convolution, with the output size being... The two-dimensional feature map is used. In order to take into account both fine-grained strokes and the morphology of connected characters, the convolutional layer uses stacked small convolutional kernels, such as multiple small kernels replacing large kernels and maintaining gradient stability through residual connections in several stages. In the last one or two stages, dilated convolution is used to cover the connected stroke area across characters. However, the kernel size, number of stages and downsampling position are all preset as structural parameters of the convolutional feature extraction sub-network and remain unchanged during training and inference.

[0061] It should be noted that the temporal context modeling subnetwork uses the temporal index in the width direction as the main axis to perform context fusion on the two-dimensional feature map. It employs a multi-layer self-attention encoding structure, adding positional encoding to the feature vector at each temporal position, and then sequentially passing it through a multi-head self-attention sub-layer and a feedforward sub-layer. Normalization is applied after each sub-layer to form a stable deep representation. Specifically, self-attention explicitly establishes long-range dependencies in the temporal dimension, enabling the model to utilize adjacent and distant stroke contexts to resolve local ambiguities caused by ligatures or connections, ultimately maintaining the output as... In order to achieve highly integrated docking with the skeleton-guided framework.

[0062] It should also be noted that by introducing a high-dimensional weight construction mechanism based on skeleton graphs into the shared coding network and performing high-dimensional weighted aggregation on the feature maps, the two-dimensional image features can be effectively converted into an aggregated feature sequence aligned with the width temporal sequence without relying on character boundary segmentation, thereby enhancing the correspondence between temporal features and the real text structure.

[0063] S3. Based on the aggregated features, construct the CTC branch and the autoregressive branch respectively. The CTC branch outputs the emission probability sequence, and the autoregressive branch outputs the character probability and forms the attention distribution.

[0064] Furthermore, the aggregated feature sequence is arranged temporally along the width direction, with each temporal position corresponding to a feature vector.

[0065] Establish a set of identification symbols to limit the range of characters in the output. The set of identification symbols includes character symbols required by the business scenario, and also includes feature tags when it is necessary to express structural semantics such as word breaks and line breaks.

[0066] In addition to the set of identification symbols, blank symbols are set, and the blank symbols and the set of identification symbols are merged to form an extended symbol set.

[0067] The set of recognized symbols is used as the output space of the autoregressive branch, and the set of extended symbols is used as the output space of the connection time classification (CTC) branch.

[0068] Furthermore, based on the aggregated feature sequence, a connection-time classification (CTC) branch is constructed. Specifically, the aggregated feature sequence is input into the CTC branch, which is used to output the corresponding symbol probability distribution for each temporal position without relying on character segmentation.

[0069] The CTC branch includes a network layer for temporal feature transformation and a linear mapping layer for symbol mapping. The linear mapping layer maps the feature vector of each temporal position to the dimensional space of the extended symbol set and generates the logit vector corresponding to the temporal position. Then, the logit vector is normalized to obtain the emission probability of each symbol in the extended symbol set. Finally, the emission probabilities of all temporal positions are sorted and combined in temporal order to obtain the emission probability sequence containing the whitespace symbol.

[0070] Furthermore, while constructing the CTC branch, an autoregressive branch is constructed in parallel based on the same aggregated feature sequence; the autoregressive branch takes the generated character prefix and aggregated feature sequence as input and outputs the recognized character in a step-by-step manner.

[0071] Specifically, at the start of generation, a starting symbol is input to the autoregressive branch. Subsequently, in each generation step, the currently generated character prefix is ​​mapped to an embedding vector and input into the decoding structure for state updates. During the update process, the decoding structure first uses a self-attention mechanism to model the generated character prefixes to characterize the contextual dependencies within the character sequence. Simultaneously, a cross-attention mechanism is used to select temporal features related to the current generation position from the aggregated feature sequence. The cross-attention mechanism calculates the correlation between the current decoding state and the features at each temporal position, forming an attention distribution in the width temporal dimension. Based on the attention distribution, the aggregated feature sequence is weighted and converged to obtain the contextual features corresponding to the current generation step. Subsequently, the contextual features and the current decoding state are fused and mapped to a character probability distribution to obtain the generation probability of each candidate character in the current generation step. This process is repeated until a termination symbol is generated. The autoregressive branch outputs the character probability sequence sequentially and synchronously forms an attention distribution corresponding to each generation step throughout the entire generation process, which is used to characterize the correspondence between the current character and each temporal position in the aggregated feature sequence.

[0072] Furthermore, to facilitate the constraint and coordinated processing of the temporal correspondence between the CTC branch and the autoregressive branch, after the CTC branch emission probability sequence is output, non-blank confidence information with width temporal alignment is constructed based on the emission probability sequence.

[0073] Specifically, the probability values ​​corresponding to blank symbols are extracted from the symbol probability distribution output by the CTC branch at each temporal position, and the probability values ​​of blank symbols are used as the confidence level that the temporal position does not contain valid character information. Subsequently, the probability of blank symbols is reverse-mapped to obtain the non-blank confidence level indicating that the temporal position belongs to the valid text region. Thus, without relying on character boundary labels, a confidence sequence arranged along the width temporal sequence is constructed to characterize the degree of correlation between each temporal position and the actual text content. The non-blank confidence sequence, as an additional output of the CTC branch, can be used in conjunction with the formed attention distribution and temporal stroke energy information.

[0074] It should be noted that the training process of the CTC branch specifically involves the following steps: During the training phase, the aggregated feature sequence is input into the CTC branch, and the emission probability sequence containing whitespace characters is obtained through linear mapping and normalization. The target text sequence corresponding to the training samples is used as the supervision signal, and forward and backward dynamic programming of connection-series classification is used to calculate the total probability of the target text on all feasible alignment paths. The negative logarithm of the total probability is used as the CTC loss. The parameters of the shared coding network and the CTC branch are updated through backpropagation.

[0075] It should be noted that the training process of the autoregressive branch specifically involves the following steps during the training phase: the same aggregated feature sequence is input into the autoregressive branch, and a teacher-mandated approach is used to use the prefix of the target text sequence as the input for each step, enabling the decoder to progressively output the conditional probability distribution of the next character under the cross-attention mechanism; the actual next character of the target text sequence is used as the supervision signal, the progressive cross-entropy loss is calculated, and the losses at each step are summed to form the autoregressive loss; the parameters of the shared coding network and the autoregressive branch are updated through backpropagation.

[0076] It should also be noted that the training order is to first train the shared encoder network and the CTC branch, and then freeze the shared encoder network to train the autoregressive branch.

[0077] It should also be noted that by constructing CTC branches and autoregressive branches in parallel based on the same aggregated features, and outputting emission probability sequences, character conditional probabilities, and attention distributions respectively, collaborative modeling of temporal classification information and sequence generation information is achieved, enabling the decoding process to simultaneously possess overall temporal consistency and local generation flexibility.

[0078] S4. Determine the alignment path from the emission probability sequence and obtain the character time interval. Use the character time interval and the temporal stroke energy sequence as constraints to perform a constrained bundle search on the autoregressive branch. Combine the CTC sequence score and the autoregressive score to output the recognized text sequence.

[0079] Furthermore, the emission probability sequence output by the CTC branch is obtained, and the emission probability sequence sequentially gives the probability distribution of each symbol in the extended symbol set according to the temporal position.

[0080] For each time position, the symbol with the highest emission probability is selected as the greedy symbol for the time position, thereby obtaining a greedy symbol sequence with a length equal to the time length.

[0081] Perform a folding process on the greedy symbol sequence, where the folding process includes deleting adjacent duplicate symbols and deleting whitespace characters to obtain an initial character sequence that does not contain whitespace characters.

[0082] Based on the initial character sequence, a CTC extended label sequence is constructed, which involves inserting whitespace characters between adjacent characters and appending whitespace characters at the beginning and end of the sequence to form an extended label sequence in which whitespace characters and characters alternate. The extended label sequence is used to express the allowed pause, advance, and jump relationships of CTC in dynamic programming.

[0083] With the extended label sequence as the target and the emission probability sequence as the observation, dynamic programming is performed according to the legal transition rules of CTC. The legal transition rules include allowing staying at the current label position, moving to the next label position, and jumping to the next label position after that under the condition of legality. The maximum cumulative probability of reaching each extended label position at each time position is calculated.

[0084] After completing all time-series position calculations, a path with the highest probability, i.e., the alignment path, is obtained by backtracking.

[0085] It should be noted that the legal transfer rules of CTC are determined based on the restricted incrementing method of the extended tag sequence index. Specifically, the target sequence is first constructed into an extended tag sequence by alternating whitespace and characters, and it is assumed that the current time is at the extended index. Then, in the next moment, only items at the same index are allowed. , Advance to adjacent index Or jump to when the conditions are met. Among them, jump to Two constraints must be satisfied: first, the skipped... The second is the position of the inserted whitespace character, and the third is the target extended sequence. and The corresponding character labels are different.

[0086] Furthermore, based on the constructed extended label sequence, the corresponding position index of each character in the extended label sequence is determined. The position index corresponds one-to-one with the order of the characters in the output sequence, which is used to locate the set of temporal positions corresponding to the characters in the alignment path.

[0087] The alignment path is traversed sequentially to find all temporal positions in the path that have the same extended label index as the character. The traversal process is a sequential scan, which can determine all positions in the temporal dimension where the character is selected by the path.

[0088] For each character, the earliest position among all found temporal positions is determined as the start position of the character's time interval, and the latest position is determined as the end position of the character's time interval, thus forming a continuous time interval to characterize the temporal coverage of the character.

[0089] In rare cases, the character is not assigned to any timing position in the alignment path. To address this abnormal situation, the effective time interval of the current character is obtained by selecting the timing position with the highest emission probability for the character from all timing positions and using this timing position as both the start and end position of the character. The timing fallback strategy satisfies the monotonically increasing timing constraint.

[0090] For each character in the output character sequence, determine the corresponding time interval according to the time interval determination process, and summarize the time intervals of all characters to form a character time interval set. The character time interval set is sorted according to the character order.

[0091] Furthermore, obtain the set of output character time intervals, where each character corresponds to a start position and an end position in the time dimension; obtain the time-series stroke energy sequence.

[0092] For each character's time interval, determine the center position and range scale of the time interval, represented as: ; ; in, Indicates the first Character time interval The center position of the interval, Indicates the first The interval range scale of a time interval for a given character. Indicates the first The starting time of each word in the time dimension Indicates the first The termination time of a word in the temporal dimension.

[0093] During the beam search generation process, each character generated by the autoregressive branch produces a corresponding attention distribution, which gives the weights for each temporal position in the temporal dimension.

[0094] Based on the attention distribution, the attention focus position is calculated to represent the temporal position of the currently generated character. The attention focus position is obtained by weighted summation of the attention distribution.

[0095] The location of attention focus is represented as: ; in, Indicates the first The attention focus position corresponding to each character Indicates the autoregressive branch at the th The character pair in the encoded segment Attention at each temporal position.

[0096] The attention focus position is compared with the time interval corresponding to the character. If the attention focus position falls within the time interval, it is considered to meet the interval consistency requirement. If the attention focus position exceeds the time interval range, a penalty value is generated that increases with the degree of deviation. The penalty value is used as the interval consistency constraint term and is used to deduct points from the candidate sequence that does not meet the time interval constraint in the candidate score of the bundle search.

[0097] The interval consistency constraint penalty is expressed as: ; in, Indicates the first The interval consistency constraint penalty value corresponding to each character This represents a very small non-zero value.

[0098] The cross-attention distribution and temporal stroke energy sequence corresponding to the current character are weighted and combined to obtain the stroke energy value of the attention coverage area. The stroke energy value of the attention coverage area is then compared with the energy threshold.

[0099] The stroke energy value of the area covered by attention is represented as: ; in, Indicates the first The stroke energy value of the attention coverage area corresponding to each character. Indicates the first The stroke energy value corresponding to each time sequence position.

[0100] If the stroke energy value is not less than the energy threshold, the attention coverage area is considered to have sufficient stroke support; if the stroke energy is less than the energy threshold, a penalty value is generated, and the penalty value increases with the degree of insufficiency.

[0101] During the beam search generation process, for each step of the candidate sequence expansion, the interval consistency constraint term and the stroke energy consistency constraint term generated in the expansion step are calculated, and the two types of constraint terms are weighted by a fixed weight and added to the cumulative penalty of the candidate sequence to form the cumulative constraint value of the candidate sequence.

[0102] The stroke energy consistency penalty is represented as follows: ; in, Indicates the first The stroke energy consistency penalty value of each character. This indicates the energy threshold.

[0103] It should be noted that the energy threshold is determined by the statistical distribution of the temporal stroke energy sequence of the training samples across all samples and all temporal positions. For example, the mean of the statistical distribution minus the standard deviation is taken as a fixed threshold and kept unchanged during the inference stage. The value range is usually [0,1]. The fixed weight is determined by jointly tuning the recognition performance on the validation set, so that the recognition result corresponding to the comprehensive scoring function in different scenarios reaches the optimal value and is then fixed.

[0104] Furthermore, using the start symbol as the initial input, the candidate sequence set for the beam search is initialized, and an initial cumulative autoregressive score, cumulative CTC score, and cumulative constraint penalty value are set for each candidate sequence.

[0105] At each generation step of the beam search, the autoregressive branch outputs the conditional probability distribution of the next symbol based on the prefix and aggregated feature sequence of the current candidate sequence. According to the conditional probability distribution, several extended candidates are generated for each candidate sequence to form a new set of candidate sequences.

[0106] For each candidate sequence, the log probabilities corresponding to the newly generated symbols are accumulated according to the conditional probabilities output by the autoregressive branch to obtain the cumulative autoregressive score of the candidate sequence, expressed as: ; in, Representing candidate sequences Cumulative autoregressive score under the autoregressive branch This represents the candidate character sequence generated through beam search. Indicates the autoregressive branch at the th In the character generation step, the already generated prefix and aggregated feature sequences Generate characters under the condition The conditional probability.

[0107] For each candidate sequence, the candidate sequence is treated as the target character sequence. Based on the emission probability sequence output by the CTC branch, the probability of the candidate sequence is calculated using the CTC forward computation method, and the logarithm is taken as the CTC sequence score, expressed as: ; in, Representing candidate sequences CTC sequence score under the CTC branch This represents the emission probability sequence output by the CTC branch.

[0108] For each candidate sequence, based on the attention distribution corresponding to the current generation step, and combining the character time interval and the temporal stroke energy sequence, calculate the interval consistency constraint penalty and the stroke energy consistency constraint penalty, and accumulate the penalty value into the cumulative constraint penalty of the candidate sequence, as shown below: ; in, Representing candidate sequences The constraint penalty value accumulated during the generation process.

[0109] For each candidate sequence, a comprehensive score is constructed based on the cumulative autoregressive score, cumulative CTC score, and cumulative constraint penalty of the candidate sequence. Then, all candidate sequences are sorted according to the comprehensive score, and only candidate sequences with high scores are retained.

[0110] The overall score is expressed as follows: ; in, Representing candidate sequences The overall score, This represents the fusion coefficient, with a value range of [0,1].

[0111] It should be noted that the fusion coefficient is determined by progressively searching the recognition indicators on the validation set to achieve the optimal final recognition result under the comprehensive scoring function. Once determined, it is fixed as the inference configuration parameter.

[0112] After the beam search is completed, the candidate sequence with the highest comprehensive score is selected from the final candidate sequence set as the final recognized text sequence output.

[0113] It should also be noted that by determining the character time interval based on the CTC emission probability sequence and introducing the character time interval and the temporal stroke energy sequence together as constraints into the autoregressive bundle search process, an explicit constraint on the autoregressive attention alignment behavior is achieved, reducing the probability of temporal offset and alignment drift during character generation.

[0114] It should also be noted that by constructing a comprehensive scoring function by combining CTC sequence scores, autoregressive sequence scores, and consistency constraint penalties during the bundle search process, and sorting and filtering candidate sequences, a stable selection of recognition results is achieved in multi-candidate decoding scenarios, thereby improving the reliability of the final recognized text in terms of temporal consistency and structural rationality.

[0115] In this embodiment, to verify the determination of character time intervals based on CTC emission probability sequences and the use of character time intervals as constraints in the autoregressive bundle search process to achieve explicit constraints on autoregressive attention alignment behavior, this is conducted in a simulation environment of character spacing compression perturbation. Using the same batch of text lines as the baseline input, i.e., through a shared encoding network and fixed training weights, character spacing compression is applied to the images to simulate the width temporal distortion caused by character adhesion and local compression common in scenarios such as invoices, waybills, and contracts. The character spacing scaling factor is gradually reduced from 1.00 to 0.60, indicating a gradual increase in perturbation intensity. Under each perturbation intensity, three decoding settings are run: G0 without character time interval consistency constraints, G1 with character time intervals determined by CTC emission probability sequences and character time interval consistency constraints enabled in the constrained bundle search, and G2 with an interval replacement strategy and consistency constraints enabled. The corresponding CER is output.

[0116] like Figure 5 The figure shows the CER curve under the character spacing compression perturbation scenario. The horizontal axis represents the perturbation intensity (character spacing scaling factor, 1.00 indicates no compression, 0.60 indicates strong compression), and the vertical axis represents the character error rate (CER). The three curves in the same figure correspond to groups G0, G1, and G2, respectively. G0 is the control group, G1 is the invention group where the character time interval is determined by the CTC emission probability sequence and the character time interval consistency constraint is enabled in the constrained bundle search, and G2 is the alternative group which adopts a weaker interval strategy and enables consistency constraints. As the character spacing scaling factor decreases, text stickiness... The concatenation and width temporal distortion are aggravated, and autoregressive attention is more prone to alignment drift such as backsliding and crossing, thus increasing the overall CER. The curves show that G1 maintains the lowest CER under all perturbation intensities, and its advantage is more significant in the strong perturbation range (0.70~0.60), indicating that the character time interval as an explicit constraint term can effectively suppress autoregressive attention alignment shift, reduce the probability of missing characters, extra characters, and misalignment errors, and achieve higher robustness to complex imaging conditions. The performance of G2 is between that of G0 and G1, indicating that the quality of the interval source has a substantial impact on the constraint effect.

[0117] like Figure 6 The image shown is a CDF plot of the character time interval length. Figure 6The statistical results of character time interval length under the condition of a character spacing scaling factor of 0.60 (strong compression) are used to explain the alignment stability of constrained bundle search from a structural level. The horizontal axis is the average length of the character time interval, and the vertical axis is the cumulative distribution (CDF). The three curves in the same figure correspond to the three groups G0, G1, and G2, respectively. The more the curve is to the left, the shorter the overall interval length and the more concentrated the distribution. The steeper the curve, the smaller the fluctuation of the interval length and the stronger the stability. Under strong perturbation, if the character time interval is too long or the dispersion is too large, it means that the alignment path is not clear enough. During decoding, multiple characters are prone to sharing the same temporal region or overlapping intervals, which can induce autoregressive attention alignment drift. In the figure, the CDF of G1 is more to the left and rises faster, indicating that its character time interval length is more concentrated and more stable, which can provide clearer temporal range constraints for constrained bundle search, thereby improving attention alignment consistency and reducing the risk of temporal offset. G2 is better than G0 but not as good as G1, which further supports the necessity and effectiveness of determining the character time interval based on the CTC emission probability sequence.

[0118] This embodiment also provides a computer device applicable to the deep learning-based image and text recognition method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the deep learning-based image and text recognition method proposed in the above embodiment.

[0119] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0120] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the deep learning-based image and text recognition method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0121] In summary, this invention, by combining a high-dimensional weight construction mechanism based on skeleton graphs in a shared coding network and performing high-dimensional weighted aggregation on feature maps, effectively converts two-dimensional image features into aggregated feature sequences aligned with width temporally, without relying on character boundary segmentation. This enhances the correspondence between temporal features and the real text structure. Furthermore, by determining the character time interval based on the CTC emission probability sequence and combining the character time interval with the temporal stroke energy sequence as constraints in the autoregressive bundle search process, explicit constraints on autoregressive attention alignment behavior are achieved, reducing the probability of temporal offset and alignment drift during character generation.

[0122] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A deep learning-based image and text recognition method, characterized in that: include, The image to be identified is obtained, scale normalized, gradient magnitude is calculated and thinned by closing operation, skeleton map is obtained, and the skeleton map is projected and converged along the width direction to generate a temporal stroke energy sequence. The normalized image is input into the shared coding network to obtain the temporal feature sequence. Weights are constructed in the height dimension based on the skeleton map, and the feature maps are weighted and aggregated to obtain aggregated features that are aligned with the width temporal sequence. Based on the aggregated features, CTC branch and autoregressive branch are constructed respectively. CTC branch outputs emission probability sequence, and autoregressive branch outputs character probability and forms attention distribution. The alignment path is determined by the emission probability sequence, and the character time interval is obtained. The character time interval and the temporal stroke energy sequence are used as constraints to perform a constrained bundle search on the autoregressive branch. The CTC sequence score and the autoregressive score are combined to output the recognized text sequence.

2. The deep learning-based image and text recognition method as described in claim 1, characterized in that: The scale normalization includes scaling the image to be recognized according to the target height while keeping the aspect ratio of the original image unchanged, and performing scale cropping on the width direction of the image according to the relationship between the width of the scaled image and the fixed width to obtain the scale-normalized image.

3. The deep learning-based image and text recognition method as described in claim 2, characterized in that: The process of obtaining the skeleton image includes converting the scale-normalized image into a grayscale image. Calculate the horizontal and vertical gradients of the grayscale image, and then calculate the gradient magnitude based on the horizontal and vertical gradients. After binarizing the gradient magnitude, perform a morphological closing operation to obtain the binary image after the closing operation. The skeleton graph is obtained by thinning the binary graph after the closing operation.

4. The deep learning-based image and text recognition method as described in claim 3, characterized in that: The step of generating a temporal stroke energy sequence by projection convergence along the width direction includes projecting the skeleton map in the height direction to obtain a skeleton density sequence in the width direction. Based on the skeleton density sequence, the skeleton density sequence is segmented and converged according to the interval length corresponding to the downsampling factor of the shared coding network in the width direction, to generate a temporal stroke energy sequence aligned with the width temporal sequence.

5. The deep learning-based image and text recognition method as described in claim 4, characterized in that: The shared coding network includes a convolutional feature extraction subnetwork that downsamples along the width direction and a temporal context modeling subnetwork that establishes contextual dependencies in the width temporal dimension; The convolutional feature extraction subnetwork outputs a two-dimensional feature map; The temporal context modeling subnetwork performs temporal modeling on the two-dimensional feature map in the width direction to obtain a temporal feature sequence.

6. The deep learning-based image and text recognition method as described in claim 5, characterized in that: The construction of weights in the height dimension based on the skeleton graph includes mapping the skeleton graph to a mapped skeleton graph corresponding to the spatial scale of the encoded feature map. Construct height weights based on the response intensity of the mapped skeleton graph in the height direction; The encoded feature maps are weighted and summed in the height dimension according to the height weight to obtain aggregated features that are temporally aligned with the width.

7. The deep learning-based image and text recognition method as described in claim 6, characterized in that: The CTC branch outputs an emission probability distribution containing blank symbols at each width temporal position based on the aggregation feature; Given the generated character prefixes, the autoregressive branch performs attention selection on the aggregated features to progressively output the conditional probability distribution of the corresponding characters.

8. The deep learning-based image and text recognition method as described in claim 7, characterized in that: The constrained search includes determining the time interval of the character in the width time sequence based on the emission probability sequence output by the CTC branch; Based on the time interval of characters in the width time sequence, during the beam search process, according to the attention distribution formed by the autoregressive branch, combined with the character time interval and the time sequence stroke energy sequence, the candidate sequence is penalized for consistency. Based on the consistency penalty, the candidate sequence with the highest comprehensive score is selected as the text sequence for recognition, based on the combined CTC score and autoregressive score.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the deep learning-based image and text recognition method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the deep learning-based image and text recognition method according to any one of claims 1 to 8.

Images