Text detection and recognition model calibration method based on spatial position and scene association

Abstract

The invention discloses a text detection and recognition model calibration method based on spatial position and scene association. The method comprises the following steps: supervising an end-to-end text detection and recognition model according to likelihood maximization of a target text sequence and absolute distance minimization of a target position to obtain recognition loss; constructing a text association sequence set through a text association sequence mining model; constructing a text smoothing position set through a spatial position smoothing method; realizing text content confidence coefficient calibration according to the text association sequence set and the regular content loss; realizing text position confidence coefficient calibration according to the text smooth position set and the regular position loss; a global calibration intensity coefficient is introduced to be combined with regular content loss and regular position loss to construct a regularization loss function, and overall controllable end-to-end confidence coefficient calibration is achieved; and combining the recognition loss and the regularization loss to obtain a final loss so as to train and obtain a calibrated end-to-end text detection and recognition model for outputting a predicted text sequence, a position and calibration confidence.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and text detection and recognition, and in particular relates to a text detection and recognition model calibration method based on spatial position and scene association. Background Art

[0002] In early text information extraction systems, the processing flow usually involves two independent steps: text detection and text recognition. The main task of the text detection stage is to locate the text region in the input image and crop it to obtain an image containing only the text region. The work of the text recognition stage is to input the text image region cropped in the previous step into a dedicated non-end-to-end text recognition system to finally extract the text content. However, this pipeline processing method has a series of shortcomings. The most obvious problem is that errors may accumulate between the two stages. If the text detection stage fails to accurately locate all relevant text regions, or the detection is not accurate enough, the quality and accuracy of the output of the non-end-to-end text recognition system will naturally be greatly reduced when dealing with these incomplete or partially erroneous inputs. In addition, since the two stages are optimized separately, there is a lack of effective synergy between them, which may lead to inefficiency and waste of resources in the entire processing flow. With the advancement of deep learning technology and the in-depth research of algorithms, more and more studies in recent years have begun to tend to adopt detection-recognition end-to-end text recognition models. The core advantage of this model is that it combines the two processes of text detection and text recognition into a unified model framework, in this way, the potential synergy between the two can be better utilized. Specifically, the end-to-end model trains the text detector and recognizer simultaneously through a joint optimization strategy, thereby achieving higher operational efficiency and lower error rate. This integrated processing approach not only reduces the consumption of computing resources, but also enables more accurate and faster recognition in practical applications because the model can share visual features and contextual information throughout the recognition process. In addition, another significant advantage of the end-to-end system is that it simplifies the architecture of the entire system. In traditional separation models, complex interfaces and post-processing methods are required between different modules, while the end-to-end model significantly reduces the design complexity of the text information extraction system by integrating all necessary functions into a unified network.

[0003] Unlike non-end-to-end text detection and recognition models that mainly focus on accurately predicting text content from the determined text area, end-to-end models need to simultaneously achieve accurate positioning of the text in the image and accurate recognition of the text content in the image. This functional mode that is closer to practical applications makes the performance of end-to-end models more comprehensive in practical applications, but it also introduces new problems. The prediction confidence of the end-to-end text detection and recognition model not only reflects the recognition accuracy of the text content, but also includes the accuracy of the text location. Therefore, the confidence reliability evaluation of the end-to-end text detection and recognition model needs to consider these two aspects comprehensively. In non-end-to-end text detection and recognition models, overconfidence usually comes from overfitting of training data and maximizing the likelihood of the target text sequence without restriction. In end-to-end text detection and recognition models, the reason for overconfidence also comes from overfitting of training data, but its manifestation is more complicated. On the one hand, the end-to-end text detection and recognition model uses the target text sequence likelihood maximization training paradigm to supervise the text content prediction. Under this supervision paradigm, the predicted probability of text content will continue to increase, eventually exceeding the actual probability, resulting in overconfidence in text content. On the other hand, the end-to-end text detection and recognition model uses the training paradigm of minimizing the absolute distance between the target text position and the predicted text position to supervise the predicted text position. The end-to-end text detection and recognition model usually adopts the position annotation form of the position coordinates of the two-dimensional point set of the polygonal text contour, and directly optimizes the text position prediction by regressing and fitting the coordinates of the contour point set of the target text position. Although this annotation method can provide certain text position information, there are subjective factors in the manually annotated text position labels, and it is inevitable that there are some annotation noises, which cannot fully and truly reflect the actual position of the text, especially in complex backgrounds or irregular text scenes. However, under the continuous supervision and optimization of these non-real text position labels, the current training paradigm will continuously optimize the text position prediction to these non-real text position labels without limit, resulting in the model overfitting these noisy text positions, making the model overconfidently believe that it fits the actual real text position, which in turn causes overconfidence in the text position. However, the current text recognition model confidence calibration algorithm is mainly designed to address the problem of overconfidence in text content, and does not consider the potential impact of text position on prediction confidence. In fact, for end-to-end text detection and recognition models, the predicted position of the text also greatly affects the evaluation of prediction confidence. Ignoring this factor may cause the calibration results to fail to fully reflect the reliability of the model prediction, resulting in suboptimal calibration results. Therefore, a calibration algorithm for an end-to-end text detection and recognition model should comprehensively consider the impact of the accuracy of the text content and the precision of the text position on the prediction confidence, in order to achieve a more accurate and reliable confidence calibration. Summary of the invention

[0004] In view of this, it is necessary to provide a text detection and recognition model calibration method based on spatial position and scene association for the technical problem of confidence calibration of the end-to-end text detection and recognition model, and the method integrates the two correlations of content and position into the final regularization as a more effective calibration constraint. Specifically, the method consists of two processes, namely a standard training process and a calibration process. In the standard training process, the end-to-end text detection and recognition model adopts the training paradigm of maximizing the likelihood of the target text sequence and the training paradigm of minimizing the absolute distance of the target text position to supervise the optimization process of text content and text position respectively, aiming to maximize the probability of the target text sequence and accurately fit the target position of the text, resulting in recognition loss. On the contrary, the calibration process is introduced to regularize the standard training process, and the overconfidence of the text content and the overconfidence of the text position of the end-to-end text detection and recognition model are alleviated by constraining the maximization of the likelihood of the target text sequence and the minimization of the absolute distance of the target text position. Specifically, the calibration process introduces the text association sequence and the text smooth position to respectively implement additional supervision on the text content and the text position to regularize the standard training process. A scene-aware text association sequence mining model is used to mine text association sequences that are more suitable for the scene in the image. At the same time, a spatial position smoothing method is used to smooth the target text position to generate a text smoothing position set. These text association sequences and text smoothing position labels are also used to supervise the end-to-end text detection and recognition model training, thereby generating a regularization loss. In addition, a global calibration strength factor is introduced to adjust the optimization weight between the recognition loss and the regularization loss to achieve controllable calibration.

[0005] The present invention discloses a text detection and recognition model calibration method based on spatial position and scene association, comprising the following steps:

[0006] Step 1: supervise the end-to-end text detection and recognition model according to the training paradigm of maximizing the likelihood of the target text sequence and minimizing the absolute distance of the target text position, obtain the text content loss and text position loss respectively, and combine them to obtain the recognition loss;

[0007] Step 2: Obtain a text association sequence set through a scene-aware text association sequence mining model, and construct an instance-specific text association sequence set that is more suitable for the current scene;

[0008] Step 3, spatially smoothing the target text position by a spatial position smoothing method to obtain a text smoothing position set, and constructing an instance-specific text smoothing position set;

[0009] Step 4: Based on the scene-aware text association sequence set, the regular content loss is combined to constrain the target text sequence likelihood to maximize, thus achieving scene-effective text content confidence calibration;

[0010] Step 5: Based on the text smoothing position set, the regular position loss is combined to constrain the absolute distance of the target text position to be minimized, so as to achieve text position confidence calibration;

[0011] Step 6: introduce the global calibration strength coefficient, combine the regularized content loss and the regularized position loss, construct the regularized loss function of spatial position smoothing and scene perception association, and realize the overall controllable end-to-end confidence calibration;

[0012] Step 7: Combine the recognition loss and the regularization loss to obtain the final loss, use the final loss to retrain the training model to be calibrated, and finally obtain the calibrated end-to-end text detection and recognition model, which is used to output the predicted text sequence, the predicted text position and the confidence of the calibration.

[0013] Further, step 1 includes the following steps:

[0014] Input sample data, the sample data comprising a text image, the text image containing a plurality of mutually independent text instances, each of the text instances consisting of a corresponding target text position and a target text sequence;

[0015] Then the text image is input into the end-to-end text detection and recognition model to obtain the predicted text sequence for each text instance. And predict text location

[0016] For any text instance, the end-to-end text detection and recognition model adopts two training paradigms: maximizing the likelihood of the target text sequence and minimizing the absolute distance of the target text position. The training process of the text content and text position is supervised respectively to obtain the text content loss and text position loss. Then, the two losses are added together to obtain the recognition loss of the text instance:

[0017]

[0018] in, is the text content loss, is the text position loss; Loss r (·) is a commonly used loss function, ||·|| is the absolute distance function, Y is the target text sequence of the current text instance, and P is the target text position of the current text instance.

[0019] Further, in step 2, the scene-aware text association sequence mining model includes an encoder and a decoder, and the encoder includes a position embedding layer, a position attention network, a tag embedding layer and a global image encoder; the specific steps are:

[0020] The character tokens are sequentially encoded into position embeddings represented as time steps through a position embedding layer;

[0021] A single text instance image is processed by the position attention network to obtain the visual feature embedding of the current time step, which is combined with the position embedding and modality embedding to obtain the perceptual representation;

[0022] The target text sequence is encoded into semantic tag feature embedding through the tag embedding layer, which is combined with position embedding and modality embedding to obtain semantic representation;

[0023] The global feature sequence representation in the text image is extracted through the global image encoder;

[0024] The perceptual representation, semantic representation, and global feature sequence representation are concatenated together and connected as keys and values. Subsequently, by modifying the attention mask in the decoder, the query selectively interacts with different feature embeddings at different time steps, thereby affecting the predicted distribution and searching for different types of related text association sequence sets.

[0025] Furthermore, the perceptual representation generation steps are as follows:

[0026] After the text instance image is processed by the position attention network, the position of each character tag in the image is determined and the visual feature embedding of each tag is extracted as v∈R T×D , by embedding the visual features v∈R T×D , position embedding v pos ∈R T×D and modal embedding v type ∈R D Sum and get the perception representation H v ∈R T×D ,as follows:

[0027] H v =v+v pos +v type

[0028] Where T is the length of the target text sequence, D is the dimension size, and R is a real number set;

[0029] The steps of generating the semantic representation are as follows:

[0030] The target text sequence Y passes through the token embedding layer to produce semantically labeled feature embeddings represented as s∈R T×D Then, by embedding the semantic tag features and position embedding s pos ∈R T×D and modality embeddings type ∈R D Sum and get the semantic representation H s ∈R T ×D ,as follows:

[0031] H s =s+s pos +s type

[0032] The global feature sequence represents the generation steps as follows:

[0033] Input text image X. Pre-trained ResNet network global image encoder Encoder (·) extracts the global feature sequence representation of the image. as follows:

[0034] H g =Encoder(X)

[0035] Among them, T g is the sum of the lengths of all text sequences in the text image.

[0036] Further, in step 2, the output of the decoder is represented by z N Converted into a prediction distribution, the prediction probability distribution p of the scene-aware text association sequence mining model is obtained c As shown below:

[0037]

[0038] in, is the predicted probability of the Tth character of the text association sequence in a single text instance, softmax is the activation function, and linear is the linear layer;

[0039] Then, the prediction probability of the scene-aware text association sequence level is calculated by multiplying the prediction probability of the character at each time step. c (Y′|X) is:

[0040]

[0041] in, It is represented as the predicted probability of the character at time step i, T is the length of the predicted text sequence, X∈X represents a single text instance image, X represents the text image, and Y′ represents the predicted scene-aware text association sequence;

[0042] Finally, the predicted probabilities of the scene-aware text association sequence Y′ predicted by the scene-aware text association sequence mining model for a single text instance image X are sorted in descending order, and the top N C The predicted text association sequence Y′ is used as the scene-aware text association sequence set C(X,Y), where Y is the target text sequence.

[0043] Furthermore, the specific steps of step 3 are as follows:

[0044] The target text position P of each text instance is composed of a series of two-point set coordinates distributed along the text contour to form a polygon and is expressed as By using the spatial position smoothing method, a certain amount of Gaussian noise is added to multiple coordinate points of the text contour to obtain the text smoothing position P′, and N S The text smoothing position P′ is taken as the text smoothing position set S(P), and the specific formula is: P′ m Indicates the mth text smoothing position;

[0045] Any smooth position P′ is expressed as follows:

[0046]

[0047]

[0048] Among them, δ x and δ y is the Gaussian noise added in the x-axis and y-axis directions respectively, Represents δ x It has a mean of 0 and a variance of Gaussian distribution, Represents δ y It has a mean of 0 and a variance of Gaussian distribution, the number of coordinate points is n, x j ,y j are the horizontal and vertical coordinates of the coordinate point respectively.

[0049] Furthermore, the specific steps of step 4 are as follows:

[0050] The scene-aware text association sequence Y′ in step 2 is used as additional supervisory information to constrain the unlimited improvement of the probability distribution of the target text sequence, and the regular content loss is obtained as:

[0051]

[0052] in, Predict the text sequence for the image of the current text instance, Loss r (·) is a commonly used loss function, C(X,Y) is a set of scene-aware text association sequences, X is a single text instance image, and Y is the target text sequence of the current text instance.

[0053] Furthermore, the specific steps of step 5 are as follows:

[0054] The text smoothing position P′ in step 3 is used as additional supervisory information to constrain the text position prediction to overfit to the target text position, and the regular position loss is obtained as:

[0055]

[0056] in Predict the text position for the image of the current text instance, S(P) is the set of smoothed text positions, and ||·|| is the absolute distance function.

[0057] Furthermore, the specific steps of step 6 are as follows:

[0058] The regular content loss Loss in step 4 reg_r and the regular position loss Loss in step 5 reg_p Combined, we get the regular content position loss Loss reg_rp for:

[0059]

[0060] Next, the global calibration strength coefficient α is introduced, and the regularized loss function is obtained as:

[0061] Loss reg =αLoss reg_rp

[0062] Here, the global calibration strength is adjusted by α.

[0063] Furthermore, the specific steps of step 7 are as follows:

[0064] The recognition loss in step 1 and the regularization loss in step 6 are combined to obtain the final loss, and the training model to be calibrated is retrained using the final loss. Finally, a calibrated end-to-end text detection and recognition model is obtained, which is used to output the predicted text sequence, predicted text position and calibrated confidence of each text instance in the text image. The function formula of the final loss is:

[0065] Loss overall =Loss rec +Loss reg .

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] The method of the present invention systematically studies the overconfidence problem in the end-to-end text detection and recognition model for the first time, provides in-depth analysis and insights, and lays a theoretical foundation for subsequent improvement work. In order to solve the overconfidence problem, an innovative text detection and recognition model calibration method based on spatial position and scene association is proposed. The scene-aware text association sequence is introduced as an additional supervisory constraint to unlimitedly improve the probability distribution of the target text sequence, thereby calibrating the text content overconfidence. On the other hand, a spatial smoothing method is designed to obtain the text smoothing position as an additional supervisory constraint. The text position prediction is overfitted to the target text position, thereby calibrating the text position overconfidence. The method of the present invention integrates the two correlations of text content and position into the final regularization as a more effective calibration constraint to achieve more accurate end-to-end text detection and recognition model confidence calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A schematic flow chart of an implementation method of the present invention is shown;

[0069] Figure 2 A schematic diagram of a scene-aware text association sequence mining model in an embodiment of the present invention is shown;

[0070] Figure 3 A schematic diagram of an attention mask in an embodiment of the present invention is shown;

[0071] Figure 4 A schematic diagram of the position of smoothed text in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] like Figure 1 As shown, the present invention discloses a text detection and recognition model calibration method based on spatial position and scene association, comprising the following steps:

[0074] The end-to-end text detection and recognition model is supervised according to the training paradigm of maximizing the likelihood of the target text sequence and minimizing the absolute distance of the target text position, and the text content loss and text position loss are obtained respectively, and the recognition loss is obtained by combining them;

[0075] Obtain text association sequences through a scene-aware text association sequence mining model, and construct instance-specific text association sequence sets that are more appropriate for the current scene;

[0076] The target text position is spatially smoothed by a spatial position smoothing method to obtain a text smoothing position, and an instance-specific text smoothing position set is constructed;

[0077] Based on the scene-aware text association sequence set, the regular content loss is combined to constrain the target text sequence likelihood to maximize, thus achieving scene-effective text content confidence calibration;

[0078] According to the text smoothing position set, the regular position loss is combined to constrain the absolute distance of the target text position to be minimized, thus achieving text position confidence calibration;

[0079] A global calibration strength coefficient is introduced, and the regularized content loss and regularized position loss are combined to construct a regularized loss function that associates spatial position smoothing with scene perception, thus achieving overall controllable end-to-end confidence calibration.

[0080] The recognition loss and regularization loss are combined to obtain the final loss, and the training model to be calibrated is retrained using the final loss. Finally, a calibrated end-to-end text detection and recognition model is obtained, which is used to output the predicted text sequence, predicted text position and calibration confidence.

[0081] Specifically, this embodiment adopts the following steps to implement the inventive method:

[0082] Step 1: Supervise the end-to-end text detection and recognition model according to the training paradigm of maximizing the likelihood of the target text sequence and minimizing the absolute distance of the target position, and obtain the text content loss and text position loss respectively, and combine them to obtain the recognition loss.

[0083] In step 1, enter the sample data Where X represents the text image in the sample data ( Figure 2 The text image X contains N independent text instances ( Figure 2 The text instance image corresponding to the text instance image in is X∈X, It represents the target text position of the i-th text instance in the text image, which is usually represented by a series of point set coordinates distributed along the text contour to form a polygon. The number of point sets is n, x ij Represents the horizontal coordinate of the jth corresponding point in the target text position point set of the i-th text instance, y ij Represents the ordinate of the jth corresponding point in the target text position point set of the i-th text instance, Y i Represents the target text sequence of the i-th text instance in the text image;

[0084] It should be noted that the polygon formed by the text outline can be a rectangle, that is, nFor a pentagon, it is 4, for a pentagon, it is 5, and for other polygonal structures, n is equal to the number of vertices of the polygon;

[0085] Specifically, considering that the target text sequence of any text instance in the text image X is Y and the target text position is P, the text image is input into the end-to-end text detection and recognition model to obtain the predicted text sequence of the text instance And predict text location

[0086] For any text instance, the end-to-end text detection and recognition model adopts two training paradigms: maximizing the likelihood of the target text sequence and minimizing the absolute distance of the target text position. The training process of the text content and text position is supervised respectively to obtain the text content loss and text position loss. Then, the two losses are added together to obtain the recognition loss of the text instance:

[0087]

[0088] in, is the text content loss, is the text position loss; Loss r (·) is the loss function commonly used in text content supervision, such as cross entropy loss, and ||·|| is the absolute distance function.

[0089] Step 2: Obtain text association sequences through a scene-aware text association sequence mining model, and construct an instance-specific text association sequence set that is more appropriate for the current scene.

[0090] like Figure 2 As shown in the figure, the proposed scene-aware text association sequence mining model consists of two parts: encoder and decoder, where the encoder includes a position embedding layer, a position attention network, a tag embedding layer and a global image encoder;

[0091] The character token sequence is encoded into position embeddings represented as time step t through the position embedding layer;

[0092] A single text instance image is processed by the position attention network to obtain the visual feature embedding of the current time step, which is combined with the position embedding and modality embedding to obtain the perceptual representation;

[0093] The target text sequence is encoded into semantic tag feature embedding through the tag embedding layer, which is combined with position embedding and modality embedding to obtain semantic representation;

[0094] The global feature sequence representation in the text image is extracted through the global image encoder;

[0095] The perceptual representation, semantic representation, and global feature sequence representation are then concatenated together and connected as keys and values. Subsequently, by modifying the attention mask in the Transformer decoder, the query can selectively interact with different feature embeddings at different time steps, thereby affecting the predicted distribution and searching for different types of related text association sequence sets.

[0096] It should be noted that the character tag sequence is usually the text tag sequence in the text. For example, the text content in the text instance image is pack, the character at time step 0 is p, the character at time step 1 is a, the character at time step 2 is c, and the character at time step 3 is k. At this time, the character tag sequence is 0123; the time step is usually referred to by a t, which means the current time step t, that is, the current moment;

[0097] Specifically, the text instance image is processed by the position attention network to determine the position of each character tag in the text image and extract the visual feature embedding of each tag as v∈R T×D , then, by embedding the visual features v∈R T×D , position embedding v pos ∈R T×D and modal embedding v type ∈R D Sum and get the perception representation H v ∈R T×D ,as follows:

[0098] H v =v+v pos +v type

[0099] Among them, T is the length of the target text sequence, D is the dimension size of the hidden layer, and R is a set of real numbers.

[0100] When training a model, it is usually necessary to project the features into a high-dimensional space, where D is the hidden layer dimension size of the high-dimensional space;

[0101] Secondly, the target text sequence Y passes through the token embedding layer to produce semantically labeled feature embeddings represented as s∈R T×D Then, by embedding the semantic tag features and position embedding s pos ∈R T×D and modality embeddings type ∈R D Sum and get the semantic representation H s ∈R T×D ,as follows:

[0102] H s =s+s pos +s type

[0103] At the same time, the input text image X is pre-trained through the ResNet network global image encoder Encoder (·) to extract the global feature sequence representation of the image as follows:

[0104] H g =Encoder(X)

[0105] Among them, T g is the sum of the lengths of all text sequences in the text image.

[0106] The Transformer decoder is specially modified to eliminate the self-attention mechanism in the decoder to prevent information leakage caused by mutual access between character tokens at different time steps. Each layer in the decoder includes a multi-head cross attention layer (MHCA), a feed-forward network layer (FFN) and a normalization layer (LN);

[0107] It should be noted that the multi-head cross-attention layer uses multiple independent attention heads to calculate the attention weights separately, and concatenates or weighted sums their results to obtain a richer representation. This calculation mechanism is also called the cross-attention mechanism, which is a common term in Transformer.

[0108] First, the character token sequence [1, 2, 3, ..., T] is processed through the position embedding layer to obtain the position embedding z0 as the query input of the cross attention mechanism to represent each time step. At the same time, the perceptual representation, semantic representation, and global feature sequence representation are concatenated together to form keys and values. T g is the sum of the lengths of all text sequences in the text image, T is the length of the target text sequence, D is the dimension of the hidden layer, and R is a real number set;

[0109] The operation in the lth layer (referring to any layer in the decoder) of the decoder can be formulated as:

[0110] z′ l-1 =MHCA(LN(z l-1 ))+z l-1

[0111] z l =FFN(LN(z′ l-1 ))+z′ l-1

[0112] Among them, z l-1 is the input representation of the current layer, z′ l-1 is the intermediate representation after the multi-head cross-attention operation at layer l, z l is the output representation of the current layer, T is the length of the target text sequence;

[0113] The calculation result of the multi-head cross attention operation is:

[0114]

[0115] Q=W q z l-1

[0116] K=W k H vsg

[0117] V=W v H vsg

[0118] Where Q is the query in the self-attention mechanism, K is the key in the self-attention mechanism, V is the value in the self-attention mechanism, D is the size of the hidden layer dimension, and W q , W k and W v are all learnable weight coefficients, is the attention mask, K T is the transpose of K, and softmax is an activation function commonly used in machine learning and deep learning;

[0119] Finally, the output of the Transformer decoder is represented by z using the classifier N Converted to predictive distribution, the predicted probability distribution p of the obtained scene-aware text association sequence mining model c As shown below:

[0120]

[0121] in, is the predicted probability of the Tth character, N is the number of text instances; softmax is the activation function, and linear is the linear layer;

[0122] Among them, the schematic diagram of the attention mask is as follows Figure 3 As shown, attention masks include visual masks, semantic masks, and visual and visual masks;

[0123] The predicted probability of the character at the current time step t is calculated by the visual mask as follows:

[0124]

[0125] When using visual masks, all semantic representation embeddings and perceptual representation embeddings of other time steps are filtered out, and the query value at the current time step t only focuses on the perceptual representation embedding at time step t. Therefore, the predicted character at time step t only depends on the text instance image feature x at the current time step t. t ,at this time The larger the value, the more the current character category k and the text instance image feature x t are more similar in perception; t Indicates the predicted sequence category of the current time step;

[0126] The predicted probability of the character at the current time step t is calculated by the semantic mask as follows:

[0127]

[0128] When using semantic mask, all perceptual representation embeddings and semantic representation embeddings at the current time step t are filtered out, where T is the length of the predicted text sequence. The larger the value, the more semantic context combination y 1:t-1 +k+y t+1:T The stronger the semantic relevance to the target text sequence Y, the 1：t-1 The predicted sequence category before the current time step, y t+1：T is the predicted sequence category after the current time step;

[0129] The predicted probability of the character at the current time step t is calculated by visual and semantic masks as follows:

[0130]

[0131] When using visual and semantic masks, T is the length of the predicted text sequence, The larger the value, the more semantic context combination y 1:t-1 +k+y t+1:T The stronger the semantic relevance to the target text sequence Y, the closer the current category k is to the text instance image feature x t are more similar in perception; 1：t-1 The predicted sequence category before the current time step, y t+1：T is the predicted sequence category after the current time step;

[0132] It should be noted that, for example, when the text content of a text instance image is PACK and the second time step t is 1, then x t ∈X represents the feature of A in the image, that is, the text instance image feature at the current time step t, and X represents the current text instance image;

[0133] At the same time, assuming that text recognition has 36 categories (lowercase English letters and numbers), in the actual encoding process, the character categories of letters and numbers need to be encoded, such as 0 for a, 1 for b, 2 for c, ..., 35 for 9. The total number of categories is generally defined as K, and the current category is referred to as k, which is understood as the current predicted character category. In other words, the corresponding text content can be obtained by predicting the character category;

[0134] Through the prediction of the attention mask and scene-aware text association sequence mining model, the probability of the character category at each time step is predicted. For example, if the text content in the text instance image is [p, a, c, k], the corresponding output prediction probability of the character category [p=15, a=0, c=2, k=10] is [0.9, 0.96, 0.97, 0.95];

[0135] Next, the prediction probability of the entire text sequence pack is calculated by multiplying the prediction probability of each character by 0.9*0.96*0.97*0.95 to obtain the sequence-level prediction probability.

[0136] According to the prediction probability of each character at each time step, the sequence-level prediction probability P is calculated by cumulative multiplication. c (Y′|X) is:

[0137]

[0138] in, It is expressed as the predicted probability of the character at time step i, T is the length of the predicted text sequence,

[0139] Next, the image X of a single text instance is sorted in descending order by the predicted probability of the scene-aware text association sequence Y′ predicted by the scene-aware text association sequence mining model, and the top N images are collected. C The predicted text association sequence Y′ is used as the scene-aware text association sequence set C(X,Y), where Y is the target text sequence.

[0140] Step 3: Perform spatial smoothing on the target text position through a spatial position smoothing method to obtain a text smoothing position, and construct an instance-specific text smoothing position set.

[0141] Considering that the target text position is represented by a series of two-point set coordinates distributed along the text contour to form a polygon By using the spatial position smoothing method, a certain amount of Gaussian noise is added to multiple coordinate points of a single contour to obtain the text smoothing position P′, and N S The text smoothing position P′ is taken as the text smoothing position set S(P), and the specific formula is: P′ m Indicates the mth text smoothing position;

[0142] Any smooth position P′ is expressed as follows:

[0143]

[0144] Among them, δ x and δ yis the Gaussian noise added in the x-axis and y-axis directions respectively, Represents δ x It has a mean of 0 and a variance of Gaussian distribution, Represents δ y It has a mean of 0 and a variance of Gaussian distribution, the total number of coordinate points is n, x j ,y j are the horizontal and vertical coordinates of the coordinate point respectively;

[0145] Among them, the schematic diagram of the smooth text position is as follows Figure 4 As shown, the solid line frame in the left image is the target text position, and the dotted line frame in the right image is the smooth text position. It can be observed that the smooth text position is more consistent with the actual position of the target text.

[0146] Step 4: Based on the scene-aware text association sequence set, the regular content loss is combined to constrain the target text sequence likelihood to maximize and achieve scene-effective text content confidence calibration.

[0147] Taking the scene-aware text association sequence Y′ as additional supervisory information, we constrain the unlimited improvement of the probability distribution of the target text sequence, and obtain the regular content loss as:

[0148]

[0149] in, For the predicted text sequence of the text instance image, Loss r (·) is the commonly used loss function in the text content supervision process, and C(X,Y) is the set of scene-aware text association sequences.

[0150] By introducing global feature sequence representation, the scene-aware text association sequence mining model can more keenly capture the scene features in the image, and then mine the text association sequence that is more suitable for the current scene, so as to more realistically reflect the actual probability distribution of the text content and achieve more accurate text content confidence calibration;

[0151] For example Figure 2 By introducing text images, the scene-aware text association sequence mining model can perceive the "mall" scene in the image, thereby reducing the possibility of mistakenly mining the third character marker "i" in the text instance image "exit" as "1", and more truly reflecting the actual probability distribution of the text content.

[0152] Step 5: Based on the text smoothing position set, the regular position loss is combined to constrain the absolute distance of the target text position to be minimized to achieve text position confidence calibration.

[0153] The text smoothing position P′ is used as additional supervision information to constrain the text position prediction to overfit to the target text position, and the regular position loss is obtained as:

[0154]

[0155] in is the predicted text position of the text instance image, S(P) is the text smoothing position set, and ||·|| is the absolute distance function;

[0156] During the training process, these noisy parameters are introduced to impose additional constraints on the text position prediction of the end-to-end text detection and recognition model. This effectively constrains the overfitting of the target text position and alleviates the problem of overconfidence in the model's text position prediction.

[0157] Step 6: Introduce the global calibration strength coefficient, combine the regularized content loss and the regularized position loss, construct the regularized loss function of spatial position smoothing and scene perception association, and achieve overall controllable end-to-end confidence calibration.

[0158] Loss of regular content reg_r And regular position loss Loss reg_p Combined, we get the regular content position loss Loss reg_rp for:

[0159]

[0160] Introducing the global calibration strength coefficient α, the regularized loss function is obtained as:

[0161] Loss reg =αLoss reg_rp

[0162] Here, the global calibration strength is adjusted by α.

[0163] Step 7: Combine the recognition loss and the regularization loss to obtain the final loss, use the final loss to retrain the training model to be calibrated, and finally obtain a calibrated end-to-end text detection and recognition model, which is used to output the predicted text sequence, predicted text position and calibrated confidence of each text instance in the text image. The function formula of the final loss is:

[0164] Loss overall =Loss rec +Loss reg

[0165] In step 7, the original loss function is adjusted according to the standard training process in steps 1, 2, 3, 4, 5, and 6, the scene-aware text association sequence mining strategy, the target text position spatial smoothing strategy, the regular content loss construction, the regular position loss construction, and the overall regularization loss construction, and the pre-trained model is retrained and calibrated so that the model can be calibrated. Since the fine-tuning model is used for training, the learning rate is set to 0.1 during the training process, and iterative training is performed. When the expected calibration error of the model does not decrease for 7 consecutive times, the training is stopped, and the predicted text sequence, position, and calibrated confidence are finally output.

[0166] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0167] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0169] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A text detection and recognition model calibration method based on spatial position and scene association, characterized in that: The following steps are involved: Step 1: supervise the end-to-end text detection and recognition model according to the training paradigm of maximizing the likelihood of the target text sequence and minimizing the absolute distance of the target text position, obtain the text content loss and text position loss respectively, and combine them to obtain the recognition loss; Step 2: Obtain text association sequences through a scene-aware text association sequence mining model to construct an instance-specific text association sequence set that is more suitable for the current scene; Step 3, spatially smoothing the target text position by a spatial position smoothing method to obtain a text smoothing position, and constructing an instance-specific text smoothing position set; Step 4: Based on the scene-aware text association sequence set, the regular content loss is combined to constrain the target text sequence likelihood to maximize, thus achieving scene-effective text content confidence calibration; Step 5: Based on the text smoothing position set, the regular position loss is combined to constrain the absolute distance of the target text position to be minimized, so as to achieve text position confidence calibration; Step 6: introduce the global calibration strength coefficient, combine the regularized content loss and the regularized position loss, construct the regularized loss function of spatial position smoothing and scene perception association, and realize the overall controllable end-to-end confidence calibration; Step 7: Combine the recognition loss and the regularization loss to obtain the final loss, use the final loss to retrain the training model to be calibrated, and finally obtain the calibrated end-to-end text detection and recognition model, which is used to output the predicted text sequence, the predicted text position and the confidence of the calibration.

2. The text detection and recognition model calibration method based on spatial position and scene association according to claim 1 is characterized in that: Step 1 includes the following steps: Input sample data, the sample data comprising a text image, the text image containing a plurality of mutually independent text instances, each of the text instances consisting of a corresponding target text position and a target text sequence; Then the text image is input into the end-to-end text detection and recognition model to obtain the predicted text sequence for each text instance. And predict text location For any text instance, the end-to-end text detection and recognition model adopts two training paradigms: maximizing the likelihood of the target text sequence and minimizing the absolute distance of the target text position. The training process of the text content and text position is supervised respectively to obtain the text content loss and text position loss. Then, the two losses are added together to obtain the recognition loss of the text instance: in, is the text content loss, is the text position loss; Loss r (·) is a commonly used loss function, ‖·‖ is the absolute distance function, Y is the target text sequence of the current text instance, and P is the target text position of the current text instance.

3. The text detection and recognition model calibration method based on spatial position and scene association according to claim 2 is characterized in that: In step 2, the scene-aware text association sequence mining model includes an encoder and a decoder, and the encoder includes a position embedding layer, a position attention network, a tag embedding layer and a global image encoder; the specific steps are: The character tokens are sequentially encoded into position embeddings represented as time steps through a position embedding layer; A single text instance image is processed by the position attention network to obtain the visual feature embedding of the current time step, which is combined with the position embedding and modality embedding to obtain the perceptual representation; The target text sequence is encoded into semantic tag feature embedding through the tag embedding layer, which is combined with position embedding and modality embedding to obtain semantic representation; The global feature sequence representation in the text image is extracted through the global image encoder; The perceptual representation, semantic representation, and global feature sequence representation are concatenated together and connected as keys and values. Subsequently, by modifying the attention mask in the decoder, the query selectively interacts with different feature embeddings at different time steps, thereby affecting the predicted distribution and searching for different types of related text association sequence sets.

4. The text detection and recognition model calibration method based on spatial position and scene association according to claim 3 is characterized in that: The perceptual representation generation steps are as follows: After the text instance image is processed by the position attention network, the position of each character tag in the image is determined and the visual feature embedding of each tag is extracted as v∈R T×D , by embedding the visual features v∈R T×D , position embedding v pos ∈R T ×D and modal embedding v type ∈R D Sum and get the perception representation H v ∈R T×D ,as follows: H v =v+v pos +v type Where T is the length of the target text sequence, D is the dimension size, and R is a real number set; The steps of generating the semantic representation are as follows: The target text sequence Y passes through the token embedding layer to produce semantically labeled feature embeddings represented as s∈R T×D Then, by embedding the semantic tag features and position embedding s pos ∈R T×D and modality embeddings type ∈R D Sum and get the semantic representation H s ∈R T×D ,as follows: H s =s+s pos +s type The global feature sequence represents the generation steps as follows: Input text image X. Pre-trained ResNet network global image encoder Encoder (·) extracts the global feature sequence representation of the image. as follows: H g =Encoder(X) Among them, T g is the sum of the lengths of all text sequences in the text image.

5. The text detection and recognition model calibration method based on spatial position and scene association according to claim 4 is characterized in that: In step 2, the output of the decoder is represented as z N Converted into a prediction distribution, the prediction probability distribution p of the scene-aware text association sequence mining model is obtained c As shown below: in, is the predicted probability of the Tth character of the text association sequence in a single text instance, softmax is the activation function, and linear is the linear layer; Then, the prediction probability of the scene-aware text association sequence level is calculated by multiplying the prediction probability of the character at each time step. c (Y′|X) is: in, It is represented as the predicted probability of the character at time step i, T is the length of the predicted text sequence, X∈X represents a single text instance image, X represents the text image, and Y′ represents the predicted scene-aware text association sequence; Finally, the predicted probabilities of the scene-aware text association sequence Y′ predicted by the scene-aware text association sequence mining model for a single text instance image X are sorted in descending order, and the top N C The predicted text association sequence Y′ is used as the scene-aware text association sequence set C(X,Y), where Y is the target text sequence.

6. The text detection and recognition model calibration method based on spatial position and scene association according to claim 3 is characterized in that: Step 3 The specific steps are as follows: The target text position P of each text instance is composed of a series of two-point set coordinates distributed along the text contour to form a polygon and is expressed as By using the spatial position smoothing method, a certain amount of Gaussian noise is added to multiple coordinate points of the text contour to obtain the text smoothing position P′, and N S The text smoothing position P′ is taken as the text smoothing position set S(P), and the specific formula is: P′ m Indicates the mth text smoothing position; Any smooth position P′ is expressed as follows: Among them, δ x and δ y is the Gaussian noise added in the x-axis and y-axis directions respectively, Represents δ x It has a mean of 0 and a variance of Gaussian distribution, Represents δ y It has a mean of 0 and a variance of Gaussian distribution, the number of coordinate points is n, x j ,y j are the horizontal and vertical coordinates of the coordinate point respectively.

7. The text detection and recognition model calibration method based on spatial position and scene association according to claim 6 is characterized in that: Step 4 The specific steps are as follows: The scene-aware text association sequence Y′ in step 2 is used as additional supervisory information to constrain the unlimited improvement of the probability distribution of the target text sequence, and the regular content loss is obtained as: in, Predict the text sequence for the image of the current text instance, Loss r (·) is a commonly used loss function, C(X,Y) is a set of scene-aware text association sequences, X is a single text instance image, and Y is the target text sequence of the current text instance.

8. The text detection and recognition model calibration method based on spatial position and scene association according to claim 7 is characterized in that: Step 5 The specific steps are as follows: The text smoothing position P′ in step 3 is used as additional supervisory information to constrain the text position prediction to overfit to the target text position, and the regular position loss is obtained as: in Predict the text position for the image of the current text instance, S(P) is the set of text smoothing positions, and ‖·‖ is the absolute distance function.

9. The text detection and recognition model calibration method based on spatial position and scene association according to claim 8, characterized in that: Step 6 The specific steps are as follows: The regular content loss Loss in step 4 reg_r and the regular position loss Loss in step 5 reg_p Combined, we get the regular content position loss Loss reg_rp for: Next, the global calibration strength coefficient α is introduced, and the regularized loss function is obtained as: Loss reg =αLoss reg_rp Here, the global calibration strength is adjusted by α.

10. The text detection and recognition model calibration method based on spatial position and scene association according to claim 9, characterized in that: Step 7 The specific steps are as follows: The recognition loss in step 1 and the regularization loss in step 6 are combined to obtain the final loss, and the training model to be calibrated is retrained using the final loss. Finally, a calibrated end-to-end text detection and recognition model is obtained, which is used to output the predicted text sequence, predicted text position and calibrated confidence of each text instance in the text image. The function formula of the final loss is: Loss overall =Loss rec +Loss reg 。

Images