A low-resource language image-text recognition method based on sample contribution perception, an electronic device and a computer readable storage medium

Abstract

The application relates to a low-resource language image-text recognition method based on sample contribution perception, an electronic device and a computer readable storage medium, and belongs to the technical field of model training. A low-resource language text image set to be recognized is acquired, and a small amount of text images in the set are labeled. A text recognition network is trained using the labeled text image set. A sample group with the highest contribution degree is selected from unlabeled samples in batches using multiple iteration steps, and is labeled. An incremental learning training of the text recognition network is performed using a new labeled sample group through an elastic weight integration algorithm, and iteration is continuously performed until a labeling cost reaches a limited value. A trained text image recognition network is used to recognize any input text image, and a recognition result is output. The application can improve the recognition accuracy of a low-resource language image-text recognition model under the condition that the labeling cost of a training set image is limited.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model training, and in particular to a low-resource language image-text recognition method based on sample contribution perception, an electronic device and a computer readable storage medium. BACKGROUND

[0002] The quantity and quality of image-text training data have a crucial impact on the training effect of image-text recognition models. However, the annotated image-text data of low-resource languages are scarce, which restricts the improvement of the recognition model performance. The current low-resource language image-text recognition method ignores the difference in contribution of different samples to model training when annotating and training the model. When annotating, the existing method annotates the text images of low-resource languages without selection in the same important degree until the annotation budget is exhausted, which cannot realize the full use of the annotation budget and maximize the training effect of the model. At the same time, the existing active learning method mixes the newly selected samples with the annotated samples after selecting the samples in batches, and then re-trains the model, which increases the model training cost. SUMMARY

[0003] The purpose of the present application is to solve the problems in the background art, and to provide a low-resource language image-text recognition method based on sample contribution perception, which can improve the recognition accuracy of low-resource language image-text recognition models under the condition of limited training set image annotation cost.

[0004] The technical solution of the present application: a low-resource language image-text recognition method based on sample contribution perception, which comprises the following steps:

[0005] S1, obtaining a low-resource language text image to be recognized to form a training image set Wherein, represents the i-th text image,

[0006] represents the number of text images in

[0007] annotating the recognition labels of a small number of images in the set Wherein represents the number of images in

[0008] represents the i-th text image in

[0009] represents the i-th text image in ​​​​​​The recognition result label corresponding to each text image;

[0010] The remaining unlabeled samples constitute a set Wherein Indicates the number of images in ,

[0011] Indicates the i-th text image in ,

[0012] Indicates the number of images in ;

[0013] S2, constructing a recognition network of a text image , comprising: a visual feature extraction module, a Transformer decoding module, and a loss function calculation module;

[0014] Each sample in is input into by the visual feature extraction module to extract visual features , is input into the Transformer decoding module to obtain a decoded character probability sequence prediction result , is compared with to calculate a loss function and perform back propagation to update the parameters in , and obtain an initial recognition network ;

[0015] During training, the Fisher information matrix of each parameter in the network is calculated by an elastic weight integration algorithm ;

[0016] S3, for i , iteratively input the samples in into the recognition network to calculate the uncertainty of each sample on ;

[0017] Combine sample diversity to select the sample set composed of the samples that contribute most to the training of ; Wherein is the number of samples selected in the i-th step,

[0018] is the maximum number of iterations;

[0019] to ​​After labeling the samples, a set is obtained Wherein is the kth sample in ;

[0020] is the kth sample in ;

[0021] S4, the input , according to the Fisher information matrix The network node parameters are set to different degrees of constraint, and the incremental learning training is carried out by using the elastic weight integration method , the network after i-step training is obtained , and the Fisher information matrix of the i-step is calculated ;

[0022] The samples existing in are removed from , and the remaining unlabeled sample set is obtained ; the i-step labeling cost is calculated Wherein, is the cost of labeling a sample;

[0023] If ,

[0024] Stop iteration and return , otherwise execute the next iteration;

[0025] When executing the next iteration,

[0026] If ,

[0027] Let ,

[0028] Otherwise let ,

[0029] Wherein, Indicates the maximum labeling cost;

[0030] S5, when the iteration stops, the trained text image model is used to predict any input text image , so as to obtain the recognition result of ;

[0031] 2. The low-resource language image-text recognition method based on sample contribution perception according to claim 1, wherein the S3 comprises the following steps:

[0032] S3.1: the each sample input recognition network in forward propagation in

[0033] get a prediction result set of wherein, represent a model the prediction result of as the pseudo label of sample ;

[0034] S3.2: for r ,

[0035] calculated by gradient embedding vectors of samples in ,

[0036] wherein, represent the parameters of the last layer,

[0037] represent the results obtained after inputting ,

[0038] represent the samples labeled with ;

[0039] the recognition loss function of the network calculated after inputting network ,

[0040] represent the gradient embedding vectors of ;

[0041] S3.3: add the sample with the largest gradient embedding vector norm in to the set ; for

[0042] , compare the norm of the gradient embedding difference between the sample in and the existing sample in ,

[0043] select the sample with the largest norm difference , and add it to ;

[0044] when , label the samples in to obtain .

[0045] ​​​​An electronic device comprises a memory for storing a program supporting a processor to execute the above-mentioned low-resource language graphic-text recognition method based on sample contribution perception, and the processor is configured to execute the program stored in the memory.

[0046] A computer-readable storage medium for running the above-mentioned electronic device, the computer-readable storage medium has a computer program stored thereon, and the computer program performs the steps in the low-resource language graphic-text recognition method based on sample contribution perception when executed by a processor.

[0047] Compared with the prior art, the present application has the following beneficial technical effects:

[0048] In the training process of the graphic-text recognition model, the present application selects unlabelled samples to be added to the training set through dynamic iteration, overcoming the deficiency of the prior art that ignores the importance difference of training samples when constructing training data, and saving the labeling cost under the condition of achieving the same recognition accuracy. At the same time, the sample selection strategy of the present application enables the model to receive a sample group with high training value for the current model in each training round, thereby realizing the training of a model with better accuracy under the condition of limited labeling budget.

[0049] In the training process of the graphic-text recognition model, the present application combines the parameter state of the current round model to realize dynamic perception of the combined contribution of samples to the graphic-text recognition model. On the one hand, the greater the uncertainty of the sample, the lower the prediction ability of the model for this type of sample at the current training stage, and using this sample for model training will bring more performance improvement. On the other hand, the difference between samples is small, and the contribution of these samples to model training is redundant. The present application combines the uncertainty of the model to the sample and the diversity of the sample to measure the contribution of the sample combination to the graphic-text recognition model, considers the contribution of a single sample to recognition, avoids the redundancy of samples in the group, establishes an effective sample selection strategy, and thus improves the accuracy of the graphic-text recognition model.

[0050] After selecting the sample group in each round, the present application trains the model on the new sample group by referring to the idea of incremental learning. When training the model in each round, if the existing samples and new samples are mixed to retrain the model, the training cost of the model will increase. If only the new samples are used to fine-tune the model, the risk of catastrophic forgetting of the model will increase.

[0051] The present application reduces the risk of forgetting existing knowledge of the model without increasing the training cost of the model, thereby further improving the training efficiency of the model and improving the recognition accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0053] Fig. 1 is a use flowchart of the low-resource language image-text recognition method based on sample contribution perception of the present application;

[0054] Fig. 2 is an implementation framework diagram of the low-resource language image-text recognition method based on sample contribution perception of the present application. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned objects, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings.

[0056] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in other manners different from those described herein without departing from the scope of the present application, and those skilled in the art can make similar generalizations without departing from the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0057] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor does it mean that the embodiment is mutually exclusive or selective with other embodiments.

[0058] Thirdly, the present application is described in detail in combination with the schematic diagram. In the detailed description of the embodiments of the present application, the cross-sectional view of the device structure is locally enlarged without the general proportion for the convenience of description, and the schematic diagram is only an example, which should not limit the scope of protection of the present application herein. In addition, the three-dimensional spatial dimensions of length, width and depth should be included in actual manufacturing.

[0059] Embodiment one

[0060] As shown in the figure, the present application proposes a low-resource language image-text recognition method based on sample contribution perception, which comprises the following steps: Figs. 1-2

[0061] S1, obtaining a low-resource language text image to be recognized to form a training image set Among them, represents the i-th text image,

[0062] ​​ denotes the number of text images in the Chinese image,

[0063] annotating the recognition labels of a small number of images in the set to obtain a labeled training image set wherein denotes the number of images in the set

[0064] denotes the i-th text image in the set

[0065] denotes the recognition result label corresponding to the i-th text image the remaining unlabeled samples constitute a set

[0066] wherein denotes the number of images in the set

[0067] denotes the i-th text image in the set

[0068] denotes the number of images in the set

[0069] is used as an initial labeled sample pool, is used as an initial unlabeled sample pool

[0070] S2, constructing a recognition network for text images , comprising: a visual feature extraction module, a Transformer decoding module, and a loss function calculation module;

[0071] each sample in the set is input into the set , and visual features are extracted by the visual feature extraction module , the set is input into the Transformer decoding module to obtain a decoded character probability sequence prediction result , the set is compared with the set to calculate a loss function and perform back propagation to update the parameters in the set , thereby obtaining an initial recognition network ;

[0072] ​​​​During training, the Fisher information matrix of each parameter in the network is calculated using the elastic weight integration algorithm ;

[0073] The Fisher information matrix measures the importance of each weight in the recognition network for the current training task. It is used to impose regularization constraints on important weights when training the network on newly labeled samples in the next round, preventing the model from forgetting the knowledge gained from training on previous data.

[0074] S3, for i , iteratively Sample input recognition network in , calculate each sample in uncertainty on ;

[0075] Combined sample diversity selection Middle pair The training that contributed most to The sample set consists of samples in is the number of samples selected in step i,

[0076] is the maximum number of iterations;

[0077] right After the samples are labeled, the set in for The kth sample in ;

[0078] for The corresponding recognition result label;

[0079] The S3 includes the following steps:

[0080] S3.1: Each sample in the input recognition network Forward propagation is performed in

[0081] get The prediction result set in, Representation Model right The prediction results are used as samples Pseudo labels of

[0082] S3.2: For r ,

[0083] Depend on calculate The gradient embedding vector of the sample in ,

[0084] wherein, denotes the parameters of the last layer,

[0085] denotes the result obtained after inputting to the network ,

[0086] denotes that the label is ,

[0087] the identification loss function of the network calculated after inputting to the network ,

[0088] denotes the gradient embedding vector of ;

[0089] S3.3: The sample with the largest gradient embedding vector norm in is added to the set ; the larger the gradient embedding vector norm, the higher the uncertainty of the current model to the sample.

[0090] For , the norm of the difference between the gradient embedding of the sample in and the gradient embedding of the existing sample in is compared, and the sample with the largest norm difference is added to ; the larger the norm difference, the greater the difference in diversity between the sample and the selected sample.

[0091] When , the sample in is labeled to obtain .

[0092] On the one hand, the greater the uncertainty of the sample, the lower the prediction ability of the model of this type of sample in the current training stage, and if this sample is used for training of the model, more performance improvement may be brought.

[0093] On the other hand, the smaller the difference between the samples, the more redundant the contribution of these samples to the model training may be. The present application jointly measures the contribution of the sample combination to the model according to the uncertainty of the sample and the diversity of the sample. After the selection of samples is completed, the samples in are labeled to obtain

[0094] S4, Input In an embodiment, the Fisher information matrix is calculated according to Different degrees of constraints are set for network node parameters, and the incremental learning is trained by using elastic weight integration , to obtain the network after the i-th step of training , and the Fisher information matrix of the i-th step is calculated ;

[0095] When training the model at each round, if the existing samples and the new samples are mixed to retrain the model, the cost of model training is increased. If the model is only fine-tuned using the new samples, the risk of catastrophic forgetting of the model is increased. The present application reduces the risk of forgetting the existing knowledge of the model without increasing the cost of model training, thereby further improving the training efficiency of the model.

[0096] After removing the samples in from , the remaining unlabeled sample set is obtained for sample selection and training in the next step.

[0097] The annotation cost of the i-th step is calculated wherein, is the cost of annotating a sample;

[0098] If ,

[0099] the iteration is stopped, and is returned, otherwise the next iteration is performed.

[0100] When performing the next iteration,

[0101] If ,

[0102] then ,

[0103] otherwise ,

[0104] wherein, represents the maximum annotation cost.

[0105] S5, when the iteration is stopped, the trained text image model is used to predict any input text image , so as to obtain the recognition result of ;

[0106] The application obtains a set of low-resource language text images to be recognized, and a small amount of text images in the set are labeled; a text recognition network is trained using the set of labeled text images; a sample group with the highest contribution is selected from unlabeled samples in batches using multiple iteration steps, and is labeled; the text recognition network is incrementally learned and trained using the newly labeled sample group through an elastic weight integration algorithm, and iteration is continued until the labeling cost reaches a limited value; and the trained text image recognition network is used to recognize any input text image and output a recognition result.

[0107] In the training process of the image-text recognition model, the application selects unlabeled samples to be added to the training set through dynamic iteration, overcomes the deficiency of the prior art that ignores the importance difference of training samples when constructing training data, and saves the labeling cost under the condition of achieving the same recognition accuracy. At the same time, the sample selection strategy of the application enables the model to receive a sample group with a higher training value for the current model in each round of training, thereby realizing training of a model with better accuracy under the condition of limited labeling budget.

[0108] In the training process of the image-text recognition model, the application combines the parameter state of the current round model to realize dynamic perception of the combined contribution of samples to the image-text recognition model. On the one hand, the greater the uncertainty of the sample, the lower the prediction ability of the model for this type of sample in the current training stage, and using this sample for model training will bring more performance improvement. On the other hand, the difference between samples is small, and the contribution of these samples to model training is redundant. The application jointly measures the contribution of sample combination to the image-text recognition model by considering the uncertainty of the model to the sample and the diversity of the sample, considers the contribution of a single sample to recognition, avoids redundancy of samples in the group, establishes an effective sample selection strategy, and thereby improves the accuracy of the image-text recognition model.

[0109] After selecting the sample group in each round, the application trains the model on the new sample group by referring to the idea of incremental learning. When the model is trained in each round, if the existing samples and the new samples are mixed to retrain the model, the training cost of the model will increase. If the model is only fine-tuned using the new samples, the risk of catastrophic forgetting of the model will increase.

[0110] The application reduces the risk of forgetting existing knowledge of the model without increasing the training cost of the model by introducing the elastic weight integration algorithm, thereby further improving the training efficiency of the model and improving the recognition accuracy of the model.

[0111] Embodiment two

[0112] As Figs. 1-2As shown, an electronic device proposed in the present invention includes a memory and a processor, characterized in that the memory is used to store a program that supports the processor to execute the low-resource language and text recognition method based on sample contribution perception according to any one of claims 1-2, and the processor is configured to execute the program stored in the memory.

[0113] Example 3

[0114] like Figs. 1-2 As shown, the present invention proposes a computer-readable storage medium for running the electronic device according to claim 3, characterized in that a computer program is stored on the computer-readable storage medium, and the computer program executes the steps in the low-resource language and text recognition method based on sample contribution perception when the processor runs the computer program.

[0115] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A method for low resource language OCR based on sample contribution awareness, the method comprising: The method comprises the following steps: S1, acquiring a low-resource language text image to be recognized to form a training image set D = {x1,..., x m ,...,x M} wherein x m represents the mth text image, M represents the number of D Chinese text images; A small number of images in the set D are labeled with recognition labels to obtain a labeled training image set wherein J represents D label the number of images in represents D label the jthtext image in b j represents the recognition result label corresponding to the jth text image; The remaining unlabeled samples constitute a set wherein R0represents the number of images in denotes the rth text image in , R0 = M - J denotes the number of images in S2, constructing a text image recognition network M(·), comprising: a visual feature extraction module, a Transformer decoding module, and a loss function calculation module; Each sample in D label is input into the visual feature extraction module The visual feature is extracted by the visual feature extraction module is input into the Transformer decoding module The decoded character probability sequence prediction result is obtained is input into the Transformer decoding module is compared with b j , used to calculate the loss function and perform back propagation, update the parameters in M(·), and obtain the initial recognition network M0(·) During training, the Fisher information matrix F0 of each parameter in the network is calculated by an elastic weight integration algorithm; S3, for i = [1,..., I max ], iteratively input samples in into the recognition network M i-1 (·), calculate the uncertainty of each sample on M i-1 (·); Combined sample diversity selection Middle to M i-1 The K that contributes most to the training of (·) i The sample set consists of samples where K i is the number of samples selected for the i-th step, I max is the maximum number of iterations; K i After annotating the K samples, a set is obtained wherein is S i the kth sample in d max,k For a corresponding recognition result label; S4, S i Input M i-1 In (·), according to Fisher information matrix F i-1 Different degrees of constraints are set for network node parameters, and the incremental learning is trained by using the elastic weight integration method i-1 (·), the network M i (·) trained in the i th step is obtained, and the Fisher information matrix F i of the i th step is calculated; S' = S i from S After removing, get the remaining unlabeled sample set Calculate the labeling cost of the i-th step C i = C i-1 + c K i Where c is the cost of labeling a sample; If, C i + c > C max , then stop iteration, return M i (·), otherwise perform the next iteration; When performing the next iteration, If C i + c K i > C max , Let Else let K i+1 = K i , where C max denotes the maximum annotation cost; S5, after the iteration stops, using the trained text image model to predict any input text image X to obtain the recognition result of X.

2. The low-resource language OCR method based on sample contribution awareness according to claim 1, characterized in that, The S3 comprises the following steps: S3.1: inputting each sample in the recognition network M i-1 forward propagation in (·)​ obtained a set of prediction results wherein, representing the model M i-1 (·) the prediction results of the model M as pseudo labels for the samples ; S3.2: For r = [1,..., R i-1 ], From Computing Gradient embedding vectors for the center samples wherein represents M i-1 (·) parameters of the last layer, indicates that input M i-1 the results obtained, denotes that the is a label, will be described below. input network M i-1 the identification loss function of the network calculated after (·) representing a gradient embedding vector; S3.3: Set samples with maximum gradient embedding vector norm Add to set S' i in the middle; For k = [2,..., K i ], compare the norm of the difference between the gradient embedding of the sample in S i and the gradient embedding of the existing samples in S and select the sample with the largest norm difference i to join S When k = K i then, annotating the samples in S i , obtaining S i .

3. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store a program supporting the processor to execute the low-resource language image-text recognition method based on sample contribution awareness according to any one of claims 1-2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium for running the electronic device according to claim 3, characterized in that, A computer program is stored on a computer readable storage medium, and the computer program is run by a processor to execute the steps in the low-resource language image-text recognition method based on sample contribution awareness.

Images