Model training method and system based on reinforcement learning, readable medium and equipment

Abstract

The invention relates to a model training method and system based on reinforcement learning, a readable medium and equipment, and relates to the technical field of image processing. According to the method, in the training process of a character recognition model, a prediction text output by a strategy model to a degraded sample can be obtained; calculating an initial reward value corresponding to each degraded sample based on the predicted text and a reference text corresponding to the original image; on the basis, relative advantages are calculated based on the initial reward values of the degradation samples in the training group, and the strategy model is updated based on the relative advantages to obtain a character recognition model; wherein the degraded sample is obtained by performing noise injection on the original image, and the training group comprises at least one degraded sample. According to the scheme, the model is updated based on the relative advantage evaluation of the degraded samples in the training group, so that the model can autonomously identify and suppress the noise influence, the dependence on the data quality is efficiently reduced at low cost, the model robustness is improved, and the computing resource consumption is reduced.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a reinforcement learning-based model training method, a reinforcement learning-based model training system, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Currently, OCR (Optical Character Recognition) model training typically relies on supervised learning methods, learning the mapping relationship between images and text on a large number of labeled samples. However, in practical applications, the training is hampered by the cost of acquiring labeled samples and the quality of the data. Especially dependent on data quality, OCR errors, image degradation, or labeling biases caused by subjective human perception can severely impact model training performance.

[0003] To address the aforementioned issues, methods such as preprocessing, data augmentation, and self-training can be used to improve data quality. However, these methods typically suffer from high costs, low efficiency, additional burdens, or negative impacts on model training performance.

[0004] Therefore, how to reduce the interference of sample noise in a low-cost and efficient manner and improve the model training effect of OCR is a technical challenge in this field.

[0005] It should be noted that the information in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore it may include not only information that constitutes prior art known to those skilled in the art, but also information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a reinforcement learning-based model training method, a reinforcement learning-based model training system, a computer-readable storage medium, and an electronic device, thereby overcoming, to at least some extent, the problem of difficulty in reducing sample noise interference and improving model training effect in OCR model training at low cost and high efficiency.

[0007] According to one aspect of this disclosure, a reinforcement learning-based model training method is provided, which may include: obtaining predicted text output by a policy model for degraded samples; obtaining degraded samples by injecting noise into an original image; calculating an initial reward value corresponding to each degraded sample based on the predicted text and reference text corresponding to the original image; calculating a relative advantage in a training group based on the initial reward values ​​of each degraded sample; the training group includes at least one degraded sample; and updating the policy model based on the relative advantage to obtain a text recognition model.

[0008] In an optional exemplary embodiment of this disclosure, before obtaining the predicted text output by the policy model for the degraded sample, the method further includes: performing noise injection on the original image based on noise parameters to obtain the corresponding degraded sample; the noise parameters include at least one of injection method and noise intensity; constructing a training group based on the degraded sample; at least one of the original image and noise parameters is different between each degraded sample.

[0009] In an optional exemplary embodiment of this disclosure, the injection method includes at least one of morphological transformation, random pixel replacement, and blurring.

[0010] In an optional exemplary embodiment of this disclosure, the noise intensity decreases as the training process progresses.

[0011] In an optional exemplary embodiment of this disclosure, calculating the initial reward value corresponding to each degraded sample based on the predicted text and the reference text corresponding to the original image includes: obtaining the reference text corresponding to the original image; and calculating the prediction accuracy corresponding to the predicted text based on the reference text to obtain the initial reward value corresponding to the degraded sample.

[0012] In an optional exemplary embodiment of this disclosure, obtaining the initial reward value corresponding to the degraded sample based on the prediction accuracy corresponding to the reference text includes: calculating the prediction error of the predicted text relative to the reference text; the prediction error includes at least one of the number of character substitution errors, the number of character deletion errors, and the number of character insertion errors; and obtaining the initial reward value corresponding to the degraded sample based on the prediction accuracy corresponding to the predicted text calculated based on the total number of characters in the reference text and the prediction error.

[0013] In an optional exemplary embodiment of this disclosure, calculating the relative advantage based on the initial reward value of each degenerate sample in the training group includes: calculating the reward mean and standard deviation of each degenerate sample in the training group using the initial reward value; and normalizing the initial reward value of each degenerate sample based on the reward mean and standard deviation to obtain the relative advantage.

[0014] In an optional exemplary embodiment of this disclosure, updating the policy model based on relative advantage to obtain a character recognition model includes: updating the policy model based on relative advantage under an iterative suppression policy to obtain a character recognition model.

[0015] In an optional exemplary embodiment of this disclosure, the iterative suppression strategy includes at least one of the following: information divergence constraint on the strategy model based on a reference model; and a pruning mechanism for fixing the parameters of the reference model to adjust the magnitude of each update of the strategy model.

[0016] In an optional exemplary embodiment of this disclosure, the policy model is obtained by supervised fine-tuning of a pre-trained model.

[0017] According to one aspect of this disclosure, a reinforcement learning-based model training system is provided. This system may include: a sample prediction module for acquiring predicted text output by a policy model for degraded samples; the degraded samples are obtained by injecting noise into the original image; a reward statistics module for calculating an initial reward value corresponding to each degraded sample based on the predicted text and reference text corresponding to the original image; an advantage calculation module for calculating a relative advantage in the training group based on the initial reward values ​​of each degraded sample; the training group includes at least one degraded sample; and a model update module for updating the policy model based on the relative advantage to obtain a text recognition model.

[0018] In an optional exemplary embodiment of this disclosure, the system may further include a group construction module for performing noise injection on the original image based on noise parameters to obtain corresponding degraded samples; the noise parameters include at least one of injection method and noise intensity; a training group is constructed based on the degraded samples; at least one of the original image and noise parameters is different between each degraded sample.

[0019] In an optional exemplary embodiment of this disclosure, the injection method includes at least one of morphological transformation, random pixel replacement, and blurring.

[0020] In an optional exemplary embodiment of this disclosure, the noise intensity decreases as the training process progresses.

[0021] In an optional exemplary embodiment of this disclosure, the reward statistics module is specifically used to obtain the reference text corresponding to the original image; and to obtain the initial reward value corresponding to the degraded sample based on the prediction accuracy of the predicted text according to the reference text.

[0022] In an optional exemplary embodiment of this disclosure, the reward statistics module is specifically used to count the prediction error of the predicted text relative to the reference text; the prediction error includes at least one of the number of character substitution errors, the number of character deletion errors, and the number of character insertion errors; based on the total number of characters in the reference text and the prediction error, the prediction accuracy corresponding to the predicted text is calculated to obtain the initial reward value corresponding to the degraded sample.

[0023] In an optional exemplary embodiment of this disclosure, the advantage calculation module is specifically used to calculate the mean and standard deviation of the reward for each degenerate sample in the training group based on the initial reward value; and to normalize the initial reward value of each degenerate sample based on the mean and standard deviation of the reward to obtain a relative advantage.

[0024] In an optional exemplary embodiment of this disclosure, the model update module is specifically used to update the strategy model based on relative advantage under the iterative suppression strategy to obtain a text recognition model.

[0025] In an optional exemplary embodiment of this disclosure, the iterative suppression strategy includes at least one of the following: information divergence constraint on the strategy model based on a reference model; and a pruning mechanism for fixing the parameters of the reference model to adjust the magnitude of each update of the strategy model.

[0026] In an optional exemplary embodiment of this disclosure, the policy model is obtained by supervised fine-tuning of a pre-trained model.

[0027] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described reinforcement learning-based model training method.

[0028] According to one aspect of this disclosure, an electronic device is provided, comprising: The processor; and memory for storing the processor's executable instructions; The processor is configured to execute the aforementioned reinforcement learning-based model training method by executing executable instructions.

[0029] This disclosure provides a reinforcement learning-based model training method, a reinforcement learning-based model training system, a computer-readable storage medium, and an electronic device. In the training process of a text recognition model, this method can obtain the predicted text output by the policy model for degraded samples; then, based on the predicted text and the reference text corresponding to the original image, it calculates the initial reward value corresponding to each degraded sample; based on this, it calculates the relative advantage in the training group based on the initial reward values ​​of each degraded sample, and updates the policy model based on the relative advantage to obtain the text recognition model; wherein, the degraded samples are obtained by injecting noise into the original image, and the training group includes at least one degraded sample. This scheme trains the character recognition model based on group relative strategy optimization. It updates the model based on the relative advantage evaluation of degraded samples within the training group, enabling the character recognition model to fully learn autonomous recognition and suppress the influence of noise in the samples. This efficiently and cost-effectively reduces dependence on sample data quality and improves the model's robustness to noise. Furthermore, this scheme only requires training the policy model during training, and updates it based on relative advantage. This accelerates the model's convergence speed in complex scenarios and reduces computational resource consumption during training, effectively lowering the training efficiency and cost of the character recognition model.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0032] Figure 1 An exemplary flowchart illustrating the steps of a reinforcement learning-based model training method provided in an embodiment of this disclosure is shown.

[0033] Figure 2 An exemplary flowchart illustrates the architecture of a reinforcement learning-based model training method provided in an embodiment of this disclosure.

[0034] Figure 3 An exemplary structural diagram of a reinforcement learning-based model training system provided in an embodiment of this disclosure is shown.

[0035] Figure 4 An electronic device for implementing a reinforcement learning-based model training method is illustrated by way of example embodiment of the present disclosure. Detailed Implementation

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0037] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0038] Figure 1This is a schematic flowchart illustrating the steps of a reinforcement learning-based model training method provided in an embodiment of this disclosure. Figure 1 As shown, the method may include the following steps 101 to 104.

[0039] Step 101: Obtain the predicted text output by the strategy model for the degraded samples; the degraded samples are obtained by injecting noise into the original image.

[0040] In this embodiment, the policy model can refer to a pre-trained model capable of extracting text information from image data, thereby matching business requirements in the subsequent model update process for text recognition inference tasks. The original image can refer to image data containing text information to support the policy model's ability to learn the mapping and transformation between images and text information. Degraded samples can be obtained by injecting noise into the original image. Degraded samples can simulate the situation where the original image contains noise information, thereby guiding the policy model to identify and suppress the interference of noise information during training, and output inference results that meet expectations.

[0041] In this embodiment of the disclosure, the predicted text is the text information output by the policy model through inference and prediction of degraded samples. Under the interference of noise, the predicted text may have a certain deviation. Based on this deviation, the parameters of the policy model can be updated to improve the robustness of the model's text recognition ability under noise interference.

[0042] Step 102: Calculate the initial reward value for each degraded sample based on the predicted text and the reference text corresponding to the original image.

[0043] In this embodiment of the disclosure, the reference text can be the text information annotated on the original image, representing the actual text content contained in the original image. Generally, the closer the predicted text output by the policy model for degraded samples is to the reference text, the more accurate the text information extracted by the policy model under noise interference is, thus indicating that the policy model has a certain ability to identify and suppress noise; conversely, the less accurate the text information extracted by the policy model under noise interference is, thus indicating that the policy model has a lower ability to identify and suppress noise.

[0044] Step 103: Calculate the relative advantage in the training group based on the initial reward value of each degenerate sample; the training group contains at least one degenerate sample.

[0045] In this embodiment of the disclosure, degraded samples can be assigned to corresponding training groups, and a training group can contain at least one degraded sample. The types of degraded samples can be categorized based on data source, generation method, etc. For example, a training group may contain only a degraded sample corresponding to one original image, thus ensuring that degraded samples within the same training group correspond to the same reference text, facilitating subsequent processing and analysis; or, a training group may include degraded samples corresponding to different original images to enhance the generalization ability of the text recognition model to different business scenarios.

[0046] In this embodiment, within the training group, each degraded sample has a corresponding initial reward value, representing the accuracy of the predicted text relative to the reference text. This allows for the calculation of the relative advantage among predicted texts within the group based on the initial reward value. The relative advantage compares the predicted texts of different pairs of degraded samples within the training group, simplifying the training process compared to traditional absolute value evaluation schemes, improving computational efficiency, and maintaining training stability. The relative advantage can be used to evaluate the performance of the policy model in noise recognition and suppression across different degraded samples; for example, a larger relative advantage indicates better noise recognition and suppression performance on degraded samples, and vice versa. Furthermore, the relative advantage is comparable across training groups.

[0047] Step 104: Update the strategy model based on relative advantage to obtain the text recognition model.

[0048] In this embodiment, relative advantage can guide the update direction of the strategy model, improving its noise recognition and suppression capabilities until convergence conditions are met, thereby obtaining a text recognition model. The convergence conditions may include restrictions on the number of updates, update magnitude, etc., or restrictions on model performance, such as the number of updates equaling a convergence threshold, the update magnitude being less than or equal to a convergence magnitude threshold, and the noise suppression rate being greater than or equal to a convergence performance threshold. The noise suppression rate can be calculated based on the distribution of the initial reward value, or other methods can be used for calculation; this embodiment does not impose specific limitations on this.

[0049] In an optional embodiment of the method disclosed herein, the preceding steps 101 may include steps A1 to A2.

[0050] Step A1: Perform noise injection on the original image based on noise parameters to obtain the corresponding degraded sample; the noise parameters include at least one of the injection method and noise intensity.

[0051] In this embodiment of the disclosure, the degradation sample is obtained by injecting noise into the original image, which can be achieved based on noise parameters. These noise parameters may include the injection method, noise intensity, etc. The injection method can define the technical means used to perform the noise injection, and the noise intensity can define the degree of transformation of the degradation sample relative to the original image.

[0052] In this embodiment of the disclosure, random sampling can be performed on the prepared image data to obtain the original image, and then a picture sample can be generated based on the original image.

[0053] In an optional embodiment of the method disclosed herein, the injection method includes at least one of morphological transformation, random pixel replacement, and blurring.

[0054] In this embodiment of the disclosure, the injection method can be morphological transformation. In morphological transformation, the effect of text lines being deformed can be simulated by dilating and eroding the original image, such as thickening or thinning the text lines. Specific operations can be shown in the following formulas (1) and / or (2):

[0055] In the above formulas (1) and (2), I is the image matrix of the original image; K is the structuring element, usually a 3×3 rectangle or circle; i and j can be the anchor points of the structuring element.

[0056] In this embodiment of the disclosure, the injection method can be pixel dropout. In pixel dropout, some pixels in the original image can be randomly replaced with black or white pixels to simulate the effect of text lines being occluded or faded. Specifically, it is shown in the following formula (3):

[0057] in, The probability that a black pixel is replaced; The probability of a white pixel being replaced.

[0058] In this embodiment of the disclosure, the injection method can be blurring. Blurring can simulate image degradation effects through Gaussian blur and motion blur. It can be shown in the following formula (4):

[0059] Where kernel is a Gaussian kernel; i and j can be the positions of the center point; and Z is the normalization factor.

[0060] Step A2: Construct a training group based on the degraded samples; at least one of the original image and noise parameters is different between each degraded sample.

[0061] In this embodiment of the disclosure, a training group can be further constructed based on the obtained degraded samples. A training group can contain a certain number of degraded samples, typically 4 to 16. The training group may also include reference samples of the original images corresponding to the degraded samples to facilitate rapid calculation of the initial reward value. The training group may contain one or more degraded samples, and each degraded sample may differ in the original image used for generation, the noise parameters, or both the original image and the noise parameters. When the noise parameters differ, it may be due to different injection methods, different noise intensities, or both injection methods and noise intensities. This embodiment of the disclosure does not impose specific limitations in this regard.

[0062] In an optional method embodiment of this disclosure, the noise intensity decreases as the training process progresses.

[0063] In this embodiment of the disclosure, the training group can also dynamically adjust the noise intensity as the training process progresses. In the early stage of training, degenerate samples with higher noise intensity can be used, and as the training process progresses, degenerate samples with lower noise intensity can be gradually collected, so that the noise intensity decreases as the training process progresses.

[0064] In an optional method embodiment of this disclosure, the policy model is obtained by supervised fine-tuning of a pre-trained model.

[0065] In this embodiment, the strategy model can be selected from an open-source OCR model, or it can be obtained by supervised fine-tuning (SFT) of a selected pre-trained model. SFT adjusts the parameters of the pre-trained model using a small amount of labeled data to establish a targeted basic text recognition capability. Based on this, the model training method provided in this embodiment includes a supervised fine-tuning stage and a reinforcement learning stage. It can first establish a targeted basic recognition capability, and then further improve the model performance through reinforcement learning to better adapt to business needs.

[0066] In an optional embodiment of the method disclosed herein, step 102 may include steps B1 to B2.

[0067] Step B1: Obtain the reference text corresponding to the original image.

[0068] In this embodiment of the disclosure, the reference text can be referred to the relevant description of step 102 above. To avoid repetition, it will not be repeated here.

[0069] Step B2: Based on the reference text, calculate the prediction accuracy of the predicted text and obtain the initial reward value corresponding to the degraded sample.

[0070] In the disclosed embodiments, the initial reward value can represent the prediction accuracy of the predicted text relative to the reference text. Based on this, the reward function can be designed according to actual calculation conditions and needs. For example, the reward function can be designed to represent the accuracy through the ratio of the number of characters, or it can be designed to represent the accuracy through the relationship of character order, position, combination, etc. This disclosed embodiment does not impose specific limitations on this.

[0071] In an optional embodiment of the method disclosed herein, step B2 includes the following steps B21 to B22.

[0072] Step B21: Calculate the prediction error of the predicted text relative to the reference text; the prediction error includes at least one of the following: the number of character substitution errors, the number of character deletion errors, and the number of character insertion errors.

[0073] In this embodiment of the disclosure, the prediction accuracy can be evaluated by statistically analyzing the prediction error of the predicted text relative to the reference text. This prediction error can be characterized by the number of erroneous characters identified, analyzed, and processed by OCR. For example, the prediction error may include the number of character substitution errors (the number of characters in the predicted text that incorrectly replace the corresponding characters in the reference text); the number of character deletion errors (the number of characters in the predicted text that incorrectly delete the corresponding characters in the reference text); and the number of character insertion errors (the number of characters in the predicted text that incorrectly insert characters other than those in the reference text). The above prediction errors are merely examples, and those skilled in the art can select or expand appropriate prediction errors according to actual needs.

[0074] Step B22: Based on the total number of characters in the reference text and the prediction error statistics, the prediction accuracy of the predicted text is calculated to obtain the initial reward value corresponding to the degraded sample.

[0075] In this embodiment of the disclosure, based on the determined prediction error, the magnitude of the prediction error can be evaluated based on the total number of characters in the reference text, thereby obtaining the prediction accuracy as the initial reward value corresponding to the degraded sample. The prediction accuracy can be calculated as a proportion of the number of characters, such as the proportion of character substitution errors, character deletion errors, and character insertion errors to the total number of characters in the reference text; a larger proportion indicates a lower prediction accuracy. Alternatively, it can be the proportion of characters other than character substitution errors, character deletion errors, and character insertion errors to the total number of characters in the reference text; a larger proportion indicates a higher prediction accuracy.

[0076] For example, for the i-th degenerate sample, the reward function for the initial reward value is designed as shown in the following formula (5):

[0077] In the above formula (5), for the i-th degenerate sample, R iLet C be the initial reward value for the i-th degenerate sample. i Total number of characters in the reference text, S i D is the number of character replacement errors. i For the number of character deletion errors, I i Count the number of character insertion errors.

[0078] In an optional embodiment of the method disclosed herein, step 103 may include steps C1 to C2.

[0079] Step C1: In the training group, calculate the mean and standard deviation of the reward based on the initial reward value of each degenerate sample.

[0080] Step C2: Normalize the initial reward value of each degraded sample based on the reward mean and standard deviation to obtain the relative advantage.

[0081] In this embodiment of the disclosure, the relative advantage can be calculated by normalizing the standard deviation of the initial reward values ​​of each degenerate sample in the training group. That is, the mean reward and standard deviation of each degenerate sample in the training group are first determined, and then the mean reward is subtracted from the initial reward value of each degenerate sample, and then normalized using the standard deviation. For example, as shown in the following formula (6):

[0082] In the above formula (6), The relative advantage of the i-th degraded sample is... For the average reward, The standard deviation is denoted as .

[0083] In an optional embodiment of the method disclosed herein, step 104 may include step D.

[0084] Step D: Under the iterative suppression strategy, the strategy model is updated based on the relative advantage to obtain the character recognition model.

[0085] In this embodiment, step D can be referred to the relevant description of step 104 above, and will not be repeated here to avoid repetition. The update based on the relative policy model can be implemented under the constraint of an iterative suppression strategy. The iterative suppression strategy can suppress the update magnitude of the policy model in each iteration, relative to the total update magnitude in the early training stage, etc., to avoid excessive deviation of the policy.

[0086] In an optional embodiment of the method disclosed herein, the iterative suppression strategy includes information divergence constraints on the strategy model based on a reference model; the parameters of the reference model are fixed.

[0087] In this embodiment of the disclosure, the reference model can be obtained by fixing the model parameters before the policy model starts training, thereby representing the initial state of the policy model. Based on this, the iterative suppression policy can constrain the policy model to the difference between the reference model and the information divergence to prevent excessive shift of the policy model during training. For example, the information divergence can be calculated as shown in the following formula (7):

[0088] In the above formula (7), For information divergence, For the strategy model, This is a reference model.

[0089] In an optional embodiment of the method disclosed herein, the iterative suppression strategy includes a pruning mechanism for the magnitude of each update of the strategy model.

[0090] In this embodiment of the disclosure, a pruning mechanism can also be set for the update magnitude of each finer iteration to limit the change range of the probability ratio and prevent the problem of training instability caused by excessive update magnitude. For example, the pruning mechanism can be shown in the following formula (8):

[0091] In the above formula (8), This is the trimming parameter, typically set to 0.1~0.2; This is the strategy model for this update; This is the strategy model from the last update.

[0092] Under the iterative suppression strategy, the objective function of the strategy model can be designed as shown in the following formula (9):

[0093] In the above formula (9), The learning loss is used to reinforce the training; G is the number of degenerate samples in the training group; This is the regularization coefficient for information divergence, typically taken as 0.01 to 0.05.

[0094] For example, Figure 2 The architecture flowchart of the reinforcement learning-based model training method provided in the embodiments of this disclosure is as follows: Figure 2 As shown, preparation includes a strategy model. and reference model For example, the policy model can adopt a CRNN architecture, which includes a CNN feature extraction layer, a BiLSTM sequence processing layer, and a CTC transcription layer. The policy model can be obtained by supervised fine-tuning of the OCR model, the objective of which is to minimize the CTC loss, as shown in the following formula (10):

[0095] In the above formula (10), T is the number of time steps, and x is the image feature. Let t be the t-th character of the reference text. The supervised fine-tuned OCR model can be used as a trainable policy model, and its parameters can be frozen and fixed as a reference model. Based on this, the hyperparameters of the policy model are set, including the number of degenerate samples in the training group G = 8, the pruning parameter ε = 0.1, the regularization coefficient of information divergence β = 0.04, the learning rate α = 5e-6, and the maximum number of iterations 200.

[0096] In supervised fine-tuning, a reinforcement learning training process can be performed by randomly sampling the original image and injecting noise I to generate G degraded samples with different noise intensities [I1, I2, ..., I...]. G Obtain the training group. The training group may also include the reference text corresponding to the original image.

[0097] The strategy model outputs predicted text for each degraded sample, and calculates the initial reward value [R1, R2, ..., R] corresponding to each degraded sample according to the aforementioned formula (5). G ].

[0098] Based on the initial reward value, the relative advantage [A1, A2, ..., A1] corresponding to each degraded sample is calculated according to the aforementioned formula (6). G ].

[0099] The reinforcement learning loss and information divergence are calculated using the aforementioned formulas (7) and (9).

[0100] Using gradient descent ▽L GRPO Update the parameters of the strategy model until the convergence condition is met to obtain the character recognition model.

[0101] This disclosure provides a reinforcement learning-based model training method. During the training of a text recognition model, this method obtains the predicted text output by the policy model for degraded samples. Then, based on the predicted text and the reference text corresponding to the original image, it calculates the initial reward value for each degraded sample. Based on this, it calculates the relative advantage in the training group based on the initial reward values ​​of each degraded sample, and updates the policy model based on the relative advantage to obtain the text recognition model. The degraded samples are obtained by injecting noise into the original image, and the training group includes at least one degraded sample. This scheme trains the text recognition model based on the group's relative policy optimization, and updates the model based on the relative advantage evaluation of degraded samples within the training group. This enables the text recognition model to fully learn the ability to autonomously recognize and suppress the influence of noise in the samples, efficiently and cost-effectively reducing the dependence on sample data quality and improving the model's robustness to noise. Furthermore, this scheme only requires training the policy model during training and updates it based on the relative advantage. This accelerates the convergence speed of the model in complex scenarios and reduces the consumption of computational resources during training, effectively reducing the training efficiency and cost of the text recognition model.

[0102] The following are system embodiments of this disclosure, which can be used to execute the method embodiments of this disclosure. For details not disclosed in the device embodiments of this disclosure, please refer to the method embodiments of this disclosure.

[0103] This disclosure also provides an example structural diagram of a reinforcement learning-based model training system 300. Specifically, refer to... Figure 3 As shown, the system 300 may include a sample prediction module 301, used to obtain the predicted text output by the policy model for degraded samples; degraded samples are obtained by injecting noise into the original image; a reward statistics module 302, used to calculate the initial reward value corresponding to each degraded sample based on the predicted text and the reference text corresponding to the original image; an advantage calculation module 303, used to calculate the relative advantage in the training group based on the initial reward value of each degraded sample; the training group contains at least one degraded sample; and a model update module 304, used to update the policy model based on the relative advantage to obtain a text recognition model.

[0104] In an optional system embodiment of this disclosure, the system may further include a group construction module for performing noise injection on the original image based on noise parameters to obtain corresponding degraded samples; the noise parameters include at least one of injection method and noise intensity; a training group is constructed based on the degraded samples; at least one of the original image and noise parameters is different between each degraded sample.

[0105] In an optional system embodiment of this disclosure, the injection method includes at least one of morphological transformation, random pixel replacement, and blurring.

[0106] In an optional system embodiment of this disclosure, the noise intensity decreases as the training process progresses.

[0107] In an optional system embodiment of this disclosure, the reward statistics module 302 is specifically used to obtain the reference text corresponding to the original image; based on the reference text, the prediction accuracy corresponding to the predicted text is statistically predicted to obtain the initial reward value corresponding to the degraded sample.

[0108] In an optional system embodiment of this disclosure, the reward statistics module 302 is specifically used to count the prediction error of the predicted text relative to the reference text; the prediction error includes at least one of the number of character replacement errors, the number of character deletion errors, and the number of character insertion errors; based on the total number of characters in the reference text and the prediction error, the prediction accuracy corresponding to the predicted text is calculated, and the initial reward value corresponding to the degraded sample is obtained.

[0109] In an optional system embodiment of this disclosure, the advantage calculation module 303 is specifically used to calculate the mean and standard deviation of the reward based on the initial reward value of each degenerate sample in the training group; and to normalize the initial reward value of each degenerate sample based on the mean and standard deviation of the reward to obtain a relative advantage.

[0110] In an optional system embodiment of this disclosure, the model update module 304 is specifically used to update the strategy model based on relative advantage under the iterative suppression strategy to obtain a text recognition model.

[0111] In an optional system embodiment of this disclosure, the iterative suppression strategy includes at least one of the following: information divergence constraint on the strategy model based on the reference model; and a pruning mechanism for fixing the parameters of the reference model to adjust the magnitude of each update of the strategy model.

[0112] In an optional system embodiment of this disclosure, the policy model is obtained by supervised fine-tuning of a pre-trained model.

[0113] This disclosure provides a reinforcement learning-based model training system. During the training of a text recognition model, the system acquires the predicted text output by the policy model for degraded samples. Then, based on the predicted text and the reference text corresponding to the original image, it calculates the initial reward value for each degraded sample. Based on this, it calculates the relative advantage in the training group based on the initial reward values ​​of each degraded sample, and updates the policy model based on the relative advantage to obtain the text recognition model. The degraded samples are obtained by injecting noise into the original image, and the training group includes at least one degraded sample. This scheme trains the text recognition model based on the group's relative policy optimization, and updates the model based on the relative advantage evaluation of degraded samples within the training group. This enables the text recognition model to fully learn the ability to autonomously recognize and suppress the influence of noise in the samples, efficiently and cost-effectively reducing the dependence on sample data quality and improving the model's robustness to noise. Furthermore, this scheme only requires training the policy model during training and updates it based on the relative advantage. This accelerates the convergence speed of the model in complex scenarios and reduces the consumption of computational resources during training, effectively reducing the training efficiency and cost of the text recognition model.

[0114] The specific details of each module in the above reinforcement learning-based model training system have been described in detail in the corresponding reinforcement learning-based model training methods, so they will not be repeated here.

[0115] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0116] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0117] In exemplary embodiments of this disclosure, an electronic device capable of implementing the above-described methods is also provided. Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0118] The following reference Figure 4 To describe an electronic device 400 according to such an embodiment of the present disclosure. Figure 4 The electronic device 400 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0119] like Figure 4 As shown, the electronic device 400 is manifested in the form of a general-purpose computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different system components (including storage unit 420 and processing unit 410), and a display unit 440.

[0120] The storage unit stores program code that can be executed by the processing unit 410, causing the processing unit 410 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 410 can perform actions such as... Figure 1 The reinforcement learning-based model training method shown is illustrated.

[0121] Storage unit 420 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 4201 and / or cache memory 4202, and may further include a read-only memory (ROM) 4203.

[0122] Storage unit 420 may also include a program / utility 4204 having a set (at least one) program module 4205, such program module 4205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0123] Bus 430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0124] Electronic device 400 can also communicate with one or more external devices 500 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 400, and / or with any device that enables electronic device 400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 450. Furthermore, electronic device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 460. As shown, network adapter 460 communicates with other modules of electronic device 400 via bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0125] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0126] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this disclosure described in the "Exemplary Methods" section above.

[0127] The program product for implementing the above-described method according to embodiments of the present disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0128] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0129] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0130] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0131] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0132] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0133] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not invented by this disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A model training method based on reinforcement learning, characterized in that, The method includes: Obtain the predicted text output by the strategy model for the degraded samples; the degraded samples are obtained by injecting noise into the original image; Calculate the initial reward value for each degraded sample based on the predicted text and the reference text corresponding to the original image; The relative advantage is calculated in the training group based on the initial reward value of each of the degenerate samples; the training group includes at least one of the degenerate samples; The strategy model is updated based on the relative advantages to obtain a text recognition model.

2. The method according to claim 1, characterized in that, Before the acquisition strategy model outputs the predicted text for degraded samples, it also includes: Noise injection is performed on the original image based on noise parameters to obtain the corresponding degraded sample; the noise parameters include at least one of injection method and noise intensity. The training group is constructed based on the degraded samples; at least one of the original image and the noise parameters is different between each degraded sample.

3. The method according to claim 2, characterized in that, The injection method includes at least one of morphological transformation, random pixel replacement, and blurring.

4. The method according to claim 2, characterized in that, The noise intensity decreases as the training progresses.

5. The method according to claim 1, characterized in that, The calculation of the initial reward value for each degraded sample based on the predicted text and the reference text corresponding to the original image includes: Obtain the reference text corresponding to the original image; Based on the reference text, the prediction accuracy corresponding to the predicted text is statistically analyzed, and the initial reward value corresponding to the degraded sample is obtained.

6. The method according to claim 5, characterized in that, The step of calculating the prediction accuracy of the predicted text based on the reference text and obtaining the initial reward value corresponding to the degraded sample includes: The prediction error of the predicted text relative to the reference text is calculated; the prediction error includes at least one of the following: the number of character substitution errors, the number of character deletion errors, and the number of character insertion errors. Based on the total number of characters in the reference text and the prediction error, the prediction accuracy corresponding to the predicted text is calculated, and the initial reward value corresponding to the degraded sample is obtained.

7. The method according to claim 1, characterized in that, The calculation of relative advantage based on the initial reward value of each of the degraded samples in the training group includes: In the training group, the mean and standard deviation of the reward are calculated using the initial reward value of each degraded sample; The initial reward value of each of the degraded samples is normalized based on the mean reward and the standard deviation to obtain the relative advantage.

8. The method according to claim 1, characterized in that, The step of updating the strategy model based on the relative advantage to obtain a text recognition model includes: Under the iterative suppression strategy, the strategy model is updated based on the relative advantage to obtain the character recognition model.

9. The method according to claim 8, characterized in that, The iterative suppression strategy includes at least one of the following: The information divergence of the policy model is constrained based on a reference model; the parameters of the reference model are fixed. A pruning mechanism for each update magnitude of the strategy model.

10. The method according to claim 1, characterized in that, The policy model is obtained by supervised fine-tuning of a pre-trained model.

11. A model training system based on reinforcement learning, characterized in that, The system includes: The sample prediction module is used to obtain the predicted text output by the policy model for the degraded samples; the degraded samples are obtained by injecting noise into the original image. The reward statistics module is used to calculate the initial reward value for each degraded sample based on the predicted text and the reference text corresponding to the original image. An advantage calculation module is used to calculate a relative advantage in a training group based on the initial reward value of each of the degenerate samples; the training group includes at least one of the degenerate samples; The model update module is used to update the strategy model based on the relative advantage to obtain a text recognition model.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the reinforcement learning-based model training method according to any one of claims 1-10.

13. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the reinforcement learning-based model training method according to any one of claims 1-10 by executing the executable instructions.

Images