Image recognition, model training method and device

Abstract

The present invention discloses an image recognition and model training method and apparatus. The method comprises: obtaining a character string in an image to be recognized; obtaining the edit distance of the character string, wherein the edit distance serves as a reward and penalty function; and performing a policy gradient calculation on the character string based on the reward and penalty function to obtain recognized text. This invention addresses the technical problem that, due to prior art practices where the accuracy of entire vocabulary prediction is not incorporated into OCR model training, the optimization objectives during the training and testing phases are inconsistent, resulting in reduced recognition performance.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to an image recognition and model training method and device. Background Art

[0002] In the current mainstream OCR process, the text area is first detected by the text detection model, and then input into the text recognition model for text recognition. The current mainstream text recognition model is mainly based on the encoder-decoder framework. The specific steps are as follows:

[0003] (1) Resize the original text area image to w*h.

[0004] (2) Feature extraction is performed through a convolutional neural network (CNN), and the output feature map size is generally w' * 1.

[0005] (3) This feature map is used to extract and encode the context features through a recurrent neural network (RNN, LSTM). The resulting feature is a sequence feature, {c_1, c_2, ..., c_{|w'|}}.

[0006] (4) At each moment, the LSTM outputs the current hidden state h_t through the sequence decoding network (with attention mechanism). This hidden state is then passed through a fully connected layer (FC) to output the probability of each word in the vocabulary:

[0007] y'_t = argmaxSoftmax(W * h_t)

[0008] Where W is the transformation matrix that maps the latent vector to the vocabulary, Softmax is the softmax function, and argmax is the maximum value function. Finally, we select the character with the highest probability as the predicted character at the current moment.

[0009] In the current solution described above, during model training, at time t (where t ranges from 0 to a set maximum number of time-steps), the hidden state h_t of the LSTM unit at time t outputs the probability y'_t of each word in the vocabulary. Using the one-hot vector y consisting of the ground truth words at that time, the cross entropy loss (loss_{h_t}) is calculated. This loss is then used to optimize the model using gradient descent.

[0010] Currently, no effective solution has been proposed to address the above-mentioned problem that the accuracy of the entire vocabulary prediction in the existing technology is not incorporated into the training of the OCR model, resulting in inconsistent optimization goals in the training and testing stages and reduced recognition performance. Summary of the Invention

[0011] The embodiments of the present invention provide an image recognition and model training method and apparatus to at least solve the technical problem that, because the accuracy of the existing technology for predicting the entire vocabulary is not incorporated into the training of the OCR model, the optimization goals of the training phase and the testing phase are inconsistent, resulting in reduced recognition performance.

[0012] According to one aspect of an embodiment of the present invention, an image recognition method is provided, comprising: obtaining a character string in an image to be recognized; obtaining an edit distance of the character string, wherein the edit distance serves as a reward and penalty function; and performing a policy gradient calculation on the character string based on the reward and penalty function to obtain recognized text.

[0013] Optionally, obtaining the edit distance of the string and using the edit distance as a reward and punishment function includes: obtaining the probability of each image in the training set being sampled, the probability of obtaining the string based on the image to be identified, and the reward value of the string; performing an expectation calculation based on the probability of each image being sampled, the probability of obtaining the string based on the image to be identified, and the reward value of the string to obtain the reward and punishment function.

[0014] Optionally, performing a policy gradient calculation on the character string based on the reward and punishment function to obtain the recognized text includes: sampling the character string based on a preset sampling algorithm and a reward and punishment function to obtain a first recognized text; performing a policy gradient calculation on the first recognized text to obtain a gradient value; optimizing the first recognized text based on the gradient value to obtain a second recognized text; and determining the second recognized text as the recognized text.

[0015] Optionally, before obtaining the edit distance of the character string, the method further includes: cropping and resizing the image to be identified; encoding the cropped and resized image to be identified to obtain an encoded character string; and decoding the encoded character string to obtain a decoded character string.

[0016] Further, optionally, cropping and resizing the image to be recognized includes: cropping and resizing the image to be recognized, and obtaining the height and width of the character string in the image to be recognized.

[0017] Optionally, decoding the encoded string to obtain the decoded string includes: decoding the encoded string through a neural network, and obtaining the decoded string.

[0018] According to another aspect of an embodiment of the present invention, a model training method is also provided, including: obtaining the edit distance of the character string in the image to be recognized; performing policy gradient calculation based on the edit distance and the character string to obtain a gradient value; and training the text recognition network model based on the gradient value to obtain an optimized text recognition network.

[0019] Optionally, performing a policy gradient calculation based on the edit distance and the string to obtain a gradient value includes: calculating a reward value of the string based on the edit distance and the string; calculating an expected value corresponding to the string based on the reward value; sampling the string using a preset sampling algorithm, and calculating a gradient based on the expected value to obtain a gradient value.

[0020] Furthermore, optionally, training the text recognition network model according to the gradient value to obtain the optimized text recognition network includes: training the text recognition network model according to the gradient value and the calculation formula of the gradient value to obtain the optimized text recognition network.

[0021] According to another aspect of an embodiment of the present invention, an image recognition method is also provided, including: obtaining a character string in an image to be recognized; obtaining an edit distance of the character string, wherein the edit distance serves as a reward and punishment function; performing a policy gradient calculation on the character string based on the reward and punishment function to obtain a recognized text; displaying the recognized text; receiving user opinion information returned based on the recognized text; and optimizing the recognized text based on the user opinion information.

[0022] According to another aspect of an embodiment of the present invention, an image recognition device is also provided, including: a first acquisition module for acquiring a character string in an image to be recognized; a second acquisition module for acquiring an edit distance of the character string, wherein the edit distance is used as a reward and punishment function; and a recognition module for performing a policy gradient calculation on the character string based on the reward and punishment function to obtain recognized text.

[0023] According to another aspect of an embodiment of the present invention, a model training device is also provided, including: an acquisition module for obtaining the edit distance of the character string in the image to be recognized; a calculation module for performing policy gradient calculation based on the edit distance and the character string to obtain a gradient value; and a training module for training the text recognition network model based on the gradient value to obtain an optimized text recognition network.

[0024] According to another aspect of an embodiment of the present invention, an image recognition device is provided, comprising: a first acquisition module for acquiring a character string in an image to be recognized; a second acquisition module for acquiring the edit distance of the character string, wherein the edit distance serves as a reward and penalty function; a recognition module for performing policy gradient calculation on the character string according to the reward and penalty function to obtain recognized text; a display module for displaying the recognized text; a receiving module for receiving user opinion information returned based on the recognized text; and an optimization module for optimizing the recognized text based on the user opinion information. According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above method.

[0025] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is configured to run a program, wherein the above method is executed when the program is run.

[0026] In an embodiment of the present invention, by obtaining a character string in an image to be recognized; obtaining the edit distance of the character string, wherein the edit distance serves as a reward and penalty function; and performing a policy gradient calculation on the character string based on the reward and penalty function to obtain the recognized text, the purpose of achieving end-to-end optimization of the text recognition model is achieved, thereby achieving the technical effect of improving recognition performance, and further solving the technical problem that the accuracy of the entire vocabulary prediction in the existing technology is not incorporated into the model training of the OCR model, resulting in inconsistent optimization targets in the training phase and the test phase, and reduced recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 This is a hardware structure block diagram of a computer terminal for an image recognition method according to an embodiment of the present invention;

[0029] Figure 2 is a flowchart of an image recognition method according to a first embodiment of the present invention;

[0030] Figure 3 1 is a flow chart of character string recognition in an image recognition method according to the first embodiment of the present invention;

[0031] Figure 4 1 is a flow chart of a model training method according to Embodiment 2 of the present invention;

[0032] Figure 5 is a flowchart of an image recognition method according to a third embodiment of the present invention;

[0033] Figure 6 is a schematic diagram of an image recognition device according to a fourth embodiment of the present invention;

[0034] Figure 7 is a schematic diagram of a model training device according to a fifth embodiment of the present invention;

[0035] Figure 8 FIG. 1 is a schematic diagram of an image recognition device according to a sixth embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] Technical terms involved in this application:

[0039] OCR: Optical Character Recognition

[0040] CNN: Convolution Neural Network

[0041] RNN: Recurrent Neural Network

[0042] LSTM: Long Short-Term Memory, long short-term memory neural network

[0043] FC: Fully Connected Network

[0044] Encoder-Decoder: encoding-decoding network, encoding-decoding network

[0045] Reward Function

[0046] Policy Gradient

[0047] Reinforcement Learning

[0048] Example 1

[0049] According to an embodiment of the present invention, an image recognition method embodiment is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a computer terminal for an image recognition method according to an embodiment of the present invention. Figure 1 As shown, the computer terminal 10 may include one or more (only one is shown) processors 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0051] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / modules corresponding to the image recognition method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the image recognition method of the above-mentioned application. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0052] Transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of computer terminal 10. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0053] Under the above operating environment, this application provides Figure 2 The image recognition method shown. Figure 2 FIG. 1 is a flow chart of an image recognition method according to the first embodiment of the present invention. Figure 2 As shown, the method includes the following steps:

[0054] Step S202, obtaining a character string in the image to be recognized;

[0055] In step S202 of the present application, the images to be identified may include: security, municipal administration, and transportation services, which require the use of massive amounts of camera video data. In addition to key objects of interest such as people, motor vehicles, and non-motor vehicles, camera video data also includes a large amount of text data. For example, the text on street banners and slogans is related to maintaining public order, changes to traffic signs require timely updates to road data on maps, and the extraction of street shop signs requires updates to the POI (point of interest) database. Therefore, further identifying and analyzing text information in camera point images can further increase the information dimension in the city brain and further enhance the level and effectiveness of video intelligent analysis.

[0056] The character string in the embodiment of the present application may include: text information in the image to be recognized; for example, the text mentioned in the camera point map, taking the text in the traffic sign as an example, when the traffic sign on the street is recognized to display "STOP", the obtained character string may be "S", "T", "O", "P".

[0057] Step S204, obtaining the edit distance of the character string, wherein the edit distance is used as a reward and penalty function;

[0058] In the above step S204 of the present application, the edit distance of the character string in the embodiment of the present application can be obtained by the following method:

[0059] (1) Hamming Distance: refers to the number of different characters in corresponding positions of two strings, that is, the number of characters that need to be replaced to transform one string into another.

[0060] (2) Levenshtein distance: refers to the minimum number of editing operations required to transform two strings from one to the other. Allowed editing operations include: replacing one character with another, inserting a character, and deleting a character.

[0061] (3) Longest Common Subsequence: refers to the longest subsequence among all sequences in a set of sequences (usually two sequences).

[0062] In addition to the three methods listed above for measuring the edit distance between two character sequences, there are also other measurement methods such as Damerau-Levenshtein distance and Jaro-Winkler distance.

[0063] Using any of the aforementioned algorithms for calculating edit distance, since edit distance loss is non-differentiable, we introduce the policy gradient algorithm from reinforcement learning to optimize the model end-to-end. By treating the edit distance as a reward function, the ultimate goal is to maximize the reward, i.e., minimize the sum of the edit distances of the training samples in the training set. To maximize the reward function, we use a negative sign, transforming the goal into maximizing the sum of negative edit distances.

[0064] In one feasible method, obtaining the edit distance of a string and using the edit distance as a reward and penalty function includes: obtaining the probability of each image in the training set being sampled, the probability of obtaining the string based on the image to be identified, and the reward value of the string; performing an expectation calculation based on the probability of each image being sampled, the probability of obtaining the string based on the image to be identified, and the reward value of the string to obtain the reward and penalty function.

[0065] The probability of obtaining each image in the training set being sampled can be recorded as P(I); the probability of obtaining a string based on the image to be recognized can be recorded as P(S pred |I), that is, in the case of image I, get sentence S pred The probability of recognizing a string can be expressed as R(I, S pred ).

[0066] Take the string "S", "T", "O", "P" as an example, according to P(I), P(S pred |I) and R(I, S pred ) is used to calculate the reward value of the string, and then the reward and punishment function is obtained.

[0067] Specifically, in the embodiment of the present application, the goal is to find a set of network parameters θ * , this parameter can maximize the final reward. The calculation process of the reward (i.e., the reward and punishment function in the embodiment of the present application) is as shown in formula (1). Formula (1) is as follows:

[0068]

[0069]

[0070] Step S206: Perform policy gradient calculation on the character string according to the reward and punishment function to obtain the recognized text.

[0071] In one feasible manner, performing a policy gradient calculation on a character string based on a reward and punishment function to obtain a recognized text includes: sampling the character string based on a preset sampling algorithm and a reward and punishment function to obtain a first recognized text; performing a policy gradient calculation on the first recognized text to obtain a gradient value; optimizing the first recognized text based on the gradient value to obtain a second recognized text; and determining the second recognized text as the recognized text.

[0072] The expectation is calculated based on the reward and penalty function obtained based on the reward value obtained from the character string in step S204. Specifically, the reward and penalty function and the expectation are combined. At the same time, because the true value of the expectation needs to be calculated, an exhaustive enumeration is required to calculate all possible recognized and generated texts under each image. In the embodiment of the present application, a Monte Carlo sampling algorithm is used to sample and obtain the recognized text (i.e., the first recognized text in the embodiment of the present application);

[0073] The first recognized text is subjected to policy gradient calculation to obtain the gradient value including:

[0074] According to the above formula (1), we can calculate the expectation and obtain (2). The specific formula (2) is as follows:

[0075]

[0076] Based on formula (2), the reward and punishment function and the expectation are combined, and formula (3) is used. At the same time, because if the true value of this expectation is to be calculated, it is necessary to exhaustively calculate all possible recognized and generated texts under each image. In the embodiment of the present application, the Monte Carlo sampling algorithm is used to sample and obtain the recognized text (that is, the first recognized text in the embodiment of the present application); in the embodiment of the present application, formula (3) is specifically as follows:

[0077]

[0078] After obtaining the approximate form of the maximum reward and punishment function, the gradient is calculated. The calculation process is as follows:

[0079]

[0080]

[0081] The gradient value is calculated according to formulas (4) and (5), and then the text recognition network model is trained and optimized end-to-end based on the gradient value to obtain the second recognized text, and the recognized text is determined based on the second recognized text.

[0082] In an embodiment of the present invention, by obtaining a character string in an image to be recognized; obtaining the edit distance of the character string, wherein the edit distance serves as a reward and penalty function; and performing a policy gradient calculation on the character string based on the reward and penalty function to obtain the recognized text, the purpose of achieving end-to-end optimization of the text recognition model is achieved, thereby achieving the technical effect of improving recognition performance, and further solving the technical problem that the accuracy of the entire vocabulary prediction in the existing technology is not incorporated into the model training of the OCR model, resulting in inconsistent optimization targets in the training phase and the test phase, and reduced recognition performance.

[0083] Optionally, before obtaining the edit distance of the character string in step S204, the method further includes: cropping and resizing the image to be identified; encoding the cropped and resized image to be identified to obtain an encoded character string; and decoding the encoded character string to obtain a decoded character string.

[0084] Further, optionally, cropping and resizing the image to be recognized includes: cropping and resizing the image to be recognized, and obtaining the height and width of the character string in the image to be recognized.

[0085] Optionally, decoding the encoded string to obtain the decoded string includes: decoding the encoded string through a neural network, and obtaining the decoded string.

[0086] Specifically, Figure 3 FIG. 1 is a flow chart of character string recognition in an image recognition method according to the first embodiment of the present invention. Figure 3 As shown, the image to be identified is cropped and the size of the image is adjusted to obtain a cropped image to be identified, wherein the size of the image is marked by the height and width (such as Figure 3 h and w shown in ), the image to be recognized after cropping and adjusting the size of the image is encoded through convolution calculation to obtain the encoded string, which is decoded through the LSTM neural network to obtain the decoded string (as shown in Figure 3 ), based on the policy gradient algorithm entered in steps S202 to S206 above, it breaks through the limitation of the non-differentiable nature of the entry recognition rate (edit distance) and is not affected by the specific mathematical form of the edit distance, thus achieving end-to-end optimization of the text recognition model.

[0087] The image recognition method provided in this embodiment integrates the text recognition rate (edit distance) into the training of advanced mainstream text recognition models. Furthermore, by introducing the policy gradient algorithm from reinforcement learning, it is not affected by the specific calculation form (mathematical formula) of the edit distance, solving the problem of the edit distance being non-differentiable and achieving end-to-end model optimization.

[0088] Example 2

[0089] According to another aspect of the present invention, a model training method is provided. Figure 4 : is a flow chart of the model training method according to the second embodiment of the present invention. Figure 4 Shown, including:

[0090] Step S402, obtaining the edit distance of the character string in the image to be recognized;

[0091] In the embodiment of the present application, the edit distance of a string can be obtained as follows:

[0092] (1) Hamming Distance: refers to the number of different characters in corresponding positions of two strings, that is, the number of characters that need to be replaced to transform one string into another.

[0093] (2) Levenshtein distance: refers to the minimum number of editing operations required to transform two strings from one to the other. Allowed editing operations include: replacing one character with another, inserting a character, and deleting a character.

[0094] (3) Longest Common Subsequence: refers to the longest subsequence among all sequences in a set of sequences (usually two sequences).

[0095] In addition to the three methods listed above for measuring the edit distance between two character sequences, there are also other measurement methods such as Damerau-Levenshtein distance and Jaro-Winkler distance.

[0096] Using any of the aforementioned algorithms for calculating edit distance, since edit distance loss is non-differentiable, we introduce the policy gradient algorithm from reinforcement learning to optimize the model end-to-end. By treating the edit distance as a reward function, the ultimate goal is to maximize the reward, i.e., minimize the sum of the edit distances of the training samples in the training set. To maximize the reward function, we use a negative sign, transforming the goal into maximizing the sum of negative edit distances.

[0097] Step S404, performing policy gradient calculation based on the edit distance and the character string to obtain a gradient value;

[0098] In one feasible method, a policy gradient is calculated based on the edit distance and the string to obtain a gradient value, including: calculating a reward value of the string based on the edit distance and the string; calculating an expected value corresponding to the string based on the reward value; and sampling the string using a preset sampling algorithm, and calculating a gradient based on the expected value to obtain a gradient value.

[0099] In one feasible method, obtaining the edit distance of a string and using the edit distance as a reward and penalty function includes: obtaining the probability of each image in the training set being sampled, the probability of obtaining the string based on the image to be identified, and the reward value of the string; performing an expectation calculation based on the probability of each image being sampled, the probability of obtaining the string based on the image to be identified, and the reward value of the string to obtain the reward and penalty function.

[0100] The probability of obtaining each image in the training set being sampled can be recorded as P(I); the probability of obtaining a string based on the image to be recognized can be recorded as P(S pred |I), that is, in the case of image I, get sentence S pred The probability of recognizing a string can be expressed as R(I, S pred ).

[0101] Take the string "S", "T", "O", "P" as an example, according to P(I), P(S pred |I) and R(I, S pred ) is used to calculate the reward value of the string, and then the reward and punishment function is obtained.

[0102] Specifically, in the embodiment of the present application, the goal is to find a set of network parameters θ * , this parameter can maximize the final reward. The calculation process of the reward (i.e., the reward and punishment function in the embodiment of the present application) is as shown in formula (1). Formula (1) is as follows:

[0103]

[0104]

[0105] The reward and penalty function obtained based on the reward value obtained from the string is used to calculate the expectation. Specifically, the reward and penalty function and the expectation are combined. At the same time, because the true value of the expectation needs to be calculated exhaustively, all possible recognized and generated texts under each image need to be calculated. In the embodiment of the present application, the Monte Carlo sampling algorithm is used to sample and obtain the recognized text (i.e., the first recognized text in the embodiment of the present application);

[0106] The first recognized text is subjected to policy gradient calculation to obtain the gradient value including:

[0107] According to the above formula (1), we can calculate the expectation and obtain (2). The specific formula (2) is as follows:

[0108]

[0109] Based on formula (2), the reward and punishment function and the expectation are combined, and formula (3) is used. At the same time, because if the true value of this expectation is to be calculated, it is necessary to exhaustively calculate all possible recognized and generated texts under each image. In the embodiment of the present application, the Monte Carlo sampling algorithm is used to sample and obtain the recognized text (that is, the first recognized text in the embodiment of the present application); in the embodiment of the present application, formula (3) is specifically as follows:

[0110]

[0111] After obtaining the approximate form of the maximum reward and punishment function, the gradient is calculated. The calculation process is as follows:

[0112]

[0113]

[0114] The gradient value is calculated according to formulas (4) and (5), and then the text recognition network model is trained and optimized end-to-end based on the gradient value.

[0115] Step S406: training the text recognition network model according to the gradient value to obtain an optimized text recognition network.

[0116] In one feasible manner, training the text recognition network model according to the gradient value to obtain the optimized text recognition network includes: training the text recognition network model according to the gradient value and the calculation formula of the gradient value to obtain the optimized text recognition network.

[0117] Specifically, based on formulas (4) and (5) in step S404, the text recognition network model is trained and optimized end-to-end to obtain an optimized text recognition network.

[0118] In an embodiment of the present invention, by obtaining the edit distance of the character string in the image to be recognized; performing policy gradient calculation based on the edit distance and the character string to obtain a gradient value; and training the text recognition network model based on the gradient value to obtain an optimized text recognition network, the purpose of achieving end-to-end optimization of the text recognition model is achieved, thereby achieving the technical effect of improving recognition performance, and further solving the technical problem that the accuracy of the entire vocabulary prediction in the existing technology is not incorporated into the model training of the OCR model, resulting in inconsistent optimization goals in the training stage and the test stage, and reduced recognition performance.

[0119] Example 3

[0120] According to another aspect of the embodiment of the present invention, an image recognition method is provided. Figure 5 FIG. 1 is a flow chart of an image recognition method according to Embodiment 3 of the present invention. Figure 5 As shown, the image recognition method provided in the embodiment of the present application includes:

[0121] Step S501, obtaining a character string in the image to be recognized;

[0122] In step S501 of the present application, the images to be identified may include: security, municipal administration, and transportation services, which require the use of massive amounts of camera video data. In addition to key objects of interest such as people, motor vehicles, and non-motor vehicles, the camera video data also includes a large amount of text data. For example, the text on street banners and propaganda slogans involves maintaining public order, changes to traffic signs involve timely updates to road data on maps, and the extraction of street shop signs involves updates to the POI (point of interest) database. Therefore, further identifying and analyzing text information in camera point images can further increase the information dimension in the city brain and further improve the level and effectiveness of video intelligent analysis.

[0123] The character string in the embodiment of the present application may include: text information in the image to be recognized; for example, the text mentioned in the camera point map, taking the text in the traffic sign as an example, when the traffic sign on the street is recognized to display "STOP", the obtained character string may be "S", "T", "O", "P".

[0124] like Figure 5 As shown, the image recognition method provided in the embodiment of the present application can be applied to an image recognition system.

[0125] Step S502, obtaining the edit distance of the character string, wherein the edit distance is used as a reward and penalty function;

[0126] In the above step S502 of the present application, the edit distance of the character string in the embodiment of the present application can be obtained by the following method:

[0127] (1) Hamming Distance: refers to the number of different characters in corresponding positions of two strings, that is, the number of characters that need to be replaced to transform one string into another.

[0128] (2) Levenshtein distance: refers to the minimum number of editing operations required to transform two strings from one to the other. Allowed editing operations include: replacing one character with another, inserting a character, and deleting a character.

[0129] (3) Longest Common Subsequence: refers to the longest word sequence among all sequences in a set of sequences (usually two sequences).

[0130] In addition to the three methods listed above for measuring the edit distance between two character sequences, there are also other measurement methods such as Damerau-Levenshtein distance and Jaro-Winkler distance.

[0131] Using any of the aforementioned algorithms for calculating edit distance, since edit distance loss is non-differentiable, we introduce the policy gradient algorithm from reinforcement learning to optimize the model end-to-end. By treating the edit distance as a reward function, the ultimate goal is to maximize the reward, i.e., minimize the sum of the edit distances of the training samples in the training set. To maximize the reward function, we use a negative sign, transforming the goal into maximizing the sum of negative edit distances.

[0132] In one feasible method, obtaining the edit distance of a string and using the edit distance as a reward and penalty function includes: obtaining the probability of each image in the training set being sampled, the probability of obtaining the string based on the image to be identified, and the reward value of the string; performing an expectation calculation based on the probability of each image being sampled, the probability of obtaining the string based on the image to be identified, and the reward value of the string to obtain the reward and penalty function.

[0133] The probability of obtaining each image in the training set being sampled can be recorded as P(I); the probability of obtaining a string based on the image to be recognized can be recorded as P(S pred |I), that is, in the case of image I, get sentence S pred The probability of recognizing a string can be expressed as R(I, S pred ).

[0134] Take the string "S", "T", "O", "P" as an example, according to P(I), P(S pred |I) and R(I, S pred ) is used to calculate the reward value of the string, and then the reward and punishment function is obtained.

[0135] Specifically, in the embodiment of the present application, the goal is to find a set of network parameters θ * , this parameter can maximize the final reward. The calculation process of the reward (i.e., the reward and punishment function in the embodiment of the present application) is as shown in formula (1). Formula (1) is as follows:

[0136]

[0137]

[0138] Step S503, performing policy gradient calculation on the character string according to the reward and punishment function to obtain the recognized text;

[0139] In one feasible manner, performing a policy gradient calculation on a character string based on a reward and punishment function to obtain a recognized text includes: sampling the character string based on a preset sampling algorithm and a reward and punishment function to obtain a first recognized text; performing a policy gradient calculation on the first recognized text to obtain a gradient value; optimizing the first recognized text based on the gradient value to obtain a second recognized text; and determining the second recognized text as the recognized text.

[0140] The expectation is calculated based on the reward and penalty function obtained based on the reward value obtained from the character string in step S502. Specifically, the reward and penalty function and the expectation are combined. At the same time, because the true value of the expectation needs to be calculated, an exhaustive enumeration is required to calculate all possible recognized and generated texts under each image. In the embodiment of the present application, a Monte Carlo sampling algorithm is used to sample and obtain the recognized text (i.e., the first recognized text in the embodiment of the present application);

[0141] The first recognized text is subjected to policy gradient calculation to obtain the gradient value including:

[0142] According to the above formula (1), we can calculate the expectation and obtain (2). The specific formula (2) is as follows:

[0143]

[0144] Based on formula (2), the reward and punishment function and the expectation are combined, and formula (3) is used. At the same time, because if the true value of this expectation is to be calculated, it is necessary to exhaustively calculate all possible recognized and generated texts under each image. In the embodiment of the present application, the Monte Carlo sampling algorithm is used to sample and obtain the recognized text (that is, the first recognized text in the embodiment of the present application); in the embodiment of the present application, formula (3) is specifically as follows:

[0145]

[0146] After obtaining the approximate form of the maximum reward and punishment function, the gradient is calculated. The calculation process is as follows:

[0147]

[0148]

[0149] The gradient value is calculated according to formulas (4) and (5), and then the text recognition network model is trained and optimized end-to-end based on the gradient value to obtain the second recognized text, and the recognized text is determined based on the second recognized text.

[0150] Step S504, displaying the recognized text;

[0151] In the above step S504 of the present application, the recognized text obtained in step S503 is displayed to the user, so that the user can provide feedback based on the recognized text on whether the image recognition requirements are met, and then execute step S505.

[0152] Step S505, receiving user opinion information returned based on the recognized text;

[0153] In the above step S505 of the present application, after the recognized text is displayed to the user in step S504, Figure 5 As shown, the image recognition method provided by the embodiment of the present application is on the image recognition system side, and the image recognition system receives user opinion information returned by the user through the client based on the recognized text, wherein the user opinion information is used to indicate whether the current recognized text meets the user's current image processing task requirements, so as to improve subsequent image recognition performance.

[0154] Step S506 , optimizing the recognized text according to the user opinion information.

[0155] In the above-mentioned step S506 of the present application, the recognized text in step S503 is further optimized based on the user opinion information obtained in step S505, so that the finally obtained optimized recognized text is more in line with the requirements of the image processing task.

[0156] Among them, the image recognition system in the embodiment of the present application can be a server cluster composed of multiple groups of servers with image data processing functions, or it can be a separate computing terminal with a neural network. The image recognition system in the embodiment of the present application is illustrated by taking the computing terminal with a neural network as an example, and is based on the image recognition method provided in the embodiment of the present application, without specific limitation.

[0157] In an embodiment of the present invention, by obtaining a character string in an image to be recognized; obtaining the edit distance of the character string, wherein the edit distance is used as a reward and penalty function; performing a policy gradient calculation on the character string based on the reward and penalty function to obtain a recognized text; displaying the recognized text; receiving user opinion information returned based on the recognized text; and optimizing the recognized text based on the user opinion information, the purpose of achieving end-to-end optimization of the text recognition model is achieved, thereby achieving the technical effect of improving recognition performance, and further solving the technical problem that the accuracy of the entire vocabulary prediction in the existing technology is not incorporated into the model training in the OCR model, resulting in inconsistent optimization targets in the training phase and the test phase, and reduced recognition performance.

[0158] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0159] Through the description of the above embodiments, those skilled in the art will clearly understand that the image recognition method according to the above embodiments can be implemented using software plus the necessary general-purpose hardware platform. Of course, it can also be implemented using hardware, but in many cases the former is the more preferred embodiment. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disk) and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0160] Example 4

[0161] According to an embodiment of the present invention, a device for implementing the above-mentioned image recognition method is also provided. Figure 6 is a schematic diagram of an image recognition device according to a fourth embodiment of the present invention. Figure 6 As shown, the device includes: a first acquisition module 62 for acquiring a character string in an image to be recognized; a second acquisition module 64 for acquiring an edit distance of the character string, wherein the edit distance is used as a reward and punishment function; and a recognition module 66 for performing a policy gradient calculation on the character string according to the reward and punishment function to obtain a recognized text.

[0162] Example 5

[0163] According to another aspect of the present invention, a model training device is provided. Figure 7 Schematic diagram of a model training device according to a fifth embodiment of the present invention. Figure 7 As shown, it includes: an acquisition module 72, which is used to obtain the edit distance of the character string in the image to be recognized; a calculation module 74, which is used to perform policy gradient calculation based on the edit distance and the character string to obtain a gradient value; and a training module 76, which is used to train the text recognition network model based on the gradient value to obtain an optimized text recognition network.

[0164] Example 6

[0165] According to another aspect of the present invention, an image recognition device is provided. Figure 8is a schematic diagram of an image recognition device according to a sixth embodiment of the present invention. Figure 8 As shown, it includes: a first acquisition module 81, used to obtain the character string in the image to be recognized; a second acquisition module 82, used to obtain the edit distance of the character string, wherein the edit distance is used as a reward and punishment function; a recognition module 83, used to perform policy gradient calculation on the character string according to the reward and punishment function to obtain the recognized text; a display module 84, used to display the recognized text; a receiving module 85, used to receive user opinion information returned based on the recognized text; an optimization module 86, used to optimize the recognized text based on the user opinion information.

[0166] Example 7

[0167] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is further provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the method described in any one of the above embodiments 1 to 3.

[0168] Example 8

[0169] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is configured to run a program, wherein when the program is run, the method described in any one of the above embodiments 1 to 3 is executed.

[0170] Example 9

[0171] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image recognition method provided in the first embodiment.

[0172] Optionally, in this embodiment, the above-mentioned storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0173] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a character string in an image to be identified; obtaining an edit distance of the character string, wherein the edit distance is used as a reward and penalty function; and performing a policy gradient calculation on the character string based on the reward and penalty function to obtain the identified text.

[0174] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining the edit distance of the string, and using the edit distance as a reward and punishment function, including: obtaining the probability of each image in the training set being sampled, the probability of obtaining the string based on the image to be identified, and the reward value of the string; performing expected calculations based on the probability of each image being sampled, the probability of obtaining the string based on the image to be identified, and the reward value of the string to obtain a reward and punishment function.

[0175] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: performing a policy gradient calculation on a character string based on a reward and punishment function to obtain a recognized text, including: sampling the character string based on a preset sampling algorithm and a reward and punishment function to obtain a first recognized text; performing a policy gradient calculation on the first recognized text to obtain a gradient value; optimizing the first recognized text based on the gradient value to obtain a second recognized text; and determining the second recognized text as the recognized text.

[0176] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: before obtaining the edit distance of the character string, the method further includes: cropping and resizing the image to be identified; encoding the cropped and resized image to be identified to obtain an encoded character string; and decoding the encoded character string to obtain a decoded character string.

[0177] Further, optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: cropping and resizing the image to be identified includes: cropping and resizing the image to be identified, and obtaining the height and width of the character string in the image to be identified.

[0178] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: decoding the encoded string to obtain the decoded string includes: decoding the encoded string through a neural network, and obtaining the decoded string.

[0179] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0180] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0182] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0183] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0184] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), a mobile hard drive, a magnetic disk, or an optical disk.

[0185] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An image recognition method, comprising: Get the string in the image to be recognized; Obtaining an edit distance of the string, wherein the edit distance serves as a reward and penalty function; Performing a policy gradient calculation on the character string according to the reward and punishment function to obtain a recognized text; Among them, the strategy gradient calculation is performed on the character string according to the reward and punishment function to obtain the recognized text, including: sampling the character string according to a preset sampling algorithm and the reward and punishment function to obtain a first recognized text; and the strategy gradient calculation is performed on the first recognized text to obtain the recognized text.

2. The method according to claim 1, wherein Obtaining the edit distance of the string and using the edit distance as a reward and penalty function includes: Obtaining the probability of each image in the training set being sampled, the probability of obtaining the string based on the image to be recognized, and the reward value of the string; The reward and penalty function is obtained by performing an expectation calculation based on the probability of each image being sampled, the probability of obtaining the character string based on the image to be recognized, and the reward value of the character string.

3. The method according to claim 1 or 2, wherein: Performing a policy gradient calculation on the first recognized text to obtain the recognized text includes: Performing a policy gradient calculation on the first recognized text to obtain a gradient value; Optimizing the first recognized text according to the gradient value to obtain a second recognized text; The second recognized text is determined as the recognized text.

4. The method according to claim 1, wherein Before obtaining the edit distance of the character string, the method further includes: Cropping and resizing the image to be recognized; Encoding the image to be recognized after cropping and resizing the image to obtain the encoded character string; The encoded character string is decoded to obtain the decoded character string.

5. The method according to claim 4, wherein Cropping and resizing the image to be recognized includes: The image to be recognized is cropped and the size of the image is adjusted to obtain the height and width of the character string in the image to be recognized.

6. The method according to claim 4, wherein: Decoding the encoded character string to obtain the decoded character string includes: The encoded character string is decoded by a neural network to obtain the decoded character string.

7. A model training method comprising: Get the edit distance of the string in the image to be recognized; Performing a policy gradient calculation based on the edit distance and the character string to obtain a gradient value; The text recognition network model is trained according to the gradient value to obtain the optimized text recognition network, wherein the optimized text recognition network is used to execute the method described in any one of claims 1 to 6.

8. The method according to claim 7, wherein: The policy gradient calculation is performed based on the edit distance and the character string to obtain the gradient value, which includes: Calculating a reward value of the character string according to the edit distance and the character string; Calculating an expected value corresponding to the character string according to the reward value; The character string is sampled using a preset sampling algorithm, and a gradient is calculated in combination with the expected value to obtain the gradient value.

9. The method according to claim 8, wherein The text recognition network model is trained according to the gradient value to obtain the optimized text recognition network, which includes: The text recognition network model is trained based on the gradient value and the calculation formula of the gradient value to obtain the optimized text recognition network.

10. An image recognition method, comprising: Get the string in the image to be recognized; Obtaining an edit distance of the string, wherein the edit distance serves as a reward and penalty function; Performing a policy gradient calculation on the character string according to the reward and punishment function to obtain a recognized text; displaying the identified text; Receiving user opinion information returned based on the recognized text; Optimizing the recognized text according to the user opinion information; Among them, the strategy gradient calculation is performed on the character string according to the reward and punishment function to obtain the recognized text, including: sampling the character string according to a preset sampling algorithm and the reward and punishment function to obtain a first recognized text; and the strategy gradient calculation is performed on the first recognized text to obtain the recognized text.

11. An image recognition device, comprising: A first acquisition module is used to acquire a character string in an image to be recognized; A second acquisition module is configured to acquire an edit distance of the character string, wherein the edit distance is used as a reward and penalty function; A recognition module, configured to perform a policy gradient calculation on the character string according to the reward and punishment function to obtain a recognized text; Among them, the recognition module is also used to sample the character string according to a preset sampling algorithm and the reward and punishment function to obtain a first recognition text; and perform policy gradient calculation on the first recognition text to obtain the recognition text.

12. A model training device comprising: An acquisition module, used to obtain the edit distance of the character string in the image to be recognized; A calculation module, configured to perform policy gradient calculation based on the edit distance and the character string to obtain a gradient value; A training module is used to train the text recognition network model according to the gradient value to obtain the optimized text recognition network, wherein the optimized text recognition network is used to execute the method described in any one of claims 1 to 6.

13. An image recognition device, comprising: A first acquisition module is used to acquire a character string in an image to be recognized; A second acquisition module is configured to acquire an edit distance of the character string, wherein the edit distance is used as a reward and penalty function; A recognition module, configured to perform a policy gradient calculation on the character string according to the reward and punishment function to obtain a recognized text; A display module, used for displaying the recognized text; A receiving module, configured to receive user opinion information returned based on the recognized text; an optimization module, configured to optimize the recognized text according to the user opinion information; Among them, the recognition module is also used to sample the character string according to a preset sampling algorithm and the reward and punishment function to obtain a first recognition text; and perform policy gradient calculation on the first recognition text to obtain the recognition text.

14. A non-volatile storage medium, wherein: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method according to any one of claims 1 to 10.

15. A processor, wherein: The processor is configured to run a program, wherein the program executes the method according to any one of claims 1 to 10 when running.

Images