Character recognition method and device based on image, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image-based character recognition method and device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the steps of obtaining a target image; inputting the target image into a pre-trained target character recognition model; performing multi-scale feature extraction on the target image based on the multi-scale feature extraction sub-model to obtain image multi-scale features; performing context feature extraction on the image multi-scale features based on a context feature extraction sub-model to obtain image context features; performing character prediction on the image context features based on a character prediction sub-model to obtain a first prediction probability value of a first prediction character; according to the fact that the first prediction probability value is smaller than or equal to a first prediction threshold value, character correction is conducted on the image context features based on a character correction sub-model, and a second prediction probability value of a second prediction character is obtained. According to the embodiment of the invention, the accuracy of character recognition in the image is improved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to an image-based character recognition method and device, an electronic device, and a storage medium. Background Art

[0002] Image-based character recognition refers to the process of converting characters in an image into an editable character format through computer technology, and the characters at this time may include text, letters, numbers, etc. For example, in the vehicle insurance claims scenario of FinTech, it is usually necessary to identify the vehicle identification number (VIN code) used to uniquely identify the vehicle in order to efficiently obtain the relevant claims information of the vehicle and improve the processing efficiency of vehicle insurance claims.

[0003] At present, related technologies usually use character recognition methods based on convolutional recurrent neural networks (CRNN) to recognize characters in vehicle images, which is prone to problems such as false detection and missed detection, thus affecting the accuracy of character recognition. Therefore, how to improve the accuracy of character recognition in images has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an image-based character recognition method and device, an electronic device and a storage medium, aiming to improve the accuracy of character recognition in an image.

[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes an image-based character recognition method, the method comprising:

[0006] Get the target image;

[0007] Inputting the target image into a pre-trained target character recognition model, wherein the target character recognition model includes a multi-scale feature extraction sub-model, a context feature extraction sub-model, a character prediction sub-model and a character correction sub-model;

[0008] Performing multi-scale feature extraction on the target image based on the multi-scale feature extraction sub-model to obtain multi-scale features of the image;

[0009] Performing context feature extraction on the multi-scale features of the image based on the context feature extraction sub-model to obtain image context features;

[0010] Performing character prediction on the image context feature based on the character prediction sub-model to obtain a first prediction probability value of a first predicted character;

[0011] According to the first prediction probability value being less than or equal to a first prediction threshold, character correction is performed on the image context feature based on the character correction sub-model to obtain a second prediction probability value of a second predicted character.

[0012] In some embodiments, the method further comprises:

[0013] If the second prediction probability value is less than or equal to a second prediction threshold, updating the second predicted character based on a preset character;

[0014] Acquire a first valid concatenated character from the first predicted characters, where the first valid concatenated character is used to represent the first predicted character whose corresponding first predicted probability value is greater than the first prediction threshold;

[0015] Acquire a second valid concatenated character from the second predicted characters, where the second valid concatenated character is used to represent the preset character or the second predicted character whose second predicted probability value is greater than the second prediction threshold;

[0016] Character splicing is performed based on the first valid splicing character and the second valid splicing character to obtain a target recognition character string of the target image.

[0017] In some embodiments, the character splicing based on the first valid splicing character and the second valid splicing character to obtain the target recognition character string of the target image includes:

[0018] Obtaining a first character position of the first valid splicing character and a second character position of the second valid splicing character;

[0019] Determining a splicing order based on the first character position and the second character position;

[0020] The first valid splicing character and the second valid splicing character are spliced ​​based on the splicing order to obtain the target recognition character string.

[0021] In some embodiments, the context feature extraction submodel includes a forward context feature extraction submodel, a reverse context feature extraction submodel and a feature concatenation layer, and the context feature extraction of the multi-scale features of the image based on the context feature extraction submodel to obtain the image context feature includes:

[0022] Performing context feature extraction on the multi-scale features of the image based on the forward context feature extraction sub-model to obtain a forward context feature, wherein the forward context feature includes forward context sub-features arranged in time step order;

[0023] Performing context feature extraction on the multi-scale features of the image based on the reverse context feature extraction sub-model to obtain reverse context features, wherein the reverse context features include reverse context sub-features arranged in reverse order according to time steps;

[0024] Based on the feature splicing layer, feature splicing is performed on the forward context sub-feature and the reverse context sub-feature corresponding to the time step to obtain the image context feature.

[0025] In some embodiments, before inputting the target image into a pre-trained target character recognition model, the method further includes: constructing the target character recognition model, specifically including:

[0026] Acquire a training sample, wherein the training sample includes a sample image and a sample character string;

[0027] Inputting the sample image into an initial character recognition model, wherein the initial character recognition model includes the multi-scale feature extraction sub-model, the context feature extraction sub-model, the character prediction sub-model and the character correction sub-model;

[0028] Performing multi-scale feature extraction on the sample image based on the multi-scale feature extraction sub-model to obtain multi-scale features of the sample image;

[0029] Performing context feature extraction on the multi-scale features of the sample image based on the context feature extraction sub-model to obtain context features of the sample image;

[0030] Performing character prediction on the sample image context feature based on the character prediction sub-model to obtain a first sample prediction probability value of a first predicted sample character;

[0031] Performing character correction on the sample image context feature based on the character correction sub-model to obtain a second sample prediction probability value of a second predicted sample character;

[0032] splicing the first predicted sample characters based on the first sample predicted probability value to obtain a first predicted character string, and splicing the second predicted sample characters based on the second sample predicted probability value to obtain a second predicted character string;

[0033] A model loss value is determined based on the first predicted character string, the second predicted character string and the sample character string, and parameters of the initial character recognition model are adjusted based on the model loss value to obtain the target character recognition model.

[0034] In some embodiments, the character correction sub-model includes a first attention layer, a feature fusion layer and a prediction layer, and the character correction is performed on the sample image context feature based on the character correction sub-model to obtain the second sample prediction probability value of the second predicted sample character, including:

[0035] Extracting sample positive context features and sample reverse context features from the sample image context features;

[0036] Based on the first attention layer, feature attention processing is performed on the sample forward context feature and the sample reverse context feature to obtain a sample context attention feature;

[0037] Based on the feature fusion layer, the sample context attention feature and the sample reverse context feature are subjected to feature fusion to obtain a sample target context feature;

[0038] Character correction is performed on the sample target context feature based on the prediction layer to obtain the second sample prediction probability value of the second prediction sample character.

[0039] In some embodiments, determining the model loss value based on the first predicted string, the second predicted string, and the sample string includes:

[0040] Obtaining a first model weight of the character prediction sub-model and a second model weight of the character correction sub-model;

[0041] Performing loss calculation based on the first predicted character string and the sample character string to obtain a first loss value;

[0042] Performing loss calculation based on the second predicted character string and the sample character string to obtain a second loss value;

[0043] The model loss value is obtained by performing a weighted sum calculation based on the first model weight, the first loss value, the second model weight and the second loss value.

[0044] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes an image-based character recognition device, the device comprising:

[0045] An acquisition module, used for acquiring a target image;

[0046] An input module, used for inputting the target image into a pre-trained target character recognition model, wherein the target character recognition model includes a multi-scale feature extraction sub-model, a context feature extraction sub-model, a character prediction sub-model and a character correction sub-model;

[0047] A first extraction module, configured to perform multi-scale feature extraction on the target image based on the multi-scale feature extraction sub-model to obtain multi-scale features of the image;

[0048] A second extraction module, configured to perform context feature extraction on the image multi-scale features based on the context feature extraction sub-model to obtain image context features;

[0049] A character prediction module, configured to perform character prediction on the image context feature based on the character prediction sub-model to obtain a first prediction probability value of a first predicted character;

[0050] A character correction module is used to perform character correction on the image context feature based on the character correction sub-model according to the first prediction probability value being less than or equal to a first prediction threshold, so as to obtain a second prediction probability value of a second predicted character.

[0051] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the image-based character recognition method described in the first aspect when executing the computer program.

[0052] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the image-based character recognition method described in the first aspect above.

[0053] The present application proposes an image-based character recognition method and device, electronic device and storage medium, which obtains a target image; inputs the target image into a pre-trained target character recognition model, the target character recognition model includes a multi-scale feature extraction sub-model, a context feature extraction sub-model, a character prediction sub-model and a character correction sub-model; further, based on the multi-scale feature extraction sub-model, multi-scale feature extraction is performed on the target image to obtain image multi-scale features; based on the context feature extraction sub-model, context feature extraction is performed on the image multi-scale features to obtain image context features; based on the character prediction sub-model, character prediction is performed on the image context features to obtain a first prediction probability value of a first predicted character; based on the first prediction probability value being less than or equal to a first prediction threshold, character correction is performed on the image context features based on the character correction sub-model to obtain a second prediction probability value of a second predicted character. The embodiment of the present application can extract deeper features from the input image through the multi-scale feature extraction sub-model and the context feature extraction sub-model, and improve the accuracy of character prediction by combining the character prediction sub-model and the character correction sub-model. Compared with the related art, which is prone to problems such as misdetection and missed detection in recognition, the present application will perform secondary character correction on the image context features through the character correction sub-model when the first predicted probability value of the first predicted character is less than or equal to the first prediction threshold, so as to obtain a second predicted character with higher accuracy. Therefore, the present application can improve the accuracy of character recognition in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is the first flow chart of the image-based character recognition method provided by the embodiment of the present application;

[0055] Figure 2 yes Figure 1 A flowchart of step S140 in FIG.

[0056] Figure 3 yes Figure 1 A flowchart of step S160 in FIG.

[0057] Figure 4 is a flow chart of constructing a target character recognition model provided by an embodiment of the present application;

[0058] Figure 5 yes Figure 4 A flowchart of step S480 in FIG.

[0059] Figure 6 is a second flow chart of the image-based character recognition method provided in an embodiment of the present application;

[0060] Figure 7 yes Figure 6 A flowchart of step S640 in FIG.

[0061] Figure 8 is a structural schematic diagram of an image-based character recognition device provided in an embodiment of the present application;

[0062] Fig. 9 It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] First, some nouns involved in this application are analyzed:

[0067] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] Normalized exponential (Softmax function): compresses a K-dimensional vector containing arbitrary real numbers into another K-dimensional vector so that each element ranges between (0,1) and the sum of all elements is 1.

[0069] Vehicle Identification Number (VIN code): A coding system consisting of 17 characters that provides a unique identity for each car. The 1st to 3rd digits of the VIN code are the manufacturer identification code, which is used to indicate which manufacturer produced the vehicle; the 4th to 8th digits are used to describe the vehicle's type, body type and other vehicle features; the 9th digit is the check digit, which is used to prevent input errors through a specific algorithm; the 10th digit is the model year, which is used to indicate the model year specified by the manufacturer; the assembly plant: the 11th digit is the corresponding identification of the assembly plant; the factory sequence number: the 12th to 17th digits are the factory sequence number. In addition, the specific composition of the VIN code can be adjusted according to actual needs and is not limited. The VIN code can be used to confirm the identity of the car. For example, in the traffic violation query, the violation record of the vehicle can be queried through the VIN code. In the process of vehicle insurance claims, the production batch of the vehicle can be understood through the VIN code, so as to accurately find the specific accessories suitable for the vehicle.

[0070] Pipeline: It is a process-based processing method that can connect a series of tasks or steps in series to form an orderly workflow.

[0071] Image-based character recognition methods refer to the process of converting characters in an image into an editable character format through computer technology, and the characters at this time may include text, letters, numbers, etc. For example, in the vehicle insurance claims scenario of financial technology, it is usually necessary to identify the vehicle identification number (VIN code) used to uniquely identify the vehicle, so as to efficiently obtain the relevant claims information of the vehicle and improve the processing efficiency of vehicle insurance claims. For another example, the car ecology is an important scenario in finance. The digital management and transaction of used cars can accelerate transactions and attract more customers to purchase used cars. The VIN code, as a car certificate, can effectively promote the technological operation of used car management through efficient recognition of the VIN code.

[0072] At present, related technologies usually use character recognition methods based on convolutional recurrent neural networks (CRNN) to recognize characters in vehicle images, which is prone to problems such as false detection and missed detection, thus affecting the accuracy of character recognition. Therefore, how to improve the accuracy of character recognition in images has become a technical problem that needs to be solved urgently.

[0073] Based on this, the embodiments of the present application provide an image-based character recognition method and device, an electronic device, and a storage medium, which can improve the accuracy of character recognition in an image.

[0074] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0075] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0076] The image-based character recognition method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The image-based character recognition method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms; the software can be an application that implements an image-based character recognition method, etc., but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0078] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on the object's item attribute information, object identity information, and other data related to the object's identity or characteristics, the object's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the sensitive personal information of the object, the object's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the object's separate permission or separate consent, the necessary object-related data used to enable the normal operation of the embodiment of the present application will be obtained.

[0079] See also Figure 1 , Figure 1 is an optional flow chart of the image-based character recognition method provided in the embodiment of the present application. In some embodiments, Figure 1 The method may include but is not limited to steps S110 to S160:

[0080] Step S110, acquiring a target image;

[0081] Step S120, inputting the target image into a pre-trained target character recognition model;

[0082] Step S130, performing multi-scale feature extraction on the target image based on the multi-scale feature extraction sub-model to obtain multi-scale features of the image;

[0083] Step S140, performing context feature extraction on the multi-scale features of the image based on the context feature extraction sub-model to obtain image context features;

[0084] Step S150, performing character prediction on the image context feature based on the character prediction sub-model to obtain a first prediction probability value of a first predicted character;

[0085] Step S160, according to the first prediction probability value being less than or equal to the first prediction threshold, character correction is performed on the image context feature based on the character correction sub-model to obtain a second prediction probability value of the second predicted character.

[0086] In steps S110 to S160 of some embodiments, compared to related technologies that are prone to problems such as misdetection and missed detection in recognition, the present application will perform secondary character correction on the image context features through the character correction sub-model when the first predicted probability value of the first predicted character is less than or equal to the first prediction threshold, so as to obtain a second predicted character with higher accuracy. In other words, the present application forms a dual-head correction recognition mechanism through the character prediction sub-model and the character correction sub-model, which can correct error-prone characters and thus be better applied to computer vision, character recognition and other scenarios. Therefore, the present application can improve the accuracy of character recognition in images.

[0087] In step S110 of some embodiments, the target image refers to an image containing characters to be recognized. For example, in the vehicle insurance claim scenario of financial technology, when the VIN code of the target claim vehicle needs to be recognized, the target image refers to an image taken of the area where the VIN code of the target claim vehicle is located, and the characters to be recognized at this time refer to the VIN code of the target claim vehicle. For another example, during the repair process of vehicle insurance claims, the production batch of the vehicle can be understood through the VIN code, so as to accurately find specific accessories suitable for the vehicle.

[0088] It should be noted that the character to be recognized refers to a glyph unit or symbol, including letters, numbers, operation symbols, punctuation marks and other symbols, as well as some functional symbols, without limitation. In electronic computers or radio communications, characters are a general term for letters, numbers, and symbols. They are the smallest data access unit in a data structure, and are usually represented by 8 binary bits (one byte).

[0089] In step S120 of some embodiments, in order to accurately identify the characters in the target image, the present application may input the captured target image into a pre-trained target character recognition model, and the target character recognition model includes a multi-scale feature extraction sub-model, a context feature extraction sub-model, a character prediction sub-model and a character correction sub-model.

[0090] It should be noted that the target character recognition model of the present application may include a backbone network model, a decoding network model and a new recognition algorithm applied according to rules. Among them, the backbone network model of the present application is a model constructed based on the Swin Transformers structure, specifically referring to the multi-scale feature extraction sub-model of the present application. The decoding network model specifically refers to the context feature extraction sub-model, character prediction sub-model and character correction sub-model constructed in the present application. Furthermore, in specific practical applications, character recognition can be performed in combination with the proposed decoding network according to the new recognition algorithm applied according to rules to improve the accuracy of character recognition, so that specific information can be input quickly and accurately, improving efficiency while reducing human errors.

[0091] In step S130 of some embodiments, after the target image is input into the target character recognition model, the target image may be first subjected to multi-scale feature extraction based on the multi-scale feature extraction sub-model to obtain the multi-scale features of the target image, so as to obtain the feature information of the target image at different scales. Among them, the present application can extract details and regional features in the target image by adjusting the scale of the target image (such as different resolutions or sizes), thereby capturing objects or structures of different sizes in the target image and enhancing the recognition ability of the model.

[0092] It should be noted that the multi-scale feature extraction sub-model of the present application may include a patch layer, a coding layer (Embedding layer) and an attention layer (Block layer). Among them, first of all, the size of each patch can be determined first (for example, 3x3 pixels, 5x5 pixels, etc.), then the patch layer can be moved on the input image through a sliding window mechanism, and each movement will intercept a patch. The windows can overlap (that is, there are some common pixels between adjacent patches) or not overlap (each pixel belongs to only one patch) to divide the input target image into small blocks of uniform size, called "patches", and each patch is regarded as a feature vector, that is, an image piece feature. The present application divides the entire target image into multiple patches. When detecting a specific object in an image, the patches that may contain the target can be extracted first, and then more sophisticated recognition algorithms can be applied to these patches, which can effectively increase the diversity of training data, thereby improving the generalization ability of the model.

[0093] Furthermore, the Embedding layer mainly includes linear transformation. At this time, each patch will be linearly mapped to a low-dimensional feature space, that is, the image patch features are linearly transformed to obtain linear patch features. This mapping process is completed by an Embedding layer.

[0094] Furthermore, the Block layer is mainly composed of a self-attention layer and a local attention layer based on a sliding window. The self-attention layer includes query, key, and value matrices, which can be matrix-multiplied with the input linear patch features respectively. Then, the output of the query and key calculations can be transformed and matrix multiplied, and the Softmax function can be calculated row by row to obtain the attention matrix J. Finally, J and the value channel output matrix are multiplied to obtain the result. The present application is based on a self-attention layer of sliding window convolution. In the original method, the input variables are translated, the mask values ​​are fixed, and each patch performs local attention calculations with a group of surrounding adjacent patches to capture local features.

[0095] It should be noted that the present application can introduce multiple levels of Transformer blocks based on local attention, and each level can be composed of multiple Transformer blocks, each of which contains self-attention and feedforward neural network layers, and these levels gradually capture feature information of different scales. In addition, the present application can stack the Block layers in proportion to form an overall backbone network. In this way, the multi-scale image features of the target image can be output in the last Block layer.

[0096] In the embodiment of the present application, the Swim Transformer is based on the multi-scale characteristics, so that it can support the processing of fine-grained features such as text, thereby improving the accuracy of character recognition.

[0097] In step S140 of some embodiments, further, the present application may determine a decoding network model based on the constructed context feature extraction sub-model, character prediction sub-model and character correction sub-model. The decoding network model may include a bottleneck layer and a double-head layer, the bottleneck layer corresponds to the context feature extraction sub-model of the present application, and the context feature extraction sub-model may specifically include a forward context feature extraction sub-model, a reverse context feature extraction sub-model and a feature splicing layer. The double-head layer may include a character prediction sub-model and a character correction sub-model, that is, character recognition is performed through two pipelines, which can increase the correction of character recognition results relative to a single-head layer, and improve the robustness of model character (such as VIN code) recognition.

[0098] It should be noted that the context feature extraction sub-model can focus on the relationship and structural information between various parts of the target image, thereby extracting more semantic features.

[0099] See also Figure 2 , Figure 2 This is an optional flow chart of step S140 provided in an embodiment of the present application. In some embodiments, step S140 may specifically include steps S210 to S230:

[0100] Step S210, extracting context features from multi-scale features of the image based on the forward context feature extraction sub-model to obtain forward context features;

[0101] Step S220, extracting context features from the multi-scale features of the image based on the reverse context feature extraction sub-model to obtain reverse context features;

[0102] Step S230 , based on the feature concatenation layer, the forward context sub-features and the reverse context sub-features corresponding to the time step are concatenated to obtain image context features.

[0103] In step S210 of some embodiments, the forward context feature extraction submodel includes multiple forward LSTM subunits, and each forward LSTM subunit is connected in time step order, which can be used to process the input feature sequence to capture the context information in the forward order. Therefore, based on the forward context feature extraction submodel, the multi-scale features of the image are subjected to context feature extraction to obtain the forward context feature, which includes the forward context subfeatures arranged in time step order. One forward context subfeature corresponds to one time step.

[0104] In step S220 of some embodiments, the reverse context feature extraction submodel includes multiple reverse LSTM subunits, and each reverse LSTM subunit is connected in reverse time step order, which can be used to process the reverse order of the input feature sequence to capture the context information in reverse order. Therefore, based on the reverse context feature extraction submodel, context feature extraction is performed on the multi-scale features of the image to obtain reverse context features, which include reverse context subfeatures arranged in reverse order of time steps. One reverse context subfeature corresponds to one time step.

[0105] It should be noted that both the forward LSTM subunit and the reverse LSTM subunit are equivalent to an LSTM subunit, and the LSTM subunit includes an input gate, a forget gate, and an output gate. The forget gate can perform a weighted summation of the input data and the state of the previous moment through a linear function (fully connected) and an activation function (usually a sigmoid function) to generate a value between 0 and 1. Similar to the forget gate, the input gate also uses a linear function and an activation function to perform a weighted summation of the input data and the state of the previous moment to generate a value between 0 and 1. The output gate solves the output value of the output and model parameters through matrix operations, so that the LSTM subunit construction can be completed.

[0106] In step S230 of some embodiments, after obtaining the forward context sub-feature and the reverse context sub-feature, the outputs of the forward context feature extraction sub-model and the reverse context feature extraction sub-model can be feature spliced ​​according to the time step to obtain the final output of the bidirectional LSTM layer, that is, to obtain the image context feature. In this way, the output features of each time step contain context information in both the forward and reverse directions. Among them, the present application can define the forward context feature extraction sub-model as the encoding layer and the reverse context feature extraction sub-model as the decoding layer to complete the construction of the bottleneck layer.

[0107] It should be noted that when the forward context sub-feature and the reverse context sub-feature corresponding to the same time step are concatenated, the feature vector of the reverse context sub-feature can be rearranged in the original time order so that it is aligned with the feature vector of the forward context sub-feature in the time step. Furthermore, the output vectors of the forward context sub-feature and the reverse context sub-feature corresponding to the time step are concatenated (usually element-wise). In this way, the final output vector of each time step contains information from both the forward and reverse directions.

[0108] In step S150 of some embodiments, the dual-head layer of the present application may include a character prediction sub-model and a character correction sub-model, that is, character recognition is performed through two pipelines, which can increase the correction of character recognition results relative to a single-head layer, and improve the robustness of model character (such as VIN code) recognition. Therefore, after obtaining the image context features, the character prediction sub-model (i.e., pipeline a) can be used to identify and predict characters based on the extracted image context features to obtain a first predicted probability value of the first predicted character. Among them, the first predicted character is used to represent the result predicted by the character prediction sub-model for a character in the target image, and the first predicted probability value is used to represent the probability value of the predicted character being the first predicted character, and can also represent the model's confidence in the occurrence of the first predicted character.

[0109] It should be noted that the character prediction sub-model can be a prediction model structure constructed using models such as Convolutional Neural Networks (CNN), Transformer, and machine learning, without limitation.

[0110] In step S160 of some embodiments, although the recognition ability of pipline a is already very strong, it may still make recognition errors. Based on this, when the first prediction probability value in the prediction result of the character prediction sub-model is less than or equal to the first prediction threshold, it means that the first predicted character obtained is not very accurate. At this time, the image context features can be corrected based on the character correction sub-model to obtain a second prediction probability value for a more accurate second predicted character. The second predicted character is used to represent the result predicted by the character correction sub-model for a character in the target image. The second prediction probability value is used to represent the probability value of the predicted character being the second predicted character, and can also represent the model's confidence in the occurrence of the second predicted character. In this way, by setting up character remedial measures such as the character correction sub-model, the accuracy of character recognition can be improved, such as improving the accuracy of VIN code recognition.

[0111] It should be noted that when the first prediction probability value in the prediction result of the character prediction sub-model is greater than the first prediction threshold, it means that the first predicted character obtained is relatively accurate, and the first predicted character can be directly used as a recognized character result in the target image. The first prediction threshold refers to a pre-set threshold for judging the first prediction probability value output by the character prediction sub-model, which can be 0.6, 0.9, etc., without limitation.

[0112] It should be noted that the character prediction sub-model can directly identify and predict characters based on image context features. Compared with the character prediction sub-model, the character correction sub-model of the present application can be constructed by introducing an attention layer into the output layer of the image context features to extract deeper feature information. Specifically, the character correction sub-model includes a first attention layer, a feature fusion layer, and a prediction layer.

[0113] See also Figure 3 , Figure 3 It is an optional flow chart of step S160 provided in the embodiment of the present application. In some embodiments, step S150 may specifically include steps S310 to S340:

[0114] Step S310, extracting positive context features and negative context features from the image context features;

[0115] Step S320, performing feature attention processing on the forward context feature and the reverse context feature based on the first attention layer to obtain a context attention feature;

[0116] Step S330, performing feature fusion on the context attention feature and the reverse context feature based on the feature fusion layer to obtain the target context feature;

[0117] Step S340: performing character correction on the target context feature based on the prediction layer to obtain a second predicted probability value of a second predicted character.

[0118] In step S310 of some embodiments, the specific method of obtaining the positive context feature and the reverse context feature has been described in detail in the above steps S210 and S220 and will not be repeated here.

[0119] In step S320 of some embodiments, the input of the first attention layer includes query, key and value, wherein query is all outputs from the above decoding layer, i.e., reverse context features, key is all outputs from the above encoding layer, i.e., forward context features, and value is the same as key. Among them, the query, key and value layers are all composed of fully connected layers, and the output of the query and the output of the key are matrix multiplied and then connected to softmax to obtain the attention matrix M, and M is matrix multiplied with the output of the value to obtain the final output. The construction of the first attention layer is completed. Based on this, feature attention processing can be performed on the forward context features and the reverse context features based on the first attention layer to obtain contextual attention features.

[0120] In step S330 of some embodiments, further, the context attention feature and the reverse context feature may be fused based on the feature fusion layer to obtain the target context feature. Specifically, the context attention feature and the reverse context feature may be feature concatenated, so that the dimension is doubled, and further, the concatenated feature may be fully connected to reduce the dimension to the original dimension to obtain the target context feature.

[0121] In step S340 of some embodiments, the prediction layer may be a structure constructed based on a fully connected layer, and character correction is performed on the target context features based on the prediction layer to obtain a second predicted probability value of the second predicted character, thereby realizing the construction of a character correction sub-model (i.e., pipeline b).

[0122] In the above embodiments, the present application can perform character recognition through multi-level feature extraction and prediction, ensuring that the grasp of the characters in the image not only depends on the first extracted features, but also takes into account the relevance of the context and the character recognition situation, thereby achieving more accurate recognition results.

[0123] See also Figure 4 , Figure 4 is another optional flow chart of the image-based character recognition method provided by the embodiment of the present application. In some embodiments, before step S120, the image-based character recognition method provided by the embodiment of the present application further includes: constructing a target character recognition model, which may specifically include steps S410 to S480:

[0124] Step S410, obtaining training samples;

[0125] Step S420, inputting the sample image into an initial character recognition model, the initial character recognition model including a multi-scale feature extraction sub-model, a context feature extraction sub-model, a character prediction sub-model and a character correction sub-model;

[0126] Step S430, performing multi-scale feature extraction on the sample image based on the multi-scale feature extraction sub-model to obtain multi-scale features of the sample image;

[0127] Step S440, performing context feature extraction on the multi-scale features of the sample image based on the context feature extraction sub-model to obtain context features of the sample image;

[0128] Step S450, performing character prediction on the sample image context feature based on the character prediction sub-model to obtain a first sample prediction probability value of a first predicted sample character;

[0129] Step S460, performing character correction on the sample image context feature based on the character correction sub-model to obtain a second sample prediction probability value of a second predicted sample character;

[0130] Step S470, concatenating the first prediction sample characters based on the first sample prediction probability value to obtain a first prediction string, and concatenating the second prediction sample characters based on the second sample prediction probability value to obtain a second prediction string;

[0131] Step S480, determining a model loss value based on the first predicted character string, the second predicted character string and the sample character string, and adjusting parameters of the initial character recognition model based on the model loss value to obtain a target character recognition model.

[0132] In step S410 of some embodiments, the training sample includes a sample image and a sample character string. The sample image refers to an image used to train the target character recognition model, and the sample image contains the characters to be recognized. The sample character string refers to the character string contained in the pre-labeled sample image, which can be used as a label for model training. The sample image can be an image extracted frame by frame from the collected sample video. For example, the sample video is a video collected from the VIN code of the claim vehicle. The sample images in the data set are obtained by frame sampling, and each image in the data set is manually annotated to obtain the sample character string.

[0133] In steps S420 to S450 of some embodiments, steps S420 to S450 correspond to the above-mentioned steps S120 to S150 respectively, except that the parameters therein are replaced with the parameters of the sample image, which will not be repeated.

[0134] In step S460 of some embodiments, during the model training process, the comparison relationship between the first prediction probability value and the first prediction threshold may be ignored, and character correction may be performed directly on the sample image context features based on the character correction sub-model to obtain multiple second sample prediction probability values ​​for multiple second prediction sample characters.

[0135] In step S470 of some embodiments, further, a first prediction string can be obtained by splicing a plurality of first prediction sample characters based on the first sample prediction probability value, and a second prediction string can be obtained by splicing a plurality of second prediction sample characters based on the second sample prediction probability value. For example, the first prediction string finally spliced ​​by the character prediction sub-model is "1022a355", and the first prediction string finally spliced ​​by the character correction sub-model is "1021a350".

[0136] In step S480 of some embodiments, further, the same or different loss functions can be set for different pipelines (i.e., the character prediction sub-model and the character correction sub-model) to determine the model loss value, and the parameters of the initial character recognition model are adjusted based on the model loss value to obtain the target character recognition model. For example, the character prediction sub-model can be calculated using the Connectionist Temporal Classification (CTC) loss function, while the character correction sub-model can be calculated using the Cross Entropy (CE) loss function.

[0137] It should be noted that during the training process, this application can use an optimizer of adam, a learning rate of 0.001, and a batch size of 64. The training samples are first imported into the pre-trained parameters of Imagenet, and then the structure of the initial character recognition model is trained until the loss converges, the model parameters are saved, and the target character recognition model is obtained.

[0138] See also Figure 5 , Figure 5 This is an optional flow chart of step S480 provided in an embodiment of the present application. In some embodiments, step S480 may specifically include steps S510 to S540:

[0139] Step S510, obtaining a first model weight of a character prediction sub-model and a second model weight of a character correction sub-model;

[0140] Step S520, performing loss calculation based on the first predicted character string and the sample character string to obtain a first loss value;

[0141] Step S530, performing loss calculation based on the second predicted character string and the sample character string to obtain a second loss value;

[0142] Step S540, performing weighted sum calculation based on the first model weight, the first loss value, the second model weight and the second loss value to obtain the model loss value.

[0143] In step S510 of some embodiments, the first model weight is used to characterize the importance of the character prediction sub-model in the initial character recognition model training process, and the second model weight is used to characterize the importance of the character correction sub-model in the initial character recognition model training process. These weights are optimized during the training process and determine the output behavior of the model for input data.

[0144] In step S520 of some embodiments, further, a loss value between a first predicted character string generated by the character prediction model and a real sample character string is calculated. Specifically, the first predicted character string can be compared with the real sample character string, and an adapted loss function (such as a CE loss function or a CTC loss function) is usually used to calculate the difference between the two character strings to obtain a first loss value. The result of this step provides an evaluation basis for the performance of the model in the prediction stage.

[0145] In step S530 of some embodiments, further, a loss value between a second predicted character string generated by the character correction model and the real sample character string is calculated. Specifically, the second predicted character string can be compared with the real sample character string, and an adapted loss function (such as a CE loss function or a CTC loss function) is usually used to calculate the difference between the two character strings to obtain a second loss value. The result of this step provides an evaluation basis for the performance of the model in the correction stage.

[0146] In step S540 of some embodiments, all relevant model weights and loss values ​​are combined to calculate the final model loss value to optimize model training. In this process, important elements such as the first model weight, the first loss value, the second model weight and the second loss value can be combined for weighting and calculation. Specifically, the contribution of each loss to the total loss may be determined based on a certain weighting strategy (such as hyperparameter setting). The final model loss value will be used in the back propagation algorithm to adjust the model weights to optimize the performance of character prediction and correction. For example, the character prediction model uses the CTC loss function, the character correction model uses the CE loss function, the first loss weight is 0.8, and the second loss weight is 0.2. The model loss value at this time is 0.8*CTCLoss+0.2*CELoss, where CTCLoss is the first loss value and CELoss is the second loss value.

[0147] In the above embodiments, the present application proposes how to combine the outputs and losses of multiple models for effective training and optimization. Through this weighted and calculated method, the overall recognition accuracy and robustness can be improved while maintaining the characteristics of the two models.

[0148] See also Figure 6 , Figure 6It is another optional flow chart of the image-based character recognition method provided by the embodiment of the present application. In some embodiments, after step S160, the image-based character recognition method provided by the embodiment of the present application may also include steps S610 to S640:

[0149] Step S610, if the second prediction probability value is less than or equal to the second prediction threshold, updating the second predicted character based on the preset character;

[0150] Step S620, obtaining a first valid concatenated character from the first predicted character;

[0151] Step S630, obtaining a second valid concatenated character from the second predicted character;

[0152] Step S640 , performing character splicing based on the first valid splicing character and the second valid splicing character to obtain a target recognition character string of the target image.

[0153] In step S610 of some embodiments, the present application can complete the post-processing correction of the model output according to the double threshold rule to achieve recognition output. Specifically, the present application will recognize the characters to be recognized one by one. If the second prediction probability value is less than or equal to the second prediction threshold, it means that the second predicted character obtained is not very accurate. Then, the position of the second predicted character can be updated based on the preset character. The predicted character can let the object know that the character at this position is not recognized and needs further evaluation. Among them, the predicted character can be a blank character or a characteristic symbol character, etc., to distinguish it from other characters.

[0154] It should be noted that if a preset character is recognized in the target recognition string, the target image can be enhanced and the above steps S120 to S150 and S610 to S640 can be repeated again until a complete and accurate target recognition string is found.

[0155] In steps S620 to S640 of some embodiments, since in the vehicle insurance claim settlement scenario such as financial technology, the VIN code to be identified is a string. Since some of the accurate characters identified may come from the character prediction sub-model (i.e., pipline a) and some may come from the character correction sub-model (i.e., pipline b), at this time, the first valid splicing character can be obtained from the first predicted character, and the first valid splicing character is used to characterize the first predicted character whose corresponding first predicted probability value is greater than the first predicted threshold, that is, the accurate first predicted character is identified. And the second valid splicing character is obtained from the second predicted character, and the second valid splicing character is used to characterize the preset character or the second predicted character whose second predicted probability value is greater than the second predicted threshold, that is, the accurate second predicted character is identified. For example, the first predicted string finally spliced ​​by the character prediction sub-model is "102a35b", where the first predicted probability value corresponding to the character "2" is less than the first predicted character, and the first predicted probability value corresponding to the character "b" is also less than the first predicted character. Correspondingly, the character correction sub-model predicts the character "c" for the position corresponding to the character "2", and the corresponding second prediction probability value is greater than the second prediction threshold, while the character correction sub-model predicts the character "d" for the position corresponding to the character "b", and the corresponding second prediction probability value is less than the second prediction threshold. In this way, it can be determined that the first valid spliced ​​characters include "1", "0", "a", "3", "5", and the second valid spliced ​​characters include "c" and the preset character "?" corresponding to the position of the character "b". Furthermore, character splicing can be performed based on the obtained first valid spliced ​​characters and the second valid spliced ​​characters to obtain the target recognition string of the target image.

[0156] See also Figure 7 , Figure 7 It is an optional flow chart of step S640 provided in an embodiment of the present application. In some embodiments, step S640 may specifically include steps S710 to S730:

[0157] Step S710, obtaining a first character position of a first valid splicing character and a second character position of a second valid splicing character;

[0158] Step S720, determining a splicing order based on the first character position and the second character position;

[0159] Step S730: performing character splicing on the first valid splicing character and the second valid splicing character based on the splicing order to obtain a target recognition character string.

[0160] In steps S710 to S730 of some embodiments, when the first valid splicing character and the second valid splicing character are spliced, the correct positions of these characters in the target image need to be considered, and although the first predicted character predicted by the character prediction sub-model is not necessarily accurate, the corresponding character position can be accurately determined. Based on this, the first character position of the first valid splicing character and the second character position of the second valid splicing character can be determined based on the position of the first predicted character. Further, the splicing order of each valid splicing character is determined based on the first character position and the second character position. Finally, the first valid splicing character and the second valid splicing character can be spliced ​​based on the splicing order to obtain the target recognition string. For example, the first valid splicing character includes "1", "0", "a", "3", "5", and the second valid splicing character includes "c" and the preset character "?" corresponding to the position of the character "b". Further, it can be determined that the second valid splicing character "c" is behind the first valid splicing character "0", and the second valid splicing character "?" is behind the first valid splicing character "5". In this way, the first valid splicing character and the second valid splicing character are spliced ​​based on the splicing order, and the target recognition character string is "10ca35?".

[0161] It should be noted that the non-Company software tools or components appearing in the embodiments of the present application are merely examples and do not represent actual use.

[0162] The embodiment of the present application provides an image-based character recognition method, which realizes accurate recognition of characters by constructing a decoding network including a character prediction submodel and a character correction submodel, and can be combined with a dual threshold method to better apply to character recognition in practical applications. Therefore, the present application can improve the accuracy of character recognition in images, such as effectively improving the accuracy of VIN code recognition and improving the overall performance of the model. In practical applications, the present application can promote the development of specific scenarios of character recognition (such as used car systems), improve the degree of digitization of the industry, help buyers and sellers quickly understand the historical records and maintenance of vehicles, thereby reducing transaction risks and further promoting market standardization. At the same time, in the new car market, it can provide fast and accurate vehicle information entry in multiple links such as new car registration, sales, insurance claims, and repair and maintenance, improve work efficiency, and reduce human errors.

[0163] See also Figure 8 The embodiment of the present application further provides an image-based character recognition device, which can implement the above-mentioned image-based character recognition method, and the device includes:

[0164] An acquisition module 810 is used to acquire a target image;

[0165] An input module 820 is used to input a target image into a pre-trained target character recognition model, where the target character recognition model includes a multi-scale feature extraction sub-model, a context feature extraction sub-model, a character prediction sub-model, and a character correction sub-model;

[0166] A first extraction module 830 is used to perform multi-scale feature extraction on the target image based on the multi-scale feature extraction sub-model to obtain multi-scale features of the image;

[0167] A second extraction module 840 is used to extract context features from the image multi-scale features based on the context feature extraction sub-model to obtain image context features;

[0168] A character prediction module 850 is used to perform character prediction on the image context feature based on the character prediction sub-model to obtain a first prediction probability value of a first predicted character;

[0169] The character correction module 860 is used to perform character correction on the image context features based on the character correction sub-model according to the first prediction probability value being less than or equal to the first prediction threshold, so as to obtain a second prediction probability value of the second predicted character.

[0170] The specific implementation of the image-based character recognition device is substantially the same as the specific implementation of the above-mentioned image-based character recognition method, and will not be described in detail here.

[0171] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image-based character recognition method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0172] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0173] The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0174] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the image-based character recognition method of the embodiment of this application;

[0175] Input / output interface 930, used to implement information input and output;

[0176] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0177] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0178] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0179] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image-based character recognition method is implemented.

[0180] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0182] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0183] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0185] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0186] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0187] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0188] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0191] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A character recognition method based on an image, characterized in that: The method comprises: Get the target image; Inputting the target image into a pre-trained target character recognition model, wherein the target character recognition model includes a multi-scale feature extraction sub-model, a context feature extraction sub-model, a character prediction sub-model and a character correction sub-model; Performing multi-scale feature extraction on the target image based on the multi-scale feature extraction sub-model to obtain multi-scale features of the image; Performing context feature extraction on the multi-scale features of the image based on the context feature extraction sub-model to obtain image context features; Performing character prediction on the image context feature based on the character prediction sub-model to obtain a first prediction probability value of a first predicted character; According to the first prediction probability value being less than or equal to a first prediction threshold, character correction is performed on the image context feature based on the character correction sub-model to obtain a second prediction probability value of a second predicted character.

2. The method according to claim 1, characterized in that The method further comprises: If the second prediction probability value is less than or equal to a second prediction threshold, updating the second predicted character based on a preset character; Acquire a first valid concatenated character from the first predicted characters, where the first valid concatenated character is used to represent the first predicted character whose corresponding first predicted probability value is greater than the first prediction threshold; Acquire a second valid concatenated character from the second predicted characters, where the second valid concatenated character is used to represent the preset character or the second predicted character whose second predicted probability value is greater than the second prediction threshold; Character splicing is performed based on the first valid splicing character and the second valid splicing character to obtain a target recognition character string of the target image.

3. The method according to claim 2, characterized in that The character splicing is performed based on the first valid splicing character and the second valid splicing character to obtain the target recognition character string of the target image, including Obtaining a first character position of the first valid splicing character and a second character position of the second valid splicing character; Determining a splicing order based on the first character position and the second character position; The first valid splicing character and the second valid splicing character are spliced ​​based on the splicing order to obtain the target recognition character string.

4. The method according to claim 1, characterized in that The context feature extraction submodel includes a forward context feature extraction submodel, a reverse context feature extraction submodel and a feature concatenation layer. The context feature extraction of the multi-scale features of the image is performed based on the context feature extraction submodel to obtain the image context feature, including: Performing context feature extraction on the multi-scale features of the image based on the forward context feature extraction sub-model to obtain a forward context feature, wherein the forward context feature includes forward context sub-features arranged in time step order; Performing context feature extraction on the multi-scale features of the image based on the reverse context feature extraction sub-model to obtain reverse context features, wherein the reverse context features include reverse context sub-features arranged in reverse order according to time steps; Based on the feature splicing layer, feature splicing is performed on the forward context sub-feature and the reverse context sub-feature corresponding to the time step to obtain the image context feature.

5. The method according to any one of claims 1 to 4, characterized in that: Before inputting the target image into a pre-trained target character recognition model, the method further includes: constructing the target character recognition model, specifically including: Acquire a training sample, wherein the training sample includes a sample image and a sample character string; Inputting the sample image into an initial character recognition model, wherein the initial character recognition model includes the multi-scale feature extraction sub-model, the context feature extraction sub-model, the character prediction sub-model and the character correction sub-model; Performing multi-scale feature extraction on the sample image based on the multi-scale feature extraction sub-model to obtain multi-scale features of the sample image; Performing context feature extraction on the multi-scale features of the sample image based on the context feature extraction sub-model to obtain context features of the sample image; Performing character prediction on the sample image context feature based on the character prediction sub-model to obtain a first sample prediction probability value of a first predicted sample character; Performing character correction on the sample image context feature based on the character correction sub-model to obtain a second sample prediction probability value of a second predicted sample character; splicing the first predicted sample characters based on the first sample predicted probability value to obtain a first predicted character string, and splicing the second predicted sample characters based on the second sample predicted probability value to obtain a second predicted character string; A model loss value is determined based on the first predicted character string, the second predicted character string and the sample character string, and parameters of the initial character recognition model are adjusted based on the model loss value to obtain the target character recognition model.

6. The method according to claim 5, characterized in that The character correction sub-model includes a first attention layer, a feature fusion layer and a prediction layer, and the character correction is performed on the sample image context feature based on the character correction sub-model to obtain the second sample prediction probability value of the second prediction sample character, including: Extracting sample positive context features and sample reverse context features from the sample image context features; Based on the first attention layer, feature attention processing is performed on the sample forward context feature and the sample reverse context feature to obtain a sample context attention feature; Based on the feature fusion layer, the sample context attention feature and the sample reverse context feature are subjected to feature fusion to obtain a sample target context feature; Character correction is performed on the sample target context feature based on the prediction layer to obtain the second sample prediction probability value of the second prediction sample character.

7. The method according to claim 5, characterized in that The determining of the model loss value based on the first predicted character string, the second predicted character string and the sample character string comprises: Obtaining a first model weight of the character prediction sub-model and a second model weight of the character correction sub-model; Performing loss calculation based on the first predicted character string and the sample character string to obtain a first loss value; Performing loss calculation based on the second predicted character string and the sample character string to obtain a second loss value; The model loss value is obtained by performing a weighted sum calculation based on the first model weight, the first loss value, the second model weight and the second loss value.

8. An image-based character recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring a target image; An input module, used for inputting the target image into a pre-trained target character recognition model, wherein the target character recognition model includes a multi-scale feature extraction sub-model, a context feature extraction sub-model, a character prediction sub-model and a character correction sub-model; A first extraction module, configured to perform multi-scale feature extraction on the target image based on the multi-scale feature extraction sub-model to obtain multi-scale features of the image; A second extraction module, configured to perform context feature extraction on the image multi-scale features based on the context feature extraction sub-model to obtain image context features; A character prediction module, configured to perform character prediction on the image context feature based on the character prediction sub-model to obtain a first prediction probability value of a first predicted character; A character correction module is used to perform character correction on the image context feature based on the character correction sub-model according to the first prediction probability value being less than or equal to a first prediction threshold, so as to obtain a second prediction probability value of a second predicted character.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images