Text image recognition method and apparatus

Abstract

Provided are a text image recognition method and apparatus. The method comprises: acquiring a text line image to be subjected to recognition; using said text line image as input to an encoder, and processing said text line image by means of several encoding units in the encoder to obtain a latent representation, wherein at least one encoding unit among the several encoding units comprises a visual prompt layer, which is used for providing local context information and global context information of said text line image; and using the latent representation as input to a decoder, and outputting recognized text. In the present application, adding a visual prompt layer in an encoding unit realizes the introduction of local context information and global context information of a text line image to be subjected to recognition in a feature extraction process, thereby facilitating the utilization of spatial context information of character shapes, and thus improving the accuracy of text recognition.

Description

Method and device for recognizing character image

[0001] The present application claims priority to the Chinese patent application No. 202411088694.9, filed on August 8, 2024, and entitled "Method and device for recognizing character image", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI), and in particular, to a method and device for recognizing character image. BACKGROUND

[0003] With the popularity of mobile Internet, the demand for recognizing Chinese handwritten text images captured by mobile phones under open conditions is increasing. The influence of different light, angle of view, and degradation in the imaging process brings more challenges to the development of handwritten character recognition technology. Therefore, handwritten character recognition under uncontrolled conditions is still a problem to be solved. SUMMARY

[0004] Embodiments of the present application provide a method and device for recognizing character image, which introduce local context information and global context information of the text line image to be recognized in the feature extraction process, which helps to utilize the spatial context information of the character shape, and further improves the accuracy of character recognition under different scenarios (e.g., uncontrolled scenarios).

[0005] In a first aspect, the present application provides a method for recognizing character image, comprising: obtaining a text line image to be recognized; taking the text line image to be recognized as an input of an encoder, and processing the text line image to be recognized by a plurality of encoding units in the encoder to obtain a hidden representation, wherein at least one of the plurality of encoding units comprises a visual cue layer, and the visual cue layer is configured to provide local context information and global context information of the text line image to be recognized; taking the hidden representation as an input of a decoder, and outputting a recognized text.

[0006] It should be noted that the local context information indicates the character shape contour information of the current recognized character, and the global context information indicates the overall contour information of the text line in which the current recognized character (or the character to be decoded and predicted at the current decoding time step) is located; for example, the text line image to be recognized is a text line image of "Danxin Zhaofeng Xie, Pifu Ri Yijinggai", and the current recognized character is "Zhaofeng", then the local context information is the character shape contour information of "Zhaofeng", and the global context information is the overall contour information of the text line "Danxin Zhaofeng Xie, Pifu Ri Yijinggai" in which the current recognized character "Zhaofeng" is located.

[0007] The application introduces local context information and global context information of the image of the text line to be recognized in the feature extraction process by adding a visual cue layer in the encoding unit, which helps to utilize the spatial context information of the glyph, and further improves the accuracy of character recognition in different scenes.

[0008] In a possible implementation, the at least one encoding unit further includes a feature extraction layer; and the processing of the input feature map by the at least one encoding unit includes: inputting the feature map as an input of the feature extraction layer to output an intermediate feature map; and inputting the intermediate feature map into the visual cue layer to obtain an output feature map, the output feature map including the local context information and the global context information of the image of the text line to be recognized, the input feature map being the image of the text line to be recognized or an output feature map output by a previous encoding unit of the at least one encoding unit in the plurality of encoding units. For example, the encoder includes an encoding unit 1, an encoding unit 2, an encoding unit 3 and an encoding unit 4 stacked in sequence, when the visual cue layer is added in the encoding unit 1, that is, the encoding unit 1 includes the feature extraction layer and the visual cue layer, at this time, the input feature map of the encoding unit 1 is the image of the text line to be recognized. For the encoding unit in which the visual cue layer is added in a non-encoding unit 1, for example, the encoding unit 3, at this time, the input feature map of the encoding unit 3 is the output feature map output by the previous encoding unit of the encoding unit 3, that is, the encoding unit 2.

[0009] In this possible implementation, the visual cue layer injects cue information into the feature map, the cue information including the local context information and the global context information of the image of the text line to be recognized, so that the spatial context information of the glyph is contained in the implicit representation output by the encoder, which is beneficial to the decoder to obtain the text information contained in the image of the text line.

[0010] In another possible implementation, the visual cue layer includes a meta cue, a local aggregation layer and a global aggregation layer, the meta cue including a plurality of cue vectors, the plurality of cue vectors being obtained based on a training data set, the training data set including a plurality of text line image samples with labels; and one specific implementation of inputting the intermediate feature map into the visual cue layer to obtain the output feature map includes: performing block processing on the intermediate feature map to obtain M feature map subblocks, M being a positive integer greater than 1; determining M initial visual cues corresponding to the M feature map subblocks based on the meta cue; inputting the M feature map subblocks and the M initial visual cues into the local aggregation layer to output M local feature map subblocks and M local visual cues; and inputting the M local feature map subblocks and the M local visual cues into the global aggregation layer to obtain the output feature map.

[0011] The local context information of the text line image to be recognized is aggregated by processing the intermediate feature maps using the local aggregation layer, which is beneficial to recognizing the local structure of the single character image, and the global context information of the text line image to be recognized is obtained by processing using the global aggregation layer, which is beneficial to obtaining the overall writing style of the text line.

[0012] In another possible implementation, M feature map subblocks and M initial visual cues are taken as inputs of the local aggregation layer, and one specific implementation of outputting M local feature map subblocks and M local visual cues is that: multi-head self-attention mechanism calculation is respectively performed on each of the M feature map subblocks to obtain the M local feature map subblocks; and multi-head cross-attention mechanism calculation is respectively performed on each of the M initial visual cues and each of the M local feature map subblocks to obtain the M local visual cues.

[0013] In another possible implementation, M local feature map subblocks and M local visual cues are taken as inputs of the global aggregation layer, and one specific implementation of obtaining the output feature map is that: multi-head self-attention mechanism calculation is performed between the M local visual cues to obtain M global visual cues; multi-head cross-attention mechanism calculation is respectively performed on each of the M local feature map subblocks and each of the M global visual cues to obtain M global feature map subblocks; and the M global feature map subblocks are spliced to obtain the output feature map.

[0014] In another possible implementation, the implicit representation is determined based on the output feature map output by the last encoding unit of the plurality of encoding units. For example, the output feature map output by the last encoding unit is averaged along the height direction to obtain the implicit representation.

[0015] In another possible implementation, the weight parameters of the visual cue layer are obtained by optimizing and solving in a training stage with a target of maximizing local mutual information and global mutual information. The local mutual information is determined based on M local visual cues, M first edge image subblocks, and M second edge image subblocks. The global mutual information is determined based on the M global visual cues, the M first edge image subblocks, and the M second edge image subblocks. The M first edge image subblocks are obtained by dividing the first edge image, and the M second edge image subblocks are obtained by dividing the second edge image. The first edge image is obtained by performing edge extraction on the input text line image sample, and the second edge image is obtained by performing edge extraction on a different text line image sample of the input text line image sample.

[0016] In the possible implementation, in the training stage, the learning of the visual cue is guided by maximizing the mutual information (including local mutual information and global mutual information) of the visual cue and the image edge of the text line, so that the visual cue learned by the visual cue layer focuses on the geometric structure information in the text line image, and the training effect is improved.

[0017] In another possible implementation, the decoder includes a decoding unit, a visual representation generation unit and a character embedding layer; and a specific implementation of outputting the recognized text by taking the implicit representation as the input of the decoder is: taking the implicit representation as the input of the visual representation generation unit and outputting a visual representation sequence; taking the recognized text output before the current decoding time step (which can also be referred to as the recognized text) as the input of the character embedding layer and outputting a character embedding sequence; taking the character embedding sequence as the input of the decoding unit and outputting the recognized text. <sos>The start symbol) as the input of the character embedding layer, and output a recognized text embedding representation sequence; the hidden representation, the visual representation sequence and the recognized text embedding representation sequence as the input of the decoding unit, and output a character prediction result of a current decoding time step, if the character prediction result is <eos>an end symbol, the decoding process ends. The character prediction results output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the text line image to be recognized.

[0018] In this possible implementation, the attention drift problem in the encoder-decoder architecture is alleviated by introducing visual feature decoding guidance, further improving the performance of character recognition.

[0019] In another possible implementation, the visual representation generation unit includes a connectionist temporal classification (CTC) decoding layer and a visual feature sampling layer; a specific implementation of taking the hidden representation as the input of the visual representation generation unit and outputting a visual representation sequence is as follows: taking the hidden representation as the input of the CTC decoding layer and outputting a character probability sequence; based on the character probability sequence, obtaining a character position estimate and a character confidence; taking the hidden representation, the character position estimate, and the character confidence as the input of the visual feature sampling layer and outputting the visual representation sequence.

[0020] In another possible implementation, the text line image to be recognized includes one or more handwritten characters. It should be noted that the meaning of the handwritten characters mentioned here is the handwritten character images in the text line image to be recognized.

[0021] In another possible implementation, before the text line image to be recognized is input into the encoder, the text line image to be recognized is further preprocessed, and the preprocessing includes at least one of height normalization processing and pixel value normalization preprocessing.

[0022] In another possible implementation, in the training phase, the multiple text line image samples in the training data set all need to be preprocessed, for example, data enhancement processing is performed on the multiple text line image samples in the training data set, and the data enhancement processing includes at least one of random rotation processing, local pixel perturbation processing, and Gaussian low-pass filtering processing.

[0023] In a second aspect, the present application provides a character image recognition device, including an acquisition module and an inference module, wherein the acquisition module is configured to acquire a text line image to be recognized; the inference module is configured to take the text line image to be recognized as the input of an encoder, and obtain a hidden representation by processing the text line image to be recognized through a plurality of encoding units in the encoder, wherein at least one of the plurality of encoding units includes a visual cue layer, and the visual cue layer is configured to provide local context information and global context information of the text line image to be recognized; and take the hidden representation as the input of a decoder, and output a recognized text.

[0024] In a possible implementation, the at least one encoding unit further includes a feature extraction layer; and processing, by the at least one encoding unit, the input feature map includes: inputting the feature map as an input of the feature extraction layer to output an intermediate feature map; and inputting the intermediate feature map into the visual cue layer to obtain an output feature map, the output feature map including local context information and global context information of the image of the text line to be recognized, the input feature map being the image of the text line to be recognized or an output feature map output by a previous encoding unit of the at least one encoding unit in the plurality of encoding units.

[0025] In another possible implementation, the visual cue layer includes a meta cue, a local aggregation layer, and a global aggregation layer, the meta cue including a plurality of cue vectors, the plurality of cue vectors being trained based on a training data set, the training data set including a plurality of text line image samples with labels; and a specific implementation of obtaining the output feature map by inputting the intermediate feature map as an input of the visual cue layer includes: performing block processing on the intermediate feature map to obtain M feature map subblocks, M being a positive integer greater than 1; determining M initial visual cues corresponding to the M feature map subblocks based on the meta cue; inputting the M feature map subblocks and the M initial visual cues as inputs of the local aggregation layer to output M local feature map subblocks and M local visual cues; and inputting the M local feature map subblocks and the M local visual cues as inputs of the global aggregation layer to obtain the output feature map.

[0026] In another possible implementation, a specific implementation of inputting the M feature map subblocks and the M initial visual cues as inputs of the local aggregation layer to output the M local feature map subblocks and the M local visual cues includes: performing multi-head self-attention mechanism calculation on each of the M feature map subblocks to obtain the M local feature map subblocks; and performing multi-head cross-attention mechanism calculation on each of the M initial visual cues and each of the M local feature map subblocks to obtain the M local visual cues.

[0027] In another possible implementation, a specific implementation of inputting the M local feature map subblocks and the M local visual cues as inputs of the global aggregation layer to obtain the output feature map includes: performing multi-head self-attention mechanism calculation on the M local visual cues to obtain M global visual cues; performing multi-head cross-attention mechanism calculation on each of the M local feature map subblocks and each of the M global visual cues to obtain M global feature map subblocks; and splicing the M global feature map subblocks to obtain the output feature map.

[0028] In another possible implementation, the implicit representation is determined based on an output feature map output by a last encoding unit in the plurality of encoding units. For example, the output feature map output by the last encoding unit is averaged along a height direction to obtain the implicit representation.

[0029] In another possible implementation, the weight parameters of the visual cue layer are obtained based on optimization in a training phase with an objective of maximizing local mutual information and global mutual information, the local mutual information is determined based on M local visual cues, M first edge image sub-blocks and M second edge image sub-blocks, the global mutual information is determined based on the M global visual cues, M first edge image sub-blocks and M second edge image sub-blocks, the M first edge image sub-blocks are obtained based on block division of a first edge image, the M second edge image sub-blocks are obtained based on block division of a second edge image, the first edge image is obtained based on edge extraction on an input text line image sample, and the second edge image is obtained based on edge extraction on a different text line image sample from the input text line image sample.

[0030] In another possible implementation, the decoder comprises a decoding unit, a visual representation generation unit and a character embedding layer; and one specific implementation of outputting the recognized text with the hidden representation as an input of the decoder comprises: inputting the hidden representation as an input of the visual representation generation unit, and outputting a visual representation sequence; inputting the recognized text output before a current decoding time step as an input of the character embedding layer, and outputting a recognized text embedding representation sequence; inputting the hidden representation, the visual representation sequence and the recognized text embedding representation sequence as inputs of the decoding unit, and outputting a character prediction result at the current decoding time step, wherein if the character prediction result is <eos>an end symbol, the decoding process ends. The character prediction results output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the image of the text line to be recognized.

[0031] In another possible implementation, the visual representation generation unit comprises a CTC decoding layer and a visual feature sampling layer; a specific implementation of outputting the visual representation sequence based on the hidden representation as the input of the visual representation generation unit comprises: inputting the hidden representation as the input of the CTC decoding layer to output a character probability sequence; obtaining a character position estimate and a character confidence based on the character probability sequence; inputting the hidden representation, the character position estimate and the character confidence as the input of the visual feature sampling layer to output the visual representation sequence.

[0032] In another possible implementation, the image of the text line to be recognized comprises one or more handwritten characters.

[0033] In another possible implementation, before inputting the image of the text line to be recognized into the encoder, the method further comprises: performing preprocessing on the image of the text line to be recognized, the preprocessing comprising at least one of height normalization processing and pixel value normalization preprocessing.

[0034] In another possible implementation, in the training phase, the multiple text line image samples in the training data set all need to perform preprocessing, for example, data enhancement processing is performed on the multiple text line image samples in the training data set, and the data enhancement processing comprises at least one of random rotation processing, local pixel disturbance processing and Gaussian low-pass filtering processing.

[0035] In a third aspect, an embodiment of the present application provides a computing device, comprising a memory and a processor, and the memory stores instructions, when the instructions are executed by the processor, the method for recognizing a text image described in the first aspect or any possible implementation manner of the first aspect is implemented.

[0036] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, when the computer program is executed by a processor, the method for recognizing a text image described in the first aspect or any possible implementation manner of the first aspect is implemented.

[0037] In a fifth aspect, an embodiment of the present application further provides a computer program or a computer program product, the computer program or the computer program product comprises instructions, when the instructions are executed, the computer executes the method for recognizing a text image described in the first aspect or any possible implementation manner of the first aspect.

[0038] In a sixth aspect, the embodiments of the present application further provide a chip comprising at least one processor and a communication interface, and the processor is configured to execute the character image recognition method described in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0039] Fig. 1 shows a schematic diagram of an artificial intelligence subject framework;

[0040] Fig. 2 is a system architecture diagram of a sample processing system provided by the embodiments of the present application;

[0041] Fig. 3 is a schematic diagram of the hardware structure of a chip provided by the embodiments of the present application;

[0042] Fig. 4 is a flowchart of a character image recognition method provided by the embodiments of the present application;

[0043] Fig. 5 shows the structure of a handwritten character recognition model provided by the embodiments of the present application, and a schematic diagram of the processing of a handwritten text line image by the handwritten character recognition model;

[0044] Fig. 6 shows the structure of an encoding visual cue learning module in the embodiments of the present application, and a schematic diagram of the processing process of the intermediate feature map X by the encoding visual cue learning module;

[0045] Fig. 7 shows the structure of a decoding guide visual feature generation module in the decoder of the embodiments of the present application, and a schematic diagram of the processing process of the implicit representation;

[0046] Fig. 8 is a schematic diagram of the handwritten character recognition results of the embodiments of the present application and the baseline model, and the visualization of the decoder attention coefficient;

[0047] Fig. 9 is a schematic diagram of the visualization of the feature map in the encoder of the embodiments of the present application and the baseline model;

[0048] Fig. 10 is a schematic diagram of the structure of a character image recognition device provided by the embodiments of the present application;

[0049] Fig. 11 is a schematic diagram of the structure of a computing device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0050] The term "and / or" mentioned herein is a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol " / " in this paper represents the relationship of or in the associated objects, for example, A / B represents A or B.

[0051] The terms "first" and "second" and the like in the description and in the claims of this patent document are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the present application described herein are, for example, capable of use in either order. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. The use of the terms "including," "comprising," or "having" and variations thereof in this description and in the claims are not limiting, such that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to only those elements but can include other elements not expressly listed or inherent to such process, method, system, product, or apparatus.

[0052] In the embodiments of the present application, the word "exemplary" or "for example" is used to mean serving as an example, instance, or illustration. Any implementation or design solution described as "exemplary" or "for example" in the embodiments of the present application is not necessarily to be construed as preferred or advantageous over other implementations or design solutions. In fact, a variety of implementations and design solutions are contemplated as can serve one or more purposes, or provide one or more benefits, or address one or more issues. The use of the word "exemplary" or "for example" is merely intended to present one or more examples in connection with the related concept.

[0053] In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0054] First, the overall workflow of the artificial intelligence system is described, please refer to Figure 1, which shows an artificial intelligence main framework diagram, which describes the overall workflow of the artificial intelligence system, which is suitable for general artificial intelligence field requirements.

[0055] The above-mentioned artificial intelligence theme framework is described below from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom". "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the bottom infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system. From the dimension of IT value chain, the artificial intelligence main framework includes the infrastructure layer, the data layer, the data processing layer, the general capability layer (some general capabilities are realized through the processing results of data, such as translation, text analysis, computer vision processing, speech recognition, image recognition, etc.), and the uppermost application layer, which is the encapsulation of the overall solution of artificial intelligence, productization of intelligent information decision-making, and realization of landing application.

[0056] (1) Infrastructure

[0057] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips, including but not limited to central processing units (CPU), neural-network processing units (NPU), graphics processing units (GPU), application specific integrated circuits (ASIC), and field programmable gate arrays (FPGA) and other hardware acceleration chips; the underlying platform includes distributed computing frameworks and related platform guarantees and support for networks, which can include cloud storage and computing, and interconnected networks. For example, sensors and external communication obtain data, which is provided to intelligent chips in the distributed computing system provided by the underlying platform for computing.

[0058] (2) Data

[0059] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. Data involves graphics, images, speech, text, and also involves Internet of Things data from traditional devices, including business data from existing systems and sensing data such as force, displacement, liquid level, temperature, and humidity.

[0060] (3) Data processing

[0061] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, and other methods.

[0062] Among them, machine learning and deep learning can model, extract, preprocess, and train symbolic and formalized intelligent information.

[0063] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to perform machine thinking and solve problems according to reasoning control strategies, and the typical function is search and matching.

[0064] Decision-making refers to the process of making decisions after intelligent information has been reasoned, and usually provides functions such as classification, sorting, and prediction.

[0065] (4) General capabilities

[0066] After the data is processed by the above-mentioned data processing, some general capabilities can be formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, and the like.

[0067] (5) Intelligent product and industry application

[0068] Intelligent product and industry application refers to the product and application of artificial intelligence system in various fields, which is the packaging of the overall solution of artificial intelligence, and realizes the application of intelligent information decision productization. Its application fields mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent medical treatment, intelligent security, automatic driving, safe city, intelligent terminal, etc.

[0069] The character image recognition method and device provided by the embodiment of the application can improve the accuracy of character recognition in different scenarios by setting a visual prompt layer in the encoder, which helps to utilize the spatial context information of the character shape and further improve the accuracy of character recognition in different scenarios. That is, the embodiment of the application improves the character recognition model, which is an encoder-decoder architecture. The visual prompt layer is set in the encoder, and the local context information and global context information of the to-be-recognized text line image are injected into the feature map coded by the encoder through the visual prompt layer, thereby improving the accuracy of character recognition in different scenarios.

[0070] The character image recognition method and device provided by the embodiment of the application can be applied to various scenarios, such as enterprise document processing and electronicization scenarios of contracts, shipping lists, and signed lists, education business, and electronicization scenarios of personal data. In the enterprise document processing and electronicization application scenarios, the character recognition model provided by the embodiment of the application can quickly convert a large amount of handwritten documents, signatures, and the like into editable electronic documents, assist electronic archiving, and also assist experts in improving efficiency and accelerating downstream risk identification. In the education business and personal data electronicization application scenarios, the character recognition model provided by the embodiment of the application can accurately extract the handwritten character content in notes and personal data containing handwritten characters, assist in document electronicization and document content modification, and improve work efficiency.

[0071] The character recognition model provided by the embodiment of the application can be deployed in the server side, website, and mobile application to meet the handwriting recognition needs of enterprise users and individual users; and can also be used for secondary development and use by developers through an open API interface.

[0072] For the convenience of understanding the scheme, first, the sample processing system provided by the embodiments of the present application is introduced in combination with FIG. 2. Referring to FIG. 2, the data acquisition device 260 is configured to acquire training data. The training data of the embodiments of the present application includes a plurality of text line images with labels. The label is the text truth value corresponding to the text line image, that is, the text corresponding to the text image contained in the text line image. For example, if the text line image is an image with the text "day after the mountain", the label of the text line image is the text sequence "day after the mountain". After the training data is acquired, the data acquisition device 260 stores the training data in the database 230. The training device 220 iteratively optimizes and solves the model parameters based on the target function based on the training data maintained in the database 230, so as to obtain the target model / rule 201.

[0073] In actual application, the training data maintained in the database 230 may not all come from the acquisition of the data acquisition device 260, but may also be received from other devices. In addition, it should be noted that the training device 220 may not train the target model / rule 201 based on the training data maintained in the database 230, but may also obtain the training data from the cloud or other places for model training. The above description should not be regarded as a limitation of the embodiments of the present application.

[0074] The target model / rule 201 trained by the training device 220 can be applied to different systems or devices, such as the execution device 210 in FIG. 2. The execution device 210 can be a terminal, such as a mobile phone, a tablet computer, a notebook computer, an augmented reality (AR), a virtual reality (VR), a wearable device, a smart robot, a vehicle terminal, etc. It can also be a server or a cloud, etc.

[0075] Specifically, the training device 220 can deploy the trained AI model to the execution device 210.

[0076] In FIG. 2, the execution device 210 is configured with an input / output (I / O) interface 212 for data exchange with external devices. The user can input data to the I / O interface 212 through the client device 240. In this embodiment, the data can include a text line image to be recognized.

[0077] The preprocessing module 213 and the preprocessing module 214 are configured to perform preprocessing on the input data received by the I / O interface 212 (such as the text line image data to be recognized input by the user). For example, the text line image to be recognized is subjected to height normalization, pixel normalization preprocessing, and data enhancement processing on the training data set in the training stage.

[0078] It should be understood that there can be no pre-processing modules 213 and 214, or only one pre-processing module. When there is no pre-processing module 213 and 214, the input data can be directly processed by the computing module 211.

[0079] The computing module 211 is configured to perform calculations and other related processes on the data input from the pre-processing modules 213 and 214 according to the target model / rule 201 described above.

[0080] During the pre-processing of the input data by the execution device 210, or during the calculation and other related processes performed by the computing module 211 of the execution device 210, the execution device 210 can call data, code, etc. of the database storage system 250 for corresponding processing, or store the data, instructions, etc. obtained by the corresponding processing in the data storage system 250.

[0081] Finally, the I / O interface 212 returns the processing result (recognized text) to the client device 240 for the user.

[0082] It should be noted that the training device 220 can generate a target model / rule 201 corresponding to a different downstream system, which can achieve the above-mentioned target or complete the above-mentioned task, thereby providing the user with the required result. It should be noted that the training device 220 can also generate a corresponding pre-processing model for the target model / rule 201 corresponding to a different downstream system, such as a corresponding pre-processing model in the pre-processing module 213 and / or the pre-processing module 214, etc.

[0083] In the case shown in FIG. 2, the user can manually specify the data input into the execution device 210 (e.g., input the image of the text line to be recognized), for example, by operating in the interface provided by the I / O interface 212. In another case, the client device 240 can automatically input data (e.g., input the image of the text line to be recognized) into the I / O interface 212 and obtain the result, and if the client device 240 needs to obtain the authorization of the user for automatic data input, the user can set the corresponding permission in the client device 240. The user can view the result output by the execution device 210 (e.g., the output recognized text, etc.) in the client device 240, and the specific presentation form can be display, sound, action, etc. The client device 240 can also act as a data collection end to collect the input data (the image of the text line to be recognized) input into the I / O interface 212 and the output result of the target model / rule 201 as shown in FIG. 2, and store them as new sample data in the database 230.

[0084] It is notable that FIG. 2 is only a schematic diagram of a system architecture provided by the embodiments of the present application, and the positional relationship between the devices, elements, modules and the like shown in the diagram does not constitute any limitation. For example, in FIG. 2, the data storage system 250 is external memory relative to the execution device 210, and in other cases, the data storage system 250 can also be placed in the execution device 210.

[0085] FIG. 3 is a schematic diagram of a hardware structure of a chip provided by the embodiments of the present application. The chip includes a neural network processing unit (NPU) 30. The chip can be arranged in the execution device 210 as shown in FIG. 2, and used to complete the computing work of the computing module 211. The chip can also be arranged in the training device 220 as shown in FIG. 2, and used to complete the training work of the training device 220 and output the target model / rule 201.

[0086] The text recognition method of the character image provided by the embodiments of the present application can be executed in the operation circuit 303 and / or the vector calculation unit 307 in the neural network processing unit 30, so as to obtain the output text recognition result.

[0087] The following briefly introduces each module and unit in the neural network processing unit 30.

[0088] The neural network processing unit 30 is mounted as a coprocessor on a host CPU, and tasks are allocated by the host CPU. The core part of the neural network processing unit 30 is the operation circuit 303. When the neural network processing unit 30 works, the controller 304 in the neural network processing unit 30 can control the operation circuit 303 to extract matrix data in the memory and perform multiplication operation.

[0089] In some implementations, the operation circuit 303 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 303 is a two-dimensional systolic array. The operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0090] For example, it is assumed that there are an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of the matrix B from the weight memory 302 and buffers it on each PE of the operation circuit. The operation circuit takes the matrix A data from the input memory 301 and performs matrix operation with the matrix B, and the partial result or final result of the obtained matrix is saved in the accumulator 308.

[0091] The vector computation unit 307 can perform further processing on the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, magnitude comparison, etc. For example, the vector computation unit 307 can be used for network computation in non-convolution / non-fully connected layers (FC) in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0092] In some implementations, the vector computation unit 307 can store the processed output vector to the unified buffer 306. For example, the vector computation unit 307 can apply a non-linear function to the output of the arithmetic circuit 303, e.g., a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 307 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 303, e.g., for use in a subsequent layer in a neural network.

[0093] The unified memory 306 is used to store input data and output data.

[0094] The weight data is transferred from external memory to the input memory 301 and / or the unified memory 306, and from the unified memory 306 to external memory, by the direct memory access controller (DMAC) 305.

[0095] The bus interface unit (BIU) 310 is used to interact between the main CPU, the DMAC, and the instruction fetch buffer 309 via a bus.

[0096] The instruction fetch buffer 309 is connected to the controller 304, and is used to store instructions used by the controller 304.

[0097] The controller 304 is used to invoke instructions stored in the instruction fetch buffer 309 to control the operation of the arithmetic accelerator.

[0098] Generally, the unified memory 306, the input memory 301, the weight memory 302, and the instruction memory 309 can all be on-chip memories. The external memory of the NPU can be a memory external to the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM), or other readable and writable memory. It should be understood that the chip hardware structure shown in FIG. 3 is only an exemplary illustration, and the present application is not limited thereto.

[0099] In recent years, deep learning-based text recognition methods have achieved better performance than traditional methods. Handwritten text line recognition methods based on deep learning can be roughly divided into two types: character segmentation-based methods and sequence modeling-based methods. Sequence modeling-based methods are mainstream methods and can also be divided into two types: CTC decoding-based methods and attention mechanism-based encoder-decoder methods. CTC decoding-based methods lack modeling of text context relationships during decoding and often require the introduction of additional language models. Attention mechanism-based encoder-decoder methods generally use autoregressive decoding with implicit modeling of text context relationships. In recent years, Transformer self-attention mechanism-based encoder-decoder methods have shown good performance in handwritten text recognition tasks, but are prone to attention drift problems during decoding of long texts.

[0100] Related technology I, a handwritten text recognition method based on character segmentation, which first divides a text line image into individual characters and then recognizes the characters. For example, after preprocessing, a handwritten Chinese text line image with a height of 128 pixels is input into the network; a backbone network generates hierarchical feature maps with step sizes of {2, 4, 8, 16} pixels relative to the input text line image. Based on the final feature map of the backbone network, a segmentation and recognition module composed of a position, detection, and classification branch simultaneously outputs segmentation and recognition results with the help of character segmentation.

[0101] However, this solution usually requires training for character segmentation, and character-level bounding box position annotations for training character segmentation models are difficult to obtain, and errors in the character segmentation link have a greater impact on subsequent recognition results.

[0102] The related art two, a handwritten character recognition scheme based on sequence modeling, such as a handwritten character recognition scheme based on CTC decoding, and a handwritten character recognition scheme based on an encoder-decoder. The implementation architecture of the handwritten character recognition scheme based on the encoder-decoder includes a visual encoder for extracting 2D visual features, and a multi-modal decoder with probabilistic masking attention for context-selective decoding.

[0103] The method based on CTC decoding lacks modeling of text context dependency, and usually needs to introduce an external language model. The method based on the encoder-decoder has self-recursive decoding with implicit modeling of text context relationships, but is prone to attention drift problems affecting recognition effect.

[0104] In practical applications, the writing style of characters varies greatly, often greatly different from the statistical distribution of training samples, and puts high requirements on the generalization ability of the model. In the field of natural language processing, prompt learning is widely used to adapt pre-trained models to downstream tasks. Prompt learning refers to selecting or generating a set of artificially designed or learnable feature vectors containing specific task information as prompts to guide the model to extract information required for specific tasks from input data, so as to improve the generalization performance of the model. At present, researchers have applied prompt learning to image classification tasks, using a lightweight neural network to generate a feature vector related to the input image as a visual prompt based on the global features of the input image, so that the model can better adapt to the classification of unknown categories in an open condition. However, the current visual prompt learning method only processes global information of the image, while in the handwritten text line recognition task, not only the local structure of a single character needs to be recognized, but also the overall writing style of the text line needs to be adapted. Therefore, a more flexible and efficient visual prompt learning method can be considered for the handwritten character recognition task.

[0105] Therefore, the embodiments of the present application provide a character image recognition method and device, which adds a visual prompt layer in the encoder to introduce local context information and global context information of the text line image to be recognized in the feature extraction process, which is helpful to utilize the spatial context information of the character shape, and further improve the accuracy of character recognition in different scenarios.

[0106] The character image recognition method and device provided in the application can be applied to effectively recognize various images containing characters, such as recognition of printed character images, recognition of natural scene character images, and recognition of handwritten character images, but not limited to the above. In particular, the method can be applied to recognize handwritten character images, and support accurate recognition of handwritten character images in various scenes. In addition, the character image recognition method and device provided in the application can be applied to recognize characters in any language, such as Chinese, English, Japanese, Korean, Arabic, and the like.

[0107] The application takes the recognition of handwritten character images as an example to introduce the specific implementation of the character image recognition method and device provided in the application. The recognition of other types of character images is similar to the recognition of handwritten character images, and the specific implementation can refer to the specific implementation of the recognition of handwritten character images. For brevity, the application will not be described again.

[0108] The character image recognition method and device provided in the application will be described in detail below with reference to the accompanying drawings.

[0109] FIG. 4 is a flowchart of a character image recognition method provided in the application. The method can be implemented by the execution device 210 in FIG. 2. In the feature extraction process, the local context information and the global context information of the to-be-recognized text line image are introduced, which is helpful to utilize the spatial context information of the character shape, and further improve the accuracy of character recognition in different scenes. As shown in FIG. 4, the character image recognition method provided in the application includes at least steps S401 to S403.

[0110] In step S401, a to-be-recognized text line image is obtained.

[0111] In the application, a handwritten character recognition model is deployed in the execution device, which can effectively recognize handwritten text line images and output recognized text. That is, the handwritten character recognition model can accurately recognize the handwritten character content in the to-be-recognized handwritten text line image, so as to assist in document digitization and document content modification, and the like. Optionally, the handwritten character recognition model can convert non-editable image format handwritten characters into editable text format. When a user needs to recognize a handwritten text line image, the user can upload the to-be-recognized handwritten text line image through a client, and then transmit the to-be-recognized handwritten text line image to the execution device through an I / O interface. The execution device obtains the to-be-recognized handwritten text line image.

[0112] In another example, the image of the text line to be recognized can also be obtained by camera shooting. For example, the client has two controls of "upload" and "shoot" on the interactive interface for obtaining the image. When it is detected that the user selects the "shoot" option, the camera is called to shoot. The text is located in the shooting angle by adjusting the angle of the camera. The image is obtained by clicking the shooting. The image is transmitted to the execution device through the I / O interface.

[0113] The execution device can be a terminal such as a mobile phone, a tablet computer, a notebook computer, an augmented reality, a virtual reality, a wearable device, a smart robot, a vehicle-mounted terminal, etc. It can also be a server or a cloud, etc.

[0114] It is easy to understand that the text line image mentioned in the embodiments of the present application refers to a character image with a certain order. The text line can be vertical or horizontal, and can contain one word or multiple words. For a text line image or multiple text images in a complex background, one or more text line images can be obtained by processing in a front-end text line detection or text line segmentation module. The embodiments of the present application do not make specific limitations.

[0115] In another example, after obtaining the image of the handwritten text line to be recognized, the image of the handwritten text line to be recognized is preprocessed, for example, image preprocessing of the image of the handwritten text line to be recognized, including but not limited to rotation, height normalization of maintaining the aspect ratio, and pixel value normalization processing, etc. For example, for a handwritten text line image with an aspect ratio greater than a preset threshold (which can be set to 4 according to experience), it is considered as a vertical text line image. It needs to be rotated counterclockwise by 90 degrees during image preprocessing.

[0116] In step S402, the image of the text line to be recognized is input as an input of the encoder. The image of the text line to be recognized is processed by a plurality of encoding units in the encoder to obtain a hidden representation. At least one of the plurality of encoding units includes a visual cue layer, which is used to provide local and global context information of the image of the text line to be recognized.

[0117] After obtaining the image of the handwritten text line to be recognized, the execution device inputs the image of the handwritten text line to be recognized into the handwritten character recognition model, so that the handwritten character recognition model performs inference according to the image of the handwritten text line to be recognized to obtain the recognized text.

[0118] An exemplary handwriting recognition model is based on an encoder-decoder architecture. After an input handwritten text line image is input into the handwriting recognition model, the input handwritten text line image is first processed by an encoder. The encoder includes N encoding units. The input handwritten text line image is processed by the N encoding units to obtain a hidden representation. That is, the first encoding unit processes the input handwritten text line image into a first feature map, the Nth encoding unit processes the (N-1)th feature map into an Nth feature map, and the Nth feature map is used to obtain the hidden representation.

[0119] Taking N equal to 4 as an example, that is, the encoder has four encoding units, including encoding unit 1, encoding unit 2, encoding unit 3, and encoding unit 4. The input handwritten text line image is input into the encoding unit 1 to extract features to obtain a first feature map; the first feature map is input into the encoding unit 2 to extract features to obtain a second feature map, and the second feature map is input into the encoding unit 3 to extract features to obtain a third feature map; the third feature map is input into the encoding unit 4 to extract features to obtain a fourth feature map; and the fourth feature map is averaged along the height direction to obtain a hidden representation.

[0120] An exemplary input handwritten text line image is a color RGB three-channel image, and the size is HxWx3, where 3 is the number of image channels, the image height H is set to 64, and the image width W is obtained by scaling according to the original aspect ratio of the image. After the image is sequentially input into the four encoding units, the first to fourth feature maps obtained have dimensions of The fourth feature map is averaged along the height direction to obtain a hidden representation with a dimension of

[0121] In the embodiments of the present application, at least one of the N encoding units in the encoder includes a visual cue layer. The visual cue layer is used to provide local and global context information of the input text line image, so that the handwriting recognition model can utilize the spatial context information of the glyphs, thereby improving the accuracy of text recognition in different scenarios. That is, the visual cue layer can be provided in one or more encoding units in the encoder.

[0122] FIG. 5 shows a structure of a handwriting character recognition model provided by an embodiment of the present application, and a schematic diagram of processing of a handwriting text line image by the handwriting character recognition model. As shown in FIG. 5, a feature extraction module (may also be referred to as a feature extraction layer) and an encoding visual cue learning module (may also be referred to as a visual cue layer) are arranged in an encoding unit 1 in the encoder. The feature extraction module in the encoding unit 1 first performs feature extraction on the input handwriting text line image to obtain an intermediate feature map, and then the visual cue layer processes the intermediate feature map based on visual cues to realize aggregation of local context information and global context information, and obtain an output feature map.

[0123] For example, the feature extraction network in the feature extraction module in the encoding unit 1 to the encoding unit 4 can include 5 layers, the first layer includes 2 convolutional layers, the second to fourth layers include 1 max-pooling layer, a plurality of basic convolutional blocks and 1 convolutional layer, the number of basic convolutional blocks in the second to fourth layers is 1, 2 and 5 respectively, and the fifth layer includes 3 basic convolutional blocks, 1 down-sampling convolutional layer and 1 convolutional layer. The basic convolutional block includes 2 convolutional layers, and the convolutional layer further corresponds to a batch normalization layer and a ReLU activation function. In order to retain more information in the handwriting text line image, the step length of the max-pooling layer in the fourth layer and the down-sampling convolutional layer in the fifth layer in the high and wide directions is (2, 1), that is, the horizontal step length is 1. The feature extraction module in the encoding unit 1 uses the first and second layers, and the feature extraction modules in the encoding unit 2 to the encoding unit 4 use the third, fourth and fifth layers respectively.

[0124] It can be understood that the above feature extraction module is only an example and does not limit the embodiments of the present application, and the feature extraction module can also use other feature extraction network structures to realize feature extraction on the input handwriting text line image.

[0125] FIG. 6 shows a structure of an encoding visual cue learning module and a schematic diagram of a processing process of an intermediate feature map X by the encoding visual cue learning module in an embodiment of the present application. As shown in FIG. 6, the intermediate feature map X obtained by the feature extraction module is input into the visual cue learning module, and the local and global context information is aggregated through the local aggregation layer and the global aggregation layer therein. The dimension size of the intermediate feature map X is h x w x C, where h, w and C are the height, width and feature channel number of the intermediate feature map respectively. The intermediate feature map is divided into a plurality of non-overlapping sub-blocks The dimension size of each sub-block is p 2 x C, where (p, p) is the width and height of the sub-block, for example, p = 4; M = hw / p 2 is the number of sub-blocks.

[0126] Optionally, the sub-blocks of the intermediate feature map can use two-dimensional sine position encoding PE p Superposition coding is performed.

[0127] Exemplarily, in each visual cue learning module, a set of learnable vectors are randomly initialized as meta cues z at the initial moment of the training stage, with a dimension size of JxC, J being the number of meta cues, for example, J=4; C being the feature dimension of the intermediate feature map. The meta cue initialization is performed by assigning a truncated normal distribution with a mean of μ, a variance of σ 2 , and a truncated interval of [-L, L], for example, μ=0, σ=0.02, L=2. The visual cue corresponding to the feature map sub-block is initialized by the meta cue z of the present coding unit. For example, the visual cue corresponding to each feature map sub-block is initialized as the meta cue z, that is, the prompt matrix composed of the J vectors with a dimension of C in the meta cue z.

[0128] The feature map sub-block and the visual cue corresponding to each feature map sub-block are input into the local pooling layer, and each feature map sub-block is updated by a multi-head self-attention (MHSA) mechanism calculation, to obtain the local feature map sub-block x l p

[0129] The visual cue corresponding to each feature map sub-block is updated by a multi-head cross-attention (CA) mechanism calculation with the local feature map sub-block, to obtain the local visual cue , that is, the visual cue after the first update: z l =z p +CA(q=z p ,k=x l ,v=x l )

[0130] The local feature map sub-block and the local visual cue are input into the global pooling layer, and the local visual cues Z l corresponding to different sub-blocks are updated by a multi-head self-attention mechanism calculation, to obtain the global visual cue Z g , that is, the visual cue after the second update: Z g =MHSA(Z l )

[0131] The local feature map sub-block is updated by a multi-head cross-attention mechanism calculation with the corresponding global visual cue , to obtain the global feature map sub-block x g =x l ​​+ CA(q = x l , k = z g , v = z g )

[0132] The updated feature map sub-blocks are spliced to obtain an output feature map X out , whose dimension size is h x w x C.

[0133] In another example, in order to make the visual cue learned by the encoding visual cue learning module focus on the geometric structure information in the image, in the training stage, the learning of the visual cue is guided by maximizing the mutual information of the visual cue and the edge of the handwritten text line image in one or more encoding units. The mutual information of the visual cue and the edge of the handwritten text line image includes local mutual information and global mutual information, and the embodiments of the present application only maximize the mutual information of the visual cue and the edge of the handwritten text line image in the first encoding unit.

[0134] Specifically, first, according to the current encoding unit feature map, a corresponding handwritten text line image and another different handwritten text line image in the same training batch are selected, and edges thereof are extracted respectively by using an edge extraction operator (such as a Canny operator) and are scaled to the same size as the current encoding unit feature map, i.e., the dimension size is h x w x 1; similarly, the scaled edge image is divided into non-overlapping sub-blocks X e , The dimension size is N x p 2 , wherein X e is an edge image sub-block obtained from the corresponding handwritten text line image, X X is an edge image sub-block obtained from the other different handwritten text line image.

[0135] For local mutual information, first, the dimension size of the local visual cue is changed from N x M x C to N x MC; then, two feedforward neural networks (FFN X , FFN Z ) are used to respectively perform feature dimension transformation on X e , and Z l , so that the feature dimensions are the same: X′ e = FFN X (X e ) Z′ l = FFN Z (Z l )

[0136] FFN X and FFN Z each include two linear layers with C output channels, and the transformed dimension size is N x C; finally, X′ e , are concatenated along the feature dimension, and the local mutual information is calculated using a feed-forward neural network FFN l and a softplus activation function according to the Jensen-Shannon mutual information estimation method: D

[0137] where σ(·) is the regular softplus activation function in deep learning, and T(·,·) = FFN Z (Concat(·,·)), where Concat(·,·) denotes the concatenation operation, and the feed-forward neural network FFN Z contains 3 linear layers with 64 neurons in the first two layers and 1 neuron in the third layer.

[0138] For the global mutual information, the edge image sub-block X e , and the global visual cue sub-block Z g are first concatenated according to the corresponding feature map sub-block positions to obtain X` e , and Z` g The dimension sizes are changed from Mxp 2 , Mxp 2 , MxJxC to Then, two convolutional blocks (CNN X , CNN Z ) are used to transform the feature dimensions of X` e , and Z` g respectively to make them have the same feature dimension: X`` e = CNN X (X` e )

[0139] Z`` g = CNN Z (Z` g )

[0140] Both CNN X and CNN Z contain 2 convolutional layers with C output channels, a 3x3 convolution kernel, and a 1x1 step size, and the transformed dimension sizes are both Finally, X` e` , are concatenated along the feature dimension with Z` l ′, and a convolutional neural network CNN D ​and a softplus activation function according to a Jensen-Shannon mutual information estimation method to calculate the global mutual information:

[0141] wherein σ(·) is a softplus activation function, T(·,·) = CNN D (Concat(·,·)), a convolutional neural network CNN D contains 3 convolutional layers with a kernel size of 3x3 and a step size of 1x1, the number of hidden feature channels is 64, the number of output feature channels is 1, and there is also a global average pooling layer between the 2nd and 3rd convolutional layers.

[0142] In step S403, the hidden representation is taken as the input of the decoder, and the recognized text is output.

[0143] The hidden representation output by the encoder is taken as the input of the decoder, and the decoder decodes it to output the recognized text.

[0144] In one example, in order to solve the attention drift problem in the decoding process, the embodiment of the present application introduces the visual representation of the current to-be-decoded character into the decoder to guide the decoding process.

[0145] Continuing to refer to FIG. 5, the decoder of the embodiment of the present application includes a decoding unit, a decoding-guided visual feature generation module (which can also be referred to as a visual representation generation unit) and a character embedding layer, wherein the decoding-guided visual feature generation module is configured to output a visual representation sequence based on the hidden representation; and the character embedding layer is configured to output a character embedding sequence based on the recognized text output before the current decoding time step (which can also be referred to as the recognized text. At the initial decoding time step, the recognized text is initialized as <sos>The output of the recognition unit is a sequence of recognized text embedding representations. The decoding unit is configured to output a character prediction result for the current decoding time step based on the latent representation, the visual representation sequence, and the sequence of recognized text embedding representations. If the character prediction result is <eos>a symbol of end, the decoding process ends. The character prediction results output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the image of the text line to be recognized.

[0146] Exemplarily, the decoding module in the embodiment of the present application adopts a Transformer decoding network based on cross-attention mechanism, contains 2 decoding layers, and the input dimension of each layer is 512, the number of neurons of the full connection layer is 2048, the number of heads in the multi-head attention mechanism is 4, the Dropout probability is set to 0.5, and the activation function uses GELU. The character set predicted by the network contains 11991 characters such as Chinese characters, English letters, symbols and 4 special characters <pad> ”、" <sos> ”、" <eos>"and" <unk> ”，" <pad>"padding symbol for padding the sequence length for data parallel computation," <sos>" represents a predicted sequence start symbol, <eos>" is a prediction sequence end symbol, <unk>" for an unknown character not included in the character set.

[0147] the special character sequence start symbol " <sos>"and the predicted text {a1,...,a1} before the current decoding time step t-1 The input is fed into the character embedding layer, through which the recognized text embedding representation sequence (also known as the recognized text embedding representation sequence) before the current decoding time step is obtained, with a dimension of t×512.

[0148] Figure 7 illustrates the structure of the decoding-guided visual feature generation module in the decoder of this application and a schematic diagram of the processing procedure for the latent representation. As shown in Figure 7, the latent representation is input into the decoding-guided visual feature generation module to obtain a visual representation sequence. For example, the latent representation X... out Dimension size is After passing through a feedforward neural network and a Softmax activation function, the signal is fed into the CTC decoding module to obtain a character probability sequence P, with a size of [missing value]. L represents the total number of characters in the character set; the feedforward neural network consists of three linear layers. The output dimension of the first two linear layers is 512, and they include corresponding layer normalization and GELU activation functions. The output dimension of the last linear layer is L. During the training phase, for the CTC decoding module, the CTC loss function is introduced into the objective function.

[0149] The character position estimate and character confidence are obtained based on the character probability sequence. Specifically, the j-th column P(:,j) of the character probability sequence P represents the probability that all positions in the latent representation are decoded and predicted as the j-th character in the character set. The positions in P(:,j) that belong to local maxima and have a probability greater than a threshold thr (e.g., 0.05) are taken as the possible positions of the j-th character in the character set in the latent representation. Therefore, based on the character probability sequence P, the predicted text {a1,…,a...} before the current decoding time step t is used to estimate the character position and confidence. t-1 The relative positions of the characters can be used to estimate the predicted text {a1,…,a1} up to the current decoding time step. t-1 } and the character to be decoded, a t The position in the implicit representation {p1,…,p t }, and use the corresponding character probability as the character's confidence level {c1,…,c t }

[0150] The hidden representation, the character position estimation, and the character confidence are input to a visual feature sampling module for feature sampling to obtain visual features corresponding to the characters. Specifically, the position estimation of a character is p, and the confidence is c. To obtain the overall features of each character, a window with a center position p and a size of 2s+1 is used to sample the visual representation sequence of the character, then a linear layer is used to aggregate the sampled features, and finally the character confidence c is used to weight the features to obtain the visual features VP corresponding to the character, with a dimension size of 1x512: VP = c Linear(X out [p-s:p+s])

[0151] In one example, a one-dimensional convolution layer with a convolution kernel size of 2s+1 and a step size of 1 is used to perform convolution operation on the hidden representation to implement the above window sampling and linear layer aggregation operation, and the convolution kernel size is 7, which is slightly smaller than the average character width 10 in the dataset; the visual features of the current decoding time step and the characters before the current decoding time step are used as the output visual representation sequence, with a dimension size of t x 512.

[0152] Then, the hidden representation, the visual representation sequence, and the recognized text embedding representation sequence are input as inputs of the decoding unit, and the recognized text of the current decoding time step is output. The recognized texts output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the text line image to be recognized.

[0153] The training of the handwriting character recognition model according to the embodiments of the present application will be described below. The handwriting character recognition model can be implemented in the training device 220 of FIG. 2.

[0154] First, a training data set is obtained from the database 230. The handwriting character recognition model is trained using the training data set in the training phase. The training data set includes a plurality of handwritten text line image samples with labels. The label of each handwritten text line image sample is the text true value corresponding to the handwritten text line image sample. For example, the handwritten text line image sample includes the handwritten text line image "White Day on the Mountain". The label of the handwritten text line image sample is the text sequence "White Day on the Mountain".

[0155] Optionally, the plurality of text line image samples in the training data set need to be preprocessed. For example, the plurality of text line image samples in the training data set are subjected to data enhancement processing, which includes at least one of random rotation processing, local pixel perturbation processing, and Gaussian low-pass filtering processing.

[0156] The pre-processed handwritten text line image sample is taken as an input of the handwritten character recognition model, and a predicted recognition text is output. The parameters of the handwritten character recognition model are optimized and solved based on the predicted recognition text and the text true value. The optimization and solving of the handwritten character recognition model includes minimizing the conventional cross-entropy loss and CTC loss, and maximizing the local mutual information and global mutual information in the encoder based on visual cue learning. The loss function is calculated as follows:

[0157] wherein, is the conventional cross-entropy loss in the character recognition task, is the CTC loss in the decoding guided visual feature generation module, is the local mutual information, is the global mutual information, and β1, β2, β3 are hyperparameters for adjusting the weights of different losses. For example, in the embodiment of the present application, β1 = 1, β2 = 0.1, and β3 = 1 are selected, which are set according to experience. Since the value of is relatively large, β2 is set to 0.1. The value of the hyperparameter can also be obtained by grid search.

[0158] In an example, the training of the handwritten character recognition model of the embodiment of the present application can be implemented on a computer with Nvidia Tesla V graphics processing unit (GPU) parallel computing support using Ubuntu 18.04 operating system and using PyTorch deep learning framework.

[0159] The training process of the handwritten character recognition model of the embodiment of the present application is briefly described below.

[0160] The publicly available Chinese handwritten character data sets SCUT-HCCDoc, SCUT-EPT, and CASIA-HWDB 2.0-2.2 in the art are used for experiments. SCUT-HCCDoc contains 93,254 training images and 23,389 test images. The SCUT-EPT data set mainly comes from test papers, containing 40,000 training images and 10,000 test images. The CASIA-HWDB 2.0-2.2 data set contains 41,781 training images and 10,449 test images. The above images are all handwritten text line images.

[0161] A specific manner of the handwriting character recognition model training provided by the embodiment of the present application is as follows: first, an initial handwriting character recognition model is obtained by using a pre-training method and using training data of three data sets of SCUT-HCCDoc, SCUT-EPT and CASIA-HWDB 2.0-2.2 to pre-train the model, and the number of training rounds is 100; then, the pre-trained initial handwriting character recognition model is loaded and trained on the training sets of SCUT-HCCDoc and SCUT-EPT respectively to obtain a trained handwriting character recognition model, and the number of training rounds is 100. In the training process, the batch processing size Batch Size of GPU parallel computing is set to 32, the initial learning rate is 0.3, the weight decay is 1x10 -6 of the AdaDelta optimizer, and the optimization objectives include minimizing the conventional cross-entropy loss and CTC loss, and maximizing the local mutual information and global mutual information. Other hyperparameters are the conventional settings in deep learning.

[0162] In order to evaluate the beneficial effects of the handwriting character image recognition method provided by the embodiment of the present application, comparative experiments of different methods are performed on the SCUT-HCCDoc and SCUT-EPT data sets.

[0163] The evaluation index is the text line recognition accuracy rate (AR), which is calculated by using the edit distance between the recognized text and the text true value corresponding to the handwriting text line image: AR=(N t -D e -S e -I e ) / N t

[0164] wherein N t represents the total number of characters in the text true value, D e , S e , I e respectively represent the total number of character deletion, replacement and insertion errors for modifying the recognized text into the text true value corresponding to the handwriting text line image.

[0165] Table 1 compares the experimental results of whether to introduce visual cue learning in the encoder and whether to introduce visual feature guidance in the decoder. The baseline model includes a ResNet-29 backbone network, a long short-term memory (LSTM) network, and a Transformer decoding network based on cross attention. The experimental results show that after introducing visual cue learning in the encoder, better recognition performance can be achieved without using the LSTM network, so the LSTM network is not used in subsequent experiments. The experimental results show that introducing visual cue learning in the encoder and introducing visual feature guidance decoding in the decoder can both improve the recognition performance of the model.

[0166] Table 1

[0167] Table 2 compares the experimental results of whether to use local mutual information or / and global mutual information as the optimization target of the model. The experimental results show that using local mutual information or global mutual information can improve the recognition performance of the model, and using local mutual information and global mutual information can achieve the best recognition performance.

[0168] Table 2

[0169] Table 3 compares the recognition results of the embodiments of the present application and other methods on the SCUT-HCCDoc test set. "Using additional training data" refers to the results obtained by pre-training on the training sets of SCUT-HCCDoc, SCUT-EPT, and CASIA-HWDB 2.0-2.2, and then fine-tuning on the training sets of the corresponding data sets.

[0170] Table 3

[0171] Table 4 compares the recognition results of the embodiments of the present application and other methods on the SCUT-EPT test set.

[0172] Table 4

[0173] The experimental results show that the embodiments of the present application have achieved better results than the reported methods on the SCUT-HCCDoc and SCUT-EPT data sets.

[0174] Figure 8 is a schematic diagram of the handwriting recognition results of the embodiments of the present application and the baseline model and the visualization of the attention coefficients of the decoder. As can be seen from Figure 8, the embodiments of the present application can effectively alleviate the attention drift problem.

[0175] FIG. 9 is a schematic diagram of feature map visualization in the baseline model encoder and the embodiment of the present application. As shown in FIG. 9, the input image is a CASIA-HWDB single character sample image. Among them, FIG. 9(a) is the input image; FIG. 9(b) and FIG. 9(c) are the feature maps before and after introducing visual cue learning, respectively; FIG. 9(d) is an example of the attention coefficient of the visual cue and the feature map in the global pooling layer. The visualization results show that the feature map after introducing visual cue learning can better reflect the information of the character edge reflecting the writing style.

[0176] Based on the same idea as the foregoing embodiment of the character image recognition method, the present application also provides a character image recognition device 1000, which can support accurate recognition of handwritten text line images in various scenarios. The character image recognition device 1000 includes units or modules for implementing each step of the character image recognition method shown in FIGS. 4-9.

[0177] FIG. 10 is a structural schematic diagram of a character image recognition device provided by an embodiment of the present application. As shown in FIG. 10, the character image recognition device 1000 includes an acquisition module 1001 and an inference module 1002, wherein the acquisition module 1001 is configured to acquire a text line image to be recognized; and the inference module 1002 is configured to take the text line image to be recognized as an input of an encoder, and obtain a hidden representation by processing the text line image to be recognized through a plurality of encoding units in the encoder, wherein at least one encoding unit of the plurality of encoding units includes a visual cue layer configured to provide local context information and global context information of the text line image to be recognized; and take the hidden representation as an input of a decoder, and output a recognized text.

[0178] In one possible implementation, the at least one encoding unit further includes a feature extraction layer; and the processing of the input feature map by the at least one encoding unit includes: taking the feature map as an input of the feature extraction layer, and outputting an intermediate feature map; inputting the intermediate feature map into the visual cue layer to obtain an output feature map, wherein the output feature map includes the local context information and the global context information of the text line image to be recognized, and the input feature map is the text line image to be recognized or an output feature map output by a previous encoding unit of the at least one encoding unit of the plurality of encoding units.

[0179] In another possible implementation, the visual cue layer includes a meta cue, a local aggregation layer, and a global aggregation layer, the meta cue includes a plurality of cue vectors, the plurality of cue vectors are trained based on a training dataset, the training dataset includes a plurality of text line image samples with labels; a specific implementation of obtaining the output feature map by taking the intermediate feature map as input of the visual cue layer is: performing block processing on the intermediate feature map to obtain M feature map subblocks, M being a positive integer greater than 1; determining M initial visual cues corresponding to the M feature map subblocks based on the meta cue; taking the M feature map subblocks and the M initial visual cues as input of the local aggregation layer to output M local feature map subblocks and M local visual cues; taking the M local feature map subblocks and the M local visual cues as input of the global aggregation layer to obtain the output feature map.

[0180] In another possible implementation, a specific implementation of taking the M feature map subblocks and the M initial visual cues as input of the local aggregation layer to output the M local feature map subblocks and the M local visual cues is: performing multi-head self-attention mechanism calculation on each of the M feature map subblocks respectively to obtain the M local feature map subblocks; performing multi-head cross-attention mechanism calculation on each of the M initial visual cues and each of the M local feature map subblocks respectively to obtain the M local visual cues.

[0181] In another possible implementation, a specific implementation of taking the M local feature map subblocks and the M local visual cues as input of the global aggregation layer to obtain the output feature map is: performing multi-head self-attention mechanism calculation between the M local visual cues to obtain M global visual cues; performing multi-head cross-attention mechanism calculation on each of the M local feature map subblocks and each of the M global visual cues respectively to obtain M global feature map subblocks; and splicing the M feature map subblocks to obtain the output feature map.

[0182] In another possible implementation, the implicit representation is determined based on an output feature map output by a last encoding unit of the plurality of encoding units. For example, the output feature map output by the last encoding unit is averaged along a height direction to obtain the implicit representation.

[0183] In another possible implementation, the weight parameters of the visual cue layer are obtained based on optimization in a training phase aiming to maximize local mutual information and global mutual information, the local mutual information is determined based on M local visual cues, M first edge image sub-blocks and M second edge image sub-blocks, the global mutual information is determined based on the M global visual cues, M first edge image sub-blocks and M second edge image sub-blocks, the M first edge image sub-blocks are obtained by partitioning a first edge image, the M second edge image sub-blocks are obtained by partitioning a second edge image, the first edge image is obtained by performing edge extraction on an input text line image sample, and the second edge image is obtained by performing edge extraction on a different text line image sample from the input text line image sample.

[0184] In another possible implementation, the decoder comprises a decoding unit, a visual representation generation unit and a character embedding layer; taking the latent representation as an input of the decoder, one specific implementation of outputting the recognized text is: taking the latent representation as an input of the visual representation generation unit, outputting a visual representation sequence; taking the recognized text output before the current decoding time step as an input of the character embedding layer, outputting a recognized text embedding representation sequence; taking the latent representation, the visual representation sequence and the recognized text embedding representation sequence as inputs of the decoding unit, outputting the recognized text at the current decoding time step, and the recognized texts output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the text line image to be recognized.

[0185] In another possible implementation, the visual representation generation unit comprises a CTC decoding layer and a visual feature sampling layer; taking the latent representation as an input of the visual representation generation unit, one specific implementation of outputting the visual representation sequence is: taking the latent representation as an input of the CTC decoding layer, outputting a character probability sequence; based on the character probability sequence, obtaining a character position estimate and a character confidence; taking the latent representation, the character position estimate and the character confidence as inputs of the visual feature sampling layer, outputting the visual representation sequence.

[0186] In another possible implementation, the text line image to be recognized comprises one or more handwritten characters.

[0187] In another possible implementation, before inputting the text line image to be recognized into the encoder, further comprising: pre-processing the text line image to be recognized, the pre-processing comprising at least one of height normalization processing and pixel value normalization pre-processing.

[0188] In another possible implementation, in the training phase, the multiple text line image samples in the training data set all need to perform pre-processing, for example, data enhancement processing is performed on the multiple text line image samples in the training data set, and the data enhancement processing comprises at least one of random rotation processing, local pixel disturbance processing and Gaussian low-pass filtering processing.

[0189] The character image recognition apparatus 1000 according to the embodiments of the present application can correspond to performing the methods described in the embodiments of the present application, and the above and other operations and / or functions of each module in the character image recognition apparatus 1000 are respectively for realizing the corresponding flow of each method in FIGS. 4-9, and for brevity, will not be repeated here.

[0190] The embodiments of the present application also provide a computing device, including at least one processor, a memory and a communication interface, the processor being configured to execute the methods described in FIGS. 4-9.

[0191] FIG. 11 is a structural schematic diagram of a computing device according to the embodiments of the present application.

[0192] As shown in FIG. 11, the computing device 1100 includes at least one processor 1101, a memory 1102 and a communication interface 1103. Among them, the processor 1101, the memory 1102 and the communication interface 1103 are communicatively connected, which can be realized by wired (such as bus) or wireless communication. The communication interface 1103 is configured to send and / or receive data sent by other devices; the memory 1102 stores computer instructions, and the processor 1101 executes the computer instructions to execute the method in the foregoing method embodiments, so as to realize accurate recognition of handwritten text line images in various scenarios.

[0193] It should be understood that in the embodiments of the present application, the processor 1101 can be a central processing unit CPU, and the processor 1101 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0194] The memory 1102 can include read-only memory and random access memory, and provide instructions and data for the processor 1101. The memory 1102 can also include non-volatile random access memory. Optionally, the random access memory can be a high bandwidth memory (HBM).

[0195] The memory 1102 can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Where the nonvolatile memory is a read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example, and not limitation, many forms of RAM are available, for example, static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0196] It should be understood that the computing device 1100 according to the embodiments of the present application can perform the method shown in FIGS. 4-9 of the embodiments of the present application, detailed description of which is described above, and for brevity, will not be repeated here

[0197] The embodiments of the present application provide a computer readable storage medium, which stores a computer program, when the computer program is executed by a processor, the above-mentioned method is implemented.

[0198] The embodiments of the present application provide a chip, which includes at least one processor and an interface, the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the above-mentioned method.

[0199] The embodiments of the present application provide a computer program or computer program product, which includes instructions, when the instructions are executed, the computer executes the above-mentioned method.

[0200] Those skilled in the art should further understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0201] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be implemented in hardware, software executed by a processor, or a combination of both. The software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0202] The above detailed description of the specific implementation is further detailed for the purpose of the present application, technical solutions and beneficial effects. It should be understood that the above description is only a specific implementation of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.< / sos> < / unk> < / eos> < / sos> < / pad> < / unk> < / eos> < / sos> < / pad> < / eos> < / sos> < / eos> < / eos> < / sos>

Claims

1. A method of recognizing a character image, characterized by, The method comprises: acquiring a to-be-recognized text line image; inputting the to-be-recognized text line image into an encoder, and obtaining a hidden representation through processing of a plurality of encoding units in the encoder, wherein at least one encoding unit of the plurality of encoding units comprises a visual cue layer, and the visual cue layer is configured to provide local context information and global context information of the to-be-recognized text line image; inputting the hidden representation into a decoder, and outputting recognized text.

2. The method of claim 1, wherein, The at least one encoding unit further comprises a feature extraction layer. The processing of the input feature map by the at least one encoding unit comprises: inputting the feature map into the feature extraction layer to obtain an intermediate feature map; inputting the intermediate feature map into the visual cue layer to obtain an output feature map, wherein the output feature map comprises the local context information and the global context information of the to-be-recognized text line image, and the input feature map is the to-be-recognized text line image or an output feature map output by a previous encoding unit of the at least one encoding unit.

3. The method of claim 2, wherein, The visual cue layer comprises a meta cue, a local aggregation layer and a global aggregation layer, the meta cue comprises a plurality of cue vectors, the plurality of cue vectors are obtained based on a training data set, and the training data set comprises a plurality of text line image samples with labels. The inputting of the intermediate feature map into the visual cue layer to obtain an output feature map comprises: performing block processing on the intermediate feature map to obtain M feature map subblocks, wherein M is a positive integer greater than 1; determining M initial visual cues corresponding to the M feature map subblocks based on the meta cue; inputting the M feature map subblocks and the M initial visual cues into the local aggregation layer to output M local feature map subblocks and M local visual cues; inputting the M local feature map subblocks and the M local visual cues into the global aggregation layer to obtain the output feature map.

4. The method of claim 3, wherein, The inputting of the M feature map subblocks and the M initial visual cues into the local aggregation layer to output M local feature map subblocks and M local visual cues comprises: performing multi-head self-attention mechanism calculation on each of the M feature map subblocks to obtain the M local feature map subblocks; performing multi-head cross-attention mechanism calculation on each of the M initial visual cues and each of the M local feature map subblocks to obtain the M local visual cues.

5. The method according to claim 3 or 4, characterized in that, The inputting of the M local feature map subblocks and the M local visual cues into the global aggregation layer to obtain the output feature map comprises: performing multi-head self-attention mechanism calculation on the M local visual cues to obtain M global visual cues; performing multi-head cross-attention mechanism calculation on each of the M local feature map subblocks and each of the M global visual cues to obtain M global feature map subblocks; splicing the M global feature map subblocks to obtain the output feature map.

6. The method according to any one of claims 2-5, characterized in that, determine the implicit representation based on an output feature map output by a last one of the plurality of encoding units.

7. The method according to claim 5 or 6, characterized in that, The weight parameters of the visual cue layer are obtained based on optimization in a training phase with a goal of maximizing local mutual information and global mutual information, the local mutual information being determined based on the M local visual cues, M first edge image sub-blocks and M second edge image sub-blocks, the global mutual information being determined based on the M global visual cues, the M first edge image sub-blocks and the M second edge image sub-blocks, the M first edge image sub-blocks being obtained by partitioning a first edge image, the M second edge image sub-blocks being obtained by partitioning a second edge image, the first edge image being obtained by edge extraction on an input text line image sample, the second edge image being obtained by edge extraction on a different text line image sample from the input text line image sample.

8. The method according to any one of claims 1 to 7, characterized in that, The decoder comprises a decoding unit, a visual representation generation unit and a character embedding layer. The implicit representation is input into the decoder to output recognized text, including: The implicit representation is input into the visual representation generation unit to output a visual representation sequence. The recognized text output at a previous decoding time step is input into the character embedding layer to output a recognized text embedding representation sequence. The implicit representation, the visual representation sequence and the recognized text embedding representation sequence are input into the decoding unit to output a character prediction result at the current decoding time step, and the character prediction results output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the text line image to be recognized.

9. The method of claim 8, wherein, The visual representation generation unit comprises a CTC decoding layer and a visual feature sampling layer. The implicit representation is input into the visual representation generation unit to output a visual representation sequence, including: The implicit representation is input into the CTC decoding layer to output a character probability sequence. Based on the character probability sequence, a character position estimate and a character confidence are obtained. The implicit representation, the character position estimate and the character confidence are input into the visual feature sampling layer to output the visual representation sequence.

10. The method according to any one of claims 1 to 9, characterized in that, The text line image to be recognized comprises one or more handwritten characters.

11. A text image recognition device, characterized in that, including: An acquisition module is configured to acquire a text line image to be recognized. An inference module is configured to input the text line image to be recognized into an encoder, and the text line image to be recognized is processed by a plurality of encoding units in the encoder to obtain an implicit representation, wherein at least one of the plurality of encoding units comprises a visual cue layer configured to provide local context information and global context information of the text line image to be recognized. and The implicit representation is input into a decoder to output recognized text.

12. A computing device comprising a memory and a processor, wherein: The memory stores instructions which, when executed by the processor, cause the method of any one of claims 1-10 to be implemented.

13. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, causes the method of any one of claims 1-10 to be implemented.

Images