Character image recognition method and device

Abstract

The invention provides a character image recognition method and device. The method comprises the steps of obtaining a to-be-recognized text line image; a to-be-recognized text line image is used as input of an encoder, the to-be-recognized text line image is processed by a plurality of coding units in the encoder to obtain implicit representation, at least one coding unit in the coding units comprises a visual prompt layer, and the visual prompt layer comprises a visual prompt layer. The visual prompt layer is used for providing local context information and global context information of the to-be-recognized text line image; and taking the implicit representation as the input of a decoder, and outputting a recognition text. According to the method, the visual prompt layer is added in the coding unit, so that the local context information and the global context information of the to-be-recognized text line image are introduced in the feature extraction process, the spatial context information of fonts can be utilized, and the accuracy of character recognition is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular, to a method and apparatus for recognizing text images. Background Art

[0002] With the popularization of the mobile Internet, the demand for recognizing Chinese handwritten text images taken by mobile phones under open conditions has been increasing. The influence of different lighting, perspectives, and degradation during the imaging process has brought more challenges to the research and development of handwritten text recognition technology. Therefore, handwritten text recognition under uncontrolled conditions remains an unsolved problem. Summary of the Invention

[0003] Embodiments of this application provide a method and apparatus for recognizing text images. By introducing local context information and global context information of the text line image to be recognized during feature extraction, it helps to utilize the spatial context information of the glyphs, thereby improving the accuracy of text recognition in different scenarios (such as uncontrolled scenarios).

[0004] In a first aspect, this application provides a method for recognizing a text image, including obtaining a text line image to be recognized; using the text line image to be recognized as the input of an encoder, and after being processed by a number of encoding units in the encoder, an implicit representation is obtained. Among them, at least one of the number of encoding units includes a visual cue layer, and the visual cue layer is used to provide local context information and global context information of the text line image to be recognized; using the implicit representation as the input of a decoder, and outputting the recognized text.

[0005] It should be noted that the local context information indicates the glyph contour information of the currently recognized text, and the global context information indicates the overall contour information of the text line where the currently recognized text (or the text to be decoded and predicted at the current decoding time step) is located; for example, if the text line image to be recognized is the text line image of "A loyal heart shines on the past phoenix, and my temples have changed day by day", and the currently recognized text is "shines", then the local context information is the glyph contour information of "shines"; the global context information is the overall contour information of the text line "A loyal heart shines on the past phoenix, and my temples have changed day by day" where the currently recognized text "shines" is located.

[0006] This application adds a visual cue layer to the encoding unit, realizing the introduction of local context information and global context information of the text line image to be recognized during feature extraction, which helps to utilize the spatial context information of the glyphs, thereby improving the accuracy of text recognition in different scenarios.

[0007] In one possible implementation, the at least one encoding unit further includes a feature extraction layer; the processing of the input feature map by the at least one encoding unit includes: using the feature map as input to the feature extraction layer to output an intermediate feature map; inputting the intermediate feature map to a visual cue layer to obtain an output feature map, the output feature map including local context information and global context information of the text line image to be recognized, and the input feature map being the text line image to be recognized or the output feature map output by the previous encoding unit of the at least one encoding unit among several encoding units. For example, the encoder includes encoding units 1, 2, 3, and 4 stacked sequentially. When a visual cue layer is added to encoding unit 1, i.e., encoding unit 1 includes both a feature extraction layer and a visual cue layer, the input feature map of encoding unit 1 is the text line image to be recognized. For encoding units that add a visual cue layer to non-encoding unit 1, such as encoding unit 3, the input feature map of encoding unit 3 is the output feature map output by the previous encoding unit, i.e., encoding unit 2.

[0008] In this possible implementation, cue information is injected into the feature map through a visual cue layer. This cue information includes local and global context information of the text line image to be recognized, so that the implicit representation output by the encoder contains spatial context information of the glyphs, which is beneficial for the decoder to decode the text information contained in the text line image.

[0009] In another possible implementation, the visual cue layer includes a meta-cue, a local convergence layer, and a global convergence layer. The meta-cue includes several cue vectors, which are trained based on a training dataset consisting of multiple labeled text line image samples. A specific implementation of using the intermediate feature map as input to the visual cue layer to obtain the output feature map involves: dividing the intermediate feature map into blocks to obtain M feature map sub-blocks, where M is a positive integer greater than 1; determining M initial visual cues corresponding to the M feature map sub-blocks based on the meta-cue; using the M feature map sub-blocks and the M initial visual cues as input to the local convergence layer to output M local feature map sub-blocks and M local visual cues; and using the M local feature map sub-blocks and the M local visual cues as input to the global convergence layer to obtain the output feature map.

[0010] By using the local pooling layer to process the intermediate feature maps, the local contextual information of the text line image to be identified is obtained, which is helpful for recognizing the local structure of a single text image. Furthermore, by using the global pooling layer to process the text line image to be identified, the global contextual information of the text line image to be identified is obtained, which is helpful for obtaining the overall writing style of the text line.

[0011] In another possible implementation, taking M feature map sub-blocks and M initial visual cues as input to the local convergence layer and outputting M local feature map sub-blocks and M local visual cues, a specific implementation is as follows: performing multi-head self-attention mechanism calculation on each feature map sub-block in the M feature map sub-blocks to obtain M local feature map sub-blocks; performing multi-head cross-attention mechanism calculation on each initial visual cue in the M initial visual cue and each local feature map sub-block in the M local feature map sub-blocks to obtain M local visual cues.

[0012] In another possible implementation, using M local feature map sub-blocks and M local visual cues as input to the global convergence layer to obtain the output feature map, a specific implementation is as follows: Multi-head self-attention mechanism is performed among the M local visual cues to obtain M global visual cues; multi-head cross-attention mechanism is performed on each local feature map sub-block and each global visual cue among the M global visual cues to obtain M global feature map sub-blocks; the M global feature map sub-blocks are concatenated to obtain the output feature map.

[0013] In another possible implementation, the latent representation is determined based on the output feature map of the last coding unit among several coding units. For example, the latent representation is obtained by averaging the output feature map of the last coding unit along the height direction.

[0014] In another possible implementation, the weight parameters of the visual cue layer are obtained by optimizing the solution during the training phase with the goal of maximizing local mutual information and global mutual information. The local mutual information is determined based on M local visual cues, M first edge image sub-blocks, and M second edge image sub-blocks. The global mutual information is determined based on the M global visual cues, M first edge image sub-blocks, and M second edge image sub-blocks. The M first edge image sub-blocks are obtained by dividing the first edge image into blocks. The M second edge image sub-blocks are obtained by dividing the second edge image into blocks. The first edge image is obtained by edge extraction from the input text line image samples. The second edge image is obtained by edge extraction from text line image samples that are different from the input text line image samples.

[0015] In this possible implementation, during the training phase, the learning of visual cues is guided by maximizing the mutual information (including local and global mutual information) between the visual cues and the edges of the text line image. This enables the visual cues learned by the visual cue layer to focus on the geometric structure information in the text line image, thereby improving the training effect.

[0016] In another possible implementation, the decoder includes a decoding unit, a visual representation generation unit, and a character embedding layer. A specific implementation that uses the latent representation as input to the decoder and outputs recognized text is as follows: the latent representation is used as input to the visual representation generation unit, and the output is a sequence of visual representations. The recognized text (also referred to as the already recognized text) output before the current decoding time step is initialized to... <sos>The start symbol is used as input to the character embedding layer, and the output is the sequence of recognized text embedding representations. The implicit representation, visual representation sequence, and the sequence of recognized text embedding representations are used as input to the decoding unit, and the output is the character prediction result at the current decoding time step. If the character prediction result is... <eos>The decoding process ends when the end symbol is displayed. The character prediction results output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the image of the text line to be recognized.

[0017] In this possible implementation, the attention drift problem in the encoder-decoder architecture is alleviated by introducing visual features for decoding guidance, thereby further improving text recognition performance.

[0018] In another possible implementation, the visual representation generation unit includes a connectionist temporal classification (CTC) decoding layer and a visual feature sampling layer. A specific implementation that uses latent representations as input to the visual representation generation unit and outputs a visual representation sequence is as follows: The latent features are used as input to the CTC decoding layer, outputting a character probability sequence; based on the character probability sequence, character position estimates and character confidence scores are obtained; the latent representations, character position estimates, and character confidence scores are used as input to the visual feature sampling layer, outputting a visual representation sequence.

[0019] In another possible implementation, the image of the text line to be recognized includes one or more handwritten characters. It should be noted that the handwritten characters mentioned here refer to the image of handwritten characters within the text line to be recognized.

[0020] In another possible implementation, before inputting the image of the text line to be recognized into the encoder, the method further includes: preprocessing the image of the text line to be recognized, which includes at least one of height normalization processing and pixel value normalization preprocessing.

[0021] In another possible implementation, during the training phase, multiple text line image samples in the training dataset need to be preprocessed, such as data augmentation of multiple text line image samples in the training dataset, which includes at least one of random rotation processing, local pixel perturbation processing, and Gaussian low-pass filtering processing.

[0022] Secondly, this application provides a text image recognition device, including an acquisition module and an inference module. The acquisition module is used to acquire a text line image to be recognized; the inference module is used to use the text line image to be recognized as input to an encoder, and the text line image to be recognized is processed by several encoding units in the encoder to obtain a hidden representation. At least one of the several encoding units includes a visual cue layer, which is used to provide local context information and global context information of the text line image to be recognized; and the hidden representation is used as input to a decoder to output recognized text.

[0023] In one possible implementation, the at least one encoding unit further includes a feature extraction layer; the processing of the input feature map by the at least one encoding unit includes: using the feature map as input to the feature extraction layer to output an intermediate feature map; inputting the intermediate feature map into a visual cue layer to obtain an output feature map, the output feature map including local context information and global context information of the text line image to be recognized, and the input feature map being the text line image to be recognized or the output feature map output by the previous encoding unit of the at least one encoding unit among several encoding units.

[0024] In another possible implementation, the visual cue layer includes a meta-cue, a local convergence layer, and a global convergence layer. The meta-cue includes several cue vectors, which are trained based on a training dataset consisting of multiple labeled text line image samples. A specific implementation of using the intermediate feature map as input to the visual cue layer to obtain the output feature map involves: dividing the intermediate feature map into blocks to obtain M feature map sub-blocks, where M is a positive integer greater than 1; determining M initial visual cues corresponding to the M feature map sub-blocks based on the meta-cue; using the M feature map sub-blocks and the M initial visual cues as input to the local convergence layer to output M local feature map sub-blocks and M local visual cues; and using the M local feature map sub-blocks and the M local visual cues as input to the global convergence layer to obtain the output feature map.

[0025] In another possible implementation, taking M feature map sub-blocks and M initial visual cues as input to the local convergence layer and outputting M local feature map sub-blocks and M local visual cues, a specific implementation is as follows: performing multi-head self-attention mechanism calculation on each feature map sub-block in the M feature map sub-blocks to obtain M local feature map sub-blocks; performing multi-head cross-attention mechanism calculation on each initial visual cue in the M initial visual cue and each local feature map sub-block in the M local feature map sub-blocks to obtain M local visual cues.

[0026] In another possible implementation, using M local feature map sub-blocks and M local visual cues as input to the global convergence layer to obtain the output feature map, a specific implementation is as follows: Multi-head self-attention mechanism is performed among the M local visual cues to obtain M global visual cues; multi-head cross-attention mechanism is performed on each local feature map sub-block and each global visual cue among the M global visual cues to obtain M global feature map sub-blocks; the M global feature map sub-blocks are concatenated to obtain the output feature map.

[0027] In another possible implementation, the latent representation is determined based on the output feature map of the last coding unit among several coding units. For example, the latent representation is obtained by averaging the output feature map of the last coding unit along the height direction.

[0028] In another possible implementation, the weight parameters of the visual cue layer are obtained by optimizing the solution during the training phase with the goal of maximizing local mutual information and global mutual information. The local mutual information is determined based on M local visual cues, M first edge image sub-blocks, and M second edge image sub-blocks. The global mutual information is determined based on the M global visual cues, M first edge image sub-blocks, and M second edge image sub-blocks. The M first edge image sub-blocks are obtained by dividing the first edge image into blocks. The M second edge image sub-blocks are obtained by dividing the second edge image into blocks. The first edge image is obtained by edge extraction from the input text line image samples. The second edge image is obtained by edge extraction from text line image samples that are different from the input text line image samples.

[0029] In another possible implementation, the decoder includes a decoding unit, a visual representation generation unit, and a character embedding layer. A specific implementation that uses the latent representation as input to the decoder and outputs the recognized text is as follows: the latent representation is used as input to the visual representation generation unit, outputting a sequence of visual representations; the recognized text output before the current decoding time step is used as input to the character embedding layer, outputting a sequence of recognized text embedding representations; the latent representation, the visual representation sequence, and the sequence of recognized text embedding representations are used as input to the decoding unit, outputting the character prediction result for the current decoding time step. If the character prediction result is... <eos>The decoding process ends when the end symbol is displayed. The character prediction results output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the image of the text line to be recognized.

[0030] In another possible implementation, the visual representation generation unit includes a CTC decoding layer and a visual feature sampling layer. A specific implementation of using the latent representation as input to the visual representation generation unit and outputting a visual representation sequence is as follows: the latent features are used as input to the CTC decoding layer to output a character probability sequence; based on the character probability sequence, character position estimates and character confidence are obtained; the latent representation, character position estimates, and character confidence are used as input to the visual feature sampling layer to output a visual representation sequence.

[0031] In another possible implementation, the image of the text line to be recognized includes one or more handwritten characters.

[0032] In another possible implementation, before inputting the image of the text line to be recognized into the encoder, the method further includes: preprocessing the image of the text line to be recognized, which includes at least one of height normalization processing and pixel value normalization preprocessing.

[0033] In another possible implementation, during the training phase, multiple text line image samples in the training dataset need to be preprocessed, such as data augmentation of multiple text line image samples in the training dataset, which includes at least one of random rotation processing, local pixel perturbation processing, and Gaussian low-pass filtering processing.

[0034] Thirdly, embodiments of this application provide a computing device, including a memory and a processor, wherein the memory stores instructions that, when executed by the processor, cause the text image recognition method described in the first aspect or any possible implementation of the first aspect to be implemented.

[0035] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the text image recognition method described in the first aspect or any possible implementation thereof to be implemented.

[0036] Fifthly, embodiments of this application also provide a computer program or computer program product, which includes instructions that, when executed, cause a computer to perform the text image recognition method described in the first aspect or any possible implementation of the first aspect.

[0037] In a sixth aspect, embodiments of this application also provide a chip, including at least one processor and a communication interface, wherein the processor is used to execute the text image recognition method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0038] Figure 1 This diagram illustrates a basic framework for artificial intelligence.

[0039] Figure 2 A system architecture diagram of the sample processing system provided in this application embodiment;

[0040] Figure 3 A schematic diagram of the hardware structure of a chip provided in an embodiment of this application;

[0041] Figure 4 A flowchart illustrating a text image recognition method provided in an embodiment of this application;

[0042] Figure 5 The diagram illustrates the structure of the handwritten character recognition model provided in this application embodiment, as well as a schematic diagram of the handwritten character recognition model processing handwritten text line images;

[0043] Figure 6 This invention illustrates the structure of an encoding visual cue learning module according to an embodiment of the present application, and the encoding visual cue learning module's processing of intermediate features. Figure X A schematic diagram of the processing procedure;

[0044] Figure 7 The diagram illustrates the structure of the decoding-guided visual feature generation module in the decoder of this application and the processing procedure for implicit representations.

[0045] Figure 8 This is a schematic diagram illustrating the handwritten character recognition results and decoder attention coefficients of the embodiments and baseline models of this application;

[0046] Figure 9 This is a schematic diagram visualizing the feature maps in the encoder of the embodiments of this application and the baseline model;

[0047] Figure 10 A schematic diagram of the structure of a text image recognition device provided in an embodiment of this application;

[0048] Figure 11 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0049] The term "and / or" used in this article describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship, for example, A / B means A or B.

[0050] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same properties in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such processes, methods, systems, products, or apparatus.

[0051] In the embodiments of this application, the words "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0052] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0053] First, the overall workflow of the artificial intelligence system will be described. Please refer to [link / reference]. Figure 1 , Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0054] The above-mentioned artificial intelligence framework is elaborated below from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technical means), to the industrial ecosystem of the system. From the perspective of the IT value chain, the main framework of artificial intelligence includes an infrastructure layer, a data layer, a data processing layer, a general capability layer (achieving general capabilities through data processing results, such as translation, text analysis, computer vision processing, speech recognition, and image recognition), and the topmost application layer, which encapsulates the overall artificial intelligence solution, productizing intelligent information decision-making and realizing its practical application.

[0055] (1) Infrastructure:

[0056] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the external world is achieved through sensors; computing power is provided by intelligent chips, including but not limited to hardware acceleration chips such as central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the external world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0057] (2) Data

[0058] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0059] (3) Data processing

[0060] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0061] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0062] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0063] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0064] (4) General ability

[0065] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0066] (5) Smart Products and Industry Applications

[0067] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0068] The text image recognition method and apparatus provided in this application improve the accuracy of text recognition in different scenarios by setting a visual cue layer in the encoder. This visual cue layer introduces local and global contextual information of the text line image to be recognized during feature extraction, which helps to utilize the spatial contextual information of the glyphs. In other words, this application improves the text recognition model, which is an encoder-decoder architecture. The visual cue layer injects local and global contextual information of the text line image to be recognized into the feature map encoded by the encoder, thereby improving the accuracy of text recognition in different scenarios.

[0069] The text image recognition method and device provided by the embodiments of this application can be applied to various scenarios, such as enterprise document processing and digitization scenarios like contracts, shipping orders, and receipt notes, as well as digitization scenarios in education services and personal information. In the enterprise document processing and digitization application scenarios, the text recognition model provided by the embodiments of this application can quickly convert a large number of handwritten documents, signatures, etc. into editable electronic documents, facilitating electronic archiving, and can also assist experts in improving efficiency and accelerating downstream risk identification. In the education service and personal information digitization application scenarios, for notes and personal information containing handwritten text, the text recognition model provided by the embodiments of this application can accurately extract the handwritten text content therein, assist in document digitization and document content modification, etc., and improve work efficiency.

[0070] The text recognition model provided by the embodiments of this application can be deployed on the server side, websites, and mobile applications to meet the handwritten text recognition needs of enterprise users and individual users; it can also be provided through an open API interface for developers to perform secondary development and use.

[0071] To facilitate understanding of this solution, in the embodiments of this application, first Figure 2 introduce the sample processing system provided by the embodiments of this application, see Appendix Figure 2 , the data acquisition device 260 is used to acquire training data, and the training data of the embodiments of this application includes multiple text line images with labels, and the label is the text truth corresponding to the text line image, that is, the text corresponding to the text image contained in the text line image. For example, if the text line image is an image with the text "The sun along the mountain bows", then the label of this text line image is the text sequence "The sun along the mountain bows". After acquiring the training data, the data acquisition device 260 stores these training data in the database 230, and the training device 220 iteratively optimizes and solves the model parameters based on the training data maintained in the database 230 to obtain the target model / rule 201.

[0072] In actual applications, the training data maintained in the database 230 may not all come from the acquisition of the data acquisition device 260, and it may also be received from other devices. Additionally, it should be noted that the training device 220 does not necessarily train the target model / rule 201 completely based on the training data maintained in the database 230, and it may also obtain training data from the cloud or other places for model training. The above description should not be regarded as a limitation to the embodiments of this application.

[0073] The target model / rule 201 trained according to the training device 220 can be applied to different systems or devices, such as applied to Figure 2 In the execution device 210, the execution device 210 can be a terminal, such as a mobile phone, tablet computer, laptop computer, augmented reality (AR), virtual reality (VR), wearable device, smart robot, vehicle terminal, etc., or it can be a server or cloud, etc.

[0074] Specifically, the training device 220 can deploy the trained AI model to the execution device 210.

[0075] exist Figure 2 In this embodiment, the execution device 210 configures the input / output (I / O) interface 212 for exchanging data with external devices. The user can input data to the I / O interface 212 through the client device 240. In this embodiment, the data may include the image of the text line to be recognized.

[0076] Preprocessing modules 213 and 214 are used to preprocess the input data (such as image data of text lines to be recognized input by the user) received by I / O interface 212. For example, they perform height normalization and pixel normalization preprocessing on the image of the text lines to be recognized, and perform data augmentation processing on the training dataset during the training phase.

[0077] It should be understood that preprocessing modules 213 and 214 may be absent, or only one preprocessing module may be used. When preprocessing modules 213 and 214 are absent, the input data can be processed directly using the calculation module 211.

[0078] The calculation module 211 is used to perform calculations and other related processing on the data input from the preprocessing module 213 and the preprocessing module 214 according to the target model / rule 201 mentioned above.

[0079] During the preprocessing of input data by the execution device 210, or during the calculation and related processing of the calculation module 211 of the execution device 210, the execution device 210 can call the data, code, etc. of the database storage system 250 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0080] Finally, I / O interface 212 returns the processing result (recognized text) to client device 240 for use by the user.

[0081] It is worth noting that the training device 220 can generate a target model / rule 201 corresponding to different downstream systems. This target model / rule 201 can achieve the above-mentioned goals or complete the above-mentioned tasks, thereby providing the user with the required results. It should also be noted that the training device 220 can generate corresponding preprocessing models for the target model / rule 201 corresponding to different downstream systems, such as the preprocessing models corresponding to preprocessing modules 213 and / or 214.

[0082] exist Figure 2 In the scenario shown, the user can manually specify the data input to the execution device 210 (e.g., input an image of the text line to be recognized), for example, by operating through the interface provided by the I / O interface 212. Alternatively, the client device 240 can automatically input data to the I / O interface 212 (e.g., input an image of the text line to be recognized) and obtain the result. If the client device 240 requires user authorization to automatically input data, the user can set the corresponding permissions in the client device 240. The user can view the output results of the execution device 210 (e.g., the output recognized text) on the client device 240, and the specific presentation format can be display, sound, motion, etc. The client device 240 can also act as a data acquisition end, collecting data such as... Figure 2 The input data (image of the text line to be recognized) of the input I / O interface 212 and the output results of the target model / rule 201 are stored in the database 230 as new sample data.

[0083] It is worth noting that Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application, and the positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2 In this context, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed within the execution device 210.

[0084] Figure 3 This is a schematic diagram of the hardware structure of a chip provided in an embodiment of this application. The chip includes a Network Processing Unit (NPU) 30. This chip can be configured as follows: Figure 2 In the execution device 210 shown, the chip is used to perform the calculations of the calculation module 211. This chip can also be located in, for example... Figure 2 The training device 220 shown is used to complete the training work of the training device 220 and output the target model / rule 201.

[0085] The text image recognition method of this application embodiment can be executed in the arithmetic circuit 303 and / or vector calculation unit 307 in the neural network processor 30, thereby obtaining the output text recognition result.

[0086] The following is a brief introduction to the various modules and units in the neural network processor 30.

[0087] The neural network processor 30 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the neural network processor 30 is the arithmetic circuit 303. When the neural network processor 30 is working, the controller 304 in the neural network processor 30 can control the arithmetic circuit 303 to retrieve matrix data from the memory and perform multiplication operations.

[0088] In some implementations, the arithmetic circuit 303 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional pulsating array. The arithmetic circuit 303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0089] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 308.

[0090] The vector computation unit 307 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit 307 can be used for network computation in non-convolutional / non-fully connected layers (FC) of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0091] In some implementations, vector computation unit 307 can store the processed output vector into unified buffer 306. For example, vector computation unit 307 can apply a nonlinear function to the output of arithmetic circuit 303, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 307 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to arithmetic circuit 303, for example, for use in subsequent layers of a neural network.

[0092] The unified memory 306 is used to store input data and output data.

[0093] The weight data is transferred from the external memory to the input memory 301 and / or the unified memory 306 via the Direct Memory Access Controller (DMAC) 305, the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0094] BIU stands for Bus Interface Unit 310, which is used to enable interaction between the main CPU, DMAC and instruction fetch memory 309 via a bus.

[0095] The instruction fetch buffer 309, which is connected to the controller 304, is used to store the instructions used by the controller 304.

[0096] The controller 304 is used to call the instructions cached in the instruction memory 309 to control the operation of the computing accelerator.

[0097] Generally, the unified memory 306, input memory 301, weighted memory 302, and instruction fetch memory 309 can all be on-chip memories. The external memory of the NPU can be memory located outside the NPU itself, such as double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory. It should be understood that... Figure 3 The chip hardware structure shown is merely illustrative and is not intended to limit the scope of this application.

[0098] In recent years, deep learning-based text recognition methods have achieved superior performance compared to traditional methods. Deep learning-based handwritten text line recognition methods can be broadly categorized into two types: character segmentation-based methods and sequence modeling-based methods. Sequence modeling-based methods are the mainstream approach and also fall into two categories: CTC decoding-based methods and encoder-decoder methods based on attention mechanisms. CTC decoding-based methods lack modeling of text context relationships during decoding, often requiring the introduction of an additional language model. Encoder-decoder methods based on attention mechanisms generally employ autoregressive decoding with implicit modeling of text context relationships. In recent years, encoder-decoder methods based on Transformer self-attention mechanisms have demonstrated good performance in handwritten text recognition tasks, but they are prone to attention drift issues when decoding long texts.

[0099] One related technology is a handwritten character recognition method based on character segmentation. This method first segments a line of text into individual characters and then recognizes them. For example, after preprocessing, this solution inputs a 128-pixel high handwritten Chinese text line image into the network. The backbone network generates hierarchical feature maps with strides of {2, 4, 8, 16} pixels relative to the input text line image. Based on the final feature maps of the backbone network, the segmentation and recognition modules, consisting of location, detection, and classification branches, simultaneously output the segmentation and recognition results using character segmentation.

[0100] However, this approach usually requires training for character segmentation. The character-level bounding box positions used to train the character segmentation model are difficult to obtain, and errors in the character segmentation process have a significant impact on subsequent recognition results.

[0101] Related technology two includes handwritten digit recognition schemes based on sequence modeling, such as handwritten digit recognition schemes based on CTC decoding and encoder-decoder schemes. The implementation architecture of encoder-decoder handwritten digit recognition schemes includes a visual encoder and a multimodal decoder. The visual encoder is used to extract 2D visual features, and the multimodal decoder with probabilistic attention masking is used for context-selective decoding.

[0102] CTC-based decoding methods lack modeling of text context dependencies and typically require the introduction of an external language model. Encoder-decoder based methods employ autoregressive decoding that implicitly models text context relationships, but are prone to attention drift issues that can negatively impact recognition performance.

[0103] In practical applications, handwriting styles vary widely and often differ significantly from the statistical distribution of training samples, placing high demands on the generalization ability of models. In the field of natural language processing, cue learning is widely used to adapt pre-trained models to downstream tasks. Cue learning refers to guiding the model to extract the information needed for a specific task from the input data by selecting or generating a set of artificially designed or learnable feature vectors containing task-specific information, thereby improving the model's generalization performance. Currently, researchers have applied cue learning to image classification tasks, using lightweight neural networks to generate input-related feature vectors as visual cues based on the global features of the input image, enabling the model to better adapt to the classification of unknown categories in open conditions. However, current visual cue learning methods only process global information of the image. In handwritten text line recognition tasks, it is necessary not only to identify the local structure of individual characters but also to adapt to the overall handwriting style of the text line. Therefore, it is worth considering designing more flexible and efficient visual cue learning methods specifically for handwritten text recognition tasks.

[0104] In view of this, embodiments of this application provide a method and apparatus for recognizing text images. By adding a visual cue layer to the encoder, local and global contextual information of the text line image to be recognized is introduced during the feature extraction process. This helps to utilize the spatial contextual information of the glyphs, thereby improving the accuracy of text recognition in different scenarios.

[0105] The text image recognition method and apparatus provided in this application can be applied to the effective recognition of various images containing text, including but not limited to the recognition of printed text images, text images in natural scenes, and handwritten text images. In particular, it can be applied to the recognition of handwritten text images, achieving accurate recognition of handwritten text images in various scenarios. Furthermore, the text image recognition method and apparatus provided in this application can be applied to the recognition of any language, including but not limited to Chinese, English, Japanese, Korean, and Arabic.

[0106] This application takes the recognition of handwritten text images as an example to introduce the specific implementation of the text image recognition method and device provided in this application embodiment. The recognition implementation of other types of text images is similar, and the specific implementation can be referred to the specific implementation of the recognition of handwritten text images. For the sake of brevity, this application embodiment will not be described in detail.

[0107] The text image recognition method and apparatus provided in this application are described in detail below with reference to the accompanying drawings.

[0108] Figure 4 This is a flowchart illustrating a text image recognition method provided in an embodiment of this application. The method can be... Figure 2 The execution device 210 implements the feature extraction process by incorporating local and global contextual information of the image of the text line to be recognized. This helps to utilize the spatial contextual information of the glyphs, thereby improving the accuracy of text recognition in different scenarios. For example... Figure 4 As shown, the text image recognition method provided in this application embodiment includes at least steps S401 to S403.

[0109] In step S401, the image of the text line to be recognized is obtained.

[0110] In this embodiment, the execution device is equipped with a handwritten text recognition model, which can effectively recognize handwritten text lines and output the recognized text. Specifically, the handwritten text recognition model can accurately identify the handwritten text content in the handwritten text line image to be recognized, thus assisting in document digitization and content modification. Optionally, the handwritten text recognition model can convert uneditable image-formatted handwritten text into editable text format. When a user needs to recognize a handwritten text line image, they can upload the image to be recognized through a client and then transmit it to the execution device via an I / O interface. The execution device then obtains the handwritten text line image.

[0111] In another example, the image of the text line to be recognized can also be obtained by capturing it with a camera. For instance, the client has a control with "upload" and "capture" options on the image acquisition interface. If the user selects the "capture" option, the camera is invoked to take a picture. By adjusting the camera's viewing angle so that the text is within the shooting angle, clicking to capture the image is obtained, and the image is transmitted to the execution device through the I / O interface.

[0112] The execution device can be a terminal, such as a mobile phone, tablet computer, laptop computer, augmented reality, virtual reality, wearable device, smart robot, vehicle terminal, etc., or it can be a server or cloud.

[0113] It is easy to understand that the text line image mentioned in the embodiments of this application refers to a text image with a certain order. The text line can be vertical or horizontal, and can contain one or more characters. For text line images or multi-line text images in complex backgrounds, a text line detection or text line segmentation module can be connected to the front end for processing to obtain one or more text line images. The embodiments of this application do not make specific limitations in this regard.

[0114] In another example, after acquiring the image of the handwritten text line to be recognized, preprocessing is performed on the image. This preprocessing includes, but is not limited to, rotation, height normalization to maintain the aspect ratio, and pixel value normalization. For instance, handwritten text lines with an aspect ratio greater than a preset threshold (which can be empirically set to 4) are considered vertical text lines and need to be rotated 90 degrees counterclockwise during image preprocessing.

[0115] In step S402, the image of the text line to be recognized is used as the input of the encoder. The image of the text line to be recognized is processed by several coding units in the encoder to obtain the implicit representation. Among the several coding units, at least one coding unit includes a visual cue layer, which is used to provide local context information and global context information of the image of the text line to be recognized.

[0116] After acquiring the image of the handwritten text line to be recognized, the execution device inputs the image of the handwritten text line to be recognized into the handwritten text recognition model, so that the handwritten text recognition model can infer based on the image of the handwritten text line to be recognized and obtain the recognized text.

[0117] For example, the handwritten text recognition model is based on an encoder-decoder architecture. After the image of the handwritten text line to be recognized is input into the handwritten text recognition model (at this point, it can be referred to as the input handwritten text line image), it is first processed by the encoder. The encoder includes N coding units. After the input handwritten text line image is processed by the N coding units, the latent representation is obtained. That is, the first coding unit processes the input handwritten text line image into the first feature map, the Nth coding unit processes the (N-1)th feature map into the Nth feature map, and then the latent representation is obtained based on the Nth feature map.

[0118] Taking N=4 as an example, the encoder has 4 encoding units, including encoding unit 1, encoding unit 2, encoding unit 3, and encoding unit 4. The input handwritten text line image is input to encoding unit 1 for feature extraction to obtain the first feature map; the first feature map is input to encoding unit 2 for feature extraction to obtain the second feature map, and the second feature map is input to encoding unit 3 for feature extraction to obtain the third feature map; the third feature map is input to encoding unit 4 for feature extraction to obtain the fourth feature map; the fourth feature map is averaged along the height direction to obtain the latent representation.

[0119] For example, the input handwritten text line image is a color RGB three-channel image with a size of H×W×3, where 3 is the number of image channels, the image height H is set to 64, and the image width W is obtained by scaling the original aspect ratio of the image. The normalized size W varies depending on the image aspect ratio. After the image passes through four encoding units sequentially, the dimensions of the first to fourth feature maps are respectively... The fourth feature map is averaged along the height direction, resulting in a latent representation dimension of size [value missing].

[0120] In this embodiment, at least one of the N encoding units in the encoder includes a visual cue layer. The visual cue layer provides local and global contextual information of the input text line image, enabling the handwritten character recognition model to utilize the spatial contextual information of the character shapes, thereby improving the accuracy of character recognition in different scenarios. In other words, the encoder may have one or more encoding units with a visual cue layer.

[0121] Figure 5 This illustration shows the structure of the handwritten character recognition model provided in an embodiment of this application, as well as a schematic diagram of the handwritten character recognition model processing handwritten text line images. Figure 5 As shown, the encoder unit 1 is equipped with a feature extraction module (also known as a feature extraction layer) and an encoding visual cue learning module (also known as a visual cue layer). The feature extraction module in the encoder unit 1 first extracts features from the input handwritten text line image to obtain an intermediate feature map. Then, the visual cue layer processes the intermediate feature map based on the visual cue to aggregate local context information and global context information to obtain an output feature map.

[0122] For example, the feature extraction network in the feature extraction modules of encoding units 1 to 4 may include 5 layers. The first layer contains 2 convolutional layers, the second to fourth layers contain 1 max pooling layer, several basic convolutional blocks, and 1 convolutional layer, wherein the number of basic convolutional blocks in the second to fourth layers are 1, 2, and 5, respectively, and the fifth layer contains 3 basic convolutional blocks, 1 downsampling convolutional layer, and 1 convolutional layer. Each basic convolutional block contains 2 convolutional layers, and each convolutional layer also corresponds to a batch normalization layer and a ReLU activation function. In order to retain more information in the handwritten text line image, in this embodiment, the stride of the max pooling layer in the fourth layer and the stride of the downsampling convolutional layer in the fifth layer are both (2,1), that is, the stride in the horizontal direction is 1. The feature extraction module in encoding unit 1 uses the first and second layers, and the feature extraction modules in encoding units 2 to 4 use the third, fourth, and fifth layers, respectively.

[0123] It is understood that the above feature extraction module is only a feasible example and does not constitute a limitation on the embodiments of this application. The feature extraction module can also use other feature extraction network structures to extract features from the input handwritten text line image.

[0124] Figure 6 This invention illustrates the structure of an encoding visual cue learning module according to an embodiment of the present application, and the encoding visual cue learning module's processing of intermediate features. Figure X A schematic diagram of the processing procedure. (For example...) Figure 6 As shown, the intermediate features obtained by the feature extraction module Figure X The input is fed into the visual cue learning module, where local and global contextual information is aggregated through local and global convergence layers, resulting in intermediate features. Figure X The dimensions are h×w×C, where h, w, and C are the height, width, and number of feature channels of the intermediate feature map, respectively. The intermediate feature map is divided into non-overlapping sub-blocks. Each sub-block has a dimension of p. 2 ×C, where (p,p) are the width and height of the sub-block, for example, p = 4; M = hw / p 2 This represents the number of sub-blocks.

[0125] Optionally, two-dimensional sinusoidal positional encoding (PE) from deep learning can be used for intermediate feature map sub-blocks. p Perform overlay encoding.

[0126] For example, in each visual cue learning module, a set of learnable vectors is randomly initialized as meta-cues z at the initial time of the training phase. The meta-cues have a dimension of J×C, where J is the number of meta-cues, for example, J=4; and C is the feature dimension of the intermediate feature map. The meta-cues are initialized with a mean of μ and a variance of σ. 2 Assign values ​​to a cutoff normal distribution with a cutoff interval of [-L, L], for example, μ = 0, σ = 0.02, L = 2. Visual cues corresponding to feature map sub-blocks. The meta-cue z of this coding unit is used for initialization. For example, the visual cue corresponding to each feature map sub-block is initialized as the meta-cue z, which is a cue matrix composed of J vectors of dimension C in the meta-cue z.

[0127] The feature map sub-blocks and their corresponding visual cues are input into the local convergence layer. The local feature map sub-blocks are updated by calculating the multi-head self-attention (MHSA) mechanism.

[0128] x l =MHSA(x p )

[0129] Visual cues corresponding to each feature map sub-block The local feature map sub-blocks are updated using a multi-head cross attention (CA) mechanism to obtain local visual cues. Visual cues after the first update:

[0130] z l =z p +CA(q=z p k=x l v = x l )

[0131] Local feature map sub-blocks and local visual cues are input into the global convergence layer, and the local visual cues Z corresponding to different sub-blocks are... l The global visual cue Z is obtained by computation and updating through a multi-head self-attention mechanism. g The visual cue after the second update:

[0132] Z g =MHSA(Z) l )

[0133] Local feature map sub-block With corresponding global visual cues The global feature map sub-blocks are updated by calculating using a multi-head cross-attention mechanism.

[0134] x g =x l +CA(q=x l k=z g v=z g )

[0135] The updated feature map sub-blocks are concatenated to obtain the output feature. Figure X out Its dimensions are h×w×C.

[0136] In another example, to ensure that the visual cue learning module learns visual cues that focus on geometric structural information in the image, during the training phase, the learning of the visual cues is guided by maximizing the mutual information between the visual cues and the edges of the handwritten text line image in one or more encoding units. The mutual information between the visual cues and the edges of the handwritten text line image includes local mutual information and global mutual information; in this embodiment, the mutual information between the visual cues and the edges of the handwritten text line image is maximized only in the first encoding unit.

[0137] Specifically, firstly, based on the feature map of the current coding unit, select the corresponding handwritten text line image and another different handwritten text line image from the same training batch, and use an edge extraction operator (such as the Canny operator) to extract the edges of each image and scale them to the same size as the feature map of the current coding unit, i.e., with a dimension of h×w×1; similarly, divide the scaled edge image into non-overlapping sub-blocks X. e , The dimension size is N×p 2 , where X e The edge image sub-blocks are obtained from the corresponding handwritten text line images. The edge image sub-block is obtained from the other different handwritten text line image.

[0138] For local mutual information, firstly, local visual cues... The dimensionality is changed from N×M×C to N×MC; then two feedforward neural networks (FFN) are used. X ,FFN Z ) respectively X e , and Z l Perform feature dimension transformation to make the feature dimensions the same:

[0139] X′ e =FFN X (X e )

[0140]

[0141] Z′ l =FFN Z (Z l )

[0142] FFN X and FFN Z Each layer contains two linear layers with C output channels, and the transformed dimension is N×C; finally, X′ e , respectively with Z l The components are concatenated along the feature dimension using a feedforward neural network (FFN). D The local mutual information of the softplus activation function is calculated using the Jensen-Shannon mutual information estimation method:

[0143]

[0144] Where σ(·) is the conventional softplus activation function in deep learning, and T(·,·)=FFN Z (Concat(·,·)), where Concat(·,·) represents the concatenation operation, and the feedforward neural network FFN Z It contains three linear layers. The first two layers have 64 neurons each, and the third layer has 1 neuron.

[0145] For global mutual information, first, the edge image sub-block X e , and global visual cues Z g By concatenating the corresponding feature map sub-blocks according to their positions, X` is obtained. e , and Z` g The dimensions are respectively M×p 2 ,M×p 2 M×J×C becomes Then, two convolutional blocks (CNN) are used. X CNN Z ) respectively X` e , and Z` g Perform feature dimension transformation to make their feature dimensions the same:

[0146] X`` e =CNN X (X` e )

[0147]

[0148] Z`` g =CNN Z (Z` g )

[0149] CNN X and CNN Z Each layer contains two convolutional layers with C output channels, a 3×3 kernel size, and a 1×1 stride. The transformed dimensions are all... Finally, X′ e` , respectively with Z l The data is concatenated along the feature dimension using a convolutional neural network (CNN). D The global mutual information is calculated using the Jensen-Shannon mutual information estimation method with the softplus activation function:

[0150]

[0151] Where σ(·) is the softplus activation function, and T(·,·) = CNN D (Concat(·,·)), Convolutional Neural Network (CNN) D It contains three convolutional layers with a kernel size of 3×3 and a stride of 1×1, with 64 hidden feature channels and 1 output feature channel. There is also a global average pooling layer between the second and third convolutional layers.

[0152] In step S403, the implicit representation is used as the input to the decoder, and the output is the recognized text.

[0153] The implicit representation output by the encoder serves as the input to the decoder, which decodes it and outputs the recognized text.

[0154] In one example, to address the attention drift problem during the decoding process, embodiments of this application introduce a visual representation of the text to be decoded into the decoder to guide the decoding process.

[0155] See also Figure 5 The decoder in this embodiment includes a decoding unit, a decoding-guided visual feature generation module (also referred to as a visual representation generation unit), and a character embedding layer. The decoding-guided visual feature generation module is used to output a visual representation sequence based on implicit representations. The character embedding layer is used to output recognized text (also referred to as recognized text) before the current decoding time step. In the initial decoding time step, the recognized text is initialized to... <sos>The starting symbol) outputs the recognized text embedding representation sequence; the decoding unit is used to output the character prediction result at the current decoding time step based on the implicit representation, visual representation sequence, and recognized text embedding representation sequence. If the character prediction result is... <eos>The decoding process ends when the end symbol is displayed. The character prediction results output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the image of the text line to be recognized.

[0156] For example, the decoding module in this embodiment employs a Transformer decoding network based on a cross-attention mechanism, containing two decoding layers, each with an input dimension of 512. The fully connected layers have 2048 neurons, the multi-head attention mechanism uses 4 heads, the Dropout probability is set to 0.5, and the activation function is GELU. The network predicts a character set containing 11991 characters, including Chinese characters, English letters, symbols, and 4 special characters. <pad> ”、" <sos> ”、" <eos>"and" <unk> ”，" <pad>"Padded symbols to facilitate parallel data computation and sequence length completion" <sos>"Represents the start symbol of the predicted sequence," <eos>"This is the end symbol for the predicted sequence." <unk>" represents an unknown character that is not included in the character set.

[0157] Start the special character sequence with the symbol " <sos>"and the predicted text {a1,...,a1} before the current decoding time step t-1 The input is fed into the character embedding layer, through which the recognized text embedding representation sequence (also known as the recognized text embedding representation sequence) before the current decoding time step is obtained, with a dimension of t×512.

[0158] Figure 7 The diagram illustrates the structure of the decoding-guided visual feature generation module in the decoder of an embodiment of this application, as well as a schematic diagram of the processing procedure for implicit representations. For example... Figure 7 As shown, the latent representation is input into the decoding-guided visual feature generation module to obtain a visual representation sequence. For example, the latent representation X... out Dimension size is After passing through a feedforward neural network and a Softmax activation function, the signal is fed into the CTC decoding module to obtain a character probability sequence P, with a size of [missing value]. L represents the total number of characters in the character set; the feedforward neural network consists of three linear layers. The output dimension of the first two linear layers is 512, and they include corresponding layer normalization and GELU activation functions. The output dimension of the last linear layer is L. During the training phase, for the CTC decoding module, the CTC loss function is introduced into the objective function. .

[0159] The character position estimate and character confidence are obtained based on the character probability sequence. Specifically, the j-th column P(:,j) of the character probability sequence P represents the probability that all positions in the latent representation are decoded and predicted as the j-th character in the character set. The positions in P(:,j) that belong to local maxima and have a probability greater than a threshold thr (e.g., 0.05) are taken as the possible positions of the j-th character in the character set in the latent representation. Therefore, based on the character probability sequence P and the predicted text {a1,…,a...} before the current decoding time step t, the character position estimate and character confidence are obtained. t-1 The relative positions of the characters can be used to estimate the predicted text {a1,…,a1} up to the current decoding time step. t-1 } and the character to be decoded, a t The position in the implicit representation {p1,…,p t }, and use the corresponding character probability as the character's confidence level {c1,…,c t }

[0160] The implicit representation, character position estimation, and character confidence are input into the visual feature sampling module for feature sampling to obtain the visual features corresponding to the characters. Specifically, the position estimation of the character is p, and the confidence is c. To obtain the overall features of each character, a window with a center position of p and a size of 2s + 1 is used to sample the visual representation sequence of the character, then a linear layer is used to aggregate the sampled features, and finally the character confidence c is used to weight the features to obtain the visual feature VP with a dimension size of 1×512:

[0161] VP = c·Linear(X out [p - s:p + s])

[0162] In one example, a one-dimensional convolutional layer with a convolutional kernel size of 2s + 1 and a stride of 1 is used to perform a convolutional operation on the implicit representation to implement the above window sampling and linear layer aggregation operations. The convolutional kernel size is 7, which is slightly smaller than the average character width of 10 in the dataset; the visual features of the current decoding time step and the characters before the current decoding time step are used as the output visual representation sequence with a dimension size of t×512.

[0163] Then, the implicit representation, the visual representation sequence, and the recognized text embedding representation sequence are used as the input of the decoding unit to output the recognized text of the current decoding time step. The recognized texts output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the handwritten text line image to be recognized.

[0164] Next, the training of the handwritten text recognition model in the embodiments of the present application is introduced. The handwritten text recognition model can be implemented in Figure 2 the training device 220.

[0165] First, a training dataset is obtained from the database 230. In the training stage, the handwritten text recognition model is trained using the training dataset. The training dataset includes multiple handwritten text line image samples with labels, and the label of each handwritten text line image sample is its corresponding text ground truth. For example, if the handwritten text line image sample includes the handwritten text line image of "白日依山尽", then the label of this handwritten text line image sample is the text sequence "白日依山尽".

[0166] Optionally, preprocessing needs to be performed on multiple text line image samples in the training dataset. For example, data augmentation processing is performed on multiple text line image samples in the training dataset, and the data augmentation processing includes at least one of random rotation processing, local pixel perturbation processing, and Gaussian low-pass filtering processing.

[0167] The preprocessed handwritten text line image samples are used as input to the handwritten text recognition model, which outputs predicted recognized text. Based on the predicted recognized text and the ground truth text, the parameters of the handwritten text recognition model are optimized. The optimization objectives of the handwritten text recognition model include minimizing the conventional cross-entropy loss and CTC loss, and maximizing the local and global mutual information in the encoder based on visual cues. The specific calculation method of the loss function is as follows:

[0168]

[0169] in, The cross-entropy loss is a common method used in character recognition tasks. To decode the CTC loss in the guided visual feature generation module, For local mutual information, For global mutual information, β1, β2, and β3 are hyperparameters used to adjust the weights of different losses. For example, in this embodiment, β1 = 1, β2 = 0.1, and β3 = 1 are selected based on experience. Since the value of β2 is relatively large, it is set to 0.1. The hyperparameter values ​​can also be obtained through grid search.

[0170] In one example, the training of the handwritten character recognition model in this application embodiment can be implemented using the PyTorch deep learning framework on a computer with parallel computing support from an Nvidia Tesla V graphics processing unit (GPU) and running the Ubuntu 18.04 operating system.

[0171] The training process of the handwritten character recognition model in this application embodiment will be briefly described below.

[0172] Experiments were conducted using three commonly used publicly available Chinese handwritten text datasets: SCUT-HCCDoc, SCUT-EPT, and CASIA-HWDB 2.0-2.2. SCUT-HCCDoc contains 93,254 training images and 23,389 test images. The SCUT-EPT dataset, primarily derived from exam papers, contains 40,000 training images and 10,000 test images. The CASIA-HWDB 2.0-2.2 dataset contains 41,781 training images and 10,449 test images. All images are lines of handwritten text.

[0173] A specific method for training the handwritten character recognition model provided in this application embodiment is as follows: First, a pre-training method is used to pre-train the model using training data from three datasets: SCUT-HCCDoc, SCUT-EPT, and CASIA-HWDB 2.0-2.2, to obtain an initial handwritten character recognition model, with 100 training rounds. Then, the pre-trained initial handwritten character recognition model is loaded and trained on the training sets of SCUT-HCCDoc and SCUT-EPT respectively to obtain a trained handwritten character recognition model, with 100 training rounds. During the training process, the batch size for GPU parallel computing is set to 32, and the initial learning rate is 0.3 with a weight decay of 1×10⁻⁶. -6 The AdaDelta optimizer optimizes for minimizing the regular cross-entropy loss and CTC loss, as well as maximizing local and global mutual information. Other hyperparameters are standard settings for deep learning.

[0174] To evaluate the beneficial effects of the text image recognition method provided in the embodiments of this application, comparative experiments of different methods were conducted on the SCUT-HCCDoc and SCUT-EPT datasets.

[0175] The evaluation metric is the text line recognition accuracy rate (AR), which is calculated using the edit distance between the recognized text and the ground truth values ​​of the corresponding handwritten text line images.

[0176] AR = (N) t -D e -S e -I e ) / N t

[0177] Where N t D represents the total number of characters in the text truth value. e S e I e These represent the total number of character deletion, replacement, and insertion errors that modified the recognized text to the true value of the corresponding text in the handwritten text line image.

[0178] Table 1 compares the experimental results with and without incorporating visual cue learning into the encoder and with and without incorporating visual feature guidance into the decoder. The baseline model includes a ResNet-29 backbone, a long short-term memory (LSTM) network, and a Transformer decoding network based on cross-attention. Experimental results show that incorporating visual cue learning into the encoder achieves better recognition performance without using an LSTM network; therefore, LSTM networks were not used in subsequent experiments. Experimental results demonstrate that both incorporating visual cue learning into the encoder and incorporating visual feature guidance into the decoder improve the model's recognition performance.

[0179]

[0180] Table 1

[0181] Table 2 compares the experimental results of using local mutual information and / or global mutual information as the model optimization objective. The experimental results show that using either local or global mutual information can improve the model's recognition performance, and using both local and global mutual information simultaneously yields the best recognition performance.

[0182] method AR(%) Do not use mutual information (prompt count is 4) 92.58 Use local mutual information (prompt count is 4) 92.65 Use global mutual information (prompt count is 4) 92.61 Use local mutual information and global mutual information (4 prompts). 92.70 Use local and global mutual information (10 prompts allowed). 92.78

[0183] Table 2

[0184] Table 3 compares the recognition results of the embodiments of this application with other methods on the SCUT-HCCDoc test set. "Using additional training data" refers to the results obtained by first pre-training on the training sets of the three datasets SCUT-HCCDoc, SCUT-EPT, and CASIA-HWDB 2.0-2.2, and then fine-tuning on the training sets of the corresponding datasets.

[0185] Model AR(%) CTC-based decoding method 92.08 Attention-based encoder-decoder methods 92.72 Baseline model 92.25 Examples of this application 92.78 This application's embodiment (using additional training data) 93.18

[0186] Table 3

[0187] Table 4 compares the recognition results of the embodiments of this application with those of other methods on the SCUT-EPT test set.

[0188] Model AR(%) CTC-based decoding method 75.97 Attention-based encoder-decoder methods 76.42 Baseline model 78.16 Examples of this application 78.48 This application's embodiment (using additional training data) 80.03

[0189] Table 4

[0190] Experimental results show that the embodiments of this application achieve better results than reported methods on the SCUT-HCCDoc and SCUT-EPT datasets.

[0191] Figure 8 This is a schematic diagram visualizing the handwritten character recognition results and decoder attention coefficients of the embodiments and baseline models of this application. Figure 8 It is evident that the embodiments of this application can effectively alleviate the problem of attention drift.

[0192] Figure 9 This is a schematic diagram visualizing the feature maps in the encoder of the embodiments of this application and the baseline model. For example... Figure 9 As shown, the input image is a CASIA-HWDB single-word sample image. Among them, Figure 9 (a) is the input image; Figure 9 (b) and Figure 9 (c) These are feature maps before and after learning with the introduction of visual cues; Figure 9 (d) shows an example of the attention coefficients between visual cues and feature maps in the global convergence layer. The visualization results show that the feature maps after incorporating visual cues better reflect the information of text edges that reflect handwriting style.

[0193] Based on the same concept as the aforementioned embodiment of a text image recognition method, this application also provides a text image recognition device 1000, which can support accurate recognition of handwritten text lines in various scenarios. The text image recognition device 1000 includes components for implementing... Figure 4-9 The units or modules of each step in the text image recognition method shown.

[0194] Figure 10 This is a schematic diagram of the structure of a text image recognition device provided in an embodiment of this application. Figure 10 As shown, the text image recognition device 1000 includes an acquisition module 1001 and an inference module 1002. The acquisition module 1001 is used to acquire an image of a text line to be recognized. The inference module 1002 is used to take the image of the text line to be recognized as the input of the encoder. The image of the text line to be recognized is processed by several encoding units in the encoder to obtain a hidden representation. At least one of the several encoding units includes a visual cue layer, which is used to provide local context information and global context information of the image of the text line to be recognized. The hidden representation is used as the input of the decoder to output the recognized text.

[0195] In one possible implementation, at least one coding unit further includes a feature extraction layer; the processing of the input feature map by the at least one coding unit includes: using the feature map as input to the feature extraction layer and outputting an intermediate feature map; inputting the intermediate feature map into a visual cue layer to obtain an output feature map, the output feature map including local context information and global context information of the text line image to be recognized, and the input feature map being the text line image to be recognized or the output feature map output by the previous coding unit of the at least one coding unit among several coding units.

[0196] In another possible implementation, the visual cue layer includes a meta-cue, a local convergence layer, and a global convergence layer. The meta-cue includes several cue vectors, which are trained based on a training dataset consisting of multiple labeled text line image samples. A specific implementation of using the intermediate feature map as input to the visual cue layer to obtain the output feature map involves: dividing the intermediate feature map into blocks to obtain M feature map sub-blocks, where M is a positive integer greater than 1; determining M initial visual cues corresponding to the M feature map sub-blocks based on the meta-cue; using the M feature map sub-blocks and the M initial visual cues as input to the local convergence layer to output M local feature map sub-blocks and M local visual cues; and using the M local feature map sub-blocks and the M local visual cues as input to the global convergence layer to obtain the output feature map.

[0197] In another possible implementation, taking M feature map sub-blocks and M initial visual cues as input to the local convergence layer and outputting M local feature map sub-blocks and M local visual cues, a specific implementation is as follows: performing multi-head self-attention mechanism calculation on each feature map sub-block in the M feature map sub-blocks to obtain M local feature map sub-blocks; performing multi-head cross-attention mechanism calculation on each initial visual cue in the M initial visual cue and each local feature map sub-block in the M local feature map sub-blocks to obtain M local visual cues.

[0198] In another possible implementation, using M local feature map sub-blocks and M local visual cues as input to the global convergence layer to obtain the output feature map, a specific implementation is as follows: Multi-head self-attention mechanism is performed among the M local visual cues to obtain M global visual cues; multi-head cross-attention mechanism is performed on each local feature map sub-block and each global visual cue among the M global visual cues to obtain M global feature map sub-blocks; the M feature map sub-blocks are then concatenated to obtain the output feature map.

[0199] In another possible implementation, the latent representation is determined based on the output feature map of the last coding unit among several coding units. For example, the latent representation is obtained by averaging the output feature map of the last coding unit along the height direction.

[0200] In another possible implementation, the weight parameters of the visual cue layer are obtained by optimizing during the training phase with the goal of maximizing local mutual information and global mutual information. The local mutual information is determined based on M local visual cues, M first edge image sub-blocks, and M second edge image sub-blocks. The global mutual information is determined based on the M global visual cues, M first edge image sub-blocks, and M second edge image sub-blocks. The M first edge image sub-blocks are obtained by dividing the first edge image into blocks, and the M second edge image sub-blocks are obtained by dividing the second edge image into blocks. The first edge image is obtained by edge extraction from input text line image samples, and the second edge image is obtained by edge extraction from text line image samples that are different from the input text line image samples.

[0201] In another possible implementation, the decoder includes a decoding unit, a visual representation generation unit, and a character embedding layer. A specific implementation that uses the implicit representation as input to the decoder and outputs recognized text is as follows: the implicit representation is used as input to the visual representation generation unit, outputting a sequence of visual representations; the recognized text output before the current decoding time step is used as input to the character embedding layer, outputting a sequence of recognized text embedding representations; the implicit representation, the visual representation sequence, and the sequence of recognized text embedding representations are used as input to the decoding unit, outputting the recognized text at the current decoding time step. The recognized text output by the decoding unit at all decoding time steps constitutes the recognized text corresponding to the image of the text line to be recognized.

[0202] In another possible implementation, the visual representation generation unit includes a CTC decoding layer and a visual feature sampling layer. A specific implementation of using the latent representation as input to the visual representation generation unit and outputting a visual representation sequence is as follows: the latent features are used as input to the CTC decoding layer to output a character probability sequence; based on the character probability sequence, character position estimates and character confidence are obtained; the latent representation, character position estimates, and character confidence are used as input to the visual feature sampling layer to output a visual representation sequence.

[0203] In another possible implementation, the image of the text line to be recognized includes one or more handwritten characters.

[0204] In another possible implementation, before inputting the image of the text line to be recognized into the encoder, the method further includes: preprocessing the image of the text line to be recognized, which includes at least one of height normalization processing and pixel value normalization preprocessing.

[0205] In another possible implementation, during the training phase, multiple text line image samples in the training dataset need to be preprocessed, such as data augmentation of multiple text line image samples in the training dataset, which includes at least one of random rotation processing, local pixel perturbation processing, and Gaussian low-pass filtering processing.

[0206] The text image recognition device 1000 according to the embodiments of this application can correspondingly execute the method described in the embodiments of this application, and the above and other operations and / or functions of each module in the text image recognition device 1000 are respectively for implementing Figure 4-9 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0207] This application embodiment also provides a computing device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute... Figure 4-9 The method described.

[0208] Figure 11 A schematic diagram of the structure of a computing device provided in an embodiment of this application.

[0209] like Figure 11 As shown, the computing device 1100 includes at least one processor 1101, a memory 1102, and a communication interface 1103. The processor 1101, memory 1102, and communication interface 1103 are communicatively connected, which can be achieved via a wired (e.g., bus) or wireless connection. The communication interface 1103 is used to send and / or receive data from other devices. The memory 1102 stores computer instructions, which the processor 1101 executes to perform the methods described in the foregoing method embodiments, thereby achieving accurate recognition of handwritten text lines in various scenarios.

[0210] It should be understood that, in the embodiments of this application, the processor 1101 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0211] The memory 1102 may include read-only memory and random access memory, and provides instructions and data to the processor 1101. The memory 1102 may also include non-volatile random access memory. Optionally, the random access memory may be, for example, high-bandwidth memory (HBM).

[0212] The memory 1102 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0213] It should be understood that the computing device 1100 according to the embodiments of this application can perform the implementation of the embodiments of this application. Figure 4-9 The method shown is described in detail above, and will not be repeated here for the sake of brevity.

[0214] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer instructions are executed by a processor, the aforementioned method is implemented.

[0215] An embodiment of this application provides a chip including at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0216] Embodiments of this application provide a computer program or computer program product that includes instructions that, when executed, cause a computer to perform the methods mentioned above.

[0217] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0218] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented using hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0219] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.< / sos> < / unk> < / eos> < / sos> < / pad> < / unk> < / eos> < / sos> < / pad> < / eos> < / sos> < / eos> < / eos> < / sos>

Claims

1. A method for recognizing text images, characterized in that, include: Obtain the image of the text line to be recognized; The image of the text line to be recognized is used as the input of the encoder. The image of the text line to be recognized is processed by several coding units in the encoder to obtain a hidden representation. At least one of the several coding units includes a visual cue layer, which is used to provide local context information and global context information of the image of the text line to be recognized. The implicit representation is used as input to the decoder to output the recognized text.

2. The method according to claim 1, characterized in that, The at least one encoding unit further includes a feature extraction layer; The processing of the input feature map by the at least one encoding unit includes: The feature map is used as input to the feature extraction layer, and an intermediate feature map is output. The intermediate feature map is input into the visual cue layer to obtain an output feature map. The output feature map includes local context information and global context information of the text line image to be identified. The input feature map is the text line image to be identified or the output feature map output by the previous coding unit of at least one coding unit among the plurality of coding units.

3. The method according to claim 2, characterized in that, The visual cue layer includes a meta-cue, a local convergence layer, and a global convergence layer. The meta-cue includes several cue vectors, which are trained based on a training dataset, which includes multiple labeled text line image samples. The step of using the intermediate feature map as input to the visual cue layer to obtain the output feature map includes: The intermediate feature map is divided into blocks to obtain M feature map sub-blocks, where M is a positive integer greater than 1; Based on the meta-cues, determine the M initial visual cues corresponding to the M feature map sub-blocks; The M feature map sub-blocks and M initial visual cues are used as input to the local convergence layer, and the output is M local feature map sub-blocks and M local visual cues; The M local feature map sub-blocks and the M local visual cues are used as inputs to the global convergence layer to obtain the output feature map.

4. The method according to claim 3, characterized in that, The step of using the M feature map sub-blocks and M initial visual cues as input to the local convergence layer and outputting M local feature map sub-blocks and M local visual cues includes: Multi-head self-attention mechanism is used to calculate the M local feature map sub-blocks respectively; The M local visual cues are obtained by performing multi-head cross-attention mechanism calculations on each of the M initial visual cues and each of the M local feature map sub-blocks.

5. The method according to claim 3 or 4, characterized in that, The step of using the M local feature map sub-blocks and the M local visual cues as input to the global convergence layer to obtain the output feature map includes: A multi-head self-attention mechanism is used to calculate the M global visual cues among the M local visual cues; A multi-head cross-attention mechanism is used to calculate M global feature map sub-blocks for each of the M local feature map sub-blocks and for each of the M global visual cues; The M global feature map sub-blocks are concatenated to obtain the output feature map.

6. The method according to any one of claims 2-5, characterized in that, The implicit representation is determined based on the output feature map of the last coding unit among the plurality of coding units.

7. The method according to claim 5 or 6, characterized in that, The weight parameters of the visual cue layer are obtained by optimizing the solution during the training phase with the goal of maximizing local mutual information and global mutual information. The local mutual information is determined based on the M local visual cues, M first edge image sub-blocks, and M second edge image sub-blocks. The global mutual information is determined based on the M global visual cues, the M first edge image sub-blocks, and the M second edge image sub-blocks. The M first edge image sub-blocks are obtained by dividing the first edge image into blocks. The M second edge image sub-blocks are obtained by dividing the second edge image into blocks. The first edge image is obtained by edge extraction from the input text line image samples. The second edge image is obtained by edge extraction from text line image samples that are different from the input text line image samples.

8. The method according to any one of claims 1-7, characterized in that, The decoder includes a decoding unit, a visual representation generation unit, and a character embedding layer; The step of using the implicit representation as input to the decoder and outputting recognized text includes: The implicit representation is used as the input to the visual representation generation unit, and a visual representation sequence is output. The recognized text output before the current decoding time step is used as the input to the character embedding layer, and the recognized text embedding representation sequence is output. The implicit representation, the visual representation sequence, and the recognized text embedding representation sequence are used as inputs to the decoding unit, which outputs the character prediction result at the current decoding time step. The character prediction results output by the decoding unit at all decoding time steps constitute the recognized text corresponding to the image of the text line to be recognized.

9. The method according to claim 8, characterized in that, The visual representation generation unit includes a CTC decoding layer and a visual feature sampling layer; The step of using the implicit representation as input to the visual representation generation unit and outputting a visual representation sequence includes: The implicit features are used as input to the CTC decoding layer, and a character probability sequence is output. Based on the character probability sequence, the character position estimate and character confidence level are obtained; The implicit representation, the character position estimation, and the character confidence are used as inputs to the visual feature sampling layer, and the visual representation sequence is output.

10. The method according to any one of claims 1-9, characterized in that, The image of the text line to be recognized includes one or more handwritten characters.

11. A text image recognition device, characterized in that, include: The acquisition module is used to acquire images of the text lines to be recognized; The inference module is used to take the text line image to be recognized as the input of the encoder. The text line image to be recognized is processed by several coding units in the encoder to obtain a hidden representation. At least one of the several coding units includes a visual cue layer. The visual cue layer is used to provide local context information and global context information of the text line image to be recognized. as well as The implicit representation is used as input to the decoder to output the recognized text.

12. A computing device, comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-10 to be implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the method as described in any one of claims 1-10 to be implemented.

Images