Text recognition method, device and storage medium

Abstract

This application provides a character recognition method, apparatus, and storage medium. The character recognition method provided in this application improves the recognition performance of Chinese characters by extracting radical features and using these features to guide the model to obtain more accurate attention on the radical structure of Chinese characters. Furthermore, this application employs a contrastive learning loss function (CC loss) when training the character recognition model, which can better cluster characters with similar character features, enabling the model to better capture common features between characters and further improving the performance of Chinese character recognition.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and natural language processing (NLP) and neural network technology, specifically to a text recognition method, device and storage medium. Background Technology

[0002] Scene text detection aims to locate text instances in an image. For a given input image, it detects and returns text instances in the image. Text detection has a wide range of applications in the real world, including office automation, visual search, and geolocation, and is therefore of great significance.

[0003] In text recognition tasks, Chinese character recognition is more difficult than English character recognition. Chinese characters are numerous and contain a large number of similar characters that are difficult to distinguish. Furthermore, Chinese characters are exceptionally complex. Existing text recognition methods perform well on clear data (such as printed documents), but their performance on Chinese character recognition is relatively poor.

[0004] Furthermore, in some application scenarios, seal text recognition is an extremely complex but necessary task, for example, Figure 1 The image shown is a document containing a seal image, which includes Chinese characters. Due to the characteristics of seal text, such as curved characters, variable text length, and significant background interference, existing text recognition methods are not entirely satisfactory.

[0005] Therefore, there is an urgent need for a text recognition method that can improve the recognition of Chinese characters in various images, including seal images. Summary of the Invention

[0006] At least one embodiment of this application provides a character recognition method, apparatus, and storage medium to improve Chinese character recognition performance.

[0007] To solve the above-mentioned technical problems, this application is implemented as follows:

[0008] In a first aspect, embodiments of this application provide a character recognition method, including:

[0009] Extract radical features and image features from the target image, wherein the target image contains Chinese characters;

[0010] The image features are input into a multi-head self-attention network to obtain the global features of the target image generated by the multi-head self-attention network; the radical features and global features are fused through a multi-head cross-attention network to obtain the fused features;

[0011] The fused features are decoded by a decoder to obtain the Chinese characters in the target image.

[0012] Optionally, extract radical features and image features from the target image, including:

[0013] The radical features of the target image are extracted using a radical feature extraction network.

[0014] Image features of the target image are extracted using an image feature extraction network.

[0015] Optionally, the radical feature extraction network includes a CRNN encoder and a decoder with an attention mechanism.

[0016] Optionally, the image feature extraction network is a two-layer convolutional neural network, including a convolutional layer and a pooling layer.

[0017] Optionally, the radical features and global features are fused using a multi-head cross-attention network to obtain fused features, including:

[0018] The radical feature, the global feature, and the global feature are respectively used as the Q vector, K vector, and V vector of the multi-head cross-attention network, and input into the multi-head cross-attention network to obtain the output result of the multi-head cross-attention network;

[0019] The output of the multi-head cross-attention network is input into a forward propagation network to obtain the fusion features output by the forward propagation network.

[0020] Optionally, before extracting the radical features and image features of the target image, the method further includes:

[0021] A character recognition model is trained using sample images containing reference Chinese characters. The character recognition model includes the radical feature extraction network, the image feature extraction network, the multi-head self-attention network, the multi-head cross-attention network, and the decoder.

[0022] During training, the loss value of the character recognition model is calculated based on the first loss and the second loss, and the character recognition model is trained with the goal of minimizing the loss value; wherein, the first loss is the cross-entropy loss between the reference Chinese characters in the recognition result of the character recognition model, and the second loss is the character comparison loss between the reference Chinese characters in the recognition result of the character recognition model.

[0023] Optionally, before extracting the radical features and image features of the target image, the method further includes:

[0024] The system receives an image containing Chinese characters uploaded by a user, performs text line detection on the image, and obtains the target image including the text line region.

[0025] Secondly, embodiments of this application provide a character recognition device, including:

[0026] A radical feature extraction network is used to extract radical features from a target image containing Chinese characters.

[0027] An image feature extraction network is used to extract image features from the target image.

[0028] A radical-guided encoder is used to input the image features into a multi-head self-attention network to obtain the global features of the target image generated by the multi-head self-attention network; the radical features and global features are fused through a multi-head cross-attention network to obtain fused features;

[0029] A decoder is used to decode the fused features to obtain the Chinese characters in the target image.

[0030] Optionally, the radical feature extraction network includes a CRNN encoder and a decoder with an attention mechanism.

[0031] Optionally, the image feature extraction network is a two-layer convolutional neural network, including a convolutional layer and a pooling layer.

[0032] Optionally, the radical-guided encoder is further configured to input the radical feature, the global feature, and the global feature as Q vector, K vector, and V vector of the multi-head cross-attention network, respectively, to the multi-head cross-attention network to obtain the output result of the multi-head cross-attention network; and input the output result of the multi-head cross-attention network into a forward propagation network to obtain the fused feature output by the forward propagation network.

[0033] Optionally, the above-mentioned device further includes:

[0034] The training module is used to train a character recognition model using sample images containing reference Chinese characters. The character recognition model includes the radical feature extraction network, the image feature extraction network, the multi-head self-attention network, the multi-head cross-attention network, and the decoder to obtain the character recognition model.

[0035] During training, the loss value of the character recognition model is calculated based on the first loss and the second loss, and the character recognition model is trained with the goal of minimizing the loss value; wherein, the first loss is the cross-entropy loss between the reference Chinese characters in the recognition result of the character recognition model, and the second loss is the character comparison loss between the reference Chinese characters in the recognition result of the character recognition model.

[0036] Optionally, the above-mentioned device further includes:

[0037] The image processing module is used to receive an image to be processed containing Chinese characters uploaded by a user, perform text line detection on the image to be processed, and obtain the target image including the text line region.

[0038] According to a third aspect of this application, at least one embodiment provides a character recognition device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in any of the first aspects.

[0039] According to a fourth aspect of this application, at least one embodiment provides a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the method as described in any of the first aspects.

[0040] According to a fifth aspect of this application, at least one embodiment provides a computer program product including computer instructions that, when executed by a processor, implement the steps of the method as described in any of the first aspects.

[0041] Compared with existing technologies, the character recognition method and apparatus provided in this application improve the recognition performance of Chinese characters by extracting radical features and using these features to guide the model to obtain more accurate attention on the radical structure of Chinese characters. Furthermore, this application employs a contrastive learning loss function (CCloss) when training the character recognition model, which can better cluster characters with the same character features, enabling the model to better capture common features between characters and further improving the performance of Chinese character recognition. Attached Figure Description

[0042] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0043] Figure 1 This is an example image of a document containing a seal image;

[0044] Figure 2 This is a flowchart of a text recognition method according to an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the structure of a radical feature extraction network according to an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the structure of a character recognition device according to an embodiment of this application;

[0047] Figure 5 This is another structural schematic diagram of the character recognition device according to an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of a text recognition system according to an embodiment of the present invention;

[0049] Figure 7 The computer described in this application is an exemplary hardware structure.

[0050] Figure 8 This is an exemplary hardware structure of a mobile terminal according to an embodiment of this application;

[0051] Figure 9 This is an example diagram illustrating the workflow of the text recognition system according to an embodiment of this application. Detailed Implementation

[0052] To make the technical problems, technical solutions, and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as particular configurations and components are provided merely to aid in a comprehensive understanding of the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Furthermore, for clarity and brevity, descriptions of known functions and structures have been omitted.

[0053] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The word "and / or" in the specification and claims indicates at least one of the connected objects.

[0054] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0055] The following description provides examples and is not intended to limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the function and arrangement of the elements discussed without departing from the spirit and scope of this disclosure. Various procedures or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with reference to certain examples may be combined in other examples.

[0056] like Figure 1 As shown, the text recognition method provided in this application includes:

[0057] Step S11: Extract the radical features and image features of the target image, wherein the target image contains Chinese characters.

[0058] Here, the target image can be the image corresponding to the text line region. For example, in this embodiment, a user-uploaded image containing Chinese characters can be received, and text line detection can be performed on the image to be processed to obtain an image (target image) including the text line region. The image to be processed can be, for example,... Figure 1The document image shown can be a picture of the seal portion of the document. Typically, when performing seal text recognition, the circular image of the seal text portion is deformed and adjusted into a rectangular text line image (target image) to facilitate text recognition.

[0059] In S11, this embodiment of the application can extract the radical features of the target image through a radical feature extraction network. Image features of the target image are extracted through an image feature extraction network. The image feature extraction network can be a two-layer convolutional neural network (CNN) used to extract image features of the target image, specifically including convolutional layers and pooling layers, typically composed of stacked convolutional and pooling layers.

[0060] The radical feature extraction network is an encoder-decoder framework that incorporates an attention mechanism to extract radical features of Chinese characters in images. Figure 3 It provides a structure for a radical feature extraction network, which includes a Convolutional Recurrent Neural Network (CRNN) encoder and a decoder with an attention mechanism. Figure 3 In this paper, the CRNN encoder employs a densely connected network (DenseNet) and a bidirectional recurrent neural network (Bi-RNN), wherein the Bi-RNN can be implemented using a bidirectional gated recurrent unit (Bi-GRU); the decoder with attention mechanism is implemented using two unidirectional GRUs.

[0061] Specifically, the target image is input into the DenseNet, and the bidirectional RNN network generates forward hidden layer states and inverse hidden layer states based on the output of the DenseNet. The forward hidden layer states and inverse hidden layer states are then concatenated to obtain the output of the CRNN encoder.

[0062] For example, the output of the DenseNet part at layer l is represented as x. l =H l ([x0,x1,…,x l-1 ]), where [x0,x1,…,x l-1 [] indicates that the feature maps from layer 0 to layer (l-1) are concatenated along the channel dimension, H l (·) represents the nonlinear transformation of the l-th layer.

[0063] In the bidirectional RNN part, during forward propagation, the hidden state... The input x at the current time tt The hidden state at the previous time t-1 Joint decision:

[0064]

[0065] Hidden state during backpropagation The input x at the current time t t The hidden state at the previous time t-1 Joint decision:

[0066]

[0067] Where f(·) is the activation function (such as tanh or ReLU), W f W b These are the weight matrices for the forward and inverse RNNs, respectively, b f b b The corresponding bias terms.

[0068] Finally, the bidirectional RNN synthesizes the hidden state h at each time t. t It is composed of forward and reverse hidden states:

[0069]

[0070] Figure 3 The decoder with attention mechanism in the code uses a single-layer GRU decoding. The goal of the decoder is to generate the radical feature Y corresponding to the Chinese character, represented as:

[0071]

[0072] Where, {y1,…,y C The} symbol represents the characteristics of the radicals after disassembling Chinese characters. Indicates the feature dimension;

[0073] Specifically, two unidirectional GRU layers are used to calculate the hidden state s. t :

[0074]

[0075] Where A is the output of the CRNN encoder, GRU(·) represents a gated recurrent unit, and f catt (·) indicates a spatial attention model based on coverage.

[0076] Additionally, it should be noted that, Figure 3 This is only one specific structure of a radical feature extraction network that can be used in this application embodiment. Other radical feature extraction networks with existing technologies can also be used in this application embodiment.

[0077] Step S12: Input the image features into a multi-head self-attention network to obtain the global features of the target image generated by the multi-head self-attention network; fuse the radical features and global features through a multi-head cross-attention network to obtain fused features.

[0078] Here, through a multi-head self-attention mechanism, the image features are further modeled on the global features. That is, the image features x′ are input into the multi-head self-attention network to obtain the global features x of the target image generated by the multi-head self-attention network. g Then, the radical feature Y and the global feature x are... g ′ and the global feature x g ′ and are used as the Q vector, K vector and V vector of the multi-head cross-attention network, respectively, and are input into the multi-head cross-attention network to obtain the output result of the multi-head cross-attention network; then the output result of the multi-head cross-attention network is input into a forward propagation network to obtain the fusion feature output by the forward propagation network.

[0079] The multi-head cross-attention network can be represented as:

[0080]

[0081] Here, CA(Q, K, V) represents multi-head attention computation, where Q, K, and V represent Query (the information to be retrieved), Key (the vector being queried), and Value (the value obtained from the query), respectively. σ represents the square root of the dimension of the Key vector, and softmax(·) represents the probability distribution transformation function.

[0082] Step S13: Decode the fused features using a decoder to obtain the Chinese characters in the target image.

[0083] Here, in S13, a transformer decoder can be used to decode the fused features to obtain a feature sequence, which is then converted into a Chinese character sequence to obtain the Chinese characters in the target image.

[0084] Through the above steps, this application embodiment considers that Chinese characters are composed of radicals and components. By extracting radical features and using these features to guide the model to obtain more accurate attention on the radical structure of Chinese characters, the recognition performance of Chinese characters is improved. This application embodiment has good Chinese character recognition performance in scenarios with large background interference and character deformation, and is particularly suitable for recognizing Chinese characters in seal text containing seal images.

[0085] Furthermore, prior to step S11 above, this embodiment of the application can pre-train a character recognition model for character recognition using training samples. This character recognition model includes the radical feature extraction network, the image feature extraction network, the multi-head self-attention network, the multi-head cross-attention network, and the decoder. Then, steps S11-S13 above are executed using the character recognition model to recognize the target image and identify the characters contained within it.

[0086] During training, embodiments of this application can calculate the loss value of the character recognition model based on the first loss and the second loss, and train the character recognition model with the goal of minimizing the loss value; wherein, the first loss is the cross-entropy loss (CE loss) between the recognition results of the character recognition model and the reference Chinese characters, and the second loss is the character contrastive loss (CC loss) between the recognition results of the character recognition model and the reference Chinese characters.

[0087] Since Transformer excels at global modeling, it lacks character-level local modeling. CC loss can effectively address this deficiency. By employing the contrastive learning loss function (CC loss), the embodiments of this application can better cluster characters with similar features, enabling the model to better capture common features between characters and further improving Chinese character recognition performance.

[0088] Based on the above methods, this application also provides an apparatus for implementing the above methods. Please refer to [link / reference]. Figure 4 This application provides a character recognition device, including:

[0089] Radical feature extraction network 11 is used to extract radical features of a target image containing Chinese characters;

[0090] Image feature extraction network 12 is used to extract image features of the target image.

[0091] The radical-guided encoder 13 is used to input the image features into a multi-head self-attention network to obtain the global features of the target image generated by the multi-head self-attention network; and to fuse the radical features and the global features through a multi-head cross-attention network to obtain fused features.

[0092] Decoder 14 is used to decode the fused features to obtain the Chinese characters in the target image.

[0093] Through the above modules, the embodiments of this application can improve the recognition performance of Chinese characters.

[0094] Optionally, the radical feature extraction network 11 includes a CRNN encoder and a decoder with an attention mechanism.

[0095] Optionally, the image feature extraction network 12 is a two-layer convolutional neural network, including a convolutional layer and a pooling layer.

[0096] Optionally, the radical guide encoder 13 is further configured to input the radical feature, the global feature, and the global feature as Q vector, K vector, and V vector of the multi-head cross-attention network, respectively, to the multi-head cross-attention network to obtain the output result of the multi-head cross-attention network; and input the output result of the multi-head cross-attention network into a forward propagation network to obtain the fused feature output by the forward propagation network.

[0097] Optionally, the above-mentioned device further includes:

[0098] The training module is used to train a character recognition model using sample images containing reference Chinese characters. The character recognition model includes the radical feature extraction network, the image feature extraction network, the multi-head self-attention network, the multi-head cross-attention network, and the decoder to obtain the character recognition model.

[0099] During training, the loss value of the character recognition model is calculated based on the first loss and the second loss, and the character recognition model is trained with the goal of minimizing the loss value; wherein, the first loss is the cross-entropy loss between the reference Chinese characters in the recognition result of the character recognition model, and the second loss is the character comparison loss between the reference Chinese characters in the recognition result of the character recognition model.

[0100] Optionally, the above-mentioned device further includes:

[0101] The image processing module is used to receive an image to be processed containing Chinese characters uploaded by a user, perform text line detection on the image to be processed, and obtain the target image including the text line region.

[0102] Please refer to Figure 5 This application also provides a hardware structure block diagram of a character recognition device, such as... Figure 5 As shown, the character recognition device 400 includes:

[0103] Processor 402; and

[0104] Memory 404, in which computer program instructions are stored.

[0105] When the computer program instructions are executed by the processor, the processor 402 performs the following steps:

[0106] Extract radical features and image features from the target image, wherein the target image contains Chinese characters;

[0107] The image features are input into a multi-head self-attention network to obtain the global features of the target image generated by the multi-head self-attention network; the radical features and global features are fused through a multi-head cross-attention network to obtain the fused features;

[0108] The fused features are decoded by a decoder to obtain the Chinese characters in the target image.

[0109] It should be noted that the systems provided in the above embodiments are devices corresponding to the above text recognition methods. The implementation methods in the above embodiments are all applicable to the embodiments of this device and can achieve the same technical effect. The device provided in this application can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0110] Furthermore, such as Figure 5 As shown, the character recognition device 400 also includes a network interface 401, an input device 403, a hard disk 405, and a display device 406.

[0111] The various interfaces and devices described above can be interconnected via a bus architecture. The bus architecture can include any number of interconnecting buses and bridges. Specifically, various circuits representing one or more central processing units (CPUs) and / or graphics processing units (GPUs), as represented by processor 402, and one or more memories, as represented by memory 404, are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. It is understood that the bus architecture is used to implement communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are well known in the art and will not be described in detail herein.

[0112] The network interface 401 can be connected to a network (such as the Internet, local area network, etc.), receive data such as images to be identified from the network, and save the received data to the hard disk 405.

[0113] The input device 403 can receive various instructions input by the operator and send them to the processor 402 for execution. The input device 403 may include a keyboard or a clicking device (e.g., a mouse, trackball, touchpad, or touchscreen).

[0114] The display device 406 can display the results obtained by the processor 402 executing instructions, such as displaying the model training progress.

[0115] The memory 404 is used to store programs and data necessary for the operation of the operating system, as well as intermediate results and other data during the calculation process of the processor 402.

[0116] It is understood that the memory 404 in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. The memory 404 of the apparatus and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0117] In some implementations, memory 404 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating system 4041 and application program 4042.

[0118] The operating system 4041 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 4042 includes various applications, such as a browser, used to implement various application functions. Programs implementing the methods of this application embodiment can be included in application program 4042.

[0119] The methods disclosed in the above embodiments of this application can be applied to processor 402, or implemented by processor 402. Processor 402 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 402 or by instructions in the form of software. The processor 402 may be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 404, and processor 402 reads the information in memory 404 and completes the steps of the above method in combination with its hardware.

[0120] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0121] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0122] Specifically, when the computer program is executed by the processor 402, it can also perform the following steps:

[0123] The radical features of the target image are extracted using a radical feature extraction network.

[0124] Image features of the target image are extracted using an image feature extraction network.

[0125] Optionally, the radical feature extraction network includes a CRNN encoder and a decoder with an attention mechanism.

[0126] Optionally, the image feature extraction network is a two-layer convolutional neural network, including a convolutional layer and a pooling layer.

[0127] Specifically, when the computer program is executed by the processor 402, it can also perform the following steps:

[0128] The radical feature, the global feature, and the global feature are respectively used as the Q vector, K vector, and V vector of the multi-head cross-attention network, and input into the multi-head cross-attention network to obtain the output result of the multi-head cross-attention network;

[0129] The output of the multi-head cross-attention network is input into a forward propagation network to obtain the fusion features output by the forward propagation network.

[0130] Specifically, when the computer program is executed by the processor 402, it can also perform the following steps:

[0131] Before extracting the radical features and image features of the target image, a character recognition model is trained using sample images containing reference Chinese characters. The character recognition model includes the radical feature extraction network, the image feature extraction network, the multi-head self-attention network, the multi-head cross-attention network, and the decoder to obtain the character recognition model.

[0132] During training, the loss value of the character recognition model is calculated based on the first loss and the second loss, and the character recognition model is trained with the goal of minimizing the loss value; wherein, the first loss is the cross-entropy loss between the reference Chinese characters in the recognition result of the character recognition model, and the second loss is the character comparison loss between the reference Chinese characters in the recognition result of the character recognition model.

[0133] Specifically, when the computer program is executed by the processor 402, it can also perform the following steps:

[0134] The system receives an image containing Chinese characters uploaded by a user, performs text line detection on the image, and obtains the target image including the text line region.

[0135] It should be noted that the device provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0136] Figure 6 A schematic diagram of the structure of a text recognition system 800 provided in an embodiment of the present invention includes: a client 801 and a text recognition device 802.

[0137] The client 801 can be a personal computer or a mobile terminal, or an application running on any of the aforementioned terminals. A mobile terminal is a user-operated terminal device. A mobile terminal can be a smartphone, a personal digital assistant (PDA) device, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, an in-vehicle device, a wearable device, or a terminal device in a next-generation communication system, such as a terminal device in an NR network or a terminal device in a future evolved Public Land Mobile Network (PLMN) network.

[0138] The text recognition device 802 can be a server system composed of one or more computers. The text recognition device 802 runs a text recognition model 803, an image processing module 805, a training module 804, and a comparison module 805. The text recognition model 803 can be trained by the training module 804.

[0139] In the aforementioned character recognition system 800, the client 801 can connect to the character recognition device 802 via a wired and / or wireless network.

[0140] The functions of the client 801 and the text recognition device 802 can both be distributed across multiple computers.

[0141] The following describes the hardware or software structure of the relevant equipment, devices, or functions.

[0142] <Hardware Structure>

[0143] computer:

[0144] The client 801 and the text recognition device 802 communicate via, for example, Figure 7 The hardware structure shown is then materialized into a computer. Figure 7 An exemplary hardware structure of a computer according to an embodiment of this application is shown.

[0145] refer to Figure 7 The computer 500 includes an input device 501, a display device 502, an external I / F 503, RAM 504, ROM 505, a CPU 506, a communication I / F 507, an HDD 508, etc., interconnected via a bus B. It is acceptable to form a structure in which the input device 501 and the display device 502 are connected when necessary.

[0146] Input device 501 includes a keyboard, mouse, touchpad, etc., through which users input various operation signals. Display device 502 includes a monitor, etc., to display the processing results obtained by computer 500.

[0147] Communication I / F 507 is an interface configured to enable computer 500 to connect to various networks. Therefore, computer 500 performs data communication via communication I / F 507.

[0148] HDD 508 is an exemplary non-volatile storage device for storing programs and data. The stored data includes the operating system (OS) that forms the basis of the software controlling the entire computer 500, application software (also referred to herein as "applications") that provides various functionalities within the OS, and so on. Computer 500 may use a drive device that uses flash memory (e.g., a solid-state drive (SSD)) as the storage medium instead of HDD 508.

[0149] External I / F 503 is an interface for external devices. These external devices include recording media 503a, etc. In this case, computer 500 reads information from and / or writes information to recording media 503a via external I / F 503. Recording media 503a may be a floppy disk, CD, DVD, SD memory card, USB storage device, etc.

[0150] ROM 505 is a non-volatile semiconductor memory (storage device) that retains programs and / or data even when the power is off. ROM 505 stores programs and data used to execute the Basic Input / Output System (BIOS), OS settings, network settings, etc., at the time of power-on of computer 500. RAM 504 is an example of a volatile semiconductor memory (storage device) for temporary storage of programs and / or data.

[0151] CPU 506 is an algorithmic device that reads programs and / or data from storage devices such as ROM 505 and HDD 508. The read program or read data performs a process, thereby materializing the control or functional capabilities of the entire computer 500.

[0152] The client 801 and the text recognition device 802 communicate via, for example Figure 7 The hardware structure of the computer 500 shown is materialized.

[0153] Mobile terminals:

[0154] For example, the client 801 accesses the client via, as follows: Figure 8 The hardware structure shown is then materialized. Figure 8 An exemplary hardware structure of the mobile terminal in this embodiment is shown. Figure 8 The mobile terminal 12 shown includes a CPU 601, ROM 602, RAM 603, EEPROM 604, CMOS sensor 605, acceleration and orientation sensor 606, and media driver 608.

[0155] The CPU 601 controls the entire operation of the mobile terminal 12. The ROM 602 stores basic input and output programs. The RAM 603 is used as the working area of ​​the CPU 601. The EEPROM 604 reads or writes data corresponding to the control of the CPU 601. The CMOS sensor 605 captures image data corresponding to the control of the CPU 601 to obtain image data. The acceleration and orientation sensor 606 is an electromagnetic compass, rotary compass, acceleration sensor, etc., that detects the Earth's magnetic field.

[0156] Media drive 608 controls the reading or writing (storage) of data from or sent to recordable medium 607, such as flash memory. Data already stored in recordable medium 607 is read out, or new data is written to recordable medium 607. Recordable medium 607 is freely attachable to or detachable from media drive 608.

[0157] EEPROM 604 stores the operating system (OS) executed by CPU 601, as well as related information necessary for network settings. Applications for performing various processes of the first embodiment are stored in EEPROM 604, recordable medium 607, etc.

[0158] The CMOS sensor 605 is a charge-coupled device that converts light into electrical charges and digitizes an image of an object. The CMOS sensor 605 can be materialized, for example, by a charge-coupled device (CCD) sensor, as long as it can capture an image of the object.

[0159] In addition, the mobile terminal 12 includes an audio input unit 609, an audio output unit 610, an antenna 611, a communication unit 612, a wireless LAN communication unit 613, a wireless communication antenna 614, a wireless communication unit 615, a display 616, a touchpad 617, and a bus 619.

[0160] Audio input unit 609 converts sound into audio signals. Audio output unit 610 converts audio signals into sound. Communication unit 612 uses antenna 611 to communicate with the nearest base station device via wireless communication signals. Wireless LAN communication unit 613 performs wireless LAN communication with the access point in accordance with the IEEE 80411 standard. Wireless communication unit 615 performs wireless communication using wireless communication antenna 614.

[0161] Display 616 is configured to display images, various icons, etc. Display 616 is made of liquid crystal, organic EL, etc. Touchpad 617 is mounted on display 616 and is formed of a pressure-sensitive plate or an electrostatic plate. Touch positions on display 616 are detected by touch with a finger or stylus. Bus 619 is an address bus, data bus, etc., electrically connecting the aforementioned units or components.

[0162] Client 801 includes a dedicated battery 618. Client 801 is powered by battery 618. Audio input unit 609 includes a microphone for inputting sound. Audio output unit 610 includes a loudspeaker for outputting sound.

[0163] For example, client 801 via, for example, Figure 8 The hardware structure shown is then materialized.

[0164] The following is combined with Figure 9 The present invention provides an example diagram of the workflow of the aforementioned character recognition system 800. In this workflow, the character recognition model trained according to the embodiments of this application is used to recognize named entities, thereby improving the performance of Chinese character recognition. The workflow specifically includes:

[0165] In S81, the user sends an image to be recognized to the character recognition device 802 via the client 801. The character recognition device 802 receives the image.

[0166] In S82, the image processing module 805 in the character recognition device 802 processes the image, generates a target image containing text line regions, and executes the character recognition model. Figure 2 S11-S13 of the embodiment are used to identify Chinese characters in the target image.

[0167] In S83, the character recognition device 802 sends the recognized Chinese characters to the client 801. The client 801 can display the received Chinese characters as the character recognition result through its display device. Specifically, the client 801 can display the entity and its entity type through the interface of its display device.

[0168] In some embodiments of this application, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, performs the following steps:

[0169] Extract radical features and image features from the target image, wherein the target image contains Chinese characters;

[0170] The image features are input into a multi-head self-attention network to obtain the global features of the target image generated by the multi-head self-attention network; the radical features and global features are fused through a multi-head cross-attention network to obtain the fused features;

[0171] The fused features are decoded by a decoder to obtain the Chinese characters in the target image.

[0172] When executed by the processor, this program can implement all the above-mentioned text recognition methods and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0173] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described text recognition method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0174] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0175] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0176] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0178] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0179] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0180] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A character recognition method, characterized in that, include: Extract radical features and image features from the target image, wherein the target image contains Chinese characters; The image features are input into a multi-head self-attention network to obtain the global features of the target image generated by the multi-head self-attention network; the radical features and global features are fused through a multi-head cross-attention network to obtain the fused features; The fused features are decoded by a decoder to obtain the Chinese characters in the target image.

2. The method according to claim 1, characterized in that, Extract radical features and image features from the target image, including: The radical features of the target image are extracted using a radical feature extraction network. Image features of the target image are extracted using an image feature extraction network.

3. The method according to claim 2, characterized in that, The radical feature extraction network includes a CRNN encoder and a decoder with an attention mechanism.

4. The method according to claim 2, characterized in that, The image feature extraction network is a two-layer convolutional neural network, including a convolutional layer and a pooling layer.

5. The method according to claim 2, characterized in that, The radical features and global features are fused using a multi-head cross-attention network to obtain fused features, including: The radical feature, the global feature, and the global feature are respectively used as the Q vector, K vector, and V vector of the multi-head cross-attention network, and input into the multi-head cross-attention network to obtain the output result of the multi-head cross-attention network; The output of the multi-head cross-attention network is input into a forward propagation network to obtain the fusion features output by the forward propagation network.

6. The method according to claim 5, characterized in that, Before extracting the radical features and image features of the target image, the method further includes: A character recognition model is trained using sample images containing reference Chinese characters. The character recognition model includes the radical feature extraction network, the image feature extraction network, the multi-head self-attention network, the multi-head cross-attention network, and the decoder. During training, the loss value of the character recognition model is calculated based on the first loss and the second loss, and the character recognition model is trained with the goal of minimizing the loss value; wherein, the first loss is the cross-entropy loss between the reference Chinese characters in the recognition result of the character recognition model, and the second loss is the character comparison loss between the reference Chinese characters in the recognition result of the character recognition model.

7. The method according to claim 1, characterized in that, Before extracting the radical features and image features of the target image, the method further includes: The system receives an image containing Chinese characters uploaded by a user, performs text line detection on the image, and obtains the target image including the text line region.

8. A character recognition device, characterized in that, include: A radical feature extraction network is used to extract radical features from a target image containing Chinese characters. An image feature extraction network is used to extract image features from the target image. A radical-guided encoder is used to input the image features into a multi-head self-attention network to obtain the global features of the target image generated by the multi-head self-attention network; The radical features and global features are fused using a multi-head cross-attention network to obtain the fused features; A decoder is used to decode the fused features to obtain the Chinese characters in the target image.

9. The apparatus according to claim 8, characterized in that, The radical-guided encoder is further configured to input the radical feature, the global feature, and the global feature as Q vector, K vector, and V vector of a multi-head cross-attention network, respectively, into the multi-head cross-attention network to obtain the output result of the multi-head cross-attention network; and input the output result of the multi-head cross-attention network into a forward propagation network to obtain the fused feature output by the forward propagation network.

10. The apparatus according to claim 9, characterized in that, Also includes: The training module is used to train a character recognition model using sample images containing reference Chinese characters. The character recognition model includes the radical feature extraction network, the image feature extraction network, the multi-head self-attention network, the multi-head cross-attention network, and the decoder to obtain the character recognition model. During training, the loss value of the character recognition model is calculated based on the first loss and the second loss, and the character recognition model is trained with the goal of minimizing the loss value; wherein, the first loss is the cross-entropy loss between the reference Chinese characters in the recognition result of the character recognition model, and the second loss is the character comparison loss between the reference Chinese characters in the recognition result of the character recognition model.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the character recognition method as described in any one of claims 1 to 7.

12. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

Images