A text recognition method and device based on a pulse neural network and a storage medium

Abstract

This invention discloses a text recognition method, apparatus, and storage medium based on a spiking neural network (SNN), relating to the field of text recognition technology. The main objective is to provide a text recognition method and apparatus based on a spiking neural network that solves the attention drift problem in existing technologies. The method includes: acquiring a text image; preprocessing the text image to obtain a pulse sequence; constructing a spiking neural network based on an attention mechanism; and inputting the pulse sequence into the spiking neural network to generate corresponding pulse outputs, thereby obtaining the text recognition result. This invention applies SNN to the field of text recognition, fully utilizing the inherent spatiotemporal feature processing capabilities of SNN, and reducing the cost and energy consumption of hardware deployment for text recognition methods.

Description

Technical Field

[0001] This invention belongs to the field of text recognition technology, specifically relating to a text recognition method, device, and storage medium based on a spiking neural network. Background Technology

[0002] With the rapid development of deep learning, text recognition algorithms have also continuously innovated and improved, achieving substantial progress. Compared to traditional text recognition algorithms, they can handle more complex scenarios and achieve better accuracy. However, deep learning-based methods suffer from problems such as insufficient real-time performance, high hardware deployment costs, and high energy consumption. Summary of the Invention

[0003] To address the aforementioned problems, the present invention aims to provide a text recognition method and apparatus based on a spiking neural network to solve the attention drift problem in existing technologies, employing the following technical solution:

[0004] A text recognition method based on a spiking neural network, characterized in that the method includes:

[0005] Get text image;

[0006] The text image is preprocessed to obtain a pulse sequence;

[0007] Construct a spiking neural network based on an attention mechanism;

[0008] The pulse sequence is input into the spiking neural network to generate a corresponding pulse output, thus obtaining the text recognition result.

[0009] Furthermore, the acquisition of the text image specifically includes:

[0010] Obtain images of various certificates and licenses by scanning or photographing;

[0011] The text image is extracted from the certificate image using a text detection method.

[0012] Furthermore, the preprocessing of the text image to obtain the pulse sequence specifically includes:

[0013] The text image is normalized so that the pixel values ​​are in the range of 0-1;

[0014] The normalized text image is delayed-coded to obtain a text image pulse sequence.

[0015] Furthermore, the text image pulse sequence is represented as a matrix. ,in:

[0016] For time step, The number of channels in the text image. The height of the text image. The width of the text image.

[0017] Furthermore, the construction of the attention-based spiking neural network specifically includes:

[0018] The temporal attention extraction module is used to calculate the attention weights over time.

[0019] A text image feature extraction module is used for the pulse sequence of the denoised text image.

[0020] The spatial attention extraction module is used to calculate attention weights in space;

[0021] The character prediction module is used to output text recognition results.

[0022] Furthermore, the temporal attention extraction module takes the text image pulse sequence as input, calculates the temporal attention weights, and obtains the temporally weighted pulse sequence features. The calculation formula is as follows:

[0023]

[0024]

[0025] in, and This is the weight matrix. express function, Represents the temporal attention weight vector. This represents the characteristics of a pulse sequence after time-weighted processing.

[0026] Furthermore, the text image feature extraction module consists of convolutional layers and fully connected layers of a spiking neural network, taking the time-weighted pulse sequence features as input to obtain the denoised text image pulse sequence. ,in All of these are dimensionality reduction factors.

[0027] Furthermore, the spatial attention extraction module and the character prediction module constitute a recurrent spiking neural network, used to progressively output text sequences of variable length, wherein:

[0028] The spatial attention extraction module takes the denoised text image pulse sequence and the character pulse sequence features output by the character prediction module in the previous text output time step as input, calculates the spatial attention weight of the current text output time step, and obtains the spatially weighted denoised pulse sequence features in the current text output time step as input to the current character prediction module. Then, the current character prediction module outputs the character prediction result and the character pulse sequence features. When the character prediction result is the output end flag [EOS], the text sequence output ends, and the text recognition result is obtained.

[0029] The text output time step is used for the sequential output of the text sequence and belongs to a different dimension than the time step in the spiking neural network.

[0030] Furthermore, the calculation formula for the spatial attention extraction module is as follows:

[0031]

[0032]

[0033] in, Represents a character pulse sequence. Represents the spatial attention matrix. This represents the characteristics of the spatially weighted pulse sequence.

[0034] A text recognition device based on a spiking neural network, characterized in that it comprises:

[0035] The text image acquisition module is used to acquire text images.

[0036] The text image preprocessing module is used to preprocess text images to obtain pulse sequences;

[0037] The temporal attention extraction module is used to calculate the attention weights over time.

[0038] A text image feature extraction module is used for the pulse sequence of the denoised text image.

[0039] The spatial attention extraction module is used to calculate attention weights in space;

[0040] The character prediction module is used to output text recognition results.

[0041] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps of the text recognition method based on a spiking neural network as described above.

[0042] A non-transitory computer-readable storage medium storing a computer program thereon, characterized in that, when executed by a processor, the computer program implements the steps of the text recognition method based on a spiking neural network as described above.

[0043] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0044] This invention applies SNN to text detection, using pulse coding to temporally separate background information from text information in a text image. It then employs a temporal attention mechanism combined with a spiking neural network to filter background information while preserving text information. Simultaneously, a progressively updated spatial attention mechanism mitigates attention drift. Due to the data sparsity of spiking neural networks, this method significantly reduces the computational complexity of the model, lowering hardware deployment costs and energy consumption. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the text recognition method based on a spiking neural network provided by the present invention.

[0046] Figure 2 This is a schematic diagram of the structure of the text recognition device based on a spiking neural network provided by the present invention. Detailed Implementation

[0047] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. Example

[0048] The text recognition method based on spiking neural networks provided by this invention includes the following steps:

[0049] (1) Obtain the text image, specifically:

[0050] Obtain images of various certificates and licenses by scanning or photographing;

[0051] The text image is extracted from the certificate image using a text detection method.

[0052] (2) The text image is preprocessed to obtain a pulse sequence, specifically:

[0053] The text image is normalized so that the pixel values ​​are in the range of 0-1;

[0054] The normalized text image is subjected to delayed coding to obtain a text image pulse sequence. Delayed coding is a common coding method in spiking neural networks, which converts continuous pixel values ​​into discrete sequences containing only 0 and 1.

[0055] The image pulse sequence encodes the pixel values ​​of the text image into a pulse sequence, which serves as a data format that can be processed by a spiking neural network.

[0056] Text image pulse sequences are represented as matrices ,in, For time step, The number of channels in the text image. The height of the text image. The width of the text image.

[0057] (3) Constructing an attention-based spiking neural network, specifically including:

[0058] The temporal attention extraction module is used to calculate the attention weights over time.

[0059] A text image feature extraction module is used for the pulse sequence of the denoised text image.

[0060] The spatial attention extraction module is used to calculate attention weights in space;

[0061] The character prediction module is used to output text recognition results.

[0062] (4) Input the pulse sequence into the spiking neural network to generate the corresponding pulse output and obtain the text recognition result.

[0063] The embodiment of the text recognition device based on spiking neural network provided by the present invention is described as follows:

[0064] (1) Text image acquisition module, used to acquire text images;

[0065] (2) Text image preprocessing module, used to preprocess text images to obtain pulse sequences;

[0066] (3) Temporal attention extraction module, used to calculate temporal attention weights;

[0067] (4) Text image feature extraction module, used for the denoised text image pulse sequence;

[0068] (5) Spatial attention extraction module, used to calculate attention weights in space;

[0069] (6) Character prediction module, used to output text recognition results.

[0070] Example 1

[0071] A text recognition method based on a spiking neural network, characterized in that the method includes:

[0072] Step S1: Obtain the text image;

[0073] Step S1.1: Obtain images of various certificates and licenses by scanning or taking pictures;

[0074] Step S1.2: Extract the text image from the certificate image using a text detection method.

[0075] Step S2: Preprocess the text image to obtain a pulse sequence;

[0076] Step S2.1: Normalize the text image so that the pixel values ​​are in the range of 0-1;

[0077] The normalized text image is delayed-coded to obtain a text image pulse sequence.

[0078] Step S2.2: Construct a spiking neural network based on an attention mechanism;

[0079] The pulse sequence is input into the spiking neural network to generate a corresponding pulse output, thereby obtaining the text recognition result;

[0080] The text image pulse sequence is represented as a matrix. ,in:

[0081] For time step, The number of channels in the text image. The height of the text image. The width of the text image.

[0082] The construction of the attention-based spiking neural network specifically includes:

[0083] The temporal attention extraction module is used to calculate the attention weights over time.

[0084] A text image feature extraction module is used for the pulse sequence of the denoised text image.

[0085] The spatial attention extraction module is used to calculate attention weights in space;

[0086] The character prediction module is used to output text recognition results.

[0087] The temporal attention extraction module takes the text image pulse sequence as input, calculates the temporal attention weights, and obtains the temporally weighted pulse sequence features. The calculation formula is as follows:

[0088]

[0089]

[0090] in, and This is the weight matrix. express function, Represents the temporal attention weight vector. This represents the characteristics of the time-weighted pulse sequence. The meaning is average pooling operation, The meaning is max pooling operation.

[0091] The text image feature extraction module consists of convolutional layers and fully connected layers of a spiking neural network. It takes the time-weighted pulse sequence features as input to obtain the denoised text image pulse sequence. ,in All are dimensionality reduction factors;

[0092] The spatial attention extraction module and the character prediction module constitute a recurrent spiking neural network, used to progressively output text sequences of variable length, wherein:

[0093] The spatial attention extraction module takes the denoised text image pulse sequence and the character pulse sequence features output by the character prediction module in the previous text output time step as input, calculates the spatial attention weight of the current text output time step, and obtains the spatially weighted denoised pulse sequence features in the current text output time step as input to the current character prediction module. Then, the current character prediction module outputs the character prediction result and the character pulse sequence features. When the character prediction result is the output end flag [EOS], the text sequence output ends, and the text recognition result is obtained.

[0094] The text output time step is used for the sequential output of the text sequence and belongs to a different dimension than the time step in the spiking neural network.

[0095] The calculation formula for the spatial attention extraction module is as follows:

[0096]

[0097]

[0098] in This represents the pulse sequence of the (n-1)th character. Represents the spatial attention matrix. This represents the spatially weighted pulse sequence features used to predict the nth character. , , The meaning of convolutional layer.

[0099] Example 2

[0100] A text recognition device based on a spiking neural network includes:

[0101] The text image acquisition module is used to acquire text images.

[0102] The text image preprocessing module is used to preprocess text images to obtain pulse sequences;

[0103] The temporal attention extraction module is used to calculate the attention weights over time.

[0104] A text image feature extraction module is used for the pulse sequence of the denoised text image.

[0105] The spatial attention extraction module is used to calculate attention weights in space;

[0106] The character prediction module is used to output text recognition results.

[0107] Example 3

[0108] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the text recognition method based on a spiking neural network as described in Embodiment 1.

[0109] Example 4

[0110] A non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the text recognition method based on a spiking neural network as described in Example 1.

Claims

1. A method of text recognition based on a spiking neural network, characterized in that, The method includes: Step S1: Obtain the text image; Step S1.1: Obtain images of various certificates and licenses by scanning or taking pictures; Step S1.2: Extract the text image from the certificate image using a text detection method; Step S2: Preprocess the text image to obtain a pulse sequence; Step S2.1: Normalize the text image so that the pixel values ​​are in the range of 0-1; The normalized text image is delayed-coded to obtain a text image pulse sequence; Step S2.2: Construct a spiking neural network based on an attention mechanism; The pulse sequence is input into the spiking neural network to generate a corresponding pulse output, thereby obtaining the text recognition result; The construction of the attention-based spiking neural network specifically includes: The temporal attention extraction module is used to calculate the attention weights over time. A text image feature extraction module is used for the pulse sequence of the denoised text image. The spatial attention extraction module is used to calculate attention weights in space; The character prediction module is used to output text recognition results; The temporal attention extraction module takes the text image pulse sequence as input, calculates the temporal attention weights, and obtains the temporally weighted pulse sequence features. The calculation formula is as follows: in, and This is the weight matrix. express function, Represents the temporal attention weight vector. This represents the characteristics of the time-weighted pulse sequence. The meaning is average pooling operation, The meaning is max pooling operation; The text image feature extraction module consists of convolutional layers and fully connected layers of a spiking neural network. It takes the time-weighted pulse sequence features as input to obtain the denoised text image pulse sequence. ,in All are dimensionality reduction factors; The spatial attention extraction module and the character prediction module constitute a recurrent spiking neural network, used to progressively output text sequences of variable length, wherein: The spatial attention extraction module takes the denoised text image pulse sequence and the character pulse sequence features output by the character prediction module in the previous text output time step as input, calculates the spatial attention weight of the current text output time step, and obtains the spatially weighted denoised pulse sequence features in the current text output time step as input to the current character prediction module. Then, the current character prediction module outputs the character prediction result and the character pulse sequence features. When the character prediction result is the output end flag [EOS], the text sequence output ends, and the text recognition result is obtained. The text output time step is used for the sequential output of the text sequence, and belongs to a different dimension than the time step in the spiking neural network; The calculation formula for the spatial attention extraction module is as follows: in This represents the pulse sequence of the (n-1)th character. Represents the spatial attention matrix. This represents the spatially weighted pulse sequence features used to predict the nth character. , , The meaning of convolutional layer.

2. The pulse-based neural network based text recognition method according to claim 1, characterized in that, The text image pulse sequence list is represented as a matrix wherein: is a time step, is a number of channels of the text image, is a height of the text image, is a width of the text image.

3. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the text recognition method based on a spiking neural network as described in any one of claims 1 to 2.

4. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the text recognition method based on a spiking neural network as described in any one of claims 1 to 2.

Images