A mathematical expression image recognition system

Abstract

The mathematical expression image recognition system provided by the present invention includes a feature encoder, a conditional network module, and a feature unified fusion decoder. The feature encoder includes multiple lightweight deep separable convolutions, and the conditional network module is a lightweight conditional network used to enhance global position information. The output of the feature encoder is fed into the feature unified fusion decoder via the conditional network module. The feature unified fusion decoder uses an attention mechanism. This design improves the generalization of the mathematical expression image recognition system, avoiding complex calculations and increasing speed while ensuring accurate recognition even when encountering long sequences of formulas.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a mathematical expression image recognition system. Background Art

[0002] Mathematics is a universal natural science language in the financial industry, and mathematical expressions, as important mathematical carriers, are widely used in various financial services-related documents. Intelligent and automated processing and management of financial mathematical expressions is a key task in FinTech. However, mathematical expressions are often presented in the form of images, which cannot be analyzed and processed by digital computers. Therefore, it is necessary to develop recognition technology to convert mathematical expressions from images into editable and processable text representations, thereby improving the processing efficiency of financial practitioners.

[0003] In recent years, the development of deep learning technology has enhanced the ability of machines and algorithms to handle complex tasks. In particular, convolutional neural networks (CNNs) have been widely used in image recognition. They maintain a certain degree of invariance to translation, scaling, rotation, and other types of image manipulation, and eliminate the need for traditional manual feature extraction and data reconstruction before image recognition. Expression images are input as pixels, and expression features are extracted through a CNN network, enabling simple and efficient expression recognition.

[0004] However, despite the significant progress of traditional optical character recognition (OCR), mathematical expression recognition, as a subcategory of OCR, still faces considerable challenges. Mathematical expressions have complex nested structures, are sensitive to spatial position information, and have small, difficult-to-read characters. Furthermore, compared to traditional OCR tasks, mathematical expressions are unique in that the formula sequences they recognize are often longer, making it even more important to capture long-term dependencies. Summary of the Invention

[0005] Based on this, it is necessary to propose an efficient mathematical expression image recognition system that can ensure accurate recognition when encountering long sequence formulas.

[0006] The present invention provides a mathematical expression image recognition system, including a feature encoder, a conditional network module and a feature unified fusion decoder. The feature encoder includes multiple lightweight deep separable convolutions. The conditional network module is a lightweight conditional network for enhancing global position information. The output of the feature encoder is sent to the feature unified fusion decoder via the conditional network module. The feature unified fusion decoder adopts an attention mechanism.

[0007] Preferably, the feature encoder divides an input formula grayscale image into several non-overlapping image blocks through a partitioning convolution layer, and the dimension of the feature map after the partitioning convolution layer becomes Where w, h are the width and height of the image respectively, c is the dimension of the hidden layer, and the convolutional layer is divided into:

[0008] z0=BN(σ{Conv(x, stride=4, kernel size=4)})

[0009] The feature map is fed into multiple deep separable convolutions, where each layer of convolution is followed by a post-activation function and a normalization layer. Each layer of deep separable convolution is expressed as:

[0010] z {l} =BN(σ{DepthwiseConv(z {l-1} )})+z {l-1}

[0011] z {l+1} =BN(σ{PointwiseConv(z {l} )})

[0012] Get the output feature map of the feature encoder.

[0013] Preferably, the conditional network module is composed of a first linear network layer and a second linear network layer. The first linear network layer performs dimensionality compression, and the second linear network layer performs dimensionality recovery. The conditional network can be expressed as:

[0014] τ=h {θ} (V)

[0015] The parameter updates of the conditional network are achieved through back-propagation.

[0016] Preferably, the feature unified fusion decoder generates a formula symbol for each timestamp based on the output of the feature encoder and the output of the conditional network module. A start symbol and an end symbol are added to each mathematical expression sequence, and the length is aligned with the longest formula sequence in this batch. Each character is mapped to a feature word vector, i.e., t= <t1,t2,…,t T >, where t i ∈R 256 ,T is the length of the mathematical expression.

[0017] Preferably, the feature unified fusion decoder includes six attention layers, each of which includes a self-attention part and a cross-attention part, and the feature unified fusion decoder is used as a language model:

[0018] P(y t |y1,…,yt-1 ,V)=Softmax(W out O t-1 )

[0019] The parameters of the feature unification fusion decoder are updated through back-propagation.

[0020] The mathematical expression image recognition system provided by the present invention includes a feature encoder, a conditional network module, and a feature unified fusion decoder. The feature encoder includes multiple lightweight deep separable convolutions, and the conditional network module is a lightweight conditional network used to enhance global position information. The output of the feature encoder is fed into the feature unified fusion decoder via the conditional network. The feature unified fusion decoder uses an attention mechanism. This design improves the generalization of the mathematical expression image recognition system, avoiding complex calculations and increasing speed while ensuring accurate recognition even when encountering long sequences of formulas. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of a mathematical expression image recognition system provided by one embodiment;

[0023] Figure 2 Schematic diagram of a feature encoder in the present invention;

[0024] Figure 3 is a schematic diagram of the conditional network module in the present invention; and

[0025] Figure 4 Schematic diagram of the feature unified fusion decoder in the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] As used in this application, the terms "component," "module," and "system" are intended to refer to a computer-related entity, which can be hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable code, a thread of execution, a program, and / or a computer. As an illustration, both an application running on a server and a server can be a component. One or more components can reside in a process and / or a thread of execution, and a component can be located within a computer and / or distributed between two or more computers.

[0028] As used herein, the terms "inference" or "inference" generally refer to the process of inferring or reasoning about the state of a system, environment, and / or user from a set of observations captured via events and / or data. For example, inference can be used to identify a specific context or action, or can generate a probability distribution over states. Inference can be probabilistic, that is, a probability distribution over states of interest is computed based on a consideration of data and events. Inference can also refer to techniques for synthesizing higher-level events from a set of events and / or data. Such inference results in the construction of new events or actions from a set of observed events and / or stored event data, regardless of whether the events are related in adjacent time and regardless of whether the events and data come from one or several event and data sources.

[0029] The Transformer architecture, proposed in 2017, has been widely used in natural language processing due to its efficient parallelization structure and self-attention mechanism. Recently, the Transformer has been used to effectively handle computer vision tasks such as object recognition, image segmentation, and image classification. Compared to classic CNN models, Transformer-based frameworks for image tasks achieve superior accuracy.

[0030] Mathematical expression images contain serialized text content and complex spatial structures. Based on the potential of Transformer in sequence processing and computer vision, this paper studies an end-to-end mathematical formula recognition method based on Transformer, mainly innovating and improving its attention mechanism, position encoder and conditional network module, and proposing an end-to-end efficient formula recognition algorithm. While ensuring the recognition accuracy, it can reduce the computational complexity required for model reasoning. Combined with the characteristics of the mathematical expression recognition task itself, it solves the problems of large parameters and slow convergence speed of existing recognition algorithms.

[0031] like Figures 1-4As shown, the present invention provides a mathematical expression image recognition system, including a feature encoder, a conditional network module and a feature unified fusion decoder. The feature encoder includes multiple lightweight deep separable convolutions, and the conditional network module is a lightweight conditional network for enhancing global position information. The output of the feature encoder is sent to the feature unified fusion decoder via the conditional network module, and the feature unified fusion decoder adopts an attention mechanism.

[0032] The feature encoder divides an input grayscale image into several non-overlapping image blocks through a partitioning convolution layer. The dimension of the feature map after the partitioning convolution layer becomes Where w, h are the width and height of the image respectively, c is the dimension of the hidden layer, and the convolutional layer is divided into:

[0033] z0=BN(σ{Conv(x, stride=4, kernel size=4)})

[0034] The feature map is fed into multiple deep separable convolutions, where each layer of convolution is followed by a post-activation function and a normalization layer. Each layer of deep separable convolution is expressed as:

[0035] z {l} =BN(σ{DepthwiseConv(z {l-1} )})+z {l-1}

[0036] z {l+1} =BN(σ{PointwiseConv(z {l} )})

[0037] Get the output feature map of the feature encoder.

[0038] The conditional network module consists of a first linear network layer and a second linear network layer. The first linear network layer performs dimensionality compression, and the second linear network layer performs dimensionality recovery. The conditional network can be expressed as:

[0039] τ=h {θ} (V)

[0040] The parameter update of the conditional network module is achieved through back propagation.

[0041] The global position information is enhanced through the conditional network module. The output of the feature encoder is sent to the feature unified fusion decoder through a lightweight conditional network, which works together with the position encoding to stabilize the initial convergence of the model and improve the robustness of the model.

[0042] The feature unified fusion decoder generates a formula symbol for each timestamp based on the output of the feature encoder and the output of the conditional network module. A start symbol and an end symbol are added to each mathematical expression sequence, and the length is aligned with the longest formula sequence in this batch. Each character is mapped to a feature word vector, i.e., t= <t1,t2,…,t T >, where t i ∈R 256 ,T is the length of the mathematical expression.

[0043] The feature unified fusion decoder consists of six attention layers, each of which includes a self-attention part and a cross-attention part. The feature unified fusion decoder is used as a language model:

[0044] P(y t |y1,…,y t-1 ,V)=Softmax(W out O t-1 )

[0045] The parameters of the feature unification fusion decoder are updated through back-propagation.

[0046] The attention mechanism employed in the feature-unified fusion decoder enables the present invention to better capture long-sequence dependencies. Compared to lightweight feature encoders, the feature-unified fusion decoder has more parameters and a more complex network structure, making it better suited for modeling mathematical expression image sequences.

[0047] The mathematical expression image recognition system provided by the present invention includes a feature encoder, a conditional network module, and a feature unified fusion decoder. The feature encoder includes multiple lightweight deep separable convolutions, and the conditional network module is a lightweight conditional network used to enhance global position information. The output of the feature encoder is fed into the feature unified fusion decoder via the conditional network. The feature unified fusion decoder uses an attention mechanism. This design improves the generalization of the mathematical expression image recognition system, avoiding complex calculations and increasing speed while ensuring accurate recognition even when encountering long sequences of formulas.

[0048] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mathematical expression image recognition system, characterized by: It includes a feature encoder, a conditional network module and a feature unified fusion decoder. The feature encoder includes multiple lightweight deep separable convolutions. The conditional network module is a lightweight conditional network for enhancing global position information. The output of the feature encoder is sent to the feature unified fusion decoder via the conditional network module. The feature unified fusion decoder adopts an attention mechanism, wherein the feature encoder divides an input formula grayscale image into several non-overlapping image blocks through a partitioning convolution layer. The feature map after the partitioning convolution layer is sent to the multiple deep separable convolutions. Each layer of convolution is followed by a post-activation function and a normalization layer to obtain the output feature map of the feature encoder. The conditional network module is composed of a first linear network layer and a second linear network layer. The first linear network layer performs dimensionality compression and the second linear network layer performs dimensionality recovery. The feature unified fusion decoder includes six attention layers, each of which includes a self-attention part and a cross-attention part.

2. The mathematical expression image recognition system according to claim 1, characterized in that: The dimension of the feature map after the convolutional layer is divided becomes Where w, h are the width and height of the image respectively, c is the dimension of the hidden layer, and the convolutional layer is divided into: z0=BN(σ{Conv(x, stride=4, kernelsize=4)}) Each layer of deep separable convolution is expressed as: With {l} =BN(σ{DepthwiseConv(z {l-1} )})+z {l-1} With {l+1} =BN(σ{PointwiseConv(z {l} )})。 3. The mathematical expression image recognition system according to claim 2, wherein: The conditional network can be expressed as: τ=h {θ} (V) The parameter updates of the conditional network are achieved through back-propagation.

4. The mathematical expression image recognition system according to claim 3, characterized in that: The feature unified fusion decoder generates a formula symbol for each timestamp based on the output of the feature encoder and the output of the conditional network module. A start symbol and an end symbol are added to each mathematical expression sequence, and the length is aligned with the longest formula sequence in this batch. Each character is mapped to a feature word vector, i.e., t= <t1,t2,…,t T >, where t i ∈R 256 ,T is the length of the mathematical expression.

5. The mathematical expression image recognition system according to claim 4, characterized in that: Feature unified fusion decoder as a language model: P(and t |y1,…,y t-1 ,V)=Softmax(W out EITHER t-1 ) The parameters of the feature unification fusion decoder are updated through back-propagation.

Images