A method and device for formula recognition and a device for formula recognition
By preprocessing and feature extraction of the trajectory points handwritten by the user, and using the recognition model to identify information, the problem of low accuracy in formula recognition in the prior art is solved, and a more efficient formula recognition effect is achieved.
Patent Information
- Application Number
- CN202110680351.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-18
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-06-18
AI Technical Summary
The existing formula recognition method is not very accurate when users write mathematical formulas by handwriting, and often has garbled codes or unrecognized situations.
By obtaining the real-time input track points for preprocessing, including resampling and normalization, eliminating redundant points, obtaining the track point sequence, and then inputting it into the pre-trained recognition model, using the encoder to extract features and decoder for Information recognition improves the accuracy of formula recognition.
It improves the accuracy and efficiency of formula recognition, and ensures the accurate recognition results of user handwritten formulas.
Smart Images

Figure CN113408417B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information recognition technology, and in particular to a formula recognition method, device and a device for formula recognition. Background Art
[0002] Mathematical formulas are very common in textbooks, scientific papers and other documents, but the process of users inputting formulas into electronic documents is cumbersome. With the development of interactive technology between handwriting and electronic devices, users can directly write formulas on electronic devices by handwriting.
[0003] However, due to the complex spatial relationship of characters in mathematical formulas, existing formula recognition methods usually produce garbled characters or are unable to recognize them, resulting in low recognition accuracy and poor recognition effect. Summary of the invention
[0004] The embodiments of the present invention provide a formula recognition method, device and device for formula recognition, which can pre-process the trajectory points input in real time when the user writes the formula by hand, and recognize the pre-processed trajectory point series based on a pre-trained recognition model, thereby improving the accuracy of formula recognition.
[0005] In order to solve the above problem, an embodiment of the present invention discloses a formula recognition method, which includes:
[0006] Acquire trajectory points input in real time, and pre-process the trajectory points to obtain a trajectory point sequence;
[0007] Inputting the trajectory point sequence into a pre-trained recognition model, extracting and encoding features of the trajectory point sequence through an encoder in the recognition model to obtain a high-dimensional feature vector;
[0008] The decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector to obtain a formula recognition result.
[0009] Optionally, the preprocessing the trajectory points to obtain a trajectory point sequence includes:
[0010] Resampling the trajectory points to obtain resampled trajectory points;
[0011] The resampled trajectory points are normalized to obtain a trajectory point sequence.
[0012] Optionally, resampling the trajectory points to obtain resampled trajectory points includes:
[0013] Determine whether there is a target point pair in the trajectory points that meets the resampling condition, where the target point pair that meets the resampling condition includes: a distance between two points of the target point pair is less than a preset distance, and / or an angle between a connecting line of the two points of the target point pair and a preset plane is less than a preset angle;
[0014] If there is a target point pair that meets the resampling condition, all points between the target point pair are deleted to obtain the resampled trajectory points.
[0015] Optionally, the normalizing the resampled trajectory points to obtain a trajectory point sequence includes:
[0016] Determine the input stroke bounding box corresponding to the resampled trajectory points;
[0017] Adjusting the size of the input stroke bounding box according to a preset bounding box;
[0018] The trajectory points are linearly scaled according to the adjusted input stroke boundary box to obtain a trajectory point sequence.
[0019] Optionally, the method further comprises:
[0020] If the newly input track point is not recognized within the preset time, a stop character is added after the current track point;
[0021] Exchange any adjacent strokes before the stop symbol according to the exchange probability to obtain the exchanged trajectory points;
[0022] The exchanged trajectory points are added to the trajectory point sequence.
[0023] Optionally, before inputting the trajectory point sequence into a pre-trained recognition model, the method further comprises:
[0024] Acquire a training sample set, the training sample set comprising a first sample set corresponding to a first type of formula and a second sample set corresponding to a second type of formula, the first type of formula comprising at least two spatial structures and / or the trajectory points corresponding to the first type of formula are continuous points; the second type of formula comprising at most one spatial structure;
[0025] Training an initial recognition model according to the first sample set to obtain a trained first model;
[0026] Training the first model according to the second sample set to obtain a second model;
[0027] Adjust the model parameters of the second model, and iteratively train the adjusted second model according to the first sample set and / or the second sample set, and obtain a trained recognition model when the loss value corresponding to the second model is less than a preset threshold.
[0028] Optionally, the performing information recognition and decoding on the high-dimensional feature vector by a decoder in the recognition model to obtain a formula recognition result includes:
[0029] Obtaining a decoding state of each subvector in the high-dimensional feature vector;
[0030] If the decoding state of the sub-vector is undecoded, setting a first weight value for the sub-vector;
[0031] If the decoding state of the sub-vector is decoded, setting a second weight value for the sub-vector, wherein the second weight value is less than the first weight value;
[0032] Perform weighted summation on each sub-vector to obtain an adjusted high-dimensional feature vector;
[0033] The decoder in the recognition model performs information recognition and decoding on the adjusted high-dimensional feature vector to obtain a formula recognition result.
[0034] Optionally, the decoder includes at least two decoding layers, and the decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector to obtain a formula recognition result, including:
[0035] Inputting the high-dimensional feature vector into a decoder in the recognition model, and decoding the high-dimensional feature vector through at least two decoding layers of the decoder;
[0036] During the decoding process of each decoding layer, the key information in the high-dimensional feature vector is identified and decoded based on the hybrid attention mechanism to obtain the target character at the current moment, until the decoding is completed to obtain the formula recognition result composed of the target characters, and the target characters include mathematical characters and structural characters.
[0037] Optionally, the hybrid attention mechanism includes one or more of a spatial attention mechanism, a temporal attention mechanism, and an attention guidance mechanism. During the decoding process of each decoding layer, the spatial attention mechanism is used to align mathematical characters in the high-dimensional feature vector, the temporal attention mechanism is used to align structural characters in the high-dimensional feature vector, and the attention guidance mechanism is used to guide the alignment processing of the spatial attention mechanism.
[0038] Optionally, the encoder comprises a first recurrent network layer and a bidirectional second recurrent network layer.
[0039] Optionally, extracting and encoding the trajectory point sequence by an encoder in the recognition model to obtain a high-dimensional feature vector includes:
[0040] The first recurrent network layer and the bidirectional second recurrent network layer sequentially extract and encode the features of the trajectory point sequence to obtain a high-dimensional feature vector.
[0041] Optionally, extracting and encoding the trajectory point sequence by an encoder in the recognition model to obtain a high-dimensional feature vector includes:
[0042] Performing feature extraction and encoding on the trajectory point sequence through the first recurrent network layer and the bidirectional second recurrent network layer, respectively, to obtain a first high-dimensional feature vector and a second high-dimensional feature vector;
[0043] The decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector to obtain a formula recognition result, including:
[0044] The decoder in the recognition model performs information recognition and decoding on the first high-dimensional feature vector and the second high-dimensional feature vector respectively to obtain a formula recognition result.
[0045] On the other hand, an embodiment of the present invention discloses a formula recognition device, the device comprising:
[0046] A trajectory point preprocessing module, used to obtain trajectory points input in real time and preprocess the trajectory points to obtain a trajectory point sequence;
[0047] An encoding module, used for inputting the trajectory point sequence into a pre-trained recognition model, extracting and encoding features of the trajectory point sequence through an encoder in the recognition model, and obtaining a high-dimensional feature vector;
[0048] The decoding module is used to perform information recognition and decoding on the high-dimensional feature vector through a decoder in the recognition model to obtain a formula recognition result.
[0049] Optionally, the trajectory point preprocessing module includes:
[0050] A resampling submodule, used for resampling the trajectory points to obtain resampled trajectory points;
[0051] The normalization processing submodule is used to normalize the resampled trajectory points to obtain a trajectory point sequence.
[0052] Optionally, the resampling submodule includes:
[0053] a target point pair determination unit, configured to determine whether there is a target point pair in the trajectory points that satisfies a resampling condition, wherein the target point pair that satisfies the resampling condition includes: a distance between two points of the target point pair is less than a preset distance, and / or an angle between a connecting line of the two points of the target point pair and a preset plane is less than a preset angle;
[0054] The resampling unit is used to delete all points between the target point pairs if there are target point pairs that meet the resampling conditions, so as to obtain resampled trajectory points.
[0055] Optionally, the normalization processing submodule includes:
[0056] An input stroke bounding box determination unit, used to determine an input stroke bounding box corresponding to the resampled trajectory point;
[0057] An input stroke boundary box adjustment unit, used for adjusting the size of the input stroke boundary box according to a preset boundary box;
[0058] The linear scaling processing unit is used to perform linear scaling processing on the trajectory points according to the adjusted input stroke boundary box to obtain a trajectory point sequence.
[0059] Optionally, the device further comprises:
[0060] The stop symbol adding module is used to add a stop symbol after the current track point if the newly input track point is not recognized within a preset time;
[0061] A stroke exchange module, used for exchanging any adjacent strokes before the stop symbol according to the exchange probability to obtain the track points after the exchange;
[0062] The track point adding module is used to add the exchanged track points to the track point sequence.
[0063] Optionally, the device further comprises:
[0064] A training sample set acquisition module, used to acquire a training sample set, the training sample set comprising a first sample set corresponding to a first type of formula and a second sample set corresponding to a second type of formula, the first type of formula comprising at least two spatial structures and / or the trajectory points corresponding to the first type of formula are continuous points; the second type of formula comprising at most one spatial structure;
[0065] A first model training module, used for training an initial recognition model according to the first sample set to obtain a trained first model;
[0066] A second model training module, used for training the first model according to the second sample set to obtain a second model;
[0067] A model parameter adjustment module is used to adjust the model parameters of the second model, and iteratively train the adjusted second model according to the first sample set and / or the second sample set, and a trained recognition model is obtained when the loss value corresponding to the second model is less than a preset threshold.
[0068] Optionally, the decoding module includes:
[0069] A decoding state acquisition submodule, used to acquire the decoding state of each subvector in the high-dimensional feature vector;
[0070] A first weight value setting submodule, configured to set a first weight value for the subvector if the decoding state of the subvector is undecoded;
[0071] a second weight value setting submodule, configured to set a second weight value for the subvector if the decoding state of the subvector is decoded, wherein the second weight value is smaller than the first weight value;
[0072] A weighted summation submodule is used to perform weighted summation on each subvector to obtain an adjusted high-dimensional feature vector;
[0073] The first formula recognition submodule is used to perform information recognition and decoding on the adjusted high-dimensional feature vector through a decoder in the recognition model to obtain a formula recognition result.
[0074] Optionally, the decoder includes at least two decoding layers, and the decoding module includes:
[0075] A first decoding submodule, configured to input the high-dimensional feature vector into a decoder in the recognition model, and decode the high-dimensional feature vector through at least two decoding layers of the decoder;
[0076] The second formula recognition submodule is used to identify and decode the key information in the high-dimensional feature vector based on the hybrid attention mechanism during the decoding process of each decoding layer to obtain the target character at the current moment, until the decoding is completed to obtain the formula recognition result composed of the target characters, and the target characters include mathematical characters and structural characters.
[0077] Optionally, the hybrid attention mechanism includes one or more of a spatial attention mechanism, a temporal attention mechanism, and an attention guidance mechanism. During the decoding process of each decoding layer, the spatial attention mechanism is used to align mathematical characters in the high-dimensional feature vector, the temporal attention mechanism is used to align structural characters in the high-dimensional feature vector, and the attention guidance mechanism is used to guide the alignment processing of the spatial attention mechanism.
[0078] Optionally, the encoder comprises a first recurrent network layer and a bidirectional second recurrent network layer.
[0079] Optionally, the encoding module includes:
[0080] The first encoding submodule is used to extract and encode the features of the trajectory point sequence in sequence through the first recurrent network layer and the bidirectional second recurrent network layer to obtain a high-dimensional feature vector.
[0081] Optionally, the encoding module includes:
[0082] A second encoding submodule is used to extract and encode the trajectory point sequence through the first recurrent network layer and the bidirectional second recurrent network layer to obtain a first high-dimensional feature vector and a second high-dimensional feature vector;
[0083] The decoding module comprises:
[0084] The second decoding submodule is used to perform information recognition and decoding on the first high-dimensional feature vector and the second high-dimensional feature vector respectively through the decoder in the recognition model to obtain a formula recognition result.
[0085] On the other hand, an embodiment of the present invention discloses a device for formula recognition, which includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for executing one or more of the formula recognition methods described above.
[0086] On the other hand, an embodiment of the present invention discloses a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, enables the device to perform one or more of the formula recognition methods described above.
[0087] The embodiments of the present invention include the following advantages:
[0088] The embodiment of the present invention obtains the trajectory points input in real time, pre-processes the trajectory points, obtains a trajectory point sequence, then inputs the trajectory point sequence into a pre-trained recognition model, extracts and encodes the trajectory point sequence through the encoder in the recognition model, obtains a high-dimensional feature vector, and performs information recognition and decoding on the high-dimensional feature vector through the decoder in the recognition model to obtain a formula recognition result. In the process of the user handwriting the formula, the embodiment of the present invention continuously obtains the trajectory points input by the user in real time, and pre-processes the acquired trajectory points to remove redundant trajectory points, thereby obtaining a trajectory point series that meets the specification and is easy to identify, thereby improving the efficiency of trajectory point recognition; then, based on the pre-trained recognition model, the encoder in the recognition model extracts and encodes the pre-processed trajectory point series, filters out irrelevant features, obtains a high-dimensional feature vector containing the main features in the trajectory point series, and the decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector, further identifies the key information corresponding to the trajectory point series, and obtains the formula recognition result. Thus, the encoder extracts features from the trajectory point series, and then further performs information recognition through the decoder, thereby improving the accuracy of formula recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0090] Figure 1 is a flow chart of steps of an embodiment of a formula recognition method of the present invention;
[0091] Figure 2 is a structural block diagram of an embodiment of a formula recognition device of the present invention;
[0092] Figure 3 is a block diagram of an apparatus 800 for formula recognition of the present invention;
[0093] Figure 4 It is a schematic diagram of the structure of the server in some embodiments of the present invention. DETAILED DESCRIPTION
[0094] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0095] Method Embodiment
[0096] Reference Figure 1 , shows a flow chart of steps of an embodiment of a formula recognition method of the present invention, and the method may specifically include the following steps:
[0097] Step 101: Acquire trajectory points input in real time, and pre-process the trajectory points to obtain a trajectory point sequence.
[0098] Step 102: input the trajectory point sequence into a pre-trained recognition model, and extract and encode the trajectory point sequence through an encoder in the recognition model to obtain a high-dimensional feature vector.
[0099] Step 103: perform information recognition and decoding on the high-dimensional feature vector through a decoder in the recognition model to obtain a formula recognition result.
[0100] The formula recognition method provided in the embodiment of the present invention can be applied to a formula recognition module of a terminal device, wherein the terminal device supports a formula writing function and has a touch screen or a handwriting board, and can be used to collect various track points generated during the user's handwriting of the formula. The terminal device includes but is not limited to a mobile terminal, a tablet computer, a personal digital assistant (PDA), a computer, etc.
[0101] It should be noted that the real-time input trajectory points obtained in the embodiments of the present invention include trajectory points generated when a user writes a formula on a terminal device through a touch screen or a handwriting tablet, and also include real-time writing trajectories captured by a camera. The real-time writing trajectory can be the writing trajectory points generated when a user writes a formula on a blackboard or paper.
[0102] The formula recognition method provided by the embodiment of the present invention can obtain in real time the trajectory points generated when a user handwrites a formula on a terminal device, and pre-process the obtained trajectory points to eliminate redundant trajectory points to obtain a trajectory point series that meets the specifications and is easy to identify, thereby improving the efficiency of trajectory point recognition.
[0103] In order to improve the accuracy of formula recognition, the embodiment of the present invention uses the recognition model pre-trained with the pre-processed trajectory point series, and uses the encoder in the recognition model to extract and encode the pre-processed trajectory point series, filter out irrelevant features, and obtain a high-dimensional feature vector containing the main features of the trajectory point series, and uses the decoder in the recognition model to perform information recognition and decoding on the high-dimensional feature vector, further identify the key information corresponding to the trajectory point series, and obtain the formula recognition result. Thus, the trajectory point series is feature extracted by the encoder, and then further information is recognized by the decoder, thereby improving the accuracy of formula recognition.
[0104] It should be noted that, in the embodiment of the present invention, the formula recognition result is usually a latex string corresponding to the trajectory point input by the user in real time, including various symbols of the mathematical formula, such as numbers, letters, specific mathematical symbols, operators, etc., and also includes various spatial relationships in the mathematical formula, such as left and right, up and down, upper left, upper right, lower left, lower right, space, etc. Based on the recognized latex string, the mathematical formula handwritten by the user can be determined.
[0105] Among them, the encoder in the recognition model can be constructed based on a bidirectional recurrent neural network, for example, an encoder is constructed based on a double-layer or multi-layer Bi-GRU (Bidirectional-Gated Recurrent Unit), Bi-GRU+GRCL (Gated Recurrent Convolution Layers) and other network models. The encoder is used to extract and encode the trajectory point sequence to obtain a high-dimensional feature vector corresponding to the trajectory point sequence. The high-dimensional feature vector contains the formula features corresponding to the trajectory point sequence, such as numbers, letters, specific formula symbols, operators and other feature information.
[0106] The decoder in the recognition model can be constructed based on a unidirectional recurrent network, for example, a decoder can be constructed based on a unidirectional double-layer or multi-layer GRU (Gated Recurrent Unit), Transformer network, etc. The attention mechanism can also be further introduced into the decoder to enhance the decoding ability of the decoder. For example, a hybrid attention mechanism is introduced into the decoder, and the hybrid attention mechanism includes one or more of the Spatial Attention mechanism, the Temporal Attention mechanism, and the Attention Guider mechanism, which is used to improve the alignment effect of each character in the formula recognition result and the decoding ability of the decoder. Among them, the Spatial Attention mechanism uses the historical information of the recognition model to solve the alignment confusion problem of mathematical characters in the high-dimensional feature vector corresponding to the user's handwritten formula. The Temporal Attention mechanism is used to handle the special alignment of structural characters in high-dimensional feature vectors. In an embodiment of the present invention, in order to reproduce the two-dimensional spatial structure in the mathematical formula language, in addition to recognizing the entity characters in the user's handwritten formula, such as numbers, letters, special mathematical symbols, operators, etc., it is also necessary to recognize the structural characters corresponding to the user's handwritten formula to determine the two-dimensional spatial structure of the mathematical formula. These special structural characters do not exist in the input mathematical formula, and the alignment of the structural characters in the mathematical formula cannot be completed by relying solely on the Spatial Attention mechanism. Therefore, in an embodiment of the present invention, the alignment of the structural characters is supplemented by the Temporal Attention mechanism, which can not only not interfere with the learning of SpatialAttention, but also improve the decoding ability of the decoder and further improve the performance. The Attention Guider mechanism is used to strengthen the learning of Spatial Attention, and the Attention Guider mechanism is used to guide the learning of SpatialAttention, further improve the alignment effect of Spatial Attention, and then improve the accuracy of formula recognition.
[0107] In an optional embodiment of the present invention, the step 101 of preprocessing the trajectory points to obtain a trajectory point sequence includes:
[0108] Step S11, resampling the trajectory points to obtain resampled trajectory points;
[0109] Step S12: normalize the resampled trajectory points to obtain a trajectory point sequence.
[0110] In the embodiment of the present invention, the trajectory points obtained by the user in real time often contain multiple redundant trajectory points, and different users have different writing habits, and the writing trajectory of the same character will also be different. Therefore, in order to eliminate redundant trajectory points and obtain standardized and easily identifiable trajectory points, the embodiment of the present invention pre-processes the obtained trajectory points. Specifically, it can include resampling the input trajectory points and normalizing the resampled trajectory points to obtain a trajectory point sequence.
[0111] It should be noted that, in the embodiment of the present invention, the order of resampling and normalization is not specifically limited. When preprocessing the input trajectory points, in addition to the processing order described in step S11 to step S12, the input trajectory points may be normalized first to obtain normalized trajectory points, and then the normalized trajectory points may be resampled to obtain a trajectory point sequence. Whether the input trajectory points are resampled first and then normalized, or the input trajectory points are normalized first and then resampled, the same effect is achieved, and redundant trajectory points can be eliminated to obtain standardized and easily identifiable trajectory points, thereby improving the efficiency of formula recognition.
[0112] In an optional embodiment of the present invention, the step S11 of resampling the trajectory points to obtain resampled trajectory points includes:
[0113] Sub-step S111, determining whether there is a target point pair in the trajectory points that meets the resampling condition, wherein the target point pair that meets the resampling condition includes: the distance between the two points of the target point pair is less than a preset distance, and / or the angle between the connecting line of the two points of the target point pair and the preset plane is less than a preset angle;
[0114] Sub-step S112: If there is a target point pair that meets the resampling condition, all points between the target point pair are deleted to obtain the resampled trajectory points.
[0115] When resampling the input trajectory points, if the distance between two points is less than the preset distance, the user's writing trajectory can be directly determined based on the line connecting the two points, and the point between the two points is a redundant point; similarly, if the angle between the connecting line between the two points and the preset plane is less than the preset angle, the user's writing trajectory can be directly determined based on the line connecting the two points, and the point between the two points is a redundant point. Therefore, in the embodiment of the present invention, by determining whether there is a target point pair in the trajectory points, all points between the target point pair can be deleted to achieve the purpose of eliminating redundant points.
[0116] In an optional embodiment of the present invention, the step S12 of normalizing the resampled trajectory points to obtain a trajectory point sequence includes:
[0117] Sub-step S121, determining the input stroke boundary box corresponding to the resampled trajectory point;
[0118] Sub-step S122, adjusting the size of the input stroke boundary box according to a preset boundary box;
[0119] Sub-step S123: performing linear scaling processing on the trajectory points according to the adjusted input stroke boundary box to obtain a trajectory point sequence.
[0120] In the embodiment of the present invention, the normalization process of the trajectory points is mainly to facilitate the determination of the spatial structural relationship between characters based on the size and position of the input characters. The size of the user's handwritten formula is determined based on the change of the input trajectory. If the judgment criteria for the size and position of the input characters are not unified, the spatial structural relationship between the characters cannot be reasonably and accurately determined. Therefore, in the embodiment of the present invention, the size of the input stroke boundary box corresponding to the trajectory point is adjusted according to the preset boundary box, and then the size of each trajectory point is linearly scaled based on the adjusted input trajectory boundary box, so that the size of each trajectory point is scaled to a standard aspect ratio, which is convenient for subsequent recognition processing.
[0121] In an optional embodiment of the present invention, the method further includes:
[0122] Step S21: if the newly input track point is not recognized within the preset time, a stop symbol is added after the current track point;
[0123] Step S22, exchanging any adjacent strokes before the stop symbol according to the exchange probability to obtain the exchanged trajectory points;
[0124] Step S23: adding the exchanged trajectory points to the trajectory point sequence.
[0125] In an embodiment of the present invention, in addition to resampling and normalizing the trajectory points to obtain a preprocessed trajectory point sequence, the trajectory points can also be exchanged by stroke. Specifically, when it is recognized that a stroke is finished, any two adjacent strokes that have been written can be exchanged. Whether the stroke is finished can be judged by the terminator. When the newly input trajectory point is not recognized within the preset time, it means that the writing of a stroke is finished, and a terminator is added after the current trajectory point. The user's writing operation can also be identified by a pressure sensor. When the detected pressure value is less than the preset threshold, it means that the user lifts the pen, and the writing of a stroke is finished, and a terminator is added after the current trajectory point. The terminator can also be set according to actual needs. For example, (-1,0) can be set as a terminator, or "00" can be set as a terminator, and so on. This embodiment of the present invention does not make specific restrictions. If it is necessary to exchange strokes, it is only necessary to identify the terminator, and any adjacent strokes before the terminator are exchanged to obtain the exchanged trajectory point, and the exchanged trajectory point is added to the trajectory point sequence, so that the formula recognition can be performed based on the trajectory point sequence later.
[0126] It should be noted that, in practical applications, the stroke exchange process described in step S21 to step S23 is not necessary, so when performing the stroke exchange process, the exchange probability can be set according to actual needs, and any adjacent strokes before the stop symbol can be exchanged according to the exchange probability. In practical applications, a pre-trained recognition model or a trajectory point preprocessing model can be used to perform stroke exchange processing on the trajectory points according to the set exchange probability.
[0127] Furthermore, in the embodiment of the present invention, in addition to resampling, normalization, and stroke exchange of the trajectory points, the trajectory points of a certain stroke may also be stretched to a certain extent and / or rotated. The size ratio of the trajectory points before and after stretching does not exceed a preset stretching threshold, and the rotation direction may be clockwise or counterclockwise, but the rotation direction and rotation angle of each trajectory point of the same stroke are the same, and the rotation angle does not exceed a preset rotation angle threshold.
[0128] In an optional embodiment of the present invention, the encoder includes a first recurrent network layer and a bidirectional second recurrent network layer.
[0129] The encoder in the recognition model can be constructed based on a double-layer or multi-layer bidirectional second recurrent network, for example, a Bi-GRU network; it can also be constructed based on a first recurrent network layer and a bidirectional second recurrent network layer, for example, GRCL+Bi-GRU, Bi-GRU+GRCL.
[0130] In an optional embodiment of the present invention, step 102 extracts and encodes the trajectory point sequence by an encoder in the recognition model to obtain a high-dimensional feature vector, including:
[0131] The first recurrent network layer and the bidirectional second recurrent network layer sequentially extract and encode the features of the trajectory point sequence to obtain a high-dimensional feature vector.
[0132] When the encoder is constructed based on the first recurrent network layer and the bidirectional second recurrent network layer, the first recurrent network layer and the bidirectional second recurrent network layer can be used to extract features and encode the trajectory point sequence in sequence. Specifically, the trajectory point sequence can be firstly extracted and encoded by the first recurrent network layer to obtain a first encoding result, and then the first encoding result can be extracted and encoded by the bidirectional second recurrent network layer to obtain a high-dimensional feature vector, such as the GRCL+Bi-GRU model; or, the trajectory point sequence can be firstly extracted and encoded by the bidirectional second recurrent network layer to obtain a second encoding result, and then the second encoding result can be extracted and encoded by the first recurrent network layer to obtain a high-dimensional feature vector, such as the Bi-GRU+GRCL model. The accuracy of formula recognition is improved by extracting features of the trajectory sequence in sequence through two recurrent network layers.
[0133] In an optional embodiment of the present invention, step 102 extracts and encodes the trajectory point sequence by an encoder in the recognition model to obtain a high-dimensional feature vector, including:
[0134] Step S31, extracting and encoding the trajectory point sequence through the first recurrent network layer and the bidirectional second recurrent network layer respectively to obtain a first high-dimensional feature vector and a second high-dimensional feature vector;
[0135] Step 103, in which the decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector to obtain a formula recognition result, includes:
[0136] Step S32: using the decoder in the recognition model to perform information recognition and decoding on the first high-dimensional feature vector and the second high-dimensional feature vector respectively to obtain a formula recognition result.
[0137] In an embodiment of the present invention, when an encoder is jointly constructed according to the first recurrent network layer and the bidirectional second recurrent network layer, the trajectory point sequence can also be feature extracted and encoded by the first recurrent network layer and the bidirectional second recurrent network layer, respectively, to obtain a first high-dimensional feature vector and a second high-dimensional feature vector, and the first high-dimensional feature vector and the second high-dimensional feature vector can be respectively information identified and decoded by the decoder, and the formula recognition result can be jointly determined based on the two decoding results obtained, which can also improve the recognition accuracy.
[0138] In an optional embodiment of the present invention, the step 103 of performing information recognition and decoding on the high-dimensional feature vector by a decoder in the recognition model to obtain a formula recognition result includes:
[0139] Step S41, obtaining the decoding status of each sub-vector in the high-dimensional feature vector;
[0140] Step S42: If the decoding state of the sub-vector is undecoded, setting a first weight value for the sub-vector;
[0141] Step S43: If the decoding state of the sub-vector is decoded, a second weight value is set for the sub-vector, and the second weight value is smaller than the first weight value;
[0142] Step S44, performing weighted summation on each sub-vector to obtain an adjusted high-dimensional feature vector;
[0143] Step S45: performing information recognition and decoding on the adjusted high-dimensional feature vector through the decoder in the recognition model to obtain a formula recognition result.
[0144] Since the decoder focuses on the undecoded part, in an embodiment of the present invention, the decoding status of each sub-vector in the high-dimensional feature vector can be analyzed, and a larger weight can be set for the undecoded sub-vector and a smaller weight can be set for the decoded sub-vector, so that the decoder can focus on processing the undecoded part and improve the decoding efficiency.
[0145] In an optional embodiment of the present invention, the decoder includes at least two decoding layers, and step 103 performs information recognition and decoding on the high-dimensional feature vector by the decoder in the recognition model to obtain a formula recognition result, including:
[0146] Step S51, inputting the high-dimensional feature vector into a decoder in the recognition model, and decoding the high-dimensional feature vector through at least two decoding layers of the decoder;
[0147] Step S52: During the decoding process of each decoding layer, the key information in the high-dimensional feature vector is identified and decoded based on the hybrid attention mechanism to obtain the target character at the current moment, until the decoding is completed to obtain the formula recognition result composed of the target characters, and the target characters include mathematical characters and structural characters.
[0148] In an embodiment of the present invention, the decoder in the recognition model can be constructed based on a unidirectional recurrent network, for example, a decoder can be constructed based on a unidirectional double-layer or multi-layer GRU, Transformer network, etc. In order to enhance the decoding capability of the decoder, a hybrid attention mechanism can be further introduced for each decoding layer of the decoder, so that in the decoding process of each decoding layer, the key information in the high-dimensional feature vector is identified and decoded based on the hybrid attention mechanism to obtain the target character at the current moment, such as the latex character at the current moment, until the decoding is completed. After the decoding is completed, the target character string corresponding to the high-dimensional feature vector can be obtained, that is, the formula recognition result.
[0149] It should be noted that the target characters include mathematical characters and structural characters. Mathematical characters are physical characters input by the user's handwriting, such as numbers, letters, specific mathematical symbols, operators, etc. Structural characters are used to represent the two-dimensional spatial structure in the user's handwritten formula, including characters corresponding to spatial structures such as left and right, up and down, upper left, upper right, lower left, lower right, and space. Based on mathematical characters and structural characters, the formula recognition result can be determined.
[0150] In an optional embodiment of the present invention, the hybrid attention mechanism includes one or more of a spatial attention mechanism, a temporal attention mechanism, and an attention guidance mechanism. During the decoding process of each decoding layer, the spatial attention mechanism is used to align mathematical characters in the high-dimensional feature vector, the temporal attention mechanism is used to align structural characters in the high-dimensional feature vector, and the attention guidance mechanism is used to guide the alignment processing of the spatial attention mechanism.
[0151] In an embodiment of the present invention, in order to enhance the decoding capability of the decoder, a hybrid attention mechanism may be further introduced in each decoding layer of the decoder. The hybrid attention mechanism may include one or more of a Spatial Attention mechanism, a Temporal Attention mechanism, and an Attention Guider mechanism, which are used to improve the alignment effect of the recognition model and the decoding capability of the decoder. Among them, the SpatialAttention mechanism uses the historical information of the recognition model to solve the alignment confusion problem of mathematical characters in the high-dimensional feature vector corresponding to the user's handwritten formula. For example, the mathematical characters corresponding to each stroke in the user's handwritten formula, as well as the order and size relationship between each mathematical character, are identified. The Temporal Attention mechanism is used to process the special alignment of structural characters in the high-dimensional feature vector, for example, the two-dimensional spatial structural relationship of each mathematical character in the user's handwritten formula is identified, and the corresponding structural characters and the mathematical characters corresponding to the structural characters are determined according to the two-dimensional spatial structural relationship. Based on the Spatial Attention mechanism and the Temporal Attention mechanism, the entity content of the user's handwritten formula and the spatial structural relationship between the entity content can be determined, thereby obtaining an accurate and complete formula recognition result. The Attention Guider mechanism is used to strengthen the learning of Spatial Attention. The Attention Guider mechanism is used to guide the learning of Spatial Attention, further improve the alignment effect of Spatial Attention, and thus improve the accuracy of formula recognition.
[0152] In an optional embodiment of the present invention, before inputting the trajectory point sequence into a pre-trained recognition model, the method further includes:
[0153] Step S61, obtaining a training sample set, the training sample set including a first sample set corresponding to a first type of formula and a second sample set corresponding to a second type of formula, the first type of formula containing at least two spatial structures and / or the trajectory points corresponding to the first type of formula are continuous points; the second type of formula containing at most one spatial structure;
[0154] Step S62: training the initial recognition model according to the first sample set to obtain a trained first model;
[0155] Step S63: training the first model according to the second sample set to obtain a second model;
[0156] Step S64, adjust the model parameters of the second model, and iteratively train the adjusted second model according to the first sample set and / or the second sample set, and obtain a trained recognition model when the loss value corresponding to the second model is less than a preset threshold.
[0157] In an embodiment of the present invention, the pre-established recognition model may be trained based on the first sample set and the second sample set in sequence. The user's handwritten formula may contain multiple spatial structures at the same time. In an example, the formula to be input is as follows:
[0158]
[0159] The above formula (1) contains both the upper right structure and the nested structure; there may also be a formula with connected strokes, that is, the obtained trajectory points are continuous points. This type of formula structure is relatively complex and is usually difficult to recognize. Therefore, in order to improve the recognition accuracy of the recognition model, in an embodiment of the present invention, the obtained training samples can be divided into a first sample set and a second sample set according to the spatial structure and trajectory point features contained in the formula, and the recognition model is first trained with difficult samples through the first sample set to obtain the first model; and then the first model is trained with simple samples through the second sample set to obtain the second model. Usually, after obtaining the second model, the training of the recognition model has been completed. In order to further improve the recognition accuracy of the recognition model, the embodiment of the present invention can further adjust the model parameters of the second model, such as learning rate, regularization parameter, sample weight, etc., and then iteratively train the adjusted second model based on the first sample set or the second sample set, and continuously determine the loss value corresponding to the second model. When the loss value is less than the preset threshold, the iteration is stopped to obtain the trained recognition model. Among them, the preset threshold can be set according to the actual recognition accuracy requirements.
[0160] In addition, in an embodiment of the present invention, a complex teacher model can be trained first, for example, a teacher model based on a multi-layer Transformer network can be trained. Then, effective information in the teacher model is extracted and transferred to a simpler student model by means of knowledge distillation. For example, by extracting effective information from a multi-layer Transformer model, a two-layer GRU small model is obtained. Since the teacher model has higher accuracy and more parameters, the student model is obtained by performing knowledge distillation on the student model through the trained teacher model, that is, the recognition model in the embodiment of the present invention, which can reduce the number of parameters of the recognition model, but the accuracy is closer to the teacher model, thereby improving the recognition accuracy; and the latency is lower, which improves the recognition efficiency of the recognition model.
[0161] In summary, the embodiment of the present invention obtains the trajectory points input in real time, pre-processes the trajectory points, obtains the trajectory point sequence, and then inputs the trajectory point sequence into the pre-trained recognition model, extracts and encodes the trajectory point sequence through the encoder in the recognition model to obtain a high-dimensional feature vector, and performs information recognition and decoding on the high-dimensional feature vector through the decoder in the recognition model to obtain the formula recognition result. In the process of the user handwriting the formula, the embodiment of the present invention continuously obtains the trajectory points input by the user in real time, and pre-processes the acquired trajectory points to eliminate redundant trajectory points, so as to obtain a trajectory point series that meets the specification and is easy to identify, thereby improving the efficiency of trajectory point recognition; then, based on the pre-trained recognition model, the encoder in the recognition model extracts and encodes the pre-processed trajectory point series, filters out irrelevant features, obtains a high-dimensional feature vector containing the main features in the trajectory point series, and the decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector, further identifies the key information corresponding to the trajectory point series, and obtains the formula recognition result. Therefore, the encoder extracts features from the trajectory point series, and then further performs information recognition through the decoder, thereby improving the accuracy of formula recognition.
[0162] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0163] Device Embodiment
[0164] Reference Figure 2 , shows a structural block diagram of an apparatus embodiment of the present invention, the apparatus may include:
[0165] A trajectory point preprocessing module 201 is used to obtain trajectory points input in real time and preprocess the trajectory points to obtain a trajectory point sequence;
[0166] An encoding module 202, used for inputting the trajectory point sequence into a pre-trained recognition model, extracting features and encoding the trajectory point sequence through an encoder in the recognition model, and obtaining a high-dimensional feature vector;
[0167] The decoding module 203 is used to perform information recognition and decoding on the high-dimensional feature vector through a decoder in the recognition model to obtain a formula recognition result.
[0168] Optionally, the trajectory point preprocessing module includes:
[0169] A resampling submodule, used for resampling the trajectory points to obtain resampled trajectory points;
[0170] The normalization processing submodule is used to normalize the resampled trajectory points to obtain a trajectory point sequence.
[0171] Optionally, the resampling submodule includes:
[0172] a target point pair determination unit, configured to determine whether there is a target point pair in the trajectory points that satisfies a resampling condition, wherein the target point pair that satisfies the resampling condition includes: a distance between two points of the target point pair is less than a preset distance, and / or an angle between a connecting line of the two points of the target point pair and a preset plane is less than a preset angle;
[0173] The resampling unit is used to delete all points between the target point pairs if there are target point pairs that meet the resampling conditions, so as to obtain resampled trajectory points.
[0174] Optionally, the normalization processing submodule includes:
[0175] An input stroke bounding box determination unit, used to determine an input stroke bounding box corresponding to the resampled trajectory point;
[0176] An input stroke boundary box adjustment unit, used for adjusting the size of the input stroke boundary box according to a preset boundary box;
[0177] The linear scaling processing unit is used to perform linear scaling processing on the trajectory points according to the adjusted input stroke boundary box to obtain a trajectory point sequence.
[0178] Optionally, the device further comprises:
[0179] The stop symbol adding module is used to add a stop symbol after the current track point if the newly input track point is not recognized within a preset time;
[0180] A stroke exchange module, used for exchanging any adjacent strokes before the stop symbol according to the exchange probability to obtain the track points after the exchange;
[0181] The track point adding module is used to add the exchanged track points to the track point sequence.
[0182] Optionally, the device further comprises:
[0183] A training sample set acquisition module, used to acquire a training sample set, the training sample set comprising a first sample set corresponding to a first type of formula and a second sample set corresponding to a second type of formula, the first type of formula comprising at least two spatial structures and / or the trajectory points corresponding to the first type of formula are continuous points; the second type of formula comprising at most one spatial structure;
[0184] A first model training module, used for training an initial recognition model according to the first sample set to obtain a trained first model;
[0185] A second model training module, used for training the first model according to the second sample set to obtain a second model;
[0186] A model parameter adjustment module is used to adjust the model parameters of the second model, and iteratively train the adjusted second model according to the first sample set and / or the second sample set, and a trained recognition model is obtained when the loss value corresponding to the second model is less than a preset threshold.
[0187] Optionally, the decoding module includes:
[0188] A decoding state acquisition submodule, used to acquire the decoding state of each subvector in the high-dimensional feature vector;
[0189] A first weight value setting submodule, configured to set a first weight value for the subvector if the decoding state of the subvector is undecoded;
[0190] a second weight value setting submodule, configured to set a second weight value for the subvector if the decoding state of the subvector is decoded, wherein the second weight value is smaller than the first weight value;
[0191] A weighted summation submodule is used to perform weighted summation on each subvector to obtain an adjusted high-dimensional feature vector;
[0192] The first formula recognition submodule is used to perform information recognition and decoding on the adjusted high-dimensional feature vector through a decoder in the recognition model to obtain a formula recognition result.
[0193] Optionally, the decoder includes at least two decoding layers, and the decoding module includes:
[0194] A first decoding submodule, configured to input the high-dimensional feature vector into a decoder in the recognition model, and decode the high-dimensional feature vector through at least two decoding layers of the decoder;
[0195] The second formula recognition submodule is used to identify and decode the key information in the high-dimensional feature vector based on the hybrid attention mechanism during the decoding process of each decoding layer to obtain the target character at the current moment, until the decoding is completed to obtain the formula recognition result composed of the target characters, and the target characters include mathematical characters and structural characters.
[0196] Optionally, the hybrid attention mechanism includes one or more of a spatial attention mechanism, a temporal attention mechanism, and an attention guidance mechanism. During the decoding process of each decoding layer, the spatial attention mechanism is used to align mathematical characters in the high-dimensional feature vector, the temporal attention mechanism is used to align structural characters in the high-dimensional feature vector, and the attention guidance mechanism is used to guide the alignment processing of the spatial attention mechanism.
[0197] Optionally, the encoder comprises a first recurrent network layer and a bidirectional second recurrent network layer.
[0198] Optionally, the encoding module includes:
[0199] The first encoding submodule is used to extract and encode the features of the trajectory point sequence in sequence through the first recurrent network layer and the bidirectional second recurrent network layer to obtain a high-dimensional feature vector.
[0200] Optionally, the encoding module includes:
[0201] A second encoding submodule is used to extract and encode the trajectory point sequence through the first recurrent network layer and the bidirectional second recurrent network layer to obtain a first high-dimensional feature vector and a second high-dimensional feature vector;
[0202] The decoding module comprises:
[0203] The second decoding submodule is used to perform information recognition and decoding on the first high-dimensional feature vector and the second high-dimensional feature vector respectively through the decoder in the recognition model to obtain a formula recognition result.
[0204] In summary, the embodiment of the present invention obtains the trajectory points input in real time, pre-processes the trajectory points, obtains the trajectory point sequence, and then inputs the trajectory point sequence into the pre-trained recognition model, extracts and encodes the trajectory point sequence through the encoder in the recognition model to obtain a high-dimensional feature vector, and performs information recognition and decoding on the high-dimensional feature vector through the decoder in the recognition model to obtain the formula recognition result. In the process of the user handwriting the formula, the embodiment of the present invention continuously obtains the trajectory points input by the user in real time, and pre-processes the acquired trajectory points to eliminate redundant trajectory points, so as to obtain a trajectory point series that meets the specification and is easy to identify, thereby improving the efficiency of trajectory point recognition; then, based on the pre-trained recognition model, the encoder in the recognition model extracts and encodes the pre-processed trajectory point series, filters out irrelevant features, obtains a high-dimensional feature vector containing the main features in the trajectory point series, and the decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector, further identifies the key information corresponding to the trajectory point series, and obtains the formula recognition result. Therefore, the encoder extracts features from the trajectory point series, and then further performs information recognition through the decoder, thereby improving the accuracy of formula recognition.
[0205] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0206] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0207] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0208] An embodiment of the present invention provides a device for formula recognition, which includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: obtaining trajectory points input in real time, and preprocessing the trajectory points to obtain a trajectory point sequence; inputting the trajectory point sequence into a pre-trained recognition model, and performing feature extraction and encoding on the trajectory point sequence through an encoder in the recognition model to obtain a high-dimensional feature vector; performing information recognition and decoding on the high-dimensional feature vector through a decoder in the recognition model to obtain a formula recognition result.
[0209] Optionally, the preprocessing the trajectory points to obtain a trajectory point sequence includes:
[0210] Resampling the trajectory points to obtain resampled trajectory points;
[0211] The resampled trajectory points are normalized to obtain a trajectory point sequence.
[0212] Optionally, resampling the trajectory points to obtain resampled trajectory points includes:
[0213] Determine whether there is a target point pair in the trajectory points that meets the resampling condition, where the target point pair that meets the resampling condition includes: a distance between two points of the target point pair is less than a preset distance, and / or an angle between a connecting line of the two points of the target point pair and a preset plane is less than a preset angle;
[0214] If there is a target point pair that meets the resampling condition, all points between the target point pair are deleted to obtain the resampled trajectory points.
[0215] Optionally, the normalizing the resampled trajectory points to obtain a trajectory point sequence includes:
[0216] Determine the input stroke bounding box corresponding to the resampled trajectory points;
[0217] Adjusting the size of the input stroke bounding box according to a preset bounding box;
[0218] The trajectory points are linearly scaled according to the adjusted input stroke boundary box to obtain a trajectory point sequence.
[0219] Optionally, the device is further configured to execute, by one or more processors, the one or more programs including instructions for performing the following operations:
[0220] If the newly input track point is not recognized within the preset time, a stop character is added after the current track point;
[0221] Exchange any adjacent strokes before the stop symbol according to the exchange probability to obtain the exchanged trajectory points;
[0222] The exchanged trajectory points are added to the trajectory point sequence.
[0223] Optionally, before inputting the trajectory point sequence into the pre-trained recognition model, the device is further configured to execute, by one or more processors, the one or more programs including instructions for performing the following operations:
[0224] Acquire a training sample set, the training sample set comprising a first sample set corresponding to a first type of formula and a second sample set corresponding to a second type of formula, the first type of formula comprising at least two spatial structures and / or the trajectory points corresponding to the first type of formula are continuous points; the second type of formula comprising at most one spatial structure;
[0225] Training an initial recognition model according to the first sample set to obtain a trained first model;
[0226] Training the first model according to the second sample set to obtain a second model;
[0227] Adjust the model parameters of the second model, and iteratively train the adjusted second model according to the first sample set and / or the second sample set, and obtain a trained recognition model when the loss value corresponding to the second model is less than a preset threshold.
[0228] Optionally, the performing information recognition and decoding on the high-dimensional feature vector by a decoder in the recognition model to obtain a formula recognition result includes:
[0229] Obtaining a decoding state of each subvector in the high-dimensional feature vector;
[0230] If the decoding state of the sub-vector is undecoded, setting a first weight value for the sub-vector;
[0231] If the decoding state of the sub-vector is decoded, setting a second weight value for the sub-vector, wherein the second weight value is less than the first weight value;
[0232] Perform weighted summation on each sub-vector to obtain an adjusted high-dimensional feature vector;
[0233] The decoder in the recognition model performs information recognition and decoding on the adjusted high-dimensional feature vector to obtain a formula recognition result.
[0234] Optionally, the decoder includes at least two decoding layers, and the decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector to obtain a formula recognition result, including:
[0235] Inputting the high-dimensional feature vector into a decoder in the recognition model, and decoding the high-dimensional feature vector through at least two decoding layers of the decoder;
[0236] During the decoding process of each decoding layer, the key information in the high-dimensional feature vector is identified and decoded based on the hybrid attention mechanism to obtain the target character at the current moment, until the decoding is completed to obtain the formula recognition result composed of the target characters, and the target characters include mathematical characters and structural characters.
[0237] Optionally, the hybrid attention mechanism includes one or more of a spatial attention mechanism, a temporal attention mechanism, and an attention guidance mechanism. During the decoding process of each decoding layer, the spatial attention mechanism is used to align mathematical characters in the high-dimensional feature vector, the temporal attention mechanism is used to align structural characters in the high-dimensional feature vector, and the attention guidance mechanism is used to guide the alignment processing of the spatial attention mechanism.
[0238] Optionally, the encoder comprises a first recurrent network layer and a bidirectional second recurrent network layer.
[0239] Optionally, extracting and encoding the trajectory point sequence by an encoder in the recognition model to obtain a high-dimensional feature vector includes:
[0240] The first recurrent network layer and the bidirectional second recurrent network layer sequentially extract and encode the features of the trajectory point sequence to obtain a high-dimensional feature vector.
[0241] Optionally, extracting and encoding the trajectory point sequence by an encoder in the recognition model to obtain a high-dimensional feature vector includes:
[0242] Performing feature extraction and encoding on the trajectory point sequence through the first recurrent network layer and the bidirectional second recurrent network layer, respectively, to obtain a first high-dimensional feature vector and a second high-dimensional feature vector;
[0243] The decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector to obtain a formula recognition result, including:
[0244] The decoder in the recognition model performs information recognition and decoding on the first high-dimensional feature vector and the second high-dimensional feature vector respectively to obtain a formula recognition result.
[0245] Figure 3 8 is a block diagram of an apparatus 800 for formula recognition according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0246] Reference Figure 3 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0247] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing element 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned formula recognition method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0248] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0249] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.
[0250] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0251] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice information processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0252] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0253] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also formulaically identify the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0254] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0255] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0256] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0257] Figure 4 1 is a schematic diagram of the structure of the server in some embodiments of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.
[0258] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0259] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to perform Figure 1 The formula identification method shown.
[0260] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to execute a formula recognition method, the method comprising: obtaining trajectory points input in real time, and preprocessing the trajectory points to obtain a trajectory point sequence; inputting the trajectory point sequence into a pre-trained recognition model, and performing feature extraction and encoding on the trajectory point sequence through an encoder in the recognition model to obtain a high-dimensional feature vector; performing information recognition and decoding on the high-dimensional feature vector through a decoder in the recognition model to obtain a formula recognition result.
[0261] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.
[0262] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
[0263] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
[0264] The above is a detailed introduction to a formula recognition method, a formula recognition device and a device for formula recognition provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A formula recognition method, characterized in that: The method comprises: Acquire trajectory points input in real time, and pre-process the trajectory points to obtain a trajectory point sequence; Inputting the trajectory point sequence into a pre-trained recognition model, extracting and encoding features of the trajectory point sequence through an encoder in the recognition model to obtain a high-dimensional feature vector; Obtaining a decoding state of each subvector in the high-dimensional feature vector; If the decoding state of the sub-vector is undecoded, setting a first weight value for the sub-vector; If the decoding state of the sub-vector is decoded, setting a second weight value for the sub-vector, wherein the second weight value is less than the first weight value; Perform weighted summation on each sub-vector to obtain an adjusted high-dimensional feature vector; The decoder in the recognition model performs information recognition and decoding on the adjusted high-dimensional feature vector to obtain a formula recognition result.
2. The method according to claim 1, characterized in that The preprocessing of the trajectory points to obtain a trajectory point sequence includes: Resampling the trajectory points to obtain resampled trajectory points; The resampled trajectory points are normalized to obtain a trajectory point sequence.
3. The method according to claim 2, characterized in that The resampling of the trajectory points to obtain resampled trajectory points includes: Determine whether there is a target point pair in the trajectory points that meets the resampling condition, where the target point pair that meets the resampling condition includes: a distance between two points of the target point pair is less than a preset distance, and / or an angle between a connecting line of the two points of the target point pair and a preset plane is less than a preset angle; If there is a target point pair that meets the resampling condition, all points between the target point pair are deleted to obtain the resampled trajectory points.
4. The method according to claim 2, characterized in that: The normalization process is performed on the resampled trajectory points to obtain a trajectory point sequence, including: Determine the input stroke bounding box corresponding to the resampled trajectory points; Adjusting the size of the input stroke bounding box according to a preset bounding box; The trajectory points are linearly scaled according to the adjusted input stroke boundary box to obtain a trajectory point sequence.
5. The method according to claim 1, characterized in that The method further comprises: If the newly input track point is not recognized within the preset time, a stop character is added after the current track point; Exchange any adjacent strokes before the stop symbol according to the exchange probability to obtain the exchanged trajectory points; The exchanged trajectory points are added to the trajectory point sequence.
6. The method according to claim 1, characterized in that Before inputting the trajectory point sequence into the pre-trained recognition model, the method further comprises: Acquire a training sample set, the training sample set comprising a first sample set corresponding to a first type of formula and a second sample set corresponding to a second type of formula, the first type of formula comprising at least two spatial structures and / or the trajectory points corresponding to the first type of formula are continuous points; the second type of formula comprising at most one spatial structure; Training an initial recognition model according to the first sample set to obtain a trained first model; Training the first model according to the second sample set to obtain a second model; Adjust the model parameters of the second model, and iteratively train the adjusted second model according to the first sample set and / or the second sample set, and obtain a trained recognition model when the loss value corresponding to the second model is less than a preset threshold.
7. The method according to claim 1, characterized in that The decoder includes at least two decoding layers, and the decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector to obtain a formula recognition result, including: Inputting the high-dimensional feature vector into a decoder in the recognition model, and decoding the high-dimensional feature vector through at least two decoding layers of the decoder; During the decoding process of each decoding layer, the key information in the high-dimensional feature vector is identified and decoded based on the hybrid attention mechanism to obtain the target character at the current moment, until the decoding is completed to obtain the formula recognition result composed of the target characters, and the target characters include mathematical characters and structural characters.
8. The method according to claim 7, characterized in that The hybrid attention mechanism includes one or more of a spatial attention mechanism, a temporal attention mechanism, and an attention guidance mechanism. During the decoding process of each decoding layer, the spatial attention mechanism is used to align mathematical characters in the high-dimensional feature vector, the temporal attention mechanism is used to align structural characters in the high-dimensional feature vector, and the attention guidance mechanism is used to guide the alignment processing of the spatial attention mechanism.
9. The method according to claim 1, characterized in that: The encoder includes a first recurrent network layer and a bidirectional second recurrent network layer.
10. The method according to claim 9, characterized in that The extracting and encoding the features of the trajectory point sequence by the encoder in the recognition model to obtain a high-dimensional feature vector includes: The first recurrent network layer and the bidirectional second recurrent network layer sequentially extract and encode the features of the trajectory point sequence to obtain a high-dimensional feature vector.
11. The method according to claim 9, characterized in that The extracting and encoding the features of the trajectory point sequence by the encoder in the recognition model to obtain a high-dimensional feature vector includes: Performing feature extraction and encoding on the trajectory point sequence through the first recurrent network layer and the bidirectional second recurrent network layer, respectively, to obtain a first high-dimensional feature vector and a second high-dimensional feature vector; The decoder in the recognition model performs information recognition and decoding on the high-dimensional feature vector to obtain a formula recognition result, including: The decoder in the recognition model performs information recognition and decoding on the first high-dimensional feature vector and the second high-dimensional feature vector respectively to obtain a formula recognition result.
12. A formula recognition device, characterized in that: The device comprises: A trajectory point preprocessing module, used to obtain trajectory points input in real time and preprocess the trajectory points to obtain a trajectory point sequence; An encoding module, used for inputting the trajectory point sequence into a pre-trained recognition model, extracting and encoding features of the trajectory point sequence through an encoder in the recognition model, and obtaining a high-dimensional feature vector; A decoding state acquisition submodule, used to acquire the decoding state of each subvector in the high-dimensional feature vector; A first weight value setting submodule, configured to set a first weight value for the subvector if the decoding state of the subvector is undecoded; a second weight value setting submodule, configured to set a second weight value for the subvector if the decoding state of the subvector is decoded, wherein the second weight value is smaller than the first weight value; A weighted summation submodule is used to perform weighted summation on each subvector to obtain an adjusted high-dimensional feature vector; The first formula recognition submodule is used to perform information recognition and decoding on the adjusted high-dimensional feature vector through a decoder in the recognition model to obtain a formula recognition result.
13. A device for formula recognition, characterized in that: The device includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for executing the formula recognition method as described in any one of claims 1 to 11.
14. A machine-readable medium having instructions stored thereon, which, when executed by one or more processors, causes a device to perform the formula recognition method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Knowledge extraction method and system based on memory neural network and device
CN108304911A
Character recognition method based on an attention mechanism and linkage time classification loss
CN109492679A
Formula identification method and device
CN111046751A
Formula identification method and device, electronic equipment and storage medium
CN111738105A
Handwriting recognition method and device, electronic equipment and storage medium
CN112686134A
