A method, device, storage medium and electronic device for text recognition

Abstract

The disclosed embodiments provide a method, device, storage medium and electronic device for text recognition, the method comprising collecting information of a text to be recognized, the information of the text to be recognized at least comprising first image data of the text to be recognized and data of all track points in the text to be recognized; inputting the first image data into a first preset network model to obtain information of a text box; matching the track point data with the information of the text box to obtain a matched text box; inputting the second image data of the matched text box and the corresponding track point data into a second preset network model to obtain a recognition result. The disclosed embodiment is based on a network model for handwriting recognition based on deep learning, and by performing feature fusion on two input data, namely, the image of handwritten text and the track point data, the detection accuracy of the text is improved, and it can adapt to more flexible layout and typesetting, and achieve a higher recognition accuracy of handwritten text.

Description

Technical Field

[0001] The present disclosure relates to the technical field of text processing, and in particular to a text recognition method, device, storage medium and electronic device. Background Art

[0002] The existing handwriting recognition methods can be divided into offline handwriting recognition methods and online handwriting recognition methods. Offline handwriting recognition is based on pictures of handwritten text, and text recognition is based on picture data. It supports the recognition of handwritten Chinese, English, letters, numbers, and common characters in the whole picture text in multiple scenes and any layout. It does not rely on the standardization of stroke order, but the disadvantage is that the recognition accuracy is low, especially for sloppy and blurred fonts; online handwriting recognition is based on the sequence track points collected by the handwriting device. The advantage is that the input data contains stroke order information and the recognition accuracy is high. The disadvantage is that it is difficult to distinguish text lines by relying only on track point data, especially for writing on any layout. Therefore, it is often used for single word or single line handwriting input data recognition.

[0003] In some application scenarios that require handwriting recognition, such as handwriting on an electronic conference whiteboard, all handwritten text content needs to be recognized for document archiving and retrieval. In order to achieve this function, it is necessary to effectively divide the text lines under any layout while taking into account the recognition accuracy. Summary of the invention

[0004] In view of this, the purpose of the embodiments of the present disclosure is to provide a text recognition method, device, storage medium and electronic device to solve the problems in the prior art of low recognition accuracy and difficulty in distinguishing text lines by relying solely on trajectory point data.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for text recognition, which includes the following steps: collecting information of a text to be recognized, the information of the text to be recognized at least including first image data of the text to be recognized and data of all trajectory points in the text to be recognized; inputting the first image data into a first preset network model to obtain information of a text box; matching the data of the trajectory points with the information of the text box to obtain a matched text box; inputting the second image data of the matched text box and the corresponding data of the trajectory points into a second preset network model to obtain a recognition result.

[0006] In some embodiments, the data of the trajectory point includes at least first coordinate information of the trajectory point in a first coordinate system, time information of the trajectory point, and state information of the trajectory point.

[0007] In some embodiments, in the step of inputting the first image data into a first preset network model to obtain information of a text box, the first preset network model includes at least a first feature extraction network, and the first feature extraction network includes a convolutional neural network, a deconvolutional neural network, a feature fusion network, and multiple convolutional layers, including the following steps: converting the first image data into a first feature image with a first resolution through the convolutional neural network; converting the first feature image into a second feature image with a second resolution through the deconvolutional neural network, wherein the second resolution is greater than the first resolution; fusing features of the first feature image with features of the second feature image through the feature fusion network to obtain a third feature image; and inputting the third feature image into multiple convolutional layers to obtain information of the text box.

[0008] In some embodiments, the information of the text box includes at least one of the following: a text score of the text box, position information of the text box, and posture information of the text box.

[0009] In some embodiments, matching the data of the trajectory points with the information of the text box to obtain the matched text box includes the following steps: obtaining a stroke set based on the data of the trajectory points; obtaining a candidate text box for each stroke in the stroke set; and obtaining a text box corresponding to the stroke in the candidate text box.

[0010] In some embodiments, obtaining a candidate text box for each stroke in the stroke set includes the following steps: obtaining a maximum circumscribed rectangular range for each text box in a first coordinate system; determining whether a starting point and / or an end point of each stroke in the stroke set falls within the maximum circumscribed rectangular range, and if so, the corresponding text box is a candidate text box for the stroke.

[0011] In some embodiments, obtaining the text box corresponding to the stroke in the candidate text box includes the following steps: establishing a second coordinate system based on the candidate text box; obtaining second coordinate information of the starting point and / or end point of each stroke in the stroke set in the second coordinate system; determining whether the second coordinate information falls within the range of the candidate text box, and if so, determining that the candidate text box is the text box corresponding to the stroke.

[0012] In some embodiments, the second image data of the matched text box and the corresponding data of the trajectory points are input into a second preset network model to obtain a recognition result, comprising the following steps: inputting the second image data of the matched text box into the second feature extraction network to obtain a first output result, and inputting the data of the trajectory points into a third feature extraction network to obtain a second output result; inputting the first output result and the second output result into a data fusion network to obtain a third output result; inputting the third output result into multiple long short-term memory networks for temporal feature extraction, and obtaining a recognition result through decoding.

[0013] In some embodiments, before inputting the second image data of the matched text box into the second feature extraction network to obtain the first output result and inputting the data of the trajectory point into the third feature extraction network to obtain the second output result, it also includes: formatting the second image data and the data of the trajectory point.

[0014] In some embodiments, the format processing includes the following steps: obtaining second image data in the first image data based on the information of the text box; converting the second image data into image data in a predetermined format; and normalizing and coordinate converting the data of the trajectory points.

[0015] In a first aspect, an embodiment of the present disclosure provides a device for recognizing text, which includes the following parts: an acquisition module, which is used to acquire information of a text to be recognized, wherein the information of the text to be recognized includes at least first image data of the text to be recognized and data of all trajectory points in the text to be recognized; a first acquisition module, which is used to input the first image data into a first preset network model to obtain information of a text box; a second acquisition module, which is used to match the data of the trajectory points with the information of the text box to obtain a matched text box; and a recognition module, which is used to input the second image data of the matched text box and the corresponding data of the trajectory points into a second preset network model to obtain a recognition result.

[0016] In a third aspect, an embodiment of the present disclosure provides a storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the steps of the method described in any one of the above technical solutions are implemented.

[0017] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, comprising at least a memory and a processor, wherein a computer program is stored in the memory, and wherein the processor implements the steps of any one of the methods described in the above technical solutions when executing the computer program in the memory.

[0018] The disclosed embodiments provide a text recognition method that simultaneously fuses two types of input data, namely, images of handwritten text and trajectory point data. The method detects handwritten text lines, uses trajectory points to screen text boxes to eliminate falsely detected text boxes, further divides the text boxes by handwritten trajectory points to correspond the text boxes to the trajectory points, and finally recognizes the handwritten text in each text box. The method uses a network model for handwriting recognition based on deep learning to achieve feature fusion of the two types of input data, further improves the detection accuracy of the text, can adapt to more flexible layout and typeset text, and can achieve a higher recognition accuracy of handwritten text.

[0019] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 A schematic diagram showing a text to be recognized according to an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram showing the steps of a text recognition method according to an embodiment of the present disclosure is shown;

[0023] Figure 3 A schematic diagram showing the steps of a text recognition method according to an embodiment of the present disclosure is shown;

[0024] Figure 4 A schematic diagram showing the structure of a first preset network model in a text recognition method according to an embodiment of the present disclosure is shown;

[0025] Figure 5 A schematic diagram showing the steps of a text recognition method according to an embodiment of the present disclosure is shown;

[0026] Figure 6 A schematic diagram showing the steps of a text recognition method according to an embodiment of the present disclosure is shown;

[0027] Figure 7 A schematic diagram showing the maximum circumscribed rectangular range of a text box in a text recognition method according to an embodiment of the present disclosure;

[0028] Figure 8 A schematic diagram showing the steps of a text recognition method according to an embodiment of the present disclosure is shown;

[0029] Fig. 9 A schematic diagram showing a second preset network model in the text recognition method according to an embodiment of the present disclosure;

[0030] Fig.10 A schematic diagram showing the steps of a text recognition method according to an embodiment of the present disclosure is shown;

[0031] Fig.11 A schematic diagram showing the steps of a text recognition method according to an embodiment of the present disclosure is shown;

[0032] Fig.12 A schematic diagram of a second feature extraction network in a text recognition method according to an embodiment of the present disclosure is shown;

[0033] Fig.13 A schematic diagram showing the structure of a text recognition device according to an embodiment of the present disclosure is shown;

[0034] Fig.14 A schematic structural diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present disclosure clearer, the technical solution of the present disclosure will be clearly and completely described below in conjunction with the drawings of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0036] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0037] In order to keep the following description of the present disclosure clear and concise, the present disclosure omits detailed descriptions of well-known functions and well-known components.

[0038] In a first aspect, an embodiment of the present disclosure provides a method for recognizing text. Figure 1 The application scenario of the text recognition method involved in the embodiment of the present disclosure is shown. The goal expected to be achieved by the embodiment of the present disclosure is to locate and recognize all text information in the user's handwritten text content, so as to facilitate the conversion of the recognized handwritten text content into a printed text layout, thereby realizing the functions of document archiving and retrieval. The text recognition method involved in the embodiment of the present disclosure, such as Figure 2 As shown, it specifically includes the following steps:

[0039] S101, collecting information of a text to be recognized, wherein the information of the text to be recognized at least includes first image data of the text to be recognized and data of all track points in the text to be recognized.

[0040] In this step, the information of the text to be recognized is collected, such as Figure 1 As shown, the text information to be recognized here can be directly derived from any media source that can be collected, such as electronic whiteboards, tablet computers and other electronic devices with handwriting functions, and users can perform handwriting operations through these electronic devices with handwriting functions. It can also come from electronic devices that can process the information of the text to be recognized, such as cameras, etc. The electronic device used for processing here can obtain processed data in various formats such as pictures of the text information to be recognized. Here, the present disclosure does not limit the source of the information of the text to be recognized.

[0041] Furthermore, the information of the text to be recognized obtained by collection at least includes the first image data of the text to be recognized and the data of all track points in the text to be recognized, wherein Figure 1 The first image data of the text to be recognized is shown. In addition, when the user completes handwriting of a text, for example, through an electronic device, the text to be recognized here will be composed of various handwritten strokes, and each of the strokes here is composed of trajectory points.

[0042] During the process of collecting and converting the handwritten text to be recognized, the current image data of the information of the text to be recognized and the data of all handwritten trajectory points in the information of the text to be recognized will be recorded and acquired, wherein, for example, the first image data of the text to be recognized and the data of all trajectory points in the text to be recognized can be acquired through sensors built into the electronic device, and the specific acquisition method can adopt any method in the prior art, which will not be repeated here; here, the first image data will be used as the input parameter of the model subsequently used for text box detection, and the first image data and the trajectory point data will be simultaneously used as the input parameters of the model subsequently used for text recognition.

[0043] Further, as described above, the content of the text to be recognized is composed of the strokes, and the strokes are composed of the trajectory points. Therefore, the data of the trajectory points here at least include the first coordinate information of the trajectory points in the first coordinate system, the time information of the trajectory points, and the state information of the trajectory points. Specifically, the first coordinate system here is a global coordinate system, and the global coordinate system can be established based on the entire text to be recognized. The first coordinate information of the trajectory points in the first coordinate system can be the x-coordinate and y-coordinate of each of the trajectory points in the global coordinate system. The time information of the trajectory points can be the timestamp information formed by each of the trajectory points. The state information of the trajectory points can be information on whether each of the trajectory points belongs to the pen-down state or the pen-up state, etc., wherein the pen-down state can be represented by 0, and the pen-up state can be represented by 1.

[0044] S102: Input the first picture data into a first preset network model to obtain information of a text box.

[0045] After collecting the information of the text to be recognized through the above step S101, the first image data in the text information to be recognized will be used as the input parameters of the model for text box detection, so as to detect and recognize the text box including text content in the first image data. The text box here can be a rectangular text box, which is used to cover all the text content in the first image data. Each of the text boxes can circle characters, words, phrases, short sentences and other texts in the direction of each row / column. The specific type and number of each text box circled can be determined based on the composition of the text content in each row / column, so as to ensure that all the text content in the text to be recognized is circled in different text boxes. Figure 1 For example, a text box can be used to circle the phrase "a snake"; specifically, the first picture data is input into a first preset network model to obtain information of the text box, wherein the first preset network model here at least includes a first feature extraction network, and the first feature extraction network is used to learn and extract image features in the picture data of the text information to be identified, which can be formed based on the network structure of Resnet, including convolutional neural network, deconvolutional neural network, feature fusion network and multiple convolutional layers, such as Figure 3 As shown, the specific steps include:

[0046] S201, converting the first picture data into a first feature picture with a first resolution through the convolutional neural network.

[0047] In this step, the first picture data is first converted into a first feature picture with a first resolution by the convolutional neural network in the first feature extraction network. Specifically, Figure 4As shown, the convolutional neural network here is composed of conv1, pool1, conv2_x, conv3_x, conv4_x, and conv5_x operations in the Resnet structure. In one embodiment, the first picture data as input has an original resolution. When the first picture data is input to the convolutional neural network, the first picture data is converted into a first feature picture through the convolutional neural network, and the resolution of the first feature picture is converted to 1 / 32 of the original resolution of the first picture data.

[0048] S202: Convert the first feature image into a second feature image with a second resolution through the deconvolution neural network, wherein the second resolution is greater than the first resolution.

[0049] In this step, the first feature image is converted into a second feature image with a second resolution by the deconvolution neural network in the first feature extraction network, wherein the second resolution is greater than the first resolution; Figure 4 As shown, the deconvolution neural network here is composed of three deconvolution deconv operations, each deconv operation is composed of a 1X1 convolution kernel and an upsampling operation unit, and the resolution of the feature image will be increased by 2 times after each deconv operation. In this embodiment, after three deconv operations, the first resolution of the first feature image is converted into a second resolution. For example, the first resolution of 1 / 32 of the original resolution of the first image data is converted into a second resolution of 1 / 4 of the original resolution of the first image data, thereby obtaining the second feature image with the second resolution converted by the first feature image.

[0050] S203: Fusing the features of the first feature image and the features of the second feature image through the feature fusion network to obtain a third feature image.

[0051] After the first feature image and the second feature image are obtained respectively through the above steps S201 and S202, in this step, the features of the first feature image and the features of the second feature image are feature fused through the feature fusion network in the first feature extraction network, thereby obtaining a third feature image. The feature fusion network here adopts the FPN algorithm (Feature Pyramid Networks for Object Detection), which fuses the low-level features of the convolution part with the high-level features of the deconvolution part on multiple scale features, that is, the low-level features of the first feature image obtained by the convolution neural network and the high-level features of the second feature image obtained by the deconvolution neural network are feature fused, so that the detailed texture information in the second feature image is retained, and the semantic information in the first feature image is reflected, and multiple resolution scales can also be fused to adapt to the detection of text targets of different sizes, thereby improving the detection effect.

[0052] S204: Input the third feature image into multiple convolutional layers to obtain information of the text box.

[0053] In this step, the third feature image obtained through feature fusion in the above step S203 is input into the multiple convolutional layers to obtain information of all text boxes. The multiple convolutional layers here may include 3 convolutional layers, and the information of the text boxes obtained includes not only the division and quantity of the text boxes, but more importantly, the information of the text boxes includes at least one of the following: the text score of the text box, the position of the text box, and the posture of the text box. The text score of the text box here can be set for each text box according to preset rules, for example, it can be the completeness of the text contained in the text box, etc. The posture of the text box here can be, for example, the tilt angle of the text box, etc.

[0054] S103, matching the data of the track point with the information of the text box to obtain a matched text box.

[0055] After obtaining the information of each text box through the above step S102, in this step, all the track points constituting the strokes generated by the handwriting operation are combined to determine the correspondence between the track points and the text boxes and to filter the text boxes based on the correspondence; specifically, in this step, the data of the track points are matched with the information of the text boxes, so as to filter the text boxes and obtain the matched text boxes, such as Figure 5 As shown, the following steps are included:

[0056] S301, acquiring a stroke set based on the data of the trajectory points.

[0057] In this step, it is first necessary to establish an association between the trajectory points and the strokes that constitute the text content. This requires obtaining a stroke set based on the data of the trajectory points. Specifically, the time information of the trajectory points (such as the timestamp information formed by the trajectory points) and the state information (such as the state of the pen lifting or pen putting down the pen) can be used to determine and divide the strokes. The stroke here refers to an object composed of a series of trajectory points arranged in chronological order from the beginning of pen putting down to the process of pen lifting. Based on the formation rules of the strokes, several of the trajectory points are associated with a formed stroke. Further, the same steps can be used to obtain all the strokes to form a stroke set.

[0058] S302: Obtain a candidate text box for each stroke in the stroke set.

[0059] In order to establish a correspondence between the strokes formed by the trajectory points and the corresponding text boxes, so as to facilitate the determination of the information of the text to be recognized, it is necessary to obtain a candidate text box for each stroke in the stroke set, such as Figure 6 As shown, the specific steps include:

[0060] S401, obtaining a maximum circumscribed rectangular range of each text box in a first coordinate system.

[0061] In this step, the maximum bounding rectangle range of each text box is obtained in the first coordinate system. Specifically, Figure 7 As shown, the first coordinate system here refers to the global coordinate system. Each text box is projected onto the global coordinate system to obtain the maximum enclosing rectangular range of each text box, namely x:[xmin,xmax], y:[ymin,ymax]. The maximum enclosing rectangular range here is the minimum rectangular range of each text box in the first coordinate system.

[0062] S402, determining whether the starting point and / or the ending point of each stroke in the stroke set falls within the maximum circumscribed rectangle, and if so, the corresponding text box is a candidate text box for the stroke.

[0063] After obtaining the maximum circumscribed rectangular range of each text box in the first coordinate system through the above step S401, determine in turn whether the starting point and / or end point of each stroke in the stroke combination falls within the maximum circumscribed rectangular range. If so, the text box corresponding to the maximum circumscribed rectangular range is used as a candidate text box for the stroke. Of course, the candidate text box corresponding to each stroke can be one or more. Further, all the candidate text boxes corresponding to all the strokes in the stroke set are obtained.

[0064] S303: Obtain a text box corresponding to the stroke in the candidate text box.

[0065] After obtaining the candidate text box of each stroke in the stroke set in step S302, it is then specifically determined whether the starting point and / or the end point of the stroke falls within the candidate text box, thereby filtering out the text box corresponding to the stroke from the candidate text box. Figure 8 As shown, the specific steps include:

[0066] S501: Establish a second coordinate system based on the candidate text box.

[0067] In this step, a second coordinate system is established based on the candidate text box to accurately determine the positional relationship between the trajectory point of the stroke and the candidate text box.

[0068] S502, acquiring second coordinate information of a starting point or an end point of each stroke in the stroke set in the second coordinate system;

[0069] In this step, in the second coordinate system established based on the candidate text box, the first coordinate information of the starting point and / or end point of each stroke in the first coordinate system is converted to obtain the second coordinate information of the trajectory point serving as the starting point or end point in the second coordinate system.

[0070] S503: Determine whether the second coordinate information falls within the range of the candidate text box. If yes, determine that the candidate text box is a text box corresponding to the stroke.

[0071] In this step, after obtaining the second coordinate information of the starting point or the end point of the stroke in the second coordinate system, if the position represented by the second coordinate information of this point falls within the range of the candidate text box in the second coordinate system, it is considered that the corresponding stroke is associated with the candidate text box, and thus it is considered that there is a corresponding relationship between the stroke and the candidate text box, and this candidate text box is the text box corresponding to the stroke. Of course, if there is a candidate text box that does not establish a corresponding relationship with any of the strokes, it is considered that the candidate text box is a false detection and will be deleted, thereby achieving the screening of text boxes.

[0072] S104, inputting the second image data of the matched text box and the corresponding data of the trajectory point into a second preset network model to obtain a recognition result.

[0073] After matching the data of the trajectory points with the information of the text box through the above step S103 and obtaining the text box after matching, in this step, the second picture data of the text box after matching and the corresponding data of the trajectory points are input into a second preset network model to obtain an identification result, where the second preset network model is as Fig. 9 shown.

[0074] Further, before inputting the second picture data of the text box after matching and the corresponding data of the trajectory points into the second preset network model, it further includes: performing format processing on the second picture data and the data of the trajectory points, as Fig.10 shown, and the format processing includes the following steps:

[0075] S601, obtaining second picture data in the first picture data based on the information of the text box after matching.

[0076] In this step, first, according to the information of the text box after matching, such as the position information of the text box, the picture of the text box after matching is cropped from the first picture data as the second picture data.

[0077] S602, converting the second picture data into picture data in a predetermined format.

[0078] In this step, assuming that the width of the text box in the second picture data of the text box after matching is w and the height is h, if w>h, that is, the landscape writing format, the second picture data is converted into picture data in a predetermined format through interpolation, for example, converted into picture data with a height h = 32 and w scaled proportionally. If w<h, that is, the portrait writing format, the second picture data is converted into picture data in a predetermined format through interpolation, for example, adjusted to a width w = 32, h scaled proportionally, and then rotated 90 degrees counterclockwise to be horizontal as the picture data for input.

[0079] S603, performing normalization processing and coordinate conversion processing on the data of the trajectory points.

[0080] In this step, the data of the trajectory points corresponding to the text box after matching are subjected to normalization processing and coordinate conversion processing. Specifically, all y coordinates are normalized to the range of 0-1; at the same time, all x coordinates are normalized to the range of 0-1, and then all the trajectory point coordinates are transformed as xnew = y, ynew = 1 - x.

[0081] After completing the format processing of the second image data and the data of the trajectory point, the second image data of the matched text box and the corresponding data of the trajectory point are input into the second preset network model to obtain the recognition result, such as Fig.11 As shown, the following steps are included:

[0082] S701, inputting the second image data of the matched text box into the second feature extraction network to obtain a first output result, and inputting the data of the trajectory point into the third feature extraction network to obtain a second output result;

[0083] In this step, the second image data of the matched text box is input into the second feature extraction network to obtain a first output result, and the data of the trajectory point is input into the third feature extraction network to obtain a second output result, wherein the second feature extraction network is used to extract image features from the second image data of the input text box, such as Fig.12 As shown, for example, the second feature extraction network here may include multiple convolution blocks (Conv Bloc), each convolution block is composed of a conventional convolution layer, a depth-separable convolution layer based on a hole convolution, etc., and this feature extraction network can increase the receptive field of the model and improve the model's ability to judge similar characters in the image data; the third feature extraction network is used to extract sequence information features from the input data of the trajectory points with a handwriting sequence, and the third feature extraction network can be composed of three convolution layers, such as 1X1, 3X3 and 1X1 convolution layers.

[0084] S702, inputting the first output result and the second output result into a data fusion network to obtain a third output result;

[0085] After respectively obtaining the first output result and the second output result through the above step S701, the first output result and the second output result are merged through, for example, a concat module and input into a data fusion network. The data fusion network fuses the image features in the first output result with the sequence features in the second output result to obtain a third output result, so that the third output result contains both image information and stroke sequence information. The data fusion network here can, for example, be composed of three convolutional layers, for example, 1X1, 3X3 and 1X1 respectively.

[0086] S703: Input the third output result into multiple long short-term memory networks to extract temporal features, and obtain the recognition result through decoding.

[0087] After obtaining the third output result containing both the image feature information and the sequence feature information of the trajectory points in the strokes through the above step S703, the third output result is input into multiple long short-term memory networks (LSTM) for temporal feature extraction. For example, the long short-term memory network here can be a bidirectional long short-term memory network, and the number of networks is adjusted according to the text category, quantity and time effect to be recognized. Finally, through the decoding module, the output prediction result is matched with the text category to obtain the final recognition result.

[0088] The disclosed embodiments provide a text recognition method that simultaneously fuses two types of input data, namely, images of handwritten text and trajectory point data. The method detects handwritten text lines, uses trajectory points to screen text boxes to eliminate falsely detected text boxes, further divides the text boxes by handwritten trajectory points to correspond the text boxes to the trajectory points, and finally recognizes the handwritten text in each text box. The method uses a network model for handwriting recognition based on deep learning to achieve feature fusion of the two types of input data, further improves the detection accuracy of the text, can adapt to more flexible layout and typeset text, and can achieve a higher recognition accuracy of handwritten text.

[0089] In a second aspect, an embodiment of the present disclosure provides a text recognition device, Figure 1 The application scenario of the text recognition device involved in the embodiment of the present disclosure is shown. The goal expected to be achieved by the embodiment of the present disclosure is to locate and recognize all text information in the user's handwritten text content, so as to facilitate the conversion of the recognized handwritten text content into a printed text layout, thereby realizing the functions of document archiving and retrieval. The text recognition device involved in the embodiment of the present disclosure, such as Fig.13 As shown, it specifically includes a collection module 10, a first acquisition module 20, a second acquisition module 30 and a recognition module 40, and the functions of the above modules correspond to the mutual coupling between the recognition modules for text in the first embodiment of the present disclosure, specifically:

[0090] The acquisition module 10 is used to acquire information of the text to be recognized, wherein the information of the text to be recognized at least includes first image data of the text to be recognized and data of all track points in the text to be recognized.

[0091] The text information to be identified here can come directly from any media source that can be collected, such as electronic whiteboards, tablet computers and other electronic devices with handwriting functions. Users can perform handwriting operations through these electronic devices with handwriting functions. It can also come from electronic devices that can process the information of the text to be identified, such as cameras, etc. The electronic device used for processing here can obtain processed data in various formats such as pictures of the text information to be identified. Here, the present disclosure does not limit the source of the information of the text to be identified.

[0092] Furthermore, the information of the text to be recognized obtained through collection includes at least the first image data of the text to be recognized and the data of all trajectory points in the text to be recognized. When the user completes the handwriting of a text, for example, through an electronic device, the text to be recognized here will be composed of various handwritten strokes, and each of the strokes here is composed of trajectory points.

[0093] During the process of collecting and converting the handwritten text to be recognized, the current image data of the information of the text to be recognized and the data of all handwritten trajectory points in the information of the text to be recognized will be recorded and acquired, wherein, for example, the first image data of the text to be recognized and the data of all trajectory points in the text to be recognized can be acquired through sensors built into the electronic device, and the specific acquisition method can adopt any method in the prior art, which will not be repeated here; here, the first image data will be used as the input parameter of the model subsequently used for text box detection, and the first image data and the trajectory point data will be simultaneously used as the input parameters of the model subsequently used for text recognition.

[0094] Further, as described above, the content of the text to be recognized is composed of the strokes, and the strokes are composed of the trajectory points. Therefore, the data of the trajectory points here at least include the first coordinate information of the trajectory points in the first coordinate system, the time information of the trajectory points, and the state information of the trajectory points. Specifically, the first coordinate system here is a global coordinate system, and the global coordinate system can be established based on the entire text to be recognized. The first coordinate information of the trajectory points in the first coordinate system can be the x-coordinate and y-coordinate of each of the trajectory points in the global coordinate system. The time information of the trajectory points can be the timestamp information formed by each of the trajectory points. The state information of the trajectory points can be information on whether each of the trajectory points belongs to the pen-down state or the pen-up state, etc., wherein the pen-down state can be represented by 0, and the pen-up state can be represented by 1.

[0095] The first acquisition module 20 is used to input the first picture data into a first preset network model to acquire information of the text box.

[0096] After collecting the information of the text to be recognized through the acquisition module 10, the first image data in the text information to be recognized will be used as the input parameters of the model for text box detection, so as to detect and recognize the text box including the text content in the first image data to ensure that all the contents in the text to be recognized are reflected in the text box; specifically, the first image data is input into the first preset network model to obtain the information of the text box, wherein the first preset network model here includes at least a first feature extraction network, and the first feature extraction network is used to learn and extract image features in the image data of the text information to be recognized, which can be formed based on the network structure of Resnet, for example, including convolutional neural network, deconvolutional neural network, feature fusion network and multiple convolutional layers.

[0097] The second acquisition module 30 is used to match the data of the track point with the information of the text box to obtain the matched text box.

[0098] After the information of each text box is acquired through the first acquisition module 10, the second acquisition module 20 combines all the trajectory points constituting the strokes generated by the handwriting operation to determine the correspondence between the trajectory points and the text boxes and to filter the text boxes based on the correspondence; specifically, the data of the trajectory points are matched with the information of the text boxes, thereby filtering the text boxes and acquiring the matched text boxes.

[0099] The recognition module 40 is used to input the second image data of the matched text box and the corresponding data of the trajectory point into a second preset network model to obtain a recognition result.

[0100] After matching the data of the trajectory point with the information of the text box through the second acquisition module 30 to obtain the matched text box, the second image data of the matched text box and the corresponding data of the trajectory point are input into the second preset network model through the recognition module 40 to obtain the recognition result.

[0101] The disclosed embodiments provide a text recognition method that simultaneously fuses two types of input data, namely, images of handwritten text and trajectory point data. The method detects handwritten text lines, uses trajectory points to screen text boxes to eliminate falsely detected text boxes, further divides the text boxes by handwritten trajectory points to correspond the text boxes to the trajectory points, and finally recognizes the handwritten text in each text box. The method uses a network model for handwriting recognition based on deep learning to achieve feature fusion of the two types of input data, further improves the detection accuracy of the text, can adapt to more flexible layout and typeset text, and can achieve a higher recognition accuracy of handwritten text.

[0102] A third aspect of the present disclosure provides a storage medium, which is a computer-readable medium and stores a computer program. When the computer program is executed by a processor, the method provided by any embodiment of the present disclosure is implemented, including the following steps S11 to S14:

[0103] S11, collecting information of the text to be recognized, wherein the information of the text to be recognized includes at least first image data of the text to be recognized and data of all track points in the text to be recognized;

[0104] S12, inputting the first picture data into a first preset network model to obtain information of a text box;

[0105] S13, matching the data of the track point with the information of the text box to obtain a matched text box.

[0106] S14, inputting the second image data of the matched text box and the corresponding data of the trajectory point into a second preset network model to obtain a recognition result.

[0107] The computer program is executed by the processor. When the first image data is input into the first preset network model to obtain the information of the text box, the first preset network model at least includes a first feature extraction network, and the first feature extraction network includes a convolutional neural network, a deconvolutional neural network, a feature fusion network and multiple convolutional layers. Specifically, the processor performs the following steps: converting the first image data into a first feature image with a first resolution through the convolutional neural network; converting the first feature image into a second feature image with a second resolution through the deconvolutional neural network, wherein the second resolution is greater than the first resolution; fusing the features of the first feature image with the features of the second feature image through the feature fusion network to obtain a third feature image; inputting the third feature image into multiple convolutional layers to obtain the information of the text box.

[0108] The computer program is executed by the processor to match the data of the trajectory point with the information of the text box. When obtaining the matched text box, the processor specifically executes the following steps: obtaining a stroke set based on the data of the trajectory point; obtaining a candidate text box for each stroke in the stroke set; and obtaining a text box corresponding to the stroke in the candidate text box.

[0109] When the computer program is executed by the processor to obtain the candidate text box for each stroke in the stroke set, the processor specifically executes the following steps: obtaining the maximum circumscribed rectangular range of each text box in the first coordinate system; determining whether the starting point and / or end point of each stroke in the stroke set falls within the maximum circumscribed rectangular range, and if so, the corresponding text box is the candidate text box for the stroke.

[0110] When the computer program is executed by the processor to obtain the text box corresponding to the stroke in the candidate text box, the processor specifically executes the following steps: establishing a second coordinate system based on the candidate text box; obtaining second coordinate information of the starting point and / or end point of each stroke in the stroke set in the second coordinate system; determining whether the second coordinate information falls within the range of the candidate text box, and if so, determining that the candidate text box is the text box corresponding to the stroke.

[0111] The computer program is executed by the processor to input the second image data of the matched text box and the corresponding data of the trajectory points into the second preset network model to obtain the recognition result. Specifically, the processor executes the following steps: input the second image data of the matched text box into the second feature extraction network to obtain the first output result, and input the data of the trajectory points into the third feature extraction network to obtain the second output result; input the first output result and the second output result into the data fusion network to obtain the third output result; input the third output result into multiple long short-term memory networks to extract temporal features, and obtain the recognition result through decoding.

[0112] The computer program is executed by the processor to input the second image data of the matched text box into the second feature extraction network to obtain the first output result, and input the data of the trajectory point into the third feature extraction network to obtain the second output result. Specifically, the processor performs the following steps: formatting the second image data and the data of the trajectory point, wherein the formatting processing includes obtaining the second image data in the first image data based on the information of the text box; converting the second image data into image data in a predetermined format; and normalizing and coordinate converting the data of the trajectory point.

[0113] The disclosed embodiments provide a text recognition method that simultaneously fuses two types of input data, namely, images of handwritten text and trajectory point data. The method detects handwritten text lines, uses trajectory points to screen text boxes to eliminate falsely detected text boxes, further divides the text boxes by handwritten trajectory points to correspond the text boxes to the trajectory points, and finally recognizes the handwritten text in each text box. The method uses a network model for handwriting recognition based on deep learning to achieve feature fusion of the two types of input data, further improves the detection accuracy of the text, can adapt to more flexible layout and typeset text, and can achieve a higher recognition accuracy of handwritten text.

[0114] A fourth aspect of the present disclosure provides an electronic device. The structural diagram of the electronic device can be as follows: Fig.13As shown, at least a memory 901 and a processor 902 are included, the memory 901 stores a computer program, and the processor 902 implements the method provided by any embodiment of the present disclosure when executing the computer program on the memory 901. Exemplarily, the steps of the electronic device computer program are as follows S21 to S24:

[0115] S21, collecting information of the text to be recognized, wherein the information of the text to be recognized at least includes first image data of the text to be recognized and data of all track points in the text to be recognized;

[0116] S22, inputting the first picture data into a first preset network model to obtain information of a text box;

[0117] S23, matching the data of the track point with the information of the text box to obtain a matched text box.

[0118] S24, inputting the second image data of the matched text box and the corresponding data of the trajectory point into a second preset network model to obtain a recognition result.

[0119] The processor executes the computer program stored in the execution memory, when the first picture data is input into the first preset network model to obtain the information of the text box, the first preset network model includes at least a first feature extraction network, the first feature extraction network includes a convolutional neural network, a deconvolutional neural network, a feature fusion network and multiple convolutional layers, and also executes the following: converting the first picture data into a first feature picture with a first resolution through the convolutional neural network; converting the first feature picture into a second feature picture with a second resolution through the deconvolutional neural network, wherein the second resolution is greater than the first resolution; fusing the features of the first feature picture with the features of the second feature picture through the feature fusion network to obtain a third feature picture; inputting the third feature picture into multiple convolutional layers to obtain the information of the text box.

[0120] When the processor matches the data of the trajectory point with the information of the text box stored in the execution memory to obtain the matched text box, it also executes the following computer program: obtaining a stroke set based on the data of the trajectory point; obtaining a candidate text box for each stroke in the stroke set; and obtaining a text box corresponding to the stroke in the candidate text box.

[0121] When the processor executes the method of obtaining the candidate text box of each stroke in the stroke set stored in the memory, it also executes the following computer program: obtaining the maximum circumscribed rectangular range of each text box in the first coordinate system; determining whether the starting point and / or end point of each stroke in the stroke set falls within the maximum circumscribed rectangular range, and if so, the corresponding text box is the candidate text box of the stroke.

[0122] When the processor executes the program stored in the memory for obtaining the text box corresponding to the stroke in the candidate text box, it also executes the following computer program: establishing a second coordinate system based on the candidate text box; obtaining second coordinate information of the starting point and / or end point of each stroke in the stroke set in the second coordinate system; determining whether the second coordinate information falls within the range of the candidate text box, and if so, determining that the candidate text box is the text box corresponding to the stroke.

[0123] When the processor inputs the second picture data of the matched text box and the corresponding data of the trajectory points stored in the execution memory into the second preset network model to obtain the recognition result, it also executes the following computer program: inputting the second picture data of the matched text box into the second feature extraction network to obtain the first output result, and inputting the data of the trajectory points into the third feature extraction network to obtain the second output result; inputting the first output result and the second output result into the data fusion network to obtain the third output result; inputting the third output result into multiple long short-term memory networks to extract temporal features, and obtaining the recognition result through decoding.

[0124] Before the processor executes the steps stored in the memory to input the second image data of the matched text box into the second feature extraction network to obtain the first output result and input the data of the trajectory point into the third feature extraction network to obtain the second output result, the processor also executes the following computer program: formatting the second image data and the data of the trajectory point, wherein the formatting processing includes obtaining the second image data in the first image data based on the information of the text box; converting the second image data into image data in a predetermined format; and normalizing and coordinate converting the data of the trajectory point.

[0125] The disclosed embodiments provide a text recognition method that simultaneously fuses two types of input data, namely, images of handwritten text and trajectory point data. The method detects handwritten text lines, uses trajectory points to screen text boxes to eliminate falsely detected text boxes, further divides the text boxes by handwritten trajectory points to correspond the text boxes to the trajectory points, and finally recognizes the handwritten text in each text box. The method uses a network model for handwriting recognition based on deep learning to achieve feature fusion of the two types of input data, further improves the detection accuracy of the text, can adapt to more flexible layout and typeset text, and can achieve a higher recognition accuracy of handwritten text.

[0126] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0127] The storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0128] The storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains at least two Internet Protocol addresses; sends a node evaluation request including at least two Internet Protocol addresses to a node evaluation device, wherein the node evaluation device selects an Internet Protocol address from the at least two Internet Protocol addresses and returns it; receives the Internet Protocol address returned by the node evaluation device; wherein the obtained Internet Protocol address indicates an edge node in a content distribution network.

[0129] Alternatively, the storage medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: receives a node evaluation request including at least two Internet Protocol addresses; selects an Internet Protocol address from the at least two Internet Protocol addresses; and returns the selected Internet Protocol address; wherein the received Internet Protocol address indicates an edge node in a content distribution network.

[0130] Computer program code for performing operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the passenger computer, partially on the passenger computer, as a separate software package, partially on the passenger computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the passenger computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] It should be noted that the storage medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any storage medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0132] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0134] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0135] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0137] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0138] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0139] Multiple embodiments of the present disclosure are described in detail above, but the present disclosure is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concepts of the present disclosure, and these variations and modifications should all fall within the scope of protection required by the present disclosure.

Claims

1. A method for text recognition, characterized in that: The following steps are involved: Collecting information of the text to be recognized, wherein the information of the text to be recognized at least includes first image data of the text to be recognized and data of all track points in the text to be recognized; Inputting the first picture data into a first preset network model to obtain information of a text box; Matching the data of the trajectory point with the information of the text box to obtain a matched text box; The second image data of the matched text box and the corresponding data of the trajectory point are input into a second preset network model to obtain a recognition result.

2. The identification method according to claim 1, characterized in that: The data of the trajectory point at least includes first coordinate information of the trajectory point in a first coordinate system, time information of the trajectory point, and state information of the trajectory point.

3. The method according to claim 1, characterized in that In the step of inputting the first picture data into a first preset network model to obtain information of a text box, the first preset network model at least includes a first feature extraction network, and the first feature extraction network includes a convolutional neural network, a deconvolutional neural network, a feature fusion network, and multiple convolutional layers, including the following steps: Converting the first picture data into a first feature picture with a first resolution by using the convolutional neural network; Converting the first feature image into a second feature image having a second resolution by the deconvolution neural network, wherein the second resolution is greater than the first resolution; The feature fusion network is used to fuse the features of the first feature image with the features of the second feature image to obtain a third feature image; The third feature image is input into multiple convolutional layers to obtain information of the text box.

4. The method according to claim 1, characterized in that The information in the text box includes at least one of the following: The text score of the text box, the position information of the text box, and the posture information of the text box.

5. The method according to claim 1, characterized in that The step of matching the data of the track point with the information of the text box to obtain the matched text box comprises the following steps: Acquire a stroke set based on the data of the trajectory points; Obtaining a candidate text box for each stroke in the stroke set; A text box corresponding to the stroke is obtained in the candidate text box.

6. The method according to claim 5, characterized in that The step of obtaining a candidate text box for each stroke in the stroke set comprises the following steps: Obtaining the maximum bounding rectangle of each text box in the first coordinate system; It is determined whether the starting point and / or the ending point of each stroke in the stroke set falls within the maximum circumscribed rectangle. If so, the corresponding text box is a candidate text box for the stroke.

7. The method according to claim 5, characterized in that The step of obtaining a text box corresponding to the stroke in the candidate text box comprises the following steps: Establishing a second coordinate system based on the candidate text box; Acquire second coordinate information of a starting point and / or an end point of each stroke in the stroke set in the second coordinate system; It is determined whether the second coordinate information falls within the range of the candidate text box, and if so, the candidate text box is determined to be a text box corresponding to the stroke.

8. The method according to claim 1, characterized in that The step of inputting the second image data of the matched text box and the corresponding data of the trajectory point into a second preset network model to obtain a recognition result comprises the following steps: Inputting the second image data of the matched text box into a second feature extraction network to obtain a first output result, and inputting the data of the trajectory point into a third feature extraction network to obtain a second output result; Inputting the first output result and the second output result into a data fusion network to obtain a third output result; The third output result is input into a plurality of long short-term memory networks for temporal feature extraction, and the recognition result is obtained by decoding.

9. The method according to claim 8, characterized in that Before inputting the second image data of the matched text box into the second feature extraction network to obtain the first output result and inputting the data of the trajectory point into the third feature extraction network to obtain the second output result, the method also includes: formatting the second image data and the data of the trajectory point.

10. The method according to claim 9, characterized in that The format processing includes the following steps: Acquire second picture data from the first picture data based on the information in the text box; Converting the second picture data into picture data in a predetermined format; The data of the trajectory points are normalized and coordinate transformed.

11. A text recognition device, characterized in that: Includes the following sections: A collection module, which is used to collect information of the text to be recognized, wherein the information of the text to be recognized at least includes first image data of the text to be recognized and data of all track points in the text to be recognized; A first acquisition module, which is used to input the first picture data into a first preset network model to obtain information of a text box; A second acquisition module, which is used to match the data of the trajectory point with the information of the text box to obtain a matched text box; The recognition module is used to input the second image data of the matched text box and the corresponding data of the trajectory point into a second preset network model to obtain a recognition result.

12. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

13. An electronic device comprising at least a memory and a processor, wherein a computer program is stored in the memory, wherein: The processor implements the steps of the method of any one of claims 1 to 10 when executing the computer program on the memory.

Images