Written character recognition method, device, terminal equipment and storage medium

Abstract

The present application discloses a method for recognizing written characters, a device for recognizing written characters, a terminal device, and a readable storage medium. The method for recognizing written characters includes: obtaining writing trajectory data, the writing trajectory data including trajectory point coordinates and corresponding writing time information, and when the writing trajectory data does not belong to a new file, performing text line segmentation according to the trajectory point coordinates and the writing time information to obtain text lines, and then merging the writing trajectory data of the text lines and the corresponding historical writing trajectory data according to the trajectory point coordinates to obtain merged trajectory data, and then saving the merged trajectory data after line segmentation, and finally using a text recognition model to recognize each line of merged trajectory data to obtain a recognition result. The present application recognizes each line of text by segmenting the writing trajectory data into multiple lines of text when it is determined that the writing trajectory data is not a new file. In this way, the recognition of the entire writing trajectory data can be completed, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to image recognition technology, and in particular to a written text recognition method, a written text recognition device, a terminal device and a storage medium. Background Art

[0002] When using a conference machine, users can record meeting content, minutes, or training content by handwriting in the corresponding application. These written contents can be saved in the form of files on the local storage, making it easier for users to consult them later. When there are too many files, it is difficult for users to find the files they want. In related technologies, written text recognition technology can be used to quickly locate the desired files. However, current written text recognition technology can only recognize single words or single lines of content, and the recognition effect is limited. Summary of the invention

[0003] The present application provides a written text recognition method for use in a written text recognition device, the written text recognition method comprising:

[0004] Acquire writing trajectory data, wherein the writing trajectory data includes trajectory point coordinates and corresponding writing time information;

[0005] In the case that the writing trajectory data does not belong to a new file, text line segmentation is performed according to the trajectory point coordinates and the writing time information to obtain text lines;

[0006] Merging the writing trajectory data of the text line and the corresponding historical writing trajectory data according to the trajectory point coordinates to obtain merged trajectory data;

[0007] Saving the merged trajectory data after branching; and

[0008] The text recognition model is used to recognize each line of the merged trajectory data to obtain a recognition result.

[0009] In some embodiments, the written word recognition method further comprises:

[0010] In the case where the writing trajectory data belongs to a new file, performing text line segmentation according to the trajectory point coordinates and the writing time information to obtain new text lines;

[0011] Saving the writing track data of the text line after line division;

[0012] The text recognition model is used to recognize each line of the writing trajectory data to obtain a recognition result.

[0013] In some implementations, the segmenting of text lines according to the track point coordinates and the writing time information to obtain text lines includes:

[0014] Dividing the writing strokes according to the pen-down state and the pen-lifting state of the writing trajectory data;

[0015] Determine the writing interval time between each of the writing strokes according to the writing time information;

[0016] Dividing the writing strokes into blocks according to the writing interval time to obtain a first text block;

[0017] Divide the first text block into second text blocks according to a first distance threshold, where the first distance threshold is determined according to the first text block;

[0018] Divide the second text block into third text blocks according to a second distance threshold, wherein the second distance threshold is determined according to the second text block;

[0019] The third text block is merged according to the overlap rate of the trajectory point coordinates in the font height direction to obtain the text line.

[0020] In some implementations, merging the writing trajectory data of the text line with corresponding historical writing trajectory data according to the trajectory point coordinates to obtain merged trajectory data includes:

[0021] After deleting the corresponding historical writing track data according to the eraser track, the text line is divided into blocks to obtain the historical writing text blocks;

[0022] The historical text blocks and the text lines are merged according to the center point coordinates of each line and the text block sequence to obtain the merged trajectory data.

[0023] In some embodiments, obtaining writing trajectory data includes:

[0024] A handwriting file uploaded by a user by calling a handwriting recognition service at a terminal is received to obtain the writing trajectory data.

[0025] In some embodiments, the handwritten character recognition method comprises:

[0026] The recognition result is returned to the terminal.

[0027] In some embodiments, the written word recognition method further comprises:

[0028] Building deep learning models;

[0029] Acquire training trajectory data, where the training trajectory data is the writing trajectory data corresponding to each line of text input by the user;

[0030] The deep learning model is trained using the training trajectory data to obtain the text recognition model.

[0031] In certain embodiments, establishing a deep learning model comprises:

[0032] The deep learning model is established based on a time recursive neural network, and the deep learning model includes a multi-layer long short-term memory neural network.

[0033] The present application also provides a written text recognition device, comprising:

[0034] An acquisition module, used to acquire writing trajectory data, wherein the writing trajectory data includes trajectory point coordinates and corresponding writing time information;

[0035] A segmentation module, used for segmenting text lines according to the track point coordinates and the writing time information to obtain text lines when the writing track data does not belong to a new file;

[0036] A merging module, used for merging the writing trajectory data of the text line and the corresponding historical writing trajectory data according to the trajectory point coordinates to obtain merged trajectory data;

[0037] A saving module, used for saving the merged trajectory data after branching; and

[0038] The recognition module is used to use a text recognition model to recognize each line of the combined trajectory data to obtain a recognition result.

[0039] The present application also provides a terminal device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, any one of the above-mentioned written text recognition methods is implemented.

[0040] The present application also provides a non-volatile computer-readable storage medium containing a computer program. When the computer program is executed by a processor, the processor executes any one of the above-mentioned written text recognition methods.

[0041] In the written text recognition method, written text recognition device, terminal device and computer-readable storage medium of the implementation mode of the present application, the writing time information of the uploaded writing trajectory data is judged to identify whether the writing trajectory data is a new file, and when the writing trajectory data does not belong to a new file, the writing trajectory data is segmented into multiple text lines according to the trajectory point coordinates and writing time information in the writing trajectory data, and each text line is then recognized. In this way, the recognition of the entire writing trajectory data can be completed, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0043] Figure 1 It is a flowchart of a written text recognition method of certain embodiments of the present application.

[0044] Figure 2 It is a module schematic diagram of a written text recognition device in certain embodiments of the present application.

[0045] Figure 3-4 It is a flowchart of a written text recognition method of certain embodiments of the present application.

[0046] Figure 5 It is a schematic diagram of a line-by-line processing scenario of a written text recognition method in certain embodiments of the present application.

[0047] Figure 6 A schematic flowchart of a method for handwritten text recognition in certain embodiments of the present application.

[0048] Figure 7 It is a schematic diagram of a scene in which recognition processing is performed based on merged trajectory data in the related art.

[0049] Figure 8 It is a schematic diagram of a scene identified based on the merged trajectory data in certain embodiments of the present application.

[0050] Fig. 9 A flowchart of a method for handwritten text recognition in certain embodiments of the present application.

[0051] Fig.10 It is a schematic diagram of the overall framework of LSMT in certain implementation modes of the present application.

[0052] Fig.11 It is a module diagram of a deep learning model of certain embodiments of the present application. DETAILED DESCRIPTION

[0053] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0054] When using the all-in-one conference machine, users will write meeting content, minutes or training blackboards in the app, and these contents will be saved in the form of files on the local storage, which is convenient for users to check them later. However, when there are too many files, how to quickly find the content that users want becomes an urgent problem to be solved.

[0055] In the related technology, the handwriting recognition function can be used to quickly find the content that the user wants. After the user writes the content, click on the handwriting recognition, and the recognized content can be saved locally as a label of the meeting file. The user can recall the content of the meeting at that time and quickly and conveniently retrieve the file he wants to view by keyword.

[0056] Handwriting recognition technology refers to the process of converting the ordered trajectory information (including two-dimensional coordinates and possibly one-dimensional writing time information) generated when writing text on a touch-screen handwriting device into text. Handwriting recognition is one of the most natural and convenient means of human-computer interaction and can be used in application fields such as office automation, machine translation, and e-commerce. At present, handwriting recognition technology is mainly used in single-word recognition or single-line recognition of input methods. There are relatively few applications for the recognition of trajectory points of the entire text, which limits the application scope of first-write recognition technology.

[0057] In view of this, please refer to Figure 1 The present application embodiment provides a written text recognition method for use in a written text recognition device. The written text recognition method comprises the following steps:

[0058] 01. Obtain writing trajectory data, which includes trajectory point coordinates and corresponding writing time information;

[0059] 02. When the writing trajectory data does not belong to a new file, the text lines are segmented according to the trajectory point coordinates and the writing time information to obtain the text lines;

[0060] 03. Merge the writing trajectory data of the text line and the corresponding historical writing trajectory data according to the trajectory point coordinates to obtain merged trajectory data;

[0061] 04, save the merged trajectory data after branching; and

[0062] 05. Use the text recognition model to recognize each line of merged trajectory data to obtain the recognition result.

[0063] Please read further Figure 2 The embodiment of the present application provides a written character recognition device 10. The written character recognition device 10 includes an acquisition module 110, a segmentation module 120, a merging module 130, a storage module 140 and a recognition module 150.

[0064] Step 01 can be implemented by the acquisition module 110, step 02 can be implemented by the segmentation module 120, step 03 can be implemented by the merging module 130, step 04 can be implemented by the saving module 140, and step 05 can be implemented by the recognition module 150. In other words, the acquisition module 110 can be used to obtain writing trajectory data, and the writing trajectory data includes trajectory point coordinates and corresponding writing time information. The segmentation module 120 can be used to segment text lines according to the trajectory point coordinates and writing time information to obtain text lines when the trajectory data does not belong to a new file. The merging module 130 can be used to merge the trajectory data of the text line and the corresponding historical writing trajectory data according to the trajectory point coordinates to obtain merged trajectory data. The saving module 140 can be used to save the merged trajectory data after line segmentation, and the recognition module 150 can be used to use the text recognition model to recognize each line of merged trajectory data separately to obtain a recognition result.

[0065] The present application also provides a terminal device, including a processor, which can be used to obtain writing trajectory data, the writing trajectory data including trajectory point coordinates and corresponding writing time information, and when the trajectory data does not belong to a new file, the text line is segmented according to the trajectory point coordinates and the writing time information to obtain text lines. The processor can also be used to merge the trajectory data of the text line and the corresponding historical writing trajectory data according to the trajectory point coordinates to obtain merged trajectory data, save the merged trajectory data after line segmentation, and use the text recognition model to recognize each line of the merged trajectory data to obtain a recognition result.

[0066] In the written text recognition method, written text recognition device 10 and terminal device of these embodiments, the writing time information of the uploaded writing trajectory data is judged to identify whether the writing trajectory data is a new file. If the writing trajectory data does not belong to a new file, the writing trajectory data is segmented into multiple text lines according to the trajectory point coordinates and writing time information in the writing trajectory data, and each text line is then recognized. In this way, the recognition of the entire writing trajectory data can be completed, thereby improving the user experience.

[0067] In some embodiments, the terminal device may be an electronic device such as a mobile phone, a tablet, or an all-in-one conference machine that can input text by handwriting. This embodiment is described by taking the terminal device being an all-in-one conference machine as an example, that is, the written text recognition method and the written text recognition device 100 are applied to but not limited to the all-in-one conference machine. The written text recognition device 100 may be hardware or software pre-installed on the all-in-one conference machine, and the written text recognition method may be executed when the machine is started and run on the all-in-one conference machine. For example, the written text recognition device 100 may be a low-level software code segment of the all-in-one conference machine or a part of the operating system.

[0068] It should be noted that the all-in-one conference machine is an intelligent interactive touch all-in-one machine used in conference offices, training and remote conferencing scenarios for large and medium-sized innovative and high-tech enterprises. It integrates computers, projectors, TVs, electronic whiteboards, advertising machines, speakers and other functions into one, and can be used for a variety of purposes.

[0069] In some embodiments, the handwriting recognition device 100 may be discrete components assembled in a certain manner to have the aforementioned functions, or a chip having the aforementioned functions in the form of an integrated circuit, or a computer software code segment that enables the computer to have the aforementioned functions when running on the computer.

[0070] In some embodiments, as hardware, the handwriting recognition device 100 can be independent or installed as an additional peripheral component to a computer or computer system. The handwriting recognition device 100 can also be integrated into a computer or computer system. For example, when the handwriting recognition device 100 is part of a terminal device, the handwriting recognition device 100 can be integrated into a processor.

[0071] The terminal device of this embodiment includes a touch device, and the user can input handwritten trajectory points through the touch device. After the touch device receives the trajectory points input by the user, it can save the x, y coordinate values ​​of each trajectory point, the timestamp of each trajectory point, and the status of each trajectory point (pen down or pen up), thereby obtaining writing trajectory data.

[0072] The processor may be a cloud server and connected to the touch device network, so as to obtain writing trajectory data from the touch device. That is, the written text recognition method of the present application can be implemented by a cloud server. It can be understood that the cloud server is based on a large-scale distributed computing system and can process massive and fast data. In this way, the written text recognition method can be implemented conveniently and quickly.

[0073] The cloud server may include a database, and the database may store historical writing trajectory data. The historical writing trajectory is saved in the form of text pages. A new file refers to a writing trajectory data whose content does not exist in the historical writing trajectory data in the database, and a non-new file refers to a writing trajectory data whose at least part of the content is the same as the content of the historical writing trajectory data in the database. For example, if the current writing trajectory data is the writing trajectory data generated by the user after partially modifying the historical writing trajectory data (deleting or adding some content), it means that the current writing trajectory data is a non-new file.

[0074] The writing track data may include a writing track and an eraser track, wherein the writing track is used to generate writing content, and the eraser track is used to remove writing content, and the writing track data are all saved in chronological order.

[0075] The writing trajectory data may be a handwritten file uploaded by the user when the terminal (touch device) calls the handwriting recognition service. That is, the processor may obtain the writing trajectory data uploaded by the user when the terminal calls the handwriting recognition service from the terminal.

[0076] The text recognition model is a neural network model (NN), which is a complex network system formed by a large number of simple processing units (called neurons) widely connected to each other. It reflects many basic characteristics of human brain functions and is a highly complex nonlinear dynamic learning system. Neural networks have large-scale parallel, distributed storage and processing, self-organization, self-adaptation and self-learning capabilities, and are particularly suitable for processing imprecise and fuzzy information processing problems that require considering many factors and conditions at the same time.

[0077] Specifically, when the processor obtains the writing trajectory data, it traverses the historical writing trajectory data stored in the database to judge whether the writing trajectory data belongs to a new file, and when the writing trajectory data contains the same content as the historical writing trajectory data in the database, it is determined that the writing trajectory data does not belong to a new file, and the writing trajectory data is segmented according to the trajectory point coordinates and writing time information to obtain the writing trajectory data of multiple text lines. Then, according to the trajectory point coordinates of the writing trajectory data, the writing trajectory data of the text line and the corresponding historical writing trajectory data are merged to obtain the merged trajectory data, and the merged trajectory data is saved to the position corresponding to the original historical writing trajectory.

[0078] Furthermore, the processor can only obtain the content in the writing trajectory data that is different from the historical writing trajectory data (i.e., the modified content part), thereby reducing the network transmission time of the writing trajectory data and improving efficiency. When merging the writing trajectory data of the text line and the corresponding historical writing trajectory data according to the trajectory point coordinates of the writing trajectory data to obtain the merged trajectory data, only the content in the writing trajectory data that is different from the historical writing trajectory data (i.e., the modified content part) can be merged with the historical writing trajectory data, thereby reducing the amount of data processing.

[0079] Furthermore, after obtaining the merged trajectory data, the processor may input each line of the merged trajectory data into the text recognition model, so that the text recognition model recognizes the merged trajectory data respectively, outputs the recognition result, and returns the recognition result to the terminal for display, so that the user can obtain the recognition result.

[0080] In addition, when the touch device receives the writing trajectory data input by the user, each writing trajectory data can be independently numbered, and a record point can be made for the current writing trajectory data each time, so that only the newly added writing trajectory data after the last record point can be uploaded during the next recognition. When the processor obtains the writing trajectory data, it can first process the trajectory point, then merge and save the result with the historical record in the database according to the number of the writing trajectory data, and finally recognize it through the text recognition model and return the result.

[0081] In this way, the entire writing trajectory data can be recognized, which improves the user experience. Moreover, when the writing trajectory data does not belong to a new file, it is merged with the historical writing trajectory data, which effectively reduces the amount of data storage.

[0082] See also Figure 3 In some embodiments, after step 01, the written text recognition method further includes:

[0083] 06. When the writing trajectory data belongs to a new file, the text lines are segmented according to the trajectory point coordinates and the writing time information to obtain new text lines;

[0084] 07, save the writing track data of the text line after the line break;

[0085] 08. Use the text recognition model to recognize each line of writing trajectory data and obtain the recognition result.

[0086] Please read further Figure 2 In some implementations, step 06 can be implemented by the segmentation module 120, step 07 can be implemented by the storage module 140, and step 08 can be implemented by the recognition module 150. In other words, the segmentation module 120 can be used to segment text lines to obtain new text lines according to the coordinates of the trajectory points and the writing time information when the trajectory data belongs to a new file, the storage module 140 can be used to save the trajectory data of the text line after the line division, and the recognition module 150 can be used to use the text recognition model to recognize each line of trajectory data to obtain a recognition result.

[0087] In some embodiments, the processor can be used to segment the text lines according to the trajectory point coordinates and writing time information to obtain new text lines when the trajectory data belongs to a new file, save the trajectory data of the text lines after segmentation, and use the text recognition model to recognize each line of trajectory data separately to obtain recognition results.

[0088] It can be understood that when the writing trajectory data belongs to a new file, it means that the writing trajectory data does not contain the same content as the historical writing trajectory data in the database, then a space can be allocated in the database to store the content of the writing trajectory data.

[0089] Specifically, the historical writing trajectory data stored in the database is traversed to judge the acquired writing trajectory data, and when it is confirmed that the writing trajectory data does not contain the same content as the historical writing trajectory data in the database, the writing trajectory data is directly segmented into text lines according to the trajectory point coordinates and the writing time information to obtain the writing trajectory data of the text line after segmentation, and then the writing trajectory data of the text line is saved in the database, and the text recognition model is used to recognize each line of writing trajectory data separately to obtain the recognition result, and then the recognition result is returned to the terminal device for display, so that the user can easily obtain the recognition result.

[0090] Please combine Figure 4 In some embodiments, step 02 includes the sub-steps of:

[0091] 021, dividing the writing strokes according to the pen-down state and the pen-lifting state of the writing trajectory data;

[0092] 022, determining the writing interval time between each writing stroke according to the writing time information;

[0093] 023, dividing the writing strokes into blocks according to the writing interval time to obtain a first text block;

[0094] 024, dividing the first text block into blocks according to a first distance threshold to obtain a second text block, wherein the first distance threshold is determined according to the first text block;

[0095] 025, dividing the second text block into blocks according to a second distance threshold to obtain a third text block, wherein the second distance threshold is determined according to the second text block;

[0096] 026, merging the third text block according to the overlap rate of the trajectory point coordinates in the font height direction to obtain a text line.

[0097] Please read further Figure 2 In some embodiments, sub-steps 021-026 can be implemented by the segmentation module 120. In other words, the segmentation module 120 can be used to divide the writing strokes according to the pen-down state and the pen-lifting state of the writing trajectory data, and determine the writing interval time between each writing stroke according to the writing time information, and divide the writing strokes into blocks according to the writing interval time to obtain a first text block. The segmentation module 120 can also be used to divide the first text block into blocks according to a first distance threshold to obtain a second text block, the first distance threshold is determined according to the first text block, and divide the second text block into blocks according to a second distance threshold to obtain a third text block, the second distance threshold is determined according to the second text block, and then merge the third text block according to the overlap rate of the trajectory point coordinates in the font height direction to obtain a text line.

[0098] In certain embodiments, the processor is used to divide the writing strokes according to the pen-down state and the pen-lifting state of the writing trajectory data, and determine the writing interval time between each writing stroke according to the writing time information, and divide the writing strokes into blocks according to the writing interval time to obtain a first text block. The processor can also be used to divide the first text block into blocks according to a first distance threshold to obtain a second text block, the first distance threshold is determined based on the first text block, and divide the second text block into blocks according to a second distance threshold to obtain a third text block, the second distance threshold is determined based on the second text block, and then merge the third text block according to the overlap rate of the trajectory point coordinates in the font height direction to obtain a text line.

[0099] Please combine Figure 5 Specifically, the trajectory points are divided into strokes according to the pen-dropping and pen-lifting states of each pen-lifting point, the time difference between all strokes is counted, and a longer writing interval time thr1 (for example, 10 seconds) is pre-set based on experience according to different usage scenarios, and then all strokes are processed in blocks according to the writing interval time thr1, so as to obtain the first text block stroke_block1. That is, when the time interval between two characters exceeds 10 seconds, they are considered to be different text lines, and thus they are processed in blocks to obtain the first text block stroke_block1.

[0100] Furthermore, three points (starting point, midpoint, and end point) are taken for each stroke, and the distances between the three points are calculated and added as the estimated length of each stroke. Then, the average length of all strokes is calculated, and 2.5 times of the average length is taken as the first distance threshold thr2. For each stroke_block1_i in the first text block stroke_block1, the distance dis 1 between the starting point of the next stroke and the end point of the previous stroke is calculated. When the distance dis 1 is greater than the first distance threshold thr2, the blocks are processed, and finally the second text block stroke_block2 is obtained.

[0101] Furthermore, considering the difference in stroke length between large characters and small characters, the average length of the strokes in each second text block stroke_block2 is calculated, and 1.5 times of the average length is used as the second distance threshold thr3. For each second text block stroke_block2_i in the second text block stroke_block2, the x-axis distance dis_2 between the starting point of the next stroke and the end point of the previous stroke is calculated. When the distance dis_2 is greater than the second distance threshold thr3, the blocks are processed, and finally the third text block stroke_block3 is obtained.

[0102] Since the third text block stroke_block3 has a probability of separating texts that are originally in the same line (such as the second line in the third text block stroke_block3), the third text block stroke_block3 needs to be merged at the end. Specifically, the overlap ratio of the y coordinates of the current third text block stroke_block3_i+1 and the previous third text block stroke_block3_i is calculated. When the overlap ratio is greater than 0.5, the two parts are merged, and so on. Finally, the text line is obtained.

[0103] Please combine Figure 6 In some embodiments, step 03 includes:

[0104] 031, after deleting the corresponding historical writing track data according to the eraser track, divide the text lines into blocks to obtain historical writing text blocks;

[0105] 032, merge the historical text blocks and text lines according to the center point coordinates of each line and the order of the text blocks to obtain the merged trajectory data.

[0106] In some embodiments, sub-steps 031 and 032 can be implemented by the merging module 130, or in other words, the merging module 130 can be used to segment the text lines into blocks after deleting the corresponding historical writing trajectory data according to the eraser trajectory to obtain historical writing text blocks, and merge the historical text blocks and text lines according to the center point coordinates of each line and the text block order to obtain merged trajectory data.

[0107] In some embodiments, the processor can be used to segment the text lines into historical writing text blocks after deleting the corresponding historical writing trajectory data according to the eraser trajectory, and merge the historical text blocks and text lines according to the center point coordinates of each line and the text block order to obtain merged trajectory data.

[0108] Please combine Figure 7 It can be understood that after the user obtains the historical writing trajectory data, he makes modifications, erases the text in the first few lines and supplements the handwritten text. Since the handwriting trajectory is recorded in chronological order, usually, in the result obtained by calling the handwriting recognition algorithm, the recognition result of the supplemented handwritten text will appear at the end of the entire document.

[0109] Please combine Figure 8 Therefore, the present application divides the text lines into blocks after deleting the corresponding historical writing tracks from the eraser tracks to obtain historical writing text blocks, and then determines whether the historical writing text blocks and the text lines are in the same line according to the vertical coordinate of the center point of each line, and determines the order of the text blocks in the same line according to the horizontal coordinate of the center point, and merges the text lines and historical text blocks to obtain merged track data. In this way, accurate recognition results can be obtained through the text recognition model pair later.

[0110] Please combine Fig. 9 In some embodiments, before step 01, the written text recognition method further includes the steps of:

[0111] 001, build a deep learning model;

[0112] 002, obtaining training trajectory data, where the training trajectory data is the writing trajectory data corresponding to each line of text input by the user;

[0113] 003. Use the training trajectory data to train the deep learning model to obtain a text recognition model.

[0114] Please further combine Figure 2 In some embodiments, the written text recognition device 10 further includes a construction module 160 and a training module 170 .

[0115] Step 001 can be implemented by the construction module 160, step 002 can be implemented by the acquisition module 110, and step 003 can be implemented by the training module 170. In other words, the construction module 160 can be used to establish a deep learning model; the acquisition module 110 can also be used to obtain training trajectory data, which is the writing trajectory data corresponding to each line of text input by the user, and the training module 170 can be used to train the deep learning model using the training trajectory data to obtain a text recognition model.

[0116] In some embodiments, the processor is also used to establish a deep learning model and obtain training trajectory data, where the training trajectory data is writing trajectory data corresponding to each line of text input by the user, and use the training trajectory data to train the deep learning model to obtain a text recognition model.

[0117] In this embodiment, the deep learning model can be established based on a time recursive neural network (RNN), and the deep learning model includes a multi-layer long short-term memory neural network (LSTM).

[0118] Those skilled in the art can understand that a time recurrent neural network is a time recurrent neural network, an artificial neural network (ANN) with a tree-like hierarchical structure and network nodes recursively process input information in the order of their connection. It is one of the deep learning algorithms.

[0119] Long short-term memory neural network is a time recurrent neural network, which is specially designed to solve the long-term dependency problem of general RNN (recurrent neural network). All RNNs have a chain form of repeated neural network modules. Due to its design characteristics, LSTM is very suitable for modeling time series data, such as text data.

[0120] Please combine Fig.10 , LSTM is composed of the input word X at time t t , cell state C t , temporary cell state Hidden state h t , forget gate f t , Memory Gate t , output gate o t The calculation process of LSTM can be summarized as follows: by forgetting the information in the cell state and memorizing new information, the information useful for subsequent moment calculations can be transmitted, while the useless information is discarded, and the hidden state h is output at each time step. t , where forgetting, memory and output are determined by the hidden state h at the previous moment t-1 and the current input X t The calculated forget gate f t , Memory Gate t , output gate o t to control.

[0121] LSTM can be a bidirectional long short-term memory neural network (Bi-directional Long Short-Term Memory, BiLSTM), and BiLSTM is composed of a forward LSTM and a backward LSTM.

[0122] See also Fig.11 In this embodiment, the deep learning model is composed of three consecutive BiLSTMs and two fully connected layers. Among them, the two fully connected layers are located after the BiLSTM, and the fully connected layers are used to classify the data processed by the Bi-LSTM. The three BiLSTMs contain 256, 256 and 512 computing units respectively, and the two fully connected layers contain 512 and 3816 neurons (computing nodes) respectively, among which 3816 is the final classification category, including 3755 Chinese first-level character libraries, 26 English letters, 10 Arabic numerals and 25 common punctuation marks. In addition, after each LSTM, a Dropout layer with a probability of 0.5 is also used, that is, some connections are randomly deleted, thereby reducing model overfitting and accelerating model convergence.

[0123] When the writing trajectory data corresponding to each line of text input by the user is obtained, the writing trajectory data can be input into the deep learning model, and the deep learning model can be continuously trained to obtain a text recognition model.

[0124] The terminal device of the implementation mode of the present application also includes a memory, and the memory stores a computer program. When the computer program is processed by the processor, the written text recognition method as described in any of the above implementation modes is implemented.

[0125] The present application provides a non-volatile computer-readable storage medium containing a computer program. When the computer program is executed by a processor, the processor executes the above-mentioned written text recognition method.

[0126] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0127] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0129] In addition, the functional modules in the various embodiments of the present application may be integrated into a combined module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0130] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for recognizing written characters, characterized in that: include: Acquire writing trajectory data, wherein the writing trajectory data includes trajectory point coordinates and corresponding writing time information; In the case that the writing trajectory data does not belong to a new file, text line segmentation is performed according to the trajectory point coordinates and the writing time information to obtain text lines; Merging the writing trajectory data of the text line and the corresponding historical writing trajectory data according to the trajectory point coordinates to obtain merged trajectory data; Saving the merged trajectory data after branching; and Using a text recognition model to recognize each line of the merged trajectory data to obtain a recognition result; The step of performing text line segmentation according to the track point coordinates and the writing time information to obtain text lines includes: Dividing the writing strokes according to the pen-down state and the pen-lifting state of the writing trajectory data; Determine the writing interval time between each of the writing strokes according to the writing time information; Dividing the writing strokes into blocks according to the writing interval time to obtain a first text block; Divide the first text block into second text blocks according to a first distance threshold, where the first distance threshold is determined according to the first text block; Divide the second text block into third text blocks according to a second distance threshold, wherein the second distance threshold is determined according to the second text block; The third text block is merged according to the overlap rate of the trajectory point coordinates in the font height direction to obtain the text line.

2. The method for recognizing written characters according to claim 1, wherein: The written word recognition method also includes: In the case where the writing trajectory data belongs to a new file, performing text line segmentation according to the trajectory point coordinates and the writing time information to obtain new text lines; Saving the writing track data of the text line after line division; The text recognition model is used to recognize each line of the writing trajectory data to obtain a recognition result.

3. The method for recognizing written characters according to claim 1, wherein: The step of merging the writing trajectory data of the text line and the corresponding historical writing trajectory data according to the trajectory point coordinates to obtain merged trajectory data includes: After deleting the corresponding historical writing track data according to the eraser track, the text line is divided into blocks to obtain the historical writing text blocks; The historically written text blocks and the text lines are merged according to the center point coordinates of each line and the order of the text blocks to obtain the merged trajectory data.

4. The method for recognizing written characters according to claim 1, wherein: The obtaining of writing trajectory data comprises: A handwriting file uploaded by a user by calling a handwriting recognition service at a terminal is received to obtain the writing trajectory data.

5. The method for recognizing written characters according to claim 4, characterized in that: The written word recognition method comprises: The recognition result is returned to the terminal.

6. The method for recognizing written characters according to claim 1, wherein: The written word recognition method also includes: Building deep learning models; Acquire training trajectory data, where the training trajectory data is the writing trajectory data corresponding to each line of text input by the user; The deep learning model is trained using the training trajectory data to obtain the text recognition model.

7. The method for recognizing written characters according to claim 6, characterized in that: The establishing of the deep learning model comprises: The deep learning model is established based on a time recursive neural network, and the deep learning model includes a multi-layer long short-term memory neural network.

8. A written character recognition device, characterized in that: include: An acquisition module, used to acquire writing trajectory data, wherein the writing trajectory data includes trajectory point coordinates and corresponding writing time information; A segmentation module, used for segmenting text lines according to the track point coordinates and the writing time information to obtain text lines when the writing track data does not belong to a new file; A merging module, used for merging the writing trajectory data of the text line and the corresponding historical writing trajectory data according to the trajectory point coordinates to obtain merged trajectory data; A saving module, used for saving the merged trajectory data after branching; A recognition module, used to use a text recognition model to recognize each line of the merged trajectory data to obtain a recognition result; The step of performing text line segmentation according to the track point coordinates and the writing time information to obtain text lines includes: Dividing the writing strokes according to the pen-down state and the pen-lifting state of the writing trajectory data; Determine the writing interval time between each of the writing strokes according to the writing time information; Dividing the writing strokes into blocks according to the writing interval time to obtain a first text block; Divide the first text block into second text blocks according to a first distance threshold, where the first distance threshold is determined according to the first text block; Divide the second text block into third text blocks according to a second distance threshold, wherein the second distance threshold is determined according to the second text block; The third text block is merged according to the overlap rate of the trajectory point coordinates in the font height direction to obtain the text line.

9. A terminal device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the written text recognition method according to any one of claims 1 to 7 is implemented.

10. A non-volatile computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to execute the written text recognition method according to any one of claims 1 to 7.

Images