Method, system and computing device for recognizing stroke order of handwritten Chinese characters

Abstract

The present invention provides a method, system and computing device for recognizing the stroke order of handwritten Chinese characters, the method comprising: obtaining a trajectory point sequence of a handwritten Chinese character to be recognized, and preprocessing the trajectory point sequence of the handwritten Chinese character to obtain a key point sequence of the handwritten Chinese character; extracting the stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, wherein the stroke features include basic features and semantic features; and using a trained stroke order recognition model to process the stroke features of the handwritten Chinese character together with the stroke features of the corresponding template Chinese character to obtain a stroke order recognition result. The present invention can obtain handwritten Chinese characters and perform real-time analysis, accurately recognize whether the stroke order of writing the handwritten Chinese character is correct or not, and feed back the recognition result to the user, which can not only greatly improve the recognition efficiency of the stroke order of handwritten Chinese characters, relieve the tutoring and teaching pressure of parents and teachers, but also assist the writer to develop the habit of writing Chinese characters according to the correct stroke order.

Description

Technical Field

[0001] The present invention relates to the field of Chinese character recognition, and more specifically, to a method, system and computing device for recognizing the stroke order of handwritten Chinese characters. Background Art

[0002] Chinese characters are the carrier of our country's culture, and the writing process is also a process of experiencing the rich heritage of Chinese traditional culture. However, with the popularization of computers and mobile phones and the rapid development of the Internet, the advent of the digital age has changed people's writing methods. Various input methods for writing Chinese characters have emerged one after another, and typing efficiency has become higher and higher. Although this input method typing does bring a lot of convenience to people's writing, while people enjoy efficient digital life, China's profound Chinese character culture has also been hit unprecedentedly. As people's opportunities to write with a pen are decreasing, "Chinese character amnesia" and "computer agraphia" are spreading, and the phenomenon of ignoring strokes and stroke order is particularly serious.

[0003] However, writing is an important part of primary school Chinese teaching, a part of quality education, and a basic project for promoting and integrating the traditional Chinese culture. The new curriculum standard for primary school Chinese requires that lower-grade primary school students learn to read and write Chinese characters "master the basic strokes of Chinese characters and commonly used radicals, and be able to write with a hard pen according to the stroke order rules." It can be seen that in the enlightenment stage of primary school students' literacy and writing, mastering the stroke order of Chinese characters is crucial. In addition, the process of writing stroke by stroke is also a process of cultivating students' patience and observation, which is very beneficial to the cultivation of students' thinking, the development of intelligence, and the formation of good behavior.

[0004] However, the above-mentioned phenomenon of "amnesia" in writing is also subtly affecting the Chinese character writing habits of primary school students in the lower grades. With the increasingly younger age of computer and mobile phone users, the Chinese character writing level of primary school students has generally declined. It is common for them to forget words when writing and have sloppy handwriting. The phenomenon of ignoring whether the stroke order of Chinese characters is correct is particularly serious. Manual tutoring and error correction is not only time-consuming and labor-intensive, but also excessive intervention by teachers or parents may cause disgust among primary school students, which further affects the cultivation of good Chinese character writing habits among primary school students. Summary of the invention

[0005] The present invention is proposed to solve the above problems. According to one aspect of the present invention, a method for identifying the stroke order of handwritten Chinese characters is provided, the method comprising: obtaining a trajectory point sequence of a handwritten Chinese character to be identified, and preprocessing the trajectory point sequence of the handwritten Chinese character to obtain a key point sequence of the handwritten Chinese character; extracting the stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, wherein the stroke features include basic features and semantic features; and processing the stroke features of the handwritten Chinese character together with the stroke features of the corresponding template Chinese character using a trained stroke order recognition model to obtain a stroke order recognition result.

[0006] In one embodiment, the trained stroke order recognition model includes a trained spatial transformation network and a stroke order recognition network, wherein the spatial transformation network is used to perform spatial transformation on the stroke features of the handwritten Chinese characters to obtain the transformed stroke features of the handwritten Chinese characters; the stroke order recognition network is used to process the transformed stroke features of the handwritten Chinese characters together with the stroke features of the template Chinese characters to obtain the stroke order recognition results.

[0007] In one embodiment, the stroke order recognition network includes a hidden feature extraction module for extracting handwritten Chinese character hidden features and template Chinese character hidden features of the template Chinese character from the transformed stroke features of the handwritten Chinese character and the stroke features of the template Chinese character, respectively.

[0008] In one embodiment, the stroke order recognition network further includes a feature fusion module for fusing the extracted hidden features of the handwritten Chinese characters with the hidden features of the template Chinese characters to obtain difference features between the hidden features of the handwritten Chinese characters and the hidden features of the template Chinese characters.

[0009] In one embodiment, the stroke order recognition network further includes a stroke order recognition module, which is used to identify the stroke order recognition result based on the difference feature.

[0010] In one embodiment, the basic features include the coordinates and writing time of each key point on the stroke segment, and the semantic features include one or more of the Bezier features, path signature features, distance between key points, curvature, and stroke direction features of the stroke segment.

[0011] In one embodiment, the step of preprocessing the trajectory point sequence of the handwritten Chinese character to obtain the key point sequence of the handwritten Chinese character includes: processing the trajectory point sequence of the handwritten Chinese character to obtain the stroke segments on which the trajectory points of the handwritten Chinese character are evenly distributed; for each stroke segment on which the trajectory points are evenly distributed, calculating the curvature angle corresponding to all the trajectory points thereon and the distance between the stroke segment and the previous trajectory point, and determining the key point sequence of the stroke segment based on the curvature angle and / or the distance; and combining the key point sequences of the stroke segments to obtain the key point sequence of the handwritten Chinese character.

[0012] In one embodiment, the method further includes: using a Chinese character recognition model to recognize the handwritten Chinese characters to be recognized, obtaining Chinese character recognition results, and matching and acquiring the corresponding template Chinese characters according to the Chinese character recognition results.

[0013] In one embodiment, the method further includes: if the stroke order recognition result indicates that the stroke order of the handwritten Chinese character to be recognized is wrong, obtaining the correct stroke order of the corresponding template Chinese character and presenting it to the user.

[0014] In one embodiment, the stroke order recognition network is a recurrent neural network.

[0015] In one embodiment, the spatial transformation network and the stroke order recognition network are trained to recognize the stroke order of handwritten Chinese characters through a joint training method.

[0016] In one embodiment, the spatial transformation network and the stroke order recognition network are trained by the following steps: obtaining handwritten Chinese characters and corresponding template Chinese characters as training samples; performing spatial transformation on the template Chinese characters by the spatial transformation network so that the transformed template Chinese characters have the same stroke order as the handwritten Chinese characters; using the hidden layer feature extraction module in the stroke order recognition network to respectively extract the handwritten Chinese character hidden layer features of the handwritten Chinese characters and the template Chinese character hidden layer features of the transformed template Chinese characters, and obtain a first loss function; using the feature fusion module in the stroke order recognition network to fuse the handwritten Chinese character hidden layer features and the template Chinese character hidden layer features to obtain the difference features between the handwritten Chinese character hidden layer features and the template Chinese character hidden layer features, and obtain a second loss function; and adjusting the weights of the spatial transformation network and the stroke order recognition network based on the first loss function and the second loss function to obtain a trained spatial transformation network and stroke order recognition network.

[0017] According to another embodiment of the present invention, a system for recognizing the stroke order of handwritten Chinese characters is provided, the system comprising: a processor for using one or more neural networks to: pre-process an acquired trajectory point sequence of a handwritten Chinese character to obtain a key point sequence of the handwritten Chinese character; extract stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, wherein the stroke features include basic features and semantic features; and process the stroke features of the handwritten Chinese character together with the stroke features of a corresponding template Chinese character using a stroke order recognition model in the neural network to obtain a stroke order recognition result, and a memory for storing network parameters of the one or more neural networks.

[0018] According to another embodiment of the present invention, a computing device is provided, which includes: a preprocessing module, used to obtain a trajectory point sequence of a handwritten Chinese character to be recognized, and preprocess the trajectory point sequence of the handwritten Chinese character to obtain a key point sequence of the handwritten Chinese character; a feature extraction module, used to extract the stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, wherein the stroke features include basic features and semantic features; and a feature processing module, used to process the stroke features of the handwritten Chinese character together with the stroke features of the stored corresponding template Chinese character to obtain a stroke order recognition result.

[0019] According to another embodiment of the present invention, a computing device is provided. The computing device includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the method described above.

[0020] According to yet another embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed, the method described above is executed.

[0021] The method, system and computing device for recognizing the stroke order of handwritten Chinese characters according to the embodiments of the present invention can obtain handwritten Chinese characters and perform real-time analysis, accurately recognize whether the stroke order of writing the handwritten Chinese characters is correct, and feed back the recognition results to the user. This can not only greatly improve the recognition efficiency of the stroke order of handwritten Chinese characters and alleviate the teaching pressure of parents and teachers, but also help the writer to develop the habit of writing Chinese characters according to the correct stroke order. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 An electronic device for implementing the method, system and computing device for recognizing the stroke order of handwritten Chinese characters according to an embodiment of the present invention is shown.

[0024] Figure 2 A flow chart showing the steps of a method for recognizing the stroke order of handwritten Chinese characters according to an embodiment of the present invention is shown.

[0025] Figure 3 A schematic diagram showing the main process of preprocessing a trajectory point sequence of handwritten Chinese characters according to an embodiment of the present invention.

[0026] Figure 4 A schematic block diagram of a stroke order recognition model according to an embodiment of the present invention is shown.

[0027] Figure 5 A schematic structural block diagram of a stroke order recognition network according to an embodiment of the present invention is shown.

[0028] Figure 6 A schematic structural block diagram of a system for recognizing the stroke order of handwritten Chinese characters according to an embodiment of the present invention is shown.

[0029] Figure 7 A schematic structural block diagram of a computing device according to an embodiment of the present invention is shown.

[0030] Figure 8 A schematic structural block diagram of a computing device according to another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0032] As mentioned above, manual tutoring of primary school students in writing Chinese characters is not only time-consuming and labor-intensive, but excessive intervention by teachers or parents may also cause resentment among primary school students.

[0033] Therefore, in order to help cultivate primary school students' good habits of writing Chinese characters, the present invention provides a method for identifying the stroke order of handwritten Chinese characters, the method comprising: obtaining a trajectory point sequence of a handwritten Chinese character to be identified, and preprocessing the trajectory point sequence of the handwritten Chinese character to obtain a key point sequence of the handwritten Chinese character; extracting the stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, wherein the stroke features include basic features and semantic features; and processing the stroke features of the handwritten Chinese character together with the stroke features of the corresponding template Chinese character using a trained stroke order recognition model to obtain a stroke order recognition result.

[0034] The present invention can obtain handwritten Chinese characters and perform real-time analysis, accurately identify whether the stroke order of the handwritten Chinese characters is correct, and feed back the recognition result to the user, which can not only greatly improve the recognition efficiency of the stroke order of handwritten Chinese characters and alleviate the tutoring and teaching pressure of parents and teachers, but also help the writer to develop the habit of writing Chinese characters according to the correct stroke order.

[0035] The method, system and computing device for recognizing the stroke order of handwritten Chinese characters according to the present invention are described in detail below in conjunction with specific embodiments.

[0036] First, refer to Figure 1 An electronic device 100 for implementing the method, system and computing device for recognizing the stroke order of handwritten Chinese characters according to an embodiment of the present invention is described.

[0037] In one embodiment, the electronic device 100 can be, for example, a laptop computer, a desktop computer, a tablet computer, a learning machine, a mobile device (such as a smart phone, a phone watch, etc.), an embedded computer, a tower server, a rack server, a blade server, or any other suitable electronic device.

[0038] In one embodiment, the electronic device 100 may include at least one processor 102 and at least one memory 104 .

[0039] Among them, the memory 104 can be a volatile memory, such as a random access memory (RAM), a cache memory (cache), a dynamic random access memory (DRAM) (including stacked DRAM) or a high bandwidth memory (HBM), etc., or it can be a non-volatile memory, such as a read-only memory (ROM), a flash memory, 3D Xpoint, etc. In one embodiment, some parts of the memory 104 can be volatile memory, while another part can be a non-volatile memory (for example, using a two-level memory hierarchy). The memory 104 is used to store a computer program, which, when executed, can implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below.

[0040] Processor 102 may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, or other processing unit having data processing capabilities and / or instruction execution capabilities. Processor 102 may be communicatively coupled to any suitable number or type of components, peripherals, modules, or devices via a communication bus. In one embodiment, the communication bus may be implemented using any suitable protocol, such as Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Express (PCIe), Accelerated Graphics Port (AGP), HyperTransport, or any other bus or one or more point-to-point communication protocols.

[0041] The electronic device 100 may further include an input device 106 and an output device 108. The input device 106 is a device for receiving user input, which may include a keyboard, a mouse, a touch pad, a microphone, etc. In addition, the input device 106 may also be any interface for receiving information. The output device 108 may output various information (such as images or sounds) to the outside (such as a user), which may include one or more of a display, a speaker, etc. In addition, the output device 108 may also be any other device with an output function, such as a printer, etc.

[0042] Reference below Figure 2 A flowchart of the steps of a method 200 for recognizing the stroke order of handwritten Chinese characters according to an embodiment of the present invention is described. In one embodiment, the method 200 for recognizing the stroke order of handwritten Chinese characters can be performed online or offline.

[0043] like Figure 2 As shown, the method 200 for recognizing the stroke order of handwritten Chinese characters may include the following steps:

[0044] In step S210: a trajectory point sequence of a handwritten Chinese character to be recognized is obtained, and the trajectory point sequence of the handwritten Chinese character is preprocessed to obtain a key point sequence of the handwritten Chinese character.

[0045] The trajectory point sequence of the handwritten Chinese characters may include various information, such as the coordinates of the trajectory points and the writing time of the trajectory points.

[0046] In one embodiment, the trajectory point sequence of the handwritten Chinese characters to be recognized can be obtained through an existing general learning machine or any other electronic device that can support handwriting input (for example, the electronic device 100 described above). For example, when a user writes Chinese characters on an electronic device, the electronic device can adaptively collect the trajectory point sequence of the handwritten Chinese characters according to the handwriting speed of the writer, the sampling rate of the hardware settings, the pressure parameters, etc., and upload it to the cloud for storage or save it locally, so that the present invention can obtain the trajectory point sequence of the input handwritten Chinese characters from the cloud or locally.

[0047] Since different electronic devices use different hardware, the sampling rate, pressure and other parameters of the trajectory points of the collected handwritten Chinese characters will be different. At the same time, since each person has different writing habits and writing speeds, this leads to different people writing the same character, or even the same person writing the same Chinese character at different times, with different shapes of the written Chinese characters and different numbers of trajectory points. Therefore, before sending the input trajectory point sequence of handwritten Chinese characters to the subsequent stroke order recognition model, the input trajectory point sequence of handwritten Chinese characters can be preprocessed to increase the robustness of the back-end stroke order recognition model.

[0048] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing the main process of preprocessing a trajectory point sequence of handwritten Chinese characters according to an embodiment of the present invention. Figure 3 As shown, in one embodiment, preprocessing the trajectory point sequence of the handwritten Chinese character to obtain the key point sequence of the handwritten Chinese character may include the following steps:

[0049] Step a: Processing the trajectory point sequence of the handwritten Chinese character to obtain stroke line segments in which the trajectory points of the handwritten Chinese character are evenly distributed.

[0050] In one embodiment, this step can be achieved by first separating the stroke segments and then resampling the stroke segments. Specifically:

[0051] First, the trajectory point sequence of each stroke segment can be separated according to the starting point and the ending point of the handwritten Chinese character.

[0052] When each person writes a Chinese character, each stroke corresponds to a starting point and an ending point. Even in the case of connected strokes, the connected strokes will correspond to a starting point and an ending point. Therefore, the trajectory point sequence of each stroke segment can be separated according to the starting point and the ending point of the input handwritten Chinese character.

[0053] Then, the trajectory point sequence of the stroke line segment is resampled to obtain the stroke line segment with uniformly distributed trajectory points of the handwritten Chinese character.

[0054] Specifically, for each separated stroke segment, the total length of the stroke segment is calculated, and the trajectory points of the total length of the stroke segment are uniformly resampled according to the set sampling rate, so that the trajectory points in each stroke segment obtained by sampling are evenly distributed.

[0055] Step b. for each stroke segment with uniformly distributed trajectory points, calculate the curvature angle corresponding to all trajectory points and the distance between the stroke segment and the previous trajectory point, and determine the key point sequence of the stroke segment based on the curvature angle and / or the distance.

[0056] Specifically, for each stroke line segment with uniformly distributed trajectory points, the curvature angle θ corresponding to all trajectory points on the stroke line segment and the distance d between the previous trajectory point are calculated.

[0057] The distance d between a trajectory point and the previous trajectory point can be directly calculated based on the coordinate difference between the two trajectory points. Specifically, assuming that the coordinates of the trajectory point are (x2, y2) and the coordinates of the previous trajectory point are (x1, y1), then the distance between the trajectory point and the previous trajectory point is The curvature angle θ refers to the angle between the current trajectory point and the line connecting the previous trajectory point and the next trajectory point.

[0058] For each trajectory point within each stroke segment, the above two quantities can be calculated. Then, a distance threshold T1 and a curvature angle threshold T2 can be set respectively. And when d < T1 or θ < T2, it can be considered that this trajectory point is not a key point and should be filtered out. When d > T1, or θ > T2, or both d > T1 and θ > T2 are satisfied, then this trajectory point is considered a key point. It should be understood that the above method for determining a trajectory point as a key point is merely exemplary and not restrictive. Other well-known determination methods in the art can also be used, and the present invention does not limit this.

[0059] After extracting the key points of each stroke segment, a key point sequence of each stroke segment is formed.

[0060] Step c. Combine the key point sequences of each stroke segment to obtain the key point sequence of the handwritten Chinese character.

[0061] Specifically, after extracting the key point sequences corresponding to each stroke of each handwritten Chinese character, they are combined into the key point sequence of this handwritten Chinese character for use in subsequent stroke feature extraction steps.

[0062] Step S220: Extract the stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, where the stroke features include basic features and semantic features.

[0063] In one embodiment, a neural network can be used to extract the stroke features of the handwritten Chinese character, where the neural network can be any neural network well-known in the art that has been trained, such as a recurrent neural network (RNN), a deep neural network (DNN), a deep belief network (DBN), etc. The present invention does not limit this.

[0064] In one embodiment, the basic features can include the coordinates and writing time of each key point on the stroke segment, and the semantic features can include one or more features among the bezier feature, path signature feature, distance between key points, curvature, and stroke direction feature of the stroke segment. The present invention does not limit this.

[0065] In addition to basic features such as coordinates, the features extracted by the present invention also include semantic features. Compared with only sending basic features without any semantic features into the backend stroke order recognition model, sending these stroke features rich in semantic features into the backend stroke order recognition model is more helpful for the model to learn the key abstract semantic features beneficial to stroke order recognition.

[0066] Step S230: Use the trained stroke order recognition model to process the stroke features of the handwritten Chinese character and the stroke features of the corresponding template Chinese character together to obtain the stroke order recognition result.

[0067] The stroke features of the template Chinese characters may be acquired in a variety of ways, which are not limited in the present invention.

[0068] In one embodiment, a template Chinese character library storing a number of template Chinese characters may be pre-established. After obtaining the handwritten Chinese character to be recognized, it is recognized and a Chinese character recognition result is obtained. Then, based on the Chinese character recognition result, the template Chinese character corresponding to the handwritten Chinese character is matched and obtained, and the template Chinese character is processed similarly to steps S210 and S220 to obtain the stroke features of the template Chinese character.

[0069] Among them, any method known in the art can be used to establish the template Chinese character library. In one embodiment, the 6763 Chinese characters included in the Chinese character encoding character set GB2312 for information exchange can be used to establish the template Chinese character library. Other methods can also be used to establish the template Chinese character library, such as web crawling, computer generation, etc., which are not limited by the present invention.

[0070] In one embodiment, the trajectory point sequence of the template Chinese character may be directly stored, and after the corresponding template Chinese character is matched and acquired, the above-mentioned processing is performed on the trajectory point sequence of the corresponding template Chinese character to obtain the stroke features of the template Chinese character.

[0071] In one embodiment, the key point sequence of the template Chinese character can also be directly stored, and after matching and obtaining the corresponding template Chinese character, the stroke features of the template Chinese character are extracted based on the key point sequence of the corresponding template Chinese character. In one embodiment, the stroke features of the template Chinese character can even be directly stored, and after matching and obtaining the corresponding template Chinese character, the stroke features of the corresponding template Chinese character are directly obtained.

[0072] refer to Figure 4 , Figure 4 FIG. 4 is a schematic block diagram of a stroke order recognition model 400 according to an embodiment of the present invention. Figure 4 As shown, in one embodiment, the trained stroke order recognition model may include a trained spatial transformation network 410 and a stroke order recognition network 420, wherein the spatial transformation network 410 is used to perform spatial transformation on the stroke features of handwritten Chinese characters to obtain the transformed stroke features of handwritten Chinese characters; the stroke order recognition network 420 is used to process the transformed stroke features of the handwritten Chinese characters together with the stroke features of the template Chinese characters to obtain the stroke order recognition results.

[0073] Due to different writing habits of different people, there will be a large difference in the morphology between the handwritten Chinese characters to be recognized and the template Chinese characters. If the Chinese characters to be recognized and the template Chinese characters are directly input into the stroke order recognition network 420, due to the capacity limitation of the stroke order recognition network, it is difficult to correctly recognize the handwritten Chinese characters with large deformation. Therefore, in order to reduce the influence of the deformation of the handwritten Chinese characters to be recognized on the recognition results, a spatial transformation network 410 is introduced before the stroke order recognition network 420 to perform spatial transformation on the stroke features of the handwritten Chinese characters, that is, to perform spatial deformation correction on the handwritten Chinese characters, so that the morphology of the transformed handwritten Chinese characters is close to the morphology of the corresponding template Chinese characters, thereby improving the accuracy of the handwritten Chinese character recognition results.

[0074] Exemplarily, the space transformation network 410 can be constructed using any deep learning network capable of adjusting space deformation, which is not limited in the present invention.

[0075] In one embodiment, the stroke order recognition network 420 may be a recurrent neural network (RNN). It should be understood that the stroke order recognition network may also be other neural networks capable of realizing stroke order recognition, and the present invention is not limited thereto.

[0076] refer to Figure 5 , Figure 5 FIG. 4 shows a schematic structural block diagram of a stroke order recognition network 420 according to an embodiment of the present invention. Figure 5 As shown, in one embodiment, the stroke order recognition network 420 may include a hidden layer feature extraction module 510, a feature fusion module 520 and a stroke order recognition module 530. It should be understood that the stroke order recognition network 420 may also include other modules that are helpful for recognizing the stroke order of handwritten Chinese characters, and the present invention is not limited to this.

[0077] Among them, the hidden layer feature extraction module 510 is used to receive the transformed stroke features of handwritten Chinese characters and the stroke features of template Chinese characters from the spatial transformation network, and extract the handwritten Chinese character hidden layer features of the handwritten Chinese characters and the template Chinese character hidden layer features of the template Chinese characters from the transformed stroke features of the handwritten Chinese characters and the stroke features of the template Chinese characters respectively.

[0078] In one embodiment, the hidden layer feature extraction module 510 can be implemented by a recurrent neural network (RNN). It should be understood that the hidden layer feature extraction module 510 can also be implemented by other neural networks capable of implementing the hidden layer feature extraction function, and the present invention is not limited to this.

[0079] Among them, the feature fusion module 520 is used to receive the extracted handwritten Chinese character hidden features and template Chinese character hidden features from the hidden feature extraction module, and fuse the handwritten Chinese character hidden features and the template Chinese character hidden features to obtain the difference features between the handwritten Chinese character hidden features and the template Chinese character hidden features.

[0080] In one embodiment, the feature fusion module 520 can be implemented by a recurrent neural network (RNN). It should be understood that the feature fusion module 520 can also be implemented by other neural networks that can achieve fusion functions, and the present invention is not limited to this.

[0081] In one embodiment, fusing the hidden features of handwritten Chinese characters with the hidden features of template Chinese characters may include: subtracting the hidden features of handwritten Chinese characters from the hidden features of template Chinese characters, so as to adaptively learn the differences between the hidden features of handwritten Chinese characters and the hidden features of template Chinese characters, thereby ensuring that the contrast features obtained after fusion eliminate features irrelevant to stroke order recognition, and only retain the difference features related to stroke order recognition as much as possible, thereby greatly reducing the recognition difficulty of subsequent recognition modules. It should be understood that the above-mentioned difference features can also be adaptively learned by adding, splicing or introducing a convolutional neural network (CNN) to the hidden features of handwritten Chinese characters and the hidden features of template Chinese characters, and the present invention is not limited to this.

[0082] Among them, the stroke order recognition module 530 is used to receive the above-mentioned difference features from the feature fusion module, and recognize the stroke order recognition result based on the difference features. The role of the stroke order recognition module 530 is to convert the difference features into a recognition result recognizable by the user, such as a binary result, for example, 0 represents that the stroke order of the handwritten Chinese character to be recognized is correct, and 1 represents that the stroke order of the handwritten Chinese character to be recognized is correct. It should be understood that other forms of recognition results can also be used, and the present invention is not limited to this.

[0083] In one embodiment, the stroke order recognition module 530 can be implemented by a recurrent neural network (RNN). It should be understood that the stroke order recognition module 530 can also be implemented by other neural networks capable of realizing recognition functions, and the present invention is not limited to this.

[0084] In one embodiment, the method 200 may further include: jointly training the spatial transformation network 410 and the stroke order recognition network 420 to train the stroke order for recognizing handwritten Chinese characters. Exemplarily, the joint training may be end-to-end joint training, which helps the spatial transformation network learn the correct spatial transformation matrix, so that the handwritten Chinese characters to be recognized output by the spatial transformation network are not only as close as possible to the template Chinese characters in terms of spatial form, but also closer to the template Chinese characters in the hidden layer feature space, thereby being more conducive to improving the accuracy of the handwritten Chinese character recognition results.

[0085] In one embodiment, the spatial transformation network 410 and the stroke order recognition network 420 may be trained by the following steps:

[0086] A handwritten Chinese character and a corresponding template Chinese character are obtained as training samples, and the template Chinese character is spatially transformed by a spatial transformation network 410 so that the transformed template Chinese character has the same stroke order as the handwritten Chinese character; a hidden feature extraction module 510 in a stroke order recognition network is used to extract the handwritten Chinese character hidden features of the handwritten Chinese character and the template Chinese character hidden features of the transformed template Chinese character, and a first loss function is obtained; a feature fusion module 520 in a stroke order recognition network is used to fuse the handwritten Chinese character hidden features and the template Chinese character hidden features to obtain the difference features between the handwritten Chinese character hidden features and the template Chinese character hidden features, and a second loss function is obtained; and the weights of the spatial transformation network 410 and the stroke order recognition network 420 are adjusted based on the first loss function and the second loss function to obtain a trained spatial transformation network and a stroke order recognition network.

[0087] The handwritten Chinese characters used as training samples include marked samples and unmarked samples, wherein the marked samples are called ground truth data. The process of obtaining the first loss function and the second loss function from the training samples is called forward propagation, and the process of adjusting the weights of the spatial transformation network 410 and the stroke order recognition network 420 based on the first loss function and the second loss function is called back propagation. After multiple forward propagations and back propagations, the trained spatial transformation network and the trained stroke order recognition network are finally obtained.

[0088] The above training process of the space transformation network 410 and the stroke order recognition network 420 is well known in the art and will not be described in detail here.

[0089] In one embodiment, the method 200 may further include: if the stroke order recognition result indicates that the stroke order of the handwritten Chinese character to be recognized is wrong, obtaining the correct stroke order of the corresponding template Chinese character and presenting it to the user.

[0090] In one embodiment, when the stroke order recognition result indicates that the stroke order of the handwritten Chinese character to be recognized is wrong, a prompt message can be issued to prompt the user that the stroke order is wrong, and the correct stroke order is obtained and presented to the user. In one embodiment, the prompt message can include sound prompts and / or visual prompts. Exemplarily, the sound prompts are such as alarm sounds, voice prompts, etc., and the visual prompts are such as warning lights, icons, etc., which are not limited by the present invention.

[0091] As described above, the method for recognizing the stroke order of handwritten Chinese characters in an embodiment of the present invention can obtain any handwritten Chinese character and perform real-time analysis, accurately recognize whether the stroke order of writing the handwritten Chinese character is correct, and feed back the recognition result to the user. When the stroke order is written incorrectly, a prompt is given and the correct stroke order is presented to the writer. This can not only greatly improve the recognition efficiency of the stroke order of handwritten Chinese characters and alleviate the tutoring and teaching pressure of parents and teachers, but also assist the writer to develop a good habit of writing Chinese characters according to the correct stroke order.

[0092] Reference below Figure 6 A schematic structural block diagram of a system 600 for recognizing the stroke order of handwritten Chinese characters according to an embodiment of the present invention is described. Figure 6 As shown, the system 600 for recognizing the stroke order of handwritten Chinese characters may include a processor 610 and a memory 620 .

[0093] Among them, the processor 610 is used to use one or more neural networks to implement the following processing steps: preprocessing the acquired trajectory point sequence of the handwritten Chinese character to obtain the key point sequence of the handwritten Chinese character; extracting the stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, wherein the stroke features include basic features and semantic features; and using the stroke order recognition model in the neural network to process the stroke features of the handwritten Chinese character together with the stroke features of the corresponding template Chinese character to obtain a stroke order recognition result.

[0094] Exemplarily, the processor 610 can be any processing device known in the art, such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, a field programmable gate array (FPGA), etc., and the present invention is not limited to this.

[0095] The memory 620 is used to store model parameters of one or more machine learning models. Exemplarily, the memory 620 can be RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage device, tape cassette, tape, magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired information and can be accessed by the processor 610.

[0096] The trajectory point sequence of handwritten Chinese characters may contain various information, such as the coordinates of all trajectory points and the writing time of a certain trajectory point.

[0097] In one embodiment, the trajectory point sequence of the handwritten Chinese characters to be recognized can be obtained through an existing general learning machine or any other electronic device that can support handwriting input (for example, the electronic device 100 described above). For example, when a user writes Chinese characters on an electronic device, the electronic device can adaptively collect the trajectory point sequence of the handwritten Chinese characters according to the handwriting speed of the writer, the sampling rate of the hardware settings, the pressure parameters, etc., and upload it to the cloud for storage or save it locally, so that the present invention can obtain the trajectory point sequence of the input handwritten Chinese characters from the cloud or locally.

[0098] Since different electronic devices use different hardware, the sampling rate, pressure and other parameters of the trajectory points of the collected handwritten Chinese characters will be different. At the same time, since each person has different writing habits and writing speeds, this leads to different people writing the same character, or even the same person writing the same Chinese character at different times, with different shapes of the written Chinese characters and different numbers of trajectory points. Therefore, before sending the input trajectory point sequence of handwritten Chinese characters to the stroke order recognition model, the input trajectory point sequence of handwritten Chinese characters can be preprocessed to increase the robustness of the stroke order recognition model.

[0099] In one embodiment, preprocessing the trajectory point sequence of a handwritten Chinese character to obtain the key point sequence of the handwritten Chinese character may include the following steps:

[0100] Step a: Processing the trajectory point sequence of the handwritten Chinese character to obtain stroke line segments in which the trajectory points of the handwritten Chinese character are evenly distributed.

[0101] In one embodiment, this step can be achieved by first separating the stroke segments and then resampling the stroke segments. Specifically:

[0102] First, the trajectory point sequence of each stroke segment can be separated according to the starting point and the ending point of the handwritten Chinese character.

[0103] When each person writes a Chinese character, each stroke corresponds to a starting point and an ending point. Even in the case of connected strokes, the connected strokes will correspond to a starting point and an ending point. Therefore, the trajectory point sequence of each stroke segment can be separated according to the starting point and the ending point of the input handwritten Chinese character.

[0104] Then, the trajectory point sequence of the stroke line segment is resampled to obtain the stroke line segment with uniformly distributed trajectory points of the handwritten Chinese character.

[0105] Specifically, for each separated stroke segment, the total length of the stroke segment is calculated, and the trajectory points of the total length of the stroke segment are uniformly resampled according to the set sampling rate, so that the trajectory points in each stroke segment obtained by sampling are evenly distributed.

[0106] Step b. For the stroke segments evenly distributed for each trajectory point, calculate the sum of the curvature angles corresponding to all the trajectory points thereon and the distance from the previous trajectory point, and determine the key point sequence of the stroke segment based on the sum of the curvature angles and / or the distance.

[0107] Specifically, for the stroke segments evenly distributed for each trajectory point, calculate the curvature angle θ corresponding to all the trajectory points on the stroke segment and the distance d from the previous trajectory point.

[0108] Among them, the distance d between a certain trajectory point and the previous trajectory point can be directly calculated according to the coordinate difference between the two trajectory points. Specifically, let the coordinates of this trajectory point be (x2, y2), and the coordinates of the previous trajectory point be (x1, y1), then the distance between this trajectory point and the previous trajectory point For the curvature angle θ, it refers to the angle size between the line connecting the current trajectory point and the previous and the next trajectory points.

[0109] For each trajectory point within each stroke segment, the above two quantities can be calculated, and then the distance threshold can be set as T1 and the curvature angle threshold can be set as T2 respectively. And when d < T1 or θ < T2, it can be considered that this trajectory point is not a key point and should be filtered out. When d > T1, or θ > T2, or both d > T1 and θ > T2 are satisfied, then it is considered that this trajectory point is a key point. It should be understood that the above method for determining a trajectory point as a key point is only exemplary and not restrictive, and other well-known judgment methods in the art can also be used, and the present invention does not make any limitations in this regard.

[0110] After extracting the key points of each stroke segment, form the key point sequence of each stroke segment.

[0111] Step c. Combine the key point sequences of each stroke segment to obtain the key point sequence of the handwritten Chinese character.

[0112] Specifically, after extracting the key point sequences corresponding to each stroke of each handwritten Chinese character, combine them into the key point sequence of the handwritten Chinese character for use in subsequent stroke feature extraction steps.

[0113] In one embodiment, a neural network can be used to extract the stroke features of handwritten Chinese characters, and the neural network can be any neural network well-known in the art that has been trained, such as a recurrent neural network (RNN), a deep neural network (DNN), a deep belief network (DBN), etc., and the present invention does not make any limitations in this regard.

[0114] In one embodiment, the basic features may include the coordinates and writing time of each key point on the stroke segment, and the semantic features may include one or more of the Bezier features, path signature features, distance between key points, curvature, and stroke direction features of the stroke segment, but the present invention is not limited to this.

[0115] In addition to basic features such as coordinates, the features extracted by the present invention also include semantic features. Compared with sending only basic features without any semantic features into the stroke order recognition model, sending these stroke features rich in semantic features into the stroke order recognition model is more helpful for the model to learn key abstract semantic features that are beneficial to stroke order recognition.

[0116] The stroke features of the template Chinese characters may be acquired in a variety of ways, which are not limited in the present invention.

[0117] In one embodiment, a template Chinese character library storing a number of template Chinese characters can be pre-established. After obtaining the handwritten Chinese character to be recognized, it is recognized and a Chinese character recognition result is obtained. Then, based on the Chinese character recognition result, the template Chinese character corresponding to the handwritten Chinese character is matched and obtained, and the template Chinese character is processed similarly to the above to obtain the stroke features of the template Chinese character.

[0118] Among them, any method known in the art can be used to establish the template Chinese character library. In one embodiment, the 6763 Chinese characters included in the Chinese character encoding character set GB2312 for information exchange can be used to establish the template Chinese character library. Other methods can also be used to establish the template Chinese character library, such as web crawling, computer generation, etc., which are not limited by the present invention.

[0119] In one embodiment, the trajectory point sequence of the template Chinese character may be directly stored, and after the corresponding template Chinese character is matched and acquired, the above-mentioned processing is performed on the trajectory point sequence of the corresponding template Chinese character to obtain the stroke features of the template Chinese character.

[0120] In one embodiment, the key point sequence of the template Chinese character can also be directly stored, and after matching and obtaining the corresponding template Chinese character, the stroke features of the template Chinese character are extracted based on the key point sequence of the corresponding template Chinese character. In one embodiment, the stroke features of the template Chinese character can even be directly stored, and after matching and obtaining the corresponding template Chinese character, the stroke features of the corresponding template Chinese character are directly obtained.

[0121] In one embodiment, the stroke order recognition model may include a spatial transformation network and a stroke order recognition network, wherein the spatial transformation network is used to perform spatial transformation on the stroke features of handwritten Chinese characters to obtain the transformed stroke features of the handwritten Chinese characters; the stroke order recognition network is used to process the transformed stroke features of the handwritten Chinese characters together with the stroke features of the template Chinese characters to obtain the stroke order recognition results.

[0122] Due to different writing habits of different people, there will be great differences in the morphology between the handwritten Chinese characters to be recognized and the template Chinese characters. If the Chinese characters to be recognized and the template Chinese characters are directly input into the stroke order recognition network, due to the capacity limitation of the stroke order recognition network, it is difficult to correctly recognize the handwritten Chinese characters with large deformation. Therefore, in order to reduce the influence of the deformation of the handwritten Chinese characters to be recognized on the recognition results, a spatial transformation network is introduced before the stroke order recognition network to perform spatial transformation on the stroke features of the handwritten Chinese characters, that is, to perform spatial deformation correction on the handwritten Chinese characters, so that the morphology of the transformed handwritten Chinese characters is close to the morphology of the corresponding template Chinese characters, thereby improving the accuracy of the handwritten Chinese character recognition results.

[0123] Exemplarily, the space transformation network can be constructed using any deep learning network capable of adjusting space deformation, and the present invention is not limited to this.

[0124] In one embodiment, the stroke order recognition network may be a recurrent neural network (RNN). It should be understood that the stroke order recognition network may also be other neural networks capable of realizing stroke order recognition, and the present invention is not limited thereto.

[0125] In one embodiment, the stroke order recognition network may include a hidden layer feature extraction module, a feature fusion module and a stroke order recognition module. It should be understood that the stroke order recognition network may also include other modules that are helpful for recognizing the stroke order of handwritten Chinese characters, and the present invention is not limited to this.

[0126] Among them, the hidden layer feature extraction module is used to receive the transformed stroke features of handwritten Chinese characters and the stroke features of template Chinese characters from the spatial transformation network, and extract the handwritten Chinese character hidden layer features of the handwritten Chinese characters and the template Chinese character hidden layer features of the template Chinese characters from the transformed stroke features of the handwritten Chinese characters and the stroke features of the template Chinese characters respectively.

[0127] In one embodiment, the hidden layer feature extraction module can be implemented by a recurrent neural network (RNN). It should be understood that the hidden layer feature extraction module can also be implemented by other neural networks that can realize the hidden layer feature extraction function, and the present invention is not limited to this.

[0128] Among them, the feature fusion module is used to receive the extracted handwritten Chinese character hidden features and template Chinese character hidden features from the hidden feature extraction module, fuse the handwritten Chinese character hidden features and the template Chinese character hidden features to obtain the difference features of the handwritten Chinese character hidden features and the template Chinese character hidden features.

[0129] In one embodiment, the feature fusion module can be implemented by a recurrent neural network (RNN). It should be understood that the feature fusion module can also be implemented by other neural networks that can achieve fusion functions, and the present invention is not limited to this.

[0130] In one embodiment, fusing the hidden features of handwritten Chinese characters with the hidden features of template Chinese characters may include: subtracting the hidden features of handwritten Chinese characters from the hidden features of template Chinese characters, so as to adaptively learn the differences between the hidden features of handwritten Chinese characters and the hidden features of template Chinese characters, thereby ensuring that the contrast features obtained after fusion eliminate features irrelevant to stroke order recognition, and only retain the difference features related to stroke order recognition as much as possible, thereby greatly reducing the recognition difficulty of subsequent recognition modules. It should be understood that the above-mentioned difference features can also be adaptively learned by adding, splicing or introducing a convolutional neural network (CNN) to the hidden features of handwritten Chinese characters and the hidden features of template Chinese characters, and the present invention is not limited to this.

[0131] Among them, the stroke order recognition module is used to receive the above-mentioned difference features from the feature fusion module, and identify the stroke order recognition result based on the difference features. The role of the stroke order recognition module is to convert the difference features into a recognition result recognizable by the user, such as a binary result, for example, 0 represents that the stroke order of the handwritten Chinese character to be recognized is correct, and 1 represents that the stroke order of the handwritten Chinese character to be recognized is correct. It should be understood that other forms of recognition results can also be used, and the present invention is not limited to this.

[0132] In one embodiment, the stroke order recognition module can be implemented by a recurrent neural network (RNN). It should be understood that the stroke order recognition module can also be implemented by other neural networks capable of realizing recognition functions, and the present invention does not limit this.

[0133] In one embodiment, the spatial transformation network and the stroke order recognition network can be jointly trained together to train the stroke order for recognizing handwritten Chinese characters. Exemplarily, the joint training can be end-to-end joint training, which helps the spatial transformation network learn the correct spatial transformation matrix, so that the handwritten Chinese characters to be recognized output by the spatial transformation network are not only as close as possible to the template Chinese characters in spatial form, but also closer to the template Chinese characters in the hidden layer feature space, thereby being more conducive to improving the accuracy of the handwritten Chinese character recognition results.

[0134] In one embodiment, the spatial transformation network and the stroke order recognition network may be trained by the following steps:

[0135] A handwritten Chinese character and a corresponding template Chinese character are obtained as training samples, and the template Chinese character is spatially transformed by a spatial transformation network so that the transformed template Chinese character has the same stroke order as the handwritten Chinese character; a hidden feature extraction module in a stroke order recognition network is used to extract the handwritten Chinese character hidden features of the handwritten Chinese character and the template Chinese character hidden features of the transformed template Chinese character, and a first loss function is obtained; a feature fusion module in a stroke order recognition network is used to fuse the handwritten Chinese character hidden features and the template Chinese character hidden features to obtain the difference features between the handwritten Chinese character hidden features and the template Chinese character hidden features, and a second loss function is obtained; and the weights of the spatial transformation network and the stroke order recognition network are adjusted based on the first loss function and the second loss function to obtain a trained spatial transformation network and a stroke order recognition network.

[0136] The handwritten Chinese characters used as training samples include marked samples and unmarked samples, wherein the marked samples are called ground truth data. The process of obtaining the first loss function and the second loss function from the training samples is called forward propagation, and the process of adjusting the weights of the spatial transformation network and the stroke order recognition network based on the first loss function and the second loss function is called back propagation. After multiple forward propagations and back propagations, the trained spatial transformation network and the trained stroke order recognition network are finally obtained.

[0137] The above training process for the space transformation network and the stroke order recognition network is well known in the art and will not be repeated here.

[0138] In one embodiment, if the stroke order recognition result indicates that the stroke order of the handwritten Chinese character to be recognized is wrong, the correct stroke order of the corresponding template Chinese character is obtained and presented to the user.

[0139] In one embodiment, when the stroke order recognition result indicates that the stroke order of the handwritten Chinese character to be recognized is wrong, a prompt message can be issued to prompt the user that the stroke order is wrong, and the correct stroke order is obtained and presented to the user. In one embodiment, the prompt message can include sound prompts and / or visual prompts. Exemplarily, the sound prompts are such as alarm sounds, voice prompts, etc., and the visual prompts are such as warning lights, icons, etc., which are not limited by the present invention.

[0140] As described above, the system for recognizing the stroke order of handwritten Chinese characters in an embodiment of the present invention can obtain any handwritten Chinese character and perform real-time analysis, accurately recognize whether the stroke order of writing the handwritten Chinese character is correct, and feed back the recognition result to the user. In the case of incorrect stroke order, a prompt is given and the correct stroke order is presented to the writer. This can not only greatly improve the recognition efficiency of the stroke order of handwritten Chinese characters and alleviate the tutoring and teaching pressure of parents and teachers, but also help the writer develop a good habit of writing Chinese characters according to the correct stroke order.

[0141] In yet another embodiment, the present invention provides a computing device. Figure 7 , Figure 7 FIG. 7 is a schematic structural block diagram of a computing device 700 according to an embodiment of the present invention. Figure 7 As shown, the computing device 700 may include a preprocessing module 710, a feature extraction module 720 and a feature processing module 730. Among them, the preprocessing module 710 is used to obtain the trajectory point sequence of the handwritten Chinese character to be recognized, and preprocess the trajectory point sequence of the handwritten Chinese character to obtain the key point sequence of the handwritten Chinese character; the feature extraction module 720 is used to extract the stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, wherein the stroke features include basic features and semantic features; the feature processing module 730 is used to process the stroke features of the handwritten Chinese character together with the stroke features of the stored corresponding template Chinese character to obtain the stroke order recognition result. The computing device 700 according to the embodiment of the present invention can execute the method 200 for identifying the stroke order of handwritten Chinese characters according to the embodiment of the present invention described above. Those skilled in the art can understand the specific operation of the computing device 700 according to the embodiment of the present invention in combination with the content described above, and for the sake of brevity, it will not be repeated here.

[0142] In yet another embodiment, the present invention provides a computing device. Figure 8 , Figure 8 FIG. 8 is a schematic structural block diagram of a computing device 800 according to another embodiment of the present invention. Figure 8 As shown, the computing device 800 may include a memory 810 and a processor 820, wherein the memory 810 stores a computer program, and when the computer program is executed by the processor 820, the processor 820 executes the method for recognizing the stroke order of handwritten Chinese characters as described above.

[0143] Those skilled in the art can understand the specific operation of the computing device 800 according to the embodiment of the present application in combination with the above contents. For the sake of brevity, the specific details are not repeated here, and only some main operations of the processor 820 are described as follows:

[0144] Acquire a trajectory point sequence of a handwritten Chinese character to be recognized, and preprocess the trajectory point sequence of the handwritten Chinese character to obtain a key point sequence of the handwritten Chinese character;

[0145] Extracting stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, wherein the stroke features include basic features and semantic features; and

[0146] The stroke features of the handwritten Chinese characters are processed together with the stroke features of the corresponding template Chinese characters using the trained stroke order recognition model to obtain a stroke order recognition result.

[0147] As described above, the computing device of an embodiment of the present invention can obtain any handwritten Chinese character and perform real-time analysis, accurately identify whether the stroke order of writing the handwritten Chinese character is correct, and feed back the recognition result to the user. If the stroke order is written incorrectly, a prompt is given and the correct stroke order is presented to the writer. This can not only greatly improve the recognition efficiency of the stroke order of handwritten Chinese characters and alleviate the tutoring and teaching pressure of parents and teachers, but also help the writer develop a good habit of writing Chinese characters in the correct stroke order.

[0148] In another embodiment, the present invention provides a computer-readable medium having a computer program stored thereon, wherein the computer program executes the method described in the above embodiment when running. Any tangible, non-temporary computer-readable medium may be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD-ROMs, DVDs, Blu-ray disks, etc.), flash memory, and / or the like. These computer program instructions may be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing device to form a machine, so that these instructions executed on a computer or other programmable data processing device may generate a device that implements a specified function. These computer program instructions may also be stored in a computer-readable memory, which may instruct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory may form a manufactured product, including an implementation device that implements a specified function. The computer program instructions may also be loaded onto a computer or other programmable data processing device, so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device may provide steps for implementing a specified function.

[0149] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0150] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0151] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0152] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0153] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0154] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0155] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for recognizing the stroke order of handwritten Chinese characters, characterized in that: The method comprises: Acquire a trajectory point sequence of a handwritten Chinese character to be recognized, and preprocess the trajectory point sequence of the handwritten Chinese character to obtain a key point sequence of the handwritten Chinese character, wherein the key points in the key point sequence are determined based on the curvature angle corresponding to each trajectory point and the distance between each trajectory point and the previous trajectory point; Extracting stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, wherein the stroke features include basic features and semantic features; and The stroke features of the handwritten Chinese characters are processed together with the stroke features of the corresponding template Chinese characters using the trained stroke order recognition model to obtain a stroke order recognition result. The trained stroke order recognition model includes a stroke order recognition network, the stroke order recognition network includes a hidden feature extraction module, a feature fusion module and a stroke order recognition module, the hidden feature extraction module is used to extract the handwritten Chinese character hidden layer features of the handwritten Chinese character and the template Chinese character hidden layer features of the template Chinese character from the stroke features of the handwritten Chinese character and the stroke features of the template Chinese character respectively; The basic features include coordinates and writing time of each key point on the stroke segment, and the semantic features include one or more of Bezier features, path signature features, distance between key points, curvature, and stroke direction features of the stroke segment; The feature fusion module is used to fuse the extracted hidden features of the handwritten Chinese characters and the hidden features of the template Chinese characters to obtain the difference features between the hidden features of the handwritten Chinese characters and the hidden features of the template Chinese characters.

2. The method according to claim 1, characterized in that The trained stroke order recognition model further includes a trained spatial transformation network, wherein: The spatial transformation network is used to perform spatial transformation on the stroke features of the handwritten Chinese characters to obtain transformed stroke features of the handwritten Chinese characters; The stroke order recognition network is used to process the stroke features of the transformed handwritten Chinese characters together with the stroke features of the template Chinese characters to obtain a stroke order recognition result.

3. The method according to claim 1, characterized in that The stroke order recognition module is used to identify the stroke order recognition result based on the difference feature.

4. The method according to claim 1, characterized in that The step of preprocessing the trajectory point sequence of the handwritten Chinese character to obtain the key point sequence of the handwritten Chinese character comprises: Processing the trajectory point sequence of the handwritten Chinese character to obtain stroke line segments in which the trajectory points of the handwritten Chinese character are evenly distributed; For each stroke line segment with uniformly distributed trajectory points, calculate the curvature angles corresponding to all trajectory points on the stroke line segment and the distances between the stroke line segment and the previous trajectory point, and determine the key point sequence of the stroke line segment based on the curvature angles and / or the distances; The key point sequences of the stroke segments are combined to obtain the key point sequence of the handwritten Chinese character.

5. The method according to claim 1, characterized in that The method further comprises: using a Chinese character recognition model to recognize the handwritten Chinese characters to be recognized, obtaining Chinese character recognition results, and matching and acquiring the corresponding template Chinese characters according to the Chinese character recognition results.

6. The method according to claim 1, characterized in that The method further includes: if the stroke order recognition result indicates that the stroke order of the handwritten Chinese character to be recognized is wrong, obtaining the correct stroke order of the corresponding template Chinese character and presenting it to the user.

7. The method according to claim 1, characterized in that The stroke order recognition network is a recurrent neural network.

8. The method according to claim 2, characterized in that The spatial transformation network and the stroke order recognition network are trained to recognize the stroke order of handwritten Chinese characters through a joint training method.

9. The method according to claim 8, characterized in that The spatial transformation network and the stroke order recognition network are trained by the following steps: Obtaining handwritten Chinese characters and corresponding template Chinese characters as training samples; The template Chinese character is spatially transformed by a spatial transformation network so that the transformed template Chinese character has the same stroke order as the handwritten Chinese character; Using the hidden layer feature extraction module in the stroke order recognition network, respectively extract the handwritten Chinese character hidden layer features of the handwritten Chinese character and the template Chinese character hidden layer features of the transformed template Chinese character, and obtain a first loss function; Using the feature fusion module in the stroke order recognition network to fuse the handwritten Chinese character hidden layer features and the template Chinese character hidden layer features, obtain the difference features between the handwritten Chinese character hidden layer features and the template Chinese character hidden layer features, and obtain a second loss function; and The weights of the spatial transformation network and the stroke order recognition network are adjusted based on the first loss function and the second loss function to obtain trained spatial transformation network and stroke order recognition network.

10. A system for recognizing the stroke order of handwritten Chinese characters, characterized in that: The system comprises: A processor configured to use one or more neural networks to: Preprocessing the acquired trajectory point sequence of the handwritten Chinese character to obtain a key point sequence of the handwritten Chinese character, wherein the key points in the key point sequence are determined based on the curvature angle corresponding to each trajectory point and the distance between each trajectory point and the previous trajectory point; Extracting stroke features of the handwritten Chinese character based on the key point sequence of the handwritten Chinese character, wherein the stroke features include basic features and semantic features; and The stroke features of the handwritten Chinese characters and the stroke features of the corresponding template Chinese characters are processed together by the stroke order recognition model in the neural network to obtain a stroke order recognition result, wherein the stroke order recognition model includes a stroke order recognition network, and the stroke order recognition network includes a hidden layer feature extraction module, a feature fusion module and a stroke order recognition module, wherein the hidden layer feature extraction module is used to extract the handwritten Chinese character hidden layer features of the handwritten Chinese characters and the template Chinese character hidden layer features of the template Chinese characters from the stroke features of the handwritten Chinese characters and the stroke features of the template Chinese characters, respectively, and A memory for storing network parameters of the one or more neural networks; The basic features include coordinates and writing time of each key point on the stroke segment, and the semantic features include one or more of Bezier features, path signature features, distance between key points, curvature, and stroke direction features of the stroke segment; The feature fusion module is used to fuse the extracted hidden features of the handwritten Chinese characters and the hidden features of the template Chinese characters to obtain the difference features between the hidden features of the handwritten Chinese characters and the hidden features of the template Chinese characters.

11. A computing device, characterized in that: The computing device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method according to any one of claims 1 to 9.

12. A computer-readable medium, characterized in that The computer readable medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 9 is executed.

Images