Text recognition method, answer text verification method, device, equipment and medium

Abstract

This invention provides a text recognition method, an answer text verification method, an apparatus, a device, and a medium. The text recognition method includes: acquiring a target trajectory point sequence; determining a first stroke feature and a second stroke feature corresponding to the text to be recognized based on the target trajectory point sequence; fusing the first and second stroke features to obtain fused features; and recognizing the text to be recognized by recognizing the fused features. This invention acquires first and second stroke features containing the text to be recognized, and fuses the first and second stroke features to achieve the fusion of offline image features on the basis of dynamic handwriting trajectory. This results in fused features containing both the temporal features of the dynamic handwriting trajectory and offline image features. Furthermore, by strongly fusing the temporal information and offline image information before the text recognition result, the accuracy of text recognition can be significantly improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and more specifically, the embodiments of the present invention relate to a text recognition method, an answer text verification method, an apparatus, a device, and a medium. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention as set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] In applications such as smart learning hardware and software products, for example, tablets with touchscreens allow users to collect information using their fingers or styluses; smart writing pens collect information using cameras and dot-matrix paper. The information collected by these smart hardware products is essentially a set of trajectory points containing coordinates, pressure values, and timestamps. Converting these trajectory points into editable text requires handwritten text recognition technology. Similarly, scanners or digital cameras use optical methods to convert handwritten text in paper documents into static image files; converting the text information stored in these image files into editable text requires handwritten text recognition technology; this text is not limited to words but also includes formulas. Summary of the Invention

[0004] However, due to the low accuracy of existing handwritten text recognition technologies, from an input perspective, handwritten text recognition can be divided into online handwriting recognition and offline handwriting recognition, depending on the acquisition device. Offline handwriting recognition is based on static images acquired through methods such as photography or scanning, possessing global visual information but lacking temporal information; online handwriting recognition is based on dynamic trajectory point sequences acquired by smart devices, containing rich temporal information, but requires temporal accuracy and is easily affected by issues such as the writing order.

[0005] To address this issue, a handwritten character recognition method integrating offline and online recognition methods has been proposed in related technologies. The offline handwritten character recognition method determines offline candidate characters by extracting the directional angular features of the Chinese character outlines and using flexible template matching. The online handwritten character recognition method determines online candidate characters by extracting the directional features of the Chinese character strokes and using flexible template matching. Finally, the recognition result is determined based on the matching scores of the offline and online candidate characters, reflecting the integration of two types of information.

[0006] Another approach proposes a handwritten character recognition method based on stroke order and OCR. This approach first acquires an offline text image using an OCR acquisition device and inputs it into a preset OCR recognition algorithm to obtain preliminary character recognition results. Then, it compares and matches the online trajectory information acquired by the OCR acquisition device with a preset character database to correct the preliminary character recognition results and thus determine the final recognition result. This also reflects the integration of two types of information.

[0007] Therefore, in the existing technology, the integration process of online trajectory information and offline image information is weakly coupled. This integration method makes the transmission of the two types of information relatively independent, with each completing the candidate text recognition process independently. They only interact when determining the final recognition result, which is a very troublesome process.

[0008] Therefore, there is a great need for an improved handwritten text recognition method to increase the accuracy of handwritten text recognition.

[0009] In this context, embodiments of the present invention aim to provide a text recognition method, an answer text verification method, an apparatus, a device, and a medium.

[0010] In a first aspect of the present invention, a text recognition method is provided, the method comprising:

[0011] Obtain a target trajectory point sequence, wherein the target trajectory point sequence includes multiple and continuous trajectory points collected from the dynamic handwriting of the text to be identified;

[0012] Based on the target trajectory point sequence, the first stroke feature corresponding to the text to be identified and the second stroke feature corresponding to the text to be identified are determined, wherein the first stroke feature contains at least the temporal features of writing the text to be identified, and the second stroke feature is an image feature extracted from a text image containing the text to be identified.

[0013] The first stroke feature and the second stroke feature are fused to obtain the fused feature;

[0014] The text to be identified is identified by recognizing the fusion features.

[0015] In another embodiment of the present invention, determining the first stroke feature corresponding to the text to be identified includes:

[0016] By extracting features from the target trajectory point sequence, multiple stroke feature sequences corresponding to multiple strokes of the text to be identified are obtained. Each stroke includes multiple trajectory points, and the stroke feature sequence corresponding to each stroke includes the position information and timestamp of all trajectory points.

[0017] The multiple stroke feature sequences are input into an LSTM-based prediction model for time-series learning to obtain the first stroke feature.

[0018] In another embodiment of the present invention, the method of extracting features from the target trajectory point sequence to obtain multiple stroke feature sequences corresponding to multiple strokes of the text to be identified includes:

[0019] Input the target trajectory point sequence into the Bessel model;

[0020] Based on the location information and timestamp of each trajectory point, all strokes corresponding to the target trajectory point sequence are fitted using Bézier curves.

[0021] The fitting parameters corresponding to each stroke are used as the stroke feature sequence of the corresponding stroke.

[0022] Output the stroke feature sequences corresponding to all strokes.

[0023] In another embodiment of the present invention, fitting all strokes corresponding to the target trajectory point sequence using Bézier curves includes:

[0024] For any target stroke among all strokes, if fitting the target stroke fails, determine the target curve corresponding to the target stroke;

[0025] Starting from the starting point of the target curve, extract a starting line segment at a preset distance from the target curve;

[0026] Fit the strokes corresponding to the starting line segment and calculate the fitting error corresponding to the starting line segment;

[0027] If the fitting error is less than or equal to the threshold, continue the fitting operation of the next line segment in the target curve excluding the starting line segment; if the fitting error is greater than the threshold, split the starting line segment when the curvature of the starting line segment reaches its maximum value, and continue fitting the starting line segment after the split starting line segment.

[0028] In another embodiment of the present invention, determining the second stroke feature corresponding to the text to be identified includes:

[0029] The text image is generated by rendering the target trajectory point sequence, wherein the text image includes the text to be identified;

[0030] The text image is input into a convolutional neural network to output the image features corresponding to the text image;

[0031] The second stroke feature is generated based on the Bézier curve corresponding to each stroke and the image features.

[0032] In another embodiment of the present invention, generating the text image by rendering the target trajectory point sequence includes:

[0033] The text image is generated by sequentially connecting the timestamps and location coordinates of each trajectory point in the target trajectory point sequence.

[0034] In another embodiment of the present invention, generating the second stroke feature based on the Bézier curve corresponding to each stroke and the image features includes:

[0035] Generate a text box corresponding to each stroke based on the Bézier curve corresponding to each stroke;

[0036] Based on the text boxes corresponding to each stroke, determine the original image size corresponding to each stroke;

[0037] Based on the original image size corresponding to each stroke, the image size information corresponding to the image feature is adjusted to obtain the second stroke feature.

[0038] In another embodiment of the present invention, the feature fusion of the first stroke feature and the second stroke feature to obtain the fused feature includes:

[0039] Associate the first stroke feature with the second stroke feature so that the first stroke feature corresponds to the second stroke feature;

[0040] Based on the stroke dimension, the first stroke feature and the second stroke feature are concatenated to obtain the fused feature.

[0041] In another embodiment of the present invention, identifying the text to be identified by recognizing the fusion features includes:

[0042] The fused features are input into a bidirectional recurrent neural network based on LSTM to obtain the prediction result for each stroke.

[0043] The prediction results of all strokes are combined to obtain the text to be identified.

[0044] In another embodiment of the present invention, the fused features are input into a bidirectional recurrent neural network based on LSTM to obtain a prediction result for each stroke, including:

[0045] The fused features and reference information are input into the bidirectional recurrent neural network, wherein the reference information is used to verify the file to be identified;

[0046] Based on the reference information, determine the verification value corresponding to each stroke;

[0047] Based on the verification value corresponding to each stroke, calculate the prediction result for each stroke.

[0048] In a second aspect of the present invention, a method for verifying answer text is provided, the method comprising:

[0049] Obtain the dynamic trajectory point sequence corresponding to the target answer text and the reference answer text corresponding to the target question, wherein the target answer text is associated with the target question;

[0050] Using the dynamic trajectory point sequence as the target trajectory point sequence and the reference answer text as reference information, the target answer text is verified using any one of the methods described in the first aspect above.

[0051] In a third aspect of the present invention, a text recognition device is provided, the device comprising:

[0052] An acquisition module is used to acquire a target trajectory point sequence, wherein the target trajectory point sequence includes multiple and continuous trajectory points collected from the dynamic handwriting of the text to be recognized;

[0053] The determining module is used to determine the first stroke feature corresponding to the text to be identified and the second stroke feature corresponding to the text to be identified based on the target trajectory point sequence, wherein the first stroke feature contains at least the temporal features of writing the text to be identified, and the second stroke feature is an image feature extracted from a text image containing the text to be identified.

[0054] The fusion module is used to fuse the first stroke feature and the second stroke feature to obtain the fused feature;

[0055] The recognition module is used to recognize the text to be recognized by recognizing the fused features.

[0056] In one embodiment of the present invention, the determining module includes:

[0057] The feature extraction unit is used to extract features from the target trajectory point sequence to obtain multiple stroke feature sequences corresponding to multiple strokes of the text to be identified, wherein each stroke includes multiple trajectory points, and the stroke feature sequence corresponding to each stroke includes the position information and timestamp of all trajectory points.

[0058] The first learning unit is used to input the multiple stroke feature sequences into an LSTM-based prediction model for time-series learning to obtain the first stroke feature.

[0059] In another embodiment of the present invention, the feature extraction unit includes:

[0060] The input subunit is used to input the target trajectory point sequence into the Bessel model;

[0061] The fitting subunit is used to fit all strokes corresponding to the target trajectory point sequence using Bézier curves based on the position information and timestamp of each trajectory point.

[0062] The first determining subunit is used to fit the fitting parameters corresponding to each stroke as the stroke feature sequence of the corresponding stroke.

[0063] The output sub-unit is used to output the stroke feature sequence corresponding to all strokes.

[0064] In yet another embodiment of the invention, the fitting subunit is used for:

[0065] For any target stroke among all strokes, if fitting the target stroke fails, determine the target curve corresponding to the target stroke;

[0066] Starting from the starting point of the target curve, extract a starting line segment at a preset distance from the target curve;

[0067] Fit the strokes corresponding to the starting line segment and calculate the fitting error corresponding to the starting line segment;

[0068] If the fitting error is less than or equal to the threshold, continue the fitting operation of the next line segment in the target curve excluding the starting line segment; if the fitting error is greater than the threshold, split the starting line segment when the curvature of the starting line segment reaches its maximum value, and continue fitting the starting line segment after the split starting line segment.

[0069] In yet another embodiment of the present invention, the determining module includes:

[0070] A rendering unit is configured to generate the text image by performing trajectory rendering on the target trajectory point sequence, wherein the text image includes the text to be recognized;

[0071] The second learning unit is used to input the text image into the convolutional neural network to output the image features corresponding to the text image;

[0072] The generation unit is used to generate the second stroke feature based on the Bézier curve corresponding to each stroke and the image features.

[0073] In yet another embodiment of the present invention, the rendering unit includes:

[0074] The first generation subunit is used to sequentially connect each trajectory point according to the timestamp and position coordinates of each trajectory point in the target trajectory point sequence to generate the text image.

[0075] In yet another embodiment of the present invention, the generation unit includes:

[0076] The second generation subunit is used to generate a text box corresponding to each stroke based on the Bézier curve corresponding to each stroke.

[0077] The first determining subunit is used to determine the original image size corresponding to each stroke based on the text box corresponding to each stroke;

[0078] The adjustment subunit is used to adjust the image size information corresponding to the image feature according to the original image size corresponding to each stroke, so as to obtain the second stroke feature.

[0079] In one embodiment of the present invention, the fusion module includes:

[0080] An association unit is used to associate the first stroke feature with the second stroke feature so that the first stroke feature corresponds to the second stroke feature;

[0081] The splicing unit is used to splice the first stroke feature and the second stroke feature based on the stroke dimension to obtain the fused feature.

[0082] In one embodiment of the present invention, the identification module includes:

[0083] The prediction unit is used to input the fused features into a bidirectional recurrent neural network based on LSTM to obtain the prediction result for each stroke.

[0084] The recognition unit is used to combine the prediction results of all strokes to obtain the text to be recognized.

[0085] In one embodiment of the present invention, the prediction unit includes:

[0086] An input subunit is used to input the fused features and reference information into the bidirectional recurrent neural network, wherein the reference information is used to verify the file to be identified;

[0087] The third determining subunit is used to determine the verification value corresponding to each stroke based on the reference information;

[0088] The calculation subunit is used to calculate the prediction result for each stroke based on the verification value corresponding to each stroke.

[0089] In a fourth aspect of the present invention, an apparatus for verifying answer text is provided, the apparatus comprising:

[0090] The acquisition module is used to acquire the dynamic trajectory point sequence corresponding to the target answer text and the reference answer text corresponding to the target question, wherein the target answer text is associated with the target question;

[0091] The verification module is used to verify the target answer text using the dynamic trajectory point sequence as the target trajectory point sequence and the reference answer text as reference information, through the device described in the third aspect above.

[0092] In a fifth aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps described in the method embodiments above.

[0093] In a sixth aspect of the present invention, a storage medium is provided that stores a computer program, wherein the computer program is configured to execute the steps described in the apparatus embodiments at runtime.

[0094] According to the text recognition method of the present invention, a target trajectory point sequence is collected from dynamic handwriting, and a first stroke feature and a second stroke feature of the text to be recognized are obtained. The first stroke feature contains the temporal features of the dynamic handwriting, and the second stroke feature contains the offline image features of the text image. Then, the first stroke feature and the second stroke feature are fused together, thereby realizing the fusion of offline image features on the basis of dynamic handwriting trajectory. This makes the fused features not only contain the temporal information of the text to be recognized, but also the offline image information of the text to be recognized. Before the text recognition result is obtained, the temporal information and the offline image information have been strongly fused together. The two types of information are shared and complementary, which can significantly improve the accuracy of text recognition. Attached Figure Description

[0095] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0096] Figure 1 An application scenario according to an embodiment of the present invention is illustrated schematically;

[0097] Figure 2 A flowchart illustrating a text recognition method according to an embodiment of the present invention is shown schematically;

[0098] Figure 3 A schematic diagram of the architecture of a handwritten text recognition method according to another embodiment of the present invention is shown.

[0099] Figure 4 A schematic diagram of a Bézier curve fitting stroke according to yet another embodiment of the present invention is shown.

[0100] Figure 5 A flowchart illustrating a method for verifying answer text according to the present invention is shown schematically.

[0101] Figure 6 A text recognition device according to an embodiment of the present invention is illustrated schematically;

[0102] Figure 7 A device for verifying answer text according to an embodiment of the present invention is illustrated schematically;

[0103] Figure 8 A schematic diagram of the structure of an electronic device according to another embodiment of the present invention is shown.

[0104] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0105] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art. Invention Overview

[0107] The inventors have discovered that in existing solutions, such as handwritten text recognition methods based on the integration of offline and online recognition methods, or handwritten text recognition methods based on stroke order and OCR, although both methods use online trajectory information and offline image information, the integration process of online trajectory information and offline image information is weakly coupled. This integration means that the transmission of the two types of information is relatively independent, and each completes the recognition process of candidate characters independently. They only interact when determining the final recognition result, and the recognition result of handwritten text is not optimal.

[0108] To address this, the present invention provides a text recognition method, which involves acquiring a target trajectory point sequence from dynamic handwriting; determining, based on the target trajectory point sequence, the first stroke feature corresponding to the text to be recognized, the temporal feature containing the handwriting, and the second stroke feature corresponding to the text to be recognized, i.e., the image feature of the offline text image; obtaining a fused feature by fusing the first stroke feature and the second stroke feature; and completing the recognition of the text to be recognized by recognizing the fused feature.

[0109] The text recognition method used in this invention extracts the temporal information of the handwriting and the offline image information of the handwriting based on the trajectory point sequence on the dynamic handwriting. Before performing text recognition, the temporal information and the offline image information are highly coupled, which can significantly improve the accuracy of text recognition.

[0110] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0111] Application Scenarios Overview

[0112] First, refer to Figure 1 , Figure 1 The application scenario according to an embodiment of the present invention is illustrated schematically. First, refer to... Figure 1 This illustrates an application scenario in which embodiments of the present invention can be implemented. Figure 1 The scenario shown includes terminal 200 and server 100. Terminal 200 can be a mobile terminal, such as a mobile phone, tablet, laptop, personal digital assistant, or desktop computer. Server 100 can be a software operator's server. Terminal 200 and server 100 can communicate via, for example, mobile internet.

[0113] After obtaining the target trajectory point sequence, detection can be performed on the server 100 side or on the terminal 200 side. More specifically, for example, a software product is installed on the terminal 200. The software product determines the first stroke feature and the second stroke feature of the text to be recognized based on the target trajectory point sequence. Then, the first stroke feature and the second stroke feature are fused to obtain the fused feature. By recognizing the fused feature, the text to be recognized is recognized. This realizes the text recognition process by coupling offline features in a highly coupled manner on the basis of dynamic trajectory.

[0114] Exemplary methods

[0115] The following is combined with Figure 1 Application scenarios, refer to Figure 2 The text recognition method according to an exemplary embodiment of the present invention will be described.

[0116] It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present invention, and the embodiments of the present invention are not limited in any way. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0117] This invention is mainly applied to handwritten text recognition, human-computer interaction, computer vision, etc., and its main application areas are intelligent learning hardware or software products.

[0118] For example, see Figure 2The diagram shown is a flowchart of a text recognition method according to an embodiment of the present invention. Figure 2 As shown, the text recognition method may include the following steps:

[0119] Step S202: Obtain the target trajectory point sequence, wherein the target trajectory point sequence includes multiple and continuous trajectory points collected from the dynamic handwriting of the text to be recognized;

[0120] In this embodiment, the target trajectory point sequence is a dynamic trajectory point sequence of handwriting trajectory, including multiple and continuous handwriting trajectory points. A large number of trajectory points can represent the strokes of the text to be recognized. The text to be recognized can be handwritten text, which is not limited to words but also includes formulas, and is not limited here.

[0121] In specific embodiments, tablet-type hardware products with touchscreens can allow users to collect information using their fingers or styluses; smart writing pen-type hardware products can collect information using camera devices and dot matrix paper, etc.

[0122] For example, when a user writes on an online smart device using their finger or a stylus, each stroke is input by completing a single pen-to-pen movement. This stroke consists of several trajectory points containing information such as coordinates, pressure values, and timestamps. Multiple pen-to-pen movements result in the input of several strokes, which together form several characters. It is understood that the input in this embodiment of the invention is a sequence of trajectory points, composed of several stroke sequences, and each stroke sequence is composed of several trajectory point sequences.

[0123] Optionally, if the pressure value information is not used in subsequent modules, {(x i ,y i ,t i )} represents the input sequence of trajectory points. Assume there are N trajectory points and M strokes. In general, N >> M.

[0124] Step S204: Based on the target trajectory point sequence, determine the first stroke feature corresponding to the text to be identified and the second stroke feature corresponding to the text to be identified.

[0125] Among them, the first stroke feature contains at least the temporal features of writing the text to be identified, and the second stroke feature is the image feature extracted from the text image containing the text to be identified;

[0126] Optionally, after obtaining the dynamic trajectory point sequence, the dynamic trajectory point sequence is sent to the online handwriting feature extraction module and the offline handwriting feature extraction module to extract the online information of the handwritten text, namely the temporal features of the dynamic handwriting of the handwritten text, and the offline information of the handwritten text, namely the offline image features of the final text image of the handwritten text.

[0127] Step S206: Perform feature fusion on the first stroke feature and the second stroke feature to obtain the fused feature;

[0128] Preferably, after obtaining online and offline information through the above steps, the online and offline information are fused in a highly coupled manner.

[0129] Step S208: Identify the text to be identified by recognizing fusion features.

[0130] According to the method of the present invention, a target trajectory point sequence is acquired from dynamic handwriting, and a first stroke feature and a second stroke feature of the text to be recognized are obtained. The first stroke feature contains the temporal features of the dynamic handwriting, and the second stroke feature contains the offline image features of the text image. Then, the first stroke feature and the second stroke feature are fused together, thereby realizing the fusion of offline image features on the basis of dynamic handwriting trajectory. This makes the fused features not only contain the temporal information of the text to be recognized, but also the offline fragment information of the text to be recognized. Before the text recognition result is obtained, the temporal information and offline image information have been strongly fused together. The two types of information are shared and complementary, which can significantly improve the accuracy of text recognition.

[0131] The present invention will be further described below with reference to specific embodiments:

[0132] Figure 3 The diagram schematically illustrates the architecture of a handwritten text recognition method according to yet another embodiment of the present invention, such as... Figure 3 As shown, the online handwriting feature extraction module uses a Bezier feature extraction module and a Long Short-Term Memory (LSTM) network, while the offline handwriting feature extraction module uses a rendering module and a convolutional neural network (such as ResNet).

[0133] Combination Figure 3 In a preferred embodiment of the provided architecture diagram, step S204, which determines the first stroke feature corresponding to the text to be identified, includes the following steps:

[0134] Step S301: By extracting features from the target trajectory point sequence, multiple stroke feature sequences corresponding to multiple strokes in the text to be identified are obtained. Each stroke includes multiple trajectory points, and the stroke feature sequence corresponding to each stroke includes the position information and timestamp of all trajectory points.

[0135] In this embodiment, the target trajectory point sequence {(x i ,y i ,t i Input the Bézier feature extraction module, which is responsible for fitting M strokes corresponding to N trajectory point sequences with Bézier curves and using the parameters of the Bézier curve fitting as the Bézier features corresponding to the strokes.

[0136] Specifically, step S301 above includes the following steps:

[0137] Step S401: Input the target trajectory point sequence into the Bessel model;

[0138] Step S402: Based on the position information and timestamp of each trajectory point, use Bézier curves to fit all strokes corresponding to the target trajectory point sequence;

[0139] In this embodiment, the Bezier curve is a curve applied to two-dimensional graphics. The curve consists of endpoints and control points; the shape of the curve can be changed by altering the coordinates of the control points. For example, Figure 4 A schematic diagram of a Bézier curve fitting stroke according to yet another embodiment of the present invention is shown. Figure 4 It includes two endpoints, the starting point and the ending point of the stroke, as well as two control points, d1 and d2.

[0140] Further, step S402 above includes the following steps: for any target stroke among all strokes, if fitting the target stroke fails, determine the target curve corresponding to the target stroke; starting from the starting point of the target curve, extract the starting line segment at a preset distance from the target curve; fit the stroke corresponding to the starting line segment and calculate the fitting error corresponding to the starting line segment; if the fitting error is less than or equal to a threshold, continue the fitting operation of the next line segment in the target curve excluding the starting line segment; if the fitting error is greater than the threshold, split the starting line segment at the maximum curvature value corresponding to the starting line segment, and continue fitting the starting line segment after the split starting line segment.

[0141] For example, see reference. Figure 4, the embodiments of the present invention use cubic Bezier curves to fit the strokes. Specifically, when a Bezier curve cannot perfectly fit a stroke, the stroke needs to be split into multiple segments for fitting. The specific fitting method is as follows: Starting from the starting point, search backward until the curve length is greater than 1.5 times the straight-line distance between the stroke endpoints, then split, and use this sequence to fit a Bezier curve. If the fitting error of the curve is lower than the threshold, continue to search backward for the next segment in this way; otherwise, split at the point with the maximum curvature, fit the first half, and then check whether the fitting error meets the conditions.

[0142] Step S403, use the fitting parameters corresponding to each fitted stroke as the stroke feature sequence of the corresponding stroke;

[0143] Step S404, output the stroke feature sequences corresponding to all strokes.

[0144] Overall, it is a greedy idea. For example, for the character "对" in Figure 3 , the first stroke is split and refitted at the point with the maximum curvature, that is, the turning point. The embodiments of the present invention extract 12-dimensional features from the Bezier curve corresponding to each stroke. For example, as shown in Figure 4 , the d x , d y features that characterize the direction and distance between two endpoints, the d1, d2, α1, α2 features that characterize the distance and angle from the control point to the endpoint, etc., are used as the Bezier feature representation of this stroke, denoted as {(w j,1 , w j,2 , …, w j,12 )}.

[0145] Step S302, input multiple stroke feature sequences into a prediction model based on LSTM for temporal learning to obtain the first stroke feature.

[0146] Finally, input the Bezier features corresponding to all strokes into a bidirectional recurrent neural network. In the embodiments of the present invention, a long short-term memory network (LSTM) is used to learn the temporal information in the strokes. Each step of the encoding uses the Bezier feature of the current stroke and the hidden layer vector representation of the previous step as inputs to obtain the hidden layer vector representation of the current step, which is the high-dimensional online feature (i.e., the above-mentioned first stroke feature) representation of the stroke.

[0147] Combined with Figure 3 shown, in another embodiment of this case, the above step S204 to determine the second stroke feature corresponding to the text to be recognized includes the following steps:

[0148] Step S501, generate a text image by performing trajectory rendering on the target trajectory point sequence, where the text image includes the text to be recognized;

[0149] In specific implementation, step S501 includes: connecting each trajectory point sequentially based on the timestamp and position coordinates of each trajectory point in the target trajectory point sequence to generate a text image.

[0150] For example, the input trajectory point sequence is sent to the trajectory rendering module. This module is responsible for connecting the trajectory points in each stroke sequence according to the timestamp of the trajectory points, using line segments of fixed width, and according to the coordinate values, to obtain the offline handwritten image (i.e., the text image mentioned above).

[0151] Step S502: Input the text image into the convolutional neural network to output the image features corresponding to the text image;

[0152] In practice, the handwritten image is input into a convolutional neural network (ResNet is used in this patent) to learn the visual information in the image and encode it into high-dimensional image feature maps. During this process, the downsampling operation in the convolutional neural network will result in the original image's dimensions being different from those of the feature maps.

[0153] Step S503: Generate the second stroke feature based on the Bézier curve and image features corresponding to each stroke.

[0154] In specific implementation, step S503 includes: generating a text box corresponding to each stroke based on the Bézier curve corresponding to each stroke; determining the original image size corresponding to each stroke based on the text box corresponding to each stroke; and adjusting the image size information corresponding to the image features based on the original image size corresponding to each stroke to obtain the second stroke feature.

[0155] Combination Figure 3 For example, based on the Bézier curves fitted to each stroke in the online handwriting feature extraction module, an outer bounding box (i.e., the text box mentioned above) can be obtained to characterize the stroke boundaries. This generates a minimum bounding box that encloses every point on the Bézier curve, described using the coordinates of its top-left and bottom-right corners. This bounding box corresponds to the original image scale and needs to be downsampled using the same downsampling factor as in the convolutional neural network to align with the scale of the feature maps.

[0156] Finally, the downsampled bounding boxes are used to outline the image feature representation corresponding to each stroke in the Feature Maps. Because each stroke is of different size, the bounding boxes also vary in size, resulting in different scales for the image feature representations corresponding to each stroke. Therefore, the ROI Align operation is needed to map the image feature representations of strokes at different scales to the same scale, thus obtaining the offline feature representation for each stroke.

[0157] Through the above steps, offline and online information are exchanged before the recognition result is determined. The handwritten text recognition process is completed through multimodal means, thereby improving the accuracy of text recognition.

[0158] Furthermore, in an optional embodiment of this case, step S206 above includes the following steps:

[0159] Step S601: Input the fused features into a bidirectional recurrent neural network based on LSTM to obtain the prediction result for each stroke.

[0160] Step S602: Combine the prediction results of all strokes to obtain the text to be recognized.

[0161] See Figure 3 As shown, in practical implementation, due to the mismatch between the number of trajectory points input by the online module and the number of image pixels input by the offline module, direct alignment and fusion are difficult. This patent cleverly introduces stroke information as a bridge, making the online stroke feature representation extracted by the online handwriting feature extraction module and the offline stroke feature representation extracted by the offline handwriting feature extraction module correspond one-to-one.

[0162] During the encoding stage, the feature representations of the two modalities are spliced ​​and fused according to the stroke dimension. That is, the stroke information is modeled in a multimodal way, which learns both the rich temporal information from online trajectory points and the global visual information from offline images. The two types of information complement each other and are jointly input into the decoding module for text content prediction.

[0163] The decoding module in the recognition process uses a Bi-LSTM-based bidirectional recurrent neural network. Generally, each step of the decoding process takes the stroke feature representation of the current step and the hidden vector representation of the previous step as input to obtain the hidden vector representation of the current step. Then, the hidden vector representation is input into the softmax layer to obtain the probability distribution representation over the vocabulary. The character with the highest probability is the prediction result of the current step, calculated using the following formula:

[0164] target=argmax{softmax{logits} i};

[0165] The final handwritten trajectory recognition result is obtained by combining the prediction results of all steps.

[0166] Through the above implementation steps, the multimodal handwriting recognition method proposed in this embodiment of the invention brings the following beneficial effects:

[0167] (1) By using Bézier curves, the online recognition scheme based on trajectory point granularity is improved to the online recognition scheme based on stroke granularity; not only can the influence of abnormal trajectory points on recognition be eliminated, but the feature dimension that needs to be modeled can also be reduced (the number of trajectory points is generally much greater than the number of strokes), thereby reducing the number of model parameters and improving decoding efficiency; thus improving the effect of text recognition.

[0168] (2) By using strokes as a bridge, online trajectory information and offline image information are fused and interacted during the encoding stage. The two types of information can complement each other. Compared with single-modal recognition schemes or schemes that only fuse recognition results in the final decoding stage, the input information can be modeled more comprehensively, thereby improving the accuracy of handwritten text recognition.

[0169] Based on the text recognition methods provided in the above embodiments, and based on the same inventive concept, this embodiment also provides a method for recognizing question answers. This method is used to implement the above embodiments and preferred embodiments, and will not be repeated hereafter.

[0170] From an application perspective, handwritten text recognition, depending on the problem it solves, includes text transcription problems where a given input (image or trajectory) is returned as a recognition result; and comparison and verification problems where a given input and a reference answer are provided.

[0171] For example, the existing solution 1 is a grading method based on English dictation text images and reference answers. It inputs an English dictation text image, obtains the original recognition result of the dictation text through a preset text detection and text recognition processing method, then replaces it according to a preset table of easily confused characters, and obtains a list of results to be replaced (including the original recognition result); finally, it judges whether the dictation result is correct by comparing it with the reference answer.

[0172] For example, the existing solution 2: using model prediction to implement a method for correcting easily confused answers. First, input the handwritten formula, and obtain the predicted LaTeX formula result through a preset OCR recognition model; then, confuse it according to a preset list of easily confused characters to obtain a list of easily confused formulas; finally, compare it with the reference answer to determine whether the answer is correct.

[0173] The existing solution 3 is a text recognition method for answering questions based on answering images and question information (including at least one of the question stem, reference answer text, and analysis text). First, features are extracted from the answering images and question information to obtain visual feature encoding information and text feature encoding information. Then, the two types of information are fused and decoded to obtain the text recognition result of the answering images.

[0174] As can be seen, Schemes 1 and 2, relying on a pre-defined list of easily confused characters, only use the reference answer at a simple "hard" matching level, affecting the accuracy of the recognition results. Scheme 3 uses additional question information that is implicitly encoded as uninterpretable feature information, which is deeply involved in the text recognition process. This makes the recognition process of the answer text "excessively" affected by the additional question information, thus generating recognition results that deviate from the answer image. That is, there may be cases where the answer is correct but is recognized as an incorrect answer, and the number of model parameters increases significantly.

[0175] In other words, the verification of the answer recognition results in the relevant technologies suffers from low accuracy.

[0176] To address the aforementioned technical problems, this embodiment provides a method for identifying the answer to a question. Figure 5 A flowchart illustrating a method for verifying answer text according to the present invention is shown schematically, such as... Figure 5 As shown, the process includes the following steps:

[0177] Step S701: Obtain the dynamic trajectory point sequence corresponding to the target answer text and the reference answer text corresponding to the target question, wherein the target answer text is associated with the target question;

[0178] Step S703: Using the dynamic trajectory point sequence as the target trajectory point sequence and the reference answer text as reference information, the target answer text is verified using the above text recognition method.

[0179] The answer text verification method provided in this invention obtains the dynamic trajectory point sequence of the target answer text. Using the aforementioned text recognition method, it obtains the offline answer image corresponding to the target answer text and the temporal information of the handwriting used to write the target answer. Information fusion and interaction are performed on the offline answer image and temporal information, enabling the fusion of offline answer images in a highly coupled manner based on online trajectory input to jointly complete the answer text recognition process, thereby improving the accuracy of answer text recognition. Furthermore, by incorporating reference answer text into the target answer text recognition process, the recognition results are automatically corrected, improving the recognition accuracy of answer text (e.g., formulas), and thus enhancing the verification accuracy.

[0180] In this embodiment of the invention, the reference answer is not implicitly encoded, but is explicitly integrated into the text recognition process. During the decoding process, a verification strategy based on the reference answer is introduced to supervise and automatically correct the decoding result of the answer text. This prevents the situation where a correct answer is identified as an incorrect answer, and the number of model parameters remains basically unchanged.

[0181] Combination Figure 3In one specific embodiment, a solution strategy is provided for decoding in an application scenario given an input trajectory and a reference answer. In this scenario, the reference answer, as additional information (i.e., the aforementioned reference information), plays a crucial guiding role in the decoding process of the input trajectory. For example, if the reference answer to a question is "2", and the user inputs a trajectory point sequence with the answer "2", but due to improper handwriting, the model predicts "z", then the service will give a "wrong answer" judgment. However, if the recognition model knows in advance that the reference answer to the question is "2" (which is very likely a question testing numerical calculation), it can correct the decoding process, thereby giving a prediction result of "2" and avoiding misjudgment by the service.

[0182] This invention proposes a verification strategy for automatically correcting recognition results based on reference answers. The specific operation is as follows:

[0183] The first step is to categorize the characters into the following main types based on the principle of "the correct or incorrect answer characters that may appear in the same type of questions":

[0184]

[0185] The second step involves obtaining a possible character set (Charset) for a given reference answer, referring to the character classification table from the first step. During each decoding step, an additional reward value (∈) is added to the confidence score (i.e., the softmax probability value) of characters belonging to this character set. The calculation formula is as follows:

[0186] target=argmax{softmax{logits} i +∈χ{i∈Charset}}}

[0187] In the example above, the model predicts "z", indicating that "z" has the highest probability value in the probability distribution of the vocabulary, assumed to be 0.5. Considering the irregularities of handwriting, the model predicts that the probability value of "2" will not be too low, assumed to be 0.4, and the probability values ​​of other characters are even lower. When the model knows in advance that the reference answer is "2", it can obtain a set of possible answer characters based on the character classification table. Then, the probability value of each character belonging to this character set will be given an extra reward value, assumed to be 0.2; after the verification strategy, the model will give a new prediction result of "2", thereby avoiding misjudgment of the recognition result.

[0188] In summary, the verification strategy proposed in this invention can explicitly integrate reference answer information into the decoding process of the answer text, automatically correct the recognition results, improve the recognition accuracy of text (e.g., formulas), and thus improve the verification accuracy.

[0189] Furthermore, the aforementioned decoding and verification strategy based on reference answers explicitly integrates the reference answers into the decoding process of the answer text, based on a predefined character classification table. The character classification table is simple and easy to understand, requiring minimal domain knowledge; and the introduction of the reference answers into the decoding process requires only a constant-level number of parameters; the entire verification strategy is simple and effective, improving both text (e.g., formula) recognition accuracy and verification accuracy.

[0190] It should be noted that the recurrent neural networks in the online feature extraction module and the decoding module of this invention are not limited to BiLSTM, but can replace BiGRU, or single-layer or multi-layer recurrent neural networks.

[0191] The convolutional neural network in the offline feature extraction module of this invention is not limited to ResNet; it can be replaced by any mature convolutional neural network, such as DenseNet, VGG, AlexNet, or a convolutional neural network with a custom structure.

[0192] The decoding module in this invention is not limited to using BiLSTM; it can be replaced with a more complex framework, such as an RNN+Attention decoding module with an attention mechanism. Furthermore, a covering mechanism, i.e., RNN+Attention+Coverage, can be added. The attention mechanism is responsible for calculating the importance between the hidden layer vector representation from the previous step and all stroke feature representations obtained from the feature fusion module, resulting in an importance matrix and a context vector, used to characterize which stroke feature(s) are helpful for the current symbol prediction. The covering mechanism focuses on the importance matrix from previously decoded moments, resulting in a coverage vector, used to characterize which strokes have been focused on and which have not. These are then incorporated into the sequence decoding process. This scheme performs better for recognizing handwritten formulas with structural information.

[0193] The feature fusion method in this invention is not limited to concatenating and fusing online and offline stroke feature representations during the encoding stage; it can also be performed during the decoding stage. For example, multimodal context feature fusion during the decoding stage involves inputting the online and offline stroke feature representations from the encoding stage into an RNN+Attention decoding module to obtain an Online Context Vector and an Offline Context Vector, which are then concatenated and fused together for use in the sequence decoding process. Another example is probabilistic fusion during the decoding stage, where the online and offline handwriting recognition modules each perform their encoding and decoding processes, and output the probability distribution representations on the vocabulary obtained from the softmax layer in the final prediction stage. These distributions are then weighted and fused, and the character with the highest probability is taken as the prediction result for the current step. The calculation formula is as follows:

[0194] target=argmax{αsoftmax{logits_online} i +(1-α)softmax{logits_offline} i}

[0195] The verification strategy in this invention is not limited to the decoding strategy based on a character classification table and reference answer guidance. It can also be replaced by a decoding strategy based on a table of similar characters (such as 2 and z, 1 and |, cross and multiplication signs, etc.) and reference answer guidance.

[0196] Exemplary device

[0197] After introducing the method of an exemplary embodiment of the present invention, the apparatus of an exemplary embodiment of the present invention will be described next.

[0198] Based on the text recognition methods provided in the above embodiments, and based on the same inventive concept, this embodiment also provides a text recognition device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0199] Figure 6A text recognition device according to an embodiment of the present invention is schematically illustrated. The device includes: an acquisition module 60, configured to acquire a target trajectory point sequence, wherein the target trajectory point sequence includes multiple and continuous trajectory points collected from the dynamic handwriting of the text to be recognized; a determination module 62, connected to the acquisition module 60, configured to determine a first stroke feature corresponding to the text to be recognized and a second stroke feature corresponding to the text to be recognized based on the target trajectory point sequence, wherein the first stroke feature contains at least temporal features of writing the text to be recognized, and the second stroke feature is an image feature extracted from a text image containing the text to be recognized; a fusion module 64, configured to fuse the first stroke feature and the second stroke feature to obtain a fused feature; and a recognition module 68, configured to recognize the text to be recognized by recognizing the fused feature.

[0200] Optionally, the determining module 62 includes: a feature extraction unit, used to extract features from the target trajectory point sequence to obtain multiple stroke feature sequences corresponding to multiple strokes in the text to be identified, wherein each stroke includes multiple trajectory points, and the stroke feature sequence corresponding to each stroke includes the position information and timestamp of all trajectory points; and a first learning unit, used to input the multiple stroke feature sequences into an LSTM-based prediction model for temporal learning to obtain a first stroke feature.

[0201] Optionally, the feature extraction unit includes: an input subunit for inputting the target trajectory point sequence into the Bezier model; a fitting subunit for fitting all strokes corresponding to the target trajectory point sequence using Bezier curves based on the position information and timestamp of each trajectory point; a first determination subunit for using the fitting parameters corresponding to each stroke as the stroke feature sequence of the corresponding stroke; and an output subunit for outputting the stroke feature sequence corresponding to all strokes.

[0202] Optionally, the fitting subunit is used to: for any target stroke among all strokes, if fitting the target stroke fails, determine the target curve corresponding to the target stroke; starting from the starting point of the target curve, extract a starting line segment at a preset distance from the target curve; fit the stroke corresponding to the starting line segment and calculate the fitting error corresponding to the starting line segment; if the fitting error is less than or equal to a threshold, continue the fitting operation of the next line segment in the target curve excluding the starting line segment; if the fitting error is greater than the threshold, split the starting line segment at the maximum curvature value corresponding to the starting line segment, and continue fitting the starting line segment after the split starting line segment.

[0203] Optionally, the determining module 62 includes: a rendering unit for generating a text image by rendering the target trajectory point sequence, wherein the text image includes the text to be recognized; a second learning unit for inputting the text image into a convolutional neural network to output image features corresponding to the text image; and a generation unit for generating second stroke features based on the Bézier curve and image features corresponding to each stroke.

[0204] Optionally, the rendering unit includes: a first generation subunit, used to sequentially connect each trajectory point according to the timestamp and position coordinates of each trajectory point in the target trajectory point sequence to generate a text image.

[0205] Optionally, the generation unit includes: a second generation subunit, used to generate a text box corresponding to each stroke based on the Bézier curve corresponding to each stroke; a first determination subunit, used to determine the original image size corresponding to each stroke based on the text box corresponding to each stroke; and an adjustment subunit, used to adjust the image size information corresponding to the image features based on the original image size corresponding to each stroke to obtain the second stroke features.

[0206] Optionally, the fusion module 66 includes: an association unit for associating the first stroke feature with the second stroke feature so that the first stroke feature corresponds to the second stroke feature; and a splicing unit for splicing the first stroke feature and the second stroke feature based on the stroke dimension to obtain the fused feature.

[0207] Optionally, the recognition module 68 includes: a prediction unit for inputting fused features into a bidirectional recurrent neural network based on LSTM to obtain a prediction result for each stroke; and a recognition unit for combining the prediction results of all strokes to obtain the text to be recognized.

[0208] Optionally, the prediction unit includes: an input subunit for inputting fused features and reference information into a bidirectional recurrent neural network, wherein the reference information is used to verify the document to be recognized; a third determination subunit for determining the verification value corresponding to each stroke based on the reference information; and a calculation subunit for calculating the prediction result corresponding to each stroke based on the verification value corresponding to each stroke.

[0209] Based on the answer text verification methods provided in the various embodiments above, and based on the same inventive concept, this embodiment also provides an answer text verification device. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0210] Figure 7 An apparatus for verifying answer text according to an embodiment of the present invention is illustrated schematically. The apparatus includes: an acquisition module 70, configured to acquire a dynamic trajectory point sequence corresponding to a target answer text and a reference answer text corresponding to a target question, wherein the target answer text is associated with the target question; and a verification module 72, connected to the acquisition module 70, configured to verify the target answer text by using the dynamic trajectory point sequence as the target trajectory point sequence and the reference answer text as reference information, through the aforementioned text recognition.

[0211] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0212] like Figure 8 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0213] Memory 113 is used to store computer programs;

[0214] In one embodiment of this application, the processor 111, when executing the program stored in the memory 113, implements the text recognition method or the answer text verification method provided in any of the foregoing method embodiments.

[0215] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the text recognition method provided in any of the foregoing method embodiments or the steps of the answer text verification method provided in any of the foregoing method embodiments.

[0216] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowchart may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0217] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that, when executed by a computer or other programmable data processing device, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0218] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0219] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0220] The use of the verbs "including" and "contains" and their inflections in the application documents does not preclude the existence of elements or steps other than those described in the application documents. The article "a" or "one" preceding an element does not preclude the existence of multiple such elements.

[0221] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A text recognition method, characterized by, The method includes: Obtain a target trajectory point sequence, wherein the target trajectory point sequence includes multiple and continuous trajectory points collected from the dynamic handwriting of the text to be identified; Based on the target trajectory point sequence, the first stroke feature corresponding to the text to be identified and the second stroke feature corresponding to the text to be identified are determined, wherein the first stroke feature contains at least the temporal features of writing the text to be identified, and the second stroke feature is an image feature extracted from a text image containing the text to be identified. The first stroke feature and the second stroke feature are fused to obtain the fused feature; The text to be identified is identified by recognizing the fusion features; The determination of the second stroke feature corresponding to the text to be identified includes: generating the text image by performing trajectory rendering on the target trajectory point sequence, the text image including the text to be identified; inputting the text image into a convolutional neural network to output the image features corresponding to the text image; and generating the second stroke feature based on the Bézier curve corresponding to each stroke and the image features.

2. The method of claim 1, wherein, The step of determining the first stroke feature corresponding to the text to be identified includes: By extracting features from the target trajectory point sequence, multiple stroke feature sequences corresponding to multiple strokes of the text to be identified are obtained. Each stroke includes multiple trajectory points, and the stroke feature sequence corresponding to each stroke includes the position information and timestamp of all trajectory points. The multiple stroke feature sequences are input into an LSTM-based prediction model for time-series learning to obtain the first stroke feature.

3. The method of claim 2, wherein, The step of extracting features from the target trajectory point sequence to obtain multiple stroke feature sequences corresponding to multiple strokes in the text to be identified includes: Input the target trajectory point sequence into the Bessel model; Based on the location information and timestamp of each trajectory point, all strokes corresponding to the target trajectory point sequence are fitted using Bézier curves. The fitting parameters corresponding to each stroke are used as the stroke feature sequence of the corresponding stroke. Output the stroke feature sequences corresponding to all strokes.

4. The method of claim 3, wherein, The step of fitting all strokes corresponding to the target trajectory point sequence using Bézier curves includes: For any target stroke among all strokes, if fitting the target stroke fails, determine the target curve corresponding to the target stroke; Starting from the starting point of the target curve, extract a starting line segment at a preset distance from the target curve; Fit the strokes corresponding to the starting line segment and calculate the fitting error corresponding to the starting line segment; If the fitting error is less than or equal to the threshold, continue the fitting operation of the next line segment in the target curve excluding the starting line segment; if the fitting error is greater than the threshold, split the starting line segment when the curvature of the starting line segment reaches its maximum value, and continue fitting the starting line segment after the split starting line segment.

5. The method of claim 1, wherein, The step of generating the text image by rendering the target trajectory point sequence includes: The text image is generated by sequentially connecting the timestamps and location coordinates of each trajectory point in the target trajectory point sequence.

6. The method of claim 1, wherein, The step of generating the second stroke feature based on the Bézier curve corresponding to each stroke and the image features includes: Generate a text box corresponding to each stroke based on the Bézier curve corresponding to each stroke; Based on the text boxes corresponding to each stroke, determine the original image size corresponding to each stroke; Based on the original image size corresponding to each stroke, the image size information corresponding to the image feature is adjusted to obtain the second stroke feature.

7. The method of claim 1, wherein, The feature fusion of the first stroke feature and the second stroke feature to obtain the fused feature includes: Associate the first stroke feature with the second stroke feature so that the first stroke feature corresponds to the second stroke feature; Based on the stroke dimension, the first stroke feature and the second stroke feature are concatenated to obtain the fused feature.

8. The method of claim 1, wherein, The process of identifying the text to be identified by recognizing the fusion features includes: The fused features are input into a bidirectional recurrent neural network based on LSTM to obtain the prediction result for each stroke. The prediction results of all strokes are combined to obtain the text to be identified.

9. The method according to claim 8, characterized in that, The step of inputting the fused features into a bidirectional recurrent neural network based on LSTM to obtain the prediction result for each stroke includes: The fused features and reference information are input into the bidirectional recurrent neural network, wherein the reference information is used to verify the text to be identified; Based on the reference information, determine the verification value corresponding to each stroke; Based on the verification value corresponding to each stroke, calculate the prediction result for each stroke.

10. A method for verifying answer text, characterized in that, The method includes: Obtain the dynamic trajectory point sequence corresponding to the target answer text and the reference answer text corresponding to the target question, wherein the target answer text is associated with the target question; Using the dynamic trajectory point sequence as the target trajectory point sequence and the reference answer text as reference information, the target answer text is verified by the method described in any one of claims 1-9.

11. A text recognition device, characterized in that, The device includes: The acquisition module acquires a target trajectory point sequence, wherein the target trajectory point sequence includes multiple and continuous trajectory points collected from the dynamic handwriting of the text to be recognized; The determining module is used to determine the first stroke feature corresponding to the text to be identified and the second stroke feature corresponding to the text to be identified based on the target trajectory point sequence, wherein the first stroke feature contains at least the temporal feature of the text to be identified, and the second stroke feature is an image feature extracted from a text image containing the text to be identified. The fusion module is used to fuse the first stroke feature and the second stroke feature to obtain the fused feature; The recognition module is used to recognize the text to be recognized by recognizing the fused features; The determining module is further configured to: generate the text image by performing trajectory rendering on the target trajectory point sequence, the text image including the text to be identified; input the text image into a convolutional neural network to output the image features corresponding to the text image; and generate the second stroke features based on the Bézier curve corresponding to each stroke and the image features.

12. A device for verifying answer text, characterized in that, The device includes: The acquisition module is used to acquire the dynamic trajectory point sequence corresponding to the target answer text and the reference answer text corresponding to the target question, wherein the target answer text is associated with the target question; The verification module is used to verify the target answer text using the dynamic trajectory point sequence as the target trajectory point sequence and the reference answer text as reference information, through the device described in claim 11.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

14. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images