Hand-drawn figure recognition method, device and system, and computer readable storage medium

Abstract

The present disclosure relates to a hand-drawn figure recognition method, device and system, and a computer readable storage medium. The hand-drawn figure recognition method comprises: receiving an input of a hand-drawn figure, the input comprising a stroke set, the stroke set comprising at least one stroke, each stroke comprising a sequence of information of a plurality of track points, and the information of each track point comprising coordinates of the track point and writing time; determining whether the stroke set constitutes a closed figure according to coordinate data of the strokes of the input; and in the case where the stroke set constitutes a closed figure, identifying a category of the closed figure based on the coordinate data of the strokes.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus and system for recognizing hand-drawn graphics, as well as a computer-readable storage medium. Background Technology

[0002] Compared to traditional keyboard input, hand-drawn input is widely used due to its simplicity and efficiency. Hand-drawn recognition facilitates natural human-computer interaction. With technological advancements, touchscreens offer users an even better hand-drawn input mode, allowing users to input various graphics online.

[0003] The relevant hand-drawn graphic recognition technology is mainly based on image recognition, outputting standard graphics corresponding to the hand-drawn graphic, such as symmetrical graphics like circles, squares, rectangles, and isosceles triangles, which cannot effectively adapt to the needs of more complex scenarios. Summary of the Invention

[0004] According to some embodiments of this disclosure, a method for recognizing hand-drawn graphics is provided, including:

[0005] Receive input of hand-drawn graphics. The input includes a set of strokes, which includes at least one stroke. Each stroke includes a sequence of information for multiple trajectory points. The information for each trajectory point includes the coordinates of the trajectory point and the writing time.

[0006] Based on the coordinate data of the input strokes, determine whether the set of strokes forms a closed figure;

[0007] When the set of strokes forms a closed figure, the category of the closed figure is identified based on the coordinate data of the strokes.

[0008] In some embodiments, the stroke set includes multiple strokes, each stroke including two endpoints, and the information of each trajectory point also includes a writing state value of the trajectory point, wherein the writing state value is used to represent the writing state of the trajectory point, and the writing state of the trajectory point includes lifting the pen and lowering the pen. Determining whether the stroke set constitutes a closed figure based on the input stroke coordinate data includes:

[0009] Determine whether the interval between the start time of the current stroke and the lift time of the previous stroke is less than the acquisition time threshold.

[0010] If the time is less than the threshold for data collection, the current stroke is taken as the j-th stroke in the stroke set. Based on the connection relationship between the endpoint of the current stroke and the previous strokes, it is determined whether the stroke set forms a closed figure, where j is a positive integer greater than 1.

[0011] In some embodiments, when the distance between the endpoint of one stroke and any endpoint of another stroke is less than a distance threshold, the endpoint of one stroke and the other stroke are considered to be connected. Determining whether the set of strokes constitutes a closed figure based on the connection relationship between the endpoint of the current stroke and previous strokes includes:

[0012] If one endpoint of the current stroke is connected to the jm-th stroke, and the other endpoint of the current stroke is connected to the jn-th stroke, then the set of strokes constitutes a closed figure. Here, m and n are positive integers, and m is greater than or equal to n.

[0013] The strokes in this figure are connected but do not form a closed shape.

[0014] In some embodiments, if neither endpoint of the current stroke is connected to any previous stroke, it is determined that the current stroke is not connected to any previous stroke and the set of strokes is not closed, and no graphic recognition is performed.

[0015] In some embodiments, if one endpoint of the current stroke is connected to one of the previous strokes, but the other endpoint of the current stroke is not connected to any other strokes, it is determined that the current stroke is connected to the previous strokes but the set of strokes does not form a closed shape, and it is determined whether the (j+1)th stroke is received within the acquisition time threshold range.

[0016] If the (j+1)th stroke is received within the acquisition time threshold, the (j+1)th stroke is taken as the current stroke of the stroke set. Based on the connection relationship between the endpoint of the current stroke and the previous strokes, it is determined whether the stroke set forms a closed figure.

[0017] If no stroke j+1 is received within the acquisition time threshold, image recognition will not be performed.

[0018] In some embodiments, the hand-drawn graphic recognition method further includes storing endpoint values ​​of strokes in a storage area, wherein:

[0019] The storage area stores two endpoint values, which include a first endpoint value and a second endpoint value.

[0020] For the first stroke, the values ​​of the two endpoints of the first stroke are the first endpoint value and the second endpoint value, respectively;

[0021] For the j-th stroke, the connection relationship between the j-th stroke and the first and second endpoint values ​​in the storage area is determined.

[0022] If the j-th stroke is connected to the previous strokes but does not form a closed shape, the endpoint value of the previous stroke that is connected to the j-th stroke is replaced in the storage area with the endpoint value of the j-th stroke that is not connected to the previous strokes.

[0023] In some embodiments, the stroke set includes multiple consecutive strokes, each stroke including two endpoints. In the multiple consecutive strokes, the interval between the start time of each stroke and the lift time of the previous stroke is less than a data acquisition time threshold. Determining whether the stroke set constitutes a closed figure based on the input stroke coordinate data includes:

[0024] Strokes from multiple consecutive strokes are sequentially selected as strokes to be processed according to the time sequence. Based on the connection relationship between the endpoints of the stroke to be processed and the previous strokes, it is determined whether the set of strokes forms a closed figure. The stroke to be processed is the j-th stroke in the set of strokes, where j is a positive integer greater than 1.

[0025] In some embodiments, when the distance between the endpoint of one stroke and any endpoint of another stroke is less than a distance threshold, the endpoint of one stroke and the other stroke are considered to be connected. Determining whether the set of strokes constitutes a closed figure based on the connection relationship between the endpoint of the stroke to be processed and the previous strokes includes:

[0026] If one endpoint of the stroke to be processed is connected to the jm-th stroke, and the other endpoint of the stroke to be processed is connected to the jn-th stroke, then the set of strokes is determined to form a closed figure, where m and n are positive integers and m is greater than or equal to n.

[0027] If no endpoint of the stroke to be processed is connected to any previous stroke, it is determined that the stroke to be processed is not connected to any previous stroke and the set of strokes does not form a closed figure.

[0028] If one endpoint of the stroke to be processed is connected to a previous stroke, but the other endpoint of the stroke to be processed is not connected to any other stroke, then it is determined that the stroke to be processed is connected to the previous strokes, but the set of strokes does not form a closed figure.

[0029] The strokes in this figure are connected but do not form a closed shape.

[0030] In some embodiments, when it is determined that the stroke to be processed is connected to the previous strokes but the set of strokes does not form a closed figure, j+1 strokes are taken as the strokes to be processed in the stroke set, and it is determined whether the set of strokes forms a closed figure based on the connection relationship between the endpoints of the stroke to be processed and the previous strokes.

[0031] If it is determined that the stroke to be processed does not form a connection with the previous strokes and the set of strokes does not form a closed figure, then no figure recognition will be performed.

[0032] In some embodiments, the hand-drawn graphic recognition method further includes: for the first stroke in the stroke set, identifying the category of the graphic formed by the stroke; when the identified category is a specified graphic, performing a preset handwriting processing operation for the specified graphic.

[0033] In some embodiments, when the identified category is not a specified shape, it is determined whether the stroke constitutes a closed shape; when it constitutes a closed shape, the category of the closed shape is identified; when it does not constitute a closed shape, it is determined whether a second stroke is received within the acquisition time threshold range; if a second stroke is received within the acquisition time threshold range, the second stroke is taken as the current stroke, and it is determined whether the second stroke and the first stroke constitute a closed shape.

[0034] In some embodiments, the handwriting processing operations include erasing, undoing, marking, or highlighting.

[0035] In some embodiments, the specified graphic includes: an inverted N-shape, a polyline, or a "Z" shape.

[0036] In some embodiments, when the specified graphic is a “Z” shape, an erasure operation is performed.

[0037] In some embodiments, the hand-drawn graphic recognition method further includes: outputting a corrected drawing graphic corresponding to the hand-drawn graphic based on the category of the closed graphic.

[0038] In some embodiments, the hand-drawn graphic recognition method further includes:

[0039] Based on the coordinate offset between trajectory points, locate the key points of the hand-drawn graphic;

[0040] Based on the category and key points of the closed shape, output the corrected drawing shape corresponding to the hand-drawn shape.

[0041] In some embodiments, the key point is a corner point.

[0042] In some embodiments, the hand-drawn graphic recognition method further includes:

[0043] Based on the input stroke coordinate data, determine whether the stroke length is within the preset length range;

[0044] When the length of the stroke is within a preset length range, the stroke is used to identify closed shapes;

[0045] If the length of a stroke exceeds the preset length range, graphic recognition will not be performed.

[0046] In some embodiments, identifying the category of a closed figure includes: using a deep learning algorithm to identify the category of the closed figure.

[0047] In some embodiments, the deep learning algorithm includes a recurrent neural network.

[0048] In some embodiments, the type of graphic includes at least one of polygon, circle, and ellipse.

[0049] In some embodiments, the hand-drawn graphic recognition method further includes:

[0050] Before identifying the category of the closed shape, sparse sampling of the trajectory points is performed according to a predetermined distance threshold to obtain a set of trajectory points; and / or

[0051] The required number of trajectory points is identified based on the category of the closed shape, and the set of trajectory points is sparsely sampled.

[0052] In some embodiments, the terminal side determines whether the set of strokes constitutes a closed figure based on the coordinate data of the input strokes, and if the set of strokes constitutes a closed figure, sends the coordinate data of the corresponding strokes to the cloud; the cloud side identifies the category of the closed figure based on the coordinate data of the strokes.

[0053] According to other embodiments of this disclosure, a method for recognizing hand-drawn graphics is provided, including:

[0054] Receive input of hand-drawn graphics. The input includes a set of strokes, which includes at least one stroke. Each stroke includes a sequence of information for multiple trajectory points. The information for each trajectory point includes the coordinates of the trajectory point and the writing time.

[0055] Based on the input stroke coordinate data, identify the category of the hand-drawn graphic;

[0056] Locate the key points of the hand-drawn graphic;

[0057] Based on the category and key points of the hand-drawn graphic, output the corresponding corrected drawing graphic.

[0058] In some embodiments, key points for locating hand-drawn graphics include:

[0059] Based on the coordinate offset between trajectory points, locate the key points of the hand-drawn graphic.

[0060] In some embodiments, locating key points of a hand-drawn graphic based on the coordinate offset between trajectory points includes:

[0061] Calculate the coordinate offset between trajectory points, including the horizontal coordinate offset and the vertical coordinate offset;

[0062] Based on the magnitude of the horizontal and vertical coordinate offsets between trajectory points, the coordinate offsets are divided into multiple intervals.

[0063] Based on the relationship between the intervals to which the coordinate offsets belong, locate the key points of the hand-drawn graphic.

[0064] In some embodiments, for a trajectory point P i P i+1 P i+2 ...P i+n Information sequence, let trajectory point P i+n+1 and P i+n The coordinate offset between them is O n Trajectory point P i+n and P i+n-1 The coordinate offset between them is O n-1 Coordinate offset O n With O n-1 For adjacent coordinate offsets, based on the relationship between the intervals to which the coordinate offsets belong, the key points of the hand-drawn graphic are located, including:

[0065] At adjacent coordinate offset O n With O n-1 When there are intervals between the intervals, the coordinate offset O is used as the basis. n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic, where i is a positive integer and n is a positive integer greater than 1.

[0066] In some embodiments, locating key points of a hand-drawn graphic based on the relationship between the intervals to which the coordinate offsets belong further includes:

[0067] At adjacent coordinate offset O n With O n-1 When the intervals are adjacent, determine O. n The corresponding trajectory point P i+n and P i+n+1 The line connecting O n-1 The corresponding trajectory point P i+n-1 and P i+n Whether the first included angle between the lines is greater than the first angle threshold;

[0068] When the first included angle is greater than the first angle threshold, based on the adjacent coordinate offset O n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic.

[0069] In some embodiments, locating key points of a hand-drawn graphic based on the relationship between the intervals to which the coordinate offsets belong further includes:

[0070] If the first included angle is not greater than the first angle threshold, or if the adjacent coordinate offset is O n With O n-1 If the intervals are the same, and the offset O n-k With O n-1 The offset between O n-1 They belong to the same interval, but with an offset of O n-k With O n-1 Determine the offset O based on the different intervals. n-k+1 The corresponding trajectory point P i+n-k+1 or P i+n-k+2 and O n The corresponding trajectory point P i+n The line connecting O n The corresponding trajectory point P i+n and P i+n+1 Whether the second included angle between the lines is greater than the second angle threshold, where k is a positive integer greater than 2 and n is a positive integer greater than 3;

[0071] When the second included angle is greater than the second angle threshold, based on the adjacent coordinate offset O n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic.

[0072] In some embodiments, locating key points of a hand-drawn graphic based on the relationship between the intervals to which the coordinate offsets belong further includes:

[0073] If the first included angle is not greater than the first angle threshold, or if the adjacent coordinate offset is O n With O n-1 If the intervals are the same, and the offset O n+k With O n The offset between O n They belong to the same interval, but with an offset of O n+k With O n Determine the offset O based on the different intervals. n+k-1 The corresponding trajectory point P i+n+k-1 or P i+n+k and O n-1 The corresponding trajectory point P i+n The line connecting O n-1 The corresponding trajectory point P i+n-1 and P i+nWhether the second included angle between the lines is greater than the second angle threshold, where k is a positive integer greater than 2 and n is a positive integer greater than 3;

[0074] When the second included angle is greater than the second angle threshold, based on the adjacent coordinate offset O n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic.

[0075] In some embodiments, locating key points of a hand-drawn graphic based on the relationship between the intervals to which the coordinate offsets belong further includes:

[0076] Based on the interval to which the coordinate offset belongs, the horizontal and vertical coordinate offsets between trajectory points are reassigned respectively;

[0077] Based on the reassignment, determine the relationship between the intervals to which the coordinate offsets belong.

[0078] In some embodiments, the key point is a corner point.

[0079] In some embodiments, the hand-drawn graphic recognition method further includes:

[0080] Before locating the key points of the hand-drawn graphic, sparse sampling of the trajectory points is performed according to a predetermined distance threshold to obtain a set of trajectory points; and / or

[0081] Based on the number of trajectory points required for locating the key points of the hand-drawn graphic, the set of trajectory points is sparsely sampled.

[0082] According to further embodiments of this disclosure, a hand-drawn graphic recognition device is provided, comprising:

[0083] Memory; and

[0084] A processor coupled to the memory is configured to execute one or more steps of the hand-drawn graphic recognition method described in any of the foregoing embodiments, based on instructions stored in the memory device.

[0085] According to further embodiments of this disclosure, a hand-drawn graphic recognition system is provided, comprising:

[0086] The touchscreen is configured to receive input of hand-drawn graphics and generate a sequence of information including multiple trajectory points when the user hand-draws graphics on the touchscreen.

[0087] The hand-drawn graphic recognition device described in the foregoing embodiments.

[0088] In some embodiments, the hand-drawn graphic recognition device includes:

[0089] The first processor, located in the terminal, is configured to determine whether the set of strokes forms a closed shape based on the coordinate data of the input strokes.

[0090] The second processor, located in the cloud, is configured to identify the category of a closed shape based on the coordinate data of the strokes when the set of strokes forms a closed shape, or to locate the key points of a hand-drawn shape.

[0091] According to other embodiments of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements one or more steps of the hand-drawn graphic recognition method described in any of the foregoing embodiments.

[0092] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0093] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0094] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0095] Figure 1 This is a flowchart illustrating a hand-drawn graphic recognition method according to an embodiment of the present disclosure;

[0096] Figure 1A This is a schematic diagram illustrating a hand-drawn graphic according to an embodiment of the present disclosure;

[0097] Figure 2 This is a flowchart illustrating a hand-drawn graphic recognition method according to another embodiment of the present disclosure;

[0098] Figure 2A This is a flowchart illustrating the positioning of key points of a hand-drawn graphic according to an embodiment of the present disclosure;

[0099] Figure 2B This is a schematic diagram illustrating the division of intervals according to an embodiment of the present disclosure;

[0100] Figure 3 This is a flowchart illustrating a hand-drawn graphic recognition method according to yet another embodiment of the present disclosure;

[0101] Figure 4A This is a flowchart illustrating a hand-drawn graphic recognition method according to another embodiment of the present disclosure;

[0102] Figure 4B This is a block diagram illustrating a hand-drawn graphic recognition device according to an embodiment of the present disclosure;

[0103] Figure 5 This is a block diagram illustrating a hand-drawn graphic recognition system according to an embodiment of the present disclosure;

[0104] Figure 6 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0105] It should be understood that the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale. Furthermore, the same or similar reference numerals denote the same or similar components. Detailed Implementation

[0106] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. The descriptions of the exemplary embodiments are merely illustrative and are in no way intended to limit the scope of the disclosure or its application or use. The present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided so that the present disclosure will be thorough and complete, and will fully express the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specifically stated, the relative arrangement of components and steps set forth in these embodiments should be construed as exemplary only and not as limiting.

[0107] All terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of ordinary skill in the art to which this disclosure pertains, unless otherwise specifically defined. It should also be understood that terms defined in a general dictionary, such as a dictionary, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0108] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0109] When users hand-draw graphics on whiteboards or other devices, they may create various irregular shapes, such as asymmetrical shapes, which cannot be effectively recognized using existing graphic recognition technologies. To correctly and effectively recognize the diverse and complex hand-drawn graphics, this disclosure proposes a new hand-drawn graphic recognition scheme.

[0110] This disclosure proposes a method to determine whether the user has completed drawing a graphic by using a hand-drawn trajectory, such as acquiring the coordinate information of trajectory points in real time during the user's drawing process, and what kind of target graphic the user is drawing. In some embodiments of this disclosure, the specific category of the target graphic can be further determined, such as whether the graphic is a regular polygon, a circle, or an oblique (e.g., a scalene) polygon. In other embodiments, key points (such as inflection points or corner points) of the target graphic can be located based on the trajectory point information. Using this information, geometric graphics that meet the user's needs can be drawn, achieving a better human-computer interaction effect.

[0111] Figure 1 This is a flowchart illustrating a hand-drawn graphic recognition method according to an embodiment of the present disclosure. Figure 1 As shown, the hand-drawn graphic recognition method includes: step S1, receiving the input of a hand-drawn graphic; step S3, determining whether the set of strokes constitutes a closed graphic based on the coordinate data of the input strokes; step S5, if the set of strokes constitutes a closed graphic, identifying the category of the closed graphic based on the coordinate data of the strokes.

[0112] In step S1, a set of strokes is input, which includes at least one stroke, and each stroke includes a sequence of information for multiple trajectory points. The information for each trajectory point includes its coordinates and writing time. The information for each trajectory point also includes its writing state value. The writing state value represents the writing state of the trajectory point, which includes lifting and lowering the pen. For example, the writing state value can be a first value representing lifting the pen or a second value representing lowering the pen. For instance, the writing state value of the endpoint trajectory point of a stroke represents the first value of lifting the pen (e.g., 1), and the writing state value of the starting trajectory point of a stroke represents the second value of lowering the pen (e.g., 0). Of course, the writing state value can also include a third value representing pen pressure or pen movement. That is, the trajectory point can be assigned to a corresponding stroke based on its writing state value.

[0113] In some embodiments, when drawing on a touchscreen, a sequence of information including multiple trajectory points is generated. For example, when writing on a touchscreen such as an electromagnetic screen or a capacitive screen, the sequence of hand-drawn trajectory points is collected and stored in chronological order. In other embodiments, terminals such as conference all-in-one machines, mobile phones, and smart e-ink tablets use software to acquire the sequence of hand-drawn trajectory points generated when drawing on the touchscreen, and can transmit the sequence of hand-drawn trajectory points to a cloud server or embedded terminal according to the actual needs of the business.

[0114] To recognize hand-drawn graphics input by a user, the first step is to locate the object to be recognized. For example, determining which strokes form a closed shape, and then recognizing each closed shape based on these strokes. In step S3, based on the coordinates of the trajectory points included in each stroke, it is determined whether the set of strokes forms a closed shape. Here, the set of strokes can include one stroke or multiple strokes. Each stroke includes two endpoints. It should be understood that a closed shape is not necessarily strictly closed. The following section will combine... Figure 1A Let me describe it in detail.

[0115] Figure 1A This is a schematic diagram illustrating a hand-drawn graphic according to an embodiment of the present disclosure.

[0116] When the input is a single stroke, such as Figure 1A As shown in (a)-(b), determining whether a stroke constitutes a closed figure includes: determining whether the distance between the starting and ending positions of the stroke is less than a distance threshold. That is, if the distance between the starting position of a stroke and the ending position of the stroke is less than the distance threshold, then... Figure 1A As shown in (a), this stroke can be considered to form a closed figure. And for... Figure 1A As shown in (b), if the distance between the starting and ending points of a stroke is large, the stroke is considered not to form a closed figure.

[0117] In some embodiments, for the first stroke in the stroke set (which can be determined based on the stroke count), it is possible to first identify the shape category without determining whether the stroke forms a closed shape. If the identified category is a specified shape, such as an inverted "N" shape, a polyline, or a "Z" shape, then a preset handwriting processing is performed for the specified shape. The handwriting processing includes: erasing, undoing, marking, or highlighting, etc. For example, if the specified shape is a "Z" shape, an erasing operation is performed. Using a specified shape to perform preset handwriting processing can provide a faster operating experience.

[0118] When the identified category is not a specified shape, determine whether the strokes form a closed shape; if they form a closed shape, identify the category of the closed shape; if they do not form a closed shape, determine whether a second stroke has been received within the acquisition time threshold range; if a second stroke is received within the acquisition time threshold range, take the second stroke as the current stroke, and determine whether the second stroke and the first stroke form a closed shape. Further processing can be performed subsequently depending on whether the stroke set includes multiple strokes.

[0119] In some embodiments, the distance threshold is a dynamically determined value based on the coordinates of the trajectory points, and therefore can be adaptively adjusted according to the characteristics of the hand-drawn graphic. Of course, the distance threshold can also be set to a fixed value as needed.

[0120] When the stroke set includes multiple strokes, such as Figure 1A As shown in (c)-(e), determining whether a set of strokes constitutes a closed figure includes: determining whether the interval between the start time of a stroke and the lift time of the previous stroke is less than a data acquisition time threshold; if it is less than the data acquisition time interval, determining whether the endpoint of a stroke is connected to the previous stroke. If the distance between the endpoint of one stroke and any endpoint of another stroke is less than a distance threshold, then the endpoint of one stroke is connected to the other stroke. That is, if the time interval between consecutive strokes is less than the data acquisition time threshold (e.g., 1 second, 3 seconds, or other times), these consecutive strokes are considered to belong to the same figure; strokes with time intervals exceeding the data acquisition time threshold are considered not to belong to the same figure. Furthermore, based on the connection relationship between the endpoint of the current stroke and the previous stroke, it is determined whether the set of strokes constitutes a closed figure. If all consecutive strokes are connected, as shown in (c)-(e ... Figure 1A As shown in (c) and (d), these consecutive strokes can be considered to form a closed figure. And for... Figure 1A As shown in (e), there are two strokes that do not form a connection, that is, there is a case where the distance between one endpoint of a stroke and the endpoints of other strokes is large. These strokes are considered not to form a closed figure.

[0121] For strokes that do not form closed shapes, they are output as is. In some embodiments, considering practical scenarios, when the length of a stroke is abnormal, such as exceeding a preset length range (i.e., the stroke is very long or very short), it can be considered abnormal data, and in this case, no shape recognition is performed. For example, the stroke can be output as is, or nothing can be displayed. That is, the hand-drawn shape recognition method also includes: determining whether the length of the stroke is within a preset length range based on the coordinate data of the input stroke. If the length of the stroke is within the preset length range, closed shape recognition is performed using the stroke; if the length of the stroke exceeds the preset length range, shape recognition is not performed. Determining whether the length of the stroke is abnormal can be done before or after determining whether consecutive strokes form a closed shape. This setting can improve the efficiency of shape recognition.

[0122] The steps described above for determining whether a set of strokes constitutes a closed figure can be performed either in real-time for each received stroke or for multiple consecutive sets of strokes, depending on the relationship between the processor's processing speed and the input speed of the hand-drawn graphic. The following describes these two scenarios in detail with reference to different embodiments.

[0123] In some embodiments, a judgment is made in real time for each received current stroke, that is, the connection relationship between the endpoint of the current stroke and the previous stroke is determined, so as to determine whether the set of strokes constitutes a closed figure.

[0124] Determining whether a set of strokes forms a closed figure based on the input stroke coordinate data includes: checking if the interval between the start time of the current stroke and the lift time of the previous stroke is less than a data acquisition time threshold; if it is less than the threshold, the current stroke is taken as the j-th stroke in the stroke set, and the connection relationship between the endpoint of the current stroke and the previous strokes is used to determine whether the set of strokes forms a closed figure, where j is a positive integer greater than 1. Here, previous strokes may form a connection but not a closed figure. For example, checking if the interval between the start time of the j-th stroke and the lift time of the (j-1)-th stroke in the stroke set is less than a data acquisition time threshold; if it is less than the threshold, the j-th stroke is added to the stroke set, and the connection relationship between the endpoint of the j-th stroke and the first to (j-1)-th strokes is used to determine whether the first to (j-1)-th strokes form a closed figure. Here, previous strokes may form a connection but not a closed figure, meaning the (j-1)-th stroke is connected to its previous strokes but not a closed figure.

[0125] If one endpoint of the current stroke is connected to the jm-th stroke, and the other endpoint of the current stroke is connected to the jn-th stroke, then the set of strokes is determined to form a closed figure, where m and n are positive integers and m is greater than or equal to n.

[0126] If neither endpoint of the current stroke is connected to any previous stroke, then the current stroke is not connected to any previous stroke and the set of strokes does not form a closed figure.

[0127] If one endpoint of the current stroke is connected to a previous stroke, but the other endpoint of the current stroke is not connected to any other stroke, then the set of strokes is determined to be connected but not to form a closed shape, and no shape recognition is performed.

[0128] If the set of strokes is determined to form a connection relationship but not a closed figure, determine whether the (j+1)th stroke is received within the acquisition threshold time range; if the (j+1)th stroke is received within the acquisition threshold time range, take the (j+1)th stroke as the current stroke of the stroke set, and determine whether the stroke set forms a closed figure based on the connection relationship between the endpoint of the current stroke and the previous strokes; if the (j+1)th stroke is not received within the acquisition threshold time range, do not perform figure recognition, and output the stroke set as is.

[0129] The following example, with j=3, m=2, and n=1, where the first and second strokes form a connection but do not constitute a closed figure, illustrates the various scenarios described above.

[0130] If one endpoint of the third stroke is connected to the first stroke, and the other endpoint is connected to the second stroke, then the first to third strokes form a closed figure.

[0131] If neither endpoint of the third stroke is connected to the first or second stroke, then the third stroke is determined to be neither connected to nor form a closed shape with the preceding strokes. In this case, the first to third strokes are output as is, and the stroke count can be reset to zero.

[0132] If one endpoint of the third stroke is connected to the first stroke, but the other endpoint of the third stroke is not connected to the second stroke, then the third stroke is connected to the previous strokes but does not form a closed figure.

[0133] If the third stroke is determined to be connected to the previous strokes but not to form a closed shape, the system further checks whether a fourth stroke has been received within the acquisition threshold time range. If a fourth stroke is received, it is taken as the current stroke, and the aforementioned check is performed again, that is, based on the connection relationship between the endpoint of the current stroke and the previous strokes, it is determined whether the stroke set forms a closed shape. If no fourth stroke is received within the acquisition threshold time range, the first to third strokes are output as is. In the case of outputting as is, the stroke count can be reset to zero to facilitate the determination of relationships between subsequent strokes.

[0134] In some embodiments, the hand-drawn graphic recognition method further includes storing endpoint values ​​of strokes in a storage area. The storage area stores two endpoint values, including a first endpoint value and a second endpoint value.

[0135] For the first stroke, the two endpoint values ​​are the first endpoint value and the second endpoint value, respectively. For the second stroke, the connection relationship between the second stroke and the first stroke is determined based on the relationship between the endpoint values ​​of the second stroke and the first and second endpoint values ​​in the storage area. If the second stroke and the first stroke are connected but do not form a closed figure, the endpoint value of the second stroke that is not connected to the first stroke is replaced by the endpoint value of the first stroke that is connected to the second stroke from the storage area.

[0136] That is, for the j-th stroke, the connection relationship between the j-th stroke and the previous strokes is determined based on the relationship between the endpoint values ​​of the j-th stroke and the first and second endpoint values ​​in the storage area; if the j-th stroke and the previous strokes form a connection relationship but do not form a closed figure, the endpoint values ​​of the previous strokes that are connected to the j-th stroke are replaced with the endpoint values ​​of the j-th stroke that are not connected to the previous strokes in the storage area.

[0137] In other embodiments, a judgment is made on multiple consecutive stroke sets, specifically by determining the connection relationship between the endpoint of a certain stroke (e.g., the j-th stroke) and previous strokes, thereby determining whether the stroke set constitutes a closed figure. That is, strokes from multiple consecutive stroke sets are sequentially selected as strokes to be processed according to chronological order, and the connection relationship between the endpoint of the stroke to be processed and previous strokes is used to determine whether the stroke set constitutes a closed figure. The stroke to be processed is the j-th stroke in the stroke set.

[0138] Based on the input stroke coordinate data, determining whether a set of strokes constitutes a closed figure includes: If one endpoint of the stroke to be processed is connected to the jm-th stroke, and the other endpoint of the stroke to be processed is connected to the jn-th stroke, then the set of strokes constitutes a closed figure, where m and n are positive integers and m is greater than or equal to n; if neither endpoint of the stroke to be processed is connected to any previous stroke, then the stroke to be processed is not connected to any previous stroke, and the set of strokes does not constitute a closed figure; if one endpoint of the stroke to be processed is connected to one of the previous strokes, but the other endpoint of the stroke to be processed is not connected to any other stroke, then the stroke to be processed is connected to any previous stroke, but the set of strokes does not constitute a closed figure. Specifically, previous strokes are connected but do not constitute a closed figure.

[0139] If it is determined that the stroke to be processed is connected to previous strokes, but the stroke set does not form a closed figure, then the (j+1)th stroke is taken as the stroke to be processed in the stroke set. Based on the connection relationship between the endpoints of the stroke to be processed and previous strokes, it is determined whether the stroke set forms a closed figure. If it is determined that the stroke to be processed is not connected to previous strokes and the stroke set does not form a closed figure, then no figure recognition is performed; for example, the stroke set is output as is. In the case of outputting as is, the stroke count can still be reset to zero to facilitate the subsequent determination of relationships between strokes.

[0140] The following example, taking a stroke set consisting of 8 consecutive strokes, j=4, m=2, n=1, where the first to third strokes form a connection but do not form a closed figure, illustrates the various situations described above.

[0141] If one endpoint of the fourth stroke is connected to the second stroke, and the other endpoint is connected to the third stroke, then the second to fourth strokes form a closed figure.

[0142] If none of the endpoints of the fourth stroke is connected to the first three strokes, then the fourth stroke is determined to be neither connected to nor form a closed shape with the preceding strokes. In this case, no shape recognition is performed; for example, the first four strokes can be output as is. Furthermore, the stroke count can be reset to zero, and the fifth stroke can be used as the first stroke for further processing.

[0143] If one endpoint of the fourth stroke is connected to the second stroke, but the other endpoint is not connected to either the first or third stroke, then the fourth stroke is determined to be connected to the previous strokes but not to form a closed figure. In this case, the fifth stroke is further treated as the stroke to be processed, and the aforementioned judgment is performed again. That is, based on the connection relationship between the endpoint of the stroke to be processed and the previous strokes, it is determined whether the set of strokes forms a closed figure.

[0144] For the strokes that form a closed shape, in step S5, the closed shape can be classified using a deep learning algorithm based on the coordinate data of the trajectory points included in the stroke.

[0145] The categories of closed figures can include various standard or non-standard figures, or regular or irregular figures, or symmetrical or asymmetrical figures. Specifically, the categories of figures can include at least one of polygons, circles, and ellipses. Polygons can include symmetrical figures such as rectangles, isosceles triangles, isosceles trapezoids, and parallelograms, and can also include asymmetrical figures such as non-isosceles triangles and non-isosceles trapezoids. It should be understood that closed figures all satisfy the basic characteristics of figures. For example, for the strokes that make up a circle, the coordinates of each trajectory point must meet certain conditions, with at least an offset in four directions (up, down, left, and right) between the trajectory points.

[0146] After identifying the category of the closed figure in step S5, the corrected drawing figure corresponding to the hand-drawn figure can be output according to the category of the closed figure.

[0147] The above-mentioned corrected drawing is a drawing after necessary corrections to the hand-drawn drawing. For example, based on the relationship between the trajectory points included in the hand-drawn line, the hand-drawn line is corrected to a straight line (such as corresponding to a polygon) or a curve (such as corresponding to a circle or ellipse).

[0148] In some embodiments, recurrent neural networks (RNNs) are used to identify the category of closed figures. For example, a bidirectional long short-term memory (BiLSTM) network combined with linear layers can be used for figure category recognition to obtain comprehensive and accurate recognition results.

[0149] The number of layers in an LSTM network can be determined based on the difficulty of the task and the time required; it can use one or more layers. The function of the linear layer is to linearly transform the output to a specified dimension. More complex RNN networks can also be used depending on the required accuracy.

[0150] Figure 2 This is a flowchart illustrating a hand-drawn graphic recognition method according to another embodiment of the present disclosure. Figure 2 and Figure 1 The difference is that, Figure 2 The hand-drawn graphic recognition method also includes steps S7 and S9. Only steps S7 and S9 will be described below. Figure 2 and Figure 1 The differences are minor, while the similarities will not be elaborated upon.

[0151] In step S7, key points of the hand-drawn graphic are located. In some embodiments, key points of the hand-drawn graphic are located based on the coordinate offset between trajectory points. Key points can also be called inflection points or corner points of the graphic, indicating a significant change in slope at that trajectory point. For example, for a polygon, the intersection of its edges is an inflection point or corner point. It should be understood that the execution order of step S7 and step S5, which identifies the category of the graphic, is not restricted; step S7 can be executed before step S5, or vice versa. Of course, identifying the category of the graphic first, and then locating the key points of the graphic based on the category, is more computationally efficient.

[0152] The following is combined with Figure 2A This describes a process for locating key points of a hand-drawn graphic according to some embodiments of this disclosure. For example... Figure 2A As shown, the process for locating key points of a hand-drawn graphic includes steps S71-S73.

[0153] In step S71, the coordinate offset between trajectory points is calculated. The coordinate offset includes the horizontal coordinate offset and the vertical coordinate offset. For trajectory point P... i P i+1 P i+2 ...P i+n Information sequence, let trajectory point P i+n+1 and P i+n The coordinate offset between them is O n Trajectory point P i+n and P i+n-1 The coordinate offset between them is O n-1 Coordinate offset O n With O n-1 This represents the offset between adjacent coordinates.

[0154] In step S72, the coordinate offset is divided into multiple intervals based on the magnitude of the horizontal and vertical coordinate offsets between trajectory points.

[0155] The calculated coordinate offsets are denoted as dx for the horizontal coordinate and dy for the vertical coordinate. That is, O n It can be represented as (dx) n ,dy n By comparing the magnitudes of dx and dy, the coordinate offsets are divided into multiple intervals. The number of intervals is determined based on the recognition difficulty of the hand-drawn graphic.

[0156] In some embodiments, to reduce computational load, the horizontal and vertical coordinate offsets between trajectory points can be reassigned based on the interval to which the coordinate offsets belong; the relationship between the intervals to which the coordinate offsets belong is determined based on the reassignment. Reassignment facilitates the identification of the interval to which the coordinate offsets belong, and can intuitively reflect the trend of the input trajectory points. Locating key points of a graphic based on coordinate offsets is simpler and requires less computation than locating key points based on slope.

[0157] The following example, using the division into four intervals, illustrates the interval division and reassignment process in conjunction with Table 1. Table 1 shows the relationship between the interval division results, the reassignment results, and the coordinate offset.

[0158] Table 1: Relationship between interval partitioning results and coordinate offset, and results of reassignment.

[0159]

[0160] As shown in Table 1, for two trajectory points, if dy > 0, then the coordinate offsets with a slope of 0 < dy / dx ≤ 1 are assigned to the first interval, the coordinate offsets with a slope of -1 ≤ dy / dx < 0 are assigned to the third interval, and the coordinate offsets with a slope of dy / dx greater than 1 or less than -1 are assigned to the second interval; if dy < 0, then the coordinate offsets with a slope of -1 ≤ dy / dx < 0 are assigned to the first interval, the coordinate offsets with a slope of 0 < dy / dx ≤ 1 are assigned to the third interval, and the coordinate offsets with a slope of dy / dx greater than 1 or less than -1 are assigned to the fourth interval.

[0161] For the first and third intervals, where the absolute value of the slope |dy / dx| is less than or equal to 1, the reassignment result is dy equal to 0 and dx equal to the original value; for the second and fourth intervals, where the absolute value of the slope |dy / dx| is greater than 1.

[0162] The case where dy = 0 can be handled in a similar manner to the cases where dy > 0 or dy < 0.

[0163] Figure 2B A schematic diagram illustrating interval division according to some embodiments of the present disclosure is shown. For example... Figure 2B As shown, the 360-degree interval can be divided into 4 intervals, with an absolute slope of 1, visually demonstrating the results of interval division based on Table 1. It should be understood that if the hand-drawn diagram is complex, more intervals, such as 8 or more, can be divided for more precise calculations.

[0164] Next, in step S73, the key points of the hand-drawn graphic are located based on the relationship between the intervals to which the coordinate offsets belong. Table 2 below explains how to locate the key points of the hand-drawn graphic. Table 2 shows the relationship between the intervals to which adjacent coordinate offsets belong, and the corresponding key point location results.

[0165] Table 2: Relationship between adjacent coordinate offset intervals and corresponding key point localization results.

[0166]

[0167]

[0168] As shown in Table 2, at adjacent coordinate offsets O n With O n-1 When there are intervals between the intervals, the coordinate offset O is used as the basis. n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic, where i is a positive integer and n is a positive integer greater than 1. For example, the offset O can be... n With O n-1 Commonly corresponding trajectory points P i+n As a key point in hand-drawn graphics. For example... Figure 2B As shown, there is an interval between the first interval and the third interval, but no interval between the first interval and the second interval or between the first interval and the fourth interval.

[0169] At adjacent coordinate offset O n With O n-1 When the intervals are adjacent, determine O. n The corresponding trajectory point P i+n and P i+n-1 The line connecting O n-1 The corresponding trajectory point P i+n+1 and P i+n Whether the first included angle between the connecting lines is greater than a first angle threshold; if the first included angle is greater than the first angle threshold, based on the adjacent coordinate offset O n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic. For example, the offset O can be... n With O n-1 Commonly corresponding trajectory points P i+n As a key point in hand-drawn graphics. For example... Figure 2B As shown, the first interval is adjacent to the second interval or the fourth interval. The first angle threshold is, for example, 30 degrees.

[0170] If the first included angle is not greater than the first angle threshold, or if the adjacent coordinate offset is O n With O n-1 If the intervals are the same, and the offset O n-k With O n-1 The offset between O n-1 They belong to the same interval, but with an offset of O n-k With O n-1 Determine the offset O based on the different intervals. n-k+1 The corresponding trajectory point P i+n-k+1 or P i+n-k+2 and O n The corresponding trajectory point P i+n The line connecting O n The corresponding trajectory point P i+n and P i+n+1 The second included angle between the lines connecting them, where k is a positive integer greater than 2 and n is a positive integer greater than 3; or if the offset O n+k With O n The offset between O n They belong to the same interval, but with an offset of O n+k With O n Determine the offset O based on the different intervals. n+k-1 The corresponding trajectory point P i+n+k-1 or P i+n+k and O n-1 The corresponding trajectory point P i+n The line connecting O n-1 The corresponding trajectory point P i+n-1 and P i+n Whether the second included angle between the lines is greater than the second angle threshold.

[0171] When the second included angle is greater than the second angle threshold, based on the offset O n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic. For example, the offset O can be... n With O n-1 Commonly corresponding trajectory points P i+n As a key point in hand-drawn graphics, the second angle threshold can be the same as or slightly different from the first angle threshold.

[0172] The embodiments disclosed herein fully consider the complex scenarios of hand-drawn graphics, such as the angle of a user-drawn zigzag line being an arc. By comprehensively analyzing the information of nearby trajectory points, the key points of the hand-drawn graphics can be located more accurately.

[0173] In some embodiments, the hand-drawn graphic recognition method further includes: sparse sampling of the input trajectory points based on coordinates. Performing graphic category recognition and key point localization after sparse sampling can reduce unnecessary computation and improve the speed of graphic recognition. In some embodiments, before recognizing the category of a closed graphic or before locating the key points of a hand-drawn graphic, sparse sampling of the trajectory points is performed according to a predetermined distance threshold to obtain a set of trajectory points. For example, if the distance between two input trajectory points is less than a predetermined threshold, either trajectory point is removed to obtain a set of trajectory points. For example, if the distance between two trajectory points is less than a predetermined threshold (e.g., 20 pixels), one of them can be deleted, thus removing some dense trajectory points. As another example, if the difference between the slopes of the lines connecting the trajectory point to its predecessor and successor is less than or equal to a change threshold (e.g., 0.1), then these three trajectory points can be considered to be on a straight line, and therefore the trajectory point can be deleted to reduce the amount of data to be recognized. For simple hand-drawn graphics, a fixed number of trajectory points (e.g., 40) can be selected, and then the number of trajectory points is fixed. For example, for a quadrilateral, each side can have 8 trajectory points.

[0174] Furthermore, the required number of trajectory points can be identified based on the category of the closed shape, and the set of trajectory points can be sparsely sampled. Alternatively, the required number of trajectory points can be located based on the key points of the hand-drawn shape, and the set of trajectory points can be sparsely sampled.

[0175] In step S9, based on the category and key points of the closed shape, a corrected drawing shape corresponding to the hand-drawn shape is output.

[0176] Corresponding to the category of the closed shape, the output corrected drawing shape may be a regular shape or an irregular shape. In some embodiments, the closed shape can be determined as a regular shape based on the category of the closed shape identified in step S3 and the coordinates of the trajectory points.

[0177] For example, if a closed shape is identified as a polygon, the side lengths and angles between adjacent sides can be calculated based on the coordinates of the trajectory points, thus determining whether the closed shape is a regular shape. If the result is a regular shape, parameters such as the center point, width, and height can be determined based on the coordinates of the trajectory points, outputting a regular shape corresponding to the hand-drawn shape. If the result is an irregular shape, the calculated side lengths, angles, and other parameters can be used to output an irregular shape corresponding to the hand-drawn shape. For example, if the hand-drawn shape is identified as an irregular triangle, corner points 1, 2, and 3 can be obtained using key point localization; connecting these three corner points will output the irregular triangle corresponding to the hand-drawn shape.

[0178] Figure 3 This is a flowchart illustrating a hand-drawn graphic recognition method according to yet another embodiment of the present disclosure. Figure 3 and Figure 2 The difference lies in the inclusion of step S2. The following will only describe... Figure 3 and Figure 2 The differences are minor, while the similarities will not be elaborated upon.

[0179] like Figure 3 As shown, in step S2, based on the input stroke coordinate data, it is determined whether the stroke length is within a preset length range. If the stroke length is within the preset length range, the stroke is used to recognize a closed shape. If the stroke length exceeds the preset length range, i.e., the stroke length is very long or very short, it can be considered abnormal data, and the stroke is output as is without shape recognition. This improves the efficiency of shape recognition.

[0180] like Figure 3 As shown, for a single stroke, if it forms a closed shape, the type of the shape is identified and the corrected drawing shape is output accordingly; if it does not form a closed shape, but is determined in step S31 to form a specified shape (such as a "Z" shape), then step S32 is entered to perform an erasure operation; if it neither forms a closed shape nor a specified shape, the stroke is output as is.

[0181] Of course, as mentioned earlier, for a single stroke, we can first determine whether it forms a closed shape, and directly identify the shape category. If it forms a specified shape, we will perform an erasure operation; if it does not form a specified shape but is identified as a shape of a certain category, we will draw the shape of the corresponding category; otherwise, we will output the stroke as is.

[0182] In the aforementioned embodiments, the terminal side can determine whether the set of strokes constitutes a closed figure based on the coordinate data of the input strokes, and if the set of strokes constitutes a closed figure, send the coordinate data of the corresponding strokes to the cloud; the cloud side identifies the category of the closed figure based on the coordinate data of the strokes.

[0183] Figure 4A This is a flowchart illustrating a hand-drawn graphic recognition method according to another embodiment of the present disclosure.

[0184] like Figure 4A As shown, the hand-drawn graphic recognition method includes: step S1, receiving the input of a hand-drawn graphic; step S5', identifying the category of the hand-drawn graphic based on the coordinate data of the input strokes; step S7, locating the key points of the hand-drawn graphic; and step S9, outputting a corrected drawing graphic corresponding to the hand-drawn graphic based on the category and key points of the hand-drawn graphic.

[0185] As you can see, Figure 4A and Figure 2The difference lies in the exclusion of step S3, and the fact that step S5' differs from step S5. Since it is not necessary to execute... Figure 2 The determination of closed figures can be done without executing step S3, so it is different from identifying the category of closed figures in step S5, which identifies the category of hand-drawn figures in step S5'.

[0186] In the above embodiments, the key points of the hand-drawn graphic can be located in the cloud.

[0187] The various method embodiments of this disclosure have been described in detail above. The corresponding product embodiments are described below. This disclosure also provides a hand-drawn graphic recognition device.

[0188] Figure 4B This is a block diagram illustrating a hand-drawn graphic recognition device according to an embodiment of the present disclosure.

[0189] like Figure 4B As shown, the hand-drawn graphic recognition device 40 includes a memory 401 and a processor 402 coupled to the memory 401. The memory 401 is configured to store instructions for executing embodiments of the hand-drawn graphic recognition method. The processor 402 is configured to execute one or more steps of the hand-drawn graphic recognition method in any of the embodiments of this disclosure based on the instructions stored in the memory 401.

[0190] It should be understood that one or more steps in the aforementioned hand-drawn graphic recognition method can be implemented by a processor, and can be implemented in any way, including software, hardware, firmware, or a combination thereof.

[0191] Besides the hand-drawn image recognition method and apparatus, embodiments of this disclosure may also take the form of a computer program product implemented on one or more non-volatile storage media containing computer program instructions. Therefore, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement one or more steps of the hand-drawn image recognition method in any of the foregoing embodiments.

[0192] Figure 5 This is a block diagram illustrating a hand-drawn graphic recognition system according to an embodiment of the present disclosure.

[0193] like Figure 5 As shown, the hand-drawn graphic recognition system 5 includes: a touch screen 50 configured to receive input of hand-drawn graphics and generate an information sequence including multiple trajectory points when the user draws graphics on the touch screen; and the aforementioned hand-drawn graphic recognition device 40.

[0194] In some embodiments, the hand-drawn graphic recognition device 40 includes: a first processor 41, located at a terminal, configured to determine whether a set of strokes constitutes a closed graphic based on the coordinate data of the input strokes; and a second processor 42, located in the cloud, configured to identify the category of the closed graphic based on the coordinate data of the strokes when the set of strokes constitutes a closed graphic, or configured to locate key points of the hand-drawn graphic.

[0195] Of course, the first processor 41 and the second processor 42 of the hand-drawn graphic recognition device 40 can both be located on the cloud server, performing hand-drawn graphic recognition on the received sequence of hand-drawn trajectory points, and transmitting the graphic recognition results to the designated receiver, such as a terminal.

[0196] Figure 6 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0197] like Figure 6 As shown, the computer system can be represented in the form of a general computing device, which can be used to implement the hand-drawn graphic recognition device of the above embodiment. The computer system includes a memory 610, a processor 620, and a bus 600 connecting different system components.

[0198] The memory 610 may include, for example, system memory, non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media may store, for example, instructions for executing corresponding embodiments of the display method. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0199] The processor 620 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each device, such as the device judgment and device determination, can be implemented by the central processing unit (CPU) running instructions in memory to execute the corresponding steps, or by dedicated circuitry to execute the corresponding steps.

[0200] Bus 600 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0201] The computer system may also include input / output interfaces 630, network interfaces 640, and storage interfaces 650. These interfaces 630, 640, and 650, as well as the memory 610 and processor 620, can be connected via a bus 600. Input / output interface 630 provides a connection interface for input / output devices such as monitors, mice, and keyboards. Network interface 640 provides a connection interface for various networked devices. Storage interface 640 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0202] Various embodiments of this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0203] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for recognizing hand-drawn graphics, comprising: The system receives input of hand-drawn graphics. The input includes a set of strokes, each stroke having two endpoints and a sequence of information for multiple trajectory points. The information for each trajectory point includes its coordinates, writing time, and writing state value. The writing state value represents the writing state of the trajectory point, including lifting and lowering the pen. Based on the input stroke coordinate data, determine whether the stroke set forms a closed figure, including: determining whether the interval between the time of the current stroke's fall and the time of the previous stroke's lift is less than the acquisition time threshold; if it is less than the acquisition time threshold, take the current stroke as the j-th stroke in the stroke set, and determine whether the stroke set forms a closed figure based on the connection relationship between the endpoint of the current stroke and the previous strokes, where j is a positive integer greater than 1. When the set of strokes forms a closed figure, the category of the closed figure is identified based on the coordinate data of the strokes.

2. The hand-drawn graphic recognition method according to claim 1, wherein, If the distance between the endpoint of one stroke and any endpoint of another stroke is less than a distance threshold, then the endpoint of one stroke and the endpoint of another stroke are considered connected. Determining whether a set of strokes constitutes a closed figure based on the connection between the endpoint of the current stroke and previous strokes includes: If one endpoint of the current stroke is connected to the jm-th stroke, and the other endpoint of the current stroke is connected to the jn-th stroke, then the set of strokes constitutes a closed figure. Here, m and n are positive integers, and m is greater than or equal to n. The strokes in this figure are connected but do not form a closed shape.

3. The hand-drawn graphic recognition method according to claim 2, wherein: If neither endpoint of the current stroke is connected to any previous stroke, it is determined that the current stroke is not connected to any previous stroke and the set of strokes is not closed, and no graphic recognition is performed.

4. The hand-drawn graphic recognition method according to claim 2, wherein: If one endpoint of the current stroke is connected to one of the previous strokes, but the other endpoint of the current stroke is not connected to any other stroke, determine that the current stroke is connected to the previous strokes but the set of strokes does not form a closed figure, and determine whether the (j+1)th stroke is received within the acquisition time threshold. If the (j+1)th stroke is received within the acquisition time threshold, the (j+1)th stroke is taken as the current stroke of the stroke set. Based on the connection relationship between the endpoint of the current stroke and the previous strokes, it is determined whether the stroke set forms a closed figure. If no stroke j+1 is received within the acquisition time threshold, image recognition will not be performed.

5. The hand-drawn graphic recognition method according to claim 2 further includes storing the endpoint values ​​of the strokes in the storage area, wherein: The storage area stores two endpoint values, which include a first endpoint value and a second endpoint value. For the first stroke, the values ​​of the two endpoints of the first stroke are the first endpoint value and the second endpoint value, respectively; For the j-th stroke, the connection relationship between the j-th stroke and the first and second endpoint values ​​in the storage area is determined. If the j-th stroke is connected to the previous strokes but does not form a closed shape, the endpoint value of the previous stroke that is connected to the j-th stroke is replaced in the storage area with the endpoint value of the j-th stroke that is not connected to the previous strokes.

6. The hand-drawn graphic recognition method according to claim 1, wherein, The stroke set includes multiple consecutive strokes, each stroke having two endpoints. In these consecutive strokes, the interval between the start of each stroke and the end of the previous stroke is less than a data acquisition time threshold. Based on the input stroke coordinate data, determining whether the stroke set constitutes a closed figure includes: Strokes from multiple consecutive strokes are sequentially selected as strokes to be processed according to the time sequence. Based on the connection relationship between the endpoints of the stroke to be processed and the previous strokes, it is determined whether the set of strokes forms a closed figure. The stroke to be processed is the j-th stroke in the set of strokes, where j is a positive integer greater than 1.

7. The hand-drawn graphic recognition method according to claim 6, wherein, If the distance between the endpoint of one stroke and any endpoint of another stroke is less than a distance threshold, then the endpoint of one stroke and the other stroke are considered connected. Determining whether the set of strokes forms a closed figure based on the connection between the endpoint of the stroke to be processed and previous strokes includes: If one endpoint of the stroke to be processed is connected to the jm-th stroke, and the other endpoint of the stroke to be processed is connected to the jn-th stroke, then the set of strokes is determined to form a closed figure, where m and n are positive integers and m is greater than or equal to n. If no endpoint of the stroke to be processed is connected to any previous stroke, it is determined that the stroke to be processed is not connected to any previous stroke and the set of strokes does not form a closed figure. If one endpoint of the stroke to be processed is connected to a previous stroke, but the other endpoint of the stroke to be processed is not connected to any other stroke, then it is determined that the stroke to be processed is connected to the previous strokes, but the set of strokes does not form a closed figure. The strokes in this figure are connected but do not form a closed shape.

8. The hand-drawn graphic recognition method according to claim 7, wherein: If it is determined that the stroke to be processed is connected to the previous strokes but the set of strokes does not form a closed figure, take the j+1 strokes as the strokes to be processed in the set of strokes, and determine whether the set of strokes forms a closed figure based on the connection relationship between the endpoints of the strokes to be processed and the previous strokes. If it is determined that the stroke to be processed does not form a connection with the previous strokes and the set of strokes does not form a closed figure, then no figure recognition will be performed.

9. The hand-drawn graphic recognition method according to claim 1 further includes: For the first stroke in the stroke set, identify the category of the shape formed by the stroke. When the identified category is a specified shape, perform the handwriting processing operation preset for the specified shape.

10. The hand-drawn graphic recognition method according to claim 9, wherein: When the identified category is not the specified shape, determine whether the strokes constitute a closed shape; When a closed figure is formed, identify the category of the closed figure; When a closed shape is not formed, determine whether the second stroke is received within the acquisition time threshold range; If the second stroke is received within the acquisition time threshold, the second stroke is taken as the current stroke, and it is determined whether the second stroke and the first stroke form a closed shape.

11. The hand-drawn graphic recognition method according to claim 9, wherein, The handwriting processing operations include: erasing, undoing, marking, or highlighting.

12. The hand-drawn graphic recognition method according to claim 9, wherein, The specified shapes include: inverted "N" shape, polyline, or "Z" shape.

13. The hand-drawn graphic recognition method according to claim 12, wherein, If the specified graphic is a "Z" shape, perform an erasure operation.

14. The hand-drawn graphic recognition method according to any one of claims 1 to 13, further comprising: Based on the category of the closed figure, output the corrected drawing figure corresponding to the hand-drawn figure.

15. The hand-drawn graphic recognition method according to any one of claims 1 to 13, further comprising: Locate the key points of the hand-drawn graphic; Based on the category and key points of the closed shape, output the corrected drawing shape corresponding to the hand-drawn shape.

16. The hand-drawn graphic recognition method according to claim 15, wherein, The key point is the corner point.

17. The hand-drawn graphic recognition method according to any one of claims 1 to 13, further comprising: Based on the input stroke coordinate data, determine whether the stroke length is within the preset length range; When the length of the stroke is within a preset length range, the stroke is used to identify closed shapes; If the length of a stroke exceeds the preset length range, graphic recognition will not be performed.

18. The hand-drawn graphic recognition method according to any one of claims 1 to 13, wherein, Categories for identifying closed figures include: Using deep learning algorithms, we can identify the categories of closed figures.

19. The hand-drawn graphic recognition method according to claim 18, wherein, Deep learning algorithms include recurrent neural networks.

20. The hand-drawn graphic recognition method according to any one of claims 1 to 13, wherein, The categories of shapes include at least one of polygons, circles, and ellipses.

21. The hand-drawn graphic recognition method according to any one of claims 1 to 13, further comprising: Before identifying the category of the closed shape, the trajectory points are sparsely sampled according to a predetermined distance threshold to obtain a set of trajectory points; and / or The required number of trajectory points is identified based on the category of the closed shape, and the set of trajectory points is sparsely sampled.

22. The hand-drawn graphic recognition method according to any one of claims 1 to 13, wherein: The terminal determines whether the set of strokes forms a closed shape based on the coordinate data of the input strokes, and if the set of strokes forms a closed shape, it sends the coordinate data of the corresponding strokes to the cloud. The cloud-based system identifies the category of closed shapes based on the coordinate data of the strokes.

23. A method for recognizing hand-drawn graphics, comprising: Receive input of hand-drawn graphics. The input includes a set of strokes, which includes at least one stroke. Each stroke includes a sequence of information for multiple trajectory points. The information for each trajectory point includes the coordinates of the trajectory point and the writing time. Based on the input stroke coordinate data, identify the category of the hand-drawn graphic; Based on the coordinate offset between trajectory points, locate the key points of the hand-drawn graphic, including: Calculate the coordinate offset between trajectory points, including the horizontal coordinate offset and the vertical coordinate offset; Based on the magnitude of the horizontal and vertical coordinate offsets between trajectory points, the coordinate offsets are divided into multiple intervals. Based on the relationship between the intervals to which the coordinate offsets belong, locate the key points of the hand-drawn graphic, wherein the key points are corner points; Based on the category and key points of the hand-drawn graphic, output the corresponding corrected drawing graphic; Among them, for the trajectory point P i P i+1 P i+2 ...P i+n Information sequence, let trajectory point P i+n+1 and P i+n The coordinate offset between them is O n Trajectory point P i+n and P i+n-1 The coordinate offset between them is O n-1 Coordinate offset O n With O n-1 For adjacent coordinate offsets, based on the relationship between the intervals to which the coordinate offsets belong, the key points of the hand-drawn graphic are located, including: At adjacent coordinate offset O n With O n-1 When there are intervals between the intervals, the coordinate offset O is used as the basis. n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic, where i is a positive integer and n is a positive integer greater than 1.

24. The hand-drawn graphic recognition method according to claim 23, wherein, Based on the relationship between the intervals to which the coordinate offsets belong, the key points for locating the hand-drawn graphic also include: At adjacent coordinate offset O n With O n-1 When the intervals are adjacent, determine O. n The corresponding trajectory point P i+n and P i+n+1 The line connecting O n-1 The corresponding trajectory point P i+n-1 and P i+n Whether the first included angle between the lines is greater than the first angle threshold; When the first included angle is greater than the first angle threshold, based on the adjacent coordinate offset O n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic.

25. The hand-drawn graphic recognition method according to claim 24, wherein, Based on the relationship between the intervals to which the coordinate offsets belong, the key points for locating the hand-drawn graphic also include: If the first included angle is not greater than the first angle threshold, or if the adjacent coordinate offset is O n With O n-1 If the intervals are the same, and the offset O n-k With O n-1 The offset between O n-1 They belong to the same interval, but with an offset of O n-k With O n-1 Determine the offset O based on the different intervals. n-k+1 The corresponding trajectory point P i+n-k+1 or P i+n-k+2 and O n The corresponding trajectory point P i+n The line connecting O n The corresponding trajectory point P i+n and P i+n+1 Whether the second included angle between the lines is greater than the second angle threshold, where k is a positive integer greater than 2 and n is a positive integer greater than 3; When the second included angle is greater than the second angle threshold, based on the adjacent coordinate offset O n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic.

26. The hand-drawn graphic recognition method according to claim 25, wherein, Based on the relationship between the intervals to which the coordinate offsets belong, the key points for locating the hand-drawn graphic also include: If the first included angle is not greater than the first angle threshold, or if the adjacent coordinate offset is O n With O n-1 If the intervals are the same, and the offset O n+k With O n The offset between O n They belong to the same interval, but with an offset of O n+k With O n Determine the offset O based on the different intervals. n+k-1 The corresponding trajectory point P i+n+k-1 or P i+n+k and O n-1 The corresponding trajectory point P i+n The line connecting O n-1 The corresponding trajectory point P i+n-1 and P i+n Whether the second included angle between the lines is greater than the second angle threshold, where k is a positive integer greater than 2 and n is a positive integer greater than 3; When the second included angle is greater than the second angle threshold, based on the adjacent coordinate offset O n With O n-1 Commonly corresponding trajectory points P i+n Locate the key points of the hand-drawn graphic.

27. The hand-drawn graphic recognition method according to any one of claims 23 to 26, wherein, Based on the relationship between the intervals to which the coordinate offsets belong, the key points for locating the hand-drawn graphic also include: Based on the interval to which the coordinate offset belongs, the horizontal and vertical coordinate offsets between trajectory points are reassigned respectively; Based on the reassignment, determine the relationship between the intervals to which the coordinate offsets belong.

28. The hand-drawn graphic recognition method according to any one of claims 23 to 26, further comprising: Before locating the key points of the hand-drawn graphic, the trajectory points are sparsely sampled according to a predetermined distance threshold to obtain a set of trajectory points. and / or Based on the number of trajectory points required for locating the key points of the hand-drawn graphic, the set of trajectory points is sparsely sampled.

29. A hand-drawn graphic recognition device, comprising: Memory; and A processor coupled to the memory, the processor being configured to perform one or more steps of the hand-drawn graphic recognition method according to any one of claims 1 to 28 based on instructions stored in the memory device.

30. A hand-drawn graphic recognition system, comprising: The touchscreen is configured to receive input of hand-drawn graphics and generate a sequence of information including multiple trajectory points when the user hand-draws graphics on the touchscreen. The hand-drawn graphic recognition device as described in claim 29.

31. The hand-drawn graphic recognition system according to claim 30, wherein, The hand-drawn graphic recognition device includes: The first processor, located in the terminal, is configured to determine whether the set of strokes forms a closed shape based on the coordinate data of the input strokes. The second processor, located in the cloud, is configured to identify the category of a closed shape based on the coordinate data of the strokes when the set of strokes forms a closed shape, or to locate the key points of a hand-drawn shape.

32. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements one or more steps of the hand-drawn graphic recognition method as described in any one of claims 1 to 28.

Images