Interaction method, electronic device and computer storage medium

Abstract

An interaction method, an electronic device and a computer storage medium; the interaction method comprises: obtaining a track point set input by a user, the track point set comprising at least one track point; determining candidate objects based on the track point set, the candidate objects comprising: a first candidate object and a second candidate object; displaying a first selection icon for representing the first candidate object and a second selection icon for representing the second candidate object; in response to a user's selection operation on the first selection icon, displaying the first candidate object or executing an operation corresponding to the first candidate object, and in response to a user's selection operation on the second selection icon, displaying the second candidate object or executing an operation corresponding to the second candidate object.

Description

Technical Field

[0001] This article relates to but is not limited to the field of computer technology, and in particular to an interaction method, electronic device and computer storage medium. Background Art

[0002] Paperless office and learning using electronic devices has gradually replaced traditional methods and become mainstream. Electronic devices such as electronic whiteboards and touch-screen mobile phones and tablets allow users to write and draw directly on the screen with their fingers or a stylus. However, when users want to input the required graphics, they need to perform a series of operations such as search, copy, paste, or insert, which is inconvenient and inefficient. Summary of the Invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] Embodiments of the present application provide an interaction method, an electronic device, and a computer storage medium.

[0005] In one aspect, an embodiment of the present application provides an interaction method, comprising:

[0006] Obtaining a track point set input by the user, where the track point set includes at least one track point;

[0007] Determine candidate objects according to the trajectory point set, the candidate objects including: a first candidate object and a second candidate object;

[0008] displaying a first selection icon for representing a first candidate object and a second selection icon for representing a second candidate object;

[0009] In response to the user's selection operation on the first selection icon, the first candidate object is displayed or the operation corresponding to the first candidate object is performed. In response to the user's selection operation on the second selection icon, the second candidate object is displayed or the operation corresponding to the second candidate object is performed.

[0010] In an exemplary embodiment, the first candidate object includes a first candidate graphic, and the second candidate object includes a second candidate graphic;

[0011] The displaying of the first candidate object in response to a user's selection operation on the first selection icon, and the displaying of the second candidate object in response to a user's selection operation on the second selection icon, comprises:

[0012] In response to a user's selection operation on the first selection icon, the first candidate graphic is displayed; in response to a user's selection operation on the second selection icon, the second candidate graphic is displayed;

[0013] The determining of candidate objects according to the trajectory point set includes:

[0014] Acquire first feature information according to the trajectory formed by the trajectory point set, and acquire a first candidate graphic that successfully matches the first feature information.

[0015] obtaining second feature information according to the trajectory formed by the set of trajectory points, and obtaining a second candidate graphic that successfully matches the second feature information;

[0016] The first feature information is different from the second feature information.

[0017] In an exemplary embodiment, the first characteristic information and the second characteristic information respectively include one or more of the following:

[0018] One or more character features corresponding to the trajectory;

[0019] Original line features of the figure formed by the trajectory;

[0020] The contour line features of the figure formed by the trajectory.

[0021] In an exemplary embodiment, the first feature information or the second feature information includes one or more character features corresponding to the trajectory;

[0022] The extracting the first feature information or the second feature information according to the trajectory formed by the trajectory point set includes:

[0023] Extracting one or more character features corresponding to the trajectory formed by the trajectory point set;

[0024] The acquiring of the first / second candidate graphics that successfully matches the first / second feature information includes:

[0025] Selecting a candidate graphic whose carried label has at least partially the same character features as those in the first / second feature information as the first / second candidate graphic;

[0026] The label carried by the candidate graphic is used to represent the content of the candidate graphic.

[0027] In an exemplary embodiment, the first feature information or the second feature information includes original line features of a graphic formed by the trajectory;

[0028] The acquiring of the first / second candidate graphics that successfully matches the first / second feature information includes:

[0029] respectively calculating a first similarity between each candidate graphic and the original line feature in the first / second feature information;

[0030] The candidate graphics whose first similarity satisfies a first preset condition are selected as the first / second candidate graphics.

[0031] In an exemplary embodiment, the first feature information or the second feature information includes original line features of a graphic formed by the trajectory;

[0032] The acquiring of the first / second candidate graphics that successfully matches the first / second feature information includes:

[0033] For each candidate graph, perform the following operations:

[0034] Calculating the similarity between the original line features in the first / second candidate graphics and the candidate graphics, as well as the similarity between the original line features in the first / second candidate graphics and the sub-graphics contained in the candidate graphics; to determine a second similarity with the candidate graphics;

[0035] selecting a candidate graphic whose second similarity satisfies a second preset condition as the first / second candidate graphic;

[0036] The subgraphs contained in the candidate graph are saved corresponding to the candidate graph.

[0037] In an exemplary embodiment, the first feature information or the second feature information includes a contour feature of a graph formed by the trajectory;

[0038] The acquiring of the first / second candidate graphics that successfully matches the first / second feature information includes:

[0039] respectively calculating a third similarity between the contour line of each candidate graphic and the contour line features in the first / second feature information;

[0040] The candidate graph whose third similarity satisfies a third preset condition is selected as the first / second candidate graph.

[0041] In an exemplary embodiment, the first candidate object includes a first candidate instruction, and the second candidate object includes a second candidate instruction;

[0042] The step of executing the operation corresponding to the first candidate object in response to the user's selection operation on the first selection icon, and executing the operation corresponding to the second candidate object in response to the user's selection operation on the second selection icon, includes:

[0043] In response to a user's selection operation on the first selection icon, executing an operation corresponding to the first candidate instruction; and in response to a user's selection operation on the second selection icon, executing an operation corresponding to the second candidate instruction;

[0044] The determining of candidate objects according to the trajectory point set includes:

[0045] The trajectory formed by the trajectory point set is identified as a standard trajectory, and different candidate instructions are obtained according to the standard trajectory as the first candidate instruction and the second candidate instruction, respectively.

[0046] In an exemplary embodiment, the first candidate object is a third candidate graphic, and the second candidate object is a third candidate instruction;

[0047] The displaying of the first candidate object in response to the user's selection operation on the first selection icon, and the performing of the operation corresponding to the second candidate object in response to the user's selection operation on the second selection icon, include:

[0048] In response to a user's selection operation on the first selection icon, displaying the third candidate graphic; and in response to a user's selection operation on the second selection icon, executing an operation corresponding to the third candidate instruction;

[0049] The determining of candidate objects according to the trajectory point set includes:

[0050] obtaining third feature information according to the trajectory formed by the set of trajectory points, and obtaining a third candidate graphic that successfully matches the third feature information;

[0051] The trajectory formed by the trajectory point set is identified as a standard trajectory, and the third candidate instruction is obtained according to the standard trajectory.

[0052] In an exemplary embodiment, determining the candidate object according to the trajectory point set further includes:

[0053] In response to a failure to identify the trajectory formed by the trajectory point set as a standard trajectory, fourth feature information is obtained based on the trajectory formed by the trajectory point set, and a fourth candidate graphic that successfully matches the fourth feature information is obtained as the second candidate object; the fourth feature information is different from the third feature information.

[0054] In an exemplary embodiment, the interaction method is applied to writing software; obtaining a set of trajectory points input by a user includes:

[0055] In response to an input operation by a user in an input interface of the writing software, obtaining a set of track points input by the user in the input interface;

[0056] The displaying of a first selection icon for representing the first candidate object and a second selection icon for representing the second candidate object comprises:

[0057] Displaying a first selection icon for representing the first candidate object and a second selection icon for representing the second candidate object in the input interface;

[0058] The displaying of the first candidate object or executing the operation corresponding to the first candidate object in response to the user's selection operation of the first selection icon, and the displaying of the second candidate object or executing the operation corresponding to the second candidate object in response to the user's selection operation of the second selection icon include:

[0059] In response to a user selecting the first selection icon in the input interface, displaying the first candidate object in the input interface, or executing an operation corresponding to the first candidate object in the writing software;

[0060] In response to the user's selection operation on the second selection icon in the input interface, the second candidate object is displayed in the input interface, or an operation corresponding to the second candidate object is executed in the writing software.

[0061] In an exemplary embodiment, displaying a first selection icon for representing a first candidate object and a second selection icon for representing a second candidate object includes:

[0062] The first candidate object is a candidate graphic, and a thumbnail of the first candidate object is displayed as the first selection icon; or the first candidate object is a candidate instruction, and description information of the first candidate object is displayed as the first selection icon;

[0063] The second candidate object is a candidate graphic, and a thumbnail of the second candidate object is displayed as the first selection icon; or the second candidate object is a candidate instruction, and description information of the second candidate object is displayed as the second selection icon.

[0064] In an exemplary embodiment, after displaying a first selection icon for representing the first candidate object and a second selection icon for representing the second candidate object, the method further includes:

[0065] In response to a predetermined operation of the user, updating the determined first candidate object and / or second candidate object;

[0066] In response to an update of the first candidate object, the displayed first selection icon is updated accordingly; in response to an update of the second candidate object, the displayed second selection icon is updated accordingly.

[0067] In an exemplary embodiment, obtaining a trajectory point set input by a user includes:

[0068] In response to a pause in the user's input operation, a preset control is displayed; wherein the detection of a pause in the user's input operation means that the length of time since the last sampling result of the input operation was received reaches or exceeds a preset time threshold;

[0069] In response to the user's operation on the preset control, track points corresponding to the user's input operation between the current detection of a pause in the user's input operation and the previous detection of a pause in the user's input operation are obtained and saved as the track point set.

[0070] On the other hand, an embodiment of the present application provides an electronic device, comprising: a memory and a processor; the memory is configured to store a program for interaction; the processor is configured to read and execute the program for interaction, and perform the above-mentioned interaction method.

[0071] On the other hand, an embodiment of the present application provides a computer storage medium configured to store an interactive program; when the interactive program is executed, the above-mentioned interactive method is performed.

[0072] The embodiments of the present application can automatically, efficiently, and quickly provide a variety of candidate objects related to the trajectory for the user to choose from based on the trajectory formed by the user input, and can display corresponding graphics or perform corresponding operations based on the user's selection, thereby greatly improving the user experience.

[0073] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings are provided to provide a further understanding of the technical solutions herein and constitute part of the specification. Together with the embodiments of the present application, they are used to explain the technical solutions herein and do not constitute a limitation of the technical solutions herein. The shapes and sizes of one or more components in the accompanying drawings do not reflect the actual scale and are intended only to illustrate the contents herein.

[0075] Figure 1 A flowchart of an interactive method according to an embodiment of the present application;

[0076] Figure 2 A schematic diagram of an electronic device according to an embodiment of the present application;

[0077] Figure 3 This is a schematic diagram of the implementation process of Example 1;

[0078] Figure 4 Schematic diagram of the trajectory formed by the trajectory point set in Example 1;

[0079] Figure 5 This is a diagram showing the selection icon in Example 1;

[0080] Figure 6Schematic diagram of the process of calculating contour similarity in Example 1;

[0081] Figure 7 This is a schematic diagram of the situation where the graph formed by the trajectory in Example 1 is similar to the candidate graph subgraph;

[0082] Figure 8 Schematic diagram of the candidate graphs formed by the subgraphs in Example 1;

[0083] Figure 9 This is a diagram showing the selection icon in Example 2. DETAILED DESCRIPTION

[0084] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments can be implemented in a variety of different forms. A person skilled in the art can easily understand that the method and content can be transformed into one or more forms without departing from the purpose and scope of this document. Therefore, this document should not be interpreted as being limited to the contents described in the following embodiments. In the absence of conflict, the embodiments and features in the embodiments herein can be combined with each other in any manner.

[0085] In the drawings, the size of one or more components, layer thicknesses, or regions may be exaggerated for clarity. Therefore, one embodiment herein is not necessarily limited to these dimensions, and the shapes and sizes of various components in the drawings do not reflect true to scale. Furthermore, the drawings schematically illustrate idealized examples, and implementations herein are not limited to the shapes or values ​​shown in the drawings.

[0086] The ordinal numbers such as "first," "second," and "third" herein are provided to avoid confusion among constituent elements, and are not intended to limit the quantity. The term "plurality" herein refers to a quantity of two or more.

[0087] For convenience, terms such as "center," "upper," "lower," "front," "back," "vertical," "horizontal," "top," "bottom," "inner," and "outer" are used herein to describe the positional relationships of components with reference to the accompanying drawings. This is intended solely to facilitate the description of this specification and simplify the description, and is not intended to indicate or imply that the device or component referred to must have a specific orientation, be constructed, or operate in a specific position. Therefore, they should not be construed as limitations of this specification. The positional relationships of components may vary depending on the orientation in which they are described. Therefore, the terms are not limited to those described in this specification and may be appropriately replaced as appropriate.

[0088] In this document, unless otherwise specified or limited, the terms "mounted," "connected," and "connected" should be interpreted broadly. For example, they can refer to fixed, removable, or integral connections; mechanical or electrical connections; direct connections, indirect connections through intermediaries, or internal communication between two components. Those skilled in the art will understand the meaning of these terms in this document based on the specific circumstances.

[0089] As used herein, "electrically connected" includes components connected together via an element having some electrical function. There are no particular limitations on the "element having some electrical function" as long as it enables the transmission of electrical signals between the connected components. Examples of "element having some electrical function" include not only electrodes and wiring, but also switching elements such as transistors, resistors, inductors, capacitors, and other components with one or more functions.

[0090] In this article, "parallel" means that the angle formed by two straight lines is greater than -10° and less than 10°, and thus includes a state where the angle is greater than -5° and less than 5°. In addition, "perpendicular" means that the angle formed by two straight lines is greater than 80° and less than 100°, and thus includes a state where the angle is greater than 85° and less than 95°.

[0091] The term "about" as used herein refers to a numerical value that is not strictly limited to a certain limit and is within a range of process and measurement errors.

[0092] An embodiment of the present application provides an interactive method, such as Figure 1 As shown, steps S110-S140 are included:

[0093] S110, obtaining a trajectory point set input by a user, where the trajectory point set includes at least one trajectory point;

[0094] S120, determining candidate objects according to the trajectory point set, where the candidate objects include: a first candidate object and a second candidate object;

[0095] S130, displaying a first selection icon for representing a first candidate object and a second selection icon for representing a second candidate object;

[0096] S140. In response to the user's selection operation on the first selection icon, display the first candidate object or execute the operation corresponding to the first candidate object; in response to the user's selection operation on the second selection icon, display the second candidate object or execute the operation corresponding to the second candidate object.

[0097] The interactive method of the embodiment of the present application can automatically, efficiently and quickly provide a variety of candidate objects related to the trajectory for the user to choose based on the trajectory formed by the user input, and can display corresponding graphics or perform corresponding operations based on the user's selection, greatly improving the user experience.

[0098] In an exemplary embodiment, the above-mentioned interaction method can be applied to electronic devices that include touch screens, such as electronic whiteboards, tablet computers, and smart phones; the touch screen may include a touchpad, a sampling unit, a display panel, and a display driving unit; wherein the touchpad is a transparent plate arranged on the upper surface of the display panel; the user uses a finger or a stylus to perform input operations, and the sampling unit samples the user's touch to obtain the real-time coordinates of the track point formed by the contact position of the finger or touch tip with the touchpad, and sends the real-time coordinates to the display driving unit, so that the track point can be displayed at the position corresponding to the real-time coordinate on the display panel; the user uses a finger or a stylus to continuously slide on the touchpad to write or draw, and through continuous sampling and sending by the sampling unit, the display panel can display the track formed by the set of track points corresponding to multiple real-time coordinates, which is consistent with the track of the user writing or drawing on the touchpad.

[0099] In one exemplary embodiment, the above-described interaction method can be applied to a handwriting tablet and computer system or a computer equipped with a touchpad. User input operations are received via the handwriting tablet or touchpad, and the resulting track points can be displayed on the computer screen. In other exemplary embodiments, the user can use a mouse or keyboard to control cursor movement instead of using a touchpad to input track points.

[0100] In an exemplary embodiment, in step S110, a set of track points input by the user in a preset input area (such as an input area corresponding to a dialog box, an editing interface, or a predetermined APP interface) can be obtained; or a set of track points input by the user in a certain time period (such as between two pauses in user operations) can be obtained.

[0101] In an exemplary embodiment, in step S120, one or more first candidate objects and one or more second candidate objects may be determined.

[0102] In an exemplary embodiment, in step S120, the determined candidate objects may further include a third candidate object or more candidate objects.

[0103] In an exemplary embodiment, there are multiple first candidate objects and multiple second candidate objects, and accordingly, there are multiple first selection icons and multiple second selection icons, which correspond one-to-one to the first candidate objects and the second candidate objects, respectively.

[0104] In one exemplary embodiment, the category of the candidate object determined in step S120 is a graphic or an instruction corresponding to an operation; the category of the first candidate object and the second candidate object can both be graphics, or both be instructions, or one can be a graphic and the other can be an instruction. Accordingly, in step S140, if the category of the candidate object is a graphic and the corresponding first or second selection icon is clicked, the corresponding graphic can be displayed; if the category of the candidate object is an instruction and the corresponding first or second selection icon is clicked, the operation corresponding to the instruction can be executed.

[0105] In an exemplary embodiment, the first selection icon and the second selection icon displayed in step S130 are for the user to click to select the corresponding first candidate object and second candidate object. The form of the first and second selection icons can be determined according to the category of the first and second candidate objects respectively; for example, the first and second selection icons can be controls that use thumbnails of graphics as icons to display content, or can be controls that use description information of instructions as icons to display content. The description information can be in the form of graphics or text. For example, if the first candidate object is an "undo" instruction, the first selection icon can be displayed as a counterclockwise arrow; for example, if the first candidate object is a "delete" instruction, the first selection icon can display "del".

[0106] In an exemplary embodiment, in step S130, the first selection icon and the second selection icon may be displayed simultaneously, or the first selection icon and the second selection icon may be displayed sequentially; the first and second selection icons may be displayed in different areas, respectively. The order and areas of display may be set by default or may be user-defined.

[0107] In one exemplary embodiment, in step S140, the user is only allowed to select one selection icon. In response to a user selection operation (such as, but not limited to, a click or long press) on a selection icon (a first selection icon or a second selection icon), the first selection icon and the second selection icon may cease to be displayed, and the candidate object corresponding to the selection icon selected by the user may be displayed, or the corresponding operation may be performed according to the corresponding candidate object. That is, by selecting a selection icon, the user may select a first candidate object or a second candidate object; and according to the category of the candidate object selected by the user, the candidate object may be displayed accordingly, or the corresponding operation may be performed.

[0108] In an exemplary embodiment, in step S140, the user may be allowed to select two or more selection icons. These two or more selection icons may all be first / second selection icons, or some may be first selection icons and some may be second selection icons. For example, if the user holds down the "ctrl" key and selects a first selection icon and a second selection icon, in response to the user's selection operation, the first and second candidate objects may be displayed simultaneously, or operations corresponding to the first and second objects may be performed sequentially, or one candidate object may be displayed and an operation corresponding to the other candidate object may be performed.

[0109] In an exemplary embodiment, in step S140 , when displaying the candidate object, it may be displayed at the current location of the cursor, or the candidate object may be used to replace the trajectory formed by the trajectory point set for display.

[0110] In an exemplary embodiment, the first candidate object includes a first candidate graphic, and the second candidate object includes a second candidate graphic;

[0111] In response to a user's selection operation on a first selection icon, a first candidate object is displayed; in response to a user's selection operation on a second selection icon, a second candidate object is displayed, comprising:

[0112] In response to a user's selection operation on a first selection icon, a first candidate graphic is displayed; in response to a user's selection operation on a second selection icon, a second candidate graphic is displayed;

[0113] Determine candidate objects based on a set of trajectory points, including:

[0114] Obtain first feature information according to the trajectory formed by the trajectory point set, and obtain a first candidate graph that successfully matches the first feature information.

[0115] Obtaining second feature information according to the trajectory formed by the trajectory point set, and obtaining a second candidate graphic that successfully matches the second feature information;

[0116] The first feature information is different from the second feature information.

[0117] The first and second candidate graphics may be, but are not limited to, graphics in a preset graphic database; there may be one or more graphic databases, which may be set locally, on a server, in the cloud, etc.

[0118] In an exemplary embodiment, the first characteristic information and the second characteristic information respectively include one or more of the following:

[0119] One or more character features corresponding to the trajectory;

[0120] The original line features of the figure formed by the trajectory;

[0121] The contour line characteristics of the figure formed by the trajectory.

[0122] In addition to the types listed above, other types of feature information can be selected for matching, and this application does not impose any restrictions on this.

[0123] The original line feature refers to the line itself formed by the trajectory points in the trajectory point set and used to form a graphic.

[0124] In an exemplary embodiment, the first candidate object and the second candidate object may be candidate graphics that successfully match different types of feature information, respectively. For example, the first candidate object is a second candidate graphic that successfully matches a character feature, and the second candidate object is a second candidate graphic that successfully matches an original line feature.

[0125] In an exemplary embodiment, the first feature information or the second feature information includes one or more character features corresponding to the trajectory;

[0126] Acquiring the first / second feature information according to the trajectory formed by the trajectory point set includes:

[0127] Obtain one or more character features corresponding to the trajectory formed by the trajectory point set;

[0128] Obtaining the first / second candidate graphics that successfully matches the first / second feature information includes:

[0129] A candidate graphic whose carried label is at least partially identical to the character features in the first / second feature information is selected as the first / second candidate graphic; wherein the label carried by the candidate graphic is used to represent the content of the candidate graphic.

[0130] In one implementation of this embodiment, one or more character features corresponding to the trajectory may be extracted through, but is not limited to, optical character recognition (OCR). Alternatively, the character features corresponding to the trajectory may be obtained through other methods, such as extracting stroke features and stroke order features from the trajectory and comparing them with corresponding features of different characters in a character library to determine the character features corresponding to the trajectory.

[0131] In one implementation of this embodiment, a candidate graphic may have one or more tags, each of which may include one or more characters. In one example, a graphic database is searched for tags with identical content based on the identified character features. If any tag of a candidate graphic matches at least one identified character feature, the candidate graphic is considered a match.

[0132] In one implementation of this embodiment, the criteria for successful matching can be set by yourself. It can be set so that the label must be the same as all the character features for the match to be successful. For example, if the acquired character features and the label are both "umbrella", the match is successful; or it can be set so that the label is the same as some of the character features for the match to be successful. For example, if the acquired character features are "umbrella" and the label is "umbrella", it can be determined that the match is successful; for example, if the acquired character features are "cat" and the label is "cat", it can be determined that the match is successful.

[0133] In one implementation of this embodiment, if there are multiple successfully matched candidate graphics, priority may be given to those with tags that are identical to the acquired character features. Alternatively, a relevance value or priority may be pre-added to the tags, giving priority to those corresponding to tags with higher relevance values ​​or higher priorities. The relevance value may be used to indicate the degree of correlation between the candidate graphics and the tag. In one example, the relevance value between an umbrella graphic and the tag "umbrella" is 100%, while the relevance value between a mushroom graphic and the tag "umbrella" is 20%. If multiple candidate graphics are successfully matched based on the character feature "umbrella," the umbrella graphic may be given priority.

[0134] In an exemplary embodiment, the first feature information or the second feature information includes original line features of a graphic formed by the trajectory;

[0135] Obtaining the first / second candidate graphics that successfully matches the first / second feature information includes:

[0136] Calculating first similarities between each candidate graphic and the original line features in the first / second feature information respectively;

[0137] The candidate graphics whose first similarity satisfies the first preset condition are selected as the first / second candidate graphics.

[0138] In one implementation of this embodiment, the first preset condition may be that the first similarity of the candidate graphics is one of the N highest first similarities; and selecting the candidate graphics whose first similarity satisfies the first preset condition includes selecting the N candidate graphics with the highest first similarities. N is a preset positive integer and may be the number of candidate graphics to be displayed for user selection.

[0139] In another implementation of this embodiment, the first preset condition may be that the first similarity exceeds or reaches a preset first similarity threshold.

[0140] In one implementation of this embodiment, the first similarity between a candidate graphic and the original line feature can be calculated as follows:

[0141] Sampling the original line feature according to a first spacing to obtain a first sampling point set, and sampling the candidate figure according to the first spacing to obtain a second sampling point set;

[0142] The sampling points in the first and second sampling point sets are paired using the Hungarian pairing algorithm; if the number of sampling points in one sampling point set to be paired is less than that in the other sampling point set, the number of sampling point pairs obtained is equal to the number of sampling points in the sampling point set with the smaller number of sampling points;

[0143] The distance of each set of sampling point pairs is calculated respectively, and the first similarity between the candidate figure and the original line feature is determined according to the sum of the distances of all sampling point pairs.

[0144] The first spacing can be set arbitrarily. For example, if relatively dense sampling is required, the first spacing can be set to be smaller.

[0145] The distance between the sampling point pairs calculated may be, but is not limited to, the Euclidean distance; the smaller the distance, the greater the similarity.

[0146] In this embodiment, the first similarity is not limited to being calculated in the above manner, and other manners may be used to determine the first similarity between the original line feature and the candidate graphics, such as calculating cosine similarity, or calculating similarity using fingerprint information.

[0147] In an exemplary embodiment, the first feature information or the second feature information includes original line features of a graphic formed by the trajectory;

[0148] Obtaining the first / second candidate graphics that successfully matches the first / second feature information includes:

[0149] For each candidate graph, perform the following operations:

[0150] Calculating the similarity between the original line feature in the first / second feature information and the candidate graphic, and the similarity between the original line feature and the sub-graph contained in the candidate graphic, to determine a second similarity with the candidate graphic;

[0151] Selecting a candidate graphic whose second similarity satisfies a second preset condition as the first / second candidate graphic;

[0152] The subgraphs contained in the candidate graph are saved corresponding to the candidate graph.

[0153] In this embodiment, the greatest similarity may be selected from the calculated similarities with the candidate graph and the similarities with the subgraph of the candidate graph as the similarity with the candidate graph.

[0154] In one implementation of this embodiment, the second preset condition may be that the second similarity of the candidate graphic is one of the N highest second similarities; and selecting the candidate graphic whose second similarity satisfies the second preset condition includes selecting the N highest second similarities. N is a preset positive integer and may be the number of candidate graphics to be displayed for user selection.

[0155] In another implementation of this embodiment, the second preset condition may be that the second similarity exceeds or reaches a preset second similarity threshold.

[0156] In this embodiment, the implementation details of calculating the similarity between the original line feature and the candidate graphics or sub-graphics may refer to the method of calculating the first similarity.

[0157] This embodiment can select candidate graphics that are similar to the graphics formed by the part and the trajectory, that is, select candidate graphics that are similar to the graphics formed by the "part" and the trajectory.

[0158] In this embodiment, in the process of storing the candidate graph, the subgraphs and the candidate graph composed of the subgraphs may be stored together, or the subgraphs may be separated from the candidate graph and stored together with the candidate graph.

[0159] In this embodiment, not every candidate graphic has a sub-graph; only some of the candidate graphics may have sub-graphs. The total number of sub-graphs contained in a candidate graphic may be smaller than the candidate graphic itself; identical parts of candidate graphics may be treated as the same sub-graph. For example, if the candidate graphic is a house with multiple identical windows, each window can use the same "window sub-graph," while the rest of the "house graphic" is not treated as a sub-graph.

[0160] In an exemplary embodiment, the first feature information or the second feature information includes a contour feature of a figure formed by the trajectory;

[0161] Obtaining the first / second candidate graphics that successfully matches the first / second feature information includes:

[0162] respectively calculating a third similarity between the contour line of each candidate graphic and the contour line feature in the first / second feature information;

[0163] The candidate graphics whose third similarity satisfies the third preset condition are selected as the first / second candidate graphics.

[0164] In one implementation of this embodiment, the third preset condition may be that the third similarity of the candidate graphics is one of the N highest third similarities; and selecting the candidate graphics whose third similarity satisfies the third preset condition includes selecting the N candidate graphics with the highest third similarities. N is a preset positive integer and may be the number of candidate graphics desired to be displayed for user selection.

[0165] In another implementation of this embodiment, the third preset condition may be that the third similarity exceeds or reaches a preset third similarity threshold.

[0166] In this embodiment, the process of calculating the third similarity between a candidate graphic and a contour feature may include:

[0167] Extracting contour features from the graph formed by the trajectory, performing normalization processing, and sampling according to the second spacing to obtain a third sampling point set;

[0168] Sampling the contour line of the candidate figure according to the second spacing to obtain a fourth sampling point set;

[0169] The sampling points in the third and fourth sampling point sets are paired using the Hungarian pairing algorithm; if the number of sampling points in one sampling point set to be paired is less than that in the other sampling point set, the number of sampling point pairs obtained is equal to the number of sampling points in the sampling point set with the smaller number of sampling points;

[0170] The distance of each pair of sampling points is calculated respectively, and the third similarity between the contour line feature and the contour line of the candidate figure is determined according to the sum of the distances of all the sampling point pairs.

[0171] The second spacing can be set arbitrarily. For example, if relatively dense sampling is required, the second spacing can be set to be smaller.

[0172] The calculated distance between the sampling point pairs may be, but is not limited to, Euclidean distance.

[0173] In this embodiment, the third similarity is not limited to being calculated in the above manner. Any method for determining the similarity between curves may be used to determine the third similarity. For example, the third similarity may be determined by discrete Fréchné distance, Hausdorff distance, Pearson correlation coefficient, and the like.

[0174] In some exemplary embodiments, the first feature information or the second feature information includes a figure formed by the trajectory or a contour line of the figure, and obtaining the first / second feature information may include:

[0175] The graphics formed by the trajectory are scaled according to multiple preset ratios;

[0176] Accordingly, the similarity between the original line features in the first / second feature information and the candidate graphic (or its sub-image), or the similarity between the contour line features in the first / second feature information and the contour line of the candidate graphic, is calculated according to the graphics scaled to different proportions; different similarities are calculated for different proportions, and the maximum value is taken as the final similarity.

[0177] For example, to calculate the first similarity between the original line feature IMG1 and the candidate figure IMG2, we first scale IMG1 according to the preset scaling ratios of 0.5, 1, and 2 to obtain IMG1. 1-1 、IMG 1-2 and IMG 1-3 About IMG 1-1 and IMG2 are densely sampled at equal intervals, and then the two sampling point sets are Hungarian paired, and the distance of each sampling point pair is calculated to obtain the total distance L 1-1 ; Similarly, for IMG 1-2 and IMG 1-3 Get the total distance L respectively 1-2 and L 1-3 The final first similarity between IMG1 and IMG2 is determined based on the minimum value (corresponding to the maximum similarity) of the three distance sums as the final distance.

[0178] This embodiment can reduce the influence of the size of the graph on the distance (or similarity) calculation.

[0179] In this embodiment, the preset ratio can be set to 0.1, 0.2, ..., 9, 10.

[0180] In an exemplary embodiment, the first candidate object includes a first candidate instruction, and the second candidate object includes a second candidate instruction;

[0181] In response to a user's selection operation on the first selection icon, executing an operation corresponding to the first candidate object; in response to a user's selection operation on the second selection icon, executing an operation corresponding to the second candidate object, including:

[0182] In response to a user's selection operation on a first selection icon, executing an operation corresponding to a first candidate instruction; and in response to a user's selection operation on a second selection icon, executing an operation corresponding to a second candidate instruction;

[0183] The determining of candidate objects according to the trajectory point set includes:

[0184] The trajectory formed by the trajectory point set is identified as a standard trajectory, and different candidate instructions are obtained according to the standard trajectory as the first candidate instruction and the second candidate instruction, respectively.

[0185] In one implementation of this embodiment, the trajectory may be identified as a first standard trajectory and a second standard trajectory, and the candidate instruction corresponding to the first standard trajectory is obtained as the first candidate instruction, and the candidate instruction corresponding to the second standard trajectory is obtained as the second standard instruction.

[0186] In another implementation of this embodiment, the trajectory may be identified as a standard trajectory, and multiple candidate instructions corresponding to the standard trajectory are obtained as the first and second candidate instructions, respectively.

[0187] In this embodiment, if multiple standard trajectories are identified, they may be sorted according to the similarity between the multiple standard trajectories and the trajectory formed by the trajectory point set, and the standard trajectory with the highest ranking may be selected to obtain the first and second candidate instructions.

[0188] In this embodiment, the standard trajectory is a trajectory with a standardized form. Because the degree of personalization of the user input trajectory is very high, even if the intention is to input the same trajectory (for example, everyone wants to input the trajectory "Z"), the trajectory forms actually input by different users will be different, and even the trajectories input twice by the same user will be different; in order to adapt to this differentiated input, it is necessary to first identify the trajectory input by the user as a standard trajectory and then obtain the corresponding candidate instructions, that is, to perform a "standardization" process first. The process of identifying the trajectory as a standard trajectory can be similar to the method of identifying the characters input by the user in handwriting input: extract the features of the trajectory and compare them with the features of the standard trajectory to identify the trajectory as one of the preset standard trajectories. If the features of the trajectory input by the user cannot be identified as any standard trajectory, it means that the trajectory input by the user does not have a corresponding candidate instruction, and it is impossible to obtain a candidate object whose category is a candidate instruction based on the trajectory input by the user.

[0189] In this embodiment, multiple standard tracks can be pre-set, and one or more candidate instructions corresponding to each standard track can be set; the style of the standard track can be any one or more of the following: graphics, symbols, and characters.

[0190] An exemplary embodiment, in this embodiment, the trajectory input by the user may be recognized as more than one standard trajectory. For example, if the user inputs a slanted "Z", it can be recognized as the standard trajectory "Z" or as the standard trajectory "N". Candidate instructions corresponding to "Z" and "N" can be obtained, and the candidate instruction corresponding to "Z" is used as the first candidate object, and the candidate instruction corresponding to "N" is used as the second candidate object.

[0191] In another exemplary embodiment, the first and second candidate objects can be different candidate instructions corresponding to a standard trajectory. For example, the standard trajectory "Z" corresponds to two instructions: delete trajectory and clear dialog box. These two instructions can be used as the first and second candidate objects, respectively.

[0192] In an exemplary embodiment, the first candidate object is the third candidate graphic, and the second candidate object is the third candidate instruction;

[0193] Determining candidate objects based on a set of trajectory points includes:

[0194] In response to a user's selection operation on a first selection icon, a first candidate object is displayed; and in response to a user's selection operation on a second selection icon, an operation corresponding to the second candidate object is performed, including:

[0195] In response to the user's selection operation on the first selection icon, displaying a third candidate graphic, and in response to the user's selection operation on the second selection icon, executing an operation corresponding to the third candidate instruction;

[0196] Determine candidate objects based on a set of trajectory points, including:

[0197] Obtaining third feature information according to the trajectory formed by the trajectory point set, and obtaining a third candidate graphic that successfully matches the third feature information;

[0198] The trajectory formed by the trajectory point set is identified as a standard trajectory, and a third candidate instruction is obtained according to the standard trajectory.

[0199] In one implementation of this embodiment, determining a candidate object based on a trajectory point set may further include:

[0200] In response to a failure to identify the trajectory formed by the trajectory point set as a standard trajectory, fourth feature information is obtained based on the trajectory formed by the trajectory point set, and a fourth candidate graphic that successfully matches the fourth feature information is obtained as the second candidate object; the fourth feature information is different from the third feature information.

[0201] In this embodiment, the first and second candidate objects are defaulted to be a candidate graphic and a candidate instruction respectively; however, if the trajectory input by the user cannot be recognized as a standard trajectory, that is, a candidate instruction cannot be obtained, the second candidate object can be adaptively changed to a candidate graphic that is successfully matched according to the fourth feature information that is different from the third feature information.

[0202] In an exemplary embodiment, the above interaction method may be applied to writing software; obtaining a set of trajectory points input by the user includes:

[0203] In response to an input operation by a user in an input interface of the writing software, obtaining a set of track points input by the user in the input interface;

[0204] Displaying a first selection icon for representing a first candidate object and a second selection icon for representing a second candidate object includes:

[0205] Displaying a first selection icon for representing a first candidate object and a second selection icon for representing a second candidate object in an input interface of the writing software;

[0206] In response to the user's selection operation on the first selection icon, displaying the first candidate object or performing the operation corresponding to the first candidate object, and in response to the user's selection operation on the second selection icon, displaying the second candidate object or performing the operation corresponding to the second candidate object includes:

[0207] In response to a user selecting a first selection icon in an input interface of the writing software, displaying a first candidate object in the input interface, or performing an operation corresponding to the first candidate object in the writing software;

[0208] In response to the user's selection operation on the second selection icon in the input interface, the second candidate object is displayed in the input interface, or an operation corresponding to the second candidate object is executed in the writing software.

[0209] In an exemplary embodiment, displaying a first selection icon for representing a first candidate object and a second selection icon for representing a second candidate object includes:

[0210] The first candidate object is a candidate graphic, and a thumbnail of the first candidate object is displayed as the first selection icon; or the first candidate object is a candidate instruction, and description information corresponding to the first candidate object is displayed as the first selection icon;

[0211] The second candidate object is a candidate graphic, and a thumbnail of the second candidate object is displayed as the first selection icon; or the second candidate object is a candidate instruction, and description information corresponding to the second candidate object is displayed as the second selection icon.

[0212] In an exemplary embodiment, after displaying a first selection icon for representing a first candidate object and a second selection icon for representing a second candidate object, the method further includes:

[0213] In response to a predetermined operation of the user, updating the determined first candidate object and / or second candidate object;

[0214] In response to an update of the first candidate object, the displayed first selection icon is updated accordingly; in response to an update of the second candidate object, the displayed second selection icon is updated accordingly.

[0215] In this embodiment, for example, if the user clicks the control representing "more" or clicks the icon representing "refresh", another batch of candidate objects can be obtained (such as candidate graphics that have been successfully matched but not displayed, candidate graphics reselected according to other feature information, candidate instructions reacquired by identifying the trajectory as other standard trajectories, etc.), and the displayed selection icon can be updated accordingly.

[0216] In an exemplary embodiment, obtaining a trajectory point set input by a user includes:

[0217] In response to a pause in the user's input operation, a preset control is displayed; wherein the detection of a pause in the user's input operation means that the length of time since the last sampling result of the input operation was received reaches or exceeds a preset time threshold;

[0218] In response to the user's operation on the preset control, the track points corresponding to the user input operation between the current detection of the user input operation pause and the previous detection of the user input operation pause are obtained and saved as a track point set.

[0219] In this embodiment, a set of track points inputted during a specific time period (between two operation pauses) is obtained. In other embodiments, a set of track points inputted within a predetermined input area may be obtained.

[0220] The present application also provides an electronic device, such as Figure 2 As shown, it includes: a memory 21 and a processor 22;

[0221] The memory 21 is configured to store the interactive program;

[0222] The processor 22 is configured to read and execute the program for interaction, and perform the interaction method in any of the above embodiments.

[0223] In an exemplary embodiment, the electronic device may further include:

[0224] A touchpad configured to receive user input;

[0225] A sampling unit, a display panel and a display driver unit; the real-time coordinates of the touch position are sampled by the sampling unit and sent to the display driver unit, which can be displayed on the display panel according to the user's input operation on the touch panel; wherein the display driver unit can also display all or part of the successfully matched alternative graphics on the display panel according to the control of the processor.

[0226] An embodiment of the present application further provides a computer storage medium configured to store an interactive program; when the interactive program is executed, the interactive method in any of the above embodiments is performed.

[0227] Two examples are used below to illustrate the embodiments of this application.

[0228] Example 1

[0229] This example is applied to an electronic whiteboard, which includes a touch screen, a memory, and a processor. The candidate objects include: a first candidate object, a second candidate object, and a third candidate object, all of which are candidate graphics.

[0230] The overall implementation process of this example is as follows Figure 3 As shown:

[0231] When a user writes or draws on an electronic whiteboard, the processor detects a pause and identifies the set of track points formed by the user's writing or drawing to obtain candidate graphics. The candidate graphics are returned to the front end as recommendation results and displayed in the form of selection icons for the user to choose.

[0232] In this example, the method for obtaining candidate graphics includes: obtaining three types of feature information from the trajectory formed by the trajectory point set, and matching them with the candidate graphics in the graphics database respectively; depending on the type of feature information obtained, the matching method is divided into:

[0233] Character matching (this method matches based on the character features corresponding to the trajectory), contour matching (this method matches based on the contour features of the graphics formed by the trajectory), and graphic matching (this method matches based on the original line features of the graphics formed by the trajectory).

[0234] In this example, the sampling unit can send the sampled real-time coordinates of the user's touch on the electronic whiteboard to the processor; the processor can trigger the start of the interactive operation when it detects a user pause. Detecting a user pause can mean that the length of time since the last real-time coordinates were received from the sampling unit exceeds a preset time threshold.

[0235] In this example, after the interactive operation begins, the processor controls the display driver unit to display a dotted frame with an interactive trigger point on the display panel to send an interactive request to the user. The dotted frame frames the trajectory between the last pause detected and the current pause detected, such as Figure 4 As shown; that is, the trajectory in the dotted box is the trajectory formed by the acquired trajectory point set, including multiple trajectory points including the first trajectory point received after the last pause was detected and the last trajectory point received before the pause was detected this time.

[0236] In addition to the dotted frame used in this example, other forms can be used to identify the acquired track point set, such as changing the color of the corresponding track, or highlighting the background color of the area containing the corresponding track.

[0237] In this example, the interaction trigger point is set to a dot with a certain area located in the lower right corner of the dotted box, and the triggering method is to click on the dot; in implementation, the interaction trigger point can adopt other display forms, be located in other positions, and the triggering method can be various; for example, the interaction trigger point can be set to an icon in the menu bar, or can be set on the frame line of the dotted box or in the area within the dotted box; for example, the interaction contact point can be not set, and the interaction can be triggered directly by double-clicking or long pressing.

[0238] In this example, after displaying the dotted box, the processor determines the user's operation based on one or more actual coordinates sampled by the sampling unit and performs corresponding processing, including the following five cases:

[0239] (1) The user continues writing or drawing within the dotted box:

[0240] The processor does not process the information, but records the time of the pause and continues monitoring until the next pause.

[0241] (2) The user continues writing or drawing outside the dotted box:

[0242] The processor does not process the information, but records the time of the pause and continues monitoring until the next pause.

[0243] (3) The user long presses or holds down any position within the dotted box and drags:

[0244] The processor changes the coordinates of the track in the dotted box according to the user's dragging, and sends the changed coordinates to the display driving unit, so that the effect displayed on the display panel is that the track in the dotted box moves accordingly with the user's dragging.

[0245] (4) The user long presses or holds down the interaction trigger point in the lower right corner of the dotted box and drags:

[0246] The processor changes the distance between adjacent track points in the track according to the user's dragging, keeping the coordinates of one track point in the track unchanged, and the coordinates of other track points change with the change of distance; the processor sends the changed coordinates to the display driver unit, so that the effect displayed on the display panel is that the track in the dotted box scales as the user drags.

[0247] (5) The user clicks the interaction trigger point in the lower right corner of the dotted box:

[0248] The processor obtains three types of feature information of the trajectory in the dotted box: character features, original line features and contour features of the formed graphics, connects to a local or remote graphics database, and matches each type of feature information obtained with the candidate graphics contained in the graphics database to obtain candidate objects corresponding to each type of feature information, that is, candidate graphics that are successfully matched according to this type of feature information. The processor returns all or part of the successfully matched candidate graphics to the front end as candidate objects for display. In this example, the first candidate object is a candidate graphic that is successfully matched based on the character features corresponding to the trajectory, the second candidate object is a candidate graphic that is successfully matched based on the contour features of the graphics formed by the trajectory, and the third candidate object is a candidate graphic that is successfully matched based on the original line features of the graphics formed by the trajectory.

[0249] like Figure 5As shown, four first selection icons, three second selection icons, and three third selection icons are shown to the user, which are the thumbnails corresponding to the first, second, and third candidate objects respectively. In this example, the characters corresponding to the trajectory are also shown.

[0250] In this example, the processor responds to the user's click on any one of the selection icons and displays the corresponding candidate object. One way is to display the candidate object selected by the user in a dashed box, replacing the original trajectory in the dashed box. In this implementation, the candidate object can be scaled according to the size of the dashed box; another way is to display the candidate object selected by the user at a predetermined position or any position in its original size for the user to drag and scale according to needs later.

[0251] In addition to the click selection method, it is possible to respond to the user's long press on any one of the selection icons to display the corresponding candidate object.

[0252] Figure 5 In [description], the graphic recommendation area 5 can include four sub-areas, which display different contents respectively:

[0253] Among them, the best recognition result obtained by performing OCR on the trajectory shown in the OCR result sub-area 51 is used as the character corresponding to the trajectory. Figure 5 Taking the Chinese character "umbrella" as an example in [description].

[0254] The first selection icon of the first candidate object obtained by matching the OCR result as feature information is shown in the character graphic recommendation sub-area 52, that is, according to the obtained character features, the corresponding candidate graphic is found for recommendation.

[0255] The second selection icon of the second candidate object obtained by matching according to the contour line features of the graphic formed by the trajectory is shown in the similar graphic recommendation sub-area 53.

[0256] The third selection icon of the third candidate object obtained by matching according to the original line features of the graphic formed by the trajectory is shown in the included graphic recommendation sub-area 54.

[0257] In this example, the boundaries of the four sub-areas can each be marked with a dotted frame; each sub-area can have its own interactive trigger point, which in this example is set at the lower right corner of the dotted frame of the sub-area. When the user clicks the interactive trigger point of the sub-area 51 displaying the OCR results, other alternative OCR results will be displayed in the sub-area or in a new area, such as a suboptimal recognition result. This can solve the problem of users being dissatisfied with the OCR results and needing to independently select the recognition result; if the user selects another OCR result, the first candidate object will be re-obtained based on the result, and the first selection icon will be refreshed. When the user clicks the interactive trigger point in any of the other three sub-areas, selection icons for more candidate objects can be displayed in the corresponding sub-area.

[0258] In this example, the details of obtaining candidate graphics based on different types of feature information are as follows:

[0259] (1) Character graphic recommendation

[0260] In this example, when recommending graphics based on the character features corresponding to the trajectory, the character corresponding to the trajectory is first recognized through OCR, and the recognition result is used as the acquired character features. The recognition result can be a single character, or multiple characters, such as a word. The characters in this example include Chinese characters, letters, etc.

[0261] Based on the recognition results, a match is performed in the image database. The candidate images contained in the image database carry a label that indicates the content of the candidate image. If the label and at least one of the acquired character features are the same, the match is considered successful.

[0262] In response to the user clicking on the interactive trigger point of the character graphic recommendation sub-area 52, some or all of the candidate graphics that have not been displayed and have successfully matched can be selected for display.

[0263] (2) Similar graphics recommendation

[0264] The similar figures referred to in this example are figures with similar contour lines.

[0265] Considering that the user's handwriting is not necessarily text, it is possible that a simple sketch is drawn to find a graphic. In this case, recommendation based on OCR results is meaningless. The method of searching for similar graphics can be used for recommendation.

[0266] In this example, the process of recommending similar graphics includes: calculating the similarity between the contour features of the graphics formed by the trajectory and the contours of each candidate graphics in the graphics database, obtaining the top three candidate graphics with the highest contour similarity as the second candidate objects, and displaying the corresponding second selection icon. Figure 5In the example above, the thumbnails of the candidate graphics ranked 1 to 3 in similarity are displayed after the similarity is sorted from high to low. In response to the user clicking the interaction trigger point in the similar graphics recommendation sub-area 53, the candidate graphics ranked 4 to 6 in similarity are retrieved as the second candidate objects according to the order of similarity, and the displayed second selection icon is updated accordingly.

[0267] like Figure 6 As shown, the contour line similarity calculation includes the following steps 601-604:

[0268] 601. First, extract the contour features of the figure formed by the trajectory in the dotted box;

[0269] 602. Perform normalization processing on the extracted contour line features;

[0270] For each candidate graph in the graph database, steps 603 and 604 are performed respectively:

[0271] 603. Extracting contour features of the candidate graphic;

[0272] 604. Calculate similarity of the contour features.

[0273] Contour features can be extracted and stored for each candidate shape in the graph database in advance, i.e., step 603 can be completed in advance. This allows step 604 to proceed directly after step 602 during matching, saving time. Candidate shapes in the graph database are generally standard shapes, so normalization is not required. However, for non-standard shapes, normalization is still required after contour feature extraction.

[0274] In this example, in step 604, the similarity between the contour lines can be calculated as follows:

[0275] The contour features of the candidate graph and the graph formed by the trajectory are densely sampled at equal intervals. The sampling points in the two sampling point sets obtained by sampling are paired with each other according to the Hungarian matching algorithm. The distance of each sampling point pair is calculated, and the similarity of the contour lines is determined based on the sum of the distances of all sampling point pairs.

[0276] The distance calculated may be, but is not limited to, the Euclidean distance. During pairing, if the number of sampling points in the two sampling point sets is different, for example, one is M and the other is N, and N is greater than M, then the number of sampling point pairs obtained is M.

[0277] (3) Includes graphic recommendations

[0278] In this example, the meaning of containing a graphic is that the original line features of the graphic formed by the trajectory are similar to the "parts" in the candidate graphic, that is, the shape of the graphic formed by the trajectory is similar to a sub-graph in the candidate graphic; Figure 7 As shown, the shape formed by the trajectory is similar to the shape of "wings" in the sub-graph 71 of the candidate graph "bird", and the candidate graph "bird" is obtained as the third candidate object.

[0279] When calculating similarity, the original line features of the trajectory graph and the line features of the candidate graph (or a subgraph of the candidate graph) can be densely sampled at equal intervals. The sampling points in the two resulting sampling point sets are paired using the Hungarian matching algorithm. The distance between each pair of sampling points is calculated, and the similarity between the two graphs is determined based on the sum of the distances of all pairs of sampling points. The calculated distance can be, but is not limited to, the Euclidean distance.

[0280] In this example, a candidate graph in the graph database can have one or more subgraphs, such as Figure 8 In the example shown, the candidate graphic "animated figure" can have four sub-graphs: head 81, torso 82, arms 83 (the left and right arms use the same sub-graph), and legs 84 (the left and right legs use the same sub-graph). If the trajectory forms a similar graphic to head 81, the candidate graphic "animated figure" can be successfully matched.

[0281] For a candidate graph with a subgraph, the similarities between the original line features of the graph formed by the trajectory and the original line features of the candidate graph and the subgraph can be calculated respectively. The maximum similarity among the calculated similarities is taken as the final similarity between the candidate graph and the graph formed by the trajectory.

[0282] Example 2

[0283] In this example, the first and second candidate objects are defaulted to the first and second candidate instructions. The displayed first and second selection icons are descriptions of the instructions. In this example, the trajectory input by the user is identified as one or more of the preset standard trajectories. Based on the identified standard trajectories, the corresponding candidate instructions are obtained as the first and second candidate objects, the corresponding selection icons are displayed, and the corresponding operation is performed in response to the user clicking the selection icons.

[0284] In this example, the selection icons displayed to the user are as follows Figure 9As shown, the trajectory formed by the trajectory point set input by the user is shown in area 91; two standard trajectories can be identified for this trajectory point set: "Z" and "N". The candidate instruction corresponding to standard trajectory "N" is obtained as the first candidate object, and the description information corresponding to "N" is obtained as the first selection icon 921, which is displayed in selection icon area 92. The candidate instruction corresponding to standard trajectory "Z" is obtained as the second candidate object, and the description information corresponding to "Z" is obtained as the second selection icon 922, which is displayed in selection icon area 92. When the user clicks the first selection icon 921, a new document is created in the current application. When the user clicks the second selection icon 922, the trajectory in area 91 is deleted.

[0285] The description information of the candidate instructions used by the selection icon in this example can be replaced by an icon of the candidate instruction or the content of the candidate instruction.

[0286] In this example, a standard track may correspond to multiple candidate instructions. For example, if the standard track "N" corresponds to multiple candidate instructions, these multiple candidate instructions may be obtained as different first candidate objects, and corresponding first selection icons may be displayed for each.

[0287] In a variation of this example, one of the first and second candidate objects can be changed to a candidate graph; all candidate instructions corresponding to the identified standard trajectories can be used as another candidate object. The candidate graph can be selected based on one or more types of feature information.

[0288] In this example, if no standard trajectory can be identified for the user-input trajectory, the candidate object can be adaptively changed to a candidate graphic. For example, if only one standard trajectory can be identified for the user-input trajectory, the candidate instruction corresponding to the standard trajectory is obtained as the first candidate object. In addition, preset feature information of the trajectory is extracted to select a candidate graphic and obtain it as the second candidate object. If no standard trajectory can be identified for the user-input trajectory, candidate graphics can be selected based on different types of feature information as the first and second candidate objects, respectively.

[0289] The drawings in this article only relate to the implementation methods involved in this article, and other implementation methods can refer to the design. In the absence of conflict, the embodiments of this application and the features in the embodiments can be combined with each other to obtain new embodiments.

[0290] It should be understood by those skilled in the art that the technical solutions of this document may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of this document, and all should be included in the scope of the claims of this document.

Claims

1. An interactive method, comprising: Acquire a trajectory point set input by a user, wherein the trajectory point set includes at least one trajectory point; Determining candidate objects based on the trajectory point set, the candidate objects including: a first candidate object and a second candidate object; at least one of the first candidate object and the second candidate object is a character graphic or instruction recommended based on the trajectory point set, the character graphic or instruction being generated by recognizing the character corresponding to the trajectory point set; the other of the first candidate object and the second candidate object is a similar graphic or a contained graphic recommended based on the trajectory point set, the similar graphic being a graphic similar to the contour line of the graphic formed by the trajectory point set, the contained graphic including at least one sub-graphic, and the sub-graphic being similar to the graphic of a portion of the original lines included in the trajectory point set; displaying a first selection icon for representing the first candidate object and a second selection icon for representing the second candidate object; In response to a user's selection operation on the first selection icon, the first candidate object is displayed or an operation corresponding to the first candidate object is executed; in response to a user's selection operation on the second selection icon, the second candidate object is displayed or an operation corresponding to the second candidate object is executed; The step of determining a candidate object based on a set of trajectory points includes: obtaining first feature information according to the trajectory formed by the set of trajectory points, obtaining a first candidate graphic that successfully matches the first feature information, obtaining second feature information according to the trajectory formed by the set of trajectory points, and obtaining a second candidate graphic that successfully matches the second feature information; the first feature information and the second feature information are different; Acquire third feature information based on the trajectory formed by the set of trajectory points, and obtain a third candidate graphic that successfully matches the third feature information; identify the trajectory formed by the set of trajectory points as a standard trajectory, and obtain a third candidate instruction based on the standard trajectory; in response to a failure to identify the trajectory formed by the set of trajectory points as a standard trajectory, acquire fourth feature information based on the trajectory formed by the set of trajectory points, and obtain a fourth candidate graphic that successfully matches the fourth feature information as the second candidate object; the fourth feature information is different from the third feature information.

2. The interactive method according to claim 1, wherein: The first candidate object includes a first candidate graphic, and the second candidate object includes a second candidate graphic; The displaying of the first candidate object in response to a user's selection operation on the first selection icon, and the displaying of the second candidate object in response to a user's selection operation on the second selection icon, comprises: In response to a user's selection operation on the first selection icon, the first candidate graphic is displayed, and in response to a user's selection operation on the second selection icon, the second candidate graphic is displayed.

3. The interactive method according to claim 2, wherein: The first characteristic information and the second characteristic information respectively include one or more of the following: One or more character features corresponding to the trajectory; Original line features of the figure formed by the trajectory; The contour line features of the figure formed by the trajectory.

4. The interactive method according to claim 3, wherein: The first feature information or the second feature information includes one or more character features corresponding to the trajectory; The extracting the first feature information or the second feature information according to the trajectory formed by the trajectory point set includes: Extracting one or more character features corresponding to the trajectory formed by the trajectory point set; The acquiring of the first / second candidate graphics that successfully matches the first / second feature information includes: Selecting a candidate graphic whose carried label has at least partially the same character features as those in the first / second feature information as the first / second candidate graphic; The label carried by the candidate graphic is used to represent the content of the candidate graphic.

5. The interactive method according to claim 3, wherein: The first feature information or the second feature information includes original line features of the figure formed by the trajectory; The acquiring of the first / second candidate graphics that successfully matches the first / second feature information includes: respectively calculating a first similarity between each candidate graphic and the original line feature in the first / second feature information; The candidate graphics whose first similarity satisfies a first preset condition are selected as the first / second candidate graphics.

6. The interactive method according to claim 3, wherein: The first feature information or the second feature information includes original line features of the figure formed by the trajectory; The acquiring of the first / second candidate graphics that successfully matches the first / second feature information includes: For each candidate graph, perform the following operations: Calculating the similarity between the original line features in the first / second candidate graphics and the candidate graphics, as well as the similarity between the original line features in the first / second candidate graphics and the sub-graphics contained in the candidate graphics; to determine a second similarity with the candidate graphics; selecting a candidate graphic whose second similarity satisfies a second preset condition as the first / second candidate graphic; The subgraphs contained in the candidate graph are saved corresponding to the candidate graph.

7. The interactive method according to claim 3, wherein: The first feature information or the second feature information includes a contour feature of a figure formed by the trajectory; The acquiring of the first / second candidate graphics that successfully matches the first / second feature information includes: respectively calculating a third similarity between the contour line of each candidate graphic and the contour line features in the first / second feature information; The candidate graph whose third similarity satisfies a third preset condition is selected as the first / second candidate graph.

8. The interactive method according to claim 1, wherein: The first candidate object includes a first candidate instruction, and the second candidate object includes a second candidate instruction; The step of executing the operation corresponding to the first candidate object in response to the user's selection operation on the first selection icon, and executing the operation corresponding to the second candidate object in response to the user's selection operation on the second selection icon, includes: In response to a user's selection operation on the first selection icon, executing an operation corresponding to the first candidate instruction; and in response to a user's selection operation on the second selection icon, executing an operation corresponding to the second candidate instruction; The determining of candidate objects according to the trajectory point set includes: The trajectory formed by the trajectory point set is identified as a standard trajectory, and different candidate instructions are obtained according to the standard trajectory as the first candidate instruction and the second candidate instruction, respectively.

9. The interactive method according to claim 1, wherein: The first candidate object is a third candidate graphic, and the second candidate object is a third candidate instruction; The displaying of the first candidate object in response to the user's selection operation on the first selection icon, and the performing of the operation corresponding to the second candidate object in response to the user's selection operation on the second selection icon, include: In response to a user's selection operation on the first selection icon, the third candidate graphic is displayed; in response to a user's selection operation on the second selection icon, an operation corresponding to the third candidate instruction is executed.

10. The interactive method according to claim 1, wherein: The interaction method is applied to writing software; The track point set obtained from the user input includes: In response to an input operation by a user in an input interface of the writing software, obtaining a set of track points input by the user in the input interface; The displaying of a first selection icon for representing the first candidate object and a second selection icon for representing the second candidate object comprises: Displaying a first selection icon for representing the first candidate object and a second selection icon for representing the second candidate object in the input interface; The displaying of the first candidate object or executing the operation corresponding to the first candidate object in response to the user's selection operation of the first selection icon, and the displaying of the second candidate object or executing the operation corresponding to the second candidate object in response to the user's selection operation of the second selection icon include: In response to a user selecting the first selection icon in the input interface, displaying the first candidate object in the input interface, or executing an operation corresponding to the first candidate object in the writing software; In response to the user's selection operation on the second selection icon in the input interface, the second candidate object is displayed in the input interface, or an operation corresponding to the second candidate object is executed in the writing software.

11. The interactive method according to any one of claims 1 to 10, wherein: The displaying of the first selection icon for representing the first candidate object and the second selection icon for representing the second candidate object includes: if the first candidate object is a candidate graphic, a thumbnail of the first candidate object is displayed as the first selection icon; or if the first candidate object is a candidate instruction, description information of the first candidate object is displayed as the first selection icon; The second candidate object is a candidate graphic, and a thumbnail of the second candidate object is displayed as the first selection icon; or the second candidate object is a candidate instruction, and description information of the second candidate object is displayed as the second selection icon.

12. The interactive method according to any one of claims 1 to 10, wherein: After displaying the first selection icon for representing the first candidate object and the second selection icon for representing the second candidate object, the method further includes: In response to a predetermined operation of the user, updating the determined first candidate object and / or second candidate object; In response to an update of the first candidate object, the displayed first selection icon is updated accordingly; in response to an update of the second candidate object, the displayed second selection icon is updated accordingly.

13. The interactive method according to any one of claims 1 to 10, wherein: The track point set obtained from the user input includes: In response to a pause in the user's input operation, a preset control is displayed; wherein the detection of a pause in the user's input operation means that the length of time since the last sampling result of the input operation was received reaches or exceeds a preset time threshold; In response to the user's operation on the preset control, track points corresponding to the user's input operation between the current detection of a pause in the user's input operation and the previous detection of a pause in the user's input operation are obtained and saved as the track point set.

14. An electronic device comprising: memory and processor; The memory is configured to store a program for performing the interaction; The processor is configured to read and execute the program for performing the interaction, and perform the interaction method according to any one of claims 1 to 13.

15. A computer storage medium configured to store an interactive program; when the interactive program is executed, the interactive method according to any one of claims 1 to 13 is performed.

Images