Operation identification method and related device

Abstract

The embodiment of the invention discloses an operation identification method and a related device, and the method comprises the steps: firstly determining starting point position information used for identifying sliding starting points corresponding to a plurality of contact media, building a plurality of position information sets based on the starting point position information, and then carrying out the arrangement of the remaining position information in operation data according to the arrangement order in the operation data, and analyzing the closeness degree of the position information included in each position information set one by one, and summarizing the closeness degree into the closest position information set. As the multiple pieces of position information in the same position information set are relatively close, the probability of identifying the position on the sliding track of the same contact medium is relatively high. Meanwhile, the arrangement sequence of the position information in the operation data is referred to during induction, and the arrangement sequence corresponds to the position sequence of the contact medium during sliding, so that the position information sequence in the same position information set can represent the position sequence of the single contact medium during sliding; therefore, accurate identification of the sliding track of each contact medium is completed.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to an operation recognition method and related devices. Background Art

[0002] During the human-computer interaction process, various operation data will be generated based on the operations of the terminal device by the operation direction. These operation data can usually represent the corresponding operations, so that subsequent processing such as analysis and reproduction of the operations can be performed based on the operation data.

[0003] Among various operation behaviors, there is a multi-finger sliding operation. A multi-finger operation refers to an operation in which the operator simultaneously uses multiple fingers or other contact media to slide on the terminal device. For example, the image can be enlarged by double-finger sliding. In order to perform subsequent processing on the multi-finger sliding operation based on the operation data, it is necessary to accurately identify the sliding trajectory of each finger on the terminal device from the operation data.

[0004] However, since the operation data is obtained in real time based on the contact positions of the operator's fingers on the terminal device and does not distinguish different contact media, the sliding trajectories of different contact media in the operation data will be mixed together. In the related art, there is a lack of a method that can accurately identify the sliding trajectories corresponding to different contact media from the operation data, resulting in the inability to perform subsequent processing on the multi-finger sliding operation based on the operation data, and the utilization degree of the operation data is relatively low. Summary of the Invention

[0005] In order to solve the above technical problems, this application provides an operation recognition method, which can accurately analyze the sliding trajectories of various contact media during the target operation based on the operation data corresponding to the target operation, so that a series of subsequent processing such as analysis and reproduction of the target operation can be performed based on the sliding trajectories.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] In a first aspect, the embodiments of this application disclose an operation recognition method, and the method includes:

[0008] Obtain the operation data to be recognized, where the operation data to be recognized is the operation data generated based on the target operation, the target operation is an operation in which M contact media slide simultaneously on the target interface, the operation data to be recognized includes multiple position information, the position information is used to identify the positions passed by the M contact media on the target interface, and the arrangement order of the multiple position information in the operation data to be recognized corresponds to the position order passed by the M contact media during sliding, and M is an integer greater than 1;

[0009] Determine the starting position information among the multiple pieces of position information, where the starting position information is used to identify the M sliding starting points corresponding to the M contact media, and the sliding starting point is the starting point of the contact media for sliding on the target interface;

[0010] Based on the arrangement order of the N pieces of position information, respectively summarize the N pieces of position information. For the i-th piece of position information among the N pieces of position information, summarize the i-th piece of position information into the target position information set, where the target position information set is the position information set that includes the position information closest to the i-th piece of position information among the position information included in the M position information sets. The N pieces of position information are the position information among the multiple pieces of position information except the starting position information, and the M position information sets are constructed based on the starting position information. Each position information set includes the starting position information corresponding to any one of the sliding starting points, and the sliding starting points identified by the starting position information included in different position information sets are different;

[0011] According to the positions respectively identified by the M position information sets after summarization, determine the positions passed by the M contact media respectively when performing the target operation.

[0012] In a possible implementation manner, the target interface is a game interface, the target operation is used to trigger a target game function in the game interface, and the operation instruction is used to trigger the target game function in the game interface.

[0013] In a second aspect, an operation recognition device is disclosed in an embodiment of the present application. The device includes a first acquisition unit, a first determination unit, a summarization unit, and a second determination unit:

[0014] The first acquisition unit is configured to acquire operation data to be recognized, where the operation data to be recognized is operation data generated based on a target operation. The target operation is an operation in which M contact media slide simultaneously on a target interface. The operation data to be recognized includes multiple pieces of position information, and the position information is used to identify the positions passed by the M contact media on the target interface. The arrangement order of the multiple pieces of position information in the operation data to be recognized corresponds to the position order passed by the M contact media during sliding, and M is an integer greater than 1;

[0015] The first determination unit is configured to determine the starting position information among the multiple pieces of position information, where the starting position information is used to identify the M sliding starting points corresponding to the M contact media, and the sliding starting point is the starting point of the contact media for sliding on the target interface;

[0016] The induction unit is configured to respectively induct the N location information based on the arrangement order of the N location information. For the i-th location information among the N location information, the i-th location information is inducted into the target location information set, where the target location information set is the location information set including the location information closest to the i-th location information among the location information included in the M location information sets. The N location information is the location information among the multiple location information except the starting point location information. The M location information sets are constructed based on the starting point location information. Each location information set includes the starting point location information corresponding to any sliding starting point, and the sliding starting points identified by the starting point location information included in different location information sets are different;

[0017] The second determination unit is configured to determine the positions respectively passed by the M contact media when performing the target operation according to the positions respectively identified by the M inducted location information sets.

[0018] In a possible implementation manner, the device further includes a second acquisition unit:

[0019] The second acquisition unit is configured to acquire multiple operation data, where the multiple operation data correspond one-to-one to the operations on the target interface, and each operation data includes multiple location information;

[0020] The first acquisition unit is specifically configured to:

[0021] Determine the operation data including multiple sets of contact location information and having the number of included location information greater than the first preset threshold among the multiple operation data as the to-be-identified operation data, where the contact location information is the location information generated by the first contact of the contact media with the target interface, and each set of contact location information is used to identify the contact location of any contact media with the target interface for the first contact.

[0022] In a possible implementation manner, the first determination unit is specifically configured to:

[0023] Determine the multiple sets of contact location information as the starting point location information, and the contact location identified by each set of contact location information is the sliding starting point.

[0024] In a possible implementation manner, the contact location information is the location information with a contact identifier, and the contact identifier is used to identify that the contact location information is the location information generated by the first contact of the contact media with the target interface.

[0025] In a possible implementation, the target interface has a corresponding interface coordinate system for positioning the contact position of the contact medium with the target interface. Each set of contact position information includes coordinate information corresponding to multiple coordinate axes of the interface coordinate system, and the coordinate information included in each set of contact position information is adjacent among the multiple position information included in the to-be-recognized operation data. The coordinate information corresponding to the multiple coordinate axes is used to identify the contact position in the interface coordinate system.

[0026] In a possible implementation, the device further includes a third acquisition unit:

[0027] The third acquisition unit is configured to acquire initial operation data, which includes position information, an operation start identifier, and an operation end identifier. The operation start identifier is used to identify the start of any operation on the target interface, and the operation end identifier is used to identify the end of any operation on the target interface;

[0028] The second acquisition unit is specifically configured to:

[0029] Determine the position information between any adjacent operation start identifier and operation end identifier in the initial operation data as an operation data, and obtain the multiple operation data.

[0030] In a possible implementation, the third acquisition unit is specifically configured to:

[0031] Acquire an operation recording start request and an operation recording end request. The operation recording start request corresponds to a first moment, and the operation recording end request corresponds to a second moment;

[0032] Acquire the operation data generated from the first moment to the second moment as the initial operation data.

[0033] In a possible implementation, the induction unit is specifically configured to:

[0034] Determine the positions respectively identified by the M position information sets according to the position information included in the M position information sets. The position order in the positions identified by the first position information set corresponds to the arrangement order of the position information included in the first position information set in the to-be-recognized operation data. The first position information set is any one of the M position information sets;

[0035] Determine the positions respectively identified by the M position information sets as the positions respectively passed by the M contact media when performing the target operation. The position order corresponding to the first position information set is the position order of the positions passed by any contact medium when performing the target operation.

[0036] In a possible implementation, the target interface has a corresponding interface coordinate system for positioning the contact position of the contact medium with the target interface. The position information is the coordinate information of any coordinate axis in the interface coordinate system. Specifically, the induction unit is configured to:

[0037] Divide the coordinate information included in the first position information set into multiple groups of coordinate information. Each group of coordinate information includes a target number of adjacent coordinate information in the first position information set. The target number of coordinate information corresponds one-to-one with the coordinate axes of the interface coordinate system, and the target number is the dimension of the interface coordinate system;

[0038] Determine the positions identified by the multiple groups of coordinate information in the interface coordinate system as the positions identified by the first position information set. The arrangement order of the multiple groups of coordinate information in the first position information set is the position order corresponding to the first position information set.

[0039] In a possible implementation, the device further includes a clustering unit:

[0040] The clustering unit is configured to cluster the positions identified by the M position information sets according to the position information respectively corresponding to the positions identified by the M position information sets, to obtain multiple position clusters. The clustering is used to make the proximity degree of the position information between the positions in the same position cluster greater than that between the positions in different position clusters;

[0041] Specifically, the second determination unit is configured to:

[0042] Based on the distribution of the positions identified by the M position information sets in the multiple position clusters being consistent with the distribution in the M position information sets, determine the positions respectively identified by the M position information sets as the positions respectively passed by the M contact media when performing the target operation.

[0043] In a possible implementation, the device further includes a fitting unit:

[0044] The fitting unit is configured to use the M position information sets as the second position information sets respectively, and fit a fitting curve corresponding to the second position information set according to the positions identified by the second position information set;

[0045] Specifically, the second determination unit is configured to:

[0046] Based on the curvature of the fitting curves corresponding to the M position information sets being less than a second preset threshold, determine the positions respectively identified by the M position information sets as the positions respectively passed by the M contact media when performing the target operation.

[0047] In a possible implementation, the target interface has a corresponding interface coordinate system for positioning the contact position of the contact medium with the target interface. The multiple position information includes coordinate information corresponding to each coordinate axis of the interface coordinate system. The i-th position information is the coordinate information corresponding to the target coordinate axis in the interface coordinate system, and the target coordinate axis is any coordinate axis in the interface coordinate system. Specifically, the induction unit is configured to:

[0048] Determine the degree of proximity between the coordinate information corresponding to the target coordinate axis included in each of the M position information sets and the coordinate information corresponding to the i-th position information;

[0049] Induce the i-th position information into the target position information set with the greatest degree of proximity.

[0050] In a possible implementation, the induction unit is specifically configured to:

[0051] Respectively use the M position information sets as the third position information sets, and determine the target position information that was most recently induced into the third position information set. The target position information is the coordinate information corresponding to the target coordinate axis;

[0052] Determine the target degree of proximity between the coordinate information corresponding to the target position information and the coordinate information corresponding to the i-th position information;

[0053] Determine the target degree of proximity as the degree of proximity between the coordinate information corresponding to the target coordinate axis included in the third position information set and the coordinate information corresponding to the i-th position information.

[0054] In a possible implementation, the device further includes a generation unit and an execution unit:

[0055] The generation unit is configured to generate an operation instruction corresponding to the target operation according to the positions respectively passed by the M contact media when performing the target operation;

[0056] The execution unit is configured to execute the operation instruction, and the operation instruction is used to make the target interface make an interface response when receiving the target operation.

[0057] In a possible implementation, the generation unit is specifically configured to:

[0058] Based on obtaining an operation reproduction request, generate an operation instruction corresponding to the target operation according to the positions respectively passed by the M contact media when performing the target operation. The operation reproduction request is used to request to reproduce the target operation for the target interface.

[0059] In a possible implementation, the generating unit is specifically configured to:

[0060] Respectively use the M position information sets as the fourth position information set, and determine the endpoint positions and non-endpoint positions identified by the fourth position information set. The endpoint positions are the starting position and the ending position among the positions passed by the contact medium corresponding to the fourth position information set when performing the target operation, and the non-endpoint positions are the other positions in the positions identified by the fourth position information set except the endpoint positions;

[0061] Determine a target position from the non-endpoint positions. The target position is a part of the non-endpoint positions, and the target position satisfies that the similarity between the first curve and the second curve is greater than a third preset threshold. The first curve is a curve fitted based on the positions identified by the fourth position information set, and the second curve is a curve fitted based on the endpoint positions and the target position;

[0062] Generate an operation instruction corresponding to the target operation according to the endpoint positions and the target position.

[0063] In a possible implementation, the target interface is a game interface, the target operation is used to trigger a target game function in the game interface, and the operation instruction is used to trigger the target game function in the game interface.

[0064] In a third aspect, an embodiment of the present application discloses a computer device, which includes a processor and a memory:

[0065] The memory is used to store a computer program and transmit the computer program to the processor;

[0066] The processor is used to execute the operation recognition method according to any one of the items in the first aspect according to the instructions in the computer program;

[0067] In a fourth aspect, an embodiment of the present application discloses a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the operation recognition method according to any one of the items in the first aspect;

[0068] In a fifth aspect, an embodiment of the present application discloses a computer program product including a computer program, which, when running on a computer device, causes the computer device to execute the operation recognition method according to any one of the items in the first aspect.

[0069] As can be seen from the above technical solution, after obtaining the operation data to be recognized generated based on the target operation, the starting position information used to identify the starting point of the swipe in the operation to be recognized can be determined first. M position information sets can be constructed according to the starting position information of the starting points corresponding to the M contact media respectively. Each position information set includes the starting position information of any starting point of the swipe. When successively generalizing the other N position information included in the operation data to be recognized except the starting position information into the M position information sets, for the i-th position information, the degree of proximity between this position information and the position information included in the M position information sets respectively can be judged. The higher the degree of proximity, the closer the i-th position information is to the position respectively identified by this position information set in the target interface, that is, the higher the probability that the i-th position information and the position information included in this position information set are on the same swipe trajectory. Therefore, the i-th position information can be generalized into the position information set whose included position information is the closest to it. The position information included in this position information set is very likely to be the position information that identifies the same swipe trajectory as the i-th position information. Since the position information in the operation data to be recognized is arranged in the order of the positions passed by the contact media during the swipe, through this method, the order of the position information in each position information set can still correspond to the order of the positions passed by the contact media corresponding to this position information set during the swipe. Thus, it can be seen that each position information set obtained by this method can include the position information corresponding to a single contact media. These position information can not only represent the positions passed by this single contact media during the target operation, but also represent the order of the positions passed, so as to accurately represent the swipe trajectory of the single contact media during the target operation. Furthermore, through this method, the swipe trajectories corresponding to multiple contact media can be accurately recognized, and the analysis method during the recognition process is relatively simple and efficient. While ensuring the operation recognition efficiency, it provides accurate data that can be used for subsequent operation processing, which helps various subsequent processing such as analyzing and reproducing the target operation. Description of the Drawings

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0071] Figure 1 Schematic diagram of an operation recognition method in an actual application scenario provided by an embodiment of the present application;

[0072] Figure 2Flowchart of an operation recognition method provided by an embodiment of the present application;

[0073] Figure 3 Schematic diagram of an operation recognition method provided by an embodiment of the present application;

[0074] Figure 4 Flowchart of an operation recognition method in an actual application scenario provided by an embodiment of the present application;

[0075] Figure 5 Schematic diagram of an operation recognition method in an actual application scenario provided by an embodiment of the present application;

[0076] Figure 6 Schematic diagram of an operation recognition method in an actual application scenario provided by an embodiment of the present application;

[0077] Figure 7 Schematic diagram of an operation recognition method in an actual application scenario provided by an embodiment of the present application;

[0078] Figure 8 Structural block diagram of an operation recognition device provided by an embodiment of the present application;

[0079] Figure 9 Structural diagram of a terminal provided by an embodiment of the present application;

[0080] Figure 10 Structural diagram of a server provided by an embodiment of the present application. Detailed implementation manners

[0081] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0082] A multi-finger sliding operation refers to an operation of sliding on an interface using multiple contact media simultaneously, where the contact media can be any medium capable of performing a sliding operation on the interface, such as a user's finger, a stylus, etc. Operation data is data generated based on the operation of the contact media on the interface, and can record the position on the interface where the contact media touches during the operation through the included position information.

[0083] In the related art, some operations on the interface can be recognized relatively accurately through operation data. For example, when the position information included in the operation data only identifies one position on the interface, the operation data most likely corresponds to a click operation on the interface; when the multiple position information included in the operation data can accurately identify multiple positions on the interface, the operation data most likely corresponds to a sliding operation of a single contact media on the interface.

[0084] However, when performing a multi-finger sliding operation, since the position information corresponding to the positions where multiple contact media contact the interface is collected simultaneously, and the ability to distinguish different contact media is not available during the collection process, the operation data corresponding to the multi-finger sliding operation usually presents intricate position information and cannot accurately represent the state of each position. For example, if the position information is the coordinate information on the plane coordinate system corresponding to the interface, each position where the contact media contacts the interface corresponds to an X-axis coordinate and a Y-axis coordinate. If there is only one contact media, according to the order of the positions passed by the contact media in the interface, operation data with alternating X-axis coordinates and Y-axis coordinates will be obtained. Each set of X-axis and Y-axis coordinates can identify a position, so that the operation data can clearly identify the sliding trajectory of the operation media during sliding; when there are multiple contact media, since multiple sets of X-axis and Y-axis coordinates are received simultaneously, a situation where the coordinate information is interspersed, such as "X-axis coordinate X-axis coordinate Y-axis coordinate Y-axis coordinate", may occur. At the same time, it is also impossible to directly know which contact media each set of coordinate information corresponds to only through the operation data. Thus, in the related art, it is impossible to accurately identify the sliding trajectories of each contact media during the multi-finger sliding operation through the operation data. Therefore, it is impossible to accurately analyze the operation process of the contact media and impossible to accurately reproduce the multi-finger sliding operation (that is, it is impossible to re-perform a consistent multi-finger sliding operation).

[0085] To solve the above technical problems, the present application provides an operation recognition method. First, the starting position information used to identify the sliding starting points corresponding to multiple contact media in the operation data can be determined, and multiple position information sets are constructed based on the starting position information. Then, the remaining position information in the operation data is analyzed one by one according to the arrangement order in the operation data to determine the degree of proximity to the position information included in each position information set, and it is classified into the position information set that is closest in terms of position information, resulting in multiple inducted position information sets. Since the multiple position information in the same position information set is relatively close, and the degree of proximity of the multiple position information between different position information sets is relatively low, the probability that the multiple position information in the same position information set identifies the sliding trajectory of the same contact media is relatively high. At the same time, since the arrangement order of the position information in the operation data is referred to during induction, and this arrangement order corresponds to the position order of the contact media during sliding, the position information order in the same position information set can represent the position order of a single contact media during sliding. Thus, the sliding trajectory of a contact media during the target operation can be determined through each position information set, and the accurate identification of the sliding trajectories of each contact media is thus completed.

[0086] It can be understood that this method can be applied to a computer device, which is a computer device capable of performing operation recognition, such as a terminal device or a server. This method can be independently executed by the terminal device or the server, or can be applied to a network scenario where the terminal device and the server communicate, and is executed in cooperation with the terminal device and the server. Among them, the terminal device can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc. The server can be understood as an application server or a Web server. In actual deployment, the server can be an independent server, a cluster server, or a cloud server, etc.

[0087] To facilitate the understanding of the technical solution provided by this application, next, an operation recognition method provided by this application will be introduced in combination with an actual application scenario.

[0088] See Figure 1 , Figure 1 FIG. is a schematic diagram of an operation recognition method in an actual application scenario provided by an embodiment of this application. In this actual application scenario, the computer device can be a server 101 with operation recognition function, and the position information is the coordinate information on the plane coordinate system corresponding to the target interface.

[0089] In this actual application scenario, a total of two contact media perform a multi-finger sliding operation on the target interface. The sliding trajectory of contact media 1 passes through points 1, 3, and 5, and the sliding trajectory of contact media 2 passes through points 2, 4, and 6. Since the two contact media slide simultaneously, the position information generated during the sliding process will be interleaved. For example, in the operation data to be recognized obtained by the server 101 based on the multi-finger sliding operation, it will include multiple coordinate information arranged in the order of X1, Y1, X2, Y2, X3, X4, Y3, X5, Y4, Y5, X6, Y6. These coordinate information are not all arranged in the form of a set of X-axis coordinates and Y-axis coordinates. Therefore, in the related art, it is impossible to directly determine which X-axis coordinates and Y-axis coordinates are in a group, and it is impossible to determine which contact media each set of X-axis coordinates and Y-axis coordinates corresponds to, so it is impossible to recognize the sliding trajectory of each contact media.

[0090] In this application, first, based on the property that there is usually a difference in sequence when multiple contact media touch the contact interface, the starting position in the operation data to be recognized usually includes multiple sets of independent X-axis coordinates and Y-axis coordinates. These coordinate information sets are the starting position information indicating the sliding starting point. For example, in Figure 1Among them, at the starting position of the operation data to be recognized, there are two sets of coordinate information, (X1, Y1) and (X2, Y2), which can identify two sliding starting points, point 1 and point 2. Based on the starting position information, two position information sets can be constructed. The coordinate information of point 1 is included in set 1, and the coordinate information of point 2 is included in set 2.

[0091] Then, the server 101 will, according to the arrangement order of the position information in the operation data to be recognized, classify the position information other than the starting position information into set 1 or set 2 based on the proximity between the position information. Taking the coordinate information X3 as an example, for set 1, the server 101 will compare the proximity between X3 and X1, and for set 2, the server 101 will compare the proximity between X3 and X2. Figure 1 It can be seen that since the proximity of set 1 is greater than that of set 2, it indicates that the probability that the coordinate information X3 and the coordinate information X1 and Y1 are the coordinate information identifying the same sliding trajectory is greater. Therefore, the server 101 can classify X3 into set 1. By analogy, finally, multiple classified position information sets are obtained. Among them, the coordinate information corresponding to point 1, point 3, and point 5 is included in set 1, and the coordinate information corresponding to point 2, point 4, and point 6 is included in set 2. Since this application classifies in sequence according to the arrangement order of the position information in the operation data to be recognized, and this arrangement order corresponds to the position order when multiple contact media slide on the target interface, the classification order of the position information in each position information set can also represent the position order when the corresponding contact medium slides. Thus, through the multiple classified position information sets, the sliding trajectories of each contact medium can be respectively identified. For example, in Figure 1 it is possible to know that the contact medium 1 slides on the target interface in the order of point 1, point 3, and point 5 through the coordinate information corresponding to the three points included in set 1 and the classification order among the three-point coordinate information.

[0092] It can be seen from this that based on the characteristic that the position information of the positions on the same trajectory is relatively close, through relatively simple position information analysis operations, this application can divide the intricate multiple position information into multiple groups of position information, enabling each group of position information to accurately represent the sliding trajectory of a single contact medium, achieving efficient recognition of multi-finger sliding operations and facilitating subsequent processing of multi-finger sliding operations.

[0093] Next, in combination with the accompanying drawings, the operation recognition method provided by the embodiments of this application will be introduced.

[0094] See Figure 2 , Figure 2 which is a flowchart of an operation recognition method provided by an embodiment of this application. In this embodiment, the computer device can be any of the above computer devices with operation recognition functions. The method includes:

[0095] S201: Obtain the operation data to be recognized.

[0096] The operation data to be recognized is operation data generated based on a target operation. The target operation is an operation in which M contact media slide simultaneously on a target interface. The contact media can be any contact media, and the target interface can also be any interface that supports the target operation. For example, it can be a hardware interface, such as a mobile phone screen interface, a tablet computer screen interface, etc.; it can also be a software interface displayed by a hardware device, such as a game interface displayed by a mobile phone, etc., which is not limited here. The target operation can be any operation executed by simultaneously sliding multiple contact media. For example, it can be a multi-finger sliding operation, etc.

[0097] The operation data to be recognized includes multiple position information, which is used to identify the positions passed by M contact media on the target interface. Among them, the arrangement order of the multiple position information in the operation data to be recognized corresponds to the position order passed by the M contact media during sliding. M is an integer greater than 1. Since the position information corresponding to the M contact media is obtained simultaneously during the sliding process, the position information corresponding to each of the M contact media will be interspersed with each other in the operation data to be recognized, resulting in the inability to directly read out the position information corresponding to each position from the operation data to be recognized. At the same time, the computer device cannot directly know which contact media the position information corresponds to, which is the pain point in the related art where the target operation cannot be accurately recognized.

[0098] S202: Determine the starting position information among the multiple position information.

[0099] It can be understood that although during the simultaneous sliding of the M contact media, it is easy for the computer device to receive the position information corresponding to the multiple contact media simultaneously, resulting in disordered position information, when the M contact media first contact the target interface, since the contact action usually corresponds to only one position and usually has a certain contact time difference, there is usually a time difference when obtaining the position information corresponding to the position where the target interface is first contacted. Therefore, the position information corresponding to these positions is relatively independent in the operation data to be recognized, that is, it can be accurately known which position information in the operation data to be recognized these positions correspond to. At the same time, the sliding operation is an operation of first contacting the interface - then sliding on the interface - and finally leaving the interface. Therefore, the position where the interface is first contacted can be regarded as the starting point of the sliding operation, and the corresponding position information can be regarded as the starting position information in this application. As mentioned above, the arrangement order of the position information in the operation data to be recognized is the position order passed by the contact media during sliding. Therefore, this starting position information is usually located at the front end of the operation data to be recognized.

[0100] In summary, on the one hand, the starting point position information has a unique data arrangement position in the operation data to be recognized. On the other hand, the starting point position information in the operation data to be recognized can accurately identify the sliding starting point, that is, it has a clear information structure. Therefore, the computer device can accurately determine the starting point position information from the operation data to be recognized. For example, in Figure 1 the starting point position information is the first two groups of coordinate information in the operation data to be recognized, and each group of coordinate information is the adjacent X-axis coordinate information and Y-axis coordinate information in the operation data to be recognized. Therefore, the position on the plane coordinate system can be effectively identified, that is, the position information corresponding to each sliding starting point is determined.

[0101] Based on this, the computer device can first use the starting point position information to identify the M sliding starting points corresponding to the M contact media. The sliding starting point is the starting point of the contact media sliding on the target interface;

[0102] S203: Induce the N position information respectively based on the arrangement order of the N position information.

[0103] Since the sliding starting point of each contact media is usually a single position, and M contact media correspond to M sliding starting points. Therefore, the computer device can construct M position information sets based on the starting point position information. Each position information set is used to record the position information generated by any contact media during the multi-finger sliding process. Each position information set will include the starting point position information corresponding to any sliding starting point. Then, the computer device can analyze which contact media the remaining position information corresponds to, and induct the remaining position information into the corresponding position information set, so that each position information set includes all the position information generated by any contact media during the multi-finger sliding process.

[0104] It can be understood that since the sliding operation is a continuous process, the position relationship of the positions passed by the same contact media during the sliding process is usually relatively close on the target interface, that is, the degree of proximity between the position information corresponding to the same contact media is relatively high. For example, in Figure 1 the position relationship between point 1, point 3 and point 5 is relatively close, and the position relationship between point 1 and point 2, point 4 is relatively far. The computer device can induct the remaining position information other than the starting point position information based on this feature.

[0105] In the actual induction process, for the i-th position information among the N position information, the computer device can determine the degree of proximity between the i-th position information and the position information respectively included in the M position information sets, and the degree of proximity can reflect whether the i-th position information and the position information included in the position information set are position information generated by the sliding of the same contact medium. The higher the degree of proximity, the closer the relationship between the position identified by the i-th position information and the position identified by the position information included in the position information set, and therefore the greater the probability that the position information is generated by the sliding of the same contact medium.

[0106] Based on this, the computer device can summarize the i-th position information into the target position information set, and the target position information set is the position information set whose position information included in the M position information sets is closest to the i-th position information, wherein the N position information is the position information other than the starting position information in the multiple position information, N is a positive integer not less than M, and i is a positive integer not greater than N. The M position information sets are constructed based on the starting position information, and each position information set includes the starting position information corresponding to any sliding starting point, and the sliding starting points identified by the starting position information included in different position information sets are different.

[0107] S204: Determine the positions that the M contact media pass through when performing the target operation according to the positions respectively identified by the summarized M position information sets.

[0108] Through the above method, the computer device can summarize the position information with a high degree of proximity into the same position information set based on the arrangement order of the position information in the sequence of operations to be identified, and the degree of proximity between the position information included in different position information sets is low. Therefore, on the one hand, the position information included in each position information set is likely to correspond to the same contact medium. On the other hand, since the order in which the position information is summarized into the position information set is the position order through which the contact medium slides, the position information used to identify each position can be distinguished first, so that the position identified by each position information set can be accurately determined. For example, when the position information is the coordinate information of the X-axis or Y-axis, after being summarized in the above method, the coordinate information in the same position information set can constitute a group of adjacent X-axis coordinate information and Y-axis coordinate information.

[0109] Secondly, through the summarized order between the position information, the order between the multiple positions identified by the same position information set can be identified, and this order is the position order of the contact medium corresponding to the position information set when sliding. It can be seen that the position information set summarized by this application can accurately reflect the position and position order of each contact medium during the target operation process, and then accurately identify the sliding trajectory of each contact medium, and realize accurate identification of the target operation.

[0110] In summary, through the operation recognition method of the present application, each obtained position information set can include the position information corresponding to a single contact medium. These position information can not only represent the positions passed by the single contact medium during the target operation, but also represent the position sequence of the passed positions, so as to accurately represent the sliding trajectory of the single contact medium during the target operation. Furthermore, through this method, the sliding trajectories corresponding to multiple contact media can be accurately recognized respectively, and the analysis method during the recognition process is relatively simple and efficient. While ensuring the operation recognition efficiency, it provides accurate data that can be used for subsequent operation processing, which is helpful for various subsequent processes such as analyzing and reproducing the target operation.

[0111] As mentioned above, since the target operation is completed by the sliding operations of multiple contact media, and the sliding operation usually starts from the first contact. It can be seen that if the operation data includes the position information corresponding to multiple first contact operations, then this operation data is probably the operation data corresponding to the target operation. At the same time, from the above content, it can be seen that the position information corresponding to the first contact is usually the position information that is relatively easy to identify in the operation data. Therefore, it is less difficult to identify the operation type corresponding to the operation data by identifying the position information corresponding to the first contact, and the recognition efficiency is relatively high.

[0112] Based on this, in a possible implementation manner, the computer device can determine the operation data generated by the target operation through the position information generated by the first contact with the target interface. The computer device can first obtain multiple operation data, where the multiple operation data correspond one-to-one to the operations on the target interface, and each operation data includes multiple position information.

[0113] When executing step S201, the computer device can execute step S2011 (not shown in the figure). Step S2011 is a possible implementation manner of step S201 and includes:

[0114] S2011: Among the multiple operation data, determine the operation data that includes multiple sets of contact position information among the multiple position information and the number of included position information is greater than the first preset threshold as the operation data to be recognized.

[0115] Among them, the contact position information is the position information generated by the first contact through the contact medium with the target interface. Each group of contact position information is used to identify the contact position of any contact medium with the target interface for the first time. How to identify the position information generated by the first contact has been introduced above, that is, these position information are usually located at the front end of the operation data, used to identify the first contact position corresponding to the contact medium, and can accurately identify the data of the position in the target interface. If a certain operation data includes multiple groups of contact position information, it means that the operation involves multiple contact media contacting the target interface, which belongs to a multi-finger operation; if the number of position information included in the operation data is greater than the first preset threshold, it means that the number of position information included in the operation data is relatively large. As mentioned above, the position information is generated by the contact between the contact medium and the target interface. Therefore, the more the position information, the more contact positions between the contact medium and the target interface, and the greater the probability that the operation corresponding to the operation data is a sliding operation. Based on this, if both of these conditions are met at the same time, it can be determined that the operation data is the to-be-identified operation data corresponding to the target operation, realizing the accurate screening of the to-be-identified operation data.

[0116] As can be seen from the above, when identifying the to-be-identified operation data corresponding to the target operation from multiple operation data, the computer device will identify the position information generated by the first contact. At the same time, in the sliding operation of the contact medium, the position where the contact medium first contacts the interface is usually the starting point of the sliding operation. Therefore, in a possible implementation manner, when performing step S202, the computer device may perform step S2021 (not shown in the figure). Step S2021 is a possible implementation manner of step S202, including:

[0117] S2021: Determine multiple groups of contact position information as the starting point position information.

[0118] The computer device can directly determine the contact position information analyzed in the previous stage as the starting point position information without analyzing the to-be-identified operation data again. The contact position identified by each group of contact position information is the sliding starting point. Thus, only one identification of the contact position information is required in the entire operation identification process, improving the identification efficiency of the operation identification.

[0119] Next, a detailed introduction will be given on how to identify contact information. In the technical field involved in this application, the position information corresponding to the first contact interface and the position information corresponding to when the contact medium leaves the interface usually have corresponding identifiers. For example, the position information corresponding to the first contact interface has a corresponding "down" identifier to indicate that the contact medium "lands" on the interface at the position identified by this position information, and the position information corresponding to when the contact medium leaves the interface can have an "up" identifier to indicate that the contact medium leaves the interface at this position information. Based on this, the computer device can identify the contact position information in the operation data through specific identifiers.

[0120] In a possible implementation manner, the contact position information is the position information with a contact identifier, and the contact identifier is used to indicate that the contact position information is the position information generated by the first contact of the contact medium with the target interface. That is, the computer device can directly determine the position information with this contact identifier among the multiple position information included in the operation data as the contact position information. Combining with the fact that the contact position information itself has an accurate data structure and can accurately identify the characteristics of the corresponding position, the computer device can determine the sliding starting points corresponding to multiple contact media based on the contact position information.

[0121] Among them, according to the different forms of the position information, the specific identification method of the contact position information can also be different. For example, in a possible implementation manner, the target interface has a corresponding interface coordinate system, and the interface coordinate system is used to locate the contact position of the contact medium with the target interface. Each group of contact position information includes the coordinate information corresponding to multiple coordinate axes of the interface coordinate system, and the coordinate information included in each group of contact position information is adjacent among the multiple position information included in the operation data to be identified. The coordinate information corresponding to multiple coordinate axes is used to identify the contact position in the interface coordinate system. That is, each group of contact position information includes one coordinate information corresponding to multiple coordinate axes respectively, and one coordinate position can be identified in the interface coordinate system through each group of contact position information, and this position is the contact position on the target interface.

[0122] As mentioned above, since it is difficult for multiple contact media to come into contact with the target interface for the first time simultaneously, there will be a time difference when obtaining the contact position information corresponding to each of the multiple contact media. As a result, the contact position information can have a clear information structure in the operation data. In this actual application scenario, it is manifested as the coordinate information included in each set of contact position information being adjacent among the multiple position information included in the operation data to be recognized, so that the computer device can quickly determine multiple sets of contact position information. The interface coordinate system can include a two-dimensional coordinate system, a three-dimensional coordinate system, etc. according to different interface types, which is not limited here. Taking the two-dimensional coordinate system as an example, the position information corresponding to the front end in the operation data to be recognized can be X1, Y1, X2, Y2, which can form two contact positions (X1, Y1) and (X2, Y2).

[0123] It can be understood that the operation data is a data continuously obtained during the operation process. In the initial data state, the operation data usually includes the operation data corresponding to multiple operations arranged in the operation sequence. The computer device first needs to segment the initial operation data with a single operation as a unit to obtain the operation data corresponding to each operation, and then identify the operation data corresponding to the target operation.

[0124] In a possible implementation manner, the computer device can first obtain the initial operation data, which includes the operation data corresponding to multiple operations on the target interface. Specifically, the initial operation data includes position information, an operation start identifier, and an operation end identifier. The operation start identifier is used to identify the start of any operation on the target interface, and the operation end identifier is used to identify the end of any operation on the target interface. That is, the position information between two adjacent operation start identifiers and operation end identifiers is the position information generated by a single operation.

[0125] Based on this, when obtaining multiple operation data, the computer device can determine the position information between any set of adjacent operation start identifiers and operation end identifiers in the initial operation data as an operation data to obtain multiple operation data, so that each operation data can correspond to a single operation. For example, in the operation data obtained by using the time acquisition tool getevent, the operation start identifier can be "ABS_MT_TRACKING_ID", etc. In other types of operation data, the operation start identifier can also be other identifiers, which is not limited here.

[0126] As mentioned above, some operations may generate a large amount of operation data due to the large number of involved positions. The recording and storage of operation data both consume certain processing resources and storage resources of the computer device. Based on this, in a possible implementation, in order to meet the operation recording requirements while saving resource occupancy, the computer device can provide a function of freely recording operation data.

[0127] In this implementation, the computer device can obtain initial operation data based on an actively initiated request. The computer device can obtain an operation recording start request and an operation recording end request. The operation recording start request is used to make the computer device start recording operation data, and the operation recording end request is used to make the computer device terminate recording operation data. Among them, the operation recording start request corresponds to the first moment, and the operation recording end request corresponds to the second moment. The operations within the time period composed of the first moment and the second moment are the operations to be recorded.

[0128] Based on this, when obtaining the above requests, the computer device can obtain the operation data generated from the first moment to the second moment as the initial operation data. Other operation data can be not recorded or not stored, and no subsequent processing is performed on other operation data. Thus, while meeting the operation recording requirements and operation recognition requirements, on the one hand, the processing resources and storage resources required for recording operation data are reduced, and on the other hand, it is not necessary to perform operation recognition on each operation data, reducing the resource consumption in the operation recognition process.

[0129] For example, in a game application scenario, a player may want to repeat a game operation in a certain game program. At this time, the player can initiate an operation recording start request through a specific operation, and the computer device will start recording operation data. The player can perform the game operation in the game interface, so that the operation data corresponding to the game operation is obtained by the computer device. After the required game operation is completed, the player can initiate an operation recording end request through a specific operation. The computer device terminates recording operation data and can perform operation recognition on the obtained operation data to determine the position information required to reproduce the game operation. Thus, the game operation can be automatically implemented based on this position information without the player repeating the game operation, thereby improving the convenience of the player's game.

[0130] Next, the specific method for using the position information included in each position information set to determine the sliding trajectory of each contact medium in step S204 will be introduced.

[0131] In a possible implementation, when performing step S204, the computer device can perform steps S2041 - S2042 (not shown in the figure). Steps S2041 - S2042 are a possible implementation of step S204 and include:

[0132] S2041: Determine the positions respectively identified by the M sets of position information according to the position information respectively included in the M sets of position information.

[0133] Taking the first set of position information among the M sets of position information as an example, the first set of position information is any one of the M sets of position information. It can be seen from the above induction process that each set of position information includes the position information generated when a single contact medium slides, and the order of the position information is the position order of the contact medium when it slides. Therefore, the position information corresponding to each position is arranged in sequence in the set of position information, and there will be no situation where the position information corresponding to multiple positions is disordered with each other. Thus, the computer device can accurately determine the multiple positions corresponding to the multiple position information coordinates included in the first set of position information, and the position order in the positions identified by the first set of position information corresponds to the arrangement order of the position information included in the first set of position information in the to-be-recognized operation data.

[0134] S2042: Determine the positions respectively identified by the M sets of position information as the positions respectively passed by the M contact media when performing the target operation.

[0135] Since the order of the position information in each set of position information is the position order of the contact medium when it slides, the order of the positions identified by the set of position information in the set of position information is the position order of the corresponding contact medium during the sliding operation. For example, the order of the positions corresponding to the first set of position information is the position order of any contact medium when performing the target operation.

[0136] For example, in Figure 1 , the coordinate information corresponding to point 1, point 3, and point 5 are arranged in sequence in the inducted set 1, so that it can be determined that the positions identified by the set 1 are point 1, point 3, and point 5, and at the same time, the contact medium 1 slides in the order of point 1, point 3, and point 5.

[0137] Among them, based on the different types of position information, the method for determining the positions identified by the set of position information may also be different. Taking the type of position information in the above interface coordinate system scenario as an example, in a possible implementation manner, the target interface has a corresponding interface coordinate system, and the interface coordinate system is used to locate the contact position between the contact medium and the target interface. The position information is the coordinate information of any coordinate axis in the corresponding interface coordinate system. When performing step S2041, the computer device may execute steps S20411 - S20412 (not shown in the figure), and steps S20411 - S20412 are a possible implementation manner of step S2041, including:

[0138] S20411: Divide the coordinate information included in the first position information set into multiple groups of coordinate information.

[0139] Among them, each group of coordinate information includes the target number of adjacent coordinate information in the first position information set. The target number of coordinate information corresponds one-to-one with the coordinate axes of the interface coordinate system, and the target number is the dimension of the interface coordinate system. That is, each group of coordinate information includes one coordinate information corresponding to multiple coordinate axes in the interface coordinate system respectively. As mentioned above, the position information corresponding to each position is arranged in sequence in the position information set, and there will be no confusion between the position information corresponding to multiple positions. Therefore, each group of coordinate information is arranged in sequence in the first position information set, and there will be no confusion of coordinate information between multiple groups of coordinate information.

[0140] S20412: Determine the positions identified by the multiple groups of coordinate information in the interface coordinate system as the positions identified by the first position information set.

[0141] Multiple positions can be identified through multiple groups of coordinate information. These positions are the positions passed by the contact medium corresponding to the first position information set during the sliding operation. The arrangement order of the multiple groups of coordinate information in the first position information set is the position order corresponding to the first position information set. Since the arrangement order of the multiple groups of coordinate information in the first position information set is determined based on the position order passed by the contact medium corresponding to the first position information during the sliding operation, the arrangement order can be determined as the position order corresponding to the first position set to identify the position order of the above multiple positions, thereby identifying the sliding trajectory of the contact medium.

[0142] In addition to directly determining the sliding trajectory corresponding to the contact medium based on the position information set itself, in some implementation manners, the computer device can also combine multiple verification methods to verify the set grouping result of multiple position information in the operation data to be recognized, so as to ensure that the position information corresponding to each contact medium can be summarized into a single position information set, thereby ensuring the recognition accuracy of the target operation recognition.

[0143] In a possible implementation manner, the computer device can verify the induction result of the position information set by means of clustering.

[0144] The computer device can cluster the positions identified by M sets of position information according to the position information corresponding to each of the positions identified by the M sets of position information, to obtain multiple position clusters. The characteristic of clustering is to cluster similar objects into the same cluster and dissimilar objects into different clusters. Since the clustering objects in this application are positions and the information is position information, clustering can be used to make the proximity of the position information between positions in the same position cluster greater than that between positions in different position clusters. And positions with a higher degree of proximity have a greater probability of being the positions passed by the same contact medium during sliding. Therefore, positions clustered into the same position cluster are likely to be the positions passed by the same contact medium during sliding.

[0145] Based on this, when performing step S2042, the computer device may execute step S20421 (not shown in the figure). Step S20421 is a possible implementation of step S2042 and includes:

[0146] S20421: Based on the distribution of the positions identified by the M sets of position information in multiple position clusters being consistent with the distribution in the M sets of position information, determine the positions respectively identified by the M sets of position information as the positions respectively passed by the M contact media when performing the target operation.

[0147] If the distribution of the positions identified by the M sets of position information in multiple position clusters is consistent with the distribution in the M sets of position information, it indicates that the analysis of the correspondence between positions and contact media through clustering is consistent with the correspondence characterized by the sets of position information, verifying that the position information included in each set of position information is likely to be the position information corresponding to a single contact medium. At this time, the computer device can further determine the positions respectively identified by the M sets of position information as the positions respectively passed by the M contact media when performing the target operation, thereby further improving the accuracy of the induction of the multiple position information, and further improving the recognition accuracy of the target operation.

[0148] In another possible implementation, the computer device can improve the induction accuracy of the set of position information by fitting a curve. It can be understood that usually, the sliding trajectory of a single contact medium on the interface is relatively smooth and it is difficult to have large turns. For example, the sliding operation of a single finger on the interface is usually a sliding operation in a single direction or a sliding operation with a small degree of curvature. The computer device can analyze whether the position information included in each set of position information corresponds to a single contact medium based on this characteristic.

[0149] The computer device can use each of the M sets of position information as the second set of position information, and fit a fitting curve corresponding to the second set of position information based on the positions identified by the second set of position information. This fitting curve is the curve that sequentially passes through the positions identified by the second set of position information according to the position order corresponding to the second set of position information, that is, this fitting curve can simulate the sliding trajectory of the contact medium corresponding to the second set of position information during the target operation.

[0150] When performing step S2042, the computer device can execute step S20422 (not shown in the figure). Step S20422 is a possible implementation of step S2042 and includes:

[0151] S20422: Based on the curvature of the fitting curves corresponding to the M sets of position information being less than a second preset threshold, determine the positions identified by each of the M sets of position information as the positions passed by each of the M contact media during the execution of the target operation.

[0152] The computer device can analyze the curvature corresponding to the fitting curve. The curvature is used to characterize the degree of bending of the fitting curve. The larger the curvature, the greater the degree of bending. As mentioned above, the sliding trajectory of a single contact medium is usually relatively smooth and has a small degree of bending. Therefore, the computer device can preset a second preset threshold, which is used to determine whether the fitting curve has a small degree of bending. If the curvature of the fitting curve is less than this second preset threshold, it indicates that the fitting curve corresponding to the second set of position information has a small degree of bending and is more likely to be the sliding trajectory of a single contact medium. Based on this, if the curvatures of the fitting curves corresponding to the M sets of position information are all less than the second preset threshold, it means that each of the M sets of position information corresponds to the sliding trajectory of a single contact medium. Therefore, it can be determined that the position information generated by the target operation has been accurately summarized, and thus the positions identified by each of the M sets of position information can be determined as the positions passed by each of the M contact media during the execution of the target operation.

[0153] It can be understood that the way of identifying positions by position information can include various types. For example, in some cases, one piece of position information can identify a position, and in other cases, a position information group composed of multiple pieces of position information is required to identify a position. For the recognition technology of operations where multiple contact media slide simultaneously, if a position is composed of multiple pieces of position information, then since the position information corresponding to multiple contact media will be acquired simultaneously, the position information corresponding to different positions may be confused, resulting in the inability to directly identify individual sliding positions based on the original operation data.

[0154] Next, based on a way of constructing positions by multiple pieces of position information, this application will be introduced on how to accurately identify the sliding trajectory of each contact medium.

[0155] In a possible implementation, the target interface has a corresponding interface coordinate system. The interface coordinate system is used to locate the contact position between the contact medium and the target interface. The multiple position information includes the coordinate information of each coordinate axis corresponding to the interface coordinate system. A coordinate position can be identified in the interface coordinate system through a coordinate information corresponding to each coordinate axis respectively. For example, a coordinate position in the plane coordinate system can be identified through an X-axis coordinate and a Y-axis coordinate.

[0156] Wherein, the i-th position information is the coordinate information of the target coordinate axis in the corresponding interface coordinate system, and the target coordinate axis is any coordinate axis in the interface coordinate system, such as the X-axis, Y-axis, Z-axis, etc. When executing step S203, the computer device can execute steps S2031 - S2032 (not shown in the figure). Steps S2031 - S2032 are a possible implementation of step S203, including:

[0157] S2031: Determine the degree of proximity between the coordinate information corresponding to the target coordinate axis included in each of the M position information sets and the coordinate information corresponding to the i-th position information.

[0158] It can be understood that in the coordinate system, the degree of proximity between coordinate information should be measured by the coordinate information corresponding to the same coordinate axis to ensure the reliability of the degree of proximity. For example, if the coordinate information corresponding to two points on the X-axis is relatively close, then compared with two points that are not close on the X-axis, the degree of proximity is usually higher. Based on this, in the present application, when analyzing the degree of proximity, the computer device can first determine the coordinate information corresponding to the target coordinate axis in each position information set, and then determine the degree of proximity between this part of the coordinate information and the coordinate information corresponding to the i-th position information. This degree of proximity can more accurately represent the degree of proximity between the i-th position information and the position information included in each position information set.

[0159] S2032: Incorporate the i-th position information into the target position information set with the highest degree of proximity.

[0160] In this way, the computer device can accurately classify the position information of the coordinate information type, so that the positions identified by the coordinate information in each position information set are relatively close in each coordinate axis dimension, thereby increasing the probability that each position information set corresponds to a single contact medium.

[0161] It can be understood that the sliding process of the contact medium on the target interface is a continuous process. Therefore, usually, the newly contacted position during the sliding process of the contact medium is usually the closest to the previous position contacted during the sliding process. Based on this, in a possible implementation, the computer device can measure whether the position information belongs to the same contact medium by the proximity between adjacent positions.

[0162] When performing step S2031, the computer device can execute steps S20311 - S20313 (not shown in the figure). Steps S20311 - S20313 are a possible implementation of step S2031, including:

[0163] S20311: Respectively use the M sets of position information as the third set of position information, and determine the target position information that is newly included in the third set of position information. The target position information is the coordinate information corresponding to the target coordinate axis.

[0164] As mentioned above, the induction order of the position information is the arrangement order of the position information in the operation data to be recognized, and this arrangement order corresponds to the position order passed by the contact medium during the sliding process. Therefore, the target position information is the position information that is the last in the sliding order among the multiple pieces of position information included in the third set of position information. That is, if the i-th piece of position information is the position information corresponding to the contact medium corresponding to the third set of position information, then theoretically, the proximity between the i-th piece of position information and the target position information is the highest. For example, in Figure 1 , point 1 is the previous point passed by point 3 during the sliding process. Then, when point 5 has not appeared yet, point 1 is the closest point to point 3.

[0165] Based on this, the computer device can determine the proximity between the i-th piece of position information and the position information included in the third set of position information based on the proximity between the i-th piece of position information and the target position information.

[0166] S20312: Determine the target proximity between the coordinate information corresponding to the target position information and the coordinate information corresponding to the i-th piece of position information.

[0167] S20313: Determine the target proximity as the proximity between the coordinate information corresponding to the target coordinate axis included in the third set of position information and the coordinate information corresponding to the i-th piece of position information.

[0168] The proximity determined in this way can better conform to the operation characteristics of the target operation, so that the i-th piece of position information can be more accurately included in the corresponding set of position information, and ultimately improve the recognition accuracy of the target operation.

[0169] Next, the application method of the identified target operation in this application will be introduced in detail.

[0170] It can be understood that for hardware devices, the control of the interface is achieved based on the received instructions one by one. In the field of interactive interfaces, these instructions are usually generated based on the operations of the contact medium on the interface. That is, after obtaining the operations implemented by the contact medium, corresponding instructions will be generated, and the hardware device will control the interface to make a response corresponding to the operation based on the instructions.

[0171] Based on this, in a possible implementation manner, the computer device can automatically reproduce the same target operation by analyzing the accurate sliding trajectory corresponding to the target operation. Operation reproduction refers to achieving the response effect achieved through the operation.

[0172] First, the computer device can generate an operation instruction corresponding to the target operation according to the positions passed by the M contact media when performing the target operation. Since these positions can accurately identify the sliding trajectories of each contact media during the target operation, these positions are the actual positions obtained by the device corresponding to the target interface when performing the target operation. That is, the device corresponding to the target interface constructs instructions based on these positions and makes a response corresponding to the target operation through the instructions. Therefore, the operation instructions generated through these positions can be used to make the target interface make an interface response when receiving the target operation. The computer device can execute the operation instruction, thereby automatically displaying the execution effect of performing the target operation. For example, in a game program, the player can zoom in on the game map in the game interface through a two-finger sliding operation. After analyzing the operation data corresponding to the two-finger sliding operation and generating the corresponding instruction, based on this instruction, the device running the game program can automatically zoom in on the game map in the game interface without the player performing the two-finger sliding operation, so as to reproduce the response effect brought by performing the two-finger sliding operation.

[0173] Specifically, in order to better meet the requirements of operation reproduction, in a possible implementation manner, the analysis of the instruction by the computer device and the reproduction of the operation response based on the instruction can be actively triggered.

[0174] If it is necessary to reproduce the operation for the target operation, an operation reproduction request can be sent to the computer device, for example, by a specific operation in the interface to send the request, etc.

[0175] When generating an operation instruction corresponding to a target operation based on the positions passed by M contact media during the execution of the target operation, the computer device may generate the operation instruction corresponding to the target operation based on obtaining an operation reproduction request, where the operation reproduction request is used to request to reproduce the target operation for a target interface. In this way, the operation reproduction for the target operation can be made into a controllable behavior, and the user can freely select the operation they want to reproduce, improving the freedom of operation recognition. For example, in a game program, a player can select the game operation they want to reproduce to meet various operation automation requirements of the player, such as automatic combat operations, automatic walking operations, etc.

[0176] It can be understood that although the response of the target interface to the target operation is brought about by the execution of the target operation, in some cases, not all of the multiple positions involved in the target operation may play a key role in the interface response. For example, when zooming in on an image in the interface by a two-finger swipe operation, as long as the two fingers are swiped in two opposite directions, the positions passed during the swipe are not important, and multiple different two-finger swipe operations can achieve the same image zoom effect. That is, in the swipe operation, the swipe direction determined by the starting point and the ending point is more important, and smaller differences can occur in the positions passed in the middle without affecting the effect of the swipe operation.

[0177] At the same time, as mentioned above, the operation instruction is generated based on the position. Therefore, the more positions used to generate the operation instruction, to a certain extent, the greater the difficulty of generating the operation instruction, the higher the complexity of the operation instruction, and the greater the difficulty of executing the operation instruction. Based on this, in a possible implementation manner, the computer device can reduce the difficulty of generating the operation instruction while ensuring the operation reproduction effect by streamlining the positions used to generate the operation instruction.

[0178] When generating an operation instruction corresponding to a target operation based on the positions passed by M contact media during the execution of the target operation, the computer device may use the M position information sets as the fourth position information set respectively, and determine the endpoint positions and non-endpoint positions identified by the fourth position information set. Among them, the endpoint positions are the starting position and the ending position among the positions passed by the contact media corresponding to the fourth position information set during the execution of the target operation, and the endpoint positions determine the general direction of the swipe operation. The non-endpoint positions are the other positions except the endpoint positions among the positions identified by the fourth position information set, that is, the non-endpoint positions determine the specific swipe details of the swipe operation.

[0179] If the computer device wants to ensure the effectiveness of the sliding operation while reducing the positions involved in the sliding trajectory, it needs to ensure that the sliding trajectory after reducing the positions has a high consistency with the sliding trajectory before reducing the positions. Based on this, the computer device can determine a target position from the non-endpoint positions. The target position is a part of the non-endpoint positions, and this target position satisfies that the similarity between the first curve and the second curve is greater than a third preset threshold. The first curve is a curve fitted based on the positions identified by the fourth position information set, that is, the sliding trajectory of the contact medium corresponding to the fourth position information set during the target operation. The second curve is a curve fitted based on the endpoint positions and the target position, that is, the sliding trajectory after position reduction. The third preset threshold is used to determine whether there is a high similarity between two curves. If the similarity between the first curve and the second curve is greater than the third preset threshold, it means that the similarity between the two curves is high. Therefore, the interface responses that can be triggered by the sliding operations with these two curves as the sliding trajectories in the target interface are probably relatively consistent. Based on this, the computer device can generate an operation instruction corresponding to the target operation according to the endpoint positions and the target position, so as to reduce the positions used to generate the operation instruction while ensuring that the operation instruction can trigger the interface response corresponding to the target operation. On the one hand, it reduces the difficulty of generating the operation instruction and the complexity of the operation instruction. On the other hand, it also reduces the difficulty of executing the operation instruction, enabling the target operation to be accurately and efficiently reproduced.

[0180] Among them, the reproduction of the target operation in this application can be applied to multiple scenarios. For example, in a possible implementation, the operation reproduction can be applied to a game scenario. The target interface can be a game interface, and the target operation is used to trigger a target game function in the game interface, such as the function of enlarging the game map, the function of controlling the movement of game objects, etc. The operation instruction can be used to trigger the target game function in the game interface. Thus, when it comes to some game functions that need to be repeatedly executed, the player can use the operation reproduction function provided by this application to make the computer device automatically repeat the execution of the game function. The player only needs to perform one operation to provide the operation data, which further simplifies the player's game operation and improves the player's game experience.

[0181] For example, in Figure 3 , the player can zoom in on the game map in the game interface through a two-finger sliding operation. At this time, the computer device can obtain the operation data corresponding to the two-finger sliding operation, identify the sliding trajectories corresponding to the two fingers from it, and then generate a corresponding operation instruction. Based on this instruction, the game map in the game interface can be zoomed in at the same or similar zoom ratio.

[0182] To facilitate the understanding of the technical solution provided by this application, next, an operation recognition method provided by an embodiment of this application will be introduced in combination with an actual application scenario.

[0183] See Figure 4 , Figure 4 is a flowchart of an operation recognition method in an actual application scenario provided by an embodiment of this application. In this actual application scenario, the computer device can be a terminal device running a game program, such as a mobile phone, a tablet computer, etc. The target interface can be a game interface, and the game interface has a corresponding two-dimensional interface coordinate system. The position information is the coordinate information corresponding to any coordinate axis on the coordinate system, such as the coordinate information corresponding to the X-axis or the coordinate information corresponding to the Y-axis, etc. The target operation can be a multi-finger sliding operation. The method includes:

[0184] S401: Split the initial operation data to obtain multiple pieces of operation data.

[0185] The computer device can obtain the initial operation data based on the multi-finger sliding operation on the game interface. Specifically, the computer device can complete this step through some software or processes for obtaining operation data. For example, in this actual application scenario, the computer device can obtain the operation data through the event acquisition tool getevent. getevent is a tool in the system and is essentially an executable file used to provide information on real-time dumps of input devices and kernel input events in the system. It can be simply understood that through this tool, the real-time information flow of operations in the system can be obtained, including interface input events (such as click, slide, long press, return to the home screen (home), back, etc.), key input events (such as power key, volume control key, etc.), and real-time information of some kernel input events. The result will be displayed as a string stream data in a specific format.

[0186] When starting to obtain the operation data, the computer device can trigger the acquisition of the operation data through specific instructions, such as through the adb shell instruction or directly through the shell ability to execute the terminal instruction of -l -t in getevent. When this instruction is executed, the device running the game program will output the relevant operation data. At this time, if the player performs an operation on the game interface, the computer device can obtain the corresponding initial operation data

[0187] First, the computer device can divide the initial operation data into multiple operation data based on the operation start identifier and the operation end identifier. Each operation data includes the position information corresponding to one operation. It can be understood that the initially obtained position information may not be in an analyzable format. For example, it may be a series of strings without explicit meaning. The computer device can first parse it. If the obtained X and Y coordinates are displayed in hexadecimal and are not the original coordinate data, the computer device can calculate according to the coordinate conversion rule to obtain the coordinate data corresponding to the actual target interface coordinate system after conversion.

[0188] S402: Identify the contact position information, and identify the operation data corresponding to the multi-finger swipe operation from the multiple operation data.

[0189] The computer device can identify the contact position information at the front section of the operation data. If the contact position information identifies multiple contact positions, it means that the operation data is generated by the operation of multiple contact media; if the contact position information only identifies a single contact position, it means that the operation data is generated by the operation of a single contact media. Then the computer device can judge the number of position information included in the operation data, and determine the operation data corresponding to the multi-finger swipe operation as the operation data with multiple contact positions identified by the contact position information and a relatively large number of position information.

[0190] Taking the two-finger swipe operation as an example, as Figure 5 shown, during the two-finger swipe operation, the position information corresponding to contact medium 1 and contact medium 2 will be obtained simultaneously. Therefore, there will be a situation where the position information during the swipe is obtained in an interleaved manner. However, since there is usually a difference in the time when contact medium 1 and contact medium 2 first touch the game interface, the contact position information "X1, Y1, X2, Y2" can be determined through adjacent coordinate information and the contact identifier "Down" to identify the first contact positions (X1, Y1) and (X2, Y2). Since the operation data includes the contact position information for identifying two first contact positions and has a relatively large amount of coordinate information, it can be determined that the operation data is the operation data corresponding to the two-finger swipe operation, and (X1, Y1) and (X2, Y2) are the starting points of the swipe trajectories corresponding to the two contact media respectively.

[0191] S403: Construct multiple position information sets based on the contact position information.

[0192] Among them, each position information set includes the coordinate information for identifying a first contact position, and each position information set is used to record the position and position order of a contact medium during the multi-finger swipe operation.

[0193] S404: Whether to summarize all the coordinate information

[0194] The computer device can determine whether all the coordinate information in the operation data has been summarized into the position information set. If not, step S405 is executed.

[0195] S405: Determine the coordinate axis corresponding to the coordinate information.

[0196] If the coordinate information to be summarized is the coordinate information on the X-axis, step S406 is executed. If it is the coordinate information on the Y-axis, step S407 is executed.

[0197] S406: Compare the coordinate information with the latest added X-axis coordinate information in each position information set.

[0198] S407: Compare the coordinate information with the latest added Y-axis coordinate information in each position information set.

[0199] Among them, the latest added X-axis / Y-axis coordinate information in the position information set is the latest coordinate information generated during the sliding of the contact medium among the coordinate information included in the position information set. Therefore, determining the proximity through these coordinate information can most truly reflect the proximity between the coordinate information to be summarized and the sliding trajectory corresponding to the position information set. The calculation of the proximity can be an absolute value calculation. For example, in Figure 6 In the scenario of a two-finger sliding operation shown, when summarizing the coordinate information X3, the corresponding latest X-axis coordinate information in the two position information sets are X1 and X2 respectively. Since |X3 - X1| < |X3 - X2|, X3 belongs to the coordinate data in set A. Therefore, at this time, X3 will be summarized into set A; after the value Y4, similarly, it will continue to be compared with the absolute value difference of Y1 in set A and Y2 in set B. Because |Y4 - Y1| > |Y4 - Y2| at this time, Y4 belongs to the coordinate data in set B. Therefore, it will be merged into set B by the algorithm.

[0200] S408: Add the coordinate information to the position information set with the highest proximity.

[0201] S409: Determine the position and position order of the position information set identifier.

[0202] In the set of location information where induction is completed, a group of adjacent X-axis coordinates and Y-axis coordinates correspond to a location, and the order of the coordinate information is the location order. The computer device can verify the induction accuracy of the set of location information based on clustering. For example, it can be verified through the Density-Based Spatial Clustering of Applications with Noise (abbreviated as DBSCAN). DBSCAN clusters objects with a high degree of proximity into the same object cluster and objects with a low degree of proximity into different object clusters. In this actual application scenario, the objects to be clustered are the locations in the set of location information (note that they are the locations identified by the location information, not the location information itself), and the basis for clustering is the location information corresponding to the locations, so that relatively close locations can be clustered into the same location cluster, that is, the locations passed by the same contact medium during the multi-finger sliding operation are clustered into the same location cluster. At the same time, the computer device will perform curve fitting on the locations identified by each set of location information and analyze the curvature of the fitted curve.

[0203] If the clustering method of the locations obtained through clustering is consistent with the distribution method of the locations in the set of location information, and the curvature of each fitted curve is small, it can be shown that the multiple coordinate information in the operation data has been accurately induced.

[0204] As Figure 7 shown, Set A is consistent with the locations in Location Cluster A, Set B is consistent with the locations in Location Cluster B, and the bending degrees of the fitted curves A and B obtained by fitting are both small. Therefore, it can be determined that the sliding trajectory of the two-finger sliding operation has been accurately recognized.

[0205] S410: Construct an operation instruction based on the locations and location order identified by the set of location information.

[0206] In this actual application scenario, the multi-finger sliding operation can be used to Figure 3 as shown, zoom in on the game map in the game interface. Therefore, this constructed operation instruction is an instruction to make the computer device zoom in and display the game map, and its zoom level fits this multi-finger sliding operation. When generating the instruction, the computer device can screen the coordinate information in the above manner and only retain the coordinate information of the key locations that can determine the shape of the sliding trajectory curve for instruction construction.

[0207] S411: Implement the operation reproduction of the multi-finger sliding operation based on the operation instruction

[0208] It can be seen that this application solves the following pain points in the related technology:

[0209] Pain Point 1: The operation data corresponding to the multi-finger sliding operation is difficult to identify

[0210] Based on the feature that there is a slight time difference in the first contact position during the multi-finger sliding operation, this application identifies the contact position information in the operation data, so as to accurately identify the operation data corresponding to the multi-finger sliding operation.

[0211] Pain point 2: Large amount of data

[0212] On the one hand, this application only needs to analyze the simple positional relationship to complete the operation recognition of the operation data, and the data processing method is relatively simple, so it is less affected by the amount of data; on the other hand, this application will screen the key positions when constructing instructions, thereby reducing the difficulty and complexity of instruction construction.

[0213] Pain point 3: High complexity of position information

[0214] This application can use the proximity between position information to accurately generalize the position information corresponding to each contact medium from the intricate multiple position information, so as to accurately identify the sliding trajectories of each contact medium.

[0215] Pain point 4: Low recognition accuracy

[0216] This application can verify the induced sliding trajectory through various verification methods such as clustering and curve fitting, ensuring the accuracy of operation recognition.

[0217] Based on the operation recognition method provided in the above embodiments, this application also provides an operation recognition device. See Figure 8 , Figure 8 is a structural block diagram of an operation recognition device 800 provided in an embodiment of this application. The device includes a first acquisition unit 801, a first determination unit 802, an induction unit 803, and a second determination unit 804:

[0218] The first acquisition unit 801 is configured to acquire operation data to be recognized. The operation data to be recognized is operation data generated based on a target operation. The target operation is an operation in which M contact media slide simultaneously on a target interface. The operation data to be recognized includes multiple position information, and the position information is used to identify the positions passed by the M contact media on the target interface. The arrangement order of the multiple position information in the operation data to be recognized corresponds to the position order passed by the M contact media during sliding. M is an integer greater than 1;

[0219] The first determination unit 802 is configured to determine the starting position information among the multiple position information. The starting position information is used to identify the M sliding starting points corresponding to the M contact media. The sliding starting point is the starting point at which the contact media slides on the target interface;

[0220] The induction unit 803 is configured to perform induction on the N location information respectively based on the arrangement order of the N location information. For the i-th location information among the N location information, the i-th location information is inducted into the target location information set, where the target location information set is the location information set including the location information closest to the i-th location information among the location information included in the M location information sets. The N location information is the location information among the multiple location information except the starting point location information. The M location information sets are constructed based on the starting point location information, and each location information set includes the starting point location information corresponding to any sliding starting point. The starting point location information included in different location information sets identifies different sliding starting points;

[0221] The second determination unit 804 is configured to determine the positions respectively passed by the M contact media when performing the target operation according to the positions respectively identified by the M location information sets after induction.

[0222] In a possible implementation manner, the apparatus further includes a second acquisition unit:

[0223] The second acquisition unit is configured to acquire a plurality of operation data, where the plurality of operation data corresponds to the operations on the target interface one by one, and each operation data includes a plurality of location information;

[0224] The first acquisition unit 801 is specifically configured to:

[0225] Determine the operation data including multiple sets of contact location information and having the number of included location information greater than a first preset threshold among the plurality of operation data as the to-be-identified operation data. The contact location information is the location information generated by the first contact of the contact medium with the target interface, and each set of contact location information is used to identify the contact location of any contact medium with the target interface for the first contact.

[0226] In a possible implementation manner, the first determination unit 802 is specifically configured to:

[0227] Determine the multiple sets of contact location information as the starting point location information, and the contact location identified by each set of contact location information is the sliding starting point.

[0228] In a possible implementation manner, the contact location information is location information with a contact identifier, and the contact identifier is used to identify that the contact location information is the location information generated by the first contact of the contact medium with the target interface.

[0229] In a possible implementation, the target interface has a corresponding interface coordinate system, which is used to locate the contact position between the contact medium and the target interface. Each set of contact position information includes the coordinate information corresponding to multiple coordinate axes of the interface coordinate system, and the coordinate information included in each set of contact position information is adjacent among the multiple position information included in the to-be-recognized operation data. The coordinate information corresponding to the multiple coordinate axes is used to identify the contact position in the interface coordinate system.

[0230] In a possible implementation, the device further includes a third acquisition unit:

[0231] The third acquisition unit is configured to acquire initial operation data, where the initial operation data includes position information, an operation start identifier, and an operation end identifier. The operation start identifier is used to identify the start of any operation on the target interface, and the operation end identifier is used to identify the end of any operation on the target interface;

[0232] Specifically, the second acquisition unit is configured to:

[0233] Determine the position information between any adjacent operation start identifier and operation end identifier in the initial operation data as an operation data, and obtain the multiple operation data.

[0234] In a possible implementation, specifically, the third acquisition unit is configured to:

[0235] Acquire an operation recording start request and an operation recording end request, where the operation recording start request corresponds to a first moment, and the operation recording end request corresponds to a second moment;

[0236] Acquire the operation data generated from the first moment to the second moment as the initial operation data.

[0237] In a possible implementation, specifically, the induction unit 803 is configured to:

[0238] According to the position information included in the M position information sets respectively, determine the positions identified by the M position information sets respectively. The position order in the positions identified by the first position information set corresponds to the arrangement order of the position information included in the first position information set in the to-be-recognized operation data. The first position information set is any one of the M position information sets;

[0239] Determine the positions identified by the M position information sets respectively as the positions passed by the M contact media respectively when performing the target operation. The position order corresponding to the first position information set is the position order of the positions passed by any contact medium when performing the target operation.

[0240] In a possible implementation, the target interface has a corresponding interface coordinate system for positioning the contact position of the contact medium with the target interface. The position information is the coordinate information of any coordinate axis in the interface coordinate system. Specifically, the induction unit 803 is configured to:

[0241] Divide the coordinate information included in the first position information set into multiple groups of coordinate information. Each group of coordinate information includes a target number of adjacent coordinate information in the first position information set. The target number of coordinate information corresponds one-to-one with the coordinate axes of the interface coordinate system, and the target number is the dimension of the interface coordinate system;

[0242] Determine the positions identified by the multiple groups of coordinate information in the interface coordinate system as the positions identified by the first position information set. The arrangement order of the multiple groups of coordinate information in the first position information set is the position order corresponding to the first position information set.

[0243] In a possible implementation, the device further includes a clustering unit:

[0244] The clustering unit is configured to cluster the positions identified by the M position information sets according to the position information corresponding to the positions identified by the M position information sets, to obtain multiple position clusters. The clustering is used to make the proximity degree of the position information between the positions in the same position cluster greater than the proximity degree of the position information between the positions in different position clusters;

[0245] Specifically, the second determination unit 804 is configured to:

[0246] Based on the distribution of the positions identified by the M position information sets in the multiple position clusters being consistent with the distribution in the M position information sets, determine the positions identified by the M position information sets as the positions respectively passed by the M contact media when performing the target operation.

[0247] In a possible implementation, the device further includes a fitting unit:

[0248] The fitting unit is configured to use the M position information sets as the second position information sets respectively, and fit a fitting curve corresponding to the second position information set according to the positions identified by the second position information set;

[0249] Specifically, the second determination unit 804 is configured to:

[0250] Based on the curvatures of the fitting curves corresponding to the M sets of position information being less than a second preset threshold, determine the positions respectively identified by the M sets of position information as the positions passed by the M contact media when performing the target operation.

[0251] In a possible implementation, the target interface has a corresponding interface coordinate system for positioning the contact position between the contact medium and the target interface. The multiple sets of position information include coordinate information corresponding to each coordinate axis of the interface coordinate system. The i-th set of position information is the coordinate information corresponding to the target coordinate axis in the interface coordinate system, and the target coordinate axis is any coordinate axis in the interface coordinate system. The induction unit 803 is specifically configured to:

[0252] Determine the degree of proximity between the coordinate information corresponding to the target coordinate axis included in each of the M sets of position information and the coordinate information corresponding to the i-th set of position information;

[0253] Induce the i-th set of position information into the set of target position information with the greatest degree of proximity.

[0254] In a possible implementation, the induction unit 803 is specifically configured to:

[0255] Take each of the M sets of position information as a third set of position information, and determine the target position information that is newly induced into the third set of position information. The target position information is the coordinate information corresponding to the target coordinate axis;

[0256] Determine the target degree of proximity between the coordinate information corresponding to the target position information and the coordinate information corresponding to the i-th set of position information;

[0257] Determine the target degree of proximity as the degree of proximity between the coordinate information corresponding to the target coordinate axis included in the third set of position information and the coordinate information corresponding to the i-th set of position information.

[0258] In a possible implementation, the device further includes a generation unit and an execution unit:

[0259] The generation unit is configured to generate an operation instruction corresponding to the target operation according to the positions passed by the M contact media when performing the target operation;

[0260] The execution unit is configured to execute the operation instruction, and the operation instruction is used to make the target interface make an interface response when receiving the target operation.

[0261] In a possible implementation, the generation unit is specifically configured to:

[0262] Based on the operation reproduction request, an operation instruction corresponding to the target operation is generated according to the positions passed by the M contact media respectively when the target operation is executed, and the operation reproduction request is used to request to reproduce the target operation for the target interface.

[0263] In a possible implementation manner, the generating unit is specifically configured to:

[0264] Respectively use the M position information sets as the fourth position information set, and determine the endpoint positions and non-endpoint positions identified by the fourth position information set. The endpoint positions are the starting position and the ending position among the positions passed by the contact media corresponding to the fourth position information set when the target operation is executed, and the non-endpoint positions are other positions in the positions identified by the fourth position information set except the endpoint positions;

[0265] Determine a target position from the non-endpoint positions. The target position is a part of the non-endpoint positions, and the target position satisfies that the similarity between the first curve and the second curve is greater than a third preset threshold. The first curve is a curve fitted based on the positions identified by the fourth position information set, and the second curve is a curve fitted based on the endpoint positions and the target position;

[0266] Generate an operation instruction corresponding to the target operation according to the endpoint positions and the target position.

[0267] In a possible implementation manner, the target interface is a game interface, the target operation is used to trigger a target game function in the game interface, and the operation instruction is used to trigger the target game function in the game interface.

[0268] The embodiments of the present application further provide a computer device. Please refer to Figure 9 as shown. This computer device may be a terminal device. Taking the terminal device as a mobile phone as an example:

[0269] Figure 9 The block diagram of a part of the structure of the mobile phone related to the terminal device provided by the embodiments of the present application is shown. Referring to Figure 9 , the mobile phone includes: a radio frequency (RF) circuit 710, a memory 720, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a wireless fidelity (WiFi) module 770, a processor 780, and a power supply 790, etc. Those skilled in the art can understand that Figure 9 the structure of the mobile phone shown in does not constitute a limitation on the mobile phone, and it may include more or fewer components than shown in the figure, or combine some components, or arrange different components.

[0270] The following will combine with Figure 9 to specifically introduce each component of the mobile phone:

[0271] The RF circuit 710 can be used for receiving and transmitting signals during information reception or call processes. Specifically, after receiving the downlink information from the base station, it is sent to the processor 780 for processing; in addition, the uplink data is sent to the base station. Generally, the RF circuit 710 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (Low Noise Amplifier, abbreviated as LNA), a duplexer, etc. In addition, the RF circuit 710 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile communication (abbreviated as GSM), General Packet Radio Service (abbreviated as GPRS), Code Division Multiple Access (abbreviated as CDMA), Wideband Code Division Multiple Access (abbreviated as WCDMA), Long Term Evolution (abbreviated as LTE), email, Short Messaging Service (abbreviated as SMS), etc.

[0272] The memory 720 can be used to store software programs and modules. The processor 780 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 720. The memory 720 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory 720 can include high-speed random access memory and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0273] The input unit 730 can be used to receive input numeric or character information and generate key signal inputs related to the user settings and function controls of the mobile phone. Specifically, the input unit 730 can include a touch panel 731 and other input devices 732. The touch panel 731, also known as a touch screen, can collect touch operations of the user thereon or nearby (such as operations of the user using any suitable object or accessory such as a finger, a stylus, etc. on or near the touch panel 731), and drive corresponding connection devices according to a preset program. Optionally, the touch panel 731 can include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the touch orientation of the user, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 780, and can receive and execute commands sent by the processor 780. In addition, various types such as resistive, capacitive, infrared, and surface acoustic wave can be used to implement the touch panel 731. In addition to the touch panel 731, the input unit 730 can also include other input devices 732. Specifically, the other input devices 732 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0274] The display unit 740 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 740 can include a display panel 741. Optionally, the display panel 741 can be configured in the form of a liquid crystal display (LCD for short), an organic light-emitting diode (OLED for short), etc. Further, the touch panel 731 can cover the display panel 741. When the touch panel 731 detects a touch operation thereon or nearby, it is transmitted to the processor 780 to determine the type of touch event. Subsequently, the processor 780 provides corresponding visual output on the display panel 741 according to the type of touch event. Although in Figure 9 the touch panel 731 and the display panel 741 are implemented as two independent components to realize the input and input functions of the mobile phone, in some embodiments, the touch panel 731 and the display panel 741 can be integrated to realize the input and output functions of the mobile phone.

[0275] The mobile phone may further include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. Among them, the ambient light sensor can adjust the brightness of the display panel 741 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 741 and / or the backlight when the mobile phone is moved to the ear. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary, and can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the mobile phone can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be elaborated here.

[0276] The audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the user and the mobile phone. The audio circuit 760 can transmit the electrical signal converted from the received audio data to the speaker 761, and the speaker 761 converts it into a sound signal for output; on the other hand, the microphone 762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 760 and then converted into audio data. After the audio data is output to the processor 780 for processing, it is sent through the RF circuit 710 to, for example, another mobile phone, or the audio data is output to the memory 720 for further processing.

[0277] WiFi belongs to short - range wireless transmission technology. The mobile phone can help users send and receive emails, browse the web, and access streaming media through the WiFi module 770, which provides users with wireless broadband Internet access. Although Figure 9 the WiFi module 770 is shown, it can be understood that it does not belong to an essential component of the mobile phone and can be omitted entirely within the scope of not changing the essence of the invention according to needs.

[0278] The processor 780 is the control center of the mobile phone, connecting various parts of the entire mobile phone using various interfaces and lines. By running or executing software programs and / or modules stored in the memory 720, and by calling the data stored in the memory 720, it executes various functions of the mobile phone and processes data, thereby performing an overall detection of the mobile phone. Optionally, the processor 780 may include one or more processing units; preferably, the processor 780 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above - mentioned modem processor may not be integrated into the processor 780 either.

[0279] The mobile phone further includes a power supply 790 (such as a battery) for powering each component. Preferably, the power supply can be logically connected to the processor 780 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system.

[0280] Although not shown, the mobile phone may further include a camera, a Bluetooth module, etc., which will not be elaborated here.

[0281] In this embodiment, the processor 780 included in the terminal device further has the following functions:

[0282] Obtain operation data to be recognized, where the operation data to be recognized is operation data generated based on a target operation, the target operation is an operation in which M contact media slide simultaneously on a target interface, the operation data to be recognized includes a plurality of position information, the position information is used to identify the positions passed by the M contact media on the target interface, and the arrangement order of the plurality of position information in the operation data to be recognized corresponds to the position order passed by the M contact media during sliding, and M is an integer greater than 1;

[0283] Determine a starting position information among the plurality of position information, where the starting position information is used to identify M sliding starting points corresponding to the M contact media, and the sliding starting point is the starting point where the contact media slides on the target interface;

[0284] Induce the N position information respectively based on the arrangement order of the N position information. For the i-th position information among the N position information, induce the i-th position information into a target position information set, where the target position information set is the position information set including the position information closest to the i-th position information among the position information included in the M position information sets, the N position information is the position information among the plurality of position information except the starting position information, the M position information sets are constructed based on the starting position information, each position information set includes the starting position information corresponding to any one sliding starting point, and the sliding starting points identified by the starting position information included in different position information sets are different;

[0285] Determine the positions passed by the M contact media respectively when performing the target operation according to the positions respectively identified by the M position information sets after induction.

[0286] An embodiment of the present application further provides a server. Please refer to Figure 10 as shown Figure 10This is a structural diagram of server 800 provided by an embodiment of the present application. Server 800 may vary significantly due to different configurations or performances, and may include one or more central processing units (CPUs) 822 (for example, one or more processors) and a memory 832, and one or more storage media 830 (for example, one or more mass storage devices) that store application programs 842 or data 844. Among them, the memory 832 and the storage media 830 may be transient storage or persistent storage. The programs stored in the storage media 830 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Further, the central processing unit 822 may be configured to communicate with the storage media 830 and execute a series of instruction operations in the storage media 830 on the server 800.

[0287] Server 800 may further include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input / output interfaces 858, and / or one or more operating systems 841, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM and so on.

[0288] The steps performed by the server in the above embodiments may be based on Figure 10 the server structure shown.

[0289] An embodiment of the present application further provides a computer-readable storage medium for storing a computer program, and the computer program is used to execute any one of the operation recognition methods described in the foregoing embodiments.

[0290] An embodiment of the present application further provides a computer program product including a computer program, and when it runs on a computer device, it causes the computer device to execute the operation recognition method described in any one of the above embodiments.

[0291] It can be understood that in the specific implementation manner of the present application, when it comes to data related to user information (such as operation data), etc., when the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0292] Those of ordinary skill in the art will understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk, or optical disc, etc., various media that can store program code.

[0293] It should be noted that the embodiments in this specification are all described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiments. The device and system embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0294] As described above, it is only a specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in this application should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. An operation recognition method, characterized in that, The method includes: Obtaining operation data to be recognized, where the operation data to be recognized is operation data generated based on a target operation. The target operation is an operation in which M contact media slide simultaneously on a target interface. The operation data to be recognized includes multiple position information, and the position information is used to identify the positions passed by the M contact media on the target interface. The arrangement order of the multiple position information in the operation data to be recognized corresponds to the position order passed by the M contact media during sliding. M is an integer greater than 1; Determining a starting position information among the multiple position information, where the starting position information is used to identify the M sliding starting points corresponding to the M contact media, and the sliding starting point is the starting point where the contact media slides on the target interface; Inductively processing the N position information respectively based on the arrangement order of the N position information. For the i-th position information among the N position information, the i-th position information is inductively processed into a target position information set. The target position information set is the position information set among the M position information sets that includes the position information closest to the i-th position information. The N position information is the position information among the multiple position information excluding the starting position information. The M position information sets are constructed based on the starting position information. Each position information set includes the starting position information corresponding to any one sliding starting point, and the sliding starting points identified by the starting position information included in different position information sets are different; Determining the positions passed by the M contact media respectively when performing the target operation according to the positions respectively identified by the M position information sets after induction.

2. The method according to claim 1, characterized in that, The method further includes: Obtaining multiple operation data, where the multiple operation data corresponds one-to-one to the operations on the target interface, and each operation data includes multiple position information; The obtaining of the operation data to be recognized includes: Determining the operation data including multiple contact position information among the multiple position information included in the multiple operation data and having the number of included position information greater than a first preset threshold as the operation data to be recognized. The contact position information is the position information generated by the first contact of the contact media with the target interface, and each group of contact position information is used to identify the contact position of any one contact media with the target interface for the first contact.

3. The method according to claim 2, wherein The determining of the starting position information among the multiple position information includes: Determining the multiple groups of contact position information as the starting position information, and the contact position identified by each group of contact position information is the sliding starting point.

4. The method according to claim 2, wherein The contact position information is position information with a contact identifier, and the contact identifier is used to identify that the contact position information is the position information generated by the first contact of the contact media with the target interface.

5. The method according to claim 2, wherein The target interface has a corresponding interface coordinate system, which is used to locate the contact position between the contact medium and the target interface. Each set of contact position information includes the coordinate information corresponding to multiple coordinate axes of the interface coordinate system, and the coordinate information included in each set of contact position information is adjacent among the multiple position information included in the operation data to be recognized. The coordinate information corresponding to the multiple coordinate axes is used to identify the contact position in the interface coordinate system.

6. The method according to claim 2, characterized in that, The method further includes: Obtaining initial operation data, where the initial operation data includes position information, an operation start flag, and an operation end flag. The operation start flag is used to indicate the start of any operation on the target interface, and the operation end flag is used to indicate the end of any operation on the target interface; The obtaining of multiple operation data includes: Determining the position information between any adjacent operation start flag and operation end flag in the initial operation data as an operation data, and obtaining the multiple operation data.

7. The method according to claim 6, characterized in that, The obtaining of the initial operation data includes: Obtaining an operation recording start request and an operation recording end request, where the operation recording start request corresponds to a first moment, and the operation recording end request corresponds to a second moment; Obtaining the operation data generated from the first moment to the second moment as the initial operation data.

8. The method according to claim 1, wherein The determining of the positions respectively passed by the M contact media when performing the target operation according to the positions respectively identified by the M sets of position information includes: Determining the positions respectively identified by the M sets of position information according to the position information respectively included in the M sets of position information. The position order in the positions identified by the first position information set corresponds to the arrangement order of the position information included in the first position information set in the operation data to be recognized. The first position information set is any one of the M sets of position information; Determining the positions respectively identified by the M sets of position information as the positions respectively passed by the M contact media when performing the target operation, and the position order corresponding to the first position information set is the position order of the positions passed by any contact medium when performing the target operation.

9. The method according to claim 8, characterized in that, The target interface has a corresponding interface coordinate system, which is used to locate the contact position between the contact medium and the target interface. The position information is the coordinate information corresponding to any coordinate axis in the interface coordinate system. The determining of the positions respectively identified by the M sets of position information according to the position information respectively included in the M sets of position information includes: Dividing the coordinate information included in the first position information set into multiple groups of coordinate information. Each group of coordinate information includes a target number of adjacent coordinate information in the first position information set. The target number of coordinate information corresponds one-to-one with the coordinate axes of the interface coordinate system, and the target number is the dimension of the interface coordinate system; Determine the positions identified by the multiple groups of coordinate information in the interface coordinate system as the positions identified by the first position information set, and the arrangement order of the multiple groups of coordinate information in the first position information set is the position order corresponding to the first position information set.

10. The method according to claim 8, characterized in that, The method further includes: Cluster the positions identified by the M position information sets according to the position information corresponding to the positions identified by the M position information sets, to obtain a plurality of position clusters, where the clustering is used to make the proximity of the position information between the positions in the same position cluster greater than the proximity of the position information between the positions in different position clusters; The step of determining the positions identified by the M position information sets as the positions passed by the M contact media respectively when performing the target operation includes: Based on the distribution of the positions identified by the M position information sets in the plurality of position clusters being consistent with the distribution in the M position information sets, determine the positions identified by the M position information sets as the positions passed by the M contact media respectively when performing the target operation.

11. The method according to claim 8, wherein The method further includes: Take the M position information sets as second position information sets respectively, and fit a fitting curve corresponding to the second position information set according to the positions identified by the second position information set; The step of determining the positions identified by the M position information sets as the positions passed by the M contact media respectively when performing the target operation includes: Based on the curvature of the fitting curves corresponding to the M position information sets being less than a second preset threshold, determine the positions identified by the M position information sets as the positions passed by the M contact media respectively when performing the target operation.

12. The method according to claim 1, wherein The target interface has a corresponding interface coordinate system, which is used to locate the contact position of the contact medium with the target interface. The multiple pieces of position information include coordinate information corresponding to each coordinate axis of the interface coordinate system. The i-th piece of position information is the coordinate information corresponding to the target coordinate axis in the interface coordinate system, and the target coordinate axis is any coordinate axis in the interface coordinate system. The step of incorporating the i-th piece of position information into the target position information set includes: Determine the proximity between the coordinate information corresponding to the target coordinate axis included in each of the M position information sets and the coordinate information corresponding to the i-th piece of position information; Incorporate the i-th piece of position information into the target position information set with the greatest proximity.

13. The method according to claim 12, characterized in that The step of determining the proximity between the coordinate information corresponding to the target coordinate axis included in each of the M position information sets and the coordinate information corresponding to the i-th piece of position information includes: Take the M position information sets as third position information sets respectively, and determine the target position information that was most recently incorporated into the third position information set, where the target position information is the coordinate information corresponding to the target coordinate axis; Determine the target proximity between the coordinate information corresponding to the target position information and the coordinate information corresponding to the i-th piece of position information; Determine the target proximity as the proximity between the coordinate information corresponding to the target coordinate axis included in the third position information set and the coordinate information corresponding to the i-th position information.

14. The method according to claim 1, wherein The method further includes: Generating an operation instruction corresponding to the target operation according to the positions respectively passed by the M contact media when performing the target operation; Executing the operation instruction, where the operation instruction is used to make the target interface make an interface response when receiving the target operation.

15. The method according to claim 14, wherein The generating an operation instruction corresponding to the target operation according to the positions respectively passed by the M contact media when performing the target operation includes: Based on obtaining an operation reproduction request, generating an operation instruction corresponding to the target operation according to the positions respectively passed by the M contact media when performing the target operation, where the operation reproduction request is used to request to reproduce the target operation for the target interface.

16. The method according to claim 14, characterized in that, The generating an operation instruction corresponding to the target operation according to the positions respectively passed by the M contact media when performing the target operation includes: Respectively taking the M position information sets as the fourth position information set, determining the end point positions and non-end point positions identified by the fourth position information set, where the end point positions are the start position and the end position among the positions passed by the contact media corresponding to the fourth position information set when performing the target operation, and the non-end point positions are the other positions except the end point positions among the positions identified by the fourth position information set; Determining a target position from the non-end point positions, where the target position is a part of the non-end point positions, and the target position satisfies that the similarity between the first curve and the second curve is greater than a third preset threshold, the first curve is a curve fitted based on the positions identified by the fourth position information set, and the second curve is a curve fitted based on the end point positions and the target position; Generating an operation instruction corresponding to the target operation according to the end point positions and the target position.

17. An operation recognition device, characterized in that, The device includes a first obtaining unit, a first determining unit, an induction unit, and a second determining unit: The first obtaining unit is used to obtain operation data to be recognized, where the operation data to be recognized is operation data generated based on a target operation, the target operation is an operation of simultaneously sliding M contact media on a target interface, the operation data to be recognized includes a plurality of position information, the position information is used to identify the positions passed by the M contact media on the target interface, and the arrangement order of the plurality of position information in the operation data to be recognized corresponds to the position order passed by the M contact media when sliding, and M is an integer greater than 1; The first determining unit is used to determine the starting position information among the plurality of position information, where the starting position information is used to identify the M sliding starting points corresponding to the M contact media, and the sliding starting point is the starting point of the contact media sliding on the target interface. The induction unit is configured to respectively induct the N location information based on the arrangement order of the N location information. For the i-th location information among the N location information, the i-th location information is inducted into the target location information set, where the target location information set is the location information set including the location information closest to the i-th location information among the location information included in the M location information sets. The N location information is the location information among the multiple location information except the starting point location information. The M location information sets are constructed based on the starting point location information, and each location information set includes the starting point location information corresponding to any sliding starting point, and the sliding starting points identified by the starting point location information included in different location information sets are different; The second determination unit is configured to determine the locations respectively passed by the M contact media when performing the target operation according to the locations respectively identified by the M location information sets after induction.

18. A computer device, characterized in that, The computer device includes a processor and a memory: The memory is configured to store a computer program and transmit the computer program to the processor; The processor is configured to execute the operation recognition method according to any one of claims 1-16 based on the instructions in the computer program.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store a computer program, and the computer program is used to execute the operation recognition method according to any one of claims 1-16.

20. A computer program product including a computer program, when running on a computer device, causes the computer device to execute the operation recognition method according to any one of claims 1-16.

Images