Interaction control method and system by monitoring operation gesture

Abstract

The embodiment of the invention provides an interaction control method and system by monitoring an operation gesture, and the method can recognize gesture features in an operation movement track after the operation movement track is obtained, and matches a target operation instruction according to the gesture features, thereby executing the target operation instruction. Wherein the gesture features are chain codes generated based on position information such as coordinate values, inter-point distances, inter-point angles, point interval time and track directions of the operation track points. According to the method, the global operation event and the control window can be monitored and captured through the asynchronous processing event, and the gesture features are calculated by processing the operation track points, so that the specific control instruction is simulated, and the problem that the operation efficiency is low due to the fact that the operation mode depends on the control layout of the user interface is solved.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction control technology, and in particular to an interactive control method and system that listens to operation gestures. Background Technology

[0002] As a computer input device, the mouse can detect user input actions and, in conjunction with interactive controls in the user interface, enable human-computer interaction. The computer can detect mouse input actions using event-driven detection. When a user operates the mouse and performs interactive actions such as moving, clicking, or scrolling, the operating system generates corresponding mouse events and passes them to the application via a message queue. The application then combines this with the layout of controls in the currently displayed user interface, generating control commands based on the detected mouse events and the corresponding controls to achieve human-computer interaction.

[0003] For example, the operating system or application can use the message loop of the Windows Application Programming Interface (Windows API) to detect mouse events. The mouse events detected by the Windows API can include mouse movement events (Wm_Mousemove), left mouse button down events (Wm_Lbuttondown), and right mouse button down events (Wm_Rbuttondown). When the input position corresponding to the left mouse button down event is within the response area of ​​the "Close" button, a control command to close the current user interface can be generated. Executing this control command then closes the user interface.

[0004] As can be seen, the above mouse-based operation methods require the combined use of mouse events and user interface controls to achieve the corresponding interactive control functions. This limits the efficiency of the operation method due to the layout of the user interface controls. For example, when refresh, close, and back controls are generally located at the top of the user interface, the user's active area for mouse-based interaction is located in the middle of the interface. This means that when performing operations such as refresh, close, and back, the user needs to move the mouse cursor to the top of the interface to click, which is inconvenient for performing interactive operations. Summary of the Invention

[0005] In view of this, embodiments of this application provide an interactive control method and system that listens to operation gestures to solve the problem of low operation efficiency caused by the operation method depending on the control layout of the user interface.

[0006] According to one aspect of this application, an interactive control method is provided that listens to operation gestures, the method comprising:

[0007] Acquire the operation movement trajectory, which is a set of operation events consisting of at least one operation trajectory point;

[0008] The gesture features in the operation movement trajectory are identified, and the gesture features are chain codes generated based on the position information of the operation trajectory points; the position information includes at least one of the coordinate values ​​of the operation trajectory points, the distance between points, the angle between points, the time interval between points, and the trajectory direction;

[0009] The target operation instruction is matched according to the gesture feature, and the target operation instruction is an operation instruction that is associated with the gesture feature; the association between the operation instruction and the gesture feature is set according to the operation shortcut key of the current user interface and / or the graphical user interface programming data.

[0010] Execute the target operation instruction.

[0011] In some embodiments, obtaining the operation movement trajectory includes:

[0012] Invoke the operation event listening process;

[0013] The operation event listening process monitors the trajectory start point and trajectory end point, where the trajectory start point is an operation event that meets a preset start condition, and the trajectory end point is an operation event that meets a preset end condition.

[0014] When the starting point of the trajectory is detected, the operation events within a preset recording period are recorded to generate an operation event set. The preset recording period is used to characterize a preset time period after the input time at the starting point of the trajectory.

[0015] When the trajectory termination point is detected, the set of operation events is output according to the trajectory termination point to obtain the operation movement trajectory.

[0016] In some embodiments, recognizing gesture features in the operation movement trajectory includes:

[0017] Extract a first operation trajectory point and a second operation trajectory point from the operation movement trajectory, wherein the first operation trajectory point and the second operation trajectory point are two adjacent operation trajectory points;

[0018] Calculate the coordinate difference between the first operation trajectory point and the second operation trajectory point, the coordinate difference including the coordinate difference in the first direction and the coordinate difference in the second direction; the first direction and the second direction are perpendicular.

[0019] Calculate the angle between the points based on the coordinate difference;

[0020] According to the angle range to which the angles between the points belong, chain codes are assigned to the operation movement trajectory to obtain the gesture features.

[0021] In some embodiments, chain codes are assigned to the operation movement trajectory according to the angle range to which the angles between the points belong, in order to obtain the gesture features, including:

[0022] Obtain a preset angle range, which includes a first angle range, a second angle range, a third angle range, and a fourth angle range;

[0023] Extract the endpoint angle values ​​of the angle range;

[0024] By comparing the angle between the points with the angle at the endpoints, the angle range to which the angle between the points belongs is determined;

[0025] If the angle between the points belongs to the first angle range, assign a first chain code to the operation movement trajectory;

[0026] If the angle between the points belongs to the second angle range, assign a second chain code to the operation movement trajectory;

[0027] If the angle between the points belongs to the third angle range, assign a third chain code to the operation movement trajectory;

[0028] If the angle between the points belongs to the fourth angle range, a fourth chain code is assigned to the operation movement trajectory.

[0029] In some embodiments, the method further includes:

[0030] Define a transformation function, which is used to assign chain codes to adjacent operation trajectory points of the operation movement trajectory;

[0031] The adjacent operation trajectory points in the operation movement trajectory are sequentially input into the transformation function to obtain a set of chain codes for the inter-point angles corresponding to multiple adjacent operation trajectory points.

[0032] Generate a chaincode list based on the chaincode set.

[0033] In some embodiments, recognizing gesture features in the operation movement trajectory includes:

[0034] Obtain the number of operation trajectory points in the operation movement trajectory;

[0035] If the number of operation trajectory points is greater than or equal to 2, extract the first operation trajectory point and the second operation trajectory point from the operation movement trajectory.

[0036] If the number of operation trajectory points is less than 2, the operation events will be continuously monitored within a preset recording period.

[0037] In some embodiments, recognizing gesture features in the operation movement trajectory includes:

[0038] Extract the coordinates of the operation trajectory points from the operation movement trajectory;

[0039] Generate a trajectory image based on the coordinates of the operation trajectory points;

[0040] The feature recognition model is invoked, which is a neural network model trained using sample trajectory data; the sample trajectory data includes sample trajectory images generated based on sample trajectory chain codes, and feature classification labels of the sample trajectory images;

[0041] The trajectory image is input into the feature recognition model to obtain the classification result output by the feature recognition model. The classification result is used to characterize the classification probability that the operation movement trajectory contains the sample trajectory chain code corresponding to the feature classification label.

[0042] The gesture feature is determined based on the classification result, and the gesture feature is the sample trajectory chain code with the highest classification probability.

[0043] In some embodiments, matching a target operation command based on the gesture features includes:

[0044] Gesture recognition information is determined based on the gesture features;

[0045] A preset instruction mapping table is obtained, which is used to store the association between operation instructions and the gesture recognition information; the instruction mapping table is established based on standard gesture recognition information and operation instruction functions specified for the standard gesture features; the standard gesture recognition information is generated by statistically analyzing the common gesture features corresponding to multiple pre-recorded standard movement trajectories.

[0046] Based on the gesture recognition information, the target operation instruction is matched in the instruction mapping table.

[0047] In some embodiments, executing the target operation instruction includes:

[0048] Define the action function to be executed;

[0049] The gesture recognition information corresponding to the target operation command is input into the execution action function so that the execution action function can query the target shortcut key based on the gesture recognition information;

[0050] The target window handle is obtained according to the operation type of the target shortcut key, and the target window handle is used to represent the foreground window in the current user interface that can respond to the target shortcut key;

[0051] By executing the target shortcut key, an execution message is sent to the foreground window corresponding to the target window handle, so that the foreground window responds to the target operation command.

[0052] According to another aspect of this application, an interactive control system for listening to operational gestures is provided, the system comprising:

[0053] The trajectory acquisition module is used to acquire the operation movement trajectory, which is a set of operation events consisting of at least one operation trajectory point;

[0054] The feature recognition module is used to recognize gesture features in the operation movement trajectory. The gesture features are chain codes generated based on the position information of the operation trajectory points. The position information includes at least one of the coordinate values ​​of the operation trajectory points, the distance between points, the angle between points, the time interval between points, and the trajectory direction.

[0055] The instruction matching module is used to match a target operation instruction based on the gesture feature, wherein the target operation instruction is an operation instruction that is associated with the gesture feature; the association between the operation instruction and the gesture feature is set according to the operation shortcut keys of the current user interface and / or the graphical user interface programming data.

[0056] The instruction execution module is used to execute the target operation instruction.

[0057] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described interactive control method by listening to operation gestures.

[0058] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described interactive control method by listening to operation gestures.

[0059] By employing the above technical solutions, embodiments of this application provide an interactive control method and system that listens to operation gestures. The method, after acquiring an operation movement trajectory, identifies gesture features within the trajectory and matches target operation commands based on these features, thereby executing the target operation command. The gesture features are chain codes generated based on positional information such as the coordinates of points on the operation trajectory, distances between points, angles between points, time intervals between points, and trajectory direction. The method can asynchronously process event listeners to capture global operation events and control windows, and calculate gesture features by processing operation trajectory points to simulate specific control commands, thus solving the problem of low operation efficiency caused by the operation method depending on the control layout of the user interface.

[0060] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0061] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0062] Figure 1 This is a schematic diagram of the interactive control system structure provided in an embodiment of this application;

[0063] Figure 2 A schematic flowchart of an interactive control method for listening to gestures provided in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of the process for generating the movement trajectory provided in an embodiment of this application;

[0065] Figure 4 This application provides a schematic diagram of the process for assigning chaincodes to the operation movement trajectory in an embodiment of the present application.

[0066] Figure 5 This is a schematic diagram of the gesture feature recognition process in the movement trajectory provided in the embodiments of this application;

[0067] Figure 6 This is a schematic diagram of the gesture input interaction interface provided in the embodiments of this application;

[0068] Figure 7 This is a schematic diagram of the interactive control system structure that listens to operation gestures, as provided in an embodiment of this application. Detailed Implementation

[0069] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0070] In some embodiments of this application, interactive control refers to a human-computer interaction process that controls and manages system behavior through interaction between the user and the system. Interactive control is a two-way information exchange process, such as... Figure 1 As shown, users send control commands or input data to the system using input devices such as keyboards, mice, and touchscreens. The system then responds to the user's input and provides feedback on the interaction results to the user through output devices such as displays and speakers.

[0071] Electronic devices with data processing capabilities and user interfaces can process user-input control commands and data through interactive control methods, transforming them into control commands that the system can recognize and execute. Based on different interactive control methods, the human-computer interaction modes supported by electronic devices can be increased.

[0072] In some embodiments, when the electronic device is a computer, users can interact with the computer using input devices such as a mouse. The mouse detects user input actions; when a user operates the mouse and performs interactive actions such as moving, clicking, or scrolling, the mouse sends the corresponding electrical signals to the computer. The computer's operating system can perform input action detection based on event-driven detection, extracting mouse operation events from the electrical signals collected by the mouse. These operation events are then passed to the application program via a message queue. The application program, combined with the control layout in the currently displayed user interface, generates control commands based on the detected mouse events and the corresponding control positions to achieve human-computer interaction.

[0073] For example, a computer's operating system or applications can use the message loop of the Windows Application Programming Interface (Windows API) to detect mouse events. The mouse events detected by the Windows API can include mouse movement events (Wm_Mousemove), left mouse button down events (Wm_Lbuttondown), and right mouse button down events (Wm_Rbuttondown). When the input position corresponding to the left mouse button down event is within the response area of ​​the "Close" button, a control command can be generated to close the current user interface. Executing this control command then closes the user interface.

[0074] However, the mouse-based operation method provided in the above embodiments requires the combined use of mouse events and user interface controls to achieve the corresponding interactive control functions. This limits the operation method to the layout of user interface controls, reducing operational efficiency. For example, when refresh, close, and back controls are generally located at the top of the user interface, while the user's active area for mouse-based interactive operations is located in the middle of the interface, the user needs to move the mouse cursor to the top of the interface to click when performing refresh, close, or back operations, which is inconvenient for interactive operations.

[0075] To address the low operational efficiency caused by the reliance on user interface control layout, some embodiments of this application provide an interactive control method that listens to user gestures. This method can be applied to electronic devices with data processing capabilities and human-computer interaction support. These electronic devices include, but are not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control machines. In some embodiments of this application, the specific process of executing the interactive control method by listening to user gestures on an electronic device is described using a display device with display functionality. It should be understood that the method can also be applied to other types of execution entities, which will not be shown individually in this application.

[0076] The method described above achieves event execution by capturing movement trajectories, recognizing gestures, and calculating gesture features. It can be implemented using simple gestures, such as closing windows, refreshing, and going back, providing users with a convenient operating method. Figure 2 As shown, the method includes:

[0077] S101, Obtain the operation movement trajectory.

[0078] To enable interactive control via gesture monitoring, the display device can asynchronously listen for user input actions while displaying the user interface, and obtain the operation movement trajectory based on the monitored actions. This operation movement trajectory is a set of operation events consisting of at least one operation trajectory point.

[0079] Depending on the interaction methods supported by the display device, the form of the operation movement trajectory acquired by the display device will also differ. For example, when the display device is a computer, and the user performs interactive operations using a mouse, the operation movement trajectory is the trajectory of the mouse movement. Therefore, the operation movement trajectory is a set of mouse events composed of mouse trajectory points. That is, the operation movement trajectory for mouse interactive operations can be represented as:

[0080] path={P1(Wm_Mousemove,x1,y1),P2(Wm_Mousemove,x2,y2),...,Pn(Wm_Mousemove,xn,yn)};

[0081] Where path represents the operation movement trajectory; P1, P2, ..., Pn represent the 1st, 2nd, ..., nth operation trajectory points respectively; Wm_Mousemove indicates that the mouse event type corresponding to the operation trajectory point is a mouse movement event; (x1, y1), (x2, y2), ..., (xn, yn) represent the position coordinates corresponding to the 1st, 2nd, ..., nth operation trajectory points.

[0082] When the display device is a mobile terminal device that supports touch interaction, and the user performs interactive operations via touch, the operation movement trajectory is the touch trajectory. That is, the operation movement trajectory is a set of touch events composed of touch trajectory points. Therefore, the operation movement trajectory for touch interaction operations can be represented as:

[0083] path={P1(touch_down,x1,y1),P2(touch_move,x1,y1),...,Pn(touch_up,xn,yn)};

[0084] Where path represents the operation movement trajectory; P1, P2, ..., Pn represent the 1st, 2nd, ..., nth operation trajectory points respectively; touch_down indicates that the touch event type corresponding to the operation trajectory point (P1) is a touch down event; touch_move indicates that the touch event type corresponding to the operation trajectory point (P2) is a touch move event; touch_up indicates that the touch event type corresponding to the operation trajectory point (Pn) is a touch up event; (x1, y1), (x2, y2), ..., (xn, yn) represent the position coordinates corresponding to the 1st, 2nd, ..., nth operation trajectory points.

[0085] An operation movement trajectory can include multiple operation trajectory points. Depending on the desired interactive control method, the types of operation events between these points can also differ. Taking a mouse as an example, mouse events can include mouse movement events, mouse press events, mouse release events, mouse double-click events, mouse wheel events, and mouse drag events. To achieve more complex interactive operations, the operation trajectory points can have two or more mouse event types. For example, a mouse movement trajectory can include the following:

[0086] path={P1(Wm_Lbuttondown, x1, y1), P2(Wm_Mousemove, x2, y2), P3(Wm_Mousemove, x3, y3), ..., Pn-1(Wm_Mousemove, xn-1, yn-1), Pn(Wm_Wm_Lbuttonup, xn, yn)};

[0087] The mouse movement trajectory represents the continuous dragging interaction process in which the user presses the left mouse button at (x1, y1), moves the mouse (or mouse cursor) from (x2, y2) to (xn-1, yn-1), and releases the left mouse button at (xn, yn).

[0088] like Figure 3 As shown, in order to acquire the operation movement trajectory, in some embodiments, the display device may first invoke an operation event listening process when acquiring the operation movement trajectory, and listen for the trajectory start point and trajectory end point through the operation event listening process. The trajectory start point is an operation event that meets a preset start condition; the trajectory end point is an operation event that meets a preset end condition. The preset start condition and preset end condition can be set according to the specific interactive control method.

[0089] For example, when monitoring mouse dragging, the default start condition is a left mouse button press event, i.e., the Wm_Lbuttondown event; the default end condition is a left mouse button release event after the left mouse button press event is detected, i.e., the Wm_Lbuttonup event. After the display device invokes the operation event monitoring process, it can listen for signal characteristics representing a mouse press event in the electrical signals transmitted by the mouse. If the mouse transmits signals containing these characteristics, the starting point of the monitored trajectory can be determined.

[0090] Upon detecting the trajectory start point, the display device can record operation events within a preset recording period to generate an operation event set. The preset recording period represents a preset time interval after the input time at the trajectory start point. The preset recording period can be a fixed value pre-set according to user interaction habits. For example, if the preset recording period is 10 seconds, then when the user detects an operation event including a Wm_Lbuttondown event via mouse input, the display device can record all Wm_Mousemove events within 10 seconds after the Wm_Lbuttondown event is input to form an operation event set.

[0091] The preset recording period can also be continuously updated based on the input process of operation events. For example, if the preset recording period is te+10s, where te represents the input time of the last operation event, then when the user's mouse input includes a Wm_Lbuttondown event, the display device can record the Wm_Mousemove event input within 10 seconds after the Wm_Lbuttondown event, and after recording the Wm_Mousemove event, record the operation events input within 10 seconds after the Wm_Mousemove event, forming a set of operation events by continuously updating the recording period.

[0092] By continuously recording operation events to obtain a set of operation events, the display device can listen for operation events that meet preset termination conditions, i.e., trajectory termination points. Upon detecting a trajectory termination point, the device outputs the set of operation events according to that termination point to obtain the operation movement trajectory.

[0093] For example, when the display device listens to the Wm_Lbuttonup event (mouse release event) during operation event recording, since the Wm_Lbuttonup event meets the preset termination condition, the operation event corresponding to the trajectory termination point can be added to the operation event set and output to obtain the operation movement trajectory, i.e.:

[0094] path={P1(Wm_Lbuttondown, x1, y1), P2(Wm_Mousemove, x2, y2), P3(Wm_Mousemove, x3, y3), ..., Pn-1(Wm_Mousemove, xn-1, yn-1), Pn(Wm_Wm_Lbuttonup, xn, yn)};

[0095] The obtained operation movement trajectory can be analyzed as independent data to realize interactive control methods by listening to mouse gestures.

[0096] S102. Identify the gesture features in the operation movement trajectory.

[0097] After acquiring the operation movement trajectory, the display device can also perform feature recognition on the operation movement trajectory, that is, recognize the gesture features in the operation movement trajectory. The gesture features are chain codes generated based on the position information of the operation trajectory points. Chain codes are a type of encoding used to describe path direction and can be used to recognize operation input patterns.

[0098] Gesture features can be obtained by recognizing the positional information of points on the operation trajectory. This positional information may include at least one of the following: coordinates of the operation trajectory points, distances between points, angles between points, time intervals between points, and trajectory direction. The coordinates can be obtained directly by reading the coordinates of the operation trajectory points.

[0099] The distance between points can be obtained by calculating the distance between two operation trajectory points. For example, for operation trajectory points P1(x1, y1) and P2(x2, y2) in the operation movement trajectory, the distance between points can be calculated according to the following formula:

[0100]

[0101] Where L represents the distance between points; (x1, y1) and (x2, y2) represent two operation trajectory points.

[0102] The angle between points can also be calculated using the coordinates of two operation trajectory points. For example, for operation trajectory points P1(x1, y1) and P2(x2, y2) in an operation movement trajectory, the coordinate differences between the two operation trajectory points on the x-axis and y-axis can be calculated, i.e., Δx = |x1 - x2|, Δy = |y1 - y2|. Then, the angle between points can be calculated based on the coordinate difference, such as θ = arctan(Δy / Δx).

[0103] The trajectory direction can be determined by statistically analyzing the angles between multiple adjacent points in the operation's movement trajectory. That is, the trajectory direction can be a multi-dimensional vector formed by arranging and combining the angles between points according to the operation trajectory points. For example, the trajectory direction can be represented as:

[0104] Td=(θ 1，2 θ 2，3 , ..., θ n-1，n );

[0105] Where Td represents the trajectory direction; θ 1，2 θ represents the angle between points P1 and P2 on the operational trajectory. 2，3 θ represents the angle between points P2 and P3 on the operational trajectory. n-1，n This represents the angle between the operation trajectory points Pn-1 and Pn.

[0106] The point interval time can be obtained by recording the input time of each operation trajectory point in the operation movement trajectory and calculating it by comparing the input times of two operation trajectory points. That is, when acquiring the operation movement trajectory, the input time of each operation trajectory point can be recorded to form an operation movement trajectory that includes the input time.

[0107] path={P1(Wm_Lbuttondown,x1,y1,t1),P2(Wm_Mousemove,x2,y2,t2),...,Pn(Wm_Wm_Lbuttonup,xn,yn,tn)};

[0108] Where t1, t2, ..., tn represent the input times of operation trajectory points P1, P2, ..., Pn, respectively. Then, for two operation trajectory points P1(x1, y1, t1) and P2(x2, y2, t2), the point interval time can be obtained by calculating Δt = |t1 - t2|.

[0109] Depending on the specific interactive control requirements, display devices can recognize gesture features based on different types of positional information. For example, when the gesture feature is an arc-shaped gesture, the display device needs to determine whether the operation movement trajectory is an arc shape based on positional information such as coordinate values, angles between points, and trajectory direction.

[0110] To identify gesture features in the operation movement trajectory, the display device can run a preset feature extraction process. The feature extraction process executes feature extraction steps according to pre-written program code; that is, in some embodiments, the display device can first define a conversion function when performing gesture feature recognition in the operation movement trajectory. This conversion function is used to assign chain codes to adjacent operation trajectory points. For example, the display device can define a conversion function using "def path_to_chain_code(path)", that is, define a conversion function named "path_to_chain_code", which can accept a parameter "path".

[0111] After defining the conversion function, the display device can also perform a path length check, i.e., as follows: Figure 4 As shown, in some embodiments, when recognizing gesture features in the operation movement trajectory, the display device can first obtain the number of operation trajectory points in the operation movement trajectory. For example, a length checking procedure can be defined using the "len(path)" command to check the number of operation trajectory points len(path) in the operation movement trajectory path. After obtaining the number of operation trajectory points in the operation movement trajectory, the display device can compare the number of operation trajectory points with a preset trajectory point number threshold.

[0112] Since at least two operation trajectory points are required to determine an operation gesture, in some embodiments, the preset threshold for the number of trajectory points can be set to 2. If the number of operation trajectory points is less than 2, it means that the current operation movement trajectory cannot accurately determine the gesture type. Therefore, continuous monitoring can be performed, that is, continuous monitoring of operation events within a preset recording period until the number of operation trajectory points in the operation movement trajectory is greater than or equal to 2. For example, the display device can determine the number of operation trajectory points by defining the execution command "if len(path)<2; return[]", and execute the return action when len(path)<2, so as to continuously monitor operation events within a preset recording period.

[0113] If the number of operation trajectory points is greater than or equal to 2, it indicates that the current operation movement trajectory can identify operation trajectory points. Therefore, the first operation trajectory point and the second operation trajectory point can be extracted from the operation movement trajectory. The first operation trajectory point and the second operation trajectory point are two adjacent operation trajectory points.

[0114] For example, when two adjacent operation trajectory points in the operation movement trajectory path are the first operation trajectory point P1(x1, y1) and the second operation trajectory point P2(x2, y2), the display device can extract the first operation trajectory point P1(x1, y1) and the second operation trajectory point P2(x2, y2) from the operation movement trajectory path by running "for i in range(1, len(path))".

[0115] Next, calculate the coordinate difference between the first operation trajectory point and the second operation trajectory point. This coordinate difference includes the coordinate difference in the first direction and the coordinate difference in the second direction. The first direction and the second direction are two mutually perpendicular reference coordinate directions; that is, when the first direction is the x-axis, the second direction is the y-axis; and when the first direction is the y-axis, the second direction is the x-axis.

[0116] For example, the display device can calculate the difference between the current point (second operation trajectory point P2) and the previous operation trajectory point (first operation trajectory point P1) in the x-axis and y-axis directions by using the method of "dx = path[i][0] - path[i-1][0]; dy = path[i][1] - path[i-1][1]".

[0117] After calculating the coordinate difference, the display device can calculate the angle between the points based on the coordinate difference. For example, by running "angle = np.arctan2(dy, dx) * 180 / np.pi", the np.arctan2 function is used to calculate the angle from the x-axis to the point (dx, dy) in radians, and then it is converted to degrees.

[0118] Then, according to the angle range to which the angles between the points belong, chain codes are assigned to the operation movement trajectory to obtain the gesture features. For example... Figure 5 As shown, in some embodiments, in order to obtain the gesture features, the display device can first acquire a preset angle range. Multiple angle ranges can be preset according to a set angle interval size and a total angle value; that is, the angle ranges may include a first angle range, a second angle range, a third angle range, and a fourth angle range.

[0119] For example, if the total angle value is 360°, meaning the sum of the first angle range, the second angle range, the third angle range, and the fourth angle range is 360°, then each angle interval is 90°. Therefore, the first angle range can be set to [-45°, 45°); the second angle range to [45°, 135°); the third angle range to [135°, 180°] and [-180°, -135°]; and the fourth angle range to [-135°, -45°].

[0120] Next, extract the endpoint angle values ​​of the angle ranges. These endpoint angle values ​​are the minimum and maximum angle values ​​for each angle interval. For example, for the first angle range [-45°, 45°), its minimum angle value is -45° and its maximum angle value is 45°. Therefore, the endpoint angle values ​​of the first angle range can be extracted as -45° and 45°. Similarly, the endpoint angle values ​​of the second angle range are 45° and 135°; the endpoint angle values ​​of the third angle range are 135° and -135°; and the endpoint angle values ​​of the fourth angle range are -135° and -45°.

[0121] After extracting the endpoint angle values ​​of the angle range, the display device can determine the angle range to which the inter-point angle belongs by comparing the inter-point angle with the endpoint angle values. For example, when the inter-point angle θ between operation trajectory points P1 and P2... 1，2 When the angle is 12°, due to the angle θ between these points... 1，2 Satisfying -45°≤θ 1，2 <45°, therefore the angle θ between these points can be determined. 1，2 It belongs to the first angle range.

[0122] After determining the angle range to which the inter-point angles belong, the display device assigns chain codes to the operation movement trajectory based on the assigned angle range. That is, if the inter-point angles belong to the first angle range, a first chain code is assigned to the operation movement trajectory; if the inter-point angles belong to the second angle range, a second chain code is assigned to the operation movement trajectory; if the inter-point angles belong to the third angle range, a third chain code is assigned to the operation movement trajectory; and if the inter-point angles belong to the fourth angle range, a fourth chain code is assigned to the operation movement trajectory.

[0123] For example, a display device can assign a chain code to the movement trajectory corresponding to the angle by executing "if -45 ≤ angle < 45: chain.append(0); elif 45 ≤ angle < 135: chain.append(1); elif 135 ≤ angle ≤ 180 or -180 ≤ angle < -135: chain.append(2); else: chain.append(3)". This means that the direction of the path is encoded as an integer between 0 and 3 based on the calculated angle: if the angle is between -45 and 45 degrees, the chain code is chain.append(0), representing the first chain code. If the angle is between 45 and 135 degrees, the chain code is chain.append(1), representing the second chain code. If the angle is between 135 and 180 degrees or between -180 and -135 degrees, the chain code is chain.append(2), representing the third chain code. In other cases (i.e., between -135 and -45 degrees), the chain code is chain.append(3), representing the fourth chain code.

[0124] The display device can obtain a set of chain codes for the angles between multiple adjacent operation trajectory points by sequentially inputting adjacent operation trajectory points into the conversion function. For example, during the process of recognizing gesture features in the operation trajectory, if it is determined that len(path) ≥ 2, the chain code list can be initialized by defining the command "chain = []", that is, an empty list "chain" is initialized to store the chain codes of the path. Then, the path points are traversed by the command "for i in range(1, len(path))". Each point in the path is traversed using a for loop, starting from the second point (index 1), and the difference between adjacent points is calculated, the angle is calculated and converted into chain codes according to the method provided in the above embodiment. Then, the chain codes are allocated according to the angle range.

[0125] After traversing each operation trajectory point in the path, the chain codes corresponding to any two adjacent operation trajectory points can be obtained and combined to form the entire chain code set. A chain code list can then be generated based on this chain code set and returned. For example, after executing a for loop to traverse each point in the path, the chain code allocation results corresponding to adjacent points can be recorded in "chain = []" according to the traversal process. After completing the traversal, the "return chain" step is executed to return a list containing path direction codes.

[0126] As can be seen, through the gesture feature recognition method provided in the above embodiments, the display device can make a judgment based on the coordinates, angles, distances and other positional information of adjacent operation trajectory points in the operation movement trajectory, thereby converting a two-dimensional path into a chain code sequence to describe the direction of the path, simplifying the representation and processing of the path, and recognizing gesture features based on the path direction representation.

[0127] The display device can also recognize gesture features in the operation movement trajectory based on an artificial intelligence model. Specifically, in some embodiments, the display device can train the feature recognition model before recognizing the gesture features in the operation movement trajectory; that is, it can train a neural network model using sample trajectory data. The sample trajectory data includes sample trajectory images generated based on sample trajectory chaincode, and feature classification labels for the sample trajectory images.

[0128] For example, a display device can collect key points of operation trajectories and gesture history data through an application interface. That is, after a user controls the display device to enter key point recording mode, they can perform specific gestures in the interactive interface under the recording model. The display device can save the current hand key points and class IDs as the user interacts. Then, through data preprocessing, the collected raw key point data is converted into a standardized representation, i.e., sample trajectory data is obtained. For example, in the dynamic gesture recognition process, based on the coordinates of points tracked by the user in the gesture-related data, these coordinates represent the movement characteristics of the user's fingers or input devices such as a mouse. Therefore, the display device can acquire the specific gestures performed in the interactive interface under the recording model, and determine the conversion of the trajectory chain code into a sample trajectory image according to the trajectory chain code determination method provided in the above embodiments. Then, feature classification labels are set for the trajectory image according to the gesture function specified in the interactive interface.

[0129] After collecting sample trajectory data, the display device can use this data to train a neural network model. For example, the sample trajectory data can be input into the trained neural network model, which will then compile the model. During model compilation, the classification cross-entropy loss function, the Adam optimizer, and accuracy can be used as evaluation metrics. Training hyperparameters include batch size, number of iterations, and early stopping to prevent overfitting. After training, the model is evaluated on a validation set. Once the evaluation is successful, the trained model can be converted to TFLite format for deployment.

[0130] After obtaining a feature recognition model through training, the display device can recognize gesture features based on the feature recognition model. Specifically, when recognizing gesture features in the operation movement trajectory, the coordinates of operation trajectory points can be extracted from the operation movement trajectory first, and then a trajectory image can be generated based on the coordinates of the operation trajectory points. By calling the feature recognition model and inputting the trajectory image into the feature recognition model, a classification result output by the feature recognition model is obtained. This classification result characterizes the probability that the operation movement trajectory contains the sample trajectory chain code corresponding to the feature classification label. Therefore, the gesture feature is determined based on the classification result; that is, the sample trajectory chain code with the highest classification probability is identified as the gesture feature.

[0131] S103. Match the target operation command according to the gesture features.

[0132] After obtaining gesture features through gesture feature recognition, the display device can match the recognized gesture features with operation commands to obtain target operation commands, wherein the target operation commands are operation commands that are associated with the gesture features. The association between the operation commands and the gesture features can be set according to the operation shortcut keys of the current user interface and / or graphical user interface programming data.

[0133] The association between operation commands and gesture features can be represented in the form of a mapping table, that is, the display device can establish a command mapping table based on operation commands and gesture features. Specifically, the command mapping table stores the association between operation commands and the gesture recognition information; the command mapping table is established based on standard gesture recognition information and the operation command function specified for the standard gesture features; the standard gesture recognition information is generated by statistically analyzing the common gesture features corresponding to multiple pre-recorded standard movement trajectories.

[0134] The display device can pre-record multiple standard movement trajectories through a gesture input interface. The gesture input interface can include a gesture input area and controls for specifying operation commands. For example, ... Figure 6As shown, the gesture input interface can include a drawing area and a drop-down list box for specifying operation commands, used to input standard movement trajectories. After the user controls the display device to display the gesture input interface, they can perform multiple trajectory drawing operations in the drawing area to input a standard movement trajectory.

[0135] The display device can extract common gesture features corresponding to multiple standard movement trajectories. Specifically, based on the positional information of operation trajectory points within the standard movement trajectory, after determining the gesture features, the device compares the gesture features corresponding to multiple standard movement trajectories to identify common gesture features. While inputting multiple standard movement trajectories, the user can also specify the target operation command corresponding to multiple standard movement trajectories through an operation command specification control.

[0136] For example, after a user selects the "refresh" option from the drop-down list in the gesture input interface, they can draw arc trajectories multiple times in the drawing area. The display device can then extract common features from these multiple input arc trajectories to identify operation trajectory points that share the common feature of an arc shape. A correlation is then established between these common gesture features and the target operation command, and this correlation is stored in a table to construct a command mapping table.

[0137] In order to match the target operation command, in some embodiments, when the display device executes the target operation command matched according to the gesture features, the display device can determine the gesture recognition information according to the gesture features, then obtain a preset command mapping table, and match the target operation command in the command mapping table based on the gesture recognition information.

[0138] For example, when a gesture feature identifies multiple operation trajectory points forming a circle, the gesture recognition information corresponding to this feature can be determined as "circle," indicating that the gesture name is circular. Then, based on the gesture recognition information "circle," an instruction match is performed in the instruction mapping table to determine that the target operation instruction associated with "circle" is "Refresh," meaning the target operation instruction is a UI refresh instruction. Similarly, when the gesture feature is a cross shape, the gesture recognition information can be determined as "x_shape," and the target operation instruction "wm_Close" used to close the user interface can be matched. When the gesture feature is a left arrow gesture, the gesture recognition information can be determined as "left_arrow," and the operation instruction "Back" used to perform the back operation can be matched.

[0139] S104. Execute the target operation instruction.

[0140] After obtaining the target operation instruction, it can be executed, thereby performing interactive control by listening to the operation gesture. In some embodiments, when the display device executes the target operation instruction, it can first define an execution action function. Then, the gesture recognition information corresponding to the target operation instruction is input into the execution action function, so that the execution action function can query the target shortcut key based on the gesture recognition information.

[0141] For example, a display device defines an action function named "execute_action" by executing "def execute_action(gesture_name)". This action function can accept a parameter representing the gesture name, i.e., "gesture_name". The target shortcut key can then be queried based on the action function, and an interactive action can be executed based on the target shortcut key: "if gesture_name = "left_arrow":keyboard.press_and_release("alt+left"); print("Action:Back")". By executing this control command, when the user's input movement trajectory contains the left arrow gesture feature, based on the name "left_arrow" corresponding to the gesture feature, the "alt+left" key combination is simulated to be pressed and released, and the action "Action:Back" is executed. This is used to perform the "back" operation in user interfaces such as browsers.

[0142] Similarly, when determining that the user's input movement trajectory contains a circular gesture feature, the command "elifgesture_name = "circle":keyboard.press_and_release("f5"); print("Action:Refresh")" can be executed to simulate pressing and releasing the F5 key based on the gesture name "circle", thereby executing the refresh action "Action:Refresh" to refresh the currently displayed user interface.

[0143] In some embodiments, to execute a specific target operation instruction, the display device may also obtain a target window handle based on the operation type of the target shortcut key when querying and obtaining the target shortcut key. The target window handle represents the foreground window in the current user interface that can respond to the target shortcut key. By executing the target shortcut key, an execution message is sent to the foreground window corresponding to the target window handle, causing the foreground window to respond to the target operation instruction.

[0144] For example, when determining that the user's input movement trajectory contains gesture features with intersecting trajectories, the display device can execute "elifgesture_name = "x_shape": hwnd = win32gui.GetForegroundWindow(); win32gui.PostMessage(hwnd, win32con.WM_CLOSE, 0, 0)". That is, based on the gesture features with intersecting trajectories, the gesture recognition information is determined to be "x_shape". This allows the device to obtain the handle of the currently foreground window and send a close message to that window, thereby closing the foreground window in the current user interface.

[0145] In some embodiments, as a specific implementation of the interactive control method for listening to operation gestures described in the above embodiments, some embodiments of this application also provide an interactive control system for listening to operation gestures, such as... Figure 7 As shown, the system includes:

[0146] The trajectory acquisition module is used to acquire the operation movement trajectory, which is a set of operation events consisting of at least one operation trajectory point;

[0147] The feature recognition module is used to recognize gesture features in the operation movement trajectory. The gesture features are chain codes generated based on the position information of the operation trajectory points. The position information includes at least one of the coordinate values ​​of the operation trajectory points, the distance between points, the angle between points, the time interval between points, and the trajectory direction.

[0148] The instruction matching module is used to match a target operation instruction based on the gesture feature, wherein the target operation instruction is an operation instruction that is associated with the gesture feature; the association between the operation instruction and the gesture feature is set according to the operation shortcut keys of the current user interface and / or the graphical user interface programming data.

[0149] The instruction execution module is used to execute the target operation instruction.

[0150] It should be noted that other corresponding descriptions of the functional units involved in the interactive control system for listening to operation gestures provided in the embodiments of this application can be found in the corresponding descriptions in the interactive control method for listening to operation gestures provided in the above embodiments, and will not be repeated here.

[0151] By implementing the technical solutions provided in the above embodiments, an interactive control system for listening to operation gestures provided in this application may include a trajectory acquisition module, a feature recognition module, an instruction matching module, and an instruction execution module. After the trajectory acquisition module acquires the operation movement trajectory, the feature recognition module can identify the gesture features in the operation movement trajectory, and the instruction matching module can match the target operation instruction according to the gesture features, thereby allowing the instruction execution module to execute the target operation instruction. The gesture features are chain codes generated based on positional information such as the coordinates of the operation trajectory points, the distance between points, the angle between points, the time interval between points, and the trajectory direction. The system can asynchronously process event listening to capture global operation events and control windows, and calculate gesture features by processing operation trajectory points, thereby simulating specific control instructions to solve the problem of low operation efficiency caused by the operation method depending on the control layout of the user interface.

[0152] This application also provides a computer device, specifically a personal computer, server, network device, etc. The computer device includes a bus, processor, memory, and communication interface, and may also include input / output interfaces and a display device. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores location information. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0153] Those skilled in the art will understand that the structure of the computer device described above is only a partial structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.

[0154] In one embodiment, a computer-readable storage medium is also provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0155] In one embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0156] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0157] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0158] Any references to memory, database, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc.

[0159] Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0160] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An interactive control method that listens to user gestures, characterized in that, The method includes: Acquire the operation movement trajectory, which is a set of operation events consisting of at least one operation trajectory point; The gesture features in the operation movement trajectory are identified, and the gesture features are chain codes generated based on the position information of the operation trajectory points; the position information includes at least one of the coordinate values ​​of the operation trajectory points, the distance between points, the angle between points, the time interval between points, and the trajectory direction; The target operation instruction is matched according to the gesture feature, and the target operation instruction is an operation instruction that is associated with the gesture feature; the association between the operation instruction and the gesture feature is set according to the operation shortcut key of the current user interface and / or the graphical user interface programming data. Execute the target operation instruction.

2. The method according to claim 1, characterized in that, Obtain the operation movement trajectory, including: Invoke the operation event listening process; The operation event listening process monitors the trajectory start point and trajectory end point, where the trajectory start point is an operation event that meets a preset start condition, and the trajectory end point is an operation event that meets a preset end condition. When the starting point of the trajectory is detected, the operation events within a preset recording period are recorded to generate an operation event set. The preset recording period is used to characterize a preset time period after the input time at the starting point of the trajectory. When the trajectory termination point is detected, the set of operation events is output according to the trajectory termination point to obtain the operation movement trajectory.

3. The method according to claim 1, characterized in that, Identifying gesture features in the movement trajectory of the operation includes: Extract a first operation trajectory point and a second operation trajectory point from the operation movement trajectory, wherein the first operation trajectory point and the second operation trajectory point are two adjacent operation trajectory points; Calculate the coordinate difference between the first operation trajectory point and the second operation trajectory point, the coordinate difference including the coordinate difference in the first direction and the coordinate difference in the second direction; the first direction and the second direction are perpendicular. Calculate the angle between the points based on the coordinate difference; According to the angle range to which the angles between the points belong, chain codes are assigned to the operation movement trajectory to obtain the gesture features.

4. The method according to claim 3, characterized in that, According to the angle range to which the angles between the points belong, chain codes are assigned to the operation movement trajectory to obtain the gesture features, including: Obtain a preset angle range, which includes a first angle range, a second angle range, a third angle range, and a fourth angle range; Extract the endpoint angle values ​​of the angle range; By comparing the angle between the points with the angle at the endpoints, the angle range to which the angle between the points belongs is determined; If the angle between the points belongs to the first angle range, assign a first chain code to the operation movement trajectory; If the angle between the points belongs to the second angle range, assign a second chain code to the operation movement trajectory; If the angle between the points belongs to the third angle range, assign a third chain code to the operation movement trajectory; If the angle between the points belongs to the fourth angle range, a fourth chain code is assigned to the operation movement trajectory.

5. The method according to claim 3, characterized in that, The method further includes: Define a transformation function, which is used to assign chain codes to adjacent operation trajectory points of the operation movement trajectory; The adjacent operation trajectory points in the operation movement trajectory are sequentially input into the transformation function to obtain a set of chain codes for the inter-point angles corresponding to multiple adjacent operation trajectory points. Generate a chaincode list based on the chaincode set.

6. The method according to claim 3, characterized in that, Identifying gesture features in the movement trajectory of the operation includes: Obtain the number of operation trajectory points in the operation movement trajectory; If the number of operation trajectory points is greater than or equal to 2, extract the first operation trajectory point and the second operation trajectory point from the operation movement trajectory. If the number of operation trajectory points is less than 2, the operation events will be continuously monitored within a preset recording period.

7. The method according to claim 1, characterized in that, Identifying gesture features in the movement trajectory of the operation includes: Extract the coordinates of the operation trajectory points from the operation movement trajectory; Generate a trajectory image based on the coordinates of the operation trajectory points; The feature recognition model is invoked, which is a neural network model trained using sample trajectory data; the sample trajectory data includes sample trajectory images generated based on sample trajectory chain codes, and feature classification labels of the sample trajectory images; The trajectory image is input into the feature recognition model to obtain the classification result output by the feature recognition model. The classification result is used to characterize the classification probability that the operation movement trajectory contains the sample trajectory chain code corresponding to the feature classification label. The gesture feature is determined based on the classification result, and the gesture feature is the sample trajectory chain code with the highest classification probability.

8. The method according to claim 1, characterized in that, Matching target operation commands based on the gesture features includes: Gesture recognition information is determined based on the gesture features; A preset instruction mapping table is obtained, which is used to store the association between operation instructions and the gesture recognition information; the instruction mapping table is established based on standard gesture recognition information and operation instruction functions specified for the standard gesture features; the standard gesture recognition information is generated by statistically analyzing the common gesture features corresponding to multiple pre-recorded standard movement trajectories. Based on the gesture recognition information, the target operation instruction is matched in the instruction mapping table.

9. The method according to claim 8, characterized in that, Executing the target operation instruction includes: Define the action function to be executed; The gesture recognition information corresponding to the target operation command is input into the execution action function so that the execution action function can query the target shortcut key based on the gesture recognition information; The target window handle is obtained according to the operation type of the target shortcut key, and the target window handle is used to represent the foreground window in the current user interface that can respond to the target shortcut key; By executing the target shortcut key, an execution message is sent to the foreground window corresponding to the target window handle, so that the foreground window responds to the target operation command.

10. An interactive control system that listens to user gestures, characterized in that, The system includes: The trajectory acquisition module is used to acquire the operation movement trajectory, which is a set of operation events consisting of at least one operation trajectory point; The feature recognition module is used to recognize gesture features in the operation movement trajectory. The gesture features are chain codes generated based on the position information of the operation trajectory points. The position information includes at least one of the coordinate values ​​of the operation trajectory points, the distance between points, the angle between points, the time interval between points, and the trajectory direction. The instruction matching module is used to match a target operation instruction based on the gesture feature, wherein the target operation instruction is an operation instruction that is associated with the gesture feature; the association between the operation instruction and the gesture feature is set according to the operation shortcut keys of the current user interface and / or the graphical user interface programming data. The instruction execution module is used to execute the target operation instruction.

Images