Mouse control monitoring method and device

Abstract

The invention provides a mouse control monitoring method and device, and the method comprises the steps: monitoring the movement of a mouse through a GCMouse module when a mouse mode of a cloud application is a mouse-free mode, obtaining the offset of the movement of the mouse, and determining first intermediate data according to the offset of the movement of the mouse; and sending the first intermediate data to a cloud server, so that the cloud server determines a first mouse movement event according to the first intermediate data, and sending the first mouse movement event to a system of the cloud server through the virtual mouse, the first intermediate data comprises source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of mouse movement; in the mouse-free mode scene of the cloud application, when the mouse cursor moves to the edge of the iPhone screen and cannot continue to move outwards, the movement of the mouse can still be monitored by moving the mouse outwards, and then the cloud application is controlled, and the mouse control response speed is superior to that of an H5 scheme.

Description

Technical Field

[0001] The present invention relates to the field of cloud applications, and particularly to a method and device for monitoring mouse manipulation. Background Art

[0002] Currently, Apple's mobile devices include iPhones, iPads, and iWatches. However, the system API of the Apple mouse is designed for iPads and cannot be perfectly compatible with iPhones. Although a transparent WebView control that covers the touch input layer of the iPhone and loads H5 web page code can be used to monitor the mouse and redefine mouse events, and the redefined mouse events are used to manipulate cloud applications, the H5 solution has the following defects: (1) When using the mouse mode on the iPhone, the AssistiveTouch function must be enabled. When the mouse cursor moves to the edge of the screen, since the mouse cursor cannot move outwards any further, the WebView control cannot monitor new movement events. This will affect the usage experience of some cloud applications. For example, in the mouse-free mode scenario of some games, rotating the perspective of the game character by moving the mouse, in the H5 solution, when the mouse cursor moves to the edge of the screen and cannot move outwards any further, the game character cannot rotate the perspective 360 degrees, which will greatly affect the game experience and even cause the user to be unable to continue the game; (2) In the H5 solution, the WebView control is a heavyweight component, which will occupy more system resources, and mouse events need to be converted through the H5 web page code of the WebView control and the iOS system, increasing the latency. Frequent event communication will further affect the response speed.

[0003] Therefore, how to provide a solution that can still monitor the movement of the mouse and then manipulate the cloud application when the mouse cursor moves to the edge of the iPhone screen and cannot move outwards any further in the mouse-free mode scenario of the cloud application, and make the mouse manipulation response speed of this solution better than that of the H5 solution has become a technical problem to be solved urgently. Summary of the Invention

[0004] In view of the technical problems existing in the prior art, an embodiment of the present application provides a method and device for monitoring mouse manipulation.

[0005] In a first aspect, an embodiment of the present application provides a method for monitoring mouse manipulation, which is applied to a mobile device and includes: When the mouse mode of the cloud application is the mouse-free mode, monitor the movement of the mouse through the GCMouse module, obtain the offset of the mouse movement, and determine first intermediate data according to the offset of the mouse movement; Send the first intermediate data to the cloud server so that the cloud server determines the first mouse movement event based on the first intermediate data, and send the first mouse movement event to the system of the cloud server through a virtual mouse, where the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0006] In a second aspect, an embodiment of the present application further provides a mouse control monitoring method, which is applied to a cloud server and includes: When the mouse mode of the cloud application is the no-mouse mode, receive the first intermediate data sent by the mobile device, determine the first mouse movement event based on the first intermediate data, and send the first mouse movement event to the system of the cloud server through a virtual mouse, where the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0007] In a third aspect, an embodiment of the present application further provides a mouse control monitoring device, which is applied to a mobile device and includes: An acquisition unit, configured to monitor the movement of the mouse through the GCMouse module and acquire the offset of the mouse movement when the mouse mode of the cloud application is the no-mouse mode, and determine the first intermediate data according to the offset of the mouse movement; A sending unit, configured to send the first intermediate data to the cloud server so that the cloud server determines the first mouse movement event based on the first intermediate data, and send the first mouse movement event to the system of the cloud server through a virtual mouse, where the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0008] In a fourth aspect, an embodiment of the present application further provides a mouse control monitoring device, which is applied to a cloud server and includes: A receiving unit, configured to receive the first intermediate data sent by the mobile device when the mouse mode of the cloud application is the no-mouse mode, determine the first mouse movement event based on the first intermediate data, and send the first mouse movement event to the system of the cloud server through a virtual mouse, where the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0009] In summary, for the mouse control monitoring method and device provided in the embodiments of the present application, when the mouse mode of the cloud application is the mouse - less mode, the GCMouse module monitors the movement of the mouse, obtains the offset of the mouse movement, determines the first intermediate data according to the offset of the mouse movement, determines the first mouse movement event according to the first intermediate data, and sends the first mouse movement event to the system of the cloud server through the virtual mouse to control the cloud application. As a result, when the mouse cursor moves to the edge of the iPhone screen and cannot move further outwards, the mouse can still be monitored when it moves outwards, and then the cloud application can be controlled. Moreover, in the whole solution, the GCMouse module belongs to the system module, and the mouse event does not need to be converted by the H5 web page code of the WebView control and the iOS system. Therefore, the mouse control response speed is better than that of the H5 solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic flowchart of a mouse control monitoring method provided in an embodiment of the present application; Figure 2 It is a schematic flowchart of another mouse control monitoring method provided in an embodiment of the present application; Figure 3 It is a schematic structural diagram of a mouse control monitoring device provided in an embodiment of the present application; Figure 4 It is a schematic structural diagram of another mouse control monitoring device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0011] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only for the purposes of illustration and description, and are not used to limit the protection scope of the present application. Additionally, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the present application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations in the flowchart may not be implemented in sequence, and steps without logical context relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

[0012] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application usually described and illustrated in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application claimed, but only represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0013] It should be noted that the term "including" will be used in the embodiments of the present application to indicate the existence of the features stated thereafter, but does not exclude the addition of other features.

[0014] Referring to Figure 1 As shown, the embodiments of the present application provide a schematic flowchart of a mouse operation monitoring method, which is applied to a mobile device and includes: S10. When the mouse mode of the cloud application is the no-mouse mode, monitor the movement of the mouse through the GCMouse module, obtain the offset of the mouse movement, and determine the first intermediate data according to the offset of the mouse movement; In this embodiment, it should be noted that the mobile device can be an iPhone with a system version of iOS 14 or above. The iPhone is connected to a physical mouse, and the system will generate a mouse movement event during the movement of the mouse. When the mouse mode of the cloud application is the no-mouse mode (the mouse mode of the cloud application can be informed to the execution entity of step S10 by the cloud server), monitor the mouse movement event through the GCMouse module (a module of the system). After the mouse movement event is monitored, the GCMouse module will call the system callback method to return the offset of the mouse movement to the execution entity of step S10.

[0015] S11. Send the first intermediate data to the cloud server so that the cloud server determines the first mouse movement event according to the first intermediate data, and send the first mouse movement event to the system of the cloud server through the virtual mouse, where the first intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0016] In this embodiment, it should be noted that the first intermediate data includes the source data, intermediate result data, or target result data in the process of obtaining the first mouse movement event based on the offset of mouse movement. That is, the first intermediate data includes the offset of mouse movement or the result of any intermediate step in the process of obtaining the first mouse movement event based on the offset of mouse movement, or the first mouse movement event. The first mouse movement event may include a movement identifier and the offset of the target mouse movement. Among them, the offset of the target mouse movement may be the same as the offset of mouse movement, or the offset of the target mouse movement may be obtained by processing the offset of mouse movement. For example, it may be obtained by magnifying or reducing the offset of mouse movement. By determining the first mouse movement event and sending the first mouse movement event to the system of the cloud server through the virtual mouse, it is possible to simulate mouse movement on the cloud server to control the cloud application running in the cloud server. For example, the perspective of the game character in the cloud game can be rotated.

[0017] In the mouse control monitoring method provided by the embodiment of the present application, when the mouse mode of the cloud application is the no-mouse mode, the GCMouse module monitors the movement of the mouse and obtains the offset of the mouse movement. The first intermediate data is determined according to the offset of the mouse movement, and the first intermediate data is sent to the cloud server so that the cloud server determines the first mouse movement event according to the first intermediate data, and sends the first mouse movement event to the system of the cloud server through the virtual mouse to control the cloud application. Thus, when the mouse cursor moves to the edge of the iPhone screen and cannot move further outwards, the mouse can still be monitored when moving outwards, and then the cloud application can be controlled. Moreover, in the entire solution, the GCMouse module belongs to the system module, and the mouse event does not need to be converted by the H5 web page code of the WebView control and the iOS system. Therefore, the mouse control response speed is better than the H5 solution.

[0018] Based on the foregoing method embodiment, the method may further include: When the mouse mode of the cloud application is the with-mouse mode, the movement of the mouse cursor is monitored through the UIHoverGestureRecognizer of the hover gesture added on the touch input layer, and the movement position of the mouse cursor on the touch input layer is obtained. The second intermediate data is determined according to the movement position; The second intermediate data is sent to the cloud server so that the cloud server determines the second mouse movement event according to the second intermediate data, and sends the second mouse movement event to the system of the cloud server through the virtual mouse, where the second intermediate data includes the source data, intermediate result data, or target result data in the process of obtaining the second mouse movement event according to the movement position.

[0019] In this embodiment, it should be noted that when the mouse mode of the cloud application is the mouse-present mode, the cloud server will not render the mouse cursor. That is, there is no mouse cursor in the cloud application screen rendered by the cloud server, and the mouse cursor on the cloud application screen seen by the user on the screen of the mobile device is the mouse cursor of the mouse connected to the mobile device. The hover gesture UIHoverGestureRecognizer is added to the touch input layer. When the mouse mode of the cloud application is the mouse-present mode, the movement of the mouse cursor on the touch input layer is monitored through the hover gesture UIHoverGestureRecognizer. After the movement of the mouse cursor on the touch input layer is detected, the hover gesture UIHoverGestureRecognizer will call the hover gesture callback method, and the hover gesture callback method will return a gesture object to the execution entity of this embodiment. If the state attribute of the gesture object (indicating the state of the mouse cursor hovering) indicates that the mouse cursor is moving, the position of the cursor on the touch input layer can be obtained by calling the method of the hover gesture through the gesture object, that is, the movement position of the mouse cursor on the touch input layer. The second intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the second mouse movement event based on the movement position, that is, the second intermediate data includes the movement position or the result of any intermediate step in the process of obtaining the second mouse movement event based on the movement position or the second mouse movement event. The second mouse movement event may include a movement identifier and a target movement position. It can be understood that the sizes of the touch input layer of the iPhone and the cloud application screen rendered by the cloud server may not be the same. Therefore, it is necessary to find the corresponding position on the cloud application screen rendered by the cloud server based on the movement position on the touch input layer, and this position is the target movement position. For example, the size of the touch input layer is the same as that of the rendering screen layer of the iPhone, and the aspect ratio of the cloud application screen displayed on the iPhone is the same as that of the cloud application screen rendered by the cloud server. Taking the position corresponding to the top left vertex of the touch input layer as the coordinate origin, the axis where the width of the touch input layer is located as the x-axis, and the axis where the height of the touch input layer is located as the y-axis, denoting the width and height of the cloud application screen displayed on the iPhone as a1 and b1 respectively, and the width and height of the cloud application screen rendered by the cloud server as a2 and b2 respectively, and the coordinates of the movement position as (x1, y1), and the coordinates of the target movement position as (x2, y2), then x2 = x1 × a2 / a1, y2 = y1 × b2 / b1. By determining the second mouse movement event and sending the second mouse movement event to the system of the cloud server through the virtual mouse, it is possible to simulate mouse movement on the cloud server to control the cloud application.

[0020] Based on the foregoing method embodiments, the method may further include: Obtain a touch event. If it is determined according to the touch event that the manipulation type is mouse manipulation, create a mouse event touch model, add the mouse event touch model into a preset queue, and start a timer. The mouse event touch model includes manipulation coordinates, manipulation action type, creation time, status, and mouse button type, and the status is initially in a waiting state; For each mouse event touch model in the queue, determine whether the corresponding mouse button type returned by the system callback method called by the GCMouse module is received before the time set by the timer corresponding to the mouse event touch model arrives; If the corresponding mouse button type returned by the system callback method called by the GCMouse module is not received before the time set by the timer corresponding to the mouse event touch model arrives, then after the time set by the timer corresponding to the mouse event touch model arrives, determine whether there is a first mouse event touch model in the queue. If there is no first mouse event touch model in the queue, determine third intermediate data according to the mouse event touch model, and send the third intermediate data to the cloud server, so that the cloud server determines a first mouse left button event according to the third intermediate data, and sends the first mouse left button event to the system of the cloud server through a virtual mouse. Or if there is a first mouse event touch model in the queue, determine fourth intermediate data according to the mouse event touch model, determine fifth intermediate data according to the first mouse event touch model, and send the fourth intermediate data and the fifth intermediate data to the cloud server, so that the cloud server determines a second mouse left button event according to the fourth intermediate data, determines a first mouse button event according to the fifth intermediate data, and sends the second mouse left button event and the first mouse button event to the system of the cloud server through a virtual mouse in the order of the corresponding creation times. The first mouse event touch model is a mouse event touch model in the queue whose creation time is later than the creation time of this mouse event touch model, whose status is in a waiting for previous event state, and there is no second mouse event touch model between it and this mouse event touch model. The second mouse event touch model is a mouse event touch model in the queue whose status is not in a waiting for previous event state. The third intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse left button event according to this mouse event touch model. The fourth intermediate data includes source data or intermediate result data or target result data in the process of obtaining the second mouse left button event according to this mouse event touch model. The fifth intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse button event according to the first mouse event touch model; Or If, before the time set by the timer corresponding to the touch model of the mouse event arrives, the corresponding mouse button type returned by the system callback method called by the GCMouse module is received, the mouse button type in the touch model of the mouse event is set to the corresponding mouse button type returned by the system callback method, and it is determined whether there is a third touch model of the mouse event in the queue. If there is no third touch model of the mouse event in the queue, the sixth intermediate data is determined according to the touch model of the mouse event, and the sixth intermediate data is sent to the cloud server so that the cloud server determines the second mouse button event according to the sixth intermediate data and sends the second mouse button event to the system of the cloud server through the virtual mouse, or if there is a third touch model of the mouse event in the queue, the state in the touch model of the mouse event is modified to the waiting previous event state, and the timer corresponding to the touch model of the mouse event is stopped, where the third touch model of the mouse event is the touch model of the mouse event in the queue whose creation time is earlier than the creation time of the touch model of the mouse event and whose state is the waiting state, and the sixth intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the second mouse button event according to the touch model of the mouse event.

[0021] In this embodiment, it should be noted that when the touch input layer is manipulated by a finger or a mouse, the system will generate a touch event. Specifically, when the touch input layer is clicked by a finger or the left, middle, and right mouse buttons are clicked on the touch input layer, the system will generate a touch event. GCMouse can monitor the touch event, but based on the touch event, the GCMouse module can only recognize the middle mouse button operation and the right mouse button operation, and cannot recognize the left mouse button operation. Therefore, in this embodiment, the left mouse button operation, the middle mouse button operation, and the right mouse button operation are defined based on the touch event and the GCMouse module. Specifically, after each touch event is generated by the system, the generated touch event is returned to the execution entity of this embodiment through the system callback method. Then, the manipulation type can be determined according to the touch event. Specifically, when it is determined that the type in the touch event is not UITouchTypeDirect direct click or the majorRadiusTolerance radius ≤ 0, the manipulation type is determined to be mouse manipulation; in other cases, the manipulation type is determined to be finger manipulation. After the manipulation type is determined to be mouse manipulation, a mouse event touch model is created according to the information of the touch event, the mouse event touch model is added to a preset queue, and a timer is started. The mouse event touch model can include the manipulation coordinates, the manipulation action type, the creation time, the status, and the mouse button type. The manipulation coordinates are the position of the mouse cursor on the touch input layer (which can be obtained by calling the method of the touch event object), the creation time is the time when the mouse event touch model is created (which can be a timestamp), and the status is initially the waiting state. The manipulation action type includes press down and lift up. For the manipulation action type, after the touch event is generated, if it is determined that the touch begin method is called, the manipulation action type is determined to be press down; if it is determined that the touchend method and the touch cancel method are called, the manipulation action type is determined to be lift up. It should be noted that if after the touch event is generated, none of the touch begin method, the touch end method, and the touch cancel method are called, but the touch move method is called, no corresponding mouse event touch model will be created for the generated touch event this time. The timer is started for the corresponding mouse event touch model, and the time set by the timer can be set as needed, such as set to 10 ms.

[0022] Each time after adding a mouse event touch model (hereinafter simply referred to as "model M" for ease of description) to a preset queue and starting a timer (hereinafter simply referred to as "timer t"), it is necessary to determine whether the corresponding mouse button type returned by the system callback method called by the GCMouse module is received before the time set by timer t. If the corresponding mouse button type returned by the system callback method called by the GCMouse module is not received, then after the time set by timer t arrives, it is judged whether there is a first mouse event touch model in the queue. If there is no first mouse event touch model in the queue, then the third intermediate data is determined according to model M, and the third intermediate data is sent to the cloud server so that the cloud server determines the first mouse left button event according to the third intermediate data, and sends the first mouse left button event to the system of the cloud server through the virtual mouse. Among them, the first mouse event touch model is a mouse event touch model in the queue whose creation time is later than the creation time of model M, whose state is the waiting previous event state, and there is no second mouse event touch model between it and model M. The second mouse event touch model is a mouse event touch model in the queue whose state is not the waiting previous event state. The third intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse left button event according to model M, that is, the third intermediate data includes model M or the result of any intermediate step in the process of obtaining the first mouse left button event according to model M or the first mouse left button event. The first mouse left button event may include a mouse left button identifier, an identifier of the manipulation action type in model M, and the coordinates corresponding to the manipulation coordinates in model M on the cloud application screen rendered by the cloud server (the determination process of this coordinate is the same as that of the target movement position coordinate in the foregoing embodiment and will not be elaborated here). After sending the third intermediate data to the cloud server, the state in model M can be modified to the sent state, model M in the queue can be deleted, the timer corresponding to model M can be stopped, and the timer resources can be released. The mouse button type in each mouse event touch model in the queue can be initially the mouse left button, then the mouse left button identifier in the first mouse left button event can be determined according to the mouse button type in model M, for example, it can be the mouse button type in model M or convert the mouse button type in model M into a mouse left button identifier recognizable by the system of the cloud server, or the mouse button type in each mouse event touch model in the queue can be initially a null value, then when determining the third intermediate data according to model M, the mouse button type in model M can be set to the mouse left button, and the mouse left button identifier in the first mouse left button event can be determined according to the mouse button type in model M. After determining the first mouse left button event and sending the first mouse left button event to the system of the cloud server through the virtual mouse, it is possible to simulate the mouse left button manipulation on the cloud server to manipulate the cloud application.If there is a first mouse event touch model in the queue, the fourth intermediate data is determined according to model M, the fifth intermediate data is determined according to the first mouse event touch model, and the fourth intermediate data and the fifth intermediate data are sent to the cloud server, so that the cloud server determines the second left mouse button event according to the fourth intermediate data, determines the first mouse button event according to the fifth intermediate data, and sends the second left mouse button event and the first mouse button event to the system of the cloud server through the virtual mouse in the order of the creation time. For example, there are 2 first mouse event touch models in the queue: the first mouse event touch model 1 and the first mouse event touch model 2. The creation time of the first mouse event touch model 1 is earlier than that of the first mouse event touch model 2. Then, the fourth intermediate data is determined according to model M, the fifth intermediate data 1 is determined according to the first mouse event touch model 1, the fifth intermediate data 2 is determined according to the first mouse event touch model 2, and the fourth intermediate data, the fifth intermediate data 1 and the fifth intermediate data 2 are sent to the cloud server. After receiving the fourth intermediate data, the cloud server will determine the second left mouse button event according to the fourth intermediate data. After receiving the fifth intermediate data 1, the cloud server will determine the first mouse button event 1 according to the fifth intermediate data 1. After receiving the fifth intermediate data 2, the cloud server will determine the first mouse button event 2 according to the fifth intermediate data 2. When the cloud server sends data, it will send the second left mouse button event, the first mouse button event 1 and the first mouse button event 2 to the system of the cloud server through the virtual mouse in the order of sending the second left mouse button event first, then the first mouse button event 1, and finally the first mouse button event 2. The fourth intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the second left mouse button event according to model M, that is, the fourth intermediate data includes model M or the result of any intermediate step in the process of obtaining the second left mouse button event according to model M or the second left mouse button event. The second left mouse button event may include the left mouse button identifier, the identifier of the manipulation action type in model M, and the coordinates corresponding to the manipulation coordinates in model M on the cloud application screen rendered by the cloud server.The fifth intermediate data includes the source data, intermediate result data, or target result data during the process of obtaining the first mouse button event based on the first mouse event touch model. That is, the fifth intermediate data includes the first mouse event touch model, or the result of any intermediate step during the process of obtaining the first mouse button event based on the first mouse event touch model, or the first mouse button event. The first mouse button event may include the mouse button type identifier corresponding to the mouse button type in the first mouse event touch model (for example, if the mouse button type in the first mouse event touch model is the right mouse button, then the corresponding mouse button type identifier is the right mouse button identifier recognizable by the system of the cloud server; another example is that if the mouse button type in the first mouse event touch model is the middle mouse button, then the corresponding mouse button type identifier is the middle mouse button identifier recognizable by the system of the cloud server), the identifier of the manipulation action type in the first mouse event touch model, and the coordinates corresponding to the manipulation coordinates in the first mouse event touch model on the cloud application screen rendered by the cloud server. After sending the fourth intermediate data to the cloud server, the state in model M can be modified to the sent state, model M in the queue can be deleted, the timer corresponding to model M can be stopped, and the timer resources can be released. After sending the fifth intermediate data to the cloud server, the state in the first mouse event touch model can be modified to the sent state, and the first mouse event touch model in the queue can be deleted. Through the logic of this embodiment, it can be ensured that the manipulation order of the left, right, and middle mouse buttons simulated on the cloud server is consistent with the actual manipulation order.

[0023] If, before the time set by the timer t arrives, the corresponding mouse button type (such as the right mouse button or the middle mouse button) returned by the system callback method called by the GCMouse module is received, the mouse button type in the model M is set to the corresponding mouse button type returned by the system callback method, and it is determined whether there is a mouse event touch model (i.e., the third mouse event touch model) in the queue whose creation time is earlier than the creation time in the model M and whose status is the waiting state. If there is no third mouse event touch model in the queue, the sixth intermediate data is determined according to the model M, and the sixth intermediate data is sent to the cloud server so that the cloud server can determine the second mouse button event according to the sixth intermediate data, and the second mouse button event is sent to the system of the cloud server through the virtual mouse. Among them, the determination process of the sixth intermediate data can be the same as the determination process of the fifth intermediate data, and the determination process of the second mouse button event can be the same as the determination process of the first mouse button event, which will not be elaborated here. After sending the sixth intermediate data to the cloud server, the status in the model M can be modified to the sent state, the model M in the queue can be deleted, the timer corresponding to the model M can be stopped, and the timer resources can be released. If there is a third mouse event touch model in the queue, the status in the model M is modified to the waiting for previous event state, and the timer corresponding to the model M is stopped, and the timer resources are released. In addition, it should be noted that for a certain touch event generated by the system, the time interval between the result returned by the GCMouse module calling the system callback method after monitoring the touch event and the system calling the system callback method to return the touch event to the execution entity of this embodiment is generally no more than 1 millisecond. Therefore, after receiving the result returned by the GCMouse module calling the system callback method, it can be matched with a mouse event touch model in the queue. Specifically, the mouse event touch model in the queue whose creation time is not later than the first time (the first time is the time when the result returned by the GCMouse module calling the system callback method is received) and is the closest to the first time can be used as the target mouse event touch model. Then, the result returned by the GCMouse module calling the system callback method can be used to set the mouse button type in the target mouse event touch model, and the mouse button type in the model M is set to the mouse button type (right mouse button or middle mouse button) returned by the GCMouse module calling the system callback method.

[0024] Based on the foregoing method embodiments, the method may further include: Monitor the sliding of the mouse wheel through the GCMouse module, obtain the amplitude of the mouse wheel sliding, and determine the seventh intermediate data according to the amplitude of the mouse wheel sliding; Send the seventh intermediate data to the cloud server so that the cloud server determines a mouse wheel scrolling event based on the seventh intermediate data, and send the mouse wheel scrolling event to the system of the cloud server through a virtual mouse, where the seventh intermediate data includes source data or intermediate result data or target result data in the process of obtaining the mouse wheel scrolling event according to the amplitude of the mouse wheel scrolling.

[0025] In this embodiment, it should be noted that when the mouse wheel is scrolled, the system will generate a wheel event, and the GCMouse module can monitor the wheel event. After monitoring the wheel event, the GCMouse module will call the system callback method, and the callback method will return the amplitude of the mouse wheel scrolling of the execution subject in this embodiment. The seventh intermediate data includes source data or intermediate result data or target result data in the process of obtaining the mouse wheel scrolling event according to the amplitude of the mouse wheel scrolling, that is, the seventh intermediate data includes the amplitude of the mouse wheel scrolling or the result of any intermediate step in the process of obtaining the mouse wheel scrolling event according to the amplitude of the mouse wheel scrolling or the mouse wheel scrolling event. The mouse wheel scrolling event may include the identifier of the mouse wheel and the target amplitude of the mouse wheel scrolling. The target amplitude of the mouse wheel scrolling may be the amplitude of the mouse wheel scrolling or obtained by processing the amplitude of the mouse wheel scrolling, and the processing includes zooming in or out, etc. After determining the mouse wheel scrolling event and sending the mouse wheel scrolling event to the system of the cloud server through a virtual mouse, it is possible to simulate mouse wheel control on the cloud server to control cloud applications.

[0026] Refer to Figure 2 As shown, an embodiment of the present application provides a flowchart of a mouse control monitoring method, which is applied to a cloud server and includes: S20. When the mouse mode of the cloud application is the no-mouse mode, receive the first intermediate data sent by the mobile device, determine a first mouse movement event based on the first intermediate data, and send the first mouse movement event to the system of the cloud server through a virtual mouse, where the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0027] The mouse control monitoring method provided by the embodiment of the present application, when the mouse mode of the cloud application is the no-mouse mode, receives the first intermediate data sent by the mobile device, determines the first mouse movement event according to the first intermediate data, and sends the first mouse movement event to the system of the cloud server through the virtual mouse to control the cloud application. Since the first intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement, when the mouse cursor moves to the edge of the iPhone screen and cannot move outwards any further, moving the mouse outwards can still monitor the movement of the mouse and then control the cloud application. Moreover, in the whole solution, the GCMouse module belongs to the system module, and the mouse event does not need to be converted by the H5 web page code of the WebView control and the iOS system. Therefore, the mouse control response speed is better than the H5 solution.

[0028] Based on the foregoing method embodiment, the method may further include: When the mouse mode of the cloud application is the with-mouse mode, receive the second intermediate data sent by the mobile device, determine the second mouse movement event according to the second intermediate data, and send the second mouse movement event to the system of the cloud server through the virtual mouse, where the second intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the second mouse movement event according to the movement position of the mouse cursor on the touch input layer.

[0029] Based on the foregoing method embodiment, the method may further include: Receive the third intermediate data sent by the mobile device, determine the first mouse left button event according to the third intermediate data, and send the first mouse left button event to the system of the cloud server through the virtual mouse, where the third intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the first mouse left button event according to a mouse event touch model in a preset queue; or A system that receives the fourth intermediate data and the fifth intermediate data sent by a mobile device, determines a second left mouse button event based on the fourth intermediate data, determines a first mouse button event based on the fifth intermediate data, and sends the second left mouse button event and the first mouse button event to a cloud server through a virtual mouse in the order of the corresponding creation times. Among them, the fourth intermediate data includes source data or intermediate result data or target result data in the process of obtaining the second left mouse button event according to a mouse event touch model in a preset queue, and the fifth intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse button event according to the first mouse event touch model. The first mouse event touch model is a mouse event touch model in the queue whose creation time is later than that of the one mouse event touch model, whose state is waiting for a previous event state, and there is no second mouse event touch model between it and the one mouse event touch model. The second mouse event touch model is a mouse event touch model in the queue whose state is not waiting for a previous event state; or A system that receives the sixth intermediate data sent by a mobile device, determines a second mouse button event based on the sixth intermediate data, and sends the second mouse button event to a cloud server through a virtual mouse. Among them, the sixth intermediate data includes source data or intermediate result data or target result data in the process of obtaining the second mouse button event according to a mouse event touch model in a preset queue.

[0030] Based on the foregoing method embodiments, the method may further include: A system that receives the seventh intermediate data sent by a mobile device, determines a mouse wheel scrolling event based on the seventh intermediate data, and sends the mouse wheel scrolling event to a cloud server through a virtual mouse. Among them, the seventh intermediate data includes source data or intermediate result data or target result data in the process of obtaining the mouse wheel scrolling event according to the amplitude of the mouse wheel scrolling.

[0031] Refer to Figure 3 As shown, an embodiment of the present application provides a structural schematic diagram of a mouse control monitoring device, which is applied to a mobile device and includes: An obtaining unit 30, configured to monitor the movement of a mouse through a GCMouse module and obtain the offset of the mouse movement when the mouse mode of the cloud application is the no-mouse mode, and determine first intermediate data according to the offset of the mouse movement; A sending unit 31, configured to send first intermediate data to a cloud server, so that the cloud server determines a first mouse movement event according to the first intermediate data, and sends the first mouse movement event to a system of the cloud server through a virtual mouse, where the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of mouse movement.

[0032] When the mouse mode of the cloud application is the no-mouse mode, the mouse control monitoring device provided in the embodiment of the present application monitors the movement of the mouse through the GCMouse module, obtains the offset of the mouse movement, determines the first intermediate data according to the offset of the mouse movement, and sends the first intermediate data to the cloud server, so that the cloud server determines the first mouse movement event according to the first intermediate data, and sends the first mouse movement event to a system of the cloud server through the virtual mouse to control the cloud application. Therefore, when the mouse cursor moves to the edge of the iPhone screen and cannot move outwards any further, moving the mouse outwards can still monitor the mouse movement and then control the cloud application. Moreover, in the whole solution, the GCMouse module belongs to the system module, and the mouse event does not need to go through the H5 web page code of the WebView control and the conversion of the iOS system. Therefore, the mouse control response speed is better than the H5 solution.

[0033] Refer to Figure 4 As shown in A receiving unit 40, configured to receive the first intermediate data sent by a mobile device when the mouse mode of the cloud application is the no-mouse mode, determine the first mouse movement event according to the first intermediate data, and send the first mouse movement event to a system of the cloud server through the virtual mouse, where the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of mouse movement.

[0034] When the mouse mode of the cloud application is the no-mouse mode, the mouse control monitoring device provided in the embodiment of the present application receives the first intermediate data sent by the mobile device, determines the first mouse movement event according to the first intermediate data, and sends the first mouse movement event to a system of the cloud server through the virtual mouse to control the cloud application. Since the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of mouse movement, when the mouse cursor moves to the edge of the iPhone screen and cannot move outwards any further, moving the mouse outwards can still monitor the mouse movement and then control the cloud application. Moreover, in the whole solution, the GCMouse module belongs to the system module, and the mouse event does not need to go through the H5 web page code of the WebView control and the conversion of the iOS system. Therefore, the mouse control response speed is better than the H5 solution.

[0035] The mouse control monitoring device provided by the embodiment of the present application has the same implementation process as the mouse control monitoring method provided by the embodiment of the present application, and can achieve the same effects as the mouse control monitoring method provided by the embodiment of the present application, which will not be elaborated here.

[0036] The above are only specific implementation manners of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A mouse control monitoring method, characterized in that: Applicable to mobile devices, including: When the mouse mode of the cloud application is the mouse-free mode, the movement of the mouse is monitored through the GCMouse module, and the offset of the mouse movement is obtained, and the first intermediate data is determined according to the offset of the mouse movement; The first intermediate data is sent to a cloud server so that the cloud server determines a first mouse movement event based on the first intermediate data, and the first mouse movement event is sent to a system of the cloud server via a virtual mouse, wherein the first intermediate data includes source data or intermediate result data or target result data in a process of obtaining the first mouse movement event based on an offset of the mouse movement.

2. The method according to claim 1, characterized in that Also includes: When the mouse mode of the cloud application is the mouse mode, the movement of the mouse cursor is monitored through the hover gesture UIHoverGestureRecognizer added on the touch input layer, and the moving position of the mouse cursor on the touch input layer is obtained, and the second intermediate data is determined according to the moving position; The second intermediate data is sent to the cloud server so that the cloud server determines the second mouse movement event based on the second intermediate data, and the second mouse movement event is sent to the system of the cloud server through the virtual mouse, wherein the second intermediate data includes source data or intermediate result data or target result data in the process of obtaining the second mouse movement event based on the moving position.

3. The method according to claim 1 or 2, characterized in that Also includes: Get the touch event. If the control type is determined to be mouse control according to the touch event, create a mouse event touch model, add the mouse event touch model to the preset queue, and start a timer. The mouse event touch model includes the control coordinates, control action type, creation time, state, and mouse button type. The initial state is the waiting state. For each mouse event touch model in the queue, determine whether the corresponding mouse button type returned by the system callback method called by the GCMouse module is received before the time set by the timer corresponding to the mouse event touch model arrives; If the corresponding mouse button type returned by the system callback method called by the GCMouse module is not received before the time set by the timer corresponding to the mouse event touch model arrives, then after the time set by the timer corresponding to the mouse event touch model arrives, it is determined whether there is a first mouse event touch model in the queue; if there is no first mouse event touch model in the queue, third intermediate data is determined according to the mouse event touch model, and the third intermediate data is sent to the cloud server, so that the cloud server determines the first mouse left button event according to the third intermediate data, and sends the first mouse left button event to the cloud server system through the virtual mouse; or if there is a first mouse event touch model in the queue, fourth intermediate data is determined according to the mouse event touch model, fifth intermediate data is determined according to the first mouse event touch model, and the fourth intermediate data and the fifth intermediate data are sent to the cloud server, so that the cloud server determines the second mouse left button event according to the fourth intermediate data, and determines the first mouse button according to the fifth intermediate data. event, and sends the second mouse left button event and the first mouse button event to the cloud server system through the virtual mouse in the order of the corresponding creation time, wherein the first mouse event touch model is a mouse event touch model in the queue whose creation time is later than the creation time of the mouse event touch model and whose state is waiting for the previous event state and is not separated from the mouse event touch model by the second mouse event touch model, the second mouse event touch model is a mouse event touch model in the queue whose state is not waiting for the previous event state, the third intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the first mouse left button event according to the mouse event touch model, the fourth intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the second mouse left button event according to the mouse event touch model, and the fifth intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the first mouse button event according to the first mouse event touch model; or If the corresponding mouse button type returned by the system callback method called by the GCMouse module is received before the time set by the timer corresponding to the mouse event touch model arrives, the mouse button type in the mouse event touch model is set to the corresponding mouse button type returned by the system callback method, and it is determined whether there is a third mouse event touch model in the queue. If there is no third mouse event touch model in the queue, sixth intermediate data is determined according to the mouse event touch model, and the sixth intermediate data is sent to the cloud server, so that the cloud server determines the second mouse button event according to the sixth intermediate data, and sends the second mouse button event to the system of the cloud server through the virtual mouse, or if there is a third mouse event touch model in the queue, the state in the mouse event touch model is modified to the waiting state for the preceding event, and the timer corresponding to the mouse event touch model is stopped, wherein the third mouse event touch model is a mouse event touch model in the queue whose creation time is earlier than the creation time of the mouse event touch model and whose state is the waiting state, and the sixth intermediate data includes source data or intermediate result data or target result data in the process of obtaining the second mouse button event according to the mouse event touch model.

4. The method according to claim 1, characterized in that Also includes: Monitor the sliding of the mouse wheel through the GCMouse module, obtain the sliding amplitude of the mouse wheel, and determine the seventh intermediate data according to the sliding amplitude of the mouse wheel; The seventh intermediate data is sent to the cloud server so that the cloud server determines the mouse wheel sliding event based on the seventh intermediate data, and sends the mouse wheel sliding event to the system of the cloud server through the virtual mouse, wherein the seventh intermediate data includes source data or intermediate result data or target result data in the process of obtaining the mouse wheel sliding event based on the amplitude of the mouse wheel sliding.

5. A mouse control monitoring method, characterized in that: Applied to cloud servers, including: When the mouse mode of the cloud application is a mouse-free mode, first intermediate data sent by a mobile device is received, a first mouse movement event is determined based on the first intermediate data, and the first mouse movement event is sent to the system of the cloud server through a virtual mouse, wherein the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event based on an offset of the mouse movement.

6. The method according to claim 5, characterized in that Also includes: When the mouse mode of the cloud application is a mouse mode, second intermediate data sent by a mobile device is received, a second mouse movement event is determined based on the second intermediate data, and the second mouse movement event is sent to the system of the cloud server through a virtual mouse, wherein the second intermediate data includes source data or intermediate result data or target result data in the process of obtaining the second mouse movement event based on the moving position of the mouse cursor on the touch input layer.

7. The method according to claim 5 or 6, characterized in that Also includes: A system for receiving third intermediate data sent by a mobile device, determining a first left mouse button event based on the third intermediate data, and sending the first left mouse button event to a cloud server through a virtual mouse, wherein the third intermediate data includes source data or intermediate result data or target result data in a process of obtaining the first left mouse button event based on a mouse event touch model in a preset queue; or A system for receiving fourth intermediate data and fifth intermediate data sent by a mobile device, determining a second left mouse button event according to the fourth intermediate data, determining a first mouse button event according to the fifth intermediate data, and sending the second left mouse button event and the first mouse button event to a cloud server through a virtual mouse in the order of corresponding creation time, wherein the fourth intermediate data includes source data or intermediate result data or target result data in a process of obtaining the second left mouse button event according to a mouse event touch model in a preset queue, and the fifth intermediate data includes source data or intermediate result data or target result data in a process of obtaining the first mouse button event according to the first mouse event touch model, the first mouse event touch model is a mouse event touch model in the queue whose creation time is later than the creation time of the one mouse event touch model and whose state is waiting for a preceding event and whose second mouse event touch model is not separated from the one mouse event touch model, and the second mouse event touch model is a mouse event touch model in the queue whose state is not waiting for a preceding event; or A system for receiving sixth intermediate data sent by a mobile device, determining a second mouse button event based on the sixth intermediate data, and sending the second mouse button event to a cloud server via a virtual mouse, wherein the sixth intermediate data includes source data or intermediate result data or target result data in the process of obtaining the second mouse button event based on a mouse event touch model in a preset queue.

8. The method according to claim 5, characterized in that Also includes: A system for receiving the seventh intermediate data sent by a mobile device, determining a mouse wheel sliding event based on the seventh intermediate data, and sending the mouse wheel sliding event to a cloud server via a virtual mouse, wherein the seventh intermediate data includes source data or intermediate result data or target result data in a process of obtaining the mouse wheel sliding event based on the amplitude of the mouse wheel sliding.

9. A mouse control monitoring device, characterized in that: Applicable to mobile devices, including: an acquisition unit, configured to monitor the movement of the mouse through the GCMouse module when the mouse mode of the cloud application is the mouse-free mode, and acquire the offset of the mouse movement, and determine the first intermediate data according to the offset of the mouse movement; A sending unit is used to send first intermediate data to a cloud server so that the cloud server determines a first mouse movement event based on the first intermediate data, and sends the first mouse movement event to the cloud server system through a virtual mouse, wherein the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event based on an offset of the mouse movement.

10. A mouse-controlled monitoring device, characterized in that: Applied to cloud servers, including: A receiving unit is used to receive first intermediate data sent by a mobile device when the mouse mode of the cloud application is a mouse-free mode, determine a first mouse movement event based on the first intermediate data, and send the first mouse movement event to a cloud server system through a virtual mouse, wherein the first intermediate data includes source data or intermediate result data or target result data in a process of obtaining the first mouse movement event based on an offset of the mouse movement.

Images