Mouse control monitoring method and device

Abstract

The present application provides a method and device for monitoring mouse operations. The method includes: when the mouse mode of a cloud application is the mouse - less mode, monitoring the movement of the mouse through the GCMouse module, obtaining the offset of the mouse movement, and determining first intermediate data according to the offset of the mouse movement; sending the first intermediate data to a cloud server so that the cloud server determines a first mouse movement event according to the first intermediate data and sends the first mouse movement event to the system of the cloud server through a virtual mouse. Among them, the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement. In the mouse - less mode scenario of the cloud application, when the mouse cursor moves to the edge of the iPhone screen and cannot move further outwards, moving the mouse outwards can still monitor the mouse movement and then control the cloud application, and the mouse operation response speed is better than that of the H5 solution.

Description

Technical Field

[0001] The present invention relates to the field of cloud applications, and particularly to a method and device for monitoring mouse manipulation. Background Art

[0002] Currently, Apple mobile devices include iPhone, iPad, and iWatch. However, the system API of the Apple mouse is designed for iPad and cannot be perfectly compatible with iPhone. Although a transparent WebView control covering the touch input layer of iPhone and loaded with H5 web page code can be used to monitor the mouse and redefine mouse events, and the redefined mouse events are used to control cloud applications, the H5 solution has the following defects: (1) When using the mouse mode on iPhone, the AssistiveTouch function must be enabled. When the mouse cursor moves to the edge of the screen, since the mouse cursor cannot move further outwards, the WebView control cannot monitor new movement events. This will affect the usage experience of some cloud applications. For example, in the mouse-free mode scenario of some games, rotating the perspective of the game character by moving the mouse, in the H5 solution, when the mouse cursor moves to the edge of the screen and cannot move further outwards, the game character cannot rotate the perspective 360 degrees, which will greatly affect the game experience and even cause the user to be unable to continue the game; (2) In the H5 solution, the WebView control is a heavyweight component, which will occupy more system resources, and mouse events need to be converted through the H5 web page code of the WebView control and the iOS system, increasing the latency. Frequent event communication will further affect the response speed.

[0003] Therefore, how to provide a solution that can still monitor the movement of the mouse and then control the cloud application when the mouse cursor moves to the edge of the iPhone screen and cannot move further outwards in the mouse-free mode scenario of the cloud application, and make the mouse manipulation response speed of this solution better than that of the H5 solution, has become a technical problem to be solved urgently. Summary of the Invention

[0004] In view of the technical problems existing in the prior art, an embodiment of the present application provides a method and device for monitoring mouse manipulation.

[0005] In a first aspect, an embodiment of the present application provides a method for monitoring mouse manipulation, which is applied to a mobile device and includes:

[0006] When the mouse mode of the cloud application is the mouse-free mode, monitor the movement of the mouse through the GCMouse module, obtain the offset of the mouse movement, and determine the first intermediate data according to the offset of the mouse movement;

[0007] Send the first intermediate data to the cloud server so that the cloud server determines the first mouse movement event based on the first intermediate data and sends the first mouse movement event to the system of the cloud server through the virtual mouse, where the first intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0008] In a second aspect, an embodiment of the present application further provides a method for monitoring mouse manipulation, which is applied to a cloud server and includes:

[0009] When the mouse mode of the cloud application is the no-mouse mode, receive the first intermediate data sent by the mobile device, determine the first mouse movement event based on the first intermediate data, and send the first mouse movement event to the system of the cloud server through the virtual mouse, where the first intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0010] In a third aspect, an embodiment of the present application further provides a device for monitoring mouse manipulation, which is applied to a mobile device and includes:

[0011] An acquisition unit, configured to monitor the movement of the mouse through the GCMouse module and acquire the offset of the mouse movement when the mouse mode of the cloud application is the no-mouse mode, and determine the first intermediate data according to the offset of the mouse movement;

[0012] A sending unit, configured to send the first intermediate data to the cloud server so that the cloud server determines the first mouse movement event based on the first intermediate data and sends the first mouse movement event to the system of the cloud server through the virtual mouse, where the first intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0013] In a fourth aspect, an embodiment of the present application further provides a device for monitoring mouse manipulation, which is applied to a cloud server and includes:

[0014] A receiving unit, configured to receive the first intermediate data sent by the mobile device when the mouse mode of the cloud application is the no-mouse mode, determine the first mouse movement event based on the first intermediate data, and send the first mouse movement event to the system of the cloud server through the virtual mouse, where the first intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0015] In summary, for the mouse control monitoring method and device provided by the embodiments of the present application, when the mouse mode of the cloud application is the mouse - less mode, the GCMouse module monitors the movement of the mouse, obtains the offset of the mouse movement, determines the first intermediate data according to the offset of the mouse movement, determines the first mouse movement event according to the first intermediate data, and sends the first mouse movement event to the system of the cloud server through the virtual mouse to control the cloud application. Thus, when the mouse cursor moves to the edge of the iPhone screen and cannot move further outwards, the mouse can still be monitored when moved outwards, and then the cloud application can be controlled. Moreover, in the whole solution, the GCMouse module belongs to the system module, and the mouse event does not need to go through the H5 web page code of the WebView control and the conversion of the iOS system. Therefore, the mouse control response speed is better than that of the H5 solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic flow chart of a mouse control monitoring method provided by an embodiment of the present application;

[0017] Figure 2 It is a schematic flow chart of another mouse control monitoring method provided by an embodiment of the present application;

[0018] Figure 3 It is a schematic structural diagram of a mouse control monitoring device provided by an embodiment of the present application;

[0019] Figure 4 It is a schematic structural diagram of another mouse control monitoring device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only for the purposes of illustration and description, and are not used to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn to actual scale. The flowcharts used in the present application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and the steps without logical context relationships may be reversed or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

[0021] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and illustrated in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application claimed, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0022] It should be noted that the term "including" will be used in the embodiments of the present application to indicate the existence of the features stated thereafter, but does not exclude the addition of other features.

[0023] Referring to Figure 1 As shown, the embodiments of the present application provide a schematic flowchart of a mouse control monitoring method, which is applied to a mobile device and includes:

[0024] S10. When the mouse mode of the cloud application is the no-mouse mode, monitor the movement of the mouse through the GCMouse module, obtain the offset of the mouse movement, and determine the first intermediate data according to the offset of the mouse movement.

[0025] In this embodiment, it should be noted that the mobile device can be an iPhone with a system version of iOS 14 or above. The iPhone is connected to a physical mouse, and the system will generate a mouse movement event during the movement of the mouse. When the mouse mode of the cloud application is the no-mouse mode (the mouse mode of the cloud application can be informed to the execution entity of step S10 by the cloud server), monitor the mouse movement event through the GCMouse module (a module of the system). After the mouse movement event is monitored, the GCMouse module will call the system callback method to return the offset of the mouse movement to the execution entity of step S10.

[0026] S11. Send the first intermediate data to the cloud server so that the cloud server determines the first mouse movement event according to the first intermediate data, and send the first mouse movement event to the system of the cloud server through the virtual mouse, where the first intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

[0027] In this embodiment, it should be noted that the first intermediate data includes the source data, intermediate result data, or target result data in the process of obtaining the first mouse movement event based on the offset of mouse movement. That is, the first intermediate data includes the offset of mouse movement, or the result of any intermediate step in the process of obtaining the first mouse movement event based on the offset of mouse movement, or the first mouse movement event. The first mouse movement event may include a movement identifier and the offset of the target mouse movement. Among them, the offset of the target mouse movement may be the same as the offset of mouse movement, or the offset of the target mouse movement may be obtained by processing the offset of mouse movement. For example, it may be obtained by magnifying or reducing the offset of mouse movement. By determining the first mouse movement event and sending the first mouse movement event to the system of the cloud server through the virtual mouse, it is possible to simulate mouse movement on the cloud server to control the cloud application running in the cloud server. For example, the perspective of the game character in the cloud game can be rotated.

[0028] In the mouse control monitoring method provided by the embodiment of the present application, when the mouse mode of the cloud application is the no-mouse mode, the GCMouse module monitors the movement of the mouse and obtains the offset of mouse movement. The first intermediate data is determined based on the offset of mouse movement, and the first intermediate data is sent to the cloud server so that the cloud server determines the first mouse movement event based on the first intermediate data. The first mouse movement event is sent to the system of the cloud server through the virtual mouse to control the cloud application. Thus, when the mouse cursor moves to the edge of the iPhone screen and cannot move further outwards, the mouse can still be monitored when it moves outwards, and then the cloud application can be controlled. Moreover, in the whole solution, the GCMouse module belongs to the system module, and the mouse event does not need to go through the H5 web page code of the WebView control and the conversion of the iOS system. Therefore, the mouse control response speed is better than the H5 solution.

[0029] Based on the foregoing method embodiment, the method may further include:

[0030] When the mouse mode of the cloud application is the with-mouse mode, the movement of the mouse cursor is monitored through the UIHoverGestureRecognizer of the hover gesture added on the touch input layer, and the movement position of the mouse cursor on the touch input layer is obtained. The second intermediate data is determined based on the movement position;

[0031] The second intermediate data is sent to the cloud server so that the cloud server determines the second mouse movement event based on the second intermediate data, and the second mouse movement event is sent to the system of the cloud server through the virtual mouse. Among them, the second intermediate data includes the source data, intermediate result data, or target result data in the process of obtaining the second mouse movement event based on the movement position.

[0032] In this embodiment, it should be noted that when the mouse mode of the cloud application is the "with mouse" mode, the cloud server will not render the mouse cursor. That is, there is no mouse cursor in the cloud application screen rendered by the cloud server, and the mouse cursor on the cloud application screen that the user sees on the screen of the mobile device is the mouse cursor of the mouse connected to the mobile device. The hover gesture UIHoverGestureRecognizer is added to the touch input layer. When the mouse mode of the cloud application is the "with mouse" mode, the movement of the mouse cursor on the touch input layer is monitored through the hover gesture UIHoverGestureRecognizer. After the movement of the mouse cursor on the touch input layer is monitored, the hover gesture UIHoverGestureRecognizer will call the hover gesture callback method, and the hover gesture callback method will return a gesture object to the execution entity of this embodiment. If the state attribute of the gesture object (indicating the state of the mouse cursor hovering) characterizes that the mouse cursor is moving, the method of the hover gesture can be called through the gesture object to obtain the position of the cursor on the touch input layer, that is, the movement position of the mouse cursor on the touch input layer. The second intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the second mouse movement event according to the movement position. That is, the second intermediate data includes the movement position or the result of any intermediate step in the process of obtaining the second mouse movement event according to the movement position or the second mouse movement event. The second mouse movement event may include a movement identifier and a target movement position. It can be understood that the sizes of the touch input layer of the iPhone and the cloud application screen rendered by the cloud server may not be the same. Therefore, it is necessary to find the corresponding position on the cloud application screen rendered by the cloud server according to the movement position on the touch input layer, and this position is the target movement position. For example, the size of the touch input layer is the same as that of the rendering screen layer of the iPhone, and the aspect ratio of the cloud application screen displayed on the iPhone is the same as that of the cloud application screen rendered by the cloud server. Taking the position corresponding to the vertex in the upper left corner of the touch input layer as the coordinate origin, the axis where the width of the touch input layer is located as the x-axis, and the axis where the height of the touch input layer is located as the y-axis, denoting the width and height of the cloud application screen displayed on the iPhone as a1 and b1 respectively, and the width and height of the cloud application screen rendered by the cloud server as a2 and b2 respectively, the coordinates of the movement position are (x1, y1), and the coordinates of the target movement position are (x2, y2), then x2 = x1 × a2 / a1, y2 = y1 × b2 / b1. By determining the second mouse movement event and sending the second mouse movement event to the system of the cloud server through the virtual mouse, it is possible to simulate mouse movement on the cloud server to control the cloud application.

[0033] Based on the foregoing method embodiment, the method may further include:

[0034] Obtain a touch event. If it is determined according to the touch event that the control type is mouse control, create a mouse event touch model, add the mouse event touch model to a preset queue, and start a timer. The mouse event touch model includes control coordinates, control action type, creation time, status, and mouse button type, and the status is initially in a waiting state;

[0035] For each mouse event touch model in the queue, determine whether the corresponding mouse button type returned by the system callback method called by the GCMouse module is received before the time set by the timer corresponding to the mouse event touch model arrives;

[0036] If the corresponding mouse button type returned by the system callback method called by the GCMouse module is not received before the time set by the timer corresponding to the mouse event touch model arrives, then after the time set by the timer corresponding to the mouse event touch model arrives, it is determined whether there is a first mouse event touch model in the queue. If there is no first mouse event touch model in the queue, then third intermediate data is determined according to the mouse event touch model, and the third intermediate data is sent to the cloud server so that the cloud server determines a first left mouse button event according to the third intermediate data and sends the first left mouse button event to the system of the cloud server through the virtual mouse. Or if there is a first mouse event touch model in the queue, then fourth intermediate data is determined according to the mouse event touch model, fifth intermediate data is determined according to the first mouse event touch model, and the fourth intermediate data and the fifth intermediate data are sent to the cloud server so that the cloud server determines a second left mouse button event according to the fourth intermediate data, determines a first mouse button event according to the fifth intermediate data, and sends the second left mouse button event and the first mouse button event to the system of the cloud server through the virtual mouse in the order of the creation time. Among them, the first mouse event touch model is a mouse event touch model in the queue whose creation time is later than the creation time of the mouse event touch model, whose state is the waiting previous event state, and there is no second mouse event touch model between it and the mouse event touch model. The second mouse event touch model is a mouse event touch model in the queue whose state is not the waiting previous event state. The third intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first left mouse button event according to the mouse event touch model. The fourth intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the second left mouse button event according to the mouse event touch model. The fifth intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse button event according to the first mouse event touch model; or

[0037] If, before the time set by the timer corresponding to the touch model of the mouse event arrives, the corresponding mouse button type returned by the system callback method called by the GCMouse module is received, the mouse button type in the touch model of the mouse event is set to the corresponding mouse button type returned by the system callback method, and it is determined whether there is a third touch model of the mouse event in the queue. If there is no third touch model of the mouse event in the queue, the sixth intermediate data is determined according to the touch model of the mouse event, and the sixth intermediate data is sent to the cloud server so that the cloud server can determine the second mouse button event according to the sixth intermediate data and send the second mouse button event to the system of the cloud server through the virtual mouse. Or if there is a third touch model of the mouse event in the queue, the status in the touch model of the mouse event is modified to the waiting previous event status, and the timer corresponding to the touch model of the mouse event is stopped, where the third touch model of the mouse event is the touch model of the mouse event in the queue whose creation time is earlier than the creation time of the touch model of the mouse event and whose status is the waiting status, and the sixth intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the second mouse button event according to the touch model of the mouse event.

[0038] In this embodiment, it should be noted that when the touch input layer is manipulated by a finger or a mouse, the system will generate a touch event. Specifically, when the touch input layer is clicked by a finger, or the left, middle, and right mouse buttons are clicked on the touch input layer, the system will generate a touch event. GCMouse can monitor the touch event, but based on the touch event, the GCMouse module can only recognize the middle mouse button operation and the right mouse button operation, and cannot recognize the left mouse button operation. Therefore, in this embodiment, the left mouse button operation, the middle mouse button operation, and the right mouse button operation are defined based on the touch event and the GCMouse module. Specifically, after each touch event is generated by the system, the generated touch event is returned to the execution entity of this embodiment through the system callback method. Then, the manipulation type can be determined according to the touch event. Specifically, when it is determined that the type in the touch event is not UITouchTypeDirect direct click or the majorRadiusTolerance radius ≤ 0, the manipulation type is determined to be mouse manipulation; in other cases, the manipulation type is determined to be finger manipulation. After the manipulation type is determined to be mouse manipulation, a mouse event touch model is created according to the information of the touch event, and the mouse event touch model is added to a preset queue, and a timer is started. The mouse event touch model can include the manipulation coordinates, the manipulation action type, the creation time, the status, and the mouse button type. The manipulation coordinates are the position of the mouse cursor on the touch input layer (which can be obtained by calling the method of the touch event object), the creation time is the time when the mouse event touch model is created (which can be a timestamp), and the status is initially the waiting state. The manipulation action type includes down and up. For the manipulation action type, after the touch event is generated, if it is determined that the touch begin method is called, the manipulation action type is determined to be down; if it is determined that the touchend method and the touch cancel method are called, the manipulation action type is determined to be up. It should be noted that if after the touch event is generated, none of the touch begin method, the touch end method, and the touch cancel method are called, and the touch move method is called, no corresponding mouse event touch model will be created for the generated touch event this time. The timer is started for the corresponding mouse event touch model, and the time set by the timer can be set as needed, such as set to 10 ms.

[0039] Each time after adding a mouse event touch model (hereinafter simply referred to as "model M" for ease of description) into a preset queue and starting a timer (hereinafter simply referred to as "timer t"), it is necessary to determine whether the corresponding mouse button type returned by the system callback method called by the GCMouse module is received before the time set by timer t. If the corresponding mouse button type returned by the system callback method called by the GCMouse module is not received, then after the time set by timer t arrives, it is judged whether there is a first mouse event touch model in the queue. If there is no first mouse event touch model in the queue, then the third intermediate data is determined according to model M, and the third intermediate data is sent to the cloud server so that the cloud server determines the first mouse left button event according to the third intermediate data, and sends the first mouse left button event to the system of the cloud server through the virtual mouse. Among them, the first mouse event touch model is a mouse event touch model in the queue whose creation time is later than the creation time of model M, whose state is the waiting previous event state, and there is no second mouse event touch model between it and model M. The second mouse event touch model is a mouse event touch model in the queue whose state is not the waiting previous event state. The third intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse left button event according to model M, that is, the third intermediate data includes model M or the result of any intermediate step in the process of obtaining the first mouse left button event according to model M or the first mouse left button event. The first mouse left button event may include a mouse left button identifier, an identifier of the manipulation action type in model M, and the coordinates corresponding to the manipulation coordinates in model M on the cloud application screen rendered by the cloud server (the determination process of this coordinate is the same as that of the target movement position coordinate in the foregoing embodiment and will not be elaborated here). After sending the third intermediate data to the cloud server, the state in model M can be modified to the sent state, model M in the queue can be deleted, the timer corresponding to model M can be stopped, and the timer resources can be released. The mouse button type in each mouse event touch model in the queue can be initially the mouse left button, then the mouse left button identifier in the first mouse left button event can be determined according to the mouse button type in model M. For example, it can be the mouse button type in model M or the mouse button type in model M is converted into a mouse left button identifier recognizable by the system of the cloud server. Or the mouse button type in each mouse event touch model in the queue can be initially a null value, then when determining the third intermediate data according to model M, the mouse button type in model M can be set to the mouse left button, and the mouse left button identifier in the first mouse left button event can be determined according to the mouse button type in model M. After determining the first mouse left button event and sending the first mouse left button event to the system of the cloud server through the virtual mouse, it is possible to simulate the mouse left button manipulation on the cloud server to manipulate the cloud application.If there is a first mouse event touch model in the queue, the fourth intermediate data is determined according to model M, the fifth intermediate data is determined according to the first mouse event touch model, and the fourth intermediate data and the fifth intermediate data are sent to the cloud server, so that the cloud server determines the second left mouse button event according to the fourth intermediate data, determines the first mouse button event according to the fifth intermediate data, and sends the second left mouse button event and the first mouse button event to the system of the cloud server through the virtual mouse in the order of the creation time. For example, there are 2 first mouse event touch models in the queue: the first mouse event touch model 1 and the first mouse event touch model 2. The creation time of the first mouse event touch model 1 is earlier than that of the first mouse event touch model 2. Then, the fourth intermediate data is determined according to model M, the fifth intermediate data 1 is determined according to the first mouse event touch model 1, the fifth intermediate data 2 is determined according to the first mouse event touch model 2, and the fourth intermediate data, the fifth intermediate data 1 and the fifth intermediate data 2 are sent to the cloud server. After receiving the fourth intermediate data, the cloud server will determine the second left mouse button event according to the fourth intermediate data. After receiving the fifth intermediate data 1, the cloud server will determine the first mouse button event 1 according to the fifth intermediate data 1. After receiving the fifth intermediate data 2, the cloud server will determine the first mouse button event 2 according to the fifth intermediate data 2. When the cloud server sends data, it will send the second left mouse button event first, then the first mouse button event 1, and finally the first mouse button event 2 in sequence to the system of the cloud server through the virtual mouse. The fourth intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the second left mouse button event according to model M, that is, the fourth intermediate data includes model M or the result of any intermediate step in the process of obtaining the second left mouse button event according to model M or the second left mouse button event. The second left mouse button event may include the left mouse button identifier, the identifier of the manipulation action type in model M, and the coordinates corresponding to the manipulation coordinates in model M on the cloud application screen rendered by the cloud server.The fifth intermediate data includes the source data, intermediate result data, or target result data during the process of obtaining the first mouse button event based on the first mouse event touch model. That is, the fifth intermediate data includes the first mouse event touch model, or the result of any intermediate step during the process of obtaining the first mouse button event based on the first mouse event touch model, or the first mouse button event. The first mouse button event may include the mouse button type identifier corresponding to the mouse button type in the first mouse event touch model (for example, if the mouse button type in the first mouse event touch model is the right mouse button, then the corresponding mouse button type identifier is the right mouse button identifier recognizable by the system of the cloud server; another example is that if the mouse button type in the first mouse event touch model is the middle mouse button, then the corresponding mouse button type identifier is the middle mouse button identifier recognizable by the system of the cloud server), the identifier of the manipulation action type in the first mouse event touch model, and the coordinates corresponding to the manipulation coordinates in the first mouse event touch model on the cloud application screen rendered by the cloud server. After sending the fourth intermediate data to the cloud server, the state in model M can be modified to the sent state, model M in the queue can be deleted, the timer corresponding to model M can be stopped, and the timer resources can be released. After sending the fifth intermediate data to the cloud server, the state in the first mouse event touch model can be modified to the sent state, and the first mouse event touch model in the queue can be deleted. Through the logic of this embodiment, it can be ensured that the manipulation order of the left, right, and middle mouse buttons simulated on the cloud server is consistent with the actual manipulation order.

[0040] If the corresponding mouse button type (such as the right mouse button or the middle mouse button) returned by the system callback method called by the GCMouse module is received before the time set by the timer t arrives, the mouse button type in the model M is set to the corresponding mouse button type returned by the system callback method, and it is determined whether there is a mouse event touch model (i.e., the third mouse event touch model) in the queue whose creation time is earlier than the creation time in the model M and whose status is the waiting state. If there is no third mouse event touch model in the queue, the sixth intermediate data is determined according to the model M, and the sixth intermediate data is sent to the cloud server so that the cloud server can determine the second mouse button event according to the sixth intermediate data, and the second mouse button event is sent to the system of the cloud server through the virtual mouse. Among them, the determination process of the sixth intermediate data can be the same as the determination process of the fifth intermediate data, and the determination process of the second mouse button event can be the same as the determination process of the first mouse button event, which will not be elaborated here. After sending the sixth intermediate data to the cloud server, the status in the model M can be modified to the sent state, the model M in the queue is deleted, the timer corresponding to the model M is stopped, and the timer resources are released. If there is a third mouse event touch model in the queue, the status in the model M is modified to the waiting for previous event state, and the timer corresponding to the model M is stopped, and the timer resources are released. In addition, it should be noted that for a certain touch event generated by the system, the time interval between the result returned by the GCMouse module calling the system callback method after monitoring the touch event and the system calling the system callback method to return the touch event to the execution subject of this embodiment is generally no more than 1 millisecond. Therefore, after receiving the result returned by the GCMouse module calling the system callback method, it can be matched with a mouse event touch model in the queue. Specifically, the mouse event touch model in the queue whose creation time is not later than the first time (the first time is the time when the result returned by the GCMouse module calling the system callback method is received) and the closest to the first time can be used as the target mouse event touch model. Then, the result returned by the GCMouse module calling the system callback method can be used to set the mouse button type in the target mouse event touch model, and the mouse button type in the model M is set to the mouse button type (right mouse button or middle mouse button) returned by the GCMouse module calling the system callback method.

[0041] Based on the foregoing method embodiments, the method may further include:

[0042] Monitor the sliding of the mouse wheel through the GCMouse module, obtain the amplitude of the mouse wheel sliding, and determine the seventh intermediate data according to the amplitude of the mouse wheel sliding;

[0043] Send the seventh intermediate data to the cloud server so that the cloud server determines a mouse wheel scrolling event based on the seventh intermediate data and sends the mouse wheel scrolling event to the system of the cloud server through a virtual mouse. The seventh intermediate data includes source data or intermediate result data or target result data in the process of obtaining the mouse wheel scrolling event based on the amplitude of the mouse wheel scrolling.

[0044] In this embodiment, it should be noted that when the mouse wheel is scrolled, the system will generate a wheel event, and the GCMouse module can monitor the wheel event. After monitoring the wheel event, the GCMouse module will call the system callback method, and the callback method will return the amplitude of the mouse wheel scrolling of the execution subject in this embodiment. The seventh intermediate data includes source data or intermediate result data or target result data in the process of obtaining the mouse wheel scrolling event based on the amplitude of the mouse wheel scrolling, that is, the seventh intermediate data includes the amplitude of the mouse wheel scrolling or the result of any intermediate step in the process of obtaining the mouse wheel scrolling event based on the amplitude of the mouse wheel scrolling or the mouse wheel scrolling event. The mouse wheel scrolling event may include the identifier of the mouse wheel and the target amplitude of the mouse wheel scrolling. The target amplitude of the mouse wheel scrolling may be the amplitude of the mouse wheel scrolling or obtained by processing the amplitude of the mouse wheel scrolling, and the processing includes zooming in or out, etc. After determining the mouse wheel scrolling event and sending the mouse wheel scrolling event to the system of the cloud server through a virtual mouse, it is possible to simulate mouse wheel control on the cloud server to control cloud applications.

[0045] Refer to Figure 2 As shown, an embodiment of the present application provides a flowchart of a method for monitoring mouse control, which is applied to a cloud server and includes:

[0046] S20. When the mouse mode of the cloud application is the no-mouse mode, receive the first intermediate data sent by the mobile device, determine a first mouse movement event based on the first intermediate data, and send the first mouse movement event to the system of the cloud server through a virtual mouse. The first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event based on the offset of the mouse movement.

[0047] The mouse control monitoring method provided by the embodiment of the present application, when the mouse mode of the cloud application is the no-mouse mode, receives the first intermediate data sent by the mobile device, determines the first mouse movement event according to the first intermediate data, and sends the first mouse movement event to the system of the cloud server through the virtual mouse to control the cloud application. Since the first intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement, when the mouse cursor moves to the edge of the iPhone screen and cannot move outwards any further, moving the mouse outwards can still monitor the movement of the mouse and then control the cloud application. Moreover, in the whole solution, the GCMouse module belongs to the system module, and the mouse event does not need to be converted by the H5 web page code of the WebView control and the iOS system. Therefore, the mouse control response speed is better than the H5 solution.

[0048] Based on the foregoing method embodiment, the method may further include:

[0049] When the mouse mode of the cloud application is the with-mouse mode, receive the second intermediate data sent by the mobile device, determine the second mouse movement event according to the second intermediate data, and send the second mouse movement event to the system of the cloud server through the virtual mouse, where the second intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the second mouse movement event according to the movement position of the mouse cursor on the touch input layer.

[0050] Based on the foregoing method embodiment, the method may further include:

[0051] Receive the third intermediate data sent by the mobile device, determine the first mouse left button event according to the third intermediate data, and send the first mouse left button event to the system of the cloud server through the virtual mouse, where the third intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the first mouse left button event according to a mouse event touch model in a preset queue; or

[0052] A system that receives the fourth intermediate data and the fifth intermediate data sent by a mobile device, determines a second left mouse button event based on the fourth intermediate data, determines a first mouse button event based on the fifth intermediate data, and sends the second left mouse button event and the first mouse button event to a cloud server through a virtual mouse in the order of their corresponding creation times. The fourth intermediate data includes source data or intermediate result data or target result data in the process of obtaining the second left mouse button event according to a touch model of a mouse event in a preset queue. The fifth intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse button event according to a touch model of a first mouse event. The touch model of the first mouse event is a touch model of a mouse event in the queue whose creation time is later than that of the touch model of the one mouse event, whose status is waiting for a previous event status, and there is no second mouse event touch model between it and the touch model of the one mouse event. The touch model of the second mouse event is a touch model of a mouse event in the queue whose status is not waiting for a previous event status; or

[0053] A system that receives the sixth intermediate data sent by a mobile device, determines a second mouse button event based on the sixth intermediate data, and sends the second mouse button event to a cloud server through a virtual mouse. The sixth intermediate data includes source data or intermediate result data or target result data in the process of obtaining the second mouse button event according to a touch model of a mouse event in a preset queue.

[0054] Based on the foregoing method embodiments, the method may further include:

[0055] A system that receives the seventh intermediate data sent by a mobile device, determines a mouse wheel scrolling event based on the seventh intermediate data, and sends the mouse wheel scrolling event to a cloud server through a virtual mouse. The seventh intermediate data includes source data or intermediate result data or target result data in the process of obtaining the mouse wheel scrolling event according to the amplitude of the mouse wheel scrolling.

[0056] Refer to Figure 3 As shown, an embodiment of the present application provides a structural schematic diagram of a mouse control monitoring device, which is applied to a mobile device and includes:

[0057] An obtaining unit 30, configured to monitor the movement of a mouse through a GCMouse module and obtain the offset of the mouse movement when the mouse mode of the cloud application is the no mouse mode, and determine first intermediate data according to the offset of the mouse movement;

[0058] A sending unit 31, configured to send first intermediate data to a cloud server, so that the cloud server determines a first mouse movement event according to the first intermediate data, and sends the first mouse movement event to a system of the cloud server through a virtual mouse, where the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of mouse movement.

[0059] When the mouse mode of a cloud application is a mouse - less mode, the mouse control monitoring device provided in an embodiment of this application monitors the movement of a mouse through a GCMouse module, obtains the offset of mouse movement, determines first intermediate data according to the offset of mouse movement, and sends the first intermediate data to a cloud server, so that the cloud server determines a first mouse movement event according to the first intermediate data, and sends the first mouse movement event to a system of the cloud server through a virtual mouse to control the cloud application. Thus, when the mouse cursor moves to the edge of the iPhone screen and cannot move further outwards, moving the mouse outwards can still monitor the mouse movement and then control the cloud application. Moreover, in the whole solution, the GCMouse module belongs to a system module, and mouse events do not need to go through the H5 web page code of a WebView control and the conversion of the iOS system. Therefore, the mouse control response speed is better than that of the H5 solution.

[0060] Refer to Figure 4 As shown in the figure, an embodiment of this application provides a schematic structural diagram of a mouse control monitoring device, which is applied to a cloud server and includes:

[0061] A receiving unit 40, configured to receive first intermediate data sent by a mobile device when the mouse mode of a cloud application is a mouse - less mode, determine a first mouse movement event according to the first intermediate data, and send the first mouse movement event to a system of the cloud server through a virtual mouse, where the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of mouse movement.

[0062] The mouse control monitoring device provided by the embodiment of the present application receives the first intermediate data sent by a mobile device when the mouse mode of a cloud application is the mouse-free mode, determines a first mouse movement event according to the first intermediate data, and sends the first mouse movement event to the system of a cloud server through a virtual mouse to control the cloud application. Since the first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of mouse movement, when the mouse cursor moves to the edge of the iPhone screen and cannot move further outwards, moving the mouse outwards can still monitor the movement of the mouse and then control the cloud application. Moreover, in the whole solution, the GCMouse module belongs to the system module, and the mouse event does not need to be converted by the H5 web page code of the WebView control and the iOS system. Therefore, the mouse control response speed is better than that of the H5 solution.

[0063] The implementation process of the mouse control monitoring device provided by the embodiment of the present application is the same as that of the mouse control monitoring method provided by the embodiment of the present application, and the achieved effects are also the same as those of the mouse control monitoring method provided by the embodiment of the present application, and will not be elaborated here.

[0064] The above are only the specific implementation manners of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A mouse control monitoring method, characterized in that, Applied to a mobile device, including: When the mouse mode of the cloud application is the no - mouse mode, listen for the movement of the mouse through the GCMouse module, obtain the offset of the mouse movement, and determine the first intermediate data according to the offset of the mouse movement; Send the first intermediate data to the cloud server so that the cloud server determines the first mouse movement event according to the first intermediate data, and send the first mouse movement event to the system of the cloud server through the virtual mouse. Among them, the first intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement.

2. The method according to claim 1, characterized in that, It also includes: When the mouse mode of the cloud application is the with - mouse mode, listen for the movement of the mouse cursor through the UIHoverGestureRecognizer added on the touch input layer, obtain the movement position of the mouse cursor on the touch input layer, and determine the second intermediate data according to the movement position; Send the second intermediate data to the cloud server so that the cloud server determines the second mouse movement event according to the second intermediate data, and send the second mouse movement event to the system of the cloud server through the virtual mouse. Among them, the second intermediate data includes the source data or the intermediate result data or the target result data in the process of obtaining the second mouse movement event according to the movement position.

3. The method according to claim 1 or 2, characterized in that, It also includes: Obtain the touch event. If it is determined that the control type is mouse control according to the touch event, create a mouse event touch model, add the mouse event touch model to the preset queue, and start a timer. Among them, the mouse event touch model includes the control coordinates, the control action type, the creation time, the status, and the mouse button type, and the status is initially the waiting state; For each mouse event touch model in the queue, determine whether the corresponding mouse button type returned by the system callback method called by the GCMouse module is received before the time set by the timer corresponding to the mouse event touch model arrives; If the corresponding mouse button type returned by the system callback method called by the GCMouse module is not received before the time set by the timer corresponding to the touch model of the mouse event arrives, then after the time set by the timer corresponding to the touch model of the mouse event arrives, it is determined whether there is a first mouse event touch model in the queue. If there is no first mouse event touch model in the queue, the third intermediate data is determined according to the touch model of the mouse event, and the third intermediate data is sent to the cloud server, so that the cloud server determines the first left mouse button event according to the third intermediate data, and sends the first left mouse button event to the system of the cloud server through the virtual mouse. Or if there is a first mouse event touch model in the queue, the fourth intermediate data is determined according to the touch model of the mouse event, the fifth intermediate data is determined according to the first mouse event touch model, and the fourth intermediate data and the fifth intermediate data are sent to the cloud server, so that the cloud server determines the second left mouse button event according to the fourth intermediate data, determines the first mouse button event according to the fifth intermediate data, and sends the second left mouse button event and the first mouse button event to the system of the cloud server through the virtual mouse in the order of the creation time. Among them, the first mouse event touch model is the mouse event touch model in the queue whose creation time is later than the creation time of the touch model of the mouse event, the state is the waiting previous event state, and there is no second mouse event touch model between it and the touch model of the mouse event. The second mouse event touch model is the mouse event touch model in the queue whose state is not the waiting previous event state. The third intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first left mouse button event according to the touch model of the mouse event. The fourth intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the second left mouse button event according to the touch model of the mouse event. The fifth intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse button event according to the first mouse event touch model; or If, before the time set by the timer corresponding to the touch model of the mouse event arrives, the corresponding mouse button type returned by the system callback method called by the GCMouse module is received, the mouse button type in the touch model of the mouse event is set to the corresponding mouse button type returned by the system callback method, and it is determined whether there is a third touch model of the mouse event in the queue. If there is no third touch model of the mouse event in the queue, the sixth intermediate data is determined according to the touch model of the mouse event, and the sixth intermediate data is sent to the cloud server so that the cloud server determines the second mouse button event according to the sixth intermediate data, and the second mouse button event is sent to the system of the cloud server through the virtual mouse. Or if there is a third touch model of the mouse event in the queue, the state in the touch model of the mouse event is modified to the waiting previous event state, and the timer corresponding to the touch model of the mouse event is stopped, where the third touch model of the mouse event is the touch model of the mouse event in the queue whose creation time is earlier than the creation time of the touch model of the mouse event and whose state is the waiting state, and the sixth intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the second mouse button event according to the touch model of the mouse event.

4. The method according to claim 1, wherein It further includes: Listening to the scrolling of the mouse wheel through the GCMouse module, obtaining the amplitude of the mouse wheel scrolling, and determining the seventh intermediate data according to the amplitude of the mouse wheel scrolling; Sending the seventh intermediate data to the cloud server so that the cloud server determines the mouse wheel scrolling event according to the seventh intermediate data, and sending the mouse wheel scrolling event to the system of the cloud server through the virtual mouse, where the seventh intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the mouse wheel scrolling event according to the amplitude of the mouse wheel scrolling.

5. A mouse control monitoring method, characterized in that, Applied to the cloud server, it includes: When the mouse mode of the cloud application is the no-mouse mode, receiving the first intermediate data sent by the mobile device, determining the first mouse movement event according to the first intermediate data, and sending the first mouse movement event to the system of the cloud server through the virtual mouse, where the first intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of the mouse movement, and the first intermediate data is determined by the mobile device listening to the movement of the mouse through the GCMouse module and obtaining the offset of the mouse movement.

6. The method according to claim 5, characterized in that, It further includes: When the mouse mode of the cloud application is the with-mouse mode, receiving the second intermediate data sent by the mobile device, determining the second mouse movement event according to the second intermediate data, and sending the second mouse movement event to the system of the cloud server through the virtual mouse, where the second intermediate data includes the source data or intermediate result data or target result data in the process of obtaining the second mouse movement event according to the movement position of the mouse cursor on the touch input layer.

7. The method according to claim 5 or 6, characterized in that, It further includes: A system that receives the third intermediate data sent by a mobile device, determines a first left mouse button event based on the third intermediate data, and sends the first left mouse button event to a cloud server via a virtual mouse. The third intermediate data includes source data, intermediate result data, or target result data during the process of obtaining the first left mouse button event according to a mouse event touch model in a preset queue; or A system that receives the fourth intermediate data and the fifth intermediate data sent by a mobile device, determines a second left mouse button event based on the fourth intermediate data, determines a first mouse button event based on the fifth intermediate data, and sends the second left mouse button event and the first mouse button event to a cloud server via a virtual mouse in the order of their corresponding creation times. The fourth intermediate data includes source data, intermediate result data, or target result data during the process of obtaining the second left mouse button event according to a mouse event touch model in a preset queue. The fifth intermediate data includes source data, intermediate result data, or target result data during the process of obtaining the first mouse button event according to a first mouse event touch model. The first mouse event touch model is a mouse event touch model in the queue whose creation time is later than that of the mouse event touch model, whose status is waiting for a previous event status, and there is no second mouse event touch model between it and the mouse event touch model. The second mouse event touch model is a mouse event touch model in the queue whose status is not waiting for a previous event status; or A system that receives the sixth intermediate data sent by a mobile device, determines a second mouse button event based on the sixth intermediate data, and sends the second mouse button event to a cloud server via a virtual mouse. The sixth intermediate data includes source data, intermediate result data, or target result data during the process of obtaining the second mouse button event according to a mouse event touch model in a preset queue.

8. The method according to claim 5, wherein It further includes: A system that receives the seventh intermediate data sent by a mobile device, determines a mouse wheel scrolling event based on the seventh intermediate data, and sends the mouse wheel scrolling event to a cloud server via a virtual mouse. The seventh intermediate data includes source data, intermediate result data, or target result data during the process of obtaining the mouse wheel scrolling event according to the amplitude of the mouse wheel scrolling.

9. A mouse control monitoring device, characterized in that, Applied to a mobile device, it includes: An acquisition unit, which is used to listen for the movement of the mouse through the GCMouse module and acquire the offset of the mouse movement when the mouse mode of the cloud application is the no-mouse mode, and determine the first intermediate data according to the offset of the mouse movement; A sending unit, which is used to send the first intermediate data to a cloud server, so that the cloud server determines a first mouse movement event based on the first intermediate data and sends the first mouse movement event to a cloud server via a virtual mouse. The first intermediate data includes source data, intermediate result data, or target result data during the process of obtaining the first mouse movement event according to the offset of the mouse movement.

10. A mouse control monitoring device, characterized in that, Applied to a cloud server, it includes: A receiving unit, which is configured to receive first intermediate data sent by a mobile device when the mouse mode of a cloud application is the mouse-free mode, determine a first mouse movement event according to the first intermediate data, and send the first mouse movement event to the system of a cloud server through a virtual mouse. The first intermediate data includes source data or intermediate result data or target result data in the process of obtaining the first mouse movement event according to the offset of mouse movement. The first intermediate data is determined by the mobile device listening to the movement of the mouse through the GCMouse module, obtaining the offset of mouse movement, and determining according to the offset of mouse movement.

Images