Cross-terminal unified operation transmission system and method in cloud game scene

Abstract

The invention discloses a cross-terminal unified operation transmission system and method in a cloud game scene. The system comprises a client and a cloud game server; the client comprises a unified input capture module, an operation abstraction module, a coordinate scaling module and a binary coding module; the cloud game server comprises a binary decoding module, an event state management module, a coordinate restoration module and an event injection module. The method has the advantages that the operation delay is remarkably reduced, the data volume of a single operation message can be reduced by more than 75% by replacing JSON with a customized binary protocol, the network transmission time is shortened, the end-to-end operation delay is remarkably reduced, and the response speed and the hand feeling of the operation are greatly improved. Through operation abstraction and coordinate scaling of the client, the server does not need to perceive which equipment or input mode the front end is, all inputs are normalized, and logic complexity and code coupling degree of the server are greatly reduced.

Description

Technical Field

[0001] This invention relates to the fields of computer network communication and cloud computing technology, specifically to a unified operation and transmission system and method for cross-terminal operation in a cloud gaming scenario. Background Technology

[0002] Cloud gaming is an interactive application that performs game computation and rendering on cloud servers and transmits the rendered game footage to the user's terminal as a video stream. In this mode, the user's input commands on the user's terminal need to be transmitted quickly and accurately to the cloud server to drive the game process. The latency of input transmission is a core indicator that determines the cloud gaming experience, especially the responsiveness.

[0003] In existing technologies, the commonly used operation data transmission scheme is to use JSON (JavaScript Object Notation) format. For example, a user's touch press operation on the user's terminal may be encoded as {"action": "down", "pointerId": 1, "x": 100, "y": 100}.

[0004] The above solution has the following serious flaws:

[0005] 1. Data Redundancy and High Latency: JSON, as a text format, contains a large number of structured characters, such as {}, "", and :, as well as repeated key names, such as action and pointerId. In multi-touch or high-frequency operation scenarios, such as rapid swiping in first-person shooter games, a large amount of such operation data is generated instantaneously, resulting in excessively large operation data packets. In environments with limited network bandwidth, especially mobile networks, this significantly increases the burden and time of network transmission of operation data, causing users to experience severe operation delays and stuttering, i.e., "unresponsive operation".

[0006] 2. Poor cross-platform compatibility and high maintenance costs: Cloud gaming services have expanded from the single Android platform to iOS, H5 web pages (running on Windows / macOS), etc. The user input sources differ fundamentally across platforms: mobile devices primarily use multi-touch, while PC web pages mainly use keyboards and mice. Keyboard operations involve numerous key presses and releases, while mouse operations involve clicking, moving, and dragging. Furthermore, different terminal devices have varying screen sizes, resolutions, and aspect ratios. If the game screen is scaled non-uniformly to fit the terminal screen, such as filling the entire screen horizontally or vertically, directly transmitting the terminal's absolute coordinates will result in the game not being mapped to the correct in-game position in the cloud. Implementing a separate set of event parsing, coordinate transformation, and event injection logic on the cloud gaming server for each newly added platform leads to extremely complex and highly coupled code, resulting in high maintenance and iteration costs.

[0007] Therefore, there is an urgent need for a new technical solution that can significantly reduce operation latency and uniformly handle input from multiple platforms and devices in a low-cost manner. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a unified operation and transmission system and method for cross-terminal cloud gaming scenarios, which can effectively solve the above-mentioned problems.

[0009] The technical solution adopted in this invention is as follows:

[0010] This invention provides a unified operation and transmission system across terminals in a cloud gaming scenario, including a client and a cloud gaming server;

[0011] The client includes a unified input capture module, an operation abstraction module, a coordinate scaling module, and a binary encoding module;

[0012] The unified input capture module is used to listen for and capture raw physical input events generated by the user's interaction with various types of input devices; the raw physical input events include the following types: touch screen input events, mouse input events, and keyboard input events.

[0013] The operation abstraction module is used to convert the captured raw physical input events of various types into predefined standardized operation events; the standardized operation events are represented as {action type, touch ID, absolute coordinates}; wherein, the action type includes press DOWN, release UP, and move MOVE types;

[0014] The coordinate scaling module is used to convert the absolute coordinates in the standardized operation event into scaled coordinates relative to the effective display area of ​​the game, thereby obtaining the standardized operation event {action type, touch ID, scaled coordinates} after coordinate scaling.

[0015] The binary encoding module is used to encode the standardized operation event {action type, touch point ID, scaled coordinates} into a binary format according to a predefined compact binary format, generating a compact binary data stream; the binary data stream is sent to the cloud gaming server through the client's network interface;

[0016] The cloud gaming server includes a binary decoding module, an event state management module, a coordinate restoration module, and an event injection module;

[0017] The binary decoding module is used to receive the binary data stream from the client, and parse the binary data stream according to a predefined binary format to extract the action type, touch point ID and scaled coordinates;

[0018] The event state management module is used to track the state of each touch point in memory according to the touch point ID, and then reconstruct a series of discrete single-point action messages into a complete action gesture that conforms to the operating system specifications.

[0019] The coordinate restoration module is used to restore the extracted proportional coordinates to absolute coordinates in the cloud virtual environment screen.

[0020] The event injection module is used to call the underlying system interface of the cloud virtual environment, simulate an operation containing the restored absolute coordinates and complete action gestures as one or a series of native touch events, and inject them into the target game application.

[0021] Furthermore, the touchscreen input event is a touch event of the touchscreen; the mouse input event is a click and move event of the mouse; and the keyboard input event is a key press event of the keyboard.

[0022] Furthermore, the operation abstraction module is specifically used for:

[0023] The standardized operation events are standardized touch operation events;

[0024] If the captured raw physical input event is a touchscreen input event, then the standardized touch operation event is obtained directly;

[0025] If the captured original physical input event is a keyboard input event or a mouse input event, then perform logical judgment and data conversion of keyboard coordinates or mouse finger representation to convert it into the standardized touch operation event.

[0026] Furthermore, the keyboard coordinate mapping is defined as mapping discrete keyboard key events to continuous touch events at a preset or user-defined coordinate point on the screen; wherein the keyboard key events include keyboard key press events and keyboard key release events; and the touch events are DOWN and UP actions occurring at the touch point.

[0027] The mouse finger simulation is as follows: the mouse trajectory and mouse button states are simulated as single-finger touch behavior, pressing the left mouse button corresponds to the DOWN action, pressing and moving corresponds to the continuous MOVE action, and releasing the left mouse button corresponds to the UP action.

[0028] Furthermore, the coordinate scaling module converts the absolute coordinates (x, y) in the standardized operation event into scaled coordinates (axialX, axialY) relative to the effective display area of ​​the game; where axialX represents the X coordinate scale and axialY represents the Y coordinate scale; the specific conversion method is as follows:

[0029] Set the screen resolution to width L1 x height H1; the effective display area size for the game is width L2 x height H2; the center of the effective display area for the game coincides with the center of the screen; the absolute coordinates (x, y) of the touch point ID are the screen coordinates;

[0030] a. If the effective display area size of the game is the same as the screen resolution, then axialX = x / L²; axialY = y / H².

[0031] b. If the game screen resolution and the screen resolution are not perfectly matched, with the game screen height filling the entire screen but the game screen width not filling the entire screen (i.e., height H1 and height H2 are the same, and width L2 is less than width L1), then:

[0032] (1) If x does not enter the game's effective display area and is to the left of the game's effective display area, then reset x=0 and the y value remains unchanged; then: axialX = 0 / L 2 = 0; axialY = y / H2;

[0033] (2) If x enters the effective display area of ​​the game, the actual horizontal coordinate within the effective display area of ​​the game needs to be x minus the length outside the game screen area, that is, realx = x - (L1 - L2) / 2; then: axialX = realx / L2; axialY = y / H2;

[0034] (3) If x does not enter the effective display area of ​​the game and is on the right side of the effective display area of ​​the game, then reset x = width L 2, and the y value remains unchanged; then: axialX = L 2 / L 2 = 1; axialY = y / H2;

[0035] c. If the game screen resolution and the screen resolution are not perfectly matched, with the game screen height filling the entire screen but the game screen height not filling the entire screen (i.e., height H2 is less than height H1, and width L2 equals width L1), then:

[0036] (1) If y does not enter the game's effective display area and is above the effective display area, then reset y=0 and keep the value of x unchanged; then: axialX = x / L 2; axialY = 0 / H2=0;

[0037] (2) If y enters the effective display area of ​​the game, then the actual vertical coordinate within the effective display area of ​​the game needs to be y minus the length outside the game screen area, that is, realy = y - (H1 - H2) / 2; axialy = realy / H2; axialX = x / L2;

[0038] (3) If y does not enter the effective display area of ​​the game and is below the effective display area of ​​the game, then reset y=heightH2 and the value of x remains unchanged; then: axialX = x / L 2; axialy= H2 / H2=1.

[0039] Furthermore, the coordinate restoration module is used to restore the extracted scaled coordinates to absolute coordinates in the cloud virtual environment screen, specifically:

[0040] The coordinate restoration module performs inverse mathematical operations on the scaled coordinates based on the screen resolution of the cloud virtual environment, restoring them to absolute coordinates (absoluteX, absoluteY) in the cloud virtual environment screen. The formula is as follows:

[0041] absoluteX = axialX * virtual screen width; absoluteY = axialY * virtual screen height;

[0042] Among them, the virtual screen width and virtual screen height are known fixed parameters of the cloud gaming server.

[0043] The present invention also provides a method for a unified operation transmission system across terminals in a cloud gaming scenario, comprising:

[0044] Step S1, Client-side data processing and transformation process:

[0045] Step S11: The client's unified input capture module listens for and captures raw physical input events generated by the user's interaction with various types of input devices; the raw physical input events include the following types: touch screen input events, mouse input events, and keyboard input events;

[0046] Step S12, the client's operation abstraction module converts the captured raw physical input events of various types into predefined standardized operation events; the standardized operation events are represented as {action type, touch point ID, absolute coordinates}; wherein, the action type includes press DOWN, release UP, and move MOVE types;

[0047] Step S13: The coordinate scaling module of the client converts the absolute coordinates in the standardized operation event into scaled coordinates relative to the effective display area of ​​the game, thereby obtaining the standardized operation event {action type, touch ID, scaled coordinates} after coordinate scaling.

[0048] Step S14: The binary encoding module of the client performs binary encoding on the standardized operation event {action type, touch point ID, scaled coordinates} according to a predefined compact binary format to generate a compact binary data stream; the binary data stream is sent to the cloud gaming server through the client's network interface.

[0049] Step S2, Cloud gaming server-side data reception and injection process:

[0050] Step S21: The binary decoding module of the cloud gaming server receives the binary data stream from the client and parses the binary data stream according to the predefined binary format to extract the action type, touch point ID and scaled coordinates.

[0051] Step S22: The event state management module of the cloud gaming server tracks the state of each touch point in memory according to the touch point ID, and then reconstructs a series of discrete single-point action messages into a complete action gesture that conforms to the operating system specifications.

[0052] Step S23: The coordinate restoration module of the cloud gaming server restores the extracted proportional coordinates to absolute coordinates in the cloud virtual environment screen.

[0053] In step S24, the event injection module of the cloud gaming server calls the underlying system interface of the cloud virtual environment to simulate the operation containing the restored absolute coordinates and complete action gestures as one or a series of native touch events and inject them into the target game application.

[0054] The unified operation and transmission system and method for cloud gaming scenarios provided by this invention have the following advantages:

[0055] Significantly reduced operation latency: By adopting a custom binary protocol instead of JSON, the data size of a single operation message can be reduced by more than 75%, thereby directly reducing network transmission time and significantly reducing end-to-end operation latency from 270ms, thus greatly improving the response speed and responsiveness of the operation.

[0056] Achieving true cross-platform uniformity: By abstracting client-side operations and scaling coordinates, the server does not need to be aware of the front-end's device or input method. All inputs are normalized, greatly reducing the server's logical complexity and code coupling. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 The flowchart illustrates the unified operation and transmission method across terminals in a cloud gaming scenario provided by this invention. Detailed Implementation

[0059] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0060] This invention aims to solve the problems of high operation latency and high cross-platform adaptation and maintenance costs existing in the aforementioned background art. Specifically, the technical problems to be solved by this invention include:

[0061] Question 1: How to reduce the data volume of operation data, reduce network transmission latency, and improve the operation response speed and responsiveness of cloud games?

[0062] Question 2: How to design a unified operation data model and processing flow so that the cloud gaming server can use a single logic to process input from different platforms (such as Android / iOS touch, H5 keyboard and mouse), thereby reducing system complexity and maintenance costs.

[0063] Question 3: How to solve the problem of inaccurate coordinate mapping caused by the inconsistency in screen resolution and aspect ratio between the terminal device and the cloud virtual environment.

[0064] To address the aforementioned technical problems, this invention proposes a computer-implemented system and method. The system includes a client program deployed on a user terminal device and a cloud gaming server program deployed in the cloud, providing a computer-implemented method and system for processing user input, optimizing user operation data transmission, and achieving cross-platform compatibility.

[0065] This invention provides a unified operation and transmission system across terminals in a cloud gaming scenario, including a client and a cloud gaming server.

[0066] The client, whose internal processor executes instructions to implement the following functional modules:

[0067] The unified input capture module is responsible for listening to and capturing the user's raw physical input events on the current terminal, i.e., the raw physical input events generated by the user's interaction with various types of input devices. These raw physical input events include the following types: touchscreen input events, mouse input events, and keyboard input events; wherein, touchscreen input events are touch events of the touchscreen; mouse input events are click and movement events of the mouse; and keyboard input events are key press events of the keyboard. Its function is to serve as a unified entry point for all inputs.

[0068] Operation Abstraction Module: This module is one of the key inventive features that distinguishes it from existing technologies. It is responsible for converting the captured raw physical input events of various types into predefined standardized operation events; its function is to smooth out the differences in input sources across platforms. The standardized operation events are represented as {Action Type, Touch ID, Absolute Coordinates}; where the action type includes press (DOWN), release (UP), and move (MOVE); the standardized operation events are more specifically standardized touch operation events.

[0069] If the captured raw physical input event is a touchscreen input event, then the standardized touch operation event is obtained directly;

[0070] If the captured original physical input event is a keyboard input event or a mouse input event, then perform logical judgment and data conversion of keyboard coordinates or mouse finger representation to convert it into the standardized touch operation event.

[0071] The keyboard coordinate system is defined as mapping discrete keyboard key events to continuous touch events (DOWN and UP actions) at a preset or user-defined coordinate point on the screen. The keyboard key events include key press events and key release events (e.g., pressing / releasing the W key). The touch events are DOWN and UP actions occurring at the touch point. For example, when a user presses the W key, this module converts it into a DOWN operation on the center point of the virtual joystick.

[0072] The mouse finger simulation refers to simulating the mouse trajectory and mouse button states as single-finger touch behavior. For example, pressing the left mouse button corresponds to the DOWN action, pressing and moving corresponds to the continuous MOVE action, and releasing the left mouse button corresponds to the UP action.

[0073] The coordinate scaling module is another key inventive feature that distinguishes it from existing technologies. It converts the absolute coordinates (x, y) in the standardized operation event into scaled coordinates (axialX, axialY) relative to the effective display area of ​​the game, thus obtaining the scaled standardized operation event {action type, touch ID, scaled coordinates}. Its function is to decouple the terminal's physical resolution from the cloud's virtual resolution, ensuring the universality of operation positioning.

[0074] In this invention, the coordinate scaling module converts the absolute coordinates (x, y) in the standardized operation event into scaled coordinates (axialX, axialY) relative to the effective display area of ​​the game; where axialX represents the X coordinate scale and axialY represents the Y coordinate scale; the specific conversion method is as follows:

[0075] Set the screen resolution to width L1 x height H1; the effective display area size for the game is width L2 x height H2; the center of the effective display area for the game coincides with the center of the screen; the absolute coordinates (x, y) of the touch point ID are the screen coordinates;

[0076] a. If the effective display area size of the game is the same as the screen resolution, then axialX = x / L²; axialY = y / H².

[0077] b. If the game screen resolution and the screen resolution are not perfectly matched, with the game screen height filling the entire screen but the game screen width not filling the entire screen (i.e., height H1 and height H2 are the same, and width L2 is less than width L1), then:

[0078] (1) If x does not enter the game's effective display area and is to the left of the game's effective display area, then reset x=0 and the y value remains unchanged; then: axialX = 0 / L 2 = 0; axialY = y / H2;

[0079] (2) If x enters the effective display area of ​​the game, the actual horizontal coordinate within the effective display area of ​​the game needs to be x minus the length outside the game screen area, that is, realx = x - (L1 - L2) / 2; then: axialX = realx / L2; axialY = y / H2;

[0080] (3) If x does not enter the effective display area of ​​the game and is on the right side of the effective display area of ​​the game, then reset x = width L 2, and the y value remains unchanged; then: axialX = L 2 / L 2 = 1; axialY = y / H2;

[0081] c. If the game screen resolution and the screen resolution are not perfectly matched, with the game screen height filling the entire screen but the game screen height not filling the entire screen (i.e., height H2 is less than height H1, and width L2 equals width L1), then:

[0082] (1) If y does not enter the game's effective display area and is above the effective display area, then reset y=0 and keep the value of x unchanged; then: axialX = x / L 2; axialY = 0 / H2=0;

[0083] (2) If y enters the effective display area of ​​the game, then the actual vertical coordinate within the effective display area of ​​the game needs to be y minus the length outside the game screen area, that is, realy = y - (H1 - H2) / 2; axialy = realy / H2; axialX = x / L2;

[0084] (3) If y does not enter the effective display area of ​​the game and is below the effective display area of ​​the game, then reset y=heightH2 and the value of x remains unchanged; then: axialX = x / L 2; axialy= H2 / H2=1.

[0085] Binary encoding module: used to encode the standardized operation event {action type, touch ID, scaled coordinates} into binary format according to a predefined compact binary format, generating a compact binary data stream to be sent; the binary data stream is sent to the cloud gaming server through the client's network interface; its function is to maximize data compression and reduce network load.

[0086] The cloud gaming server's internal processor executes instructions to implement the following functional modules:

[0087] Binary decoding module: used to receive the binary data stream from the client, and parse the binary data stream according to a predefined binary format to extract the action type, touch point ID and scaled coordinates;

[0088] Event State Management Module: Used to track the state of each touch point in memory (such as whether it is pressed, current position) based on the touch point ID, and then reconstruct a series of discrete single-point action messages (such as DOWN, MOVE, ..., UP) into a complete action gesture that conforms to the operating system specifications;

[0089] The coordinate restoration module is used to restore the extracted proportional coordinates (axialX, axialY) to absolute coordinates (absoluteX, absoluteY) on the screen of a cloud virtual environment (such as an ARM cloud phone).

[0090] The coordinate restoration module is specifically used to: perform inverse mathematical operations on the scaled coordinates according to the screen resolution of the cloud virtual environment, restoring them to absolute coordinates (absoluteX, absoluteY) in the cloud virtual environment screen, using the following formula:

[0091] absoluteX = axialX * virtual screen width; absoluteY = axialY * virtual screen height;

[0092] Among them, the virtual screen width and virtual screen height are known fixed parameters of the cloud gaming server.

[0093] Event Injection Module: Used to call the underlying system interface of the cloud virtual environment, simulate an operation containing the restored absolute coordinates and complete action gestures as one or a series of native touch events, and inject them into the target game application.

[0094] For example, by calling the underlying system interfaces of the cloud virtual environment (such as Android's InputManager, Windows' HID, etc.), the operation containing the restored absolute coordinates and action type can be simulated as one or a series of native touch events (MotionEvent) and injected into the target game application.

[0095] This invention also provides a unified operation and transmission method across terminals in a cloud gaming scenario. This method is executed by a computer and includes the following physical steps, demonstrating the conversion of data between different physical devices and processing stages:

[0096] Client-side data processing and transformation process:

[0097] a. Physical Input to Digital Signal: The unified input capture module of the client device captures the physical operations generated by the user's interaction with the input device (touchscreen, mouse, keyboard) and converts them into initial digital event signals.

[0098] Specifically, the unified input capture module listens for and captures raw physical input events generated by the user's interaction with various types of input devices; the raw physical input events include the following types: touch screen input events, mouse input events, and keyboard input events.

[0099] b. Heterogeneous Data to Standard Data: The operation abstraction module receives the initial digital event signal. If it is a keyboard or mouse event, it performs logical judgment and data conversion based on keyboard coordinates or mouse finger representation, unifying it into a standardized touch event data structure containing {action type, touch point ID, absolute coordinates}.

[0100] Specifically, the client's operation abstraction module converts the captured raw physical input events of various types into predefined standardized operation events; the standardized operation events are represented as {action type, touch point ID, absolute coordinates}; wherein, the action type includes press DOWN, release UP, and move MOVE types;

[0101] c. Absolute coordinates to relative coordinates: The coordinate scaling module obtains the absolute coordinates in the standardized event and performs scaling mathematical operations based on the actual size of the current game screen on the terminal screen, converting them into {scaling X coordinates, scaling Y coordinates}.

[0102] Specifically, the client's coordinate scaling module converts the absolute coordinates in the standardized operation event into scaled coordinates relative to the effective display area of ​​the game, thereby obtaining the standardized operation event {action type, touch point ID, scaled coordinates} after coordinate scaling.

[0103] d. Structured data to binary stream: The binary encoding module encodes {action type, touch ID, scaled coordinates} into a compact binary data stream according to a preset bit or byte layout.

[0104] Specifically, the client's binary encoding module encodes the standardized operation event {action type, touch point ID, scaled coordinates} into a compact binary format to generate a compact binary data stream; the binary data stream is then sent to the cloud gaming server through the client's network interface.

[0105] e. Data on local machine to data on network: The client sends the binary data stream to the cloud gaming server via a network interface.

[0106] Server-side data reception and injection process:

[0107] f. Data from network to server-side memory: The cloud gaming server receives this binary data stream through a network interface.

[0108] g. Binary stream to structured data: The binary decoding module parses it to restore the structured data {action type, touch ID, scaled coordinates}.

[0109] Specifically: the binary decoding module of the cloud gaming server receives the binary data stream from the client and parses the binary data stream according to a predefined binary format to extract the action type, touch point ID and scaled coordinates; the event state management module of the cloud gaming server tracks the state of each touch point in memory according to the touch point ID, and then reconstructs a series of discrete single-point action messages into a complete action gesture that conforms to the operating system specifications.

[0110] h. Relative coordinates to absolute coordinates: The coordinate restoration module performs inverse mathematical operations on the proportional coordinates based on the screen resolution of the cloud virtual environment, restoring them to the absolute coordinates on the cloud virtual screen.

[0111] i. Internal data to system events: The event injection module constructs an input event object (such as Android's MotionEvent) that conforms to the cloud operating system specification based on the parsed action type, touch point ID and restored absolute coordinates.

[0112] j. Injecting the event into the game response: By calling the operating system API, the input event is injected into the system event queue. Specifically, the underlying system interface of the cloud virtual environment is called to simulate one or a series of native touch events, including the restored absolute coordinates and complete gesture, and then injected into the target game application.

[0113] The game application running on the system receives and processes this event, thereby generating physical effects in the game screen that correspond to the user's original operation. These effects are ultimately transmitted back to the user's terminal via video stream.

[0114] This embodiment uses an example of a user operating a cloud-based Android shooting game via keyboard and mouse on an H5 webpage to illustrate the complete process of the present invention.

[0115] 1. Design Concept and Technical Features

[0116] The core design concept of this invention is to move the work of handling complexity and differences to the client side, so that the server-side logic remains as simple and uniform as possible. Its key technical features are: (1) a set of ultra-low redundancy binary communication protocols are defined; (2) heterogeneous input (keyboard and mouse) is abstracted into standardized touch input on the client side; (3) a coordinate scaling mechanism is used to solve the multi-resolution adaptation problem.

[0117] 2. Computer program flow description:

[0118] The specific computer program flow of this method is as follows: Figure 1 :

[0119] Step 1: Predefine binary message format

[0120] To achieve optimal data compression, a 10-byte fixed-length binary message format is defined as follows:

[0121] Byte 0: Event type, 1 byte, reserved for future use;

[0122] Byte 1: Action type, 1 byte, 0x00=DOWN, 0x01=UP, 0x02=MOVE;

[0123] Bytes 2-3: Touch ID, 2 bytes, used to distinguish different fingers or simulated touch points in multi-touch;

[0124] Bytes 4-7: X-axis scale (axialX), 4-byte single-precision floating-point number;

[0125] Bytes 8-11: Y-axis scale (axialY), 4-byte single-precision floating-point number;

[0126] Byte 12: Operation user identifier, 1 byte.

[0127]

[0128] Step 2: Client Program Execution Flow

[0129] Assume a user opens the game in a browser, and the game screen display area is 1600x900.

[0130] Keyboard operation example:

[0131] a. When the user presses the W key, the client's unified input capture module captures the keydown event.

[0132] b. The operation abstraction module queries the preset key mapping table and finds that the W key is mapped to screen coordinates (200, 700), which is used to control the character's forward movement. The module assigns a touch point ID, such as 2.

[0133] c. The coordinate scaling module performs mathematical calculations: axialX = 200 / 1600 = 0.125; axialY = 700 / 900 ≈ 0.778

[0134] d. The binary encoding module generates a 13-byte binary message based on the above data: [0x00, 0x00, 0x02, ...<float_0.125> ,<float_0.778> , 0x00).

[0135] e. The client sends this message over the network. When the user releases the W key, a message with action type UP (0x01) will be generated and sent: [0x00, 0x01, 0x02, ...<float_0.125> ,<float_0.778> ,0x00]

[0136] Mouse operation example:

[0137] a. The user moves the mouse to the (1200, 450) position in the game screen and presses the left mouse button.

[0138] b. After the unified input capture module captures mouse events, the operation abstraction module converts them into DOWN actions and assigns touch point IDs.

[0139] c. Coordinate scaling module calculation: axialX = 1200 / 1600 = 0.75, axialY = 450 / 900 = 0.5

[0140] d. The binary encoding module encodes and sends the binary message: [0x00, 0x00, 0x01,<float_0.75> ,<float_0.5> , 0x00).

[0141] e. If the user holds down the mouse button and drags, messages of action type MOVE (0x02) will be continuously generated and sent; binary messages will be encoded and sent: [0x00, 0x02, 0x01,<float_0.75> ,<float_0.51> , 0x00]

[0142] Example of mobile phone / tablet operation:

[0143] Scene 1:

[0144] Suppose a user opens a cloud game on their phone. The game's display area is 1080x720, while the phone's screen resolution is 1600x720. Because the game's display area and the phone's resolution are not perfectly matched, the game screen may either fill the entire screen at a height or at a length. This scenario is an example of the game screen filling the entire screen at a height, while the game screen does not fill the entire screen at a length, with the center of the game screen coinciding with the center of the phone screen. Let's assume the rectangular area ABCD represents the game screen area.

[0145] a. When a user touches the phone screen, the unified input capture module captures the down event, with its screen coordinates being (x, y);

[0146] b. If the coordinates (x, y) are not within the game screen:

[0147] (1) If x does not enter the game screen of the rectangle ABCD area and is to the left of the rectangle ABCD area, then set x=0 and the y value remains unchanged. The coordinate scaling module calculates: axialX = 0 / 1080 = 0; axialY = y / 720.

[0148] (2) If x is within the rectangular ABCD area of ​​the game screen, then the real coordinates within the game screen area need to be x minus the length outside the game screen area, i.e., realx = x - (1600 - 1080) / 2 = x - 260. The coordinate scaling module calculates: axialX = (x - 260) / 1080; axialY = y / 720;

[0149] (3) If x does not enter the game screen of the rectangle ABCD area and is on the right side of the rectangle ABCD area, then set x = game screen length, and the y value remains unchanged. The coordinate scaling module calculates: axialX = 1080 / 1080 = 1; axialY = y / 720.

[0150] c. The binary encoding module encodes and sends the binary message: [0x00, 0x00, 0x01,<float_axialX> ,<float_axialY> , 0x00).

[0151] e. If the user's finger moves on the touchscreen, messages of action type MOVE (0x02) will be continuously generated and sent: encoded and sent binary messages: [0x00, 0x02, 0x01,<float_axialX> ,<float_axialY> ,0x00).

[0152] Scenario 2: Suppose a user opens a cloud game on their phone. The game's display area is 720x1080, while the phone's screen resolution is 720x1600. Because the game's display area and the phone's resolution are not perfectly matched, the game screen may either fill the entire height of the screen or partially fill it. This scenario is a case of partially filling the screen; the game screen's height does not fill the entire screen, and the center of the game screen coincides with the center of the phone screen. Let's assume the rectangular area ABCD represents the game screen area.

[0153] a. When a user touches the phone screen, the unified input capture module captures the down event, with its screen coordinates being (x, y);

[0154] b. If the coordinates (x, y) are not within the game screen:

[0155] (1) If y does not enter the game screen of the rectangular ABCD area and is above the rectangular ABCD area, then set x to remain unchanged and y=0. The coordinate scaling module calculates: axialX = x / 720, axialY = 0 / 1080=0.

[0156] (2) If y is within the rectangular ABCD area of ​​the game screen, then the actual coordinates within the game screen area need to be calculated by subtracting the length outside the game screen area from y, i.e., realy = y - (1600 - 1080) / 2 = y - 260. The coordinate scaling module calculates: axialy = (y - 260) / 1080; axialx = x / 720

[0157] (3) If y does not enter the game screen of the rectangle ABCD area and is below the rectangle ABCD area, then set x to remain unchanged and y = game screen height. The coordinate scaling module calculates: axialX = x / 720; axialY = 1080 / 1080 = 1.

[0158] c. The binary encoding module encodes and sends the binary message: [0x00, 0x00, 0x01,<float_axialX> ,<float_axialY> , 0x00).

[0159] e. If the user's finger moves on the touchscreen, messages of action type MOVE (0x02) will be continuously generated and sent: encoded and sent as binary messages: [0x00, 0x02, 0x01,<float_axialX> ,<float_axialY> ,0x00]

[0160] Step 3: Server-side program execution flow

[0161] The Android system runs in a cloud-based ARM virtual environment with a screen resolution of 1920x1080.

[0162] a. The server receives the binary message [0x00, 0x00, 0x01,<float_0.75> ,<float_0.5> , 0x00).

[0163] b. The binary decoding module parses the operation event information:

[0164] Action type = DOWN, Contact ID = 1, axialX = 0.75, axialY = 0.5.

[0165] c. The event status management module records that the contact with contact ID 1 has been pressed.

[0166] d. The coordinate restoration module performs reverse mathematical calculations: absoluteX = axialX * virtual screen width = 0.75 * 1920 = 1440; absoluteY = axialY * virtual screen height = 0.5 * 1080 = 540; The mathematical model here is a simple linear multiplication, and the virtual screen width and virtual screen height are known fixed parameters on the server side.

[0167] e. The event injection module uses the Android system API to construct a MotionEvent object, set its action to ACTION_DOWN, and its pointer coordinates to (1440, 540). Then, it calls InputManager.injectInputEvent() to inject the event into the system.

[0168] The unified operation and transmission system and method for cloud gaming scenarios provided by this invention have the following advantages:

[0169] (1) Technical effects:

[0170] Significantly reduced operation latency: By adopting a custom binary protocol instead of JSON, the data size of a single operation message can be reduced by more than 75%, thereby directly reducing network transmission time and significantly reducing end-to-end operation latency from 270ms, thus greatly improving the response speed and responsiveness of the operation.

[0171] Achieving true cross-platform uniformity: By abstracting client-side operations and scaling coordinates, the server does not need to be aware of the front-end's device or input method. All inputs are normalized, greatly reducing the server's logical complexity and code coupling.

[0172] (2) Economic effects:

[0173] Reduced R&D and maintenance costs: The server only needs to maintain one core logic to support all existing and future client platforms. Compared to developing a separate solution for each platform, this is expected to save more than 80% of cross-platform adaptation development and maintenance costs.

[0174] Accelerate product iteration: Access to new platforms becomes extremely fast, enabling products to be launched to market more quickly and seize the initiative.

[0175] (3) User experience:

[0176] It provides cloud gaming users with a smooth operating experience close to that of local games, improving user satisfaction and retention rates.

[0177] Features such as keyboard coordinates enable PC users to enjoy a good gaming experience on cloud gaming platforms, thus broadening the audience of cloud gaming.

[0178] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A unified operation and transmission system across terminals in a cloud gaming scenario, characterized in that, This includes client-side and cloud gaming server-side components; The client includes a unified input capture module, an operation abstraction module, a coordinate scaling module, and a binary encoding module; The unified input capture module is used to listen to and capture the raw physical input events generated by the user's interaction with various types of input devices; The raw physical input events include the following types: touch screen input events, mouse input events, and keyboard input events; The operation abstraction module is used to convert the captured raw physical input events of various types into predefined standardized operation events; the standardized operation events are represented as {action type, touch ID, absolute coordinates}; wherein, the action type includes press DOWN, release UP, and move MOVE types; The coordinate scaling module is used to convert the absolute coordinates in the standardized operation event into scaled coordinates relative to the effective display area of ​​the game, thereby obtaining the standardized operation event {action type, touch ID, scaled coordinates} after coordinate scaling. The binary encoding module is used to encode the standardized operation event {action type, touch point ID, scaled coordinates} into a binary format according to a predefined compact binary format, generating a compact binary data stream; the binary data stream is sent to the cloud gaming server through the client's network interface; The cloud gaming server includes a binary decoding module, an event state management module, a coordinate restoration module, and an event injection module; The binary decoding module is used to receive the binary data stream from the client, and parse the binary data stream according to a predefined binary format to extract the action type, touch point ID and scaled coordinates; The event state management module is used to track the state of each touch point in memory according to the touch point ID, and then reconstruct a series of discrete single-point action messages into a complete action gesture that conforms to the operating system specifications. The coordinate restoration module is used to restore the extracted proportional coordinates to absolute coordinates in the cloud virtual environment screen. The event injection module is used to call the underlying system interface of the cloud virtual environment, simulate an operation containing the restored absolute coordinates and complete action gestures as one or a series of native touch events, and inject them into the target game application.

2. The unified operation and transmission system across terminals in a cloud gaming scenario according to claim 1, characterized in that, The touchscreen input event is a touch event of the touchscreen; the mouse input event is a click and move event of the mouse; the keyboard input event is a key press event of the keyboard.

3. The unified operation and transmission system across terminals in a cloud gaming scenario according to claim 1, characterized in that, The operation abstraction module is specifically used for: The standardized operation events are standardized touch operation events; If the captured raw physical input event is a touchscreen input event, then the standardized touch operation event is obtained directly; If the captured original physical input event is a keyboard input event or a mouse input event, then perform logical judgment and data conversion of keyboard coordinates or mouse finger representation to convert it into the standardized touch operation event.

4. The unified operation and transmission system across terminals in a cloud gaming scenario according to claim 3, characterized in that, The keyboard coordinate mapping is defined as mapping discrete keyboard key events to continuous touch events at a preset or user-defined coordinate point on the screen; wherein, the keyboard key events include keyboard key press events and keyboard key release events; and the touch events are DOWN and UP actions occurring at the touch point. The mouse finger simulation is as follows: the mouse trajectory and mouse button states are simulated as single-finger touch behavior, pressing the left mouse button corresponds to the DOWN action, pressing and moving corresponds to the continuous MOVE action, and releasing the left mouse button corresponds to the UP action.

5. A unified operation and transmission system across terminals in a cloud gaming scenario according to claim 1, characterized in that, The coordinate scaling module converts the absolute coordinates (x, y) in the standardized operation event into scaled coordinates (axialX, axialY) relative to the effective display area of ​​the game; where axialX represents the X coordinate scale and axialY represents the Y coordinate scale; the specific conversion method is as follows: Set the screen resolution to width L1 x height H1; the effective display area size for the game is width L2 x height H2; the center of the effective display area for the game coincides with the center of the screen; the absolute coordinates (x, y) of the touch point ID are the screen coordinates; a. If the effective display area size of the game is the same as the screen resolution, then axialX = x / L²; axialY = y / H². b. If the game screen resolution and the screen resolution are not perfectly matched, with the game screen height filling the entire screen but the game screen width not filling the entire screen (i.e., height H1 and height H2 are the same, and width L2 is less than width L1), then: (1) If x does not enter the game's effective display area and is to the left of the game's effective display area, then reset x=0 and the y value remains unchanged; then: axialX = 0 / L 2 = 0; axialY = y / H2; (2) If x enters the effective display area of ​​the game, the actual horizontal coordinate within the effective display area of ​​the game needs to be x minus the length outside the game screen area, that is, realx = x - (L1 - L2) / 2; then: axialX = realx / L2; axialY = y / H2; (3) If x does not enter the game's effective display area and is on the right side of the game's effective display area, then reset x = width L2, and keep the y value unchanged; then: axialX = L2 / L2 = 1; axialY = y / H2; c. If the game screen resolution and the screen resolution are not perfectly matched, with the game screen height filling the entire screen but the game screen height not filling the entire screen (i.e., height H2 is less than height H1, and width L2 equals width L1), then: (1) If y does not enter the game's effective display area and is above the effective display area, then reset y=0 and keep the value of x unchanged; then: axialX = x / L 2; axialY = 0 / H2=0; (2) If y enters the effective display area of ​​the game, then the actual vertical coordinate within the effective display area of ​​the game needs to be y minus the length outside the game screen area, that is, realy = y - (H1 - H2) / 2; axialy = realy / H2; axialX = x / L2; (3) If y does not enter the effective display area of ​​the game and is below the effective display area of ​​the game, then reset y=heightH2 and the value of x remains unchanged; then: axialX = x / L 2; axialy= H2 / H2=1.

6. A unified operation and transmission system across terminals in a cloud gaming scenario according to claim 1, characterized in that, The coordinate restoration module is used to restore the extracted scaled coordinates to absolute coordinates in the cloud virtual environment screen, specifically: The coordinate restoration module performs inverse mathematical operations on the scaled coordinates based on the screen resolution of the cloud virtual environment, restoring them to absolute coordinates (absoluteX, absoluteY) in the cloud virtual environment screen. The formula is as follows: absoluteX = axialX * virtual screen width; absoluteY = axialY * virtual screen height; Among them, the virtual screen width and virtual screen height are known fixed parameters of the cloud gaming server.

7. A method for a unified operation transmission system across terminals in a cloud gaming scenario as described in any one of claims 1-6, characterized in that, include: Step S1, Client-side data processing and transformation process: Step S11: The client's unified input capture module listens for and captures the raw physical input events generated by the user's interaction with various types of input devices; The raw physical input events include the following types: touch screen input events, mouse input events, and keyboard input events; Step S12, the client's operation abstraction module converts the captured raw physical input events of various types into predefined standardized operation events; the standardized operation events are represented as {action type, touch point ID, absolute coordinates}; wherein, the action type includes press DOWN, release UP, and move MOVE types; Step S13: The coordinate scaling module of the client converts the absolute coordinates in the standardized operation event into scaled coordinates relative to the effective display area of ​​the game, thereby obtaining the standardized operation event {action type, touch ID, scaled coordinates} after coordinate scaling. Step S14: The binary encoding module of the client performs binary encoding on the standardized operation event {action type, touch point ID, scaled coordinates} according to a predefined compact binary format to generate a compact binary data stream; the binary data stream is sent to the cloud gaming server through the client's network interface. Step S2, Cloud gaming server-side data reception and injection process: Step S21: The binary decoding module of the cloud gaming server receives the binary data stream from the client and parses the binary data stream according to the predefined binary format to extract the action type, touch point ID and scaled coordinates. Step S22: The event state management module of the cloud gaming server tracks the state of each touch point in memory according to the touch point ID, and then reconstructs a series of discrete single-point action messages into a complete action gesture that conforms to the operating system specifications. Step S23: The coordinate restoration module of the cloud gaming server restores the extracted proportional coordinates to absolute coordinates in the cloud virtual environment screen. In step S24, the event injection module of the cloud gaming server calls the underlying system interface of the cloud virtual environment to simulate the operation containing the restored absolute coordinates and complete action gestures as one or a series of native touch events and inject them into the target game application.

Images