A picture rendering method and apparatus

Abstract

The application relates to the technical field of image rendering, and discloses a picture rendering method, which comprises the following steps: in response to a software window click event triggered by a user, a cloud server acquires geometric attribute information of the software window and sends the information to a client; the client extracts window coordinates based on the received information, performs partial screenshot of the software window, stores the screenshot in the local and uploads the screenshot to the cloud server; the client listens to mouse moving track coordinates in real time when the user moves the software window and uploads the coordinates to the cloud server, calculates inertia prediction coordinates of window movement in combination with the mouse moving track coordinates and mouse acceleration; the client calls a user operation habit model, acquires current software window information and historical mouse moving track coordinate data, obtains prediction window moving track coordinates, dynamically adjusts a display strategy of a rendering layer in a visual persistence time window according to the deviation of the two coordinates, makes the user basically unable to perceive changes in the picture and trailing images with naked eyes, and improves the fluency of interaction between the cloud server and the client.

Description

Technical Field

[0001] This invention relates to the field of image rendering technology, and specifically to a method and apparatus for rendering images. Background Technology

[0002] Cloud server products have penetrated various industries, evolving from simple office work to complex scenarios such as high-tech research and development, traditional industrial production, and infrastructure construction. Currently, cloud server application clients render software windows by: monitoring the user's mouse movement on the cloud server system to determine the mouse coordinates; then, using a driver to obtain the width and height of the selected software window; finally, rendering the window's wireframe onto the mouse coordinates; and then sending the image data to the cloud server client for decoding and display via the network. Two main factors affect software window rendering and ghosting: 1. The virtual hardware configuration of the cloud server system; 2. The network environment of the cloud server client. If the cloud server virtual machine runs slowly or the network environment of the cloud server client is poor, it will lead to rendering delays, reduced image clarity, and thus ghosting when dragging the software window.

[0003] Therefore, the current way cloud servers handle screen rendering when software windows are moved does not provide users with the experience of a native computer. In daily use, there is still a significant difference compared to a native computer, resulting in a poor user experience. Summary of the Invention

[0004] In view of this, in order to solve the problems of ghosting and latency when moving software windows on cloud servers and to provide users with a native computer operating experience, this invention proposes a screen rendering method and apparatus that can solve the problems existing in current rendering methods to the greatest extent.

[0005] In a first aspect, the present invention provides a method for rendering a screen, the method comprising:

[0006] In response to a user-triggered click event on the software window, the cloud server obtains the geometric attribute information of the software window and sends it to the client.

[0007] The client extracts the window coordinates based on the received geometric attribute information, takes a screenshot of the software window portion of the current screen, stores the captured software window image data in local storage, and synchronously uploads it to the cloud server.

[0008] The client monitors the mouse movement trajectory coordinates in real time when the user moves the software window and uploads them to the cloud server. It then combines the mouse movement trajectory coordinates and mouse acceleration information to calculate the inertial prediction coordinates of the window movement.

[0009] The client calls the user operation habit model bound to the user login ID and obtains the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates; based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates, the display strategy of the rendering layer is dynamically adjusted within the visual persistence time window.

[0010] The image rendering method provided in this invention significantly reduces reliance on the network environment while ensuring image quality by reusing a single screenshot locally on the client side. Especially in poor network conditions, current cloud server image rendering, which transmits images from the server to the client via image streaming, suffers from significant latency and image data distortion. This rendering and transmission method has a clear advantage in terms of signal reception and parsing latency. By accurately calculating the user's mouse movement trajectory coordinates, the next coordinate of the mouse movement under inertia can be predicted, allowing pre-rendering at the next coordinate point. This pre-rendering method utilizes the ergonomic principle of visual persistence, making image changes and ghosting virtually imperceptible to the user's naked eye, ensuring a smooth experience on the cloud server. Furthermore, by training a user operation habit model on the cloud server and combining it with current software window information and historical mouse movement trajectory coordinate data, the method predicts the window movement trajectory coordinates. Based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates, the display strategy of the rendering layer is dynamically adjusted within the visual persistence time window. This better adapts to different user operating styles and various complex operating scenarios, improving the system's adaptability and the intelligence of interaction.

[0011] In one optional implementation, the cloud server acquires the geometric attribute information of the software window and sends it to the client; the client extracts the window coordinates based on the received geometric attribute information, including:

[0012] The graphics driver listens for user click events on software windows, executes a first preset instruction to obtain the width, height, and XY coordinates of the currently active foreground software window from the system window process, and executes a second preset command to obtain the screen resolution, thus obtaining the screen width and height.

[0013] The client calculates the vertex coordinates of the software window using the local terminal processor based on the width, height, and XY coordinates of the received software window, as well as the screen width and height.

[0014] This invention provides a rapid response to user actions, promptly providing necessary window information when the user prepares to drag the window, making the dragging operation smoother and more natural. The cloud server only acquires and sends key geometric attribute information of the target software window, rather than detailed image data of the entire screen. This significantly reduces the amount of data transmitted from the cloud server to the client, lowering network bandwidth requirements. The client only needs to perform simple calculations based on the received limited geometric attribute information to determine the vertex coordinates of the software window. Compared to processing large amounts of image data, this calculation method places lower demands on the performance of the client's local terminal processor, reducing the client's computational burden and lowering processing costs.

[0015] In one optional implementation, before obtaining the geometric attribute information of the software window, the method further includes: sending an authorization request to the user; after obtaining the user's confirmation of authorization, the cloud server sends the program information currently being used by the user through the agent, converts it into an array and serializes it into a string, and sends the serialized string to the client. After receiving it, the client saves it to its local storage.

[0016] This invention sends an authorization request to the user before acquiring information about the programs the user is currently using, thus fully respecting the user's privacy rights. Users can decide whether to authorize the cloud server to collect relevant information based on their own wishes, avoiding unauthorized data collection, giving users more control over their personal information, and enhancing user trust in the cloud server's services. The cloud server collects information about the programs the user is currently using through an agent and saves it to the client's local storage, which helps to better understand the user's usage habits and needs, thereby providing more precise services.

[0017] In one optional implementation, the inertial prediction coordinates of window movement are calculated by combining mouse movement trajectory coordinates and mouse acceleration information, including:

[0018] The mouse listener monitors the coordinates of the mouse movement trajectory when the user moves the software window in real time.

[0019] Acceleration data is acquired through the microelectromechanical system accelerometer device built into the mouse;

[0020] The direction and distance of inertial motion are calculated based on the coordinates of the mouse movement trajectory and the acceleration vector, and the next coordinate point is predicted as information for the user's software usage.

[0021] This invention provides real-time monitoring of mouse movement trajectory coordinates and acceleration data, enabling the system to quickly capture the user's operational intentions. The precise acceleration data acquired by the accelerometer, combined with the mouse movement trajectory coordinates, allows the system to more accurately predict the window's movement position, allowing users to more precisely control the window's position when operating the software window and avoiding misoperations or unexpected window movement.

[0022] In one optional implementation, the process of the cloud server training the user operation habit model includes:

[0023] The text information is obtained by performing image and text recognition based on the screenshot information of the software window sent by the client. The text information and the user's software usage information are used as input to a preset machine learning model, and the corresponding mouse movement trajectory coordinates are used as the model output to train a user operation habit model.

[0024] This invention trains a model by using text information from a screenshot of a software window and user information about the software as input, and mouse movement trajectory coordinates as output. This model can learn the unique habits of each user in a specific software interface and operating scenario. As users use the software for longer periods and accumulate more operation data, the model can accurately predict user operating habits and provide continuously optimized services to users.

[0025] In one optional implementation, the client invokes a user operation habit model bound to the user login ID, including:

[0026] After a user logs in to the client, the client sends the user ID to the cloud server. The cloud server then sends the latest user operation habit model corresponding to the user ID to the client, which stores it on its local storage for later retrieval.

[0027] In this embodiment of the invention, the cloud server sends the latest user operation habit model, which can reflect the user's recent operation behavior and preference changes in a timely manner. The client can use the latest model to provide users with more personalized services that meet their current needs, thereby improving the accuracy and quality of the service.

[0028] In one optional implementation, the step of dynamically adjusting the display strategy of the rendering layer within the visual persistence time window based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates includes:

[0029] The client creates a new first hidden layer based on the inertial predicted coordinates and pre-renders the window image onto the first hidden layer;

[0030] Create a second hidden layer based on the predicted window movement trajectory coordinates, and pre-render the window image onto the second hidden layer;

[0031] When the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is less than a preset threshold, the first hidden layer is deleted and the second hidden layer is set to the display state.

[0032] When the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is not less than a preset threshold, the second hidden layer is deleted and the first hidden layer is set to the display state.

[0033] The state switching operation of the above layers is completed within the visual persistence time of 0.1 seconds.

[0034] This invention utilizes the persistence of vision to complete the layer state switching operation within 0.1 seconds, a process imperceptible to the user. The rendering layer is dynamically adjusted based on deviations, ensuring the window's display position more accurately matches the user's expected operation. This avoids jumps or discontinuities in window display, providing a smoother and more natural window movement visual experience, making the user feel the operation is smoother and improving the overall fluidity of the interaction. Furthermore, the display strategy of the rendering layer is automatically adjusted based on the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates, enabling the system to adapt to different users' operating habits and various complex operating scenarios.

[0035] In one optional implementation, the deviation is the standard deviation, and the preset threshold is determined using a sliding window method, including:

[0036] Obtain the sequence of differences between predicted coordinates and inertial coordinates for a preset number of iterations, and calculate the dynamic threshold using the following formula:

[0037]

[0038] Where σ represents the standard deviation, x i Let represent the i-th data point, μ represent the population mean, and n represent the number of data points in the population.

[0039] This invention, through obtaining a sequence of differences between predicted coordinates and inertial coordinates over a preset number of iterations, and using a sliding window method to dynamically calculate the standard deviation as a preset threshold, can capture changes in user operating habits in real time. When the user's operating mode changes, the difference sequence also changes accordingly, allowing the calculated standard deviation to be adjusted promptly to adapt to the user's current operating characteristics. This ensures that the system can always dynamically adjust the display strategy of the rendering layer based on the user's actual operating situation, providing services that better meet user needs.

[0040] Secondly, the present invention provides a screen rendering apparatus, comprising: a cloud server and at least one client interacting with the cloud server, wherein:

[0041] The cloud server includes:

[0042] The window geometry attribute information acquisition module is used to respond to the software window click event triggered by the user. The cloud server acquires the geometric attribute information of the software window and sends it to the client.

[0043] The window image and motion trajectory acquisition module is used to acquire software window image data and mouse movement trajectory coordinate data captured by the client.

[0044] The user operation habit model training module is used to train a preset machine learning model based on the software window image data and mouse movement trajectory coordinate data corresponding to the user login ID, so as to obtain the user operation habit model.

[0045] The client includes:

[0046] The window image acquisition module is used to extract the window coordinates based on the geometric attribute information sent by the cloud server, take a screenshot of the software window portion of the current screen, and upload it to the cloud server.

[0047] The inertial prediction coordinate acquisition module is used to monitor the mouse movement trajectory coordinates when the user moves the software window in real time and upload them to the cloud server, and calculate the inertial prediction coordinates of the window movement by combining the mouse acceleration information.

[0048] The rendering layer control module is used to call the user operation habit model bound to the user login ID, and obtain the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates; based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates, the display strategy of the rendering layer is dynamically adjusted within the visual persistence time window. Attached Figure Description

[0049] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating a screen rendering method according to an embodiment of the present invention;

[0051] Figure 2 This is a flowchart illustrating the dynamic adjustment strategy for the display of rendering layers according to an embodiment of the present invention;

[0052] Figure 3 This is a structural block diagram of a screen rendering apparatus according to an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] To reduce screen rendering artifacts caused by virtual hardware configuration and network environment in cloud server systems, and to provide users with a better user experience, this invention provides an embodiment of a screen rendering method. Figure 1 This is a flowchart of a screen rendering method according to an embodiment of the present invention, which includes the following steps:

[0055] S101, in response to a user-triggered click event on the software window, the cloud server obtains the geometric attribute information of the software window and sends it to the client.

[0056] Specifically, the geometric attributes of the software window include coordinates, dimensions, and screen resolution. Within the cloud server system, the graphics driver listens for user clicks on the software window and uses the "xwininfo" command to retrieve the width, height, and XY coordinates of the currently active, foreground software window from the system window processes. The "xrandr" command is used to obtain the screen resolution, resulting in the screen's width and height. The agent then sends this information to the client.

[0057] S102, the client extracts the window coordinates based on the received geometric attribute information, takes a screenshot of the software window portion of the current screen, stores the captured software window image data in local storage, and synchronously uploads it to the cloud server.

[0058] Specifically, after receiving the information, the client extracts the width, height, XY coordinates of the software window, and the screen width and height. It then calculates the coordinates of the software window's vertices using the local terminal processor and immediately captures a screenshot of the currently displayed image. The screenshot and software window coordinates are then extracted a second time using canvas, and only the portion of the image data representing the software window is saved to local storage using the toBlob method.

[0059] In one optional embodiment, an authorization request is sent to the user. After obtaining the user's confirmation of authorization, the cloud server sends the user's currently used program information through the agent, converts it into an array, serializes it into a string, and sends the serialized string to the client. The client receives the string and saves it to its local storage.

[0060] By sending authorization requests to users, we ensure that they have a clear right to know and control over the sharing of information about the programs they are using. Data transmission only occurs after the user confirms their authorization, demonstrating respect for user privacy and helping to enhance user trust in the system. The serialized string can be easily stored on the client's local storage. Local storage makes data access more convenient, eliminating the need for a network connection, and allowing the data to be read and used whenever needed.

[0061] S103: The client monitors the mouse movement trajectory coordinates in real time when the user moves the software window and uploads them to the cloud server. It then calculates the inertial prediction coordinates of the window movement by combining the mouse movement trajectory coordinates and mouse acceleration information.

[0062] Specifically, in this embodiment of the invention, the client sets up a mouse listener. When the user moves the mouse and drags the software window, the client saves the coordinate data of the mouse movement trajectory. Simultaneously, based on feedback information from the mouse's built-in accelerometer, it calculates the next coordinate point the mouse will move to due to inertial motion. In an optional embodiment, the client uses `addEventListener('mousemove', handleMouseMove)` to listen for mouse movement events. Within the `handleMouseMove` function, `event.clientX` and `event.clientY` are used to obtain the coordinates of the mouse movement and save these coordinates to local storage.

[0063] In this embodiment of the invention, the mouse incorporates a microelectromechanical system (MEMS) accelerometer, which is externally connected to a client terminal via physical wiring. Feedback from the accelerometer is transmitted to the cloud server client terminal via wired signals. The client receives the acceleration feedback and performs calculations. When the user moves the mouse, it generates acceleration in the current plane. Since acceleration is a vector quantity, and the mouse's position is a two-dimensional plane, the magnitude and direction of the acceleration can be calculated using a Cartesian coordinate system with X and Y axes. Because objects have inertia, before changing direction, they need to move a short distance along the original direction. The inertial motion in the direction of acceleration can be used to calculate the next coordinate point the mouse is about to reach, serving as the predicted inertial coordinates.

[0064] S104, the client calls the user operation habit model bound to the user login ID, and obtains the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates;

[0065] Specifically, after a user logs into the client, the client sends their user ID to the cloud server. The cloud server then sends a user operation habit model corresponding to that user ID back to the client, which stores it locally for later retrieval. This user operation habit model is generated based on the user's unique behavioral patterns. By storing it locally on the client, the client can provide highly personalized services based on this model. Because the client can access the user operation habit model locally, even if the cloud server experiences a brief failure or network fluctuation, the client can still continue to provide some services to the user based on the locally stored model. This avoids situations where client functionality is unusable due to server issues, thereby enhancing the stability and reliability of the entire system.

[0066] The process of training a user operation habit model on a cloud server in this embodiment of the invention includes: performing image and text recognition based on screenshots of software windows sent by the client to obtain text information, using the text information and user software usage information as inputs to a preset machine learning model, and using the corresponding mouse movement trajectory coordinates as the model output to train a user operation habit model.

[0067] In an optional embodiment, the pre-defined machine learning model structure includes: an input layer receiving text information and user software usage information; a convolutional neural network processing image features and a recurrent neural network processing text features; and a fully connected network for fusion and output. During training, gradient descent, Adam, and other optimization algorithms are used to adjust model parameters to improve prediction accuracy. Cross-validation and early stopping strategies are employed to prevent overfitting. The trained model is evaluated using a test set, calculating metrics such as accuracy and recall when predicting mouse movement trajectory coordinates. The user operation habit model generated after training is downloaded from the network each time a user logs into the cloud computer client.

[0068] S105, the client dynamically adjusts the display strategy of the rendering layer within the visual persistence time window based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates.

[0069] Specifically, this invention, by analyzing the deviation between the two coordinate systems in real time and dynamically adjusting the rendering layer, enables the window's display position to more accurately follow user operations. Completing these adjustments within the visual persistence time avoids perceptible pauses or jumps during window movement, resulting in a more natural and smooth visual effect, effectively reducing visual stuttering, and providing a smoother user experience when operating the software.

[0070] In an optional embodiment, step S105 above, as follows: Figure 2 As shown, it includes the following steps:

[0071] S1051, The client creates a new first hidden layer based on the inertial prediction coordinates and pre-renders the window image onto the first hidden layer;

[0072] S1052, Create a second hidden layer based on the predicted window movement trajectory coordinates, and pre-render the window image to the second hidden layer;

[0073] S1053, determine whether the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is less than a preset threshold;

[0074] S1054, when the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is less than a preset threshold, delete the first hidden layer and set the second hidden layer to the display state;

[0075] S1055, when the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is not less than a preset threshold, delete the second hidden layer and set the first hidden layer to the display state.

[0076] In this embodiment of the invention, by pre-rendering the window image to a hidden layer, the image can be quickly displayed when it is actually needed, reducing screen stuttering or delays caused by real-time rendering. Especially when the window needs to move, the pre-rendering mechanism allows for smoother movement, providing users with a more fluid visual experience and improving the application's interactivity and responsiveness. Based on the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates, corresponding hidden layers are selectively deleted and displayed. This strategy avoids unnecessary layer retention and rendering, making efficient use of the system's graphics processing resources.

[0077] By combining inertial prediction and window movement trajectory prediction, the system can adapt to different user operation scenarios. Inertial prediction is suitable for situations where users quickly slide or drag a window and then release the mouse, while window movement trajectory prediction focuses more on the actual movement path of the window. Switching layers based on deviations ensures that the window image is presented well in different scenarios, improving the system's adaptability to various operating conditions.

[0078] In this embodiment of the invention, the state switching operation of the aforementioned layers is completed within a visual persistence time of 0.1 seconds, so the user will not be aware of the creation, deletion, and switching of hidden layers. However, through this mechanism, the display and movement of windows are more natural and smooth, reducing visual abruptness and thus enhancing the user's comfort and satisfaction when using the application.

[0079] In one optional embodiment, the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is the standard deviation. If the standard deviation is less than 0.1, the layer under the model predicted coordinates is set to the display state. If the standard deviation is greater than 0.1, the pre-rendered layer on the predicted point will be deleted, and the layer where the inertial motion coordinates calculated based on mouse acceleration are located will be set to the display state.

[0080] In an optional embodiment, the threshold for standard deviation is determined using a sliding window method, including: obtaining a sequence of differences between predicted coordinates and inertial coordinates for a preset number of iterations, and calculating a dynamic threshold using the following formula:

[0081]

[0082] Where σ represents the standard deviation, x i Let represent the i-th data point, μ represent the overall mean, and n represent the total number of data points (in practice, this should be set appropriately according to requirements).

[0083] This embodiment also provides a screen rendering apparatus, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0084] This embodiment provides a screen rendering device, such as... Figure 3 As shown, it includes: a cloud server and at least one client that interacts with the cloud server. The cloud server includes:

[0085] The window geometry attribute information acquisition module 11 is used to respond to the software window click event triggered by the user, and the cloud server acquires the geometry attribute information of the software window and sends it to the client.

[0086] The window image and motion trajectory acquisition module 12 is used to acquire software window image data captured by the client and mouse movement trajectory coordinate data.

[0087] User operation habit model training module 13 is used to train a preset machine learning model based on the software window image data and mouse movement trajectory coordinate data corresponding to the user login ID, so as to obtain the user operation habit model.

[0088] The client includes:

[0089] The window image acquisition module 21 is used to extract the window coordinates based on the geometric attribute information sent by the cloud server, take a screenshot of the software window portion of the current screen, and upload it to the cloud server.

[0090] The inertial prediction coordinate acquisition module 22 is used to monitor the mouse movement trajectory coordinates when the user moves the software window in real time and upload them to the cloud server, and calculate the inertial prediction coordinates of the window movement by combining the mouse acceleration information.

[0091] The model prediction coordinate acquisition module 23 is used to call the user operation habit model bound to the user login ID, and obtain the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates.

[0092] The rendering layer control module 24 is used to dynamically adjust the display strategy of the rendering layer within the visual persistence time window based on the deviation analysis between the prediction window movement trajectory coordinates and the inertial prediction coordinates.

[0093] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0094] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A screen rendering method, applied to an interactive cloud server and client, characterized in that, include: In response to a user-triggered click event on the software window, the cloud server obtains the geometric attribute information of the software window and sends it to the client. The client extracts the window coordinates based on the received geometric attribute information, takes a screenshot of the software window portion of the current screen, stores the captured software window image data in local storage, and synchronously uploads it to the cloud server. The client monitors the mouse movement trajectory coordinates in real time when the user moves the software window and uploads them to the cloud server. It then combines the mouse movement trajectory coordinates and mouse acceleration information to calculate the inertial prediction coordinates of the window movement. The client calls the user operation habit model bound to the user login ID and obtains the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates; The client dynamically adjusts the display strategy of the rendering layer within the visual persistence time window based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates.

2. The method according to claim 1, characterized in that, The cloud server obtains the geometric attribute information of the software window and sends it to the client; The client extracts the window coordinates based on the received geometric attribute information, including: The graphics driver listens for user click events on software windows, executes a first preset instruction to obtain the width, height, and XY coordinates of the currently active foreground software window from the system window process, and executes a second preset command to obtain the screen resolution, thus obtaining the screen width and height. The client calculates the vertex coordinates of the software window using the local terminal processor based on the width, height, and XY coordinates of the received software window, as well as the screen width and height.

3. The method according to claim 1 or 2, characterized in that, Before obtaining the geometric attribute information of the software window, the process also includes: The cloud server sends an authorization request to the user. After receiving confirmation of authorization from the user, the cloud server sends the user's currently used program information through the agent, converts it into an array, serializes it into a string, and sends the serialized string to the client. The client receives the string and saves it to its local storage.

4. The method according to claim 1, characterized in that, The calculation of the inertial prediction coordinates for window movement by combining mouse movement trajectory coordinates and mouse acceleration information includes: The mouse listener monitors the coordinates of the mouse movement trajectory when the user moves the software window in real time. Acceleration data is acquired through the microelectromechanical system accelerometer device built into the mouse; The direction and distance of inertial motion are calculated based on the coordinates of the mouse movement trajectory and the acceleration vector, and the next coordinate point is predicted as the inertial prediction coordinate.

5. The method according to claim 1, characterized in that, The process of training a user operation habit model on the cloud server includes: The text information is obtained by performing image and text recognition based on the screenshot information of the software window sent by the client. The text information and the user's software usage information are used as input to a preset machine learning model, and the corresponding mouse movement trajectory coordinates are used as the model output to train a user operation habit model.

6. The method according to claim 5, characterized in that, The structure of the preset machine learning model includes: an input layer that receives text information and user software usage information; a convolutional neural network that processes image features and a recurrent neural network that processes text features; and a fully connected network for fusion and output.

7. The method according to claim 1, characterized in that, The client calls the user operation habit model bound to the user login ID, including: After a user logs in to the client, the client sends the user ID to the cloud server. The cloud server then sends the user's operation habit model corresponding to the user ID to the client, which stores it on its local storage for later retrieval.

8. The method according to claim 1 or 7, characterized in that, The method of dynamically adjusting the display strategy of the rendering layer within the visual persistence time window based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates includes: Create a new first hidden layer based on the inertial prediction coordinates, and pre-render the window image onto the first hidden layer; Create a second hidden layer based on the predicted window movement trajectory coordinates, and pre-render the window image onto the second hidden layer; Determine whether the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is less than a preset threshold; When the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is less than a preset threshold, the first hidden layer is deleted and the second hidden layer is set to the display state. When the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is not less than a preset threshold, the second hidden layer is deleted and the first hidden layer is set to the display state. The state switching operation of the above layers is completed within the visual persistence time of 0.1 seconds.

9. The method according to claim 8, characterized in that, The deviation is the standard deviation, and the preset threshold is determined using the sliding window method, including: Obtain the sequence of differences between predicted coordinates and inertial coordinates for a preset number of iterations, and calculate the dynamic threshold using the following formula: Where σ represents the standard deviation, x i Let represent the i-th data point, μ represent the population mean, and n represent the number of data points in the population.

10. A screen rendering device, characterized in that, include: A cloud server and at least one client that interacts with the cloud server, wherein: The cloud server includes: The window geometry attribute information acquisition module is used to respond to the software window click event triggered by the user. The cloud server acquires the geometric attribute information of the software window and sends it to the client. The window image and motion trajectory acquisition module is used to acquire software window image data and mouse movement trajectory coordinate data captured by the client. The user operation habit model training module is used to train a preset machine learning model based on the software window image data and mouse movement trajectory coordinate data corresponding to the user login ID, so as to obtain the user operation habit model. The client includes: The window image acquisition module is used to extract the window coordinates based on the geometric attribute information sent by the cloud server, take a screenshot of the software window portion of the current screen, and upload it to the cloud server. The inertial prediction coordinate acquisition module is used to monitor the mouse movement trajectory coordinates when the user moves the software window in real time and upload them to the cloud server, and calculate the inertial prediction coordinates of the window movement by combining the mouse acceleration information. The rendering layer control module is used to call the user operation habit model bound to the user login ID, and obtain the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates; based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates, the display strategy of the rendering layer is dynamically adjusted within the visual persistence time window.

Images