Picture rendering method and device

Abstract

The invention relates to the technical field of image rendering, and discloses a picture rendering method, which comprises the following steps that: in response to a software window click event triggered by a user, a cloud server acquires geometric attribute information of a software window and sends the geometric attribute information to a client; the client extracts window coordinates based on the received information, then performs software window partial screenshot, stores the screenshot locally and uploads the screenshot to the cloud server; the client monitors mouse movement track coordinates in real time when a user moves a software window and uploads the mouse movement track coordinates to the cloud server, and inertia prediction coordinates of window movement are calculated in combination with the mouse movement track coordinates and the mouse acceleration; the client calls a user operation habit model and obtains current software window information and historical mouse movement track coordinate data to obtain predicted window movement track coordinates, and a display strategy for dynamically adjusting a rendering layer in a visual persistence time window is analyzed according to the deviation of the two coordinates; therefore, the change and smear of the picture can not be perceived basically by naked eyes of the user, and the interaction fluency of the cloud server and the client is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image rendering, and in particular to a method and device for image rendering. Background Art

[0002] Cloud server products have penetrated into various industries, from simple clerical office work to various complex scenarios such as high-tech research and development, traditional industrial production, and basic system construction. In the existing technology, the cloud server application client renders the software window in the following way: by monitoring the user's mouse trajectory on the cloud server system to obtain the mouse coordinate position, then obtain the width and height of the software window selected by the user through the driver, and finally render the window wireframe image to the mouse coordinate position, and then send the image data to the cloud server client through the network for decoding and display. There are two main factors that affect the rendering and ghosting of software windows: 1. The virtual hardware configuration of the cloud server system; 2. The network environment where the cloud server client is located; if the cloud server virtual machine runs slowly or the network environment where the cloud server client is located is poor, it will cause rendering delays, reduced image clarity and other problems, resulting in ghosting when dragging the software window.

[0003] Therefore, the current way cloud servers handle screen rendering when software windows move cannot provide users with the experience of a native computer. In daily use, there is still a significant gap with native computers, and the user experience is poor. Summary of the Invention

[0004] In view of this, in order to solve the problem of ghosting and delay in the movement of cloud server software windows and provide users with a native computer operating experience, the embodiments of the present invention provide a screen rendering method and device that can maximize the solution to the problems existing in existing rendering methods.

[0005] In a first aspect, the present invention provides a method for rendering a picture, the method comprising:

[0006] In response to a software window click event triggered by a user, the cloud server obtains geometric attribute information of the software window and sends it to the client;

[0007] The client extracts the window coordinates based on the received geometric attribute information and takes a partial screenshot of the software window on the current screen. The captured software window image data is stored in the local storage and uploaded to the cloud server simultaneously.

[0008] The client monitors the coordinates of the mouse movement trajectory when the user moves the software window in real time and uploads them to the cloud server. It then calculates the inertial prediction coordinates of the window movement by combining the mouse movement trajectory coordinates and mouse acceleration information.

[0009] The client calls the user operation habit model bound to the user login ID, and obtains the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates; based on the deviation analysis of the predicted window movement trajectory coordinates and the inertial predicted coordinates, the display strategy of the rendering layer is dynamically adjusted within the visual persistence time window.

[0010] The screen rendering method provided by the embodiment of the present invention is based on a method of multiple reuse of a single screenshot on the client side, which greatly reduces the dependence on the network environment while ensuring the image quality. Especially in the case of poor network conditions, the current cloud server screen rendering will be transmitted from the server to the client through the screen stream, which will result in a relatively large delay, and the screen data will be distorted. The rendering and transmission method has an absolute advantage in the delay time of signal reception and analysis. By accurately calculating the trajectory coordinates of the user's mouse movement, the next coordinate of the mouse movement under the inertia of the movement can also be predicted, so that pre-rendering is performed at the next coordinate point. The pre-rendering method utilizes the principle of visual persistence in ergonomics, making the user's naked eye basically imperceptible to the changes and ghosting of the screen during use, ensuring a smooth experience when using the cloud server. The user operation habit model obtained by cloud server training is combined with the current software window information and historical mouse movement trajectory coordinate data to predict the window movement trajectory coordinates. Based on the deviation analysis of the predicted window movement trajectory coordinates and the inertial predicted coordinates, the display strategy of the rendering layer is dynamically adjusted within the visual persistence time window, which can better adapt to different users' operating styles and various complex operating scenarios, thereby improving the system's adaptability and interactive intelligence.

[0011] In an optional embodiment, the cloud server obtains geometric attribute information of the software window and sends it to the client; the client extracts the window coordinates based on the received geometric attribute information, including:

[0012] The graphics driver monitors the event of a user clicking a software window, executes a first preset instruction to obtain the width, height, and XY coordinates of the frontmost software window currently activated in the system from the system window process, and executes a second preset command to obtain the screen resolution to obtain the width and height of the screen;

[0013] The client calculates the vertex coordinates of the software window through the local terminal processor based on the received width, height, and XY coordinates of the software window and the screen width and height.

[0014] The embodiments of the present invention can quickly respond to user operations and provide necessary window information in a timely manner when the user is about to drag the window, making the window dragging operation smoother and more natural. The cloud server only obtains and sends the key geometric attribute information of the target software window, rather than the detailed image data of the entire screen. This method greatly reduces the amount of data transmitted from the cloud server to the client and reduces the demand for network bandwidth. The client only needs to perform simple calculations based on the limited geometric attribute information received to obtain the vertex coordinates of the software window. Compared with processing large amounts of image data, this calculation method has lower performance requirements for the client's local terminal processor, reducing the client's computing burden and processing costs.

[0015] In an optional embodiment, before obtaining the geometric attribute information of the software window, the method further includes: sending authorization request information to the user, and after obtaining the user's confirmation authorization feedback, the cloud server sends the program information currently being used by the user through the agent, converts it into an array and serializes it into a string, and sends the serialized string to the client, which saves it to the local storage after receiving it.

[0016] This embodiment of the present invention sends a user an authorization request before obtaining information about the programs they are currently using. This fully respects the user's privacy rights. Users can decide whether to authorize the cloud server to collect relevant information, preventing unauthorized data collection. This gives users greater control over their personal information and strengthens their trust in the cloud server service. The cloud server uses an agent to collect information about the programs currently in use and saves it to the client's local storage. This helps better understand the user's usage habits and needs, thereby providing more targeted services.

[0017] In an optional embodiment, calculating the inertial predicted coordinates of the window movement by combining the mouse movement trajectory coordinates and the mouse acceleration information includes:

[0018] Use the mouse monitor to monitor the coordinates of the mouse movement trajectory when the user moves the software window in real time;

[0019] Acceleration data is obtained through a micro-electromechanical system accelerometer device built into the mouse;

[0020] The inertial motion direction and distance are calculated based on the coordinates of the mouse movement trajectory and the acceleration vector, and the next coordinate point is predicted as the user's software usage information.

[0021] The present invention monitors the coordinates and acceleration data of mouse movement paths in real time, enabling the system to quickly detect the user's intended operation. The precise acceleration data captured by the accelerometer, combined with the mouse movement path coordinates, enables the system to more accurately predict the movement position of windows. This allows users to more precisely control the position of software windows when operating them, avoiding misoperations or unexpected window movements.

[0022] In an optional embodiment, the process of the cloud server training the user operation habit model includes:

[0023] Based on the software window screenshot information sent by the client, image and text recognition is performed to obtain text information, and the text information and user software usage information are used as the input of the preset machine learning model. The corresponding mouse movement trajectory coordinates are used as the model output to train a user operation habit model.

[0024] The embodiment of the present invention uses the text information in the software window screenshot information and the user's software usage information as input and the mouse movement trajectory coordinates as output for model training. It can learn the unique habits of each user in a specific software interface and operation scenario. As the user uses the software for a longer time and the operation data continues to accumulate, it maintains accurate prediction of the user's operation habits and provides users with continuously optimized services.

[0025] In an optional implementation, the client calls the user operation habit model bound to the user login ID, including:

[0026] After the user logs in to the client, the client sends the user ID to the cloud server. The cloud server sends the latest user operation habit model corresponding to the user ID to the client, and the client stores it in the local storage for calling.

[0027] The cloud server in the embodiment of the present invention sends the latest user operation habit model, which can timely reflect the user's recent operation behavior and preference changes. The client uses the latest model to provide users with personalized services that are more in line with their current needs, thereby improving the accuracy and quality of the service.

[0028] In an optional embodiment, dynamically adjusting the display strategy of the rendering layer within the visual persistence time window based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates includes:

[0029] The client creates a first hidden layer according to the inertial prediction coordinates and pre-renders the window image to the first hidden layer;

[0030] Create a new second hidden layer according to the predicted window movement trajectory coordinates, and pre-render the window image to the second hidden layer;

[0031] When the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is less than a preset threshold, deleting the first hidden layer and setting the second hidden layer to a display state;

[0032] When the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is not less than a preset threshold, the second hidden layer is deleted and the first hidden layer is set to a display state;

[0033] The state switching operation of the above layers is completed within the visual persistence time of 0.1 seconds.

[0034] The embodiment of the present invention utilizes the characteristics of visual persistence to complete the layer state switching operation within 0.1 seconds. The user cannot perceive this process. The rendering layer is dynamically adjusted according to the deviation, so that the display position of the window can more accurately match the user's operation expectations, avoiding jumping or discontinuity of window display, thereby providing a smoother and more natural window movement visual experience, making the user feel that the operation is smoother, and improving the smoothness of the overall interaction; according to the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates, the display strategy of the rendering layer is automatically adjusted, so that the system can adapt to the operating habits of different users and various complex operating scenarios.

[0035] In an optional implementation, the deviation is a standard deviation, and the preset threshold is determined using a sliding window method, including:

[0036] Obtain the difference sequence between the predicted coordinates and the inertial coordinates for a preset number of times, and calculate the dynamic threshold using the following formula:

[0037]

[0038] Where σ represents the standard deviation, x i represents the i-th data point, μ represents the population mean, and n represents the number of population data.

[0039] This embodiment of the present invention captures changes in user operating habits in real time by acquiring a preset number of difference sequences between predicted coordinates and inertial coordinates and using a sliding window method to dynamically calculate the standard deviation as a preset threshold. As the user's operating mode changes, the difference sequence also changes accordingly, allowing the calculated standard deviation to be adjusted in a timely manner to suit the user's current operating characteristics. This ensures that the system can always dynamically adjust the display strategy of the rendering layer based on the user's actual operating conditions, providing services that better meet user needs.

[0040] In a second aspect, the present invention provides a screen rendering device, comprising: a cloud server and at least one client interacting with the cloud server, wherein:

[0041] The cloud server includes:

[0042] The window geometric attribute information acquisition module is used to respond to the software window click event triggered by the user, and the cloud server obtains the geometric attribute information of the software window and sends it to the client;

[0043] The window image and motion trajectory acquisition module is used to obtain the software window image data and mouse movement trajectory coordinate data captured by the client;

[0044] The user operation habit model training module is used to train a preset machine learning model based on the software window image data corresponding to the user login ID and the mouse movement trajectory coordinate data to obtain a user operation habit model;

[0045] The client includes:

[0046] The window image acquisition module is used to extract the window coordinates based on the geometric attribute information sent by the cloud server, take a screenshot of the software window on the current screen, and upload it to the cloud server;

[0047] The inertial prediction coordinate acquisition module is used to monitor the coordinates of the mouse movement trajectory when the user moves the software window in real time and upload them to the cloud server. It also calculates the inertial prediction coordinates of the window movement based on the mouse acceleration information.

[0048] The rendering layer control module is used to call the user operation habit model bound to the user login ID, and obtain the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates; based on the deviation analysis of the predicted window movement trajectory coordinates and the inertial predicted coordinates, the display strategy of the rendering layer is dynamically adjusted within the visual persistence time window. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 is a flowchart of a picture rendering method according to an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of a process for dynamically adjusting a display strategy for a rendering layer according to an embodiment of the present invention;

[0052] Figure 3 4 is a structural block diagram of a picture rendering device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0054] In order to reduce the problem of screen rendering smear caused by the virtual hardware configuration and network environment of the cloud server system and provide users with a better operating experience, an embodiment of the present invention provides an embodiment of a screen rendering method. Figure 1 1 is a flowchart of a method for rendering a picture according to an embodiment of the present invention, and the process includes the following steps:

[0055] S101, in response to a software window click event triggered by a user, the cloud server obtains geometric attribute information of the software window and sends it to the client.

[0056] Specifically, the geometric attributes of software windows include coordinates, dimensions, and screen resolution. The cloud server system's graphics driver monitors user clicks on software windows. Using the "xwininfo" command, the system window process retrieves the width, height, and XY coordinates of the currently active front-most software window. The "xrandr" command retrieves the screen resolution and width and height. The agent then sends this information to the client.

[0057] S102, the client extracts the window coordinates based on the received geometric attribute information and takes a partial screenshot of the software window on the current screen, stores the captured software window image data in the local memory and simultaneously uploads it to the cloud server.

[0058] Specifically, after receiving the information, the client extracts the software window's width and height, XY coordinates, and screen width and height. The local terminal processor calculates the coordinates of the software window's vertices, and then immediately takes a screenshot of the current screen display. The screenshot and software window coordinates are then extracted again using canvas, and only the software window portion of the image data is saved to local storage using the toBlob method.

[0059] In an optional embodiment, an authorization request message is sent to the user. After obtaining the user's confirmation authorization feedback, the cloud server sends the program information currently being used by the user through the agent, converts it into an array and serializes it into a string, and sends the serialized string to the client. After receiving it, the client saves it to the local storage.

[0060] By sending authorization requests to users, we ensure they have clear knowledge and control over the sharing of information about the programs they are using. Data is only transferred after the user confirms authorization, demonstrating respect for user privacy and helping to strengthen user trust in the system. Serialized strings can be conveniently stored in the client's local storage. Local storage makes data access more convenient, eliminating the need for network connectivity and allowing the data to be read and used at any time.

[0061] S103, the client monitors the mouse movement trajectory coordinates when the user moves the software window in real time and uploads them to the cloud server, and calculates the inertial prediction coordinates of the window movement by combining the mouse movement trajectory coordinates and the mouse acceleration information.

[0062] Specifically, in one embodiment of the present invention, a client sets a mouse listener. When a user moves the mouse to drag a software window, the client saves the coordinate data of the mouse's movement trajectory and calculates the next coordinate point to which the mouse will move due to inertial motion based on feedback from the mouse's built-in accelerometer. In an optional embodiment, the client uses addEventListener('mousemove', handleMouseMove) to listen for mouse movement events. Within the handleMouseMove function, event.clientX and event.clientY are used to obtain the coordinates of the mouse movement and save the coordinates to local storage.

[0063] The mouse in the embodiment of the present invention has a built-in micro-electromechanical system accelerometer device, which is connected to the client terminal through a physical line. The feedback information of the accelerometer device is transmitted to the cloud server client terminal through a wired electrical signal. The client receives the acceleration feedback information and performs calculations. When the user moves the mouse, the mouse will generate an acceleration in the current plane. Since the acceleration is a vector and the position of the mouse is a two-dimensional plane, the magnitude and direction of the acceleration can be calculated using the plane rectangular coordinate system of the X and Y axes. Since the movement of an object is inertial, before changing the direction of movement, the object needs to move a short distance in the original direction. The inertial motion in the direction of acceleration can be used to calculate the next coordinate point that the mouse will reach as the inertial predicted coordinate.

[0064] S104, the client calls the user operation habit model bound to the user login ID, and obtains the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates;

[0065] Specifically, after the user logs in to the client, the client sends the user ID to the cloud server. The cloud server sends the user operation habit model corresponding to the user ID to the client, and the client stores it in the local storage for calling. The user operation habit model is generated based on the user's unique behavior pattern. By storing it locally on the client, the client can provide highly personalized services to the user based on this model. Since the client can call the user operation habit model locally, even if the cloud server has a temporary failure or network fluctuations, the client can still rely on the locally stored model to continue to provide some services to the user, avoiding the situation where the client function cannot be used normally due to server problems, thereby enhancing the stability and reliability of the entire system.

[0066] The process of training the user operation habit model by the cloud server in an embodiment of the present invention includes: performing image and text recognition based on the software window screenshot information sent by the client to obtain text information, and using the text information and user software usage information as preset machine learning model inputs, and using the corresponding mouse movement trajectory coordinates as model outputs to train the user operation habit model.

[0067] In an optional embodiment, the structure of the preset machine learning model includes: an input layer that receives text information and user software usage information; a convolutional neural network that processes image features and a recurrent neural network that processes text features; and a fully connected network that performs fusion and output. During the training process, gradient descent, Adam, and other optimization algorithms are used to adjust model parameters to improve model prediction accuracy. Cross-validation, early stopping, and other strategies are used to prevent model overfitting. The trained model is evaluated using a test set to calculate the model's accuracy, recall, and other indicators when predicting the coordinate points of the mouse movement trajectory. The user operation habit model generated after training will be downloaded through the network each time the user logs in to the cloud computer client.

[0068] S105: The client dynamically adjusts the display strategy of the rendering layer within the visual persistence time window based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates.

[0069] Specifically, the present invention analyzes the deviation between the two coordinates in real time and dynamically adjusts the rendering layer, allowing the window's display position to more accurately follow user actions. By completing these adjustments within the persistence of vision, the user can avoid perceived pauses or jumps during window movement, making the window movement more natural and smooth, effectively reducing visual stuttering and providing a smoother experience for users when operating the software.

[0070] In an optional embodiment, the above step S105, such as Figure 2 As shown, the following steps are included:

[0071] S1051: The client creates a first hidden layer based on the inertial prediction coordinates and pre-renders the window image to the first hidden layer;

[0072] S1052: Create a second hidden layer based on the predicted window movement trajectory coordinates, and pre-render the window image to the second hidden layer;

[0073] S1053, determining whether the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is less than a preset threshold;

[0074] S1054, when the deviation between the predicted window movement trajectory coordinates and the inertial prediction coordinates is less than a preset threshold, deleting the first hidden layer and setting the second hidden layer to a display state;

[0075] S1055: When the deviation between the predicted window moving trajectory coordinates and the inertial predicted coordinates is not less than a preset threshold, the second hidden layer is deleted, and the first hidden layer is set to a display state.

[0076] In the embodiment of the present invention, by pre-rendering the window image to a hidden layer, the window image can be quickly presented when it is actually needed, reducing the screen freeze or delay caused by real-time rendering. In particular, when the window needs to be moved, the pre-rendering mechanism enables the window to move more smoothly, giving the user a smoother visual experience and improving the interactivity and response speed of the application. According to the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates, the corresponding hidden layers are selectively deleted and displayed. This strategy avoids unnecessary layer retention and rendering and rationally utilizes the system's graphics processing resources.

[0077] A combination of inertial prediction and window trajectory prediction adapts to diverse user scenarios. Inertial prediction is suitable for situations where the user quickly slides or drags a window and then releases the mouse, while window trajectory prediction focuses more on the actual movement path of the window. By switching layers based on deviations, the system can better present window images in different scenarios, improving its adaptability to various operating conditions.

[0078] This embodiment of the present invention completes the aforementioned layer state switching operation within a visual persistence time of 0.1 seconds, preventing users from noticing the creation, deletion, and switching of hidden layers. However, this mechanism makes the display and movement of windows more natural and smooth, reducing visual intrusion and enhancing user comfort and satisfaction when using the application.

[0079] In an optional embodiment, the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is the standard deviation. If the standard deviation is less than 0.1, the layer under the model predicted coordinates is set to display status. If the standard deviation with the predicted value is greater than 0.1, the pre-rendered layer on the predicted point will be deleted, and the layer where the inertial motion coordinates calculated according to the mouse acceleration are located will be set to display status.

[0080] In an optional embodiment, the threshold of the standard deviation is determined using a sliding window method, including: obtaining a sequence of differences between the predicted coordinates and the inertial coordinates for a preset number of times, and calculating a dynamic threshold using the following formula:

[0081]

[0082] Where σ represents the standard deviation, x i Represents the i-th data point, μ represents the overall mean, and n represents the number of overall data (in practice, reasonable settings should be made according to needs).

[0083] In this embodiment, a screen rendering device is also provided. The system is used to implement the above-mentioned embodiments and preferred embodiments. The details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0084] This embodiment provides a picture rendering device, such as Figure 3 As shown, it includes: a cloud server and at least one client that interacts with the cloud server, and the cloud server includes:

[0085] The window geometric attribute information acquisition module 11 is used to respond to a software window click event triggered by a user, and the cloud server acquires the geometric attribute information of the software window and sends it to the client;

[0086] The window image and motion trajectory acquisition module 12 is used to obtain the software window image data and mouse movement trajectory coordinate data captured by the client;

[0087] The user operation habit model training module 13 is used to train a preset machine learning model based on the software window image data corresponding to the user login ID and the mouse movement trajectory coordinate data to obtain a user operation habit model;

[0088] The client includes:

[0089] The window image acquisition module 21 is used to extract the window coordinates based on the geometric attribute information sent by the cloud server, take a screenshot of the software window on the current screen, and upload it to the cloud server;

[0090] The inertial prediction coordinate acquisition module 22 is used to monitor the coordinates of the mouse movement trajectory when the user moves the software window in real time and upload it to the cloud server, and calculate the inertial prediction coordinates of the window movement in combination with the mouse acceleration information;

[0091] The model prediction coordinate acquisition module 23 is used to call the user operation habit model bound to the user login ID, and obtain the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates;

[0092] The rendering layer control module 24 is used to dynamically adjust the display strategy of the rendering layer within the visual persistence time window based on the deviation analysis between the prediction window movement trajectory coordinates and the inertial prediction coordinates.

[0093] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0094] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A screen rendering method, applied to an interactive cloud server and client, characterized in that: include: In response to a software window click event triggered by a user, the cloud server obtains geometric attribute information of the software window and sends it to the client; The client extracts the window coordinates based on the received geometric attribute information and takes a partial screenshot of the software window on the current screen. The captured software window image data is stored in the local storage and uploaded to the cloud server simultaneously. The client monitors the coordinates of the mouse movement trajectory when the user moves the software window in real time and uploads them to the cloud server. It then calculates the inertial prediction coordinates of the window movement by combining the mouse movement trajectory coordinates and mouse acceleration information. The client calls the user operation habit model bound to the user login ID, and obtains the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates; The client dynamically adjusts the display strategy of the rendering layer within the visual persistence time window based on the deviation analysis of the predicted window movement trajectory coordinates and the inertial predicted coordinates.

2. The method according to claim 1, characterized in that The cloud server obtains geometric attribute information of the software window and sends it to the client; The client extracts the window coordinates based on the received geometric attribute information, including: The graphics driver monitors the event of a user clicking a software window, executes a first preset instruction to obtain the width, height, and XY coordinates of the frontmost software window currently activated in the system from the system window process, and executes a second preset command to obtain the screen resolution to obtain the width and height of the screen; The client calculates the vertex coordinates of the software window through the local terminal processor based on the received width, height, and XY coordinates of the software window and the screen width and height.

3. The method according to claim 1 or 2, characterized in that Before obtaining the geometric attribute information of the software window, the method further includes: Send authorization request information to the user. After obtaining the user's confirmation authorization feedback, the cloud server sends the user's currently used program information through the agent, converts it into an array and serializes it into a string, and sends the serialized string to the client. After receiving it, the client saves it to the local storage.

4. The method according to claim 1, wherein The calculation of the inertial prediction coordinates of the window movement by combining the mouse movement trajectory coordinates and the mouse acceleration information includes: Use the mouse monitor to monitor the coordinates of the mouse movement trajectory when the user moves the software window in real time; Acceleration data is obtained through a micro-electromechanical system accelerometer device built into the mouse; The inertial motion direction and distance are calculated based on the mouse movement trajectory coordinates and acceleration vector, and the next coordinate point is predicted as the inertial predicted coordinate.

5. The method according to claim 1, wherein The process of the cloud server training the user operation habit model includes: Based on the software window screenshot information sent by the client, image and text recognition is performed to obtain text information, and the text information and user software usage information are used as the input of the preset machine learning model. The corresponding mouse movement trajectory coordinates are used as the model output to train a user operation habit model.

6. The method according to claim 5, characterized in that The structure of the preset machine learning model includes: an input layer receives text information and user software usage information; a convolutional neural network is used to process image features, and a recurrent neural network is used to process text features; and a fully connected network is used for fusion and output.

7. The method according to claim 1, characterized in that The client calls the user operation habit model bound to the user login ID, including: After the user logs in to the client, the client sends the user ID to the cloud server. The cloud server sends the user operation habit model corresponding to the user ID to the client, and the client stores it in the local storage for calling.

8. The method according to claim 1 or 7, characterized in that The method of dynamically adjusting the display strategy of the rendering layer within the visual persistence time window based on the deviation analysis between the predicted window movement trajectory coordinates and the inertial predicted coordinates includes: Create a new first hidden layer according to the inertial prediction coordinates, and pre-render the window image to the first hidden layer; Create a new second hidden layer according to the predicted window movement trajectory coordinates, and pre-render the window image to the second hidden layer; Determine whether the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is less than a preset threshold; When the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is less than a preset threshold, deleting the first hidden layer and setting the second hidden layer to a display state; When the deviation between the predicted window movement trajectory coordinates and the inertial predicted coordinates is not less than a preset threshold, the second hidden layer is deleted and the first hidden layer is set to a display state; The state switching operation of the above layers is completed within the visual persistence time of 0.1 seconds.

9. The method according to claim 8, characterized in that The deviation is the standard deviation, and the preset threshold is determined using a sliding window method, including: Obtain the difference sequence between the predicted coordinates and the inertial coordinates for a preset number of times, and calculate the dynamic threshold using the following formula: Where σ represents the standard deviation, x i represents the i-th data point, μ represents the population mean, and n represents the number of population data.

10. A picture rendering device, characterized in that: include: A cloud server and at least one client interacting with the cloud server, wherein: The cloud server includes: The window geometric attribute information acquisition module is used to respond to the software window click event triggered by the user, and the cloud server obtains the geometric attribute information of the software window and sends it to the client; The window image and motion trajectory acquisition module is used to obtain the software window image data and mouse movement trajectory coordinate data captured by the client; The user operation habit model training module is used to train a preset machine learning model based on the software window image data corresponding to the user login ID and the mouse movement trajectory coordinate data to obtain a user operation habit model; The client includes: The window image acquisition module is used to extract the window coordinates based on the geometric attribute information sent by the cloud server, take a screenshot of the software window on the current screen, and upload it to the cloud server; The inertial prediction coordinate acquisition module is used to monitor the coordinates of the mouse movement trajectory when the user moves the software window in real time and upload them to the cloud server. It also calculates the inertial prediction coordinates of the window movement based on the mouse acceleration information. The rendering layer control module is used to call the user operation habit model bound to the user login ID, and obtain the current software window information and historical mouse movement trajectory coordinate data to obtain the predicted window movement trajectory coordinates; based on the deviation analysis of the predicted window movement trajectory coordinates and the inertial predicted coordinates, the display strategy of the rendering layer is dynamically adjusted within the visual persistence time window.

Images