Cloud environment security screen recording method and system based on virtual network console VNC

Abstract

The invention discloses a cloud environment safe screen recording method and system based on a VNC (Virtual Network Console), and relates to the field of cloud computing or other related technical fields, the method comprises the following steps: a user identity token in an access request initiated by a VNC terminal passes verification, server information and meta-information of cloud resources are read, based on the server information, the VNC terminal receives the meta-information of the cloud resources, and the VNC terminal receives the meta-information of the cloud resources; executing real-time two-way traffic proxy by using the started traffic copy coroutine, drawing a local change picture from the VNC server to the VNC terminal to the canvas, extracting a snapshot from the canvas at a preset configuration frame rate, and adding a watermark in the snapshot; and performing real-time encoding on the snapshot added with the watermark by using a target video encoder, pushing a video stream obtained by encoding to a real-time transmission server, and transmitting the video stream to a plurality of clients for playing. According to the method and the device, the technical problems that real-time multi-terminal sharing cannot be realized and the multi-terminal cooperation working efficiency is reduced when screen recording is carried out in a cloud environment in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing technology, fintech, or other related fields. Specifically, it relates to a secure screen recording method and system for cloud environments based on the Virtual Network Control (VNC) console. Background Technology

[0002] With the rapid development of information technology, cloud computing has become an indispensable infrastructure for modern enterprises. The elastic computing resources and convenient service models it provides have greatly improved the utilization rate of IT resources and the flexibility of business. For example, in the application scenario of Elastic Cloud Server (ECS), enterprises can allocate computing resources on demand, respond quickly to market changes, and effectively reduce IT costs.

[0003] However, security, business process compliance, operational standardization, and risk prevention in the cloud environment have become issues that cannot be ignored. Remote access and management of cloud servers primarily rely on protocols such as SSH (Secure Shell). While SSH provides basic security, it also has significant limitations. For example, it requires key or password authentication, posing potential security risks; it cannot monitor modifications made by users with high privileges (high-privilege users accessing core data, bypassing financial approval processes, and tampering with transaction data, among other high-risk operations); and it cannot monitor erroneous operations within financial institutions, potentially leading to significant service termination or financial losses.

[0004] To address these issues, some related technologies employ screen recording methods, such as capturing and forwarding graphical data from remote desktops via proxy servers to record the operation process for subsequent analysis. However, the post-processing of screen recording data in these technologies is complex and cannot achieve real-time or near real-time multi-device sharing. This impacts the efficiency of operational risk monitoring, error tracing, and multi-device collaboration for financial institutions. Furthermore, the protection of sensitive information is insufficient; screen recording files may contain sensitive user operations or important corporate data, posing a risk of leakage.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a secure screen recording method and system for cloud environments based on the Virtual Network Control (VNC) console, which at least solves the technical problem in related technologies that screen recording in a cloud environment cannot be shared in real time across multiple devices, thus reducing the efficiency of multi-device collaborative work.

[0007] To achieve the above objectives, according to one aspect of this application, a secure screen recording method for a cloud environment based on a Virtual Network Console (VNC) is provided, applied to a VNC agent, comprising: upon successful verification of the user identity token in an access request initiated by the VNC terminal, reading server information and cloud resource metadata, wherein the server information includes at least a unique identifier for the cloud host, and the metadata includes at least a screen recording instruction, with each VNC server corresponding to one cloud resource; based on the server information, executing real-time bidirectional traffic proxy using a started traffic replication coroutine, wherein the real-time bidirectional traffic proxy includes: replicating traffic from the VNC terminal to the VNC server, or replicating traffic from the VNC server to the VNC terminal; drawing a partial change in the screen from the VNC server to the VNC terminal onto a canvas, extracting a snapshot from the canvas at a predetermined frame rate, and adding a watermark to the snapshot; using a target video encoder to encode the watermarked snapshot in real time, and pushing the encoded video stream to a real-time transmission server, wherein the real-time transmission server is used to transmit the video stream to multiple clients for playback.

[0008] Optionally, the step of performing real-time bidirectional traffic proxy using the initiated traffic replication coroutine includes: establishing a network link between the VNC agent and the VNC terminal via a full-duplex communication protocol, and initiating a first traffic replication coroutine to replicate the traffic of the VNC terminal to the VNC server; establishing a network link between the VNC agent and the VNC server via a transmission control protocol, and initiating a second traffic replication coroutine to replicate the traffic of the VNC server to the VNC terminal.

[0009] Optionally, the step of drawing the partial change screen from the VNC server to the VNC terminal onto the canvas includes: preparing a blank canvas when the screen recording command indicates that the current cloud server needs to record the screen; parsing the partial change traffic from the VNC server to the VNC terminal using the remote desktop access protocol; capturing multiple frames from the partial change traffic to obtain the partial change screen; and drawing the partial change screen onto the canvas.

[0010] Optionally, the step of adding a watermark to the snapshot includes: obtaining the timestamp corresponding to the current moment, the terminal IP of the VNC terminal, and user information; and adding a watermark containing the timestamp, terminal IP, cloud host unique identifier, and user information to the snapshot.

[0011] Optionally, before verifying the user identity token in the access request initiated by the VNC terminal, the method further includes: receiving a VNC login request initiated by the VNC terminal, wherein the VNC login request includes: cloud platform authentication information and cloud server information selected by the user terminal, wherein the cloud platform authentication information is used to determine the user's identity, and the cloud server information includes at least one of the following: a unique identifier for the cloud host, a physical node, a VNC address and a VNC port, and an instruction for screen recording; generating a user identity token based on the cloud platform authentication information and the cloud server information; and sending the user identity token to the VNC terminal, wherein after receiving the user identity token, the VNC terminal uses a browser to redirect to a fixed VNC entry point and initiates the access request carrying the user identity token.

[0012] Optionally, after the encoded video stream is pushed to the real-time transmission server, the method further includes: obtaining the traffic interval configuration pre-configured for the access request, wherein the traffic interval configuration includes at least: a traffic interval threshold; and blocking the network link between the VNC agent and the VNC terminal if the video stream transmission traffic interval is greater than the traffic interval threshold.

[0013] Optionally, after blocking the network link between the VNC agent and the VNC terminal, the method further includes: listening for a connection disconnection event; upon detecting that the network link between the VNC agent and the VNC terminal has been disconnected, closing the target video encoder to obtain a complete video file; and pushing the complete video file to the client's browser for playback of the recorded video through the browser.

[0014] According to another aspect of the present invention, a secure screen recording system for a cloud environment based on a Virtual Network Console (VNC) is also provided, comprising: a VNC terminal, which receives information about a cloud server selected by a user, generates a login request based on the cloud server information and cloud platform authentication information, and sends the login request to a VNC agent, wherein the VNC agent generates a user identity token based on the cloud platform authentication information and the cloud server information, returns the user identity token to the VNC terminal, and the VNC terminal initiates an access request based on the user identity token; a VNC server, a cloud platform underlying virtualization component, which provides independent cloud resource access for each network connection; and a VNC agent, which establishes a network link with the VNC terminal through a full-duplex communication protocol, establishes a network link with the VNC server through a transmission control protocol, and executes the secure screen recording method for a cloud environment based on a Virtual Network Console (VNC) as described above.

[0015] According to another aspect of the present invention, a cloud environment secure screen recording device based on a Virtual Network Console (VNC) is also provided, applied to a VNC agent, comprising: an information reading unit, used to read server information and cloud resource metadata when the user identity token in the access request initiated by the VNC terminal is verified, wherein the server information includes at least a cloud host unique identifier, and the metadata includes at least a screen recording instruction, with each VNC server corresponding to one cloud resource; a bidirectional traffic proxy unit, used to perform real-time bidirectional traffic proxy based on the server information using a started traffic replication coroutine, wherein the real-time bidirectional traffic proxy includes: copying the traffic of the VNC terminal to the VNC server, or copying the traffic of the VNC server to the VNC terminal; a canvas drawing unit, used to draw the partial changing screen from the VNC server to the VNC terminal onto a canvas, extract a snapshot from the canvas at a predetermined frame rate, and add a watermark to the snapshot; and a video stream pushing unit, used to encode the watermarked snapshot in real time using a target video encoder, and push the encoded video stream to a real-time transmission server, wherein the real-time transmission server is used to transmit the video stream to multiple clients for playback.

[0016] Optionally, the bidirectional traffic proxy unit includes: a first network link establishment module, used to establish a network link between the VNC proxy and the VNC terminal through a full-duplex communication protocol, and to start a first traffic replication coroutine to replicate the traffic of the VNC terminal to the VNC server; and a second network link establishment module, used to establish a network link between the VNC proxy and the VNC server through a transmission control protocol, and to start a second traffic replication coroutine to replicate the traffic of the VNC server to the VNC terminal.

[0017] Optionally, the canvas drawing unit includes: a canvas preparation module, used to prepare a blank canvas when the screen recording command indicates that the current cloud server needs to record the screen; a traffic parsing module, used to parse the local variable traffic from the VNC server to the VNC terminal using the remote desktop access protocol; a screen capture module, used to capture multiple screens from the local variable traffic to obtain the local variable screen; and a canvas drawing module, used to draw the local variable screen onto the canvas.

[0018] Optionally, when adding a watermark to the snapshot, the canvas drawing unit includes: a current terminal information acquisition module, used to acquire the timestamp corresponding to the current moment, the terminal IP of the VNC terminal, and user information; and a watermark adding module, used to add a watermark containing the timestamp, terminal IP, cloud host unique identifier, and user information to the snapshot.

[0019] Optionally, the cloud environment secure screen recording device based on the Virtual Network Console (VNC) further includes: a login request receiving unit, used to receive a VNC login request initiated by the VNC terminal before verifying the user identity token in the access request initiated by the VNC terminal, wherein the VNC login request includes: cloud platform authentication information and cloud server information selected by the user terminal, the cloud platform authentication information is used to determine the user identity, and the cloud server information includes at least one of the following: a unique identifier for the cloud host, a physical node, a VNC address and a VNC port, and an instruction for whether to record the screen; a token generation unit, used to generate a user identity token based on the cloud platform authentication information and the cloud server information; and a token sending unit, used to send the user identity token to the VNC terminal, wherein after receiving the user identity token, the VNC terminal uses a browser to redirect to a fixed VNC entry point and initiates the access request carrying the user identity token.

[0020] Optionally, the cloud environment secure screen recording device based on the Virtual Network Control (VNC) console further includes: a traffic interval configuration unit, used to obtain the traffic interval configuration pre-configured for the access request after the encoded video stream is pushed to the real-time transmission server, wherein the traffic interval configuration includes at least: a traffic interval threshold; and a network link blocking unit, used to block the network link between the VNC agent and the VNC terminal when the video stream transmission traffic interval is greater than the traffic interval threshold.

[0021] Optionally, the cloud-based secure screen recording device based on the Virtual Network Control (VNC) console further includes: an event listening unit, used to listen for connection disconnection events after blocking the network link between the VNC agent and the VNC terminal; an encoder shutdown unit, used to shut down the target video encoder and obtain a complete video file when the network link between the VNC agent and the VNC terminal is detected to be disconnected; and a video playback unit, used to push the complete video file to the client's browser and play the recorded video through the browser.

[0022] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute any of the above-described cloud environment secure screen recording methods based on Virtual Network Control (VNC).

[0023] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the cloud environment secure screen recording method based on Virtual Network Console (VNC) as described above.

[0024] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the cloud environment secure screen recording method based on a Virtual Network Console (VNC) as described above.

[0025] In this disclosure, server information and cloud resource metadata can be read after the user identity token in the access request initiated by the VNC terminal is verified. The server information includes at least a unique identifier for the cloud host, and the metadata includes at least a screen recording command. Each VNC server corresponds to one cloud resource. Based on the server information, a real-time bidirectional traffic proxy is executed using a started traffic replication coroutine. The real-time bidirectional traffic proxy includes: copying traffic from the VNC terminal to the VNC server, or copying traffic from the VNC server to the VNC terminal. The local changes from the VNC server to the VNC terminal are drawn onto a canvas, and a snapshot is extracted from the canvas at a predetermined frame rate. A watermark is added to the snapshot. The watermarked snapshot is encoded in real time using a target video encoder, and the encoded video stream is pushed to a real-time transmission server. The real-time transmission server is used to transmit the video stream to multiple clients for playback.

[0026] Based on the aforementioned publicly available information, a unified identity token can be used for identity verification, shielding against interference from other backend terminals. Real-time video streaming can be rendered on a canvas without requiring full screen information, making it lightweight and efficient. It supports pushing to a real-time transmission server for multi-device real-time viewing, preventing user operation risks, tracing user errors, improving business process compliance and operational standardization, and significantly enhancing the efficiency of multi-device collaborative work. This solves the technical problem in related technologies where screen recording in a cloud environment cannot be shared in real-time across multiple devices, reducing the efficiency of multi-device collaborative work. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0028] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a secure screen recording method in a cloud environment based on the Virtual Network Control (VNC) console is shown.

[0029] Figure 2 This is a flowchart of an optional cloud environment secure screen recording method based on a Virtual Network Control (VNC) according to an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of another optional cloud environment secure screen recording system based on a Virtual Network Control (VNC) according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of an optional cloud environment secure screen recording device based on a Virtual Network Control (VNC) according to an embodiment of the present invention;

[0032] Figure 5 This is a structural block diagram of an electronic device for executing a secure screen recording method for a cloud environment based on a Virtual Network Console (VNC) according to an embodiment of this application. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0036] Virtual Network Computing (VNC) is a graphical desktop sharing system that allows users to remotely control the desktop of another computer over a network.

[0037] A VNC proxy is a middleware layer that sits between a VNC client and a VNC server. Its main function is to forward and manage communication between the two, providing additional security, feature enhancements, or network optimizations.

[0038] Remote Frame Buffer (RFB) is a communication protocol used to implement remote desktop access. It describes and transmits screen image data, allowing users to control the desktop of a remote computer over a network, transmit the screen image data of the remote computer to the local end, and transmit keyboard and mouse events from the local end to the remote computer.

[0039] High Efficiency Video Coding (H.264) is a widely used video compression standard that effectively compresses video data, reducing storage space and network bandwidth requirements while maintaining high-quality video output.

[0040] A Universally Unique Identifier (UUID) is a 128-bit numerical identifier used to uniquely identify information in a distributed network environment. In a cloud environment, a UUID can be used to uniquely identify each cloud resource, such as an Elastic Cloud Server (ECS), to enable resource tracking and management.

[0041] Elastic Cloud Server (ECS) is a part of cloud computing services that provides users with virtual server resources that can be used on demand. Users can freely scale up and down computing resources, such as CPU, memory, disk space, and network bandwidth, according to their needs, making it ideal for dynamic and highly available application deployments.

[0042] Real-Time Messaging Protocol (RTMP) is a protocol used for real-time transmission of audio, video, and data. It enables the broadcasting of real-time streaming media and can be used in scenarios such as online live streaming and video conferencing.

[0043] It should be noted that the cloud environment secure screen recording method and device based on the Virtual Network Control (VNC) disclosed herein can be used in the field of cloud computing technology for secure screen recording in a cloud environment based on a VNC agent, and can also be used in any field other than cloud computing technology for secure screen recording in a cloud environment based on a VNC agent. This disclosure does not limit the application field of the cloud environment secure screen recording method and device based on the Virtual Network Control (VNC).

[0044] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) collected in this public disclosure are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0045] It should be noted that in this disclosure, customer information is collected and analyzed, and users are provided with corresponding operation entry points to choose whether to agree to or reject the automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0046] The following embodiments of the present invention can be applied to various cloud-based secure screen recording systems / applications / devices based on Virtual Network Control (VNC). This invention can be applied to cloud computing scenarios to prevent operational risks (monitoring high-privilege users' modification records, access to core data, bypassing financial approval processes to tamper with transaction data, and other high-risk operations), and to trace user errors, enabling rapid loss mitigation (due to the high complexity of financial institutions' systems, errors such as incorrect execution of batch scripts or accidental deletion of production databases can cause significant service interruptions or financial losses; this invention allows for rapid location of errors through screen recording, accelerating recovery and clarifying responsibility), thereby ensuring business process compliance and operational standardization. It can ensure that the operation of core transaction systems (such as payment and settlement systems, credit approval systems, etc.) conforms to established processes and standards, avoiding systemic risks caused by unauthorized operations.

[0047] This invention, through real-time screen recording, can also introduce traffic monitoring and proactive blocking mechanisms, effectively preventing and responding to abnormal operations, protecting cloud resources from external attacks, and using identity token authentication mechanisms to avoid directly exposing cloud resource information, thereby increasing the system's security level.

[0048] This invention supports pushing screen-recorded videos to a streaming media server in real time via a real-time messaging protocol, enabling real-time viewing and off-site backup across multiple terminals, thus enhancing the practicality of remote collaboration and remote monitoring.

[0049] The present invention will now be described in detail with reference to various embodiments.

[0050] Example 1

[0051] According to an embodiment of the present invention, an embodiment of a secure screen recording method for a cloud environment based on a Virtual Network Control (VNC) console is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0052] The cloud environment secure screen recording method based on the Virtual Network Control (VNC) provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a secure screen recording method in a cloud environment based on the Virtual Network Control (VNC) console is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1 (Illustrated using 102a, 102b, ..., 102n) Processor 102 (processor 102 may include, but is not limited to, a microprocessor MCU (Microcontroller Unit) or a programmable gate array (FPGA), etc.), memory 104 for storing data, and transmission device 106 for communication functions. In addition, it may include: a display, input / output interface (I / O interface), Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), network interface, power supply, and / or camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0053] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits may be referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0054] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the cloud environment secure screen recording method based on the Virtual Network Console (VNC) in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned cloud environment secure screen recording method based on the Virtual Network Console (VNC). The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0055] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0056] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0057] According to one aspect of the present invention, a secure screen recording system for a cloud environment based on a Virtual Network Console (VNC) is first described, comprising: a VNC terminal, which receives information about a cloud server selected by a user, generates a login request based on the cloud server information and cloud platform authentication information, and sends the login request to a VNC agent; the VNC agent generates a user identity token based on the cloud platform authentication information and cloud server information, returns the user identity token to the VNC terminal, and the VNC terminal initiates an access request based on the user identity token; a VNC server, a cloud platform underlying virtualization component, which provides independent cloud resource access for each network connection; and a VNC agent, which establishes a network link with the VNC terminal through a full-duplex communication protocol, establishes a network link with the VNC server through a transmission control protocol, and executes a secure screen recording method for a cloud environment based on a Virtual Network Console (VNC).

[0058] For the VNC terminal: acting as the front-end interface between the user and cloud resources, it collects information related to the cloud server selected by the user, including the cloud host's unique identifier, physical node, VNC address, VNC port, and whether screen recording is required. This information constitutes part of the VNC login request, informing the system of the specific cloud resource and its configuration that the user intends to access. Then, based on the collected cloud server information and the cloud platform's user authentication information, the VNC terminal generates a login request, combining the user's identity with the access request for the selected cloud resource to form a complete request message. This message is then sent to the VNC agent, serving as the starting point for accessing the cloud resource.

[0059] The VNC proxy, upon receiving a login request from the VNC terminal, generates a user identity token based on the cloud platform authentication information and cloud server information contained in the request. This token serves as the pass for all subsequent VNC access operations, containing user access permissions and the identifier of the target cloud resource, but without directly exposing the network location information of the cloud resource, thus improving the security of the access process. The VNC proxy establishes a persistent network connection with the VNC terminal via a full-duplex communication protocol, such as WebSocket. Simultaneously, it uses a transmission control protocol (such as TCP) to establish a connection with the underlying VNC server of the cloud platform, forming a bridge between the VNC terminal and the target cloud resource.

[0060] VNC server: Located in the underlying virtualization components of the cloud platform, its main responsibility is to provide independent access to cloud resources for each network connection initiated through the VNC agent. Whenever a VNC terminal requests access through the VNC agent, the VNC server creates a dedicated session to ensure that each user's operations are performed in a secure and isolated environment, preventing mutual interference between operations.

[0061] This embodiment uses user identity tokens for access control, avoiding the direct transmission of sensitive cloud server information, such as IP addresses and port numbers, over the network. This reduces the risk of man-in-the-middle attacks or data leaks. In addition, this embodiment also allows the establishment of encrypted network links between the VNC agent and the VNC terminal, ensuring the secure transmission of all communication data.

[0062] By involving a VNC agent, this embodiment simplifies the process of VNC terminals accessing specific cloud resources. Users do not need to directly deal with complex network configurations, but only need to interact with the VNC agent, which is responsible for transparently managing cloud server information and security authentication processes.

[0063] To achieve the above objectives, according to one aspect of this application, a cloud environment secure screen recording method based on the Virtual Network Console (VNC) is provided, which is applied to the VNC agent.

[0064] Under the aforementioned operating environment, this application provides the following: Figure 2 The method shown is a secure screen recording method for cloud environments based on the Virtual Network Control (VNC) console. Figure 2 This is a flowchart of an optional cloud environment secure screen recording method based on a Virtual Network Control (VNC) according to an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps:

[0065] Optionally, before verifying the user identity token in the access request initiated by the VNC terminal, the method further includes: receiving a VNC login request initiated by the VNC terminal, wherein the VNC login request includes: cloud platform authentication information and cloud server information selected by the user terminal, the cloud platform authentication information is used to determine the user's identity, and the cloud server information includes at least one of the following: unique identifier of the cloud host, physical node, VNC address and VNC port, and a command for screen recording; generating a user identity token based on the cloud platform authentication information and the cloud server information; and sending the user identity token to the VNC terminal, wherein after receiving the user identity token, the VNC terminal uses a browser to redirect to a fixed VNC entry point and initiates an access request carrying the user identity token.

[0066] In this embodiment, the cloud platform first receives a VNC login request initiated by the VNC terminal, enabling the cloud platform to identify the user and determine the specific cloud server resource being accessed. The request information may include two parts: cloud platform authentication information and cloud server information selected by the user. The cloud platform authentication information verifies the user's identity, ensuring that only authorized users can access the specified cloud resource. In a typical cloud computing environment, authentication information may include user ID, password, API key, access token, or other forms of identity verification. Through a rigorous authentication process, this embodiment ensures the traceability and security of the operation. The cloud server information refers to the information of the cloud server that the user specifies to access when initiating the login request. Optionally, the cloud server information includes the following: a unique identifier for the cloud host, such as a universally unique identifier (UUID), used to uniquely distinguish and locate different cloud resources within the cloud platform; and a physical node, which refers to the actual physical server or data center location where the cloud resource runs. VNC Address and VNC Port: The network address and port used by the VNC protocol to establish a remote control connection. By specifying the correct address and port, users can find and control the target cloud server on the network; Screen Recording Command: Users can choose whether to enable the screen recording function and decide whether to record subsequent session operations.

[0067] Furthermore, based on the received cloud platform authentication information and cloud server information, the cloud platform can generate a user identity token (defined as a Token in this embodiment). The token is a key credential used to verify the user's identity in subsequent sessions, allowing the VNC terminal to access specific cloud servers without exposing detailed cloud resource information. The token contains the necessary information for the VNC agent to identify the user and the target cloud server, but masks sensitive details such as specific IP addresses and port numbers, thereby improving the overall security of the system.

[0068] After the token is generated, the cloud platform sends it back to the VNC terminal. The VNC terminal then uses its browser to redirect to a fixed VNC entry point URL (Uniform Resource Locator). This URL is a pre-defined unified access point provided to all users by the cloud platform or VNC agent, simplifying the remote access process and enhancing the user experience. During the redirection process, the VNC terminal carries the previously obtained user identity token as authentication credentials for subsequent VNC sessions. When the VNC terminal initiates an access request carrying the user identity token, the VNC agent verifies it to confirm the user's legitimacy and the requested cloud resource access permissions.

[0069] In cloud computing environments, ensuring data security and user privacy is crucial. This embodiment effectively enhances system security and reduces the risk of man-in-the-middle attacks by using user identity tokens instead of directly exposing the cloud server's network information. Furthermore, the use of tokens prevents sensitive information from being transmitted over the network, providing an extra layer of protection. In addition, the token-based access control mechanism not only limits access permissions but also simplifies the interaction process between the cloud platform and the VNC terminal, enabling the cloud platform to flexibly manage and allocate tokens. Even when faced with a large number of concurrent VNC login requests, it ensures the security and independence of each session.

[0070] Step S201: If the user identity token in the access request initiated by the VNC terminal is verified, read the server information and the metadata of the cloud resource. The server information includes at least the unique identifier of the cloud host, and the metadata includes at least the screen recording command. Each VNC server corresponds to one cloud resource.

[0071] In this embodiment, when a VNC terminal (which can refer to a user terminal) initiates an access request, it first checks the user identity token sent with the request. This token may contain the user's identity information and access permissions. If the token passes verification, it indicates that the user has legitimate authorization to access specific cloud resources. At this time, the VNC proxy (which can be defined as VNCProxy in this embodiment) will read the server information and cloud resource metadata related to the request. In addition to basic login credentials, the user's request to access the VNC server may also include an instruction to enable screen recording. Based on this instruction, the VNC proxy can dynamically decide whether to record the operation between the user and the cloud resource. This allows for full recording of the operation process without affecting normal operation, preventing operational risks, tracing user errors, quickly stopping losses, ensuring business process compliance and operational standardization, and avoiding systemic risks caused by unauthorized operations.

[0072] When a user attempts to access these cloud resources, the VNC agent checks the screen recording command in the token and initiates the appropriate screen recording mechanism accordingly. This allows the screen recording operation to be executed automatically without additional manual intervention. Furthermore, since each VNC server corresponds to a cloud resource (such as ECS (Elastic Cloud Server) or cloud desktop), this ensures the accuracy and relevance of the screen recording data.

[0073] Step S202: Based on the server information, execute real-time bidirectional traffic proxy using the started traffic replication coroutine. The real-time bidirectional traffic proxy includes: replicating the traffic of the VNC terminal to the VNC server, or replicating the traffic of the VNC server to the VNC terminal.

[0074] Optionally, step S202 includes: establishing a network link between the VNC agent and the VNC terminal through a full-duplex communication protocol, and starting a first traffic replication coroutine to replicate the traffic of the VNC terminal to the VNC server; establishing a network link between the VNC agent and the VNC server through a transmission control protocol, and starting a second traffic replication coroutine to replicate the traffic of the VNC server to the VNC terminal.

[0075] In this embodiment, the VNC proxy server first establishes a network connection with the VNC terminal via a full-duplex communication protocol (e.g., WebSocket protocol). This protocol provides a persistent connection, enabling real-time, bidirectional data transmission. Next, this embodiment initiates a first traffic replication coroutine, which is responsible for capturing and replicating all traffic originating from the VNC terminal. This traffic may contain user commands to the VNC server, such as keyboard input, mouse clicks, and drag-and-drop actions. The first traffic replication coroutine accurately transmits these commands to the VNC server, ensuring that every user operation on the VNC terminal is reflected promptly and accurately on the remote server, maintaining a consistent user experience and operational effectiveness.

[0076] Similarly, in this embodiment, the VNC proxy server and the VNC server establish a network link through a transmission control protocol, such as TCP (Transmission Control Protocol). As a reliable, connection-oriented transport layer protocol, TCP can ensure that data packets arrive at their destination in the correct order and handle the retransmission and acknowledgment of data packets, making it very suitable for transmitting large or sensitive data.

[0077] Subsequently, this embodiment initiates a second traffic replication coroutine to capture and replicate the traffic sent back from the VNC server to the VNC terminal. This traffic may include image data of the server desktop and feedback results in response to user actions. The second traffic replication coroutine parses RFB packets, transmitting only the changed screen portions rather than the entire desktop refresh. This not only saves bandwidth but also improves transmission efficiency. The replicated traffic is transmitted to the VNC terminal in real time, allowing the user to see desktop changes synchronized with their actions, enhancing the realism and efficiency of remote operation.

[0078] Step S203: Draw the local changes from the VNC server to the VNC terminal onto the canvas, extract a snapshot from the canvas at a predetermined frame rate, and add a watermark to the snapshot.

[0079] Optionally, the step of drawing the partial change screen from the VNC server to the VNC terminal onto the canvas includes: preparing a blank canvas when the screen recording command indicates that the current cloud server needs to record the screen; parsing the partial change traffic from the VNC server to the VNC terminal using the remote desktop access protocol; capturing multiple frames from the partial change traffic to obtain the partial change screen; and drawing the partial change screen onto the canvas.

[0080] When the received screen recording command indicates that the current cloud server needs to be recorded, this embodiment first prepares a blank canvas to carry the screen information to be drawn. The canvas is located in the VNC proxy server, and its size matches the screen resolution of the cloud server to ensure the integrity and detail of the screen recording.

[0081] It should be noted that this embodiment primarily processes traffic sent from the VNC server to the VNC terminal, rather than the entire screen information. It effectively transmits the changed parts of the screen. By parsing these locally changing traffic segments, this embodiment can identify which areas of the screen have changed, and thus update only the corresponding portions of the canvas, rather than the entire screen, significantly reducing computational load and network traffic. After parsing the locally changing traffic, this embodiment further extracts multiple locally changing frames from the traffic, extracts and reassembles the transmitted image data to form a clear and coherent sequence of changing frames. Finally, these locally changing frames are drawn onto a previously prepared blank canvas. The purpose of this is to gradually construct a complete screen operation video stream through accumulated local updates, rather than processing the entire screen all at once.

[0082] Optionally, the step of adding a watermark to the snapshot includes: obtaining the timestamp corresponding to the current moment, the terminal IP of the VNC terminal, and user information; and adding a watermark containing the timestamp, terminal IP, cloud host unique identifier, and user information to the snapshot.

[0083] While generating video snapshots, this embodiment collects a series of key metadata, including the timestamp corresponding to the current moment, the terminal IP of the VNC terminal, and user information. The timestamp helps determine the specific capture time of the image; the terminal IP information can be used to trace the source of the operation; and the user information is crucial data to ensure that the operation can be traced back to a specific individual. Using the acquired metadata, this embodiment embeds a watermark on each video snapshot. The watermark content includes at least the timestamp, the terminal IP of the VNC terminal, the unique identifier of the cloud host, and the user information. In this way, each snapshot not only captures the instant of operation on the screen but also carries the operator's identity tag and operation timestamp, greatly improving the ability to prevent operational risks. It enables complete tracing of user misoperations, achieving rapid loss mitigation, and ensuring business process compliance and operational standardization.

[0084] To protect screen recording data from tampering, this embodiment employs a secure watermarking mechanism. The watermark not only contains the aforementioned metadata but can also be generated using encryption algorithms, ensuring that the watermark information remains unbreakable even if the video is captured or modified.

[0085] Additionally, it should be noted that, considering the varying requirements for screen recording quality and resource consumption in different scenarios, this embodiment supports the function of dynamically adjusting the pre-configured frame rate. The frame rate can be intelligently adjusted based on factors such as network conditions and system load to balance screen recording quality and system resource usage.

[0086] Step S204: Use the target video encoder to encode the watermarked snapshot in real time, and push the encoded video stream to the real-time transmission server, whereby the real-time transmission server is used to transmit the video stream to multiple clients for playback.

[0087] In step S204, this embodiment uses a target video encoder to encode the watermarked screen snapshot in real time. The target video encoder can adopt an efficient video encoding standard to convert each frame of the screen snapshot into a compressed video stream so that it can be transmitted over the network without consuming too much bandwidth resources. After encoding, this embodiment can push the video stream to a real-time transmission server. The real-time transmission server is responsible for transmitting the encoded video stream to multiple clients for real-time playback. It also supports training observers or other authorized users to access and play the video stream from different locations, thereby realizing multi-terminal sharing.

[0088] Optionally, after the encoded video stream is pushed to the real-time transmission server, the method further includes: obtaining the traffic interval configuration pre-configured for the access request, wherein the traffic interval configuration includes at least: a traffic interval threshold; and blocking the network link between the VNC agent and the VNC terminal if the video stream transmission traffic interval is greater than the traffic interval threshold.

[0089] In this embodiment, to further enhance security, a traffic interval configuration is pre-configured for access requests. This configuration item includes at least a traffic interval threshold, used to detect whether the video stream's transmission traffic is within the normal range. When the video stream's transmission traffic interval exceeds the set traffic interval threshold, it indicates a possible abnormal operation or network problem. In this case, this embodiment will take proactive security control measures, namely blocking the network link between the VNC proxy server and the VNC terminal. This helps prevent unauthorized access, detect potential hacking or network attacks, and quickly isolate problems in abnormal situations, reducing security risks. By setting a reasonable traffic interval threshold, potential threats can be detected and dealt with in a timely manner without affecting normal operation.

[0090] The characteristic of proactive security control is that it can achieve instant response without waiting for regular security approval checks or manual intervention, thereby enhancing the overall security and response speed of the system. For industries such as finance that have high requirements for data confidentiality and integrity, it can prevent unauthorized access or data leakage.

[0091] Optionally, after blocking the network link between the VNC agent and the VNC terminal, the method further includes: listening for connection disconnection events; upon detecting that the network link between the VNC agent and the VNC terminal has been disconnected, closing the target video encoder to obtain the complete video file; and pushing the complete video file to the client's browser for playback of the recorded video through the browser.

[0092] In this embodiment, when the network link between the VNC proxy server and the VNC terminal is blocked, the connection disconnection can be monitored further. If the connection between the VNC proxy server and the VNC terminal is indeed disconnected, the working video encoder is shut down to generate a complete video file. Finally, the video file is pushed to the client's browser, allowing the previously recorded video to be played directly through the browser.

[0093] In this embodiment, the video encoder is continuously invoked to encode the video stream of screen snapshots in real time. When a connection loss notification is received, the video encoder immediately stops the encoding process and integrates the encoded video segments into a single complete video file, ensuring that a complete record of user actions can still be obtained even in the event of an unexpected connection interruption.

[0094] By directly pushing the complete video file to the client's browser, this embodiment enables the playback of recorded videos without additional software or plugins. Users can easily view and analyze operation recordings on any device with a web browser, improving the usability and access convenience of operation recordings, while also reducing reliance on dedicated players and lowering the overall complexity of the system.

[0095] Through the above steps, if the user identity token in the access request initiated by the VNC terminal is verified, server information and cloud resource metadata can be read. The server information includes at least a unique identifier for the cloud host, and the metadata includes at least a screen recording command. Each VNC server corresponds to one cloud resource. Based on the server information, a real-time bidirectional traffic proxy is executed using the started traffic replication coroutine. The real-time bidirectional traffic proxy includes: copying traffic from the VNC terminal to the VNC server, or copying traffic from the VNC server to the VNC terminal. The local changes from the VNC server to the VNC terminal are drawn onto a canvas, and snapshots are extracted from the canvas at a predetermined frame rate. A watermark is added to the snapshots. The watermarked snapshots are encoded in real time using a target video encoder, and the encoded video stream is pushed to a real-time transmission server. The real-time transmission server is used to transmit the video stream to multiple clients for playback. In this embodiment, a unified identity token is used for authentication, shielding the interference from other backend terminals. The video stream is drawn in real time based on the canvas, without requiring full screen information. It is lightweight and efficient, supports pushing to a real-time transmission server for multi-terminal real-time viewing, can prevent user operation risks, trace user misoperations, improve business process compliance and operational standardization, and greatly improve the efficiency of multi-terminal collaborative work. This solves the technical problem in related technologies where screen recording in a cloud environment cannot be shared in real time across multiple terminals, reducing the efficiency of multi-terminal collaborative work.

[0096] The following describes in detail another optional implementation method.

[0097] Figure 3 This is a schematic diagram of another optional cloud environment secure screen recording system based on a Virtual Network Control (VNC) according to an embodiment of the present invention, such as... Figure 3 As shown, it mainly includes: a VNC client (corresponding to the VNC terminal mentioned above, which connects to the VNC agent through No VNC), a VNC agent, a VNC server, and a real-time transmission server (such as an H.264 audio and video streaming server). Each module is described below.

[0098] For the first part, the VNC client.

[0099] First, it should be noted that the VNC client is provided by noVNC (a lightweight, web-based VNC client that allows users to access and control remote computers or cloud server desktops via a web browser without downloading or installing any additional software (like traditional VNC clients). noVNC communicates with the VNC server by implementing the RFB (Remote Frame Buffer) protocol). Its operations include:

[0100] (1) Users select Elastic Cloud Server (ECS) within the cloud platform to log in via VNC;

[0101] (2) The VNC login request is sent to the VNC agent;

[0102] (3) The VNC agent generates a user identity token (VNC Token) based on the cloud platform authentication information (user) and specific cloud server information (such as UUID, physical node, VNC address and VNC port, whether recording, whether active blocking, etc.).

[0103] (4) The browser redirects to a fixed VNC entry point and carries the user's identity token (the entire VNC connection process will not expose ECS information, thus improving security).

[0104] For the second part, the VNC agent.

[0105] It should be noted that the VNC proxy can achieve VNC proxy, ECS IP and port information masking, and screen recording functions. Specifically, it can include the following steps:

[0106] (1) The VNC agent receives the request from the VNC client and establishes a full-duplex communication connection;

[0107] (2) The VNC agent verifies the token information and reads the VNC server information and metadata (used for watermarking and control);

[0108] (3) Connect to the VNC server via Transmission Control Protocol (TCP);

[0109] (4) Start the coroutine to copy the traffic of the full-duplex communication connection to the transmission control protocol connection (client traffic to server);

[0110] (5) Initiate a coroutine to copy the traffic of the Transmission Control Protocol connection to the full-duplex communication connection (server traffic to client);

[0111] (6) Determine whether the current cloud server needs screen recording;

[0112] (7) Start a coroutine to copy traffic from the server to the client in a cached manner;

[0113] (8) Prepare the Canvas;

[0114] (9) Use a remote framebuffer protocol (e.g., RFB protocol) to parse the copied server-to-client partial change traffic and draw it onto the canvas (without capturing and displaying the entire amount, only the changed area needs to be parsed and drawn each time). Figure 3The diagram illustrates the remote frame decoder (JI 1RFB Decode), which decodes locally varying traffic (such as...). Figure 3 (The metadata in the image is drawn onto the canvas.)

[0115] (10) Prepare a high-resolution video encoder (such as H.264Encode) to generate an H.264Stream, which is an H.264 video stream (which can be generated in a data storage server & real-time message transmission protocol server (i.e., Stroage & RTMP Server)) and send it to each playback end or auditing end (such as Play / Audit);

[0116] (11) Customize the acquisition of canvas snapshots by configuring the frame rate (which is faster and has very low resource consumption than reading desktop data and encoding it into images, because the canvas itself is in memory and is already the latest display).

[0117] (12) Draw the time, client IP, cloud host UUID (Universally Unique Identifier), and user information onto the snapshot;

[0118] (13) Put the snapshot into the video encoder to generate video in real time;

[0119] (14) Push video streams to real-time transmission servers (e.g., based on the RTMP (Real-Time Messaging Protocol) server) for real-time viewing and remote archiving of approval processes, ensuring compliance and standardization of financial business processes.

[0120] (15) Detect the traffic interval on the VNC client side according to the configuration to determine whether to actively block the connection;

[0121] (16) Provide an interface for risk control or operation and maintenance personnel to actively disconnect specific VNC agent links.

[0122] This implementation supports configuring safe operation intervals to actively block VNC connections and passively blocking VNC connections via interfaces.

[0123] Part Three: VNC Server.

[0124] It should be noted that the VNC server in this embodiment is provided by the underlying virtualization of the cloud platform. This invention proposes a secure screen recording solution for cloud environments based on the Virtual Network Control (VNC) console, performing protocol parsing only on unidirectional traffic. The solution includes:

[0125] (1) Each VNC server corresponds to one cloud resource (ECS Elastic Cloud Host or Cloud Desktop);

[0126] (2) Using noVNC, the browser can be used as a VNC client for cloud server management without any additional dependencies;

[0127] (3) Unify the VNC agent entry point to access different cloud backends and identify them according to the identity token;

[0128] (4) The VNC agent uses the WebSocket protocol to receive connection requests from the VNC client, uses the TCP protocol to connect to the VNC server, and starts two task goroutines to copy traffic from the VNC client to the VNC server and copy traffic from the VNC server to the VNC client, respectively.

[0129] (5) When copying traffic from the VNC server to the VNC client, determine whether screen recording is required;

[0130] (6) Copy the VNC server traffic for RFB protocol parsing, and only parse the screen change area each time and then draw it on the canvas to construct the screen display of cloud resources in real time.

[0131] (7) Create a video encoder;

[0132] (8) Get canvas snapshots periodically according to the set frame rate;

[0133] (9) Draw the time, client IP, cloud host UUID and user information watermark on the snapshot, and pass it to the encoder for real-time encoding;

[0134] (10) Push the canvas stream to the RTMP service according to the optional configuration to enable real-time viewing on multiple devices;

[0135] (11) Based on the optional configuration, if the VNC client traffic interval is greater than the threshold, actively disconnect;

[0136] (12) Listen for disconnection events and close the encoder to obtain the complete video file;

[0137] (13) The browser plays the recorded video directly.

[0138] Through the above embodiments, a unified user identity token can be used to shield the backend IP and port, and the watermark information is rendered in real time without post-processing, making it more secure and reliable.

[0139] Meanwhile, embodiments of the present invention can use asynchronous coroutines to parse data packets of unidirectional local frames based on a memory canvas and draw them in real time without requiring full frame information, making it lightweight and efficient, and encoding them into high-definition video format according to the frame rate.

[0140] In addition, when encoding video according to the frame rate, it supports pushing screen recording videos to the streaming media server in real time via a real-time message transmission protocol, enabling real-time viewing and off-site backup on multiple terminals, and enhancing the practicality of remote collaboration and remote monitoring.

[0141] The following is a detailed description with reference to another embodiment.

[0142] Example 2

[0143] The cloud environment secure screen recording device based on the Virtual Network Control (VNC) provided in this embodiment includes multiple implementation units, each of which corresponds to the implementation steps in Embodiment 1 above. The specific implementation methods and beneficial effects can be referred to the aforementioned method embodiments, and will not be repeated here.

[0144] According to another aspect of the present invention, a cloud environment secure screen recording device based on a Virtual Network Console (VNC) is also provided, which is applied to a VNC agent.

[0145] Figure 4 This is a schematic diagram of an optional cloud environment secure screen recording device based on a Virtual Network Control (VNC) according to an embodiment of the present invention, such as... Figure 4 As shown, the cloud environment secure screen recording device based on the Virtual Network Control (VNC) console may include: an information reading unit 41, a bidirectional traffic proxy unit 42, a canvas drawing unit 43, and a video stream pushing unit 44.

[0146] The information reading unit 41 is used to read server information and cloud resource metadata when the user identity token in the access request initiated by the VNC terminal is verified. The server information includes at least the unique identifier of the cloud host, and the metadata includes at least the screen recording command. Each VNC server corresponds to one cloud resource.

[0147] The bidirectional traffic proxy unit 42 is used to perform real-time bidirectional traffic proxy based on server information and by using the started traffic replication coroutine. The real-time bidirectional traffic proxy includes: copying the traffic of the VNC terminal to the VNC server, or copying the traffic of the VNC server to the VNC terminal.

[0148] The canvas drawing unit 43 is used to draw the local changes from the VNC server to the VNC terminal onto the canvas, extract snapshots from the canvas at a predetermined frame rate, and add watermarks to the snapshots.

[0149] The video stream push unit 44 is used to encode the watermarked snapshot in real time using the target video encoder and push the encoded video stream to the real-time transmission server, whereby the real-time transmission server is used to transmit the video stream to multiple clients for playback.

[0150] The aforementioned cloud environment secure screen recording device based on the Virtual Network Control (VNC) console can read server information and cloud resource metadata through the information reading unit 41 when the user identity token in the access request initiated by the VNC terminal is verified. The server information includes at least a unique identifier for the cloud host, and the metadata includes at least a screen recording instruction. Each VNC server corresponds to one cloud resource. Based on the server information, the bidirectional traffic proxy unit 42 performs real-time bidirectional traffic proxy using the started traffic replication coroutine. The real-time bidirectional traffic proxy includes copying traffic from the VNC terminal to the VNC server, or copying traffic from the VNC server to the VNC terminal. The canvas drawing unit 43 draws the partial changes from the VNC server to the VNC terminal onto the canvas, extracts snapshots from the canvas at a predetermined frame rate, adds watermarks to the snapshots, and uses the target video encoder 44 to encode the watermarked snapshots in real time. The encoded video stream is then pushed to the real-time transmission server, which transmits the video stream to multiple clients for playback. In this embodiment, a unified identity token is used for authentication, shielding the interference from other backend terminals. The video stream is drawn in real time based on the canvas, without requiring full screen information. It is lightweight and efficient, supports pushing to a real-time transmission server for multi-terminal real-time viewing, can prevent user operation risks, trace user misoperations, improve business process compliance and operational standardization, and greatly improve the efficiency of multi-terminal collaborative work. This solves the technical problem in related technologies where screen recording in a cloud environment cannot be shared in real time across multiple terminals, reducing the efficiency of multi-terminal collaborative work.

[0151] Optionally, the bidirectional traffic proxy unit includes: a first network link establishment module, used to establish a network link between the VNC proxy and the VNC terminal through a full-duplex communication protocol, and to start a first traffic replication coroutine to replicate the traffic of the VNC terminal to the VNC server; and a second network link establishment module, used to establish a network link between the VNC proxy and the VNC server through a transmission control protocol, and to start a second traffic replication coroutine to replicate the traffic of the VNC server to the VNC terminal.

[0152] Optionally, the canvas drawing unit includes: a canvas preparation module, used to prepare a blank canvas when the screen recording command instructs the current cloud server to record the screen; a traffic parsing module, used to parse the local variable traffic from the VNC server to the VNC terminal using the remote desktop access protocol; a screen capture module, used to capture multiple frames from the local variable traffic to obtain the local variable screen; and a canvas drawing module, used to draw the local variable screen onto the canvas.

[0153] Optionally, when the canvas drawing unit adds a watermark to the snapshot, it includes: a current terminal information acquisition module, used to acquire the timestamp corresponding to the current moment, the terminal IP of the VNC terminal, and user information; and a watermark adding module, used to add a watermark containing the timestamp, terminal IP, cloud host unique identifier, and user information to the snapshot.

[0154] Optionally, the cloud environment secure screen recording device based on the Virtual Network Console (VNC) further includes: a login request receiving unit, used to receive a VNC login request initiated by the VNC terminal before verifying the user identity token in the access request initiated by the VNC terminal, wherein the VNC login request includes: cloud platform authentication information and cloud server information selected by the user terminal, the cloud platform authentication information is used to determine the user identity, and the cloud server information includes at least one of the following: unique identifier of the cloud host, physical node, VNC address and VNC port, and a command for whether to record the screen; a token generation unit, used to generate a user identity token based on the cloud platform authentication information and the cloud server information; and a token sending unit, used to send the user identity token to the VNC terminal, wherein after receiving the user identity token, the VNC terminal uses a browser to redirect to a fixed VNC entry point and initiates an access request carrying the user identity token.

[0155] Optionally, the cloud environment secure screen recording device based on the Virtual Network Control (VNC) console further includes: a traffic interval configuration unit, used to obtain the traffic interval configuration pre-configured for the access request after the encoded video stream is pushed to the real-time transmission server, wherein the traffic interval configuration includes at least: a traffic interval threshold; and a network link blocking unit, used to block the network link between the VNC agent and the VNC terminal when the video stream transmission traffic interval is greater than the traffic interval threshold.

[0156] Optionally, the cloud-based secure screen recording device based on the Virtual Network Control (VNC) console further includes: an event listening unit, used to listen for connection disconnection events after the network link between the VNC agent and the VNC terminal is blocked; an encoder shutdown unit, used to shut down the target video encoder and obtain a complete video file when the network link between the VNC agent and the VNC terminal is detected to be disconnected; and a video playback unit, used to push the complete video file to the client's browser for playback of the recorded video through the browser.

[0157] The aforementioned cloud environment secure screen recording device based on the Virtual Network Control (VNC) console may also include a processor and a memory. The aforementioned information reading unit 41, bidirectional traffic proxy unit 42, canvas drawing unit 43, video stream push unit 44, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0158] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, secure screen recording in a cloud environment based on the Virtual Network Control (VNC) console can be achieved.

[0159] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0160] Example 3

[0161] Embodiments of this application may provide an electronic device. Figure 5 This is a structural block diagram of an electronic device for executing a secure screen recording method in a cloud environment based on a Virtual Network Control (VNC) console, according to an embodiment of this application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 Only one of the components is shown: processor 502, memory 504, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module, and display.

[0162] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the cloud environment secure screen recording method and device based on the Virtual Network Console (VNC) in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned cloud environment secure screen recording method based on the Virtual Network Console (VNC). The memory may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0163] The processor can access information and applications stored in memory via the transmission device to perform the following steps: If the user identity token in the access request initiated by the VNC terminal is verified, read server information and cloud resource metadata, wherein the server information includes at least: a unique identifier for the cloud host, and the metadata includes at least: a screen recording command, with each VNC server corresponding to one cloud resource; based on the server information, execute real-time bidirectional traffic proxy using the started traffic replication coroutine, wherein the real-time bidirectional traffic proxy includes: copying traffic from the VNC terminal to the VNC server, or copying traffic from the VNC server to the VNC terminal; draw the localized changes from the VNC server to the VNC terminal onto a canvas, extract snapshots from the canvas at a predetermined frame rate, and add watermarks to the snapshots; use a target video encoder to encode the watermarked snapshots in real time, and push the encoded video stream to a real-time transmission server, wherein the real-time transmission server is used to transmit the video stream to multiple clients for playback.

[0164] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0165] Those skilled in the art will understand that all or part of the steps in the various cloud environment secure screen recording methods based on the Virtual Network Control (VNC) in the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0166] Example 4

[0167] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the cloud environment secure screen recording method based on the Virtual Network Console (VNC) provided in Embodiment 1.

[0168] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute any one of the cloud environment secure screen recording methods based on the Virtual Network Control (VNC) described in Embodiment 1 above.

[0169] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0170] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the cloud environment secure screen recording method based on Virtual Network Console (VNC) as described in various embodiments of this application.

[0171] This application also provides a computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the cloud environment secure screen recording method based on a Virtual Network Console (VNC) as described in various embodiments of this application.

[0172] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0178] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A secure screen recording method for a cloud environment based on the Virtual Network Control (VNC) console, characterized in that, Applications to the VNC agent side of the virtual network console include: If the user identity token in the access request initiated by the VNC terminal is verified, the server information and cloud resource metadata are read. The server information includes at least: a unique identifier for the cloud host, and the metadata includes at least: a screen recording command. Each VNC server corresponds to one cloud resource. Based on the server information, a real-time bidirectional traffic proxy is executed using the started traffic replication coroutine, wherein the real-time bidirectional traffic proxy includes: replicating the traffic of the VNC terminal to the VNC server, or replicating the traffic of the VNC server to the VNC terminal. The local changes from the VNC server to the VNC terminal are drawn onto the canvas, and a snapshot is extracted from the canvas at a predetermined frame rate, and a watermark is added to the snapshot; The watermarked snapshot is encoded in real time using a target video encoder, and the encoded video stream is pushed to a real-time transmission server, which is used to transmit the video stream to multiple clients for playback.

2. The cloud environment secure screen recording method based on Virtual Network Control (VNC) according to claim 1, characterized in that, The steps for performing real-time bidirectional traffic proxying using the initiated traffic replication coroutine include: A network link is established between the VNC agent and the VNC terminal using a full-duplex communication protocol, and the first traffic replication coroutine is started to replicate the traffic of the VNC terminal to the VNC server. A network link is established between the VNC agent and the VNC server using the Transmission Control Protocol (TCP), and a second traffic replication coroutine is started to replicate the traffic of the VNC server to the VNC terminal.

3. The cloud environment secure screen recording method based on Virtual Network Control (VNC) according to claim 1, characterized in that, The step of drawing the partial changes from the VNC server to the VNC terminal onto the canvas includes: When the screen recording command instructs the current cloud server to record the screen, prepare a blank canvas; The localized traffic changes from the VNC server to the VNC terminal are parsed using the Remote Desktop Access Protocol. Multiple images from the localized variable flow are captured to obtain the localized variable image; The localized changes are drawn onto the canvas.

4. The cloud environment secure screen recording method based on Virtual Network Control (VNC) according to claim 1, characterized in that, The steps for adding a watermark to the snapshot include: Obtain the current timestamp, the VNC terminal's IP address, and user information; Add a watermark to the snapshot containing a timestamp, terminal IP, cloud host unique identifier, and user information.

5. The cloud environment secure screen recording method based on Virtual Network Control (VNC) according to claim 1, characterized in that, Before verifying the user identity token in the access request initiated by the VNC terminal, the following steps are also included: Receive a VNC login request initiated by a VNC terminal, wherein the VNC login request includes: cloud platform authentication information and cloud server information selected by the user terminal, the cloud platform authentication information is used to determine the user's identity, and the cloud server information includes at least one of the following: cloud host unique identifier, physical node, VNC address and VNC port, and a command to record the screen; A user identity token is generated based on the cloud platform authentication information and the cloud server information; The user identity token is sent to the VNC terminal. After receiving the user identity token, the VNC terminal uses a browser to redirect to a fixed VNC entry point and initiates an access request carrying the user identity token.

6. The cloud environment secure screen recording method based on Virtual Network Control (VNC) according to claim 1, characterized in that, After the encoded video stream is pushed to the live transmission server, the following steps are also included: Obtain the traffic interval configuration pre-configured for the access request, wherein the traffic interval configuration includes at least: a traffic interval threshold; If the video stream transmission interval is greater than the specified interval threshold, the network link between the VNC agent and the VNC terminal is blocked.

7. The cloud environment secure screen recording method based on Virtual Network Control (VNC) according to claim 6, characterized in that, After blocking the network link between the VNC agent and the VNC terminal, the following is also included: Listen for connection disconnection events; If the network connection between the VNC agent and the VNC terminal is detected to be disconnected, the target video encoder is shut down to obtain the complete video file. The complete video file is pushed to the client's browser, and the recorded video is played through the browser.

8. A secure screen recording system for a cloud environment based on a Virtual Network Control (VNC) console, characterized in that, include: The VNC terminal receives information about the cloud server selected by the user, generates a login request based on the cloud server information and cloud platform authentication information, and sends the login request to the VNC agent. The VNC agent generates a user identity token based on the cloud platform authentication information and cloud server information, returns the user identity token to the VNC terminal, and the VNC terminal initiates an access request based on the user identity token. VNC server, a cloud platform underlying virtualization component, provides independent access to cloud resources for each network connection; The VNC agent establishes a network link with the VNC terminal through a full-duplex communication protocol and establishes a network link with the VNC server through a transmission control protocol, and executes the cloud environment secure screen recording method based on the virtual network console VNC as described in any one of claims 1 to 7.

9. A cloud environment secure screen recording device based on Virtual Network Control (VNC), characterized in that, Applications to the VNC agent side of the virtual network console include: The information reading unit is used to read server information and cloud resource metadata when the user identity token in the access request initiated by the VNC terminal is verified. The server information includes at least: a unique identifier for the cloud host, and the metadata includes at least: a screen recording instruction. Each VNC server corresponds to one cloud resource. A bidirectional traffic proxy unit is used to perform real-time bidirectional traffic proxying using a started traffic replication coroutine based on the server information, wherein the real-time bidirectional traffic proxying includes: replicating the traffic of the VNC terminal to the VNC server, or replicating the traffic of the VNC server to the VNC terminal. The canvas drawing unit is used to draw the local changes in the VNC server to the VNC terminal onto the canvas, extract snapshots from the canvas at a predetermined frame rate, and add watermarks to the snapshots. The video stream push unit is used to encode the watermarked snapshot in real time using a target video encoder and push the encoded video stream to a real-time transmission server, wherein the real-time transmission server is used to transmit the video stream to multiple clients for playback.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the cloud environment secure screen recording method based on the Virtual Network Console (VNC) as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the cloud environment secure screen recording method based on the Virtual Network Console (VNC) as described in any one of claims 1 to 7.

Images