Method for remote access

Abstract

A method of preventing sensitive data received via a remote access protocol from being output to a human operator. The method comprises: receiving, via a remote access protocol, a user interface of an application executing on a remote server from the remote server; receiving a text-based representation of the application from the remote server; identifying sensitive data in the user interface and / or the text-based representation of the application using one or more pre-selected filters; modifying the user interface to remove the identified sensitive data; outputting the modified user interface to one or more output peripherals of a computer for receipt by a human operator.

Description

Technical Field

[0001] This invention relates to a method for preventing sensitive data received via a remote access protocol from being output to a human operator. Background Technology

[0002] Remote access protocols allow human operators to interact remotely with applications running on remote servers. This remote interaction is achieved by sending the application's user interface (UI) from the remote server to the human operator's computer. Using the received UI, the human operator can understand the actions the application can perform and the controls available within it. The human operator can then decide which actions should be performed by the application and input the appropriate control signals into their computer. These control signals are then sent from the human operator's computer to the remote server, where the desired actions are executed.

[0003] Several known remote access protocols exist. One example is the Remote Frame Buffer (RFB) protocol, which sends the rendered graphical user interface (GUI) image from a remote server to the human operator's computer. A second example of a remote access protocol is Microsoft's Remote Desktop Protocol (RDP). RDP is similar to the RFB protocol, except that instead of sending the rendered GUI image from a remote server to the human operator's computer, it sends the data used to render the GUI image.

[0004] The problem with these remote access protocols is that the user interface (UI) received from the remote server onto the operator's computer can include sensitive data (i.e., personal information, financial information, medical information). This sensitive data is problematic if it is received on the human operator's computer and then output to the human operator, as they may not be authorized to view it. Furthermore, there is a risk of data corruption by the human operator. Data corruption of sensitive data can lead to accidental or illegal destruction, loss, alteration, unauthorized disclosure, or access, resulting in serious human consequences. Moreover, under the regulations of various jurisdictions, access to sensitive data by a human operator may be illegal. Therefore, methods to prevent sensitive data from being output to human operators are desirable. Summary of the Invention

[0005] This invention is defined by the independent claims, and further optional features are defined by the dependent claims.

[0006] In a first aspect of the invention, a method is provided to prevent sensitive data received via a remote access protocol from being output to a human operator's computer for execution. The method includes: receiving a user interface of an application executing on a remote server from a remote server via the remote access protocol; determining or receiving a text-based representation of the application from the remote server; identifying sensitive data in the user interface and / or the text-based representation of the application using one or more pre-selected filters; modifying the user interface to remove the identified sensitive data; and outputting the modified user interface to one or more output peripheral devices of the computer for reception by a human operator. In this manner, the output of sensitive data to a human operator using the application's UI and / or the application's text-based representation is prevented.

[0007] In one embodiment, the remote access protocol is a remote framebuffering protocol. For example, the remote server may be a Virtual Network Computing (VNC) server, and the computer may be a VNC client. In this embodiment, the received user interface is in the form of an image of the user interface, the modification step includes modifying the image of the user interface, and the output step includes outputting the modified image of the user interface. Therefore, this method can be used with a remote framebuffering protocol.

[0008] In another embodiment, the remote access protocol is a remote desktop protocol. In this embodiment, the received user interface is in the form of data for presenting an image of the user interface, the modification step includes modifying the data for presenting the image of the user interface, and the output step includes outputting the modified data to present an image of the modified user interface. Therefore, this method can be used with a remote desktop protocol.

[0009] The text-based representation of an application may include text extracted from the user interface and / or the application. Additionally or alternatively, the text-based representation of the application may include a programmatic representation of the application. In some embodiments, the programmatic representation includes the application's document object model. The text-based representation of the application is used to identify additional sensitive data that may not have been identified using the application's user interface. A particular advantage of using the text-based representation of the application is that information at the application's underlying level can be used to identify sensitive data, meaning more accurate identification of sensitive data.

[0010] When sensitive data is in text form, pre-selected filters include rules, such as regular expression matching rules. Alternatively, pre-selected filters include machine learning models configured to recognize text. Both alternatives allow for the identification of sensitive data in text form and thus prevent its output to human operators.

[0011] When sensitive data is in the form of images, pre-selected filters include machine learning models configured to recognize images. This allows for the identification of sensitive data in image form and thus prevents it from being output to human operators.

[0012] The method may also include pre-selecting one or more filters based on sensitive data in the application prior to the step of determining or receiving data from the remote server. Sensitive data may include personal information such as one or more of the following: name, address, date of birth, phone number, image of identification document, facial image. Additionally or alternatively, sensitive data may include financial information such as one or more of the following: credit card number, bank account number. Additionally or alternatively, sensitive data may include medical information. This allows one or more filters to be customized based on the type of sensitive data expected in the application.

[0013] The modified user interface can be configured to interact with the human operator via one or more input peripherals of the computer. This allows the human operator to send control input to a remote server via a remote access protocol. The unmodified user interface is not output to one or more output peripherals of the computer to ensure that sensitive data is not leaked to the human operator.

[0014] Typically, for remote framebuffering protocols and remote desktop protocols, the user interface is a graphical user interface. Furthermore, the step of receiving the user interface includes receiving the user interface of the desktop from the remote server, where the application's user interface forms part of the desktop's user interface. The application can be a web browser, but any type of desktop application is suitable for use with this method.

[0015] In a second aspect of the invention, a computer program comprising instructions is provided, which, when executed by a processor, cause the processor to perform the method of the first aspect of the invention.

[0016] In a third aspect of the invention, a computer-readable medium including instructions that, when executed by a processor, cause the processor to perform the method of the first aspect of the invention.

[0017] In a fourth aspect of the invention, a processor configured to perform the method of the first aspect of the invention is provided.

[0018] In a fifth aspect of the invention, a computing system is provided, comprising: a processor configured to perform the method of the first aspect of the invention; a remote server communicatively coupled to the processor; and a computer communicatively coupled to the processor and the remote server, the computer having one or more output peripherals configured to output a modified user interface and one or more input peripherals for reception by a human operator, wherein the modified user interface is configured to interact with the human operator via the one or more input peripherals.

[0019] In some embodiments, the computing system is a virtual network computing system, the remote server is a VNC server, and the computer is a VNC client. The input peripherals may include a keyboard and / or a mouse. The output peripherals may include a monitor. Attached Figure Description

[0020] Embodiments of the present invention are described below by way of example with reference to the accompanying drawings, wherein:

[0021] Figure 1 A computing system for implementing the present invention is described.

[0022] Figure 2 Depicting Figure 1 A detailed view of the computing system.

[0023] Figure 3A and Figure 3B An example remote access protocol for use in this invention is described.

[0024] Figure 4 An example user interface with sensitive data is depicted for use in this invention.

[0025] Figure 5 A flowchart illustrating the method according to the present invention is provided.

[0026] Figure 6A and Figure 6B Each depicted based on Figure 4 Example of a modified user interface.

[0027] Figure 7 An example implementation of the method of the present invention in robotic process automation (RPA) is described. Detailed Implementation

[0028] Figure 1A computing system 10 is shown, in which the method of the present invention is implemented according to one embodiment. The computing system 10 includes one or more computers 20, which are physically accessible to one or more operators 60. The computing system 10 also includes one or more remote servers 40. The remote server 40 is "remote" because it is located in a different location from the computers 20, making it not physically accessible to the human operator 60. In some cases, this may be because the remote server 40 is a virtual remote server. The computers 20 and the remote servers 40 are communicatively coupled to each other via at least one communication network 50. This communication coupling enables data to be communicated between the computers 20 and the remote servers 40. The at least one communication network 50 may include the Internet (i.e., IP, IPv4, IPv6), cellular networks (i.e., 3G, 4G, LTE, 5G), local area networks, cloud networks, wireless networks, or any other known communication networks. An administrator computer 70 is also present in the computing system 10 and is accessible to a human administrator 80 of the computing system 10. Administrator computer 70 is communicatively coupled to computer 20 and remote server 40, enabling human administrator 80 to maintain and control policies for interactions between computer 20 and remote server 40. The described computing system 10 is merely exemplary and modifications, including the removal or addition of system components, are possible.

[0029] Figure 2 It shows Figure 1 Selected aspects of the computing system 10 shown. Specifically, Figure 2 A computer 20 is shown communicating with a remote server 40 via a communication network 50. The computer 20 includes one or more software applications 21, a processor 22, a memory 23, one or more input peripherals 24, and one or more output peripherals 25. The processor 22 includes a central processing unit (CPU) and / or a graphics processing unit (GPU). The memory 23 includes data storage devices and / or semiconductor memory. Data storage devices may take the form of hard disk drives, solid-state drives, external drives, removable optical discs, and / or memory cards. Semiconductor memory may take the form of volatile memory (e.g., random access memory (RAM)) for temporary data storage and non-volatile memory (e.g., read-only memory (ROM), flash memory) for long-term data storage.

[0030] One or more applications 21 are stored as computer programs in memory 23 and executed on computer 20 via processor 22. These applications, which facilitate direct interaction with human operator 60 via input peripherals 24 and output peripherals 25, include operating systems (OS) and desktop applications. Examples of known operating systems include Microsoft Windows, macOS, and Linux. Examples of known desktop applications for computer 20' include web browsers such as Google Chrome, document applications such as Microsoft Word, and remote access applications, which will be discussed further herein. However, it should be understood that the invention is not limited to the specific applications mentioned herein.

[0031] As described above, computer 20 includes one or more input peripheral devices 24. The purpose of the input peripheral devices 24 is to enable a human operator 60 to send instructions to computer 20. Examples of input peripheral devices 24 include a mouse, keyboard, touchscreen, image scanner, barcode reader, game controller, microphone, digital camera, webcam, etc. Input peripheral devices 24 can be integrated with computer 20, such as those found in laptop computers, or can be external to computer 20, such as desktop computers. Human operator 60 uses input peripheral devices 24 to send instructions to computer 20 by interacting with application 21. In particular, human operator 60 uses input peripheral devices 24 to interact with the user interface (UI) of application 21. In the case of a graphical user interface (GUI), this interaction is achieved, for example, by pressing buttons, clicking, dragging, scrolling, etc., on the GUI via input peripheral devices 24.

[0032] Computer 20 also includes one or more output peripheral devices 25. The purpose of output peripheral devices 25 is to enable human operator 60 to receive information from computer 20. Examples of output peripheral devices 25 include display devices (e.g., computer monitors or projectors), printers, headphones, and computer speakers. Similar to input peripheral devices 24, output peripheral devices 25 may be integrated with computer 20 or may be external to computer 20. Human operator 60 receives information from computer 20 using output peripheral devices 25 by using their senses, such as sight or hearing, to understand the UI of application 21.

[0033] Other components exist in computer 20 ( Figure 2 (Not shown in the image). For example, computer 20 includes one or more of the following: a network adapter card, a power supply, a motherboard, a sound card, etc., for enabling communication through communication network 50.

[0034] Figure 2A remote server 40 is also shown. The remote server 40 includes one or more applications 41, a processor 42, a memory 43, and a machine interface 44. The one or more applications 41 are stored as computer programs in the memory 43 and executed on the remote server 40 via the processor 42. These applications do not have direct interaction with the human operator 60 via input peripherals 24 and output peripherals 25 (although indirect interaction via remote access protocols, as described below). Instead, the one or more applications 41 are applications that interact directly with the computer 20 via a communication network 50 and the machine interface 44. Example applications of the remote server 40 include the applications described above for the computer 20, as well as robotic process automation (RPA) applications, such as those described in U.S. Patent Application No. 14 / 053319 and U.S. Patent Application No. 10,469,572.

[0035] The remote server 40 can take the form of a single server or multiple servers, or optionally, a distributed server. Distributed servers operate by distributing processing and data across their constituent components.

[0036] Remote server 40 can be a physical remote server or a virtual remote server. When remote server 40 is a virtual remote server, application 41, processor 42, memory 43, and machine interface 44 are all virtual entities.

[0037] Figure 1 An optional remote platform 90 included in the computing system 10 is also shown. The remote platform 90 is a type of physical remote server that hosts one or more virtual remote servers 40. In the same manner as the remote server 40, the remote platform 90 is "remote" because it is located in a different location from the computer 20, making it physically inaccessible to a human operator 60. The computer 20 and the remote platform 90 are communicatively coupled to each other via at least one communication network 50. This communication coupling enables data to be communicated between the computer 20 and the remote platform 90 (and any one of the one or more virtual remote servers 40 hosted thereon). In addition to the one or more virtual remote servers 40, the remote platform 90 includes one or more applications, a processor, memory, and a machine interface (not shown). Example applications for the remote platform 90 include the applications mentioned above for the computer 20 and the remote server 40, as well as virtual machine applications for providing the one or more virtual remote servers 40. Remote access protocol

[0038] Although, as described above, human operator 60 cannot directly interact with remote server 40, human operator 60 can indirectly interact with remote server 40 via computer 20 using a remote access protocol. Specifically, the remote access protocol allows operator 60 to remotely interact with application 41 on remote server 40. This is achieved by receiving the UI of application 41 on computer 20 in one form or another, enabling human operator 60 to understand the controls available in application 41. Human operator 60 then inputs control signals into computer 20 via input peripheral device 24, and these control signals are transmitted from computer 20 to remote server 40 via communication network 50. Therefore, remote server 40 executes the control signals via processor 42 to induce interaction with application 41 on remote server 40.

[0039] Several known remote access protocols exist that operate in the manner described above, enabling their use with this invention. A first example of a remote access protocol used in this invention is the Remote Frame Buffer (RFB) protocol, which... Figure 3A As described in the text. The RFB protocol operates at the framebuffer level corresponding to the rendered GUI image, in this case, the rendered GUI image of the remote server 40. Therefore, the RFB protocol can be applied to any application 41 running on the remote server 40, including operating systems (i.e., Microsoft Windows, MacOS, etc.) and other applications such as web browsers.

[0040] The basic operations of the RFB protocol are as follows. Figure 3A As shown, computer 20 receives images from the framebuffer of remote server 40, i.e., GUI images presented by remote server 40. "Image" means the data is pixel data. The format and encoding of this pixel data vary as they are negotiated by computer 20 and server 40 to suit the specific scenario in which the RFB protocol is executed. These GUI images are sent sequentially, forming a "video" of the GUI of remote server 40 for understanding by human operator 60 via computer 20. This means that once operator 60 sends control signals to remote server 40 via input peripheral device 24 and computer 20, human operator 60 can see the result of the control signals in subsequent GUI images received from remote server 40. For example, if input peripheral device 24 is a mouse, and the control signal sent by human operator 60 is to move the mouse, then human operator 60 will see the mouse cursor moving on the desktop of remote server 40 in the subsequently received GUI image.

[0041] Figure 4 A specific example of a GUI image 400 sent via the RFB protocol is shown. Specifically, Figure 4A GUI image 400 depicting the desktop of server 40 is shown, illustrating multiple applications 41, including an operating system 410 (Microsoft Windows), a web browser 411 (Google Chrome), an RPA application 412 (Blue Prism), and a remote access application 413 (Real VNC). The web browser 411 has an open window 415 in which an electronic health record, including sensitive data 416, is being viewed. The GUI image 400 is typically sent as a whole to computer 20 for viewing by a human operator 60. Note that... Figure 4 The gray box in window 415 is used for simplification in the accompanying drawing and will be filled with text in the actual GUI image.

[0042] In the RFB protocol, computer 20 is referred to as the "client" and remote server 40 is referred to as the "server". Therefore, on computer 20, application 21 includes a client-side remote access application, while on server 40, application 41 includes a server-side remote access application. Typically, Virtual Network Computing (VNC) applications, such as Real VNC, are used to execute the RFB protocol. Therefore, remote server 40 can be a VNC server, and computer 20 can be a VNC client. Further explanation of the RFB protocol can be found in Richardson, T. and Levine, J., 2011, "The remote framebuffer protocol", IETF.

[0043] A second example of a remote access protocol used in this invention is Microsoft's Remote Desktop Protocol (RDP), which... Figure 3B As shown in the diagram. RDP is similar to the RFB protocol, except that instead of sending the rendered GUI image from the remote server 40 to the computer 20, it sends data used to render the GUI image. In other words, in RDP, the data from the remote server 40 is not pixel data. Once the computer 20 receives the data used to render the GUI image, the GUI is rendered by the computer 20 and displayed on the output peripheral device 25 for human operator 60 to receive. Additionally, RDP allows human operator 60 to interact with the application 41 appearing in the rendered GUI image using operating system commands including selection, copying, etc. In this example, the application 21 on the computer 20 includes a remote desktop connection to perform RDP.

[0044] Although the data received by computer 20 in the two examples described herein relates to the GUI of remote server 40, the present invention is not limited to GUIs and can be applied to other types of UIs. Furthermore, other remote access protocols, including Serial Line Internet Protocol (SLIP), Point-to-Point Protocol (PPP), Point-to-Point over Ethernet (PPPoE), Remote Access Service (RAS), Point-to-Point Tunneling Protocol (PPTP), etc., work with the present invention. Sensitive data

[0045] like Figure 4 As shown in the example, UI 400 received at computer 20 from remote server 40 via a remote access protocol may include sensitive data 416. Sensitive data, as referred to herein, is a special type of data that requires enhanced security considerations due to its cognitive content. Damage to sensitive data can result in accidental or unlawful destruction, loss, alteration, unauthorized disclosure, or access, which can have serious human consequences. For example, the permanent deletion of a person's medical records can have significant and lasting consequences for that person's health. Therefore, the storage and processing of sensitive data are regulated in various jurisdictions, such as through the EU's General Data Protection Regulation (GDPR) and the UK's Data Protection Act 2018.

[0046] Sensitive data may take the form of text or images. Sensitive data can include personal information, that is, information relating to an identified or identifiable natural person. For example, sensitive data may include names, addresses, dates of birth, telephone numbers, images of identification documents, and facial images. Other types of sensitive data include a person's location data, online identifiers, or one or more factors specific to a person's physical, physiological, genetic, mental, economic, cultural, or social identity. Additionally or alternatively, sensitive data may include financial information, such as credit card numbers and bank account numbers. As a further option, sensitive data may include medical information.

[0047] Return to reference Figure 4 This depicts a sample UI 400 of a remote server 40, which displays sensitive data 416. Specifically, Figure 4 A window 415 of a web browser 411 is shown, which has opened an electronic health record webpage containing sensitive data 416. In this example, the sensitive data 416 takes the form of personal information, including an identification document containing a facial image (shown on the left side of box 416) and personal details such as name, address, date of birth, and telephone number (shown on the right side of box 416).

[0048] Since human operator 60 may not be authorized to view the sensitive data, sensitive data received at computer 20 via a remote access protocol is problematic if it is output to human operator 60. Furthermore, there is a risk of data corruption via human operator 60. In addition, human operator 60's access to the sensitive data may be illegal under the regulations of various jurisdictions. Therefore, methods to prevent sensitive data from being output to operator 60 are desirable. Overview of the method

[0049] Figure 5 The present invention illustrates a method for preventing sensitive data received via a remote access protocol from being output to operator 60. Figure 5 The method involves a computer 20, a remote server 40, and a processor 30. The processor 30 can be one of processors 22 and 42, or a different processor than processors 22 and 42. Preferably, the processor 30 is the processor 42 of the remote server 40 (i.e., the method is executed on the server side) to reduce potential security risks. Furthermore, the processor 30, as processor 42, reduces the computational load on the computer 20.

[0050] like Figure 5 As shown, the method includes the following steps performed at processor 30: - Receive the UI of application 41 running on remote server 40 from remote server 40 via remote access protocol (step 510); - Determine or receive a text-based representation of application 41 from remote server 40 (step 520); - Use one or more pre-selected filters to identify sensitive data in the UI and / or text-based representation of application 41 (step 530); - Modify the UI to remove the identified sensitive data (step 540); - Output the modified UI to one or more output peripheral devices 25 of computer 20 so that it can be received by human operator 60 (step 550).

[0051] Therefore, the method of the present invention centers on a processor 30 located between a remote server 40 and a human operator 60, and controls what the human operator 60 receives from the UI of application 41. Specifically, the processor 30 uses the UI of application 41 and / or the text-based representation of application 41 to identify sensitive data, and thus modifies the UI to remove the sensitive data so that a UI not containing sensitive data is output to the human operator 60 on computer 20. In this way, the human operator 60 does not access sensitive data on the remote server 40 via remote access protocols, thereby reducing the risk of data corruption.

[0052] More specifically, in step 510, processor 30 receives the UI of application 41, which is executing on remote server 40, from remote server 40. As described above, this step is typical for remote access protocols, except that the UI is received at processor 30 instead of necessarily at computer 20. The form of the UI of application 41 depends on the application and the remote access protocol used. Application 41 can be any of the applications mentioned herein, including operating system applications or other applications such as web browsers, virtual machine applications, RPA applications, etc.

[0053] When using the RFB or RDP protocol, step 510 involves receiving the desktop GUI of the remote server 40 at the processor 30. However, in some instances, the application of interest 41 for interacting with the human operator 60 will be an application GUI residing on the desktop GUI, such as... Figure 4 The example is a web browser 411. In these examples, the GUI of the application 41 of interest is formed as part of the desktop GUI. This enables... Figure 4 As can be seen, the window 415 of the web browser 411 only forms a part of the desktop GUI 400.

[0054] The following steps 520 through 550 will be discussed in detail in the following sections. Specifically, step 520 will be discussed in the section titled "Text-Based Application Representation". Step 530 will be discussed under "Filtering and Filter Customization". Steps 540 and 550 will be discussed in the section titled "Modified User Interface". Text-based application representation

[0055] In step 520, the processor 30 either determines the text-based representation of application 41 or receives the text-based representation of application 41 from the remote server 40. The text-based representation of application 41 is a representation composed of alphanumeric characters including letters, numbers, punctuation marks, and other special characters. Letters can come from any writing system (e.g., Latin, Cyrillic, Korean, Arabic, Greek, etc.). Numbers can come from any number system (e.g., Arabic numerals, Chinese numerals, Japanese numerals, Roman numerals). Punctuation marks and other special characters include, for example, []! "#$%&'()*+,. / :;<=>?@\^_`|{}~-.

[0056] The purpose of the text-based representation of Application 41 is to provide additional information about the UI of Application 41, which is useful for identifying and subsequently removing sensitive data from the UI of Application 41. For example, the text-based representation of Application 41 may reveal that a field in the UI of Application 41 is a person's "name" field (e.g., <name="Betty Dixon"), while the UI itself may only reveal the person's actual name (e.g., "Betty Dixon"), and that name may not be in a form that can be easily extracted (e.g., it is pixel data rather than text). By specifically referencing the "name" and by providing the name text, the text-based representation of Application 41 provides additional contextual information, making sensitive data easier to identify than using the UI of Application 41 alone.

[0057] The text-based representation of application 41 can take many forms. In some instances, the text-based representation of application 41 includes text extracted from the UI and / or application 41. For example, in addition to being presented as gray squares... Figure 4 In addition to the text displayed in window 415, from Figure 4 The GUI image 400 text extraction will produce the following text: Electronic Health Record My Day Patient Patent Overview (My Day Patient) Problem Medications Implanted Devices Choose an Encounter... Today's Vitals Health Reminders Due Immunizations Due Date Tasks 11:45 13 / 11 / 2019".

[0058] Optical Character Recognition (OCR) is used to extract text. When using the RFB protocol, processor 30 uses OCR to extract text from the GUI image of application 41 sent from remote server 40. When using RDP, processor 30 extracts text by recognizing natural words and numbers in the data sent from remote server 40 used to render the GUI image of application 41. Alternatively, when using RDP, processor 30 uses OCR to extract text from the rendered GUI image of application 41.

[0059] In some instances, the text-based representation of Application 41 includes the programmatic representation of Application 41. The programmatic representation is a representation of the underlying program of Application 41. For example, the programmatic representation of Application 41 could be the Document Object Model (DOM) of Application 41 or the DOM of Application 41's UI. The DOM is an application programming interface for HTML and XML-based applications. The DOM defines the logical structure of Application 41. Specifically, tags (such as "name" in the above example <name="Betty Dixon">) become element nodes and form the structure of the DOM, while text (such as "Betty Dixon" in the same example) becomes text nodes. The DOM is a form of text such as JSON or XML. Further information about the DOM can be found in Hors, A. et al, 2004, "Document Object Model (DOM) Level 3 Core Specification", W3C Recommendation.

[0060] When using the RFB protocol, in order to receive the programmable representation of application 41, processor 30 sends a message to remote server 40 requesting the programmable representation. Alternatively, remote server 40 is programmed to send the programmable representation of application 41 whenever the remote access protocol is in use. As another alternative, the programmable representation can be determined by processor 30 using the UI of application 41. In the case where the programmable representation is a DOM, processor 30 uses a deep learning model to determine the DOM from the GUI image. An example of a suitable deep learning model for this is "pix2code". When using RDP, the programmable representation of application 41 is determined by processor 30 based on data sent from remote server 40 for rendering the GUI image.

[0061] Those skilled in the art will understand that, in addition to the forms mentioned herein, the text-based representation of application 41 is also applicable to this invention. Filtration and Custom Filters

[0062] Figure 5 Step 530 involves using one or more pre-selected filters to identify sensitive data in the UI of application 41 and / or in the text-based representation of application 41. The purpose of step 530 is to determine specific portions of the UI of application 41 received in step 510 that contain sensitive data, such that these portions can be modified in step 540. As mentioned above, the sensitive data may be in the form of text or images. Therefore, the pre-selected filters used for application 41 depend on whether the sensitive data is text-based, image-based, or a combination of both.

[0063] When sensitive data is in text form, a pre-selected filter includes rules. The text-based representation of application 41 is parsed to determine whether one or more parts of application 41's UI satisfy the rules and therefore contain sensitive data. For example, for application 41 related to personal data, the rules might determine the presence of country names (e.g., "United Kingdom," "United States of America," "Japan," "Korea," etc.) because these country names are likely to be a person's country of birth or residence, and are therefore sensitive data. The list of country names can be stored in a database, etc.

[0064] Combining rules across multiple DOM elements is particularly beneficial. This is because multiple element nodes across multiple DOM elements typically identify the information type of the corresponding text node. For example, in the `<name="Betty Dixon">` example above, the element node "name" identifies the text node "Betty Dixon" as a name, which is sensitive data. In this example, the rule is that if the element node is "name", then the corresponding text node is sensitive data. Thus, sensitive data can be easily identified using element nodes.

[0065] In some instances, a rule can be a regular expression (Regex) that matches a set of strings. A regular expression, often called a pattern, is an expression used to specify a set of strings in a concise way. This conciseness is achieved by using operators such as "?" indicating one or no occurrence (e.g., color matches both "color" and "colour"), "{min,max}" indicating the minimum and maximum number of matches, "[]" matching a single character enclosed in brackets, "^" matching the beginning of a string, "$" matching the end of a string, and others. For example, the format of a UK postcode is "area-selection-zone-unit", where: area is one or two letters; selection is one number, two numbers, or one number followed by a letter; zone is a single number; and unit is two letters. For example, "WA20XP", "E70 AQ", and "WC1B 5HA" are all valid UK postcodes. Searching for every possible combination of letters and numbers would be cumbersome, so the following regular expression is used instead. The basic regular expression for a UK postcode is: ^[AZ]{1,2}[0-9][A-Z0-9]? [0-9][AZ]{2}$ Here, the region is [AZ]{1,2}, the selection area is [0-9][A-Z0-9]?, the partition is [0-9], and the cell is [AZ]{2}. From this example, it can be understood that searching for matches of this regular expression in the text-based representation of application 41 is far more efficient than searching for every possible combination of letters and numbers. Furthermore, regular expressions tend to be suitable for sensitive data because the format of sensitive data is often known.

[0066] In some instances, pre-selected filters include machine learning models configured to recognize text. A suitable example machine learning model for this purpose is Natural Language Processing (NLP). NLP models use Named Entity Recognition (NEP) to identify sensitive data. NEP attempts to locate and classify named entities mentioned in unstructured text into predefined categories such as person names, locations, medical codes, time expressions, quantities, currency values, percentages, etc. For example, in the sentence “Betty Dixon has an appointment at an East London clinic on November 15, 2019 at 2:50 PM,” NEP produces a block of annotated text highlighting the names of the entities as follows: [Betty Dixon] 人 On [November 15, 2019] 时间 [14:50] 时间 At the [East London Clinic] 位置 Appointment In this example, person names, place names, and two time expressions have been detected and classified. NLP models are easy and efficient to train and run. Therefore, when sensitive data is in text form, it is preferable to use an NLP model to identify sensitive data.

[0067] When the sensitive data is in the form of images, such as identifying document images or facial images, or when the text-based representation of application 41 is not used, pre-selected filters include machine learning models configured to identify images from the GUI images of application 41. An example machine learning model suitable for this purpose is a computer vision model. The computer vision model uses contextual information to detect GUI elements (images, text boxes, buttons, icons, dropdown lists, etc.) that may contain sensitive data. For example, a text box next to the label "Name" is considered sensitive data. Object detection algorithms are used to detect these GUI elements. Furthermore, the computer vision model is capable of using object detection algorithms to detect certain objects such as facial images or images of identification documents.

[0068] Once sensitive data has been identified using one or more pre-selected filters, the location of the sensitive data in the GUI image must be identified when using the RFB protocol. This allows the GUI to be modified only at these locations in the GUI image (i.e., only at the pixels related to the sensitive data) in step 540. The identification of the location of the identified sensitive data is based on the method used to identify the sensitive data. If a machine learning model is used to identify the sensitive data from the GUI image, the machine learning model can also extract the location of the corresponding GUI element in the GUI image. Otherwise, one or more computer vision models are used to identify the location of the GUI element in the GUI image, depending on what kind of GUI element is being identified. In particular, if the GUI element corresponds to an image (e.g., an identity document image, a facial image), template matching is used on the GUI image to identify the image location. If the GUI element corresponds to text, text boxes are detected in the GUI image (e.g., using OCR) and then matched against the GUI element. The location of the sensitive data is typically in the form of bounding boxes in the GUI image, each bounding box being defined by the coordinates of the four corners of the corresponding bounding box or other pixel references.

[0069] As described above, one or more pre-selected filters are used. When more than one pre-selected filter is used (i.e., multiple pre-selected filters), each of these multiple pre-selected filters is different, making it possible to identify different sensitive data. When more than one pre-selected filter exists, there can be more than one type of pre-selected filter. For example, these multiple pre-selected filters may include a regular expression matching rule using the text-based representation of application 41, and a machine learning model using the GUI image of application 41.

[0070] Typically, the content of application 41 may be known before the remote access protocol is put into use, so filters are pre-selected to identify the expected sensitive data in the content of application 41. In other words, one or more pre-selected filters are customized based on the content of application 41. For example, if application 41 is a web browser that typically contains medical and personal information because remote server 40 is part of a medical institution, filters are pre-selected to identify that medical and personal information. Conversely, if application 41 is a webcam application, filters are pre-selected to identify facial images. To this end, the method of the present invention may optionally include pre-selecting one or more filters based on sensitive data in the application before the receiving step.

[0071] Figure 1The management computer 70 and the human administrator 80 maintain and control the policies for the interaction between the computer 20 and the remote server 40, so the human administrator 80 can use the management computer 70 to pre-select one or more pre-selected filters. To this end, the management computer 70 may contain a database of filters to allow the administrator 80 to customize one or more pre-selected filters for the content of application 41. For the human administrator 80 to select appropriate one or more filters, the management computer 70 includes a UI for setting one or more filters. For example, the UI could be a GUI that allows adding filters. The one or more pre-selected filters are then stored in a configuration file. Optionally, in step 540, the configuration file may also indicate how to modify certain types of sensitive data. For example, the configuration file may contain a command specifying "find National Insurance number and obfuscate field". Modified user interface

[0072] exist Figure 5 In steps 540 and 550, the UI received in step 510 is modified to remove the sensitive data identified in step 530, thereby forming a modified UI. The term "removal" in this context may mean removal or obfuscation. The modified UI is output to one or more output peripheral devices 25 of computer 20 for reception by human operator 60.

[0073] More specifically, for step 540, the method used to modify the UI to remove sensitive data depends on the remote access protocol used. This is because the form of the UI received from the remote server 40 in step 510 is specified by the remote access protocol. For example, when the remote access protocol is RFB, the received UI is in the form of a GUI image, so step 540 includes modifying the GUI image. In another example, when the remote access protocol is RDP, the received UI is in the form of data used to render the GUI image, such that step 540 includes modifying the data used to render the GUI image.

[0074] Several methods exist for modifying GUI images (i.e., when using the RFB protocol). One method is to use one or more image transformations to manipulate pixels related to sensitive data so that the sensitive data is no longer understandable to human operator 60. For example, the GUI image might be blurred at locations where sensitive data has been identified. The amount of blurring must be sufficient to render the sensitive data incomprehensible to human operator 60. An example of GUI image blurring is... Figure 6A The GUI image 600 is shown, which corresponds to Figure 4 The GUI image is 400. (Example) Figure 6A As shown, with Figure 4The corresponding box 616, which contains personal information including identity verification documents and personal details, has been obscured, making the sensitive data no longer understandable to human operators 60.

[0075] Applying a blurred alternative image transformation involves applying a solid color to locations within the GUI image where sensitive data has been identified. Figure 6B Example of a GUI image with an applied solid color is shown in GUI image 600. Specifically, as... Figure 6B As shown, a solid white color is applied to box 616, making the sensitive data incomprehensible to human operator 60. Other solid colors may also be used. In some cases, the chosen solid color is based on the background color of application 41, making it possible for operator 60 not to realize that the UI has been modified. Another optional image transformation is to reduce the image resolution (i.e., increase the pixelation of the GUI image). Those skilled in the art will understand that any image transformation that means the sensitive data is no longer comprehensible to human operator 60 is applicable to the method of the present invention.

[0076] Several methods exist for modifying data used to render GUI images (i.e., when using RDP). Typically, these methods edit the data to remove sensitive data, usually in text or image form, so that the rendered GUI image does not contain the sensitive data. This prevents human operator 60 from interacting with application 41 appearing in the rendered GUI image to extract sensitive data using operating system commands (e.g., select and copy). When the sensitive data is in text form, exemplary methods replace characters in the text with random characters or asterisks. For example, if the sensitive data to be removed is a name, the name "Betty Dixon" could be replaced with random characters such as "DLNN156 OLP" or "**********". Alternatively, characters could be replaced with spaces, so that human operator 60 might not be aware that the UI has been modified. When the sensitive data is in image form, image files can be removed from the data. To do this, image files are identified in the data by their file extensions (e.g., .tiff, .jpg, .gif, .png, etc.), and then removed from the data. Alternatively, one or more of the previously mentioned image transformations can be used to blur sensitive data in the image.

[0077] Typically, non-computationally intensive methods are preferred for modifying the UI to remove sensitive data, as this ensures minimal time between the processor 30 receiving the UI in step 510 and outputting the modified UI in step 550. This ensures that the UI of the remote server 40 responds quickly to input from the human operator 60 and therefore does not have high latency.

[0078] The form of the modified UI also depends on the remote access protocol used. This is because the modified UI takes the same form as the UI received from the remote server 40 in step 510. For the RFB protocol, the received UI is in the form of a GUI image, so the output of computer 20 is the modified UI in the form of a GUI image. For RDP, the received UI is in the form of data used to render a GUI image, so the output of computer 20 is modified data to render the modified GUI image.

[0079] Once the modified UI has been output to human operator 60 via output peripheral device 25 of computer 20, human operator 60 can interact with the modified UI in the same way they would interact with the original, unmodified UI without using the method of the present invention. In other words, human operator 60 can interact with application 41 on the modified UI using input peripheral device 24. Because the modifications made to the UI are specific to the type of remote access protocol, this ensures that human operator 60 cannot gain access to sensitive data by interacting with the modified UI using input peripheral device 24. For example, in the case of the RFB protocol, the UI is a GUI image, and because the image has been changed, human operator 60 cannot access sensitive data. In the case of RDP, the data used to render the GUI image is changed such that sensitive data is removed from the rendering of application 41, meaning that human operator 60 cannot (e.g., use the copy command in the operating system) select and copy sensitive data.

[0080] For the sake of completeness, it is worth noting that the original, unmodified UI is not output to one or more output peripherals 25 of computer 20. This would result in sensitive data being leaked to operator 60, thus defeating the purpose of the invention. Example

[0081] The disclosed method for preventing sensitive data received via a remote access protocol from being output to human operator 60 is useful for various implementations of the remote access protocol. For example, in outsourced technical support, it is common practice that human operator 60 (i.e., technical support staff) is not authorized to view sensitive data on remote server 40. However, human operator 60 must connect to remote server 40 via their computer 20 and the remote access protocol to provide technical support to remote server 40. Advantageously, by implementing the disclosed method, human operator 60 will not see sensitive data but will still be able to provide technical support.

[0082] One specific implementation of the remote access protocol used for the disclosed methods involves Robotic Process Automation (RPA). RPA uses automated processes that mimic the interaction of a human operator with an application's user interface (e.g., a GUI). In this way, the automated process can be considered a virtual operator. This kind of interaction with the application is necessary for applications that cannot be interacted with solely through a machine interface, such as applications without an API.

[0083] The automated processes in RPA are implemented using virtual machines, each with its own virtual desktop for interacting with virtual operators. Typically, multiple virtual machines run in parallel on a single physical server, performing the same or similar automated processes on the application. This ensures less physical hardware is required compared to using human operators (each requiring a physical computer). RPA provides secure and accurate interaction with the application because no operator or human judgment is required. Therefore, by using virtual operators instead of human operators to interact with the application, improved data security is achieved due to the reduced likelihood of data corruption. Further information on RPA can be found in U.S. Patent Application No. 14 / 053319 and U.S. Patent No. 10,469,572.

[0084] In RPA, it is sometimes useful for human operator 60 to check whether the virtual machine is operating correctly for debugging purposes. However, because virtual machines are often headless in the sense that they do not have a connected display device, and because virtual machines are often located far from human operator 60, the best way for human operator 60 to access the virtual machine is through a remote access protocol. However, this is problematic because human operator 60 can see all the data, including sensitive data, on the virtual desktop of the virtual machine. This significantly reduces the security of RPA. Therefore, the method of the present invention for preventing received sensitive data from being output to human operator 60 is useful for RPA.

[0085] Figure 7 An RPA implementation of the method of the present invention is shown. Figure 7 In this context, remote server 40 is a virtual remote server in the form of a virtual machine. Therefore, remote server 40 is capable of executing automated processes on application 41. Specifically, the automated processes are executed by a virtual operator of the virtual machine using the user interface of application 41. Typically, there is no human operator 60 involved, and it is impossible for the human operator 60 to directly see the automated processes executed by the virtual operator because the virtual machine is headless and located far from the human operator 60.

[0086] When operator 60 wants to see an automated process being executed by a virtual operator on a virtual machine (e.g., one of remote servers 40C, 40B, 40C), they do so via their computer 20 and a remote access protocol. Typically, multiple virtual machines are hosted by a remote platform 90. Figure 7 The virtual machines are shown by remote servers 40A, 40B, and 40C. A human operator 60 can use RPA application 21 via computer 20 to select which virtual machine he or she wishes to interact with. Remote desktop application 41 is able to send the user interface (including the user interface of application 41) of the virtual desktop of the selected virtual machine to computer 20. However, due to the method of the present invention, processor 30 ( Figure 7 (Not shown) Ensure that sensitive data is removed from the user interface before it is output to human operator 60 on computer 20.

[0087] For example, such as Figure 7 As shown, human operator 60 has selected remote server 40B on remote platform 90 and can therefore see the user interface of remote server 40B and interact with it via computer 20. Specifically, human operator 60 will see an automated process of interacting with application 41 on a virtual machine; in this example, the application is a web browser. However, because the user interface of remote server 40B contains sensitive data, this sensitive data has been removed from the user interface output to human operator 60 via computer 20 (this can be seen by comparing the corresponding user interfaces). Therefore, a level of security is maintained before human operator 60's involvement. Overview

[0088] When implemented in software, this invention can take the form of a computer program. A computer program can be embodied as a computer-readable medium having computer-executable code used by or in conjunction with a processor. A computer-readable medium is any tangible device that can contain, store, transmit, propagate, or transport a program used by or in conjunction with a processor. Furthermore, a computer-readable medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device or propagation medium. Examples of computer-readable media include semiconductor memory, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drives, solid-state drives, optical discs, and memory cards. Current examples of optical discs include CDs, DVDs, and Blu-rays. Current examples of memory cards include USB flash drives, SD cards, microSD cards, MMC cards, xD cards, and Memory Sticks.

[0089] When implemented in hardware, this invention is not limited to the specific hardware described herein. Those skilled in the art will understand that this invention can be implemented on hardware different from computer 20 and remote server 40, and still function in the manner described.

[0090] The flowcharts in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the method of the present invention. In some alternative embodiments, the steps mentioned in the figures may not occur in the order shown. For example, depending on the functionality involved, two steps shown consecutively may actually be performed substantially simultaneously, or these blocks may sometimes be performed in reverse order.

[0091] It should be understood that the above description is given by way of example only, and various modifications can be made by those skilled in the art. Although various embodiments have been described above with a certain degree of specificity or with reference to one or more individual examples, those skilled in the art can make many changes to the disclosed embodiments without departing from the scope of the invention. Example

[0092] The following list provides embodiments of the invention and forms part of this specification. These embodiments can be combined in any compatible combination beyond those expressly stated. Embodiments can also be combined with any compatible features described herein: Example 1. A method for preventing sensitive data received via a remote access protocol from being output to a human operator's computer, the method comprising: Receive the user interface of the application running on the remote server from the remote server via a remote access protocol; Determine or receive from the remote server a text-based representation of the application; Use one or more pre-selected filters to identify sensitive data in the application's user interface and / or text-based representation; Modify the user interface to remove the identified sensitive data; The modified user interface is output to one or more output peripherals of the computer so that it can be received by a human operator. Example 2. The method according to Example 1, wherein the remote access protocol is provided by a remote access application. Example 3. The method according to Example 1 or 2, wherein the remote access protocol is a remote frame buffer protocol. Example 4. According to the method of Example 3, wherein the received user interface is in the form of an image of the user interface, the modification step includes modifying the image of the user interface, and the output step includes outputting the modified image of the user interface. Example 5. The method according to Example 3 or 4, wherein the remote server is a Virtual Network Computing (VNC) server and the computer is a VNC client. Example 6. The method according to Example 1 or 2, wherein the remote access protocol is a remote desktop protocol. Example 7. The method according to Example 6, wherein the received user interface is in the form of data for presenting an image of the user interface, the modification step includes modifying the data for presenting the image of the user interface, and the output step includes outputting the modified data to present an image of the modified user interface. Example 8. The method according to any of the foregoing embodiments further includes pre-selecting one or more filters based on sensitive data in the application before determining or receiving from the remote server. Example 9. The method according to Example 8, wherein the sensitive data includes personal information. Example 10. According to the method described in Example 9, the personal information includes one or more of the following: name, address, date of birth, telephone number, image of identity document, and facial image. Example 11. The method according to any one of Examples 8 to 10, wherein the sensitive data includes financial information. Example 12. The method according to Example 11, wherein the financial information includes one or more of the following: credit card number, bank account number. Example 13. The method according to any one of Examples 8 to 12, wherein the sensitive data includes medical information. Example 14. The method according to any of the foregoing embodiments, wherein the step of receiving a user interface includes receiving a user interface of the desktop of the remote server, wherein the user interface of the application forms part of the user interface of the desktop. Example 15. The method according to any of the foregoing embodiments, wherein the text-based representation of the application includes text extracted from the user interface and / or the application. Example 16. The method according to any of the foregoing embodiments, wherein the text-based representation of the application includes the programmatic representation of the application. Example 17. The method according to Example 16, wherein the programming representation includes the document object model of the application. Example 18. The method according to any of the foregoing embodiments, wherein the pre-selected filter includes rules. Example 19. The method according to any of the preceding embodiments, wherein the pre-selected filter includes regular expression matching rules. Example 20. The method according to any of the foregoing embodiments, wherein the pre-selected filter comprises a machine learning model. Example 21. The method according to Example 20, wherein the machine learning model is configured to recognize text. Example 22. The method according to Example 20, wherein the machine learning model is configured to recognize images. Example 23. The method according to any of the preceding embodiments, wherein the modified user interface is configured to interact with the human operator via one or more input peripherals of the computer. Example 24. The method according to any of the preceding embodiments, wherein the user interface is a graphical user interface. Example 25. The method according to any of the foregoing embodiments, wherein the unmodified user interface is not output to one or more output peripherals of the computer. Example 26. The method according to any of the foregoing embodiments, wherein the application is a web browser. Example 27. A computer program including instructions that, when executed by a processor, cause the processor to perform the method described in any of the preceding embodiments. Example 28. A computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform the method described in any one of Examples 1 to 26. Example 29. A processor configured to perform the method according to any one of Examples 1 to 26. Example 30. A computing system, comprising: A processor configured to perform the method described in any one of embodiments 1 to 26; A remote server communicatively coupled to the processor; and A computer communicatively coupled to the processor and the remote server, the computer having one or more output peripherals configured to output a modified user interface and one or more input peripherals for reception by a human operator, wherein the modified user interface is configured to interact with the human operator via the one or more input peripherals. Example 31. The computing system according to Example 30, wherein the computing system is a virtual network computing system, the remote server is a VNC server, and the computer is a VNC client. Example 32. The computing system according to Example 30 or 31, wherein the input peripheral device includes a keyboard and / or a mouse. Example 33. A computing system according to any one of Examples 30 to 32, wherein the output peripheral device includes a display device.

Claims

1. A method for preventing sensitive data received via a remote access protocol from being output to a human operator's computer, the method comprising: The user interface of an application running on a virtual remote server is received from a virtual remote server via a remote access protocol, wherein the virtual remote server is in the form of a virtual machine, and wherein an automated process is performed by a virtual operator of the virtual machine using the user interface of the application to execute the application running on the virtual remote server. Determine or receive from the virtual remote server a text-based representation of the application; Use one or more pre-selected filters to identify sensitive data in the application's user interface and / or text-based representation; Modify the user interface to remove the identified sensitive data; The modified user interface is output to one or more output peripherals of the computer so that it can be received by the human operator.

2. The method according to claim 1, wherein, The remote access protocol is a remote frame buffer protocol.

3. The method according to claim 2, wherein, The received user interface is in the form of an image of the user interface, the modification step includes modifying the image of the user interface, and the output step includes outputting the modified image of the user interface.

4. The method according to claim 1, wherein, The remote access protocol is the Remote Desktop Protocol.

5. The method according to claim 4, wherein, The received user interface is in the form of data used to present an image of the user interface, the modification step includes modifying the data used to present the image of the user interface, and the output step includes outputting the modified data to present an image of the modified user interface.

6. The method according to claim 1, further comprising: Prior to the steps of determining or receiving from the remote server, one or more filters are pre-selected based on the sensitive data expected in the content of the application.

7. The method according to claim 1, wherein, The step of receiving a user interface includes receiving a user interface of the desktop of the remote server, wherein the user interface of the application forms part of the user interface of the desktop.

8. The method according to claim 1, wherein, The text-based representation of the application includes text extracted from the user interface and / or the application.

9. The method according to claim 1, wherein, The text-based representation of the application includes the programmatic representation of the application.

10. The method according to claim 9, wherein, The programming representation includes the document object model of the application.

11. The method according to claim 1, wherein, The pre-selected filter includes rules.

12. The method according to claim 1, wherein, The pre-selected filters include machine learning models.

13. The method according to claim 1, wherein, The modified user interface is configured to interact with the human operator via one or more input peripherals of the computer.

14. A computer program comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-13.

15. A computing system, comprising: A processor configured to perform the method according to any one of claims 1 to 13; A remote server communicatively coupled to the processor; as well as A computer communicatively coupled to the processor and the remote server, the computer having one or more output peripherals configured to output a modified user interface and one or more input peripherals for reception by a human operator, wherein the modified user interface is configured to interact with the human operator via the one or more input peripherals.

Images