System and method for ai monitoring of remote connection sessions

Abstract

To provide a system and a method for monitoring a remote connection environment.SOLUTION: The method includes capturing a screen recording of a user accessing the first computer via the second computer, extracting visual elements from the screen recording, providing the screen recording and the extracted visual elements to a computer model trained using the training screen recording, the training extracted text, and characteristics of the operation, receiving, from the computer model, an indication of whether the screen recording and the extracted text indicate a particular operation that the method is designed to intervene in, and performing an intervention action if the computer model indicates the particular operation.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 672,297, filed July 17, 2024, which is incorporated herein by reference in its entirety.

[0002] (Technical field) FIELD OF THE DISCLOSURE The present disclosure relates generally to computer monitoring systems, and more particularly to computer monitoring via screen recording in a remote connection environment. [Background technology]

[0003] (background) Remote connections (e.g., Remote Desktop Connection, VNC, SSH, and / or other proprietary protocols) are used to remotely access computers within a computer network. Remote connections can be used for a variety of purposes, including remote system control, remote work, troubleshooting, technical support, file transfer, and general collaboration. Remote connections can be useful for monitoring computers within a remote connection environment to determine whether a user has exhibited malicious (e.g., unauthorized) behavior. User behavior can jeopardize the safety and security of the network, system, or data. However, in some environments, directly monitoring operations on a computer can be difficult due to a lack of personnel monitoring every session, direct access to the remote machine being controlled, or a lack of systems and methods capable of detecting unauthorized activity. A monitoring system's only access to activity on a remote machine may be through information transferred between intermediate machines that directly or indirectly interact with the remote machine, and that information may not include information related to operations occurring only locally on the remote machine. Summary of the Invention [Means for solving the problem]

[0004] (overview) A system and method for monitoring a remote connection environment is provided. A screen recording of a user accessing a second computer via a first computer is captured. Visual elements are extracted from the screen recording. The screen recording and extracted visual elements are provided to a computer model trained using the training screen recording, the training extracted text, and characteristics of the behavior. An indication is received from the computer model of whether the screen recording and extracted text exhibit a particular behavior for which the method is designed to intervene. If the computer model exhibits the particular behavior, an intervention action is performed.

[0005] As another example, a method for preventing classified behavior in a remote connection environment includes capturing a screen recording of a user accessing a second computer through a first computer; identifying a computer interface in the screen recording; extracting visual elements from the screen recording; determining a true and a false intent value based on the computer interface and the extracted visual elements; comparing the true intent value to a first threshold and the false intent value to a second threshold; and performing an intervention action if the true intent value exceeds the first threshold or if the false intent value is below the second threshold.

[0006] As another example, a method for training a remotely connected monitoring system includes capturing a plurality of training screen workflows; identifying a computer interface in the training screen workflows; extracting visual elements from the training screen workflows; classifying one or more of the plurality of training screen workflows as a particular operation based on the identified computer interface and the extracted visual elements; and including the classified training screen workflows in a trained remotely connected monitoring system. [Brief explanation of the drawings]

[0007] DESCRIPTION OF THE DRAWINGS [Figure 1] FIG. 1 illustrates an exemplary model (e.g., an artificial intelligence (AI) model) utilizing screen capture in a remote connection environment.

[0008] [Figure 2] FIG. 2 illustrates a further implementation of an exemplary artificial intelligence (AI) screen recording capture in a remote connection environment.

[0009] [Figure 3A] FIG. 3A illustrates an exemplary timeline of a screen recording with visual element capture in a remote connection environment.

[0010] [Figure 3B] FIG. 3B shows an exemplary screen image containing extractable visual elements.

[0011] [Figure 4] FIG. 4 illustrates an exemplary flowchart of a method for monitoring a remotely connected environment using a trained computer model.

[0012] [Figure 5] FIG. 5 illustrates a further embodiment of an exemplary flowchart of a method for monitoring a remotely connected environment.

[0013] [Figure 6] FIG. 6 illustrates an exemplary flowchart of visual element recognition in a screen image in a remote connection environment.

[0014] [Figure 7] FIG. 7 illustrates a further embodiment of an exemplary flowchart for visual element recognition in a screen image in a remote connection environment.

[0015] [Figure 8] FIG. 8 illustrates an exemplary flow chart for computer interface identification in a remote connection environment.

[0016] [Figure 9] FIG. 9 illustrates an exemplary flow chart of a method for preventing classified behavior in a remotely connected environment.

[0017] [Figure 10] FIG. 10 illustrates the suspension of a user's access to a first computer in a remote connection environment.

[0018] [Figure 11] FIG. 11 is a diagram illustrating a system administrator alert in a remote connection environment.

[0019] [Figure 12] FIG. 12 illustrates an exemplary flow chart for training a remote connection monitoring system.

[0020] [Figure 13] FIG. 13 illustrates an exemplary flow chart of a method for monitoring a remotely connected environment.

[0021] [Figure 14] FIG. 14 illustrates a further embodiment of an exemplary flowchart of a method for preventing malicious behavior in a remotely connected environment.

[0022] [Figure 15] FIG. 15 illustrates an exemplary flow chart of a method for training a remote connection monitoring system. DETAILED DESCRIPTION OF THE INVENTION

[0023] (Detailed explanation) Remote connectivity is a technology that allows a user to connect to and control a remote computer from another computer over a network connection. Remote connectivity can be used for a variety of purposes, including remote system control, remote work, troubleshooting, technical support, file transfer, and general collaboration. In some instances, a remotely connected user may utilize a "jump host" to access a computer through multiple intermediate devices on the network that have encrypted payloads. In such an environment, it may be difficult or impossible to directly monitor a user's activity on the computer (e.g., via signals or commands sent by the computer to an intermediate server or the remotely accessed computer). Furthermore, operating in such an unencrypted remote connection environment may not be feasible or advisable. When encryption is present, monitoring traffic for classified activity may be impossible, and monitoring the remote connection feed may be the only alternative. In some environments, a user may remotely connect to one authorized endpoint and then, from that endpoint, connect to other devices on the network to which they are not authorized.

[0024] In such a remote connection environment, a system administrator (e.g., an authorized user) can only access screen images displayed on the screen operated by the remotely connected user. For example, the system administrator can access a virtual desktop infrastructure (VDI) module. VDI enables the operation of multiple virtual desktops on a single physical machine. For example, a virtual desktop can be generated on a central server, and a remote user can access the virtual desktop from any machine in the network. By accessing the VDI module, the system administrator can monitor (e.g., monitor to determine malicious behavior) the screen operated by the remotely connected user. Monitoring the screen, or, for example, searching for activity, can be tedious and time-consuming for a human. Furthermore, humans may make mistakes when monitoring the screen operated by the remotely connected user.

[0025] The systems and methods disclosed herein include training a computer model (e.g., an artificial intelligence (AI) model) to monitor screens accessed by remotely connected users. For example, the computer model can be trained to detect malicious behavior based on applications, graphical elements, or text identified in a screen recording. Based on specific behavior detected in a screen recording, the computer model can initiate an intervention action, such as terminating the user's access to the computer or alerting a system administrator. Monitoring malicious behavior with a computer model can improve efficiency, provide more accurate determination of classified behavior, and provide the ability to monitor classified behavior on an increased number of computers.

[0026] FIG. 1 illustrates an exemplary model (e.g., an artificial intelligence (AI) model) that utilizes screen captures recorded in a remote connection environment. The remote connection environment 100 includes a first computer 101. The first computer 101 is accessible by a user. The remote connection environment 100 further includes a second computer group 103 that includes one or more second computers 104. The remote connection environment 100 further includes a server 102. The server 102 is connectable to the first computer 101 and the second computer group 103. The server 102 is connectable to the first computer 101 and the second computer group 103, for example, via a computer network. In the example illustrated in FIG. 1 , a user accesses and controls a second computer 104 in the second computer group 103 via the first computer 101. The user can access and control one or more of the second computers 104, for example, via the server 102. Communications between the first computer 101, the server 102, and the second computer 104 may, in some cases, utilize encrypted payloads. For example, commands sent from the first computer 101 to the second computer 104 may simply be encrypted and sent to the second computer 104 via the server 102. In such cases, the server 102, which monitors the connection between the first computer 101 and the second computer 104, may not be able to directly identify the commands from 101 to 104.

[0027] The second computer 104 can take a variety of forms. In one example, the second computer 104 is a server that provides a computing service (e.g., a web server) available for access. A user can log in to the server 104 via the first computer 101 to configure and control the operation of the server 104. In another example, the second computer 104 may be associated with the control of a physical system (e.g., an infrastructure system such as a manufacturing machine, a wastewater management system, or a dam controlling a body of water). The user 101 can interact with and control such a remote system 104 via the intermediary server 102. In some cases, the remote system 104 is a legacy system that may use outdated operating systems and software that do not include current information security and other state-of-the-art software.

[0028] In one example, a user can send commands from a first computer 101 to a server 102, as indicated by the arrow between the first computer 101 and the server 102. The commands can be directly entered (e.g., via a keyboard) into a command window on the first computer 101, or can be commands sent to the server 102 based on an indirect action of the user (e.g., opening or closing an application, interacting with a graphical element on the screen of the first computer 101, or moving the computer's mouse). The server 102 can process the commands and send them to one or more second computers 104 in the second computer group 103. In addition, the second computers 104 can send information, such as the contents of a monitor screen of the second computer 104, to the server 102. The server 102 can relay this information to the first computer 101 so that the user can observe the contents of the monitor screen that they are controlling. In some cases, commands from the first computer 101 to the second computer 104 may be encrypted or otherwise obfuscated by the server 102, while the contents of a remote capture of the screen of the second computer 104 may be identified and captured by the server 102.

[0029] The remote connection environment 100 further includes an AI model 105. The AI ​​model 105 can connect to the server 102 and can be configured to capture screen recordings 106 from the server 102 or receive captures of screens recorded by the server 102 or other computing devices. As described above, the ability to monitor a user's activity within the remote connection environment may be limited to monitoring a screen recording (e.g., the screen recording 106) of a computer the user is operating. The screen recording 106 may include, for example, multiple screen images 107 displayed on a first computer 101, captured periodically to provide a time series of screen images 107 related to the user interacting with a second computer 104. For example, the screen images 107 in the screen recording 106 may be captured from the server 102 via a virtual desktop infrastructure (VDI) module. In some exemplary implementations, the screen recording 106 may be captured from the first computer 101 or from a second computer 103, rather than or in addition to the server 102. As disclosed herein, the AI ​​model 105 is configured to prevent malicious behavior in the remote connection environment 100 (e.g., at the first computer 101) using screen recording 106.

[0030] In certain cases, remote control of a computer may be even more indirect than the example of FIG. 1. FIG. 2 illustrates a further implementation of an exemplary artificial intelligence (AI) screen recording capture in a remote connection environment. In the example illustrated in FIG. 2, the remote connection environment 100 includes a third computer group 201 including one or more third computers 202. A user can access one or more third computers 202, for example, via a server 102 and one or more second computers 104. For example, a command sent by a user from a first computer 101 to the server 102 can be relayed to one or more second computers 104 in the second computer group 103 and then to one or more third computers 202 in the third computer group 201. An AI model 105 can capture a screen recording 106 from the server 102, from the first computer 101, from one or more second computers 104, or from one or more third computers 202. The remaining components of the remotely connected environment 100 shown in Figure 2 can operate substantially similarly to those shown in Figure 1. It should be understood that in some examples, additional tiers of computers (e.g., a fourth or fifth tier of computers) may be present in the remotely connected environment 100, and screen recordings 106 can be captured by the AI ​​model 105 from a computer in any of the additional tiers of computers.

[0031] FIG. 3A illustrates an exemplary timeline of a screen recording with visual element capture in a remotely connected environment. As described above, the screen recording 106 can be captured by the AI ​​model 105 from the server 102 or another computer in the remotely connected environment 100. As shown in FIG. 3A, the screen recording 106 includes multiple screen images 107 associated with different time points in a timeline 303. Visual elements 302 can be extracted from screen images 301 associated with each time point in the timeline 303. The visual elements 302 can include text and graphical elements. Text can be present, for example, in a command window. Graphic elements can be present instead of or in addition to text. Graphic elements can include buttons or interactive features (e.g., a graphical slider used to control chemical levels in a control system). For example, the screen recording 106 shown in FIG. 3A includes a screen image 107 associated with a first time point T1 304. The screen recording 106 further includes visual elements 302 associated with a first time point T1 304 extracted from a screen image 107 associated with the first time point T1 304. The screen images 107 may be captured at N times within the screen recording session, and the visual elements 302 may be extracted from each screen image 107. The number of times N may be a predetermined number and may be based on the times a user accesses a computer (e.g., the first computer 101).

[0032] FIG. 3B illustrates an exemplary screen image including extractable visual elements. In the example shown in FIG. 3B, screen image 107 includes a command window 308 and a graphic element 310. Extractable text 309 may be present in command window 308 in the form of a command entered by a user. In the example shown in FIG. 3B, graphic element 310 is a slider. The slider can be used to control system parameters accessed by a user. Extractable text 309 and graphic element 310 may include, for example, acceptable commands and malicious commands. In addition to the extractable text 309 and graphic element 310 in command window 308, extractable text and graphic elements may also be present in other areas of screen image 107. For example, an additional command window 308 may be opened containing additional extractable text 309 and graphic element 310, or another application (e.g., an internet browser) may be opened containing additional extractable text and graphic elements. As described with reference to FIG. 3A, text may be extracted consecutively at multiple points in time.

[0033] FIG. 4 illustrates an exemplary flowchart of a method for monitoring a remotely connected environment using a trained computer model. At 401, a screen recording 106 is processed to perform visual element recognition on one or more screen images 107 within the screen recording 106. The visual element recognition 401 may include both text recognition and graphic element recognition. The visual element recognition 401 may be applied to the screen recording 106 by, for example, the AI ​​model 105 shown in FIGS. 1 and 2 . Alternatively, the visual element recognition 401 may be applied to the screen recording 106 by the AI ​​model 105, for example, before receiving it from an intermediate computer or another AI model. The visual element recognition 401 may be performed by optical character recognition (OCR) using an OCR model. The screen recording 106, each screen image 107, and text and graphic elements 405 extracted from the visual element recognition 401 are received by the trained computer model 402. The trained computer model 402 was trained using training screen recordings, training extracted text and graphical elements, and indicators of whether the behavior is malicious or not.

[0034] At 403, after the trained computer model 402 receives the extracted text and graphic elements 405, each screen image 107, and the screen recording 106, it classifies the screen recording 106 as exhibiting a particular behavior. For example, the screen recording 106 may be classified as malicious (e.g., fraudulent), normal, accidental, anomalous, or other. Malicious behavior may include unauthorized actions, such as a user accessing a prohibited application or entering a prohibited command. Actions that constitute malicious behavior may vary depending on user characteristics. For example, a system administrator may have more permissions than a lower-level operator within the remote connection environment 100. In some examples, even if an individual screen image 107 itself may not constitute malicious behavior, the individual screen images 107, when considered together in the context of other screen images 107, may constitute malicious behavior. For example, a single screen image 107 showing a user entering incorrect credentials may not constitute malicious behavior, but a group of screen images 107 showing several incorrect credentials may constitute malicious behavior. Behavior may be considered normal, for example, if the screen recording 106 indicates that the user is operating within predetermined parameters. In some examples, this determination at 403 is made by a trained computer model 402.

[0035] After classifying 403 a screen recording 106 as exhibiting a particular action, the classified screen recording 106 may be stored in a non-transitory computer-readable data store 406. The classified screen recordings 106 in the data store 406 may be searched by an operator 407 (e.g., a system administrator) in the remote connection environment 100. For example, the operator 407 may search and identify a classified screen recording 106 based on a particular action associated with the classified screen recording 106. Additionally, the operator 407 may search and identify a classified screen recording 106 based on extracted text and graphical elements 405 identified in the classified screen recording 106.

[0036] As shown in FIG. 4 , if the screen recording 106 indicates a particular category of behavior (e.g., malicious behavior), an intervention action is initiated at 404. The intervention action 404 may include suspending or terminating a user's access to a computer that the user is directly or indirectly operating in the remote connection environment 100 (e.g., by terminating the computer's connection to the network). Additionally or alternatively, the intervention action 404 may include sending an alert to a system administrator. The alert may be sent immediately or after a predetermined time (e.g., at the end of the business day or over the weekend). The alert may also be sent to a security system (e.g., a security operations center (SOC), security information and event management (SIEM), or security orchestration, automation, and response (SOAR)).

[0037] FIG. 5 illustrates a further embodiment of an exemplary flowchart of a method for monitoring a remotely connected environment. In the example shown in FIG. 5, one or more computer interfaces are identified 501 within a screen recording 106. The computer interfaces may include applications, command lines, operating system elements, etc. The identification of the computer interfaces 501 may be performed by the AI ​​model 105 or another software entity, such as a second computer model (e.g., a second AI model). For example, the AI ​​model 105 may use image recognition techniques to identify specific computer interfaces (e.g., a browser, a word processing application, a document management application, a command window, etc.) within the screen recording 106. After identifying the interfaces 501, the trained computer model 402 receives interface input 502, such as a string containing a list of identified applications. In the example shown in FIG. 5, the trained computer model 402 receives the one or more interface inputs 502, the screen recording 106, and extracted text and graphic elements 405 identified in the screen recording 106.

[0038] The trained computer model 402 can classify the screen recording 106 as exhibiting a particular behavior based on the screen recording 106, the extracted text and graphic elements 405 identified in the screen recording 106, and the interface input 502 indicative of a computer interface identified in the screen recording 106 at 501. The interface input 502, the screen recording 106, and the extracted text and graphic elements 405 alone do not constitute a behavior, but when combined, may indicate a particular behavior. For example, a user may be permitted to open a browser, so the browser detection at 501 alone would not result in a detection of malicious behavior at 403. However, the user may be prohibited from accessing a prohibited website in the browser, and the detection of the browser along with the detection of text 405 matching the prohibited website may result in malicious behavior. Similarly, a user may be permitted to operate a command window, but may be prohibited from entering certain commands in the command window.

[0039] The categorized screen recordings 106 may be stored in a non-transitory computer-readable data store 406. As discussed above with reference to FIG. 4, the categorized screen recordings 106 in the data store 406 may be searchable by an operator (e.g., a system administrator) 407 in the remote connection environment 100 based on particular behaviors associated with the screen recordings 106. Additionally, the operator 407 may search for and identify categorized screen recordings 106 based on extracted text and graphical elements 405 or computer interfaces identified therein. As discussed above, if the screen recording 106 exhibits a particular category of behavior, an intervention action may be initiated at 404.

[0040] FIG. 6 illustrates an exemplary flowchart for visual element recognition on a screen image in a remote connection environment. In the example shown in FIG. 6, text recognition (e.g., optical character recognition (OCR)) is applied to screen image 107 at 401. OCR can be applied to screen image 107 by AI model 105 or another AI model. As described above, screen image 107 can be one of multiple screen images 107 in screen recording 106. Screen image 107 includes an open command window 603 on a monitor of a computer with a Microsoft Windows operating system. The text in command window 603 includes allowable commands and malicious commands. Furthermore, the text in command window 603 indicates a particular user operating the command window. After visual element recognition is applied to screen image 107 at 401, extracted text is generated at 405. In the example shown in FIG. 6, the extracted text and graphical elements include "allowable commands" and "malicious commands," which are exemplary allowable commands and malicious commands, respectively, for illustrative purposes. As described above, the screen recordings 106 may be saved to a non-transitory computer-readable data store after applying text recognition to each screen image 107 at 401. An operator may search for a particular screen recording 106 based on the particular text identified at 401.

[0041] FIG. 7 illustrates a further embodiment of an exemplary flowchart for visual element recognition in a screen image in a remote connection environment. In the example shown in FIG. 7, the screen image 107 includes an open command window 701 for monitoring a computer with a Linux operating system. It should be understood that the systems and methods described herein can prevent specific categories of behavior in a remote connection environment using additional operating systems. Furthermore, the systems and methods described herein can detect various visual elements (e.g., commands) used in the additional operating systems. As shown in FIG. 6, visual element recognition is applied to the screen image 107 by OCR at 401. Extracted text and graphic elements are then generated at 405. The extracted text and graphic elements generated at 405 may include each text string detected in each of the multiple screen images 107 of the screen recording 106. In the example shown in FIG. 7, the extracted text and graphic elements generated at 405 include permissible commands and malicious commands.

[0042] FIG. 8 illustrates an exemplary flowchart for identifying computer interfaces in a remote connection environment. In the example illustrated in FIG. 8, computer interface recognition is performed on a screen image 107 at 501. Computer interface recognition can be performed, for example, by the AI ​​model 105 illustrated in FIGS. 1 and 2. Computer interface recognition can use various image recognition techniques. After computer interface recognition is applied to the screen image 107 at 501, an interface input 502 is generated. The interface input 502 includes a representation of each detected computer interface in the screen image 107. Furthermore, the interface input 502 may include each interface identified in each of the multiple screen images 107 in the screen recording 106. In the example illustrated in FIG. 8, the interface input 502 includes a representation of open interfaces in the screen image 107, including Internet Explorer, Settings, a Paint application, a File Explorer application, a Command Window, and WordPad. As described above, an operator can search for open interfaces and access a screen recording 106 that includes a screen image 107 with a particular interface they searched for.

[0043] 9 illustrates an exemplary flowchart of a method for preventing classified behavior in a remote connection environment. In flowchart 900 shown in FIG. 9, visual elements (e.g., text and graphic elements) are extracted 401 from a screen recording 106. As described above, visual elements may be extracted from each of the screen images 107 in the screen recording 106. A computer interface is then identified 501 for each screen image 107 in the screen recording 106. In some examples, the computer interface may be identified 501 before the visual elements are extracted 401, or the interface identification and the extraction of the visual elements may occur simultaneously. Furthermore, in some examples, only one of the extraction of visual elements 401 and the identification of the computer interface 501 may occur in the screen recording 106.

[0044] After extracting visual elements at 401 and identifying computer interfaces within a screen recording 106 at 501, true and false intent values ​​are assigned to the screen recording at 901 based on the interfaces identified in the screen recording 106 and text extracted from the screen recording 106. A true intent value may reliably indicate that an activity identified in the screen recording 106 falls within a particular category of behavior (e.g., malicious behavior). A false intent value may reliably indicate that an activity identified in the screen recording 106 does not fall within a particular category of behavior. In the example shown in FIG. 9 , a variable “X” is assigned to true intent values, and a variable “Y” is assigned to false intent values. The true and false intent values ​​may be normalized values ​​between 0 and 1, or may be values ​​on a predetermined scale, e.g., between 0 and 100.

[0045] After determining the true and false intent values ​​for the screen recording 106, the true intent values ​​are compared to a first threshold and the false intent values ​​are compared to a second threshold at 902. If the true intent values ​​are greater than the first threshold, a determination is made at 902 that the threshold confidence has been exceeded and the screen recording 106 corresponds to a particular category of activity. Additionally, if the false intent values ​​are less than the second threshold, a determination is made at 902 that the threshold confidence has not been met and the screen recording 106 does not correspond to a particular category of activity.

[0046] If the intent value classified as true exceeds a first threshold or the intent value classified as false falls below a second threshold, an intervention action is initiated at 404. As described above, the intervention action may be a suspension or termination of a user's access to a computer the user is accessing in the remote connection environment 100, or a warning message to a system administrator. In some examples, the intervention action may be initiated at 404 if the intent value classified as true exceeds a first threshold and the intent value classified as false falls below a second threshold. In the example shown in FIG. 9, if the intent value classified as true is less than or equal to the first threshold and the intent value classified as false is greater than or equal to the second threshold, the screen recording continues to be monitored at 903. A method similar to that shown in FIG. 9 may be used to prevent any category of behavior, e.g., anomalous behavior.

[0047] FIG. 10 illustrates the temporary suspension of a user's access to a first computer in a remote connection environment. In the example illustrated in FIG. 10, the remote connection environment 100 may be similar to the remote connection environment 100 illustrated in FIG. 1. The AI ​​model 105 determines that the screen recording 106 indicates a particular behavior (e.g., malicious behavior) and initiates an intervention action. As described above, this determination may be made based on text or an interface identified within the screen recording 106. After the AI ​​model 105 determines that the screen recording 106 indicates a particular behavior, it suspends the first computer 101's access to the server 102, as indicated by a red "X" 1001 in FIG. 10. This suspension may be performed by the AI ​​model 105, or by a system administrator or another computer connected to the AI ​​model 105. In some examples, the intervention action may include preventing the user from remotely accessing the computer (e.g., the second computer 104) from another computer (e.g., the first computer 101) that the user is using.

[0048] FIG. 11 illustrates a system administrator alert in a remote connection environment. As described above, an intervention action may include issuing a system administrator alert 1102 to a system administrator when a particular behavior is detected in the screen recording 106. In the example shown in FIG. 11, the system administrator alert 1102 is a window that includes information about the nature of the behavior and other data regarding the event that triggered the system administrator alert 1102. For example, the system administrator alert 1102 in FIG. 11 includes the text "WARNING: USER X IS DEMONSTRATING MALICIOUS BEHAVIOR." It should be understood that alerts with more specific or different messages can be used in the system administrator alert 1102.

[0049] System administrator alert 1102 further includes an indication of the user involved in the detected behavior (e.g., the user operating the computer from which the screen recording was taken). This user can be determined, for example, based on the credentials used to sign in to the computer or from text identified in the screen recording. Furthermore, system administrator alert 1102 includes an indication of the computer from which the screen recording was taken. System administrator alert 1102 also includes the detected action that corresponds to malicious behavior. The detected action can be determined, for example, by AI model 105. In the example shown in FIG. 11 , the detected action is an illegal command. System administrator alert 1102 further indicates a classified intent value. The classified intent value can be determined based on the true and false intent values ​​described with reference to FIG. 9 . In some examples, the true and false intent values ​​can be indicated in system administrator alert 1102.

[0050] As shown in FIG. 11 , system administrator alert 1102 includes a first button 1103, a second button 1104, and a third button 1105. First button 1103 includes an option to view the screen recording that triggered system administrator alert 1102. When a system administrator selects option 1103, another window opens allowing the system administrator to replay the screen recording. The entire screen recording can be replayed, or only the portion of the screen recording that triggered system administrator alert 1102. Second button 1104 includes an option to terminate access to the computer of the user involved in the detected behavior. This access termination can be, for example, similar to the access termination described in the discussion regarding FIG. 10 .

[0051] A system administrator can also terminate a user's access based on a screen recording 106 that the system administrator accessed based on the system administrator's own search. For example, a system administrator can search a data store (e.g., data store 406) for open interfaces (e.g., applications) and access a screen recording 106 that includes a search of the interface. After viewing the screen recording 106 or simply identifying the screen recording 106 in the search results, the system administrator can terminate a user's access that involves an open interface. A third button 1105 includes an option to dismiss the system administrator alert 1102. A system administrator can select 1103 to dismiss the system administrator alert 1102, for example, if the system administrator selects 1103 to review the screen recording and determines that the user's actions do not warrant terminating the user's access.

[0052] 12 illustrates an exemplary flowchart for training a remote connection monitoring system. In the example illustrated in FIG. 12, training screen recordings 1202 are stored in a non-transitory computer-readable data store 1201. The training screen recordings 1202 are extracted from the data store 1201. At 401, visual elements are extracted from each training screen recording 1202 (e.g., each screen image within each training screen recording 1202). As described above, OCR or other text recognition techniques can be used to extract text from the training screen recordings 1202. Computer interfaces are then identified within the training screen recordings 1202 at 501. Image recognition techniques can be used to identify interfaces in the training screen recordings 1202. The text can be extracted at 401, and the interfaces can be identified at 501, for example, by an AI model (e.g., AI model 105).

[0053] Based on the interface identified in 501 and the text extracted in 401 for each training screen recording 1202, each training screen recording 1202 is classified as one or more of a plurality of action categories. In the example shown in FIG. 12 , the plurality of action categories includes a first action category 1203 and a second action category 1204. This classification may be performed by an AI model or by a human who is able to classify actions based on the training screen recordings 1202. Based on the classification of each training screen recording 1202 as one or more of the plurality of action categories 1203, 1204, an identification (e.g., by metadata) of each specified action category may be applied to the respective training screen recording 1202.

[0054] The classified training screen recordings 1202 are then received by a trained remote connection monitoring system 1205. The trained remote connection monitoring system may be, for example, the trained computer model 402 shown in FIGS. 4 and 5. In some examples, the method shown in FIG. 12 is applied to screen recordings (e.g., screen recording 106) in real time, so that the computer model used to evaluate the screen recordings for classified behavior is continually trained with additional data. Furthermore, the trained remote connection monitoring system 1205 may be continually improved by a system administrator. For example, the system administrator may approve or correct the behavior classifications applied to the training screen recordings. This may help reduce "false positives" and "false negatives" when the trained remote connection monitoring system 1205 is applied to live screen recordings.

[0055] 13 illustrates an exemplary flowchart of a method for monitoring a remote connection environment. Method 1300 includes a first step 1301 of capturing a screen recording of a user accessing a first computer through a second computer. At 1302, visual elements are extracted from the screen recording. At 1303, the screen recording and extracted visual elements are provided to a computer model trained using the training screen recording, the training extracted text and graphic elements, and the characteristics of the behavior. At 1304, an indication of whether the screen recording and extracted visual elements exhibit a particular behavior is received from the computer model. At 1305, an intervention action is performed if the computer model exhibits the particular behavior.

[0056] 14 illustrates a further embodiment of an exemplary flowchart of a method for preventing malicious behavior in a remote connection environment. Method 1400 includes step 1401 capturing a screen recording of a user accessing a first computer via a second computer. At 1402, a computer interface is identified in the screen recording. At 1403, visual elements are extracted from the screen recording. At 1404, true and false intent values ​​are determined based on the computer interface and the extracted visual elements. At 1405, the true intent values ​​are compared to a first threshold, and the false intent values ​​are compared to a second threshold. At 1406, an intervention action is performed if the true intent value is above the first threshold or if the false intent value is below the second threshold.

[0057] 15 illustrates an example flowchart of a method for training a remote connection monitoring system. Method 1500 includes a first step 1501 of capturing a plurality of training screen workflows. At 1502, computer interfaces are identified in the training screen workflows. At 1503, visual elements are extracted from the training screen workflows. At 1504, one or more of the plurality of training screen workflows are classified as specific operations based on the identified application and the extracted visual elements. At 1505, the classified training screen workflows are included in a trained remote connection monitoring system.

[0058] The methods and systems described herein can be performed by various types of processing devices by program code that includes program instructions executable by a device processing subsystem. The software program instructions can include source code, object code, machine code, or other stored data that can cause a processing system to perform the methods and operations described herein, and can be provided in any suitable language, such as, for example, C, C++, JAVA, or other suitable programming languages. However, other implementations can be used, such as firmware or even suitably designed hardware configured to perform the methods and systems described herein.

[0059] System data and method data (e.g., associations, mappings, data inputs, data outputs, intermediate data results, final data results, etc.) can be stored and implemented in one or more different types of computer-implemented data stores, e.g., different types of storage devices and programming structures (e.g., RAM, ROM, flash memory, flat files, databases, programming data structures, programming variables, IF-THEN (or similar type) statement structures, etc.). Note that data structures represent formats for use in organizing and storing data in databases, programs, memories, or other computer-readable media used in computer programs.

[0060] The computer components, software modules, functions, data stores, and data structures described herein can be directly or indirectly connected to one another to enable the flow of data necessary for operation. It should also be noted that a module or processor includes, but is not limited to, code units that perform software operations, and can be implemented, for example, as code units of subroutines, or as code units of software functions, or as objects (as in an object-oriented paradigm), or as applets, or as computer scripting languages, or as other types of computer code. Software components and / or functions may be located on a single computer or distributed across multiple computers, depending on the situation at hand.

[0061] Although the present disclosure has been described in detail with reference to specific embodiments thereof, it will be apparent to those skilled in the art that various changes and modifications can be made thereto without departing from the spirit and scope of the embodiments. Accordingly, the present disclosure is intended to cover the modifications and variations of the present disclosure provided they come within the scope of the appended claims and their equivalents.

Claims

1. 1. A method for monitoring a remote connection environment, comprising: capturing a screen recording of a user accessing a second computer through a first computer; extracting visual elements from said screen recording; providing the screen recording and the extracted visual elements to a computer model trained using the training screen recording, the training extracted text, and the behavior characteristics; receiving an indication from the computer model whether the screen recording and the extracted visual elements exhibit a particular behavior that the method is designed to intervene in; and performing an intervention action if the computer model indicates the particular behavior.

2. 10. The method of claim 1, further comprising identifying a computer interface in the screen recording, and wherein the indication is based on one or both of the computer interface and the extracted visual elements.

3. 3. The method of claim 2, further comprising: saving the screen recording to a non-transitory computer-readable data store, the non-transitory computer-readable data store being searchable based on the extracted visual elements or the identified computer interface in the screen recording.

4. The method of claim 1 , wherein the screen recording comprises a plurality of screen images.

5. The method of claim 1 , wherein the screen recording is captured at a server or endpoint within the remote connection environment.

6. The method of claim 1 , wherein the visual element comprises text.

7. The method of claim 1 , wherein the visual element comprises a graphic element.

8. The method of claim 1 , wherein the intervention action comprises terminating the user's access to the second computer.

9. The method of claim 1 , wherein the intervention action includes sending an alert to a system administrator.

10. 1. A method for preventing classified actions in a remote connection environment, comprising: capturing a screen recording of a user accessing a second computer through a first computer; identifying a computer interface in said screen recording; extracting visual elements from said screen recording; determining a true-classified intent value and a false-classified intent value based on the computer interface and the extracted visual elements; comparing the true-classified intent value to a first threshold and comparing the false-classified intent value to a second threshold; and performing an intervention action if the true-classified intent value is above the first threshold or if the false-classified intent value is below the second threshold.

11. 11. The method of claim 10, wherein the intervention action comprises terminating the user's access to the second computer.

12. The method of claim 10 , wherein the intervention action includes sending an alert to a system administrator.

13. 11. The method of claim 10, wherein the alert is sent a predetermined time after the intent value classified as true exceeds the first threshold or the intent value classified as false falls below the second threshold.

14. 11. The method of claim 10, wherein determining the true-classified intent value and the false-classified intent value comprises comparing the screen recording to a plurality of classified screen recordings.

15. 1. A method of training a remote connected monitoring system, comprising: Capturing screen workflows for multiple training; identifying a computer interface in the training screen workflow; extracting visual elements from the training screen workflow; classifying one or more of the plurality of training screen workflows as a particular operation based on the identified computer interface and the extracted visual elements; and The method further includes including the classified training screen workflow in a trained remote connected monitoring system.

16. 16. The method of claim 15, wherein each of the plurality of training screen workflows includes a plurality of training screen images.

17. 16. The method of claim 15, further comprising approving classification of one or more of the plurality of training screen workflows as the specific action, wherein the approval is performed by a system administrator.

18. 16. The method of claim 15, wherein the particular behavior is malicious or fraudulent behavior.

19. 20. The method of claim 18, wherein the visual element comprises text.

20. The method of claim 15 , wherein the visual element comprises a graphic element.

Images