Artificial-intelligence-powered remote connections auditing

Abstract

A method includes determining a start of a remote control session between a technician device and a client device. The method includes, in response to the start of the remote control session, starting a recording session. The recording session includes obtaining a set of actions performed under remote control via the technician device on the client device. The method includes determining whether one or more criteria have been met. The method includes, in response to a determination that one or more criteria have been met, storing the set of actions, analyzing the set of actions, supplying the set of actions to a machine-learning model, and generating a summary of the set of actions based on an output of the machine-learning model.

Description

ARTIFICIAL-INTELLIGENCE-POWERED REMOTE CONNECTIONS AUDITINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 670,108 filed July 11, 2024. The entire disclosure of the above application is incorporated by reference.FIELD

[0002] The present disclosure relates to the distributed computing devices and more particularly to the creation of logs for remote control sessions.BACKGROUND

[0003] In an environment filled with computers, a user of a computer may encounter various incidents with the user's computer. For example, the user may be unable to save a file to a network hard drive, unable to print a document, unable to launch a particular application, etc. The user may contact an information technology (IT) technician, who may have the knowledge and resources to resolve the incident. However, the technician may be unavailable to provide immediate assistance to the user, such as when the technician is assisting other users, the incident arises outside of normal business hours, etc.

[0004] Once the technician becomes available to provide assistance, the technician has to request and be granted permission from the user before the technician can remotely connect to the user's computer. Typically, technicians attempt to fix problems or perform troubleshooting during these remote sessions. However, these actions are often not recorded, requiring technicians to manually log their activities in tickets for auditing purposes - a task that is frequently overlooked. This omission creates a significant blind spot regarding the actions taken to resolve issues. From a security perspective, IT managers lack visibility into what technicians are doing on remote devices due to the absence of tracking or auditing of actions performed during remote connections.

[0005] The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.SUMMARY

[0006] A method includes determining a start of a remote control session between a technician device and a client device. The method includes, in response to the start of the remote control session, starting a recording session. The recording session includes obtaining a set of actions performed under remote control via the technician device on the client device. The methodincludes determining whether one or more criteria of a set of analysis criteria have been met. The method includes, in response to a determination that one or more criteria of the set of analysis criteria have been met, storing the set of actions, analyzing the set of actions, supplying the set of actions to a machine -learning model, and generating a summary of the set of actions based on an output of the machine-learning model.

[0007] In other features, the start of the remote control session is determined in response to detecting a user input. In other features, the recording session includes a set of recording segments. In other features, the set of recording segments includes a first recording segment and a second recording segment that are a first duration and separated by a second duration. In other features, the set of analysis criteria includes a first criterion that is met when an end of the remote control session is detected and a second criterion that is met when a respective recording segment of the set of recording segments is saved.

[0008] In other features, generating the summary of the set of actions includes, for a respective action of the set of actions, interpreting a set of communications from at least one of a first user of the technician device or a second user of the client device. The set of communications is associated with the respective action. In other features, generating the summary of the set of actions includes, for a respective action of the set of actions, at least one of interpreting a context of the respective action, including at least one of interpreting a user interface associated with the respective action, interpreting a user selection associated with the respective action, and interpreting a user interface element of the user interface associated with the user selection. In other features, generating the summary of the set of actions includes, for a respective action of the set of actions, generating a set of text describing the respective action.

[0009] In other features, generating the summary of the set of actions includes summarizing the set of text for each respective action. In other features, the set of actions includes at least one of a mouse event on a mouse input device, a touch event on a touch-sensitive input device, a keyboard press on a keyboard input device, a voice command on a sound-sensitive input device, and a gesture input on a gesture-sensitive input device.

[0010] In other features, the method includes receiving a service ticket corresponding to an undesired system state of the client device. In other features, the method includes assigning the service ticket to a technician. In other features, the method includes initiating the remote control session between the technician device and the client device. The start of the remote control session is determined in response to initiation of the remote control session.

[0011] In other features, each action of the set of actions is stored in a first data structure that conforms to a first data structure definition. In other features, obtaining the set of actions includes receiving the set of actions from at least one of the client device or the technician device upon completion of the remote control session. In other features, obtaining the set of actions includes recording the set of actions executed by the client device as instructed by the technician device.

[0012] A non-transitory computer-readable medium stores processor-executable instructions. The instructions include determining a start of a remote control session between a techniciandevice and a client device. The instructions include, in response to the start of the remote control session, starting a recording session. The recording session includes obtaining a set of actions performed under remote control via the technician device on the client device. The instructions include determining whether one or more criteria of a set of analysis criteria have been met. The instructions include, in response to a determination that one or more criteria of the set of analysis criteria have been met, storing the set of actions, analyzing the set of actions, supplying the set of actions to a machine-learning model, and generating a summary of the set of actions based on an output of the machine-learning model.

[0013] In other features, the recording session includes a set of recording segments. In other features, the set of recording segments includes a first recording segment and a second recording segment that are a first duration and separated by a second duration.

[0014] In other features, the set of analysis criteria includes a first criterion that is met when an end of the remote control session is detected. In other features, the set of analysis criteria includes a second criterion that is met when a respective recording segment of the set of recording segments is saved.

[0015] In other features, generating the summary of the set of actions includes, for a respective action of the set of actions, interpreting a set of communications from at least one of a first user of the technician device or a second user of the client device. The set of communications is associated with the respective action. In other features, generating the summary of the set of actions includes, for a respective action of the set of actions, interpreting a context of the respective action, including at least one of interpreting a user interface associated with the respective action, interpreting a user selection associated with the respective action, and interpreting a user interface element of the user interface associated with the user selection. In other features, generating the summary of the set of actions includes, for a respective action of the set of actions, generating a set of text describing the respective action. In other features, generating the summary of the set of actions includes summarizing the set of text for each respective action.

[0016] A system includes memory hardware configured to store instructions and processor hardware configured to execute the instructions stored by the memory hardware. The instructions include determining a start of a remote control session between a technician device and a client device. The instructions include, in response to the start of the remote control session, starting a recording session. The recording session includes obtaining a set of actions performed under remote control via the technician device on the client device. The instructions include determining whether one or more criteria of a set of analysis criteria have been met. The instructions include, in response to a determination that one or more criteria of the set of analysis criteria have been met storing the set of actions, analyzing the set of actions, supplying the set of actions to a machine-learning model, and generating a summary of the set of actions based on an output of the machine-learning model.

[0017] In other features, the recording session includes a set of recording segments. In other features, the set of recording segments includes a first recording segment and a second recordingsegment that are a first duration and separated by a second duration. In other features, the set of analysis criteria includes a first criterion that is met when an end of the remote control session is detected and a second criterion that is met when a respective recording segment of the set of recording segments is saved.

[0018] In other features, generating the summary of the set of actions includes, for a respective action of the set of actions, interpreting a set of communications from at least one of a first user of the technician device or a second user of the client device. The set of communications is associated with the respective action. In other features, generating the summary of the set of actions includes, for a respective action of the set of actions interpreting a context of the respective action, including at least one of interpreting a user interface associated with the respective action, interpreting a user selection associated with the respective action, and interpreting a user interface element of the user interface associated with the user selection. In other features, generating the summary of the set of actions includes, for a respective action of the set of actions generating a set of text describing the respective action. In other features, generating the summary of the set of actions includes summarizing the set of text for each respective action.

[0019] Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims, and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present disclosure will become more fully understood from the detailed description and the accompanying drawings.

[0021] FIG. 1 is a functional block diagram of an example remote control system that creates and analyzes logs of a remote control session.

[0022] FIG. 2 is a flowchart of an example method for creating and analyzing logs of a remote control session.

[0023] In the drawings, reference numbers may be reused to identify similar and / or identical elements.DETAILED DESCRIPTIONINTRODUCTION

[0024] According to the present disclosure, a distributed computing system can record a set of actions and accompanying screenshots of a client device during a remote control session. Each action of the set of actions describes one or more user interface (UI) interactions with the client device, such as a mouse event and / or a keyboard event. The recording begins in response to the initialization of a remote connection between a technician device and the client device. In various embodiments, the recording begins in response to detecting a manual input. In variousembodiments, the remote connection is coordinated by the distributed computing system and occurs in response to receiving a service ticket from the client device. In various embodiments, the service ticket describes an undesired state of the client device. In various embodiments, the recording begins automatically before the remote connection is established, after the remote connection has been established, or after a set time interval. In various embodiments, the recording is initialized by the technician. In various embodiments, the recording captures the entire remote control session. In various embodiments, the recording runs in recurring intervals (for example, 10 seconds of recording for every minute of remote connection, or 1 minute of recording for every 5 minutes of remote connection). In various embodiments, the recording is performed by the remote device (such as the client device or a server device) to which the technician device is connected and continues to run until the remote connection ends. In various embodiments, the recording is performed by the technician device. In various embodiments, the recording may be stored on the client device during the remote control session.

[0025] The recording tracks events such as keyboard input and mouse clicks and movements. In various embodiments, other forms of user input are tracked such as audio, track pad input, gestures, or touch inputs. The events are translated from the raw user input into a step object, which includes relevant data along with minimal identification information. In various embodiments, the step object is a composed object (in other words, assembled from various data objects) of the raw input data and includes data properties such as a step identifier, a step number, a time stamp, a session identifier, and the captured event (such as left mouse input, right mouse input, keystrokes, etc.), and various other properties. In various embodiments, the step identifier includes a string convention with indications of the process, class, and name of the step. In various embodiments, the step identifier is used in a knowledge graph to identify steps and context without using a machine learning model (such as a large language model (LLM)). For example, various processes, classes, and names may commonly occur, and the various graph edges indicate the context of the event.

[0026] In various embodiments, additional analysis is required to determine the type of detected input. For example, a mouse event hook may only indicate the position of the mouse at the time of the event. Additional analysis of the system data must be performed to determine the context of the input. For example, the system analyzes the current window, active process, subsequent windows and processes to determine whether the mouse event was a left or right button click and what occurred in response to the mouse event.

[0027] In various embodiments, the step object includes additional metadata. In various embodiments, the step object includes an event properties object which is a composed object that includes the preprocessed raw input data and readable properties. In various embodiments, the raw input data is translated into readable properties such as a title (for example, the title of the active UI window), a process name (for example, the active process name), and others. In various embodiments, the system preprocesses the raw input data before translating the data and adding the data to the step object.

[0028] In various embodiments, the raw data contains the child data of components that indicate which user-interface element has been selected. Preprocessing the child data includes fetching data associated with the parent data. Collecting the chain of parent and child data provides an accurate indication of the user-interface element. For example, if a child component is a settings element, “settings” alone does not describe the type of settings element. The parent component indicates that the child component is a “network setting.”

[0029] In various embodiments, preprocessing the raw data includes removing unnecessary properties to reduce data size. In various embodiments, unnecessary properties include deprecated legacy data that is included in the event hook or parent data. For example, the parent and child data that is used to determine the context of user inputs and user interfaces may increase the size of the step object. Therefore, removing the parent and child objects reduces overall step object size.

[0030] Analysis begins once an event is created. First, the stored steps are fetched. Next, using machine-learning models and specific prompts, the raw unreadable step data is converted into a valuable analysis and summary of the remote connection. This includes detailing the actions taken in each step, summarizing the overall remote connection, and identifying the issues the technician attempted to resolve. In various embodiments, the prompt is directed to ignore inconsequential steps (in other words, steps that do not result in a change). For example, a session may include steps that indicate a technician opening a file explorer program and then immediately closing the program without performing a task that changes the system (for example, by opening a program by mistake or to check a setting). In various embodiments, the prompt directs the LLM to exclude inconsequential steps from the session summary. In various embodiments, the prompt includes directions to only include the technical problem and solution in the session summary.BLOCK DIAGRAM

[0031] FIG. 1 is a functional block diagram of an example remote control system that creates and analyzes logs of a remote control session. The remote control system includes client device 104, technician device 106, and control server 102. Client device 104 includes web browser 108 and remote support agent 110. In various embodiments, web browser 108 includes an application (or program) with a network connection to control server 102. Web browser 108 accesses web portal 132 to create and transmit support tickets (for example, reports of errors on client device 104 and / or requests for technician support). Remote support agent 110 includes recording module 116 and client control module 112. Recording module 116 records actions executed via technician device 106 on client device 104. For example, recording module 116 records the actions of a technician while a remote connection is active. In various embodiments, recording module 116 records user interface (UI) interactions such as mouse inputs (mouse-up, mouse-down, mouse-left, mouse-right, a mouse-click, a double-click, scroll-wheel, etc.), keyboard press inputs, touch gestures from a touch-sensitive input device, gestures from a gesture sensitive input device, and / or voice input from a microphone input or other sound-sensitive input device. In various embodiments, recording module 116 records user inputs and the context of the user inputs (such as a displayed UI while an input is detected or where an input is located with respect to the UI). In various embodiments, recording module 116 records text, audio, or video communications between technician device 106 and client device 104. In various embodiments, recording module 116 records events by detecting event hooks generated by an operating system of client device 104 and / or technician device 106. Detected event hooks allow the example remote control system to retrieve additional event information. For example, a mouse event hook indicates the object of the mouse click event, and by using the position of the mouse at the time of the event, the remote control system retrieves information about the active window, active process, and raw information about the various components. In various embodiments, a session includes hundreds of steps (for example, each mouse input or keystroke is a step).

[0032] In various embodiments, recording module 116 processes each event and translates the data into a step that is transmitted to a cloud-based data storage (such as ticket database 140 or other storage area) and stored under the steps session identifier and account identifier. The end of a remote session triggers analysis module 124, which retrieves the recorded steps and prepares the steps for machine learning model (such as an LLM) analysis.

[0033] Client control module 112 manages the remote control sessions on client device 104. In various embodiments, technician device 106 connects directly to client control module 112 via remote control software 120. In various embodiments, the remote connection may pass through remote coordination module 128 and / or control server 102. On technician device 106, web browser 118 (or a similar program with connection to control server 102) reviews tickets transmitted by a user of client device 104.

[0034] Control server 102 includes web portal 132 which a user of client device 104 uses to submit tickets via web browser 108. In various embodiments, web browser 108 is an application, API or other program that connects directly to ticket management module 136. Ticket management module 136 stores tickets in ticket database 140 and coordinates tickets and the assignment to a technician. Web browser 118 displays information associated with the user created ticket. Technician device 106 coordinates a remote control session with client device 104 via remote coordination module 128 (including a handshake protocol, timing, transmitting the user interface of client device 104 to technician device 106, and / or transmitting user inputs from technician device 106 to client device 104).

[0035] In response to the remote connection (for example, before the connection begins, after the connection begins, simultaneously with the remote connection), recording module 116 starts a recording session and records user interaction on client device 104 and inputs executed on client device 104 (both locally on client device 104 and remotely by technician device 106). In response to the remote connection terminating, the recording is analyzed by analysis module 124. In various embodiments, analysis module 124 includes a large language model (LLM). Analysis module 124 translates the raw input data and user interface data into human- readable data. In various embodiments, analysis module 124 runs on control server 102. Invarious embodiments, analysis module 124 is a part of remote support agent 110 and runs on client device 104.FLOWCHART

[0036] FIG. 2 is a flowchart of an example method for creating and analyzing logs of a remote control session. Control begins at 204 and determines whether a user ticket has been opened. If no ticket has been opened, control remains at 204. If a ticket has been opened, control transfers to 208. At 208, control assigns the ticket to a technician. At 212, control determines whether a remote connection has been established (for example between client device 104 and technician device 106). If no remote connection has been established, control remains at 212. If a remote connection has been established, control transfers to 216 and begins a recording session. In various embodiments, steps 204 and 208. For example, a technician starts a remote connection without a user ticket being received or in response to a direct communication from a user of a client device. In various embodiments, step 216 occurs in response to user input (for example from the client, technician, and / or a third party), after a set time threshold after a remote connection begins, or in intervals (such as a 30 second recording session every 2 minutes). At 220, control determines if an action has been detected. If no action has been detected, control transfers to 236. If an action has been detected, control transfers to 224.

[0037] At 224, control records the action and adds it a set of actions. At 228, the action is translated into a step object format. At 232, the action (including both raw and translated data) is stored and control returns to 220. At 236, control determines whether the remote connection has been terminated. If the remote connection has not been terminated, control returns to 220. If the remote connection has been determined, control transfers to 240 and control ends the recording session. At 244, control determines if there are unanalyzed actions in the set of actions. If there are no unanalyzed actions remaining, control transfers to 256 and control generates a summary of the actions before ending. In various embodiments, instead of ending, control returns to 204. If there are unanalyzed actions remaining, control transfers to 248. At 248, an unanalyzed action is selected. At 252, the selected action is interpreted (for example, a string of text describing the action is generated) and control transfers to 244.SESSION SUMMARY EXAMPLE

[0038] In various embodiments, a session summary includes a plain-text summary of the tasks performed by the technician. For example, the summary may include: “during the remote session, the IT technician performed various tasks primarily focused on managing browser settings and system configurations. The technician accessed Google Chrome and Microsoft Edge to troubleshoot privacy errors and adjust settings related to extensions and security. The technician navigated through the Windows Settings application to modify network configurations, including DNS settings and VPN options. The session also involved checking the Task Scheduler and exploring system settings related to Wi-Fi and network properties.Throughout the session, the technician frequently interacted with the desktop environment, ensuring that the necessary adjustments were made to resolve the user’s issues effectively.”STEP OBJECT EXAMPLE

[0039] As shown in the example below, a step object can include various properties and metadata related to an event. In various embodiments, the step identifier includes a string convention that includes indications of a process, class, and name of the step. In various embodiments, the step identifier is used in a knowledge graph to identify steps without using an LLM. In various embodiments, the step object includes various types of metadata including time stamps, a step number (in other words, the chronological position of the step in relation to other steps in the session), event properties (such as data from the event hook that triggered the creation of the step) including process titles, a screenshot, event type, input type, etc.

[0040] As an example only of potential fields and placeholder data, a step object could be defined as follows:{ "Timestamp": "2025-04-11T16 : 38 : 57.5162293Z", "StepNumber " : 105, "Stepld": " { SOME_STEP_ID } " , "Sessionld" : " { SOME_SESSION_ID} ", "Accountld" : " { SOME_ACCOUNT_ID} ", "Eventproperties": { "Identifier" : " [p] chrome [ c ] #32769 [n] Desktop l>[c] Chrome WidgetWin 1 [n] Settings - Google Chrome", "Previousldentif ier " : " [p] Idle [c] #32769 [n] Desktop 1", "Screenshot": null, "Title": "Settings - Google Chrome", "ProcessName": "chrome", "EventType": 2, "AutomationElementlnf ormation" : [ { "ControlType " : "Pane", "LocalizedControlType " : null, "Name": "Desktop 1", "AcceleratorKey" : null, "AccessKey": null, "HasKeyboardFocus " : false, "IsKeyboardFocusable" : false, "IsEnabled": false, "HelpText": null, "IsControlElement" : false,"IsContentElement": false,"Automationld": null,"ItemType": null,"IsPassword": false,"ClassName": "#32769","NativeWindowHandle": 0,"Processld": 0,"IsOff screen" : false,"Frameworkld": "Win32","IsRequiredForForm": false,"ItemStatus": null},{"ControlType " : "Window","LocalizedControlType " : null,"Name": "Settings - Google Chrome","AcceleratorKey" : null,"AccessKey": null,"HasKeyboardFocus " : false,"IsKeyboardFocusable" : false,"IsEnabled": false,"HelpText": null,"IsControlElement" : false,"IsContentElement": false,"Automationld": null,"ItemType": null,"IsPassword": false,"ClassName": "Chrome WidgetWin 1","NativeWindowHandle": 0,"Processld": 0,"IsOff screen" : false,"Frameworkld": "Win32","IsRequiredForForm": false,"ItemStatus": null}] ,"MouseEventProperties " : {"MousePositionX" : 1540,"MousePositionY" : 337}} ,"Environment" : "Production"}CONCLUSION

[0041] The foregoing description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure can be implemented in a variety of forms. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, the specification, and the following claims. In the written description and claims, one or more steps within a method may be executed in a different order (or concurrently) without altering the principles of the present disclosure. Similarly, one or more instructions stored in a non-transitory computer-readable medium may be executed in a different order (or concurrently) without altering the principles of the present disclosure. Unless indicated otherwise, numbering or other labeling of instructions or method steps is done for convenient reference, not to indicate a fixed order. Numerical terms, such as “first,” “second,” and “third,” may be used in the disclosure and claims as unique labels: they are not used to imply a sequence or order unless the context clear indicates otherwise. In other words, a “second element” could be relabeled as a “first element” without departing from the principles of the present disclosure. Further, the presence of a “second element” does not imply or require the presence of a “first element.”

[0042] Unless the context clearly indicates otherwise, the singular articles “a,” “an,” and “the” before a noun do not restrict the noun to a single instance. The verbs “comprise,” “include,” and “have” are inclusive and therefore specify the presence of elements without excluding the presence of one or more additional elements.

[0043] Further, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any embodiment of the disclosure can be implemented in and / or combined with features of any of the other embodiments, even if that combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with one another remain within the scope of this disclosure.

[0044] Spatial and functional relationships between elements (for example, between modules) are described using various terms, including “connected,” “coupled,” and “engaged.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship encompasses a direct relationship where no other intervening elements are present between the first and second elements as well as an indirect relationship where one or more intervening elements are present between the first and second elements.

[0045] The term “set” generally means a grouping of one or more elements. The elements of a set do not necessarily need to have any characteristics in common or otherwise belong together.However, in various implementations a “set” may, in certain circumstances, be the empty set (in other words, the set has zero elements in those circumstances). As an example, a set of search results resulting from a query may, depending on the query, be the empty set. In contexts where it is not otherwise clear, the term “non-empty set” can be used to explicitly denote exclusion of the empty set — that is, a non-empty set will always have one or more elements.

[0046] A “subset” of a first set generally includes some of the elements of the first set. In various implementations, a subset of the first set is not necessarily a proper subset: in certain circumstances, the subset may be coextensive with (equal to) the first set (in other words, the subset may include the same elements as the first set). In contexts where it is not otherwise clear, the term “proper subset” can be used to explicitly denote that a subset of the first set must exclude at least one of the elements of the first set. Further, in various implementations, the term “subset” does not necessarily exclude the empty set. As an example, consider a set of candidates that was selected based on first criteria and a subset of the set of candidates that was selected based on second criteria; if no elements of the set of candidates met the second criteria, the subset may be the empty set. In contexts where it is not otherwise clear, the term “non-empty subset” can be used to explicitly denote exclusion of the empty set.

[0047] The phrase “at least one of A, B, and C” should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.” The phrase “at least one of A, B, or C” should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR. The phrase "A, B, and / or C" should be construed in the same way as the phrase “at least one of A, B, and C.”

[0048] In the figures, the direction of an arrow, as indicated by the arrowhead, generally demonstrates the flow of information (such as data or instructions) that is of interest to the illustration. For example, when element A and element B exchange a variety of information but information transmitted from element A to element B is relevant to the illustration, the arrow may point from element A to element B. This unidirectional arrow does not imply that no other information is transmitted from element B to element A. Further, for information sent from element A to element B, element B may send requests for, or receipt acknowledgments of, the information to element A.

[0049] In this application, including the definitions below, the term “module” can be replaced with the term “controller” or the term “circuit.” In this application, the term “controller” can be replaced with the term “module.” The term “module” may refer to, be part of, or include processor hardware (shared, dedicated, or group) that executes code coupled with memory hardware (shared, dedicated, or group) that stores code executed by the processor hardware.

[0050] The module may include one or more interface circuits. In some examples, the interface circuit(s) may implement wired or wireless interfaces that connect to a local area network (LAN) or a wireless personal area network (WPAN). Examples of a LAN are Institute of Electrical and Electronics Engineers (IEEE) Standard 802.11-2020 (also known as the WIFI wireless networking standard) and IEEE Standard 802.3-2018 (also known as the ETHERNET wired networking standard). Examples of a WPAN are IEEE Standard 802.15.4 (including theZIGBEE standard from the ZigBee Alliance) and, from the Bluetooth Special Interest Group (SIG), the BLUETOOTH wireless networking standard (including Core Specification versions 3.0, 4.0, 4.1, 4.2, 5.0, and 5.1 from the Bluetooth SIG).

[0051] The module may communicate with other modules using the interface circuit(s). Although the module may be depicted in the present disclosure as logically communicating directly with other modules, in various implementations the module may actually communicate via a communications system. The communications system includes physical and / or virtual networking equipment such as hubs, switches, routers, and gateways. In some implementations, the communications system connects to or traverses a wide area network (WAN) such as the Internet. For example, the communications system may include multiple LANs connected to each other over the Internet or point-to-point leased lines using technologies including Multiprotocol Label Switching (MPLS) and virtual private networks (VPNs).

[0052] In various implementations, the functionality of the module may be distributed among multiple modules that are connected via the communications system. For example, multiple modules may implement the same functionality distributed by a load balancing system. In a further example, the functionality of the module may be split between a server (also known as remote, or cloud) module and a client (or, user) module. For example, the client module may include a native or web application executing on a client device and in network communication with the server module.

[0053] The term code, as used above, may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. Shared processor hardware encompasses a single microprocessor that executes some or all code from multiple modules. Group processor hardware encompasses a microprocessor that, in combination with additional microprocessors, executes some or all code from one or more modules. References to multiple microprocessors encompass multiple microprocessors on discrete dies, multiple microprocessors on a single die, multiple cores of a single microprocessor, multiple threads of a single microprocessor, or a combination of the above.

[0054] The memory hardware may also store data together with or separate from the code. Shared memory hardware encompasses a single memory device that stores some or all code from multiple modules. One example of shared memory hardware may be level 1 cache on or near a microprocessor die, which may store code from multiple modules. Another example of shared memory hardware may be persistent storage, such as a solid state drive (SSD) or magnetic hard disk drive (HDD), which may store code from multiple modules. Group memory hardware encompasses a memory device that, in combination with other memory devices, stores some or all code from one or more modules. One example of group memory hardware is a storage area network (SAN), which may store code of a particular module across multiple physical devices. Another example of group memory hardware is random access memory of each of a set of servers that, in combination, store code of a particular module. The term memory hardware is a subset of the term computer-readable medium.

[0055] The apparatuses and methods described in this application may be partially or fully implemented by a special-purpose computer created by configuring a general-purpose computer to execute one or more particular functions embodied in computer programs. Such apparatuses and methods may be described as computerized or computer-implemented apparatuses and methods. The functional blocks and flowchart elements described above serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.

[0056] The computer programs include processor-executable instructions that are stored on at least one non-transitory computer-readable medium. The computer programs may also include or rely on stored data. The computer programs may encompass a basic input / output system (BIOS) that interacts with hardware of the special-purpose computer, device drivers that interact with particular devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0057] The computer programs may include: (i) descriptive text to be parsed, such as HTML (hypertext markup language), XML (extensible markup language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. As examples only, source code may be written using syntax from languages including C, C++, C#, Objective C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language 5th revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.The term non-transitory computer-readable medium does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave). Non-limiting examples of a non-transitory computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask readonly memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).

[0058] Various example embodiments of the invention are described in the following clauses.

[0059] Clause 1: A method comprising: determining a start of a remote control session between a technician device and a client device; in response to the start of the remote control session, starting a recording session, wherein the recording session includes obtaining a set of actions performed under remote control via the technician device on the client device; determining whether one or more criteria of a set of analysis criteria have been met; andin response to a determination that one or more criteria of the set of analysis criteria have been met: storing the set of actions; analyzing the set of actions; supplying the set of actions to a machine-learning model; and generating a summary of the set of actions based on an output of the machinelearning model.

[0060] Clause 2: The method of clause 1 wherein the start of the remote control session is determined in response to detecting a user input.

[0061] Clause 3: The method of any one of clauses 1-2 wherein: the recording session includes a set of recording segments, and the set of recording segments includes a first recording segment and a second recording segment that are a first duration and separated by a second duration.

[0062] Clause 4: The method of clause 3 wherein the set of analysis criteria includes: a first criterion that is met when an end of the remote control session is detected, and a second criterion that is met when a respective recording segment of the set of recording segments is saved.

[0063] Clause 5: The method of any one of clauses 1-4 wherein generating the summary of the set of actions includes, for a respective action of the set of actions, at least one of: interpreting a set of communications from a first user of the technician device and / or a second user of the client device, wherein the set of communications is associated with the respective action; interpreting a context of the respective action, including at least one of: interpreting a user interface associated with the respective action, interpreting a user selection associated with the respective action, and interpreting a user interface element of the user interface associated with the user selection; or generating a set of text describing the respective action.

[0064] Clause 6: The method of clause 5 wherein generating the summary of the set of actions includes summarizing the set of text for each respective action.

[0065] Clause 7: The method of any one of clauses 1-6 wherein the set of actions includes at least one of: a mouse event on a mouse input device, a touch event on a touch-sensitive input device, a keyboard press on a keyboard input device, a voice command on a sound-sensitive input device, and a gesture input on a gesture-sensitive input device.

[0066] Clause 8: The method of any one of clauses 1-7 further comprising: receiving a service ticket corresponding to an undesired system state of the client device; assigning the service ticket to a technician; andinitiating the remote control session between the technician device and the client device, wherein the start of the remote control session is determined in response to initiation of the remote control session.

[0067] Clause 9: The method of any one of clauses 1-8 wherein each action of the set of actions is stored in a first data structure that conforms to a first data structure definition.

[0068] Clause 10: The method of any one of clauses 1-9 wherein obtaining the set of actions includes receiving the set of actions from the client device and / or the technician device upon completion of the remote control session.

[0069] Clause 11: The method of any one of clauses 1-10 wherein obtaining the set of actions includes recording the set of actions executed by the client device as instructed by the technician device.

[0070] Clause 12: The method of any one of clauses 1-11 wherein generating the summary includes: determining whether a respective action of the set of actions modified a state of the client device, and in response to a determination that the respective action did not modify the state of the client device, excluding the respective action from the summary.

[0071] Clause 13: The method of any one of clauses 1-12 further comprising automatically detecting the start of the remote control session.

[0072] Clause 14: A non-transitory computer-readable medium storing processor-executable instructions, wherein the instructions include: determining a start of a remote control session between a technician device and a client device; in response to the start of the remote control session, starting a recording session, wherein the recording session includes obtaining a set of actions performed under remote control via the technician device on the client device; determining whether one or more criteria of a set of analysis criteria have been met; and in response to a determination that one or more criteria of the set of analysis criteria have been met: storing the set of actions; analyzing the set of actions; supplying the set of actions to a machine-learning model; and generating a summary of the set of actions based on an output of the machinelearning model.

[0073] Clause 15: The non-transitory computer-readable medium of clause 14 wherein: the recording session includes a set of recording segments, and the set of recording segments includes a first recording segment and a second recording segment that are a first duration and separated by a second duration.

[0074] Clause 16: The non-transitory computer-readable medium of clause 15 wherein the set of analysis criteria includes:a first criterion that is met when an end of the remote control session is detected, and a second criterion that is met when a respective recording segment of the set of recording segments is saved.

[0075] Clause 17: The non-transitory computer-readable medium of any one of clauses 14-16 wherein generating the summary of the set of actions includes, for a respective action of the set of actions, at least one of: interpreting a set of communications from a first user of the technician device and / or a second user of the client device, wherein the set of communications is associated with the respective action; interpreting a context of the respective action, including at least one of: interpreting a user interface associated with the respective action, interpreting a user selection associated with the respective action, and interpreting a user interface element of the user interface associated with the user selection; or generating a set of text describing the respective action.

[0076] Clause 18: The non-transitory computer-readable medium of clause 17 wherein generating the summary of the set of actions includes summarizing the set of text for each respective action.

[0077] Clause 19: A system comprising: memory hardware configured to store instructions; and processor hardware configured to execute the instructions stored by the memory hardware, wherein the instructions include: determining a start of a remote control session between a technician device and a client device; in response to the start of the remote control session, starting a recording session, wherein the recording session includes obtaining a set of actions performed under remote control via the technician device on the client device; determining whether one or more criteria of a set of analysis criteria have been met; and in response to a determination that one or more criteria of the set of analysis criteria have been met: storing the set of actions; analyzing the set of actions; supplying the set of actions to a machine-learning model; and generating a summary of the set of actions based on an output of the machine-learning model.

[0078] Clause 20: The system of clause 19 wherein: the recording session includes a set of recording segments,the set of recording segments includes a first recording segment and a second recording segment that are a first duration and separated by a second duration, and the set of analysis criteria includes: a first criterion that is met when an end of the remote control session is detected, and a second criterion that is met when a respective recording segment of the set of recording segments is saved.

[0079] Clause 21: The system of any one of clauses 19-20 wherein generating the summary of the set of actions includes, for a respective action of the set of actions, at least one of: interpreting a set of communications from a first user of the technician device and / or a second user of the client device, wherein the set of communications is associated with the respective action; interpreting a context of the respective action, including at least one of: interpreting a user interface associated with the respective action, interpreting a user selection associated with the respective action, and interpreting a user interface element of the user interface associated with the user selection; or generating a set of text describing the respective action.

[0080] Clause 22: The system of clause 21 wherein generating the summary of the set of actions includes summarizing the set of text for each respective action.

[0081] Clause 23: A non-transitory computer-readable storage medium comprising processorexecutable instructions that embody the method of any one of clauses 1-13.

[0082] Clause 24: A system that executes the method of any one of clauses 1-13.

Claims

CLAIMS1. A method comprising: determining a start of a remote control session between a technician device and a client device; in response to the start of the remote control session, starting a recording session, wherein the recording session includes obtaining a set of actions performed under remote control via the technician device on the client device; determining whether one or more criteria of a set of analysis criteria have been met; and in response to a determination that one or more criteria of the set of analysis criteria have been met: storing the set of actions; analyzing the set of actions; supplying the set of actions to a machine-learning model; and generating a summary of the set of actions based on an output of the machinelearning model.

2. The method of claim 1 wherein the start of the remote control session is determined in response to detecting a user input.

3. The method of claim 1 wherein: the recording session includes a set of recording segments, and the set of recording segments includes a first recording segment and a second recording segment that are a first duration and separated by a second duration.

4. The method of claim 3 wherein the set of analysis criteria includes: a first criterion that is met when an end of the remote control session is detected, and a second criterion that is met when a respective recording segment of the set of recording segments is saved.

5. The method of claim 1 wherein generating the summary of the set of actions includes, for a respective action of the set of actions, at least one of: interpreting a set of communications from at least one of a first user of the technician device or a second user of the client device, wherein the set of communications is associated with the respective action; interpreting a context of the respective action, including at least one of: interpreting a user interface associated with the respective action, interpreting a user selection associated with the respective action, and interpreting a user interface element of the user interface associated with the user selection; or generating a set of text describing the respective action.

6. The method of claim 5 wherein generating the summary of the set of actions includes summarizing the set of text for each respective action.

7. The method of claim 1 wherein the set of actions includes at least one of: a mouse event on a mouse input device, a touch event on a touch-sensitive input device, a keyboard stroke on a keyboard input device, a voice command on a sound-sensitive input device, and a gesture input on a gesture-sensitive input device.

8. The method of claim 1 further comprising: receiving a service ticket corresponding to an undesired system state of the client device; assigning the service ticket to a technician; and initiating the remote control session between the technician device and the client device, wherein the start of the remote control session is determined in response to initiation of the remote control session.

9. The method of claim 1 wherein each action of the set of actions is stored in a first data structure that conforms to a first data structure definition.

10. The method of claim 1 wherein obtaining the set of actions includes receiving at least one of the set of actions from the client device or the technician device upon completion of the remote control session.

11. The method of claim 1 wherein obtaining the set of actions includes recording the set of actions executed by the client device as instructed by the technician device.

12. The method of claim 1 wherein generating the summary includes: determining whether a respective action of the set of actions modified a state of the client device, and in response to a determination that the respective action did not modify the state of the client device, excluding the respective action from the summary.

13. The method of claim 1 further comprising automatically detecting the start of the remote control session.

14. A non-transitory computer-readable medium storing processor-executable instructions, wherein the instructions include: determining a start of a remote control session between a technician device and a client device; in response to the start of the remote control session, starting a recording session, wherein the recording session includes obtaining a set of actions performed under remote control via the technician device on the client device;determining whether one or more criteria of a set of analysis criteria have been met; and in response to a determination that one or more criteria of the set of analysis criteria have been met: storing the set of actions; analyzing the set of actions; supplying the set of actions to a machine-learning model; and generating a summary of the set of actions based on an output of the machinelearning model.

15. The non-transitory computer-readable medium of claim 14 wherein: the recording session includes a set of recording segments, and the set of recording segments includes a first recording segment and a second recording segment that are a first duration and separated by a second duration.

16. The non-transitory computer-readable medium of claim 15 wherein the set of analysis criteria includes: a first criterion that is met when an end of the remote control session is detected, and a second criterion that is met when a respective recording segment of the set of recording segments is saved.

17. The non-transitory computer-readable medium of claim 14 wherein generating the summary of the set of actions includes, for a respective action of the set of actions, at least one of: interpreting a set of communications from a first user of the technician device and / or a second user of the client device, wherein the set of communications is associated with the respective action; interpreting a context of the respective action, including at least one of: interpreting a user interface associated with the respective action, interpreting a user selection associated with the respective action, and interpreting a user interface element of the user interface associated with the user selection; or generating a set of text describing the respective action.

18. The non-transitory computer-readable medium of claim 17 wherein generating the summary of the set of actions includes summarizing the set of text for each respective action.

19. A system comprising: memory hardware configured to store instructions; and processor hardware configured to execute the instructions stored by the memory hardware,wherein the instructions include: determining a start of a remote control session between a technician device and a client device; in response to the start of the remote control session, starting a recording session, wherein the recording session includes obtaining a set of actions performed under remote control via the technician device on the client device; determining whether one or more criteria of a set of analysis criteria have been met; and in response to a determination that one or more criteria of the set of analysis criteria have been met: storing the set of actions; analyzing the set of actions; supplying the set of actions to a machine-learning model; and generating a summary of the set of actions based on an output of the machine-learning model.

20. The system of claim 19 wherein: the recording session includes a set of recording segments, the set of recording segments includes a first recording segment and a second recording segment that are a first duration and separated by a second duration, and the set of analysis criteria includes: a first criterion that is met when an end of the remote control session is detected, and a second criterion that is met when a respective recording segment of the set of recording segments is saved.

21. The system of claim 19 wherein generating the summary of the set of actions includes, for a respective action of the set of actions, at least one of: interpreting a set of communications from a first user of the technician device and / or a second user of the client device, wherein the set of communications is associated with the respective action; interpreting a context of the respective action, including at least one of: interpreting a user interface associated with the respective action, interpreting a user selection associated with the respective action, and interpreting a user interface element of the user interface associated with the user selection; or generating a set of text describing the respective action.

22. The system of claim 21 wherein generating the summary of the set of actions includes summarizing the set of text for each respective action.

Images