Determining Latency for Human Interface Devices

Abstract

To describe the latency of a human interface device (HID) in determining the end-to-end latency of a system.SOLUTION: When an input is received in an HID, the amount of time required for the input to reach a connected device is calculated by the HID and included in a data packet transmitted by the HID device to the connected device. Adding peripheral latency to the end-to-end latency determination yields more comprehensive latency results for the system. When HID's peripheral latency has been determined to have a non-negligible contribution to the end-to-end latency, new HID components will be implemented, configuration settings related to the HID components will be updated, and / or other actions will be taken to reduce the contribution of the peripheral latency to the total latency of the system.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] This application is related to U.S. Non-provisional Application No. 16 / 893,327, filed June 4, 2020, U.S. Provisional Application No. 63,029,343, filed May 22, 2020, and U.S. Provisional Application No. 62 / 970,096, filed February 4, 2020, each of which is incorporated by reference in its entirety herein. [Background technology]

[0002] As the number of high performance applications, e.g., gaming applications, virtual reality (VR) applications, augmented reality (AR) applications, and / or mixed reality (MR) applications, increases, the performance of the computing systems that run these applications becomes more important. For example, to run a first person shooter (FPS) type game, a computing system, e.g., a gaming console, a personal computer, a cloud gaming system, etc., should be configured such that the end-to-end latency of the system is at a level that provides a meaningful user experience. To understand and describe latency in a computing system, conventional techniques have implemented systems to measure a single end-to-end latency value. For example, a customized hardware component, such as a high-speed camera, may be implemented to detect when an input event has occurred (e.g., by detecting an LED light indicator on a peripheral device equipped with the same), to detect when a display event has occurred, and to calculate (e.g., manually) the time difference or latency between the two. However, this process of end-to-end latency measurement requires specialized equipment, e.g., peripheral devices with high-speed cameras and visual input indicators, while also yielding only a single end-to-end latency value. The drawback of a single latency value is that various factors may contribute to the end-to-end latency of a system, e.g., peripheral latency, application latency, rendering latency, and / or display latency. Thus, without knowing the individual contribution of each of these factors to the end-to-end system latency, it may prove difficult to determine configuration setting updates, necessary changes to the system hardware and / or software, and / or other actions that may be taken to reduce the end-to-end latency of the system. Summary of the Invention

[0003] Embodiments of the present disclosure relate to accounting for a human interface device (HID) in end-to-end system latency determination. Systems and methods are disclosed that use data generated and transmitted at least in part by the HID device to determine the latency contribution of the HID. For example, when an input is received, the amount of time it takes for data representing the input to reach a connected device, such as a personal computer (PC), display (e.g., a display implementing a pass-through universal serial bus (USB) port), etc. (e.g., peripheral latency) may be calculated by the HID and included in a data packet transmitted by the HID device to the connected device.

[0004] Thus, in contrast to conventional systems, specialized hardware is not required to calculate the end-to-end latency of the system, and individual contributions to the end-to-end latency may be calculated at a more granular level, for example, including peripheral latency as calculated by the HID device. When implemented in an end-to-end latency determination system such as that described in U.S. Nonprovisional Application No. 16 / 893,327, filed June 4, 2020, and incorporated herein by reference, the addition of peripheral latency to the end-to-end latency determination may result in a more comprehensive result. For example, in addition to determining the latency from when an input is received by a connected device to when the input results in a display change, the peripheral latency may include an additional amount of time from when the physical input is registered by the HID to the receipt of the input data at the connected device. Thus, if the peripheral latency of the HID is determined to have a non-negligible contribution to the end-to-end latency, new HID components may be implemented, configuration settings associated with the HID components (e.g., polling rates) may be updated, and / or other actions may be taken to reduce the contribution of the peripheral latency to the end-to-end latency of the system.

[0005] The end-to-end latency of a system may be calculated, in an embodiment, by a display device of the system. For example, a processor present within the display device may be used to calculate the end-to-end latency by factoring in peripheral latency, application latency, rendering latency, display latency, and / or other latency contributions. In one such embodiment, the display device may include pass-through functionality for the HID connection type (e.g., Universal Serial Bus (USB), serial port, parallel port, Ethernet, etc.) and may intercept (e.g., using an intercept device such as a field programmable gate array (FPGA)) peripheral latency data from data packets transmitted by the HID through the display device to a computing device (e.g., a gaming console, a desktop computer, a laptop computer, a tablet, etc.). Thus, the processor of the display device may be used to calculate the end-to-end latency of a system that is agnostic to the particular type of computing device running the application, thereby allowing for more universal applicability of the end-to-end latency determination system.

[0006] The present system and method for accounting for human interface devices in determining end-to-end system latency is described in detail below with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0007] [Figure 1A] 1 is a block diagram of an end-to-end latency determination system according to some embodiments of the present disclosure. [Figure 1B]FIG. 1 is a block diagram of an example configuration of an end-to-end latency determination system according to some embodiments of the present disclosure. [Figure 2A] FIG. 11 is a diagram of a report descriptor corresponding to a human interface device according to some embodiments of the present disclosure. [Figure 2B] 11 is a diagram of an example data field of a data packet corresponding to a human interface device according to some embodiments of the present disclosure. [Diagram 3] 1 is a flow diagram of a method for generating a data packet to include a peripheral latency value, according to some embodiments of the present disclosure. [Figure 4] 1 is a flow diagram of a method for determining peripheral latency from data packets received from a human interface device, according to some embodiments of the present disclosure. [Diagram 5] FIG. 1 is a block diagram of an example content streaming system suitable for use in implementing some embodiments of the present disclosure. [Figure 6] FIG. 1 is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 7] FIG. 1 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Systems and methods are disclosed relating to accounting for human interface devices (HIDs) in end-to-end system latency determination. The present disclosure is primarily described in the context of a computer mouse as an HID, but this is not intended to be limiting. The HID device latency determination described herein may correspond to any type of HID device, such as a computer mouse, a keyboard, a controller device, a game pad, a joystick, a remote, a microphone, a trajectory pad, a virtual reality (VR) headset, an augmented reality (AR) headset or eyewear, a mixed reality (MR) headset or eyewear, a display, a touch screen display, a barcode reader, an image scanner, a camera (e.g., a webcam, a digital camera, etc.), a light pen, a handle, a scanner, and / or other types of peripherals or HID devices. Additionally, the present disclosure is primarily described in the context of a gaming implementation, but this is not intended to be limiting. For example, peripheral device latency may be used to calculate latency for any type of application, such as gaming applications, streaming applications, computer aided design (CAD) applications, video, photo, or audio editing applications, VR, AR and / or MR applications, video conferencing applications, robotics applications, ground vehicle and / or airborne applications (e.g., autonomous, semi-autonomous, driver-assisted, etc.), simulation applications, and / or other application types. The systems and methods described herein may be implemented in a local computing system, a cloud computing system, or a combination thereof. Connections between HIDs or peripheral devices may include wired connections, wireless connections (e.g., using wireless transceivers), or a combination thereof.

[0009] Referring to FIG. 1A, FIG. 1A is a block diagram of an end-to-end latency determination system 100 (alternatively referred to herein as “system 100”) according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are described merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by an entity may be implemented by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, system 100 may be implemented using similar components, features, and / or functionality as described herein with respect to the exemplary computing device 600 of FIG. 6. In embodiments, system 100 may be part of a cloud computing architecture that may use one or more data centers (or components, features, and / or functionality thereof), such as example data center 700 of FIG. 7. Additionally, in some embodiments, for example, when system 100 includes a content streaming system (e.g., game streaming, AR / VR streaming, video streaming, video conferencing, etc.), system 100 may include similar components, features, and / or functionality described herein with respect to example content streaming system 500 of FIG.

[0010] The system 100 may include one or more computing devices 102 and / or one or more human interface devices (HIDs) 104. The computing devices 102 may include, but are not limited to, one or more of the types of computing devices described with respect to the content server 502 and / or client device 504 of FIG. 5, the exemplary computing device 600 of FIG. 6, the exemplary data center 700 of FIG. 7, and / or other computing device types. For example, the computing devices 102 may include a laptop, a desktop, a display device, a gaming console, a VR and / or AR system, a tablet, a smart phone, a cloud computing system, a virtual machine, another computing device type, or a combination thereof.

[0011] As described herein, the HID 104 may include, but is not limited to, one or more of the input devices 526 of Figure 5, one or more of the input / output (I / O) devices 614 of Figure 6, and / or one or more other HID device types. For example, the HID 104 may include a computer mouse, a keyboard, a controller, a game pad, a joystick, a remote, a microphone, a trajectory pad, a virtual reality (VR) headset, an augmented reality (AR) headset or eyewear, a mixed reality (MR) headset or eyewear, a display, a touchscreen display, a barcode reader, an image scanner, a camera (e.g., a webcam, digital camera, etc.), a light pen, a handle, a scanner, another type of peripheral or HID device, or a combination thereof.

[0012] Computing device 102 may include display 106. For example, display 106 may include similar components, features, and / or functionality as display 524 of FIG. 5 and / or presentation component 618 of FIG. 6. In some embodiments, display 106 may include a touch screen display and may correspond to HID 104 of system 100. Display 106 may include one or more components for determining end-to-end latency of system 100 in some embodiments. For example, with respect to system 100A of FIG. 1B, such as a non-limiting example embodiment of end-to-end latency determination system 100 of FIG. 1A, display 106 may include a central processing unit 150, a field programmable gate array (FPGA) 154, and / or other components, including but not limited to those described herein. In such an embodiment, the display 106 may be used to determine the end-to-end latency of the system 100A such that the latency may be determined regardless of the type of computing device 102 implemented. For example, the display 106 may determine the end-to-end latency of the system 100A when the computing device 102 is a gaming console and may also determine the end-to-end latency when the computing device 102 is a desktop computer. Similarly, the display 106 may still accurately determine the end-to-end latency of the system 100A when the computing device 102 and / or its components, features, and / or functionality are manufactured or provided by different providers or companies.Thus, if a first computing device 102 includes a graphics processing unit (GPU) 132 manufactured by a first company and a second computing device 102 includes a GPU 132 manufactured by a second company, end-to-end latency may be accurately determined for both the first and second computing devices 102. System 100A may, in some embodiments, include components, features, and / or functionality similar to those described in U.S. Non-Provisional Application No. 16 / 893,327, filed June 4, 2020, which is incorporated by reference herein in its entirety.

[0013] The packet interceptor 108 can intercept packets received from the HID 104. For example, a packet generator 124 of the HID 104 can generate packets including data representing peripheral (or HID) latency information of the HID 104, and the transceiver 120 (which may include a transmitter, a receiver, and / or a transceiver) can transmit the packets to the computing device 102 - e.g., via a wired and / or wireless connection or communication type, e.g., Universal Serial Bus (USB), Ethernet, Bluetooth, Bluetooth Low Energy (BLE), etc. The packet interceptor 108 of the computing device 102 - and / or its display 106, e.g., implemented with pass-through functionality - can intercept the packets and determine the latency information therefrom. For example, when an input is received by an input receiver 118 of the HID 104, the peripheral latency determiner 122 may determine the amount of time from receipt of the input to successful transmission—e.g., using the transceiver 120 and / or the transceiver 116 (which may include a receiver, a transmitter, and / or a transceiver)—of a packet containing data (e.g., input data) representing the input to the computing device 102. This amount of time may correspond to the peripheral latency of the HID 104—or that particular input receiver 118.

[0014] In some instances, the input receiver 118 may include a button (e.g., a mouse button, a keyboard button, a remote control button, a game controller button, etc.), a display (e.g., a touch screen), a track pad, a motion determining device (e.g., an inertial measurement unit (IMU) sensor, and / or another type of component, feature, and / or functionality for measuring rotation and / or translation of the HID 104 - e.g., for measuring movement of a mouse, joystick, control pad, controller, etc.), and / or another type of input receiver 118. As such, the input receiver 118 may receive the input and generate data representative of the input, which may be included in packets generated by the packet generator 124 and transmitted to the computing device 102.

[0015] In some embodiments, the packet interceptor 108 may intercept a packet - e.g., may determine that a packet including peripheral latency data has been received, and may obtain or receive latency data from the packet for use by the latency determination device 110 in determining the peripheral latency and / or for using the peripheral latency in determining the end-to-end latency of the system 100. Thus, the packet may be used by other components, features, and / or functionality of the system 100 other than the latency determination device 110 (e.g., for use in updating application state for rendering), and the packet interceptor 108 may intercept the packet for use by the latency determination device 110 (e.g., as described with respect to the FPGA 154 of FIG. 1B). In other embodiments, the packet interceptor 108 may not be able to intercept the packet, but may correspond to a receiver of the packet as identified in the transmission. For example, if pass-through functionality is not implemented, packets containing peripheral latency information may be sent from HID 104 to computing device 102 directly or with pass-through functionality, but without interception. In such an embodiment, packet interceptor 108 may alternatively be referred to as packet receiver 108. As such, packet receiver 108 may be implemented if computing device 102 - not a separate device, such as display 106, as described with respect to FIG. 1B - executes latency determination apparatus 110.

[0016] To determine peripheral latency, the peripheral latency determiner 122 can analyze data corresponding to receipt of input by the input receiver 118 and receipt of packets including input data by the computing device 102 and / or a pass-through device, such as the display 106 of FIG. 1B. To determine receipt of input, a timestamp can be generated upon receipt of the input. For example, a timestamp can be generated and stored by the HID 104 when a mouse button, control pad, or keyboard button is pressed, a joystick is manipulated, a mouse cursor is moved, a touchpad is touched, a display is touched or pressed, and / or another input type is received. Additionally, a timestamp can be generated when a packet including input data, such as one generated using the packet generator 124, is successfully received by and / or transmitted to the computing device 102. For example, the HID 104 can determine successful transmission and / or reception of a packet and can generate a timestamp when successful transmission and / or reception is determined. As another example, the HID 104 may determine successful transmission and / or reception of a packet based on a return signal from the computing device 102 and / or a pass-through device, such as the display 106 of FIG. 1B. In such an embodiment, a timestamp may be generated by the computing device 102 and included in the return signal. In either embodiment, the HID 104 may use a timestamp corresponding to a time of receipt of the input and a timestamp corresponding to a time of successful transmission and / or reception of the packet including the input data by the computing device 102, and the peripheral latency determiner 122 may use these timestamps to determine the peripheral latency of the HID 104. For example, a difference between the timestamps may be calculated to determine the peripheral latency.

[0017] After the peripheral latency is determined, the packet generator 124 can generate a packet that includes the peripheral latency information. The packet may include the peripheral latency information alone and / or may include input data that corresponds to the input used in determining the peripheral latency and / or input data that corresponds to a subsequent input. If the input data corresponds to a subsequent input, the packet may include data that references the input to which the peripheral latency corresponds.

[0018] The HID 104 - e.g., during initialization, at power-up, during configuration, when plugged in or otherwise communicatively coupled, periodically, etc. - may transmit a report descriptor 200 to the computing device 102. The report descriptor 200 may include information identifying the HID 104 implemented (e.g., type, model, vendor, identifier, etc.), associated latency (when applicable), the number and / or type of input receivers of the HID 104, the format of data packets transmitted by the HID 104, and / or other information. For example, the report descriptor 200 may include data that instructs the computing device 102 which bits and / or bytes of a data packet from the HID 104 correspond to which information. As such, when a data packet containing information representing the HID report 210 is received by the computing device 102, the computing device 102 understands that there are 16 buttons, and that the input information corresponding to those buttons is 8 bits long, has an offset of 0 or 8 bits, and / or has a hexadecimal value of 0x01 or 0x00. Similarly, the computing device 102 may understand data corresponding to X position or translation, Y position or translation, and wheel input information (e.g., if the HID report 210 corresponds to a mouse), as well as AC panning information, and button latency information. These data fields are merely for illustrative purposes and are not intended to be limiting. For example, for a joystick, the data fields may include X, Y movement information and / or button inputs in addition to latency information. For a game controller, the data fields in the HID report 210 may include joystick movement, button type, and / or latency information. As such, the report descriptor 200 may indicate to the computing device 102 -and / or a pass-through device, such as the display 106 of FIG. 1B -the location of the data packet containing each of the different data fields.With respect to the packet interceptor 108 (and / or packet receiver 108), the information in the latency data field may be used to determine or obtain the peripheral latency of the HID 104 from the received packet. As shown in FIG. 2A, the bolded and underlined portions of the report descriptor may correspond to the peripheral latency data field of the HID report 210. As such, where conventional systems determine peripheral latency using separate hardware, such as a high-speed camera and LED light, the system 100 may determine the peripheral latency without requiring additional hardware and may include the peripheral latency as additional information in the data packet transmitted by the HID 104 to the computing device 102.

[0019] In some embodiments, the peripheral latency determiner 122 may be executed by the computing device 102. For example, the type and / or model of the HID 104 may be determined, e.g., using a report descriptor 200 corresponding to the HID, and this information may be used to determine associated latency information for the HID 104. As such, the peripheral latency determiner 122 may reference a look-up table or other data representation that includes a list of HID types, e.g., by model number, vendor identifier, product identifier, serial number, device identifier, unique identifier, etc., and associated (e.g., predetermined) latency information. As such, the peripheral latency determiner 122 may be part of the latency determiner 110 for determining the end-to-end latency of the system 100 and / or individual contributions thereto. For example, if the end-to-end latency of the system is determined to be 38 milliseconds (ms) and the associated latency of the HID 104 is 8 ms, a report may be generated by the report generator 112 indicating that the end-to-end latency is 38 ms and that the HID 104 contributed 8 ms to that 38 ms, or that 21% of the latency is attributable to the HID 104. This information may be useful for a user—e.g., to determine whether a different type or model of HID device would reduce the end-to-end latency—and / or by the computing device 102 to update configuration settings using the configuration updater 114. For example, if the desired latency is 30 ms, the HID-related configuration settings may be updated by the system 100—e.g., automatically, periodically, and / or dynamically—to reduce the peripheral latency contribution. As such, the polling rate of the computing device 102 may be increased such that HID information is received more frequently. In such an example, if the peripheral latency is determined to exceed some threshold value - e.g., 8 ms, 12 ms, 15 ms, etc. - the polling rate may be updated (e.g., until the peripheral latency of subsequent inputs falls below the threshold value).

[0020] In addition to or instead of updating the HID-related configuration settings, the system 100 may update other configuration settings of the system - e.g., automatically, periodically, and / or dynamically. For example, the frame rate, refresh rate, and / or image quality of the system 100 may be updated. As another example, for example, one or more settings, e.g., V-sync, G-sync, and / or other settings of the display 106 may be enabled or disabled. In other examples, the peripheral latency information may be used for anti-fraud applications, e.g., if changes in the peripheral latency information do not match a predetermined or learned latency pattern of the HID 104 (e.g., if the changes in the latency information are greater than a threshold), the system 100 may determine that a user or bot is cheating.

[0021] In some embodiments, the report generator 112 may generate reports on demand, at intervals, periodically, automatically, dynamically, and / or otherwise. The reports may include information regarding the latency of the system 100, such as individual components of the latency and / or end-to-end latency values. The report generator 112 may determine changes in latency values ​​over time and may generate a report when the values ​​vary beyond a certain threshold amount. For example, if the peripheral latency varies beyond 5 ms, 8 ms, 10 ms, etc., the report generator 112 may generate a report and / or may generate and / or display a notice or indication of the detected change in the report. Similarly, if the latency value rises above a threshold or falls below a threshold, the report generator 112 may generate a report and / or may generate and / or display a notice or indication of the latency value threshold information. This information may assist the system 100 and / or the user in determining the current performance of the system 100 and whether any configuration updates have helped reduce latency and / or increased latency and / or may be useful in determining whether configuration settings need or should be updated. In an embodiment, the configuration updater 114 may use the report to automatically update configuration settings and / or generate prompts or notifications to the user of recommended updates that may be used to improve latency. In some embodiments, the user may be able to configure a threshold and a corresponding configuration setting update to occur when the threshold is met. This may include increasing the polling rate when the peripheral latency value increases above a threshold peripheral latency value or decreasing the polling rate when the peripheral latency value decreases below the threshold peripheral latency value. Another example may include decreasing image quality when the end-to-end latency exceeds a threshold or increasing image quality when the end-to-end latency falls below a threshold.In an embodiment, the thresholds may include multiple steps such that when a first high threshold is met, the image quality is reduced by a first amount, and at a second high threshold, the image quality is reduced more than at the first high threshold, etc. This gradual thresholding may also be implemented for HID-related configuration settings, display settings, rendering settings, application settings, and / or other settings that may contribute to latency of system 100.

[0022] 1B, which includes an exemplary, non-limiting embodiment of a system 100. The system 100A includes a display 106 communicatively coupled to a computing device 102. A HID 104, e.g., a mouse, may be coupled to the computing device 102 via a pass-through functionality of the display 106. As such, a USB connection (wired or wireless) may be established between the HID 104 and the display 106, and a USB connection (wired or wireless) may be established between the display 106 and the computing device 102. As such, when a packet is received from the HID 104 by a USB port 156A of the display 106, the display 106 may pass the packet to a USB port 156C of the computing device 102 via a USB port 156B. In some embodiments, when a packet is received from the HID 104, the display's FPGA 154, e.g., acting as the packet interceptor 108 of FIG. 1A, can determine that a packet has been received and / or can obtain or intercept peripheral latency information from the packet (e.g., if the location of the data field corresponding to the peripheral latency is known from the report descriptor 200). This peripheral latency information can be passed to the CPU 150 I2C / SPI 152 (e.g., half-duplex, full-duplex, or another type of communication protocol). The CPU 150 can be a component of a chip (or integrated circuit) included on the motherboard 146 of the display 146. The CPU 150 can execute a latency determination unit 110 that can use the peripheral latency information and / or one or more other components of the latency, e.g., application latency, rendering latency, and / or display latency, to determine the end-to-end latency of the system 100A.

[0023] As an example, a packet from HID 104 passing through computing device 102 may be received by computing device 102 and used to determine input information from the input data in the packet. For example, components on motherboard 130 of computing device 102, such as CPU 136, display driver 138, and / or USB driver 140, may be used to determine the input, update application state, and send application state information (e.g., via PCIe connection 134) to GPU 132 for rendering. Rendered information, such as a frame corresponding to an application and reflecting the received input data, may be sent to display 106 (e.g., via display port (DP) connections 142 and 144). Display 106 may then display the frame. In an embodiment, CPU 150 may determine additional latency information from the frame or its display. As such, system 100A may be used to determine peripheral latency associated with HID 104 and / or may be used to determine end-to-end latency of the system using the pass-through functionality of display 106.

[0024] 1B are not intended to be limiting, and different connections and device types may be used. For example, without limitation, Ethernet may be used in place of USB, HDMI may be used in place of DP, FPGA functionality may be implemented in CPU 150 itself or included in chip 148, etc. Additionally, although the components, features, and functionality of system 100A are described with respect to display 106 and computing device 102, this is not intended to be limiting. For example, all of the components, features, and / or functionality of system 100A may be performed or included in a single device - e.g., a single computing device 102, a single display 106, etc. - and / or a combination of devices in addition to or in place of display 106 and computing device 102.

[0025] Referring now to Figures 3-4, each block of the methods 300 and 400 described herein includes computational processes that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be implemented by processor-executed instructions stored in memory. The methods 300 and 400 may also be implemented as computer usable instructions stored on a computer storage medium. The methods 300 and 400 may be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, the methods 300 and 400 are described with respect to the system 100 of Figure 1A, by way of example. However, these methods 300 and 400 may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein.

[0026] 3, which is a flow diagram of a method 300 for generating a data packet to include a peripheral latency value, according to some embodiments of the present disclosure. The method 300 includes, at block B302, generating a first timestamp corresponding to a time that the input is received, in response to receiving the input. For example, the first timestamp may be generated when the input receiver 118 receives and / or registers the input.

[0027] The method 300 includes, at block B 304, transmitting input data representing the input to the computing device. For example, the packet generator 124 can generate packets representing the input data from the input, and the transceiver 120 can transmit the packets to the computing device 102.

[0028] The method 300 includes generating a second timestamp corresponding to a time when the input data was received by the computing device at block B 306. For example, the second timestamp may be generated by the computing device 102 and / or the HID 104 after successful transmission and / or reception of a packet.

[0029] The method 300 includes calculating a peripheral latency value based at least in part on the first timestamp and the second timestamp at block B308. For example, the peripheral latency determiner 122 may calculate the peripheral latency using the first timestamp and the second timestamp.

[0030] The method 300 includes generating a data packet to include the peripheral latency value at block B310. For example, the packet generator 124 can generate a packet (which may correspond to the packet from block B304 or a subsequent packet) that includes the peripheral latency information. The peripheral latency information can be represented in a data field of the packet defined by the report descriptor 200.

[0031] The method 300 includes, at block B 312, transmitting a data packet to a computing device. For example, the HID 104 can transmit a packet including the peripheral latency information to the computing device 102 and / or a pass-through device (e.g., the display 106 of FIG. 1B).

[0032] 4, which is a flow diagram of a method 400 for determining peripheral latency from a data packet received from a human interface device, according to some embodiments of the present disclosure. The method 400 includes, at block B402, determining, by the computing device, receipt of a data packet from an input device communicatively coupled to the computing device. For example, the packet interceptor 108 (and / or the packet receiver 108) may determine that the data packet was received by the HID 104.

[0033] The method 400 includes, at block B404, obtaining, by the computing device, data from the data packet indicative of a peripheral latency value associated with the input received at the input device. For example, the packet interceptor 108 of the computing device 102 may obtain and / or receive the peripheral latency value from a data field of the data packet corresponding to the peripheral latency, as defined in the report descriptor 200.

[0034] The method 400 includes performing one or more actions based at least in part on the peripheral latency value at block B406. For example, the computing device 102 (and / or the pass-through device) may generate a report using the report generator 112, update configuration settings using the configuration updater 114, and / or determine the end-to-end latency of the system 100 and / or the contribution of the HID 104 thereto.

[0035] Exemplary Content Streaming System Referring now to FIG. 5, FIG. 5 is an example system diagram of a content streaming system 500 according to some embodiments of the present disclosure. FIG. 5 includes an application server 502 (which may include components, features, and / or functionality similar to the example computing device 600 of FIG. 6), a client device 504 (which may include components, features, and / or functionality similar to the example computing device 600 of FIG. 6), and a network 506 (which may be similar to the networks described herein). In some embodiments of the present disclosure, the system 500 may be implemented. The application sessions may correspond to game streaming applications (e.g., NVIDIA GeFORCE NOW), remote desktop applications, simulation applications (e.g., autonomous or semi-autonomous vehicle simulations), computer aided design (CAD) applications, virtual reality (VR) and / or augmented reality (AR) streaming applications, deep learning applications, and / or other application types.

[0036] In the system 500, for an application session, the client device 504 may simply receive input data in response to input on an input device, transmit the input data to the application server 502, receive encoded display data from the application server 502, and display the display data on the display 524. As such, more computationally intensive calculations and processing are offloaded to the application server 502 (e.g., rendering of the application session's graphical output - particularly ray or path tracing - is performed by the GPU of the game server 502). In other words, the application session is streamed from the application server 502 to the client device 504, thereby reducing the requirements of the client device 504 for graphics processing and rendering.

[0037] For example, with respect to instantiating an application session, the client device 504 may display frames of the application session on the display 524 based on receiving display data from the application server 502. The client device 504 may receive input to one of the input devices and generate input data in response. The client device 504 may send the input data via the communication interface 520 and via the network 506 (e.g., the Internet) to the application server 502, which may receive the input data via the communication interface 518. The CPU may receive the input data, process the input data, and send data to the GPU that causes the GPU to generate a rendering of the application session. For example, the input data may represent movement of a user's character, firing a weapon, reloading, passing a ball, turning a vehicle, etc. in a game session of a game application. A rendering component 512 can render an application session (e.g., representing the results of input data), and a rendering capture component 514 can capture the rendering of the application session as display data (e.g., as image data capturing the rendered frames of the application session). The rendering of the application session can include ray or path trace lighting and / or shading effects that are calculated using one or more parallel processing units of the application server 502, such as a GPU, which can perform ray or path tracing techniques with further use of one or more dedicated hardware accelerators or processing cores. In some embodiments, one or more virtual machines (VMs) can be used by the application server 502 to support the application sessions, such as, for example, including one or more virtual components, e.g., vGPU, vCPU, etc.The encoder 516 may then encode the display data to generate encoded display data, which may be transmitted over the network 506 via the communications interface 518 to the client device 504. The client device 504 may receive the encoded display data via the communications interface 520, and the decoder 522 may decode the encoded display data to generate the display data. The client device 504 may then display the display data via the display 524.

[0038] Exemplary Computing Device 6 is a block diagram of an example computing device 600 suitable for use in implementing some embodiments of the present disclosure. Computing device 600 may include an interconnect system 602 that directly or indirectly couples the following devices: memory 604, one or more central processing units (CPUs) 606, one or more graphics processing units (GPUs) 608, a communications interface 610, input / output (I / O) ports 612, input / output components 614, a power supply 616, one or more presentation components 618 (e.g., displays), and one or more logic units 620. In at least one embodiment, computing device 600 may include one or more virtual machines (VMs) and / or any of its components may include virtual components (e.g., virtual hardware components). As non-limiting examples, one or more of GPUs 608 may include one or more vGPUs, one or more of CPUs 606 may include one or more vCPUs, and / or one or more of logic units 620 may include one or more virtual logic units. As such, computing device 600 may include discrete components (e.g., a complete GPU dedicated to computing device 600), virtual components (e.g., a portion of a GPU dedicated to computing device 600), or a combination thereof.

[0039] While the various blocks in FIG. 6 are depicted as being connected via an interconnect system 602 having lines, this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 618, e.g., a display device, may be considered an I / O component 614 (e.g., where the display is a touch screen). As another example, the CPU 606 and / or the GPU 608 may include memory (e.g., the memory 604 may represent a storage device in addition to the memory of the GPU 608, the CPU 606, and / or other components). In other words, the computing devices of FIG. 6 are merely illustrative. Categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "gaming machine," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types are not distinguished, as they are all contemplated to be within the scope of the computing devices of FIG. 6.

[0040] The interconnect system 602 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 602 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 606 may be directly connected to the memory 604. Additionally, the CPU 606 may be directly connected to the GPU 608. When there is a direct or point-to-point connection between components, the interconnect system 602 may include a PCIe link to implement the connection. In these instances, a PCI bus need not be included in the computing device 600 .

[0041] Memory 604 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 600. Computer-readable media may include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may include computer storage media and communication media.

[0042] Computer storage media may include both volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information, e.g., computer readable instructions, data structures, program modules, and / or other data types. For example, memory 604 may store computer readable instructions (e.g., programs and / or program elements, e.g., representing an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computing device 600. As used herein, computer storage media does not include signals per se.

[0043] Computer storage media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, e.g., a carrier wave or other transport mechanism, and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media may include wired media, e.g., a wired network or direct-wired connection, and wireless media, e.g., acoustic, RF, infrared and other wireless media. Combinations of any of the foregoing should also be included within the scope of computer-readable media.

[0044] The CPU 606 may be configured to execute at least some of the computer readable instructions for controlling one or more components of the computing device 600 to perform one or more of the methods and / or processes described herein. The CPU 606 may include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.), each capable of simultaneously processing multiple software threads. The CPU 606 may include any type of processor, and may include different types of processors depending on the type of computing device 600 implemented (e.g., a processor with fewer cores for a mobile device and a processor with a larger number of cores for a server). For example, depending on the type of computing device 600, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). Computing device 600 may include one or more CPUs 606 in addition to one or more microprocessors or supplemental coprocessors, such as math coprocessors.

[0045] In addition to, or instead of, CPU 606, GPU 608 may be configured to execute at least some of the computer readable instructions for controlling one or more components of computing device 600 to perform one or more of the methods and / or processes described herein. One or more of GPUs 608 may be an integrated GPU (e.g., with one or more of CPUs 606 and / or one or more of GPUs 608 may be a discrete GPU. In an embodiment, one or more of GPUs 608 may be a co-processor of one or more of CPUs 606. GPU 608 may be used by computing device 600 to render graphics (e.g., 3D graphics) or perform general purpose computations. For example, GPU 608 may be configured to perform general purpose computing on GPU (GPGPU) functions. A graphics processor (GPU) may be used for the GPU 608. The GPU 608 may include hundreds or thousands of cores capable of simultaneously processing hundreds or thousands of software threads. The GPU 608 may generate pixel data for an output image in response to rendering commands (e.g., rendering commands from the CPU 606 received via a host interface). The GPU 608 may include graphics memory, e.g., display memory, for storing the pixel data or any other suitable data, e.g., GPGPU data. The display memory may be included as part of the memory 604. GPU 608 may include two or more GPUs operating in parallel (e.g., via links). The links may connect the GPUs directly (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When coupled together, each GPU 608 may generate pixel data or GPGPU data for a different portion of the output or for a different output (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with the other GPUs.

[0046] In addition to or instead of the CPU 606 and / or GPU 608, the logic unit 620 may be configured to execute at least some of the computer readable instructions for controlling one or more components of the computing device 600 to perform one or more of the methods and / or processes described herein. In an embodiment, the CPU 606, the GPU 608, and / or the logic unit 620 may execute any combination of the methods, processes and / or portions thereof separately or together. One or more of the logic units 620 may be part of and / or integrated with one or more of the CPU 606 and / or GPU 608, and / or one or more of the logic units 620 may be separate components or otherwise external to the CPU 606 and / or GPU 608. In an embodiment, one or more of the logic units 620 may be a co-processor of one or more of the CPU 606 and / or GPU 608.

[0047] Examples of logic unit 620 include one or more processing cores and / or components thereof, such as a Tensor Core (TC), a Tensor Processing Unit (TPU), a Pixel Visual Core (PVC), a Vision Processing Unit (VPU), a Graphics Processing Cluster (GPC), a Texture Processing Cluster (TPC), a Streaming Multiprocessor (SM), a Tree Traversal Unit (TTU), an Artificial Intelligence Accelerator (AIA), a Deep Learning Accelerator (DLA), an Arithmetic-Logic Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating Point Unit (FPU), a 32-bit LSB (32-bit LSB ... The term “component interconnect” refers to a physical layer that may include a peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) elements, and / or the like.

[0048] The communications interface 610 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 600 to communicate with other computing devices over electronic communications networks, including wired and / or wireless communications. The communications interface 610 may include components and functionality to enable communication over any of a number of different networks, such as a wireless network (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (e.g., communicating over Ethernet or InfiniBand), a low power wide area network (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0049] The I / O ports 612 may enable the computing device 600 to be logically coupled to other devices, including an I / O component 614, a presentation component 618, and / or other components, some of which may be incorporated (e.g., integrated) into the computing device 600. Exemplary I / O components 614 include a microphone, a mouse, a keyboard, a joystick, a game pad, a game controller, a satellite dish, a scanner, a printer, a wireless device, and the like. The I / O component 614 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by a user. In some cases, the input may be sent to an appropriate network element for further processing. The NUI may implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, recognition of gestures both on and near the screen, air gestures, head and eye tracking, and touch recognition in conjunction with the display of the computing device 600 (as described in more detail below). The computing device 600 may include a depth camera for gesture detection and recognition, e.g., a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof. In addition, the computing device 600 may include an accelerometer or gyroscope (e.g., as part of an inertia measurement unit (IMU)) that enables detection of motion. In some instances, the output of the accelerometer or gyroscope may be used by the computing device 600 to render immersive augmented or virtual reality.

[0050] Power supply 616 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 616 may provide power to computing device 600 to enable components of computing device 600 to operate.

[0051] The presentation component 618 may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up-display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component 618 can receive data from other components (e.g., GPU 608, CPU 606, etc.) and output data (e.g., as images, video, sound, etc.).

[0052] Exemplary Data Center 7 illustrates an example data center 700 that may be used in at least one embodiment of the present disclosure. The data center 700 may include a data center infrastructure layer 710, a framework layer 720, a software layer 730, and / or an application layer 740.

[0053] 7, the data center infrastructure layer 710 can include a resource orchestrator 712, grouped computational resources 714, and node computational resources ("node CRs") 716(1)-716(N), where "N" represents any whole, positive integer. In at least one embodiment, the node CRs 716(1)-716(N) can include, but are not limited to, any number of central processing units (CPUs) or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules, etc. In some embodiments, one or more of the nodes CR716(1)-716(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, the nodes CR716(1)-716(N) may include one or more virtual components, such as a vGPU, a vCPU, and / or the like, and / or one or more of the nodes CR716(1)-716(N) may correspond to a virtual machine (VM).

[0054] In at least one embodiment, the grouped computing resources 714 may include separate groups of nodes CR716 stored in one or more racks (not shown), or multiple racks (also not shown) stored in data centers in various geographic locations. The separate groups of nodes CR716 in the grouped computing resources 714 may include grouped computing, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR716 including CPUs, GPUs, and / or other processors may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches, in any combination.

[0055] The resource orchestrator 722 may configure or otherwise control one or more nodes CR 716(1)-716(N) and / or grouped computational resources 714. In at least one embodiment, the resource orchestrator 722 may include a software design infrastructure (SDI) management entity for the data center 700. The resource orchestrator 722 may include hardware, software, or some combination thereof.

[0056] In at least one embodiment, as shown in FIG. 7 , framework layer 720 may include a job scheduler 732, a configuration manager 734, a resource manager 736, and / or a distributed file system 738. Framework layer 720 may include a framework to support software 732 in software layer 730 and / or one or more applications 742 in application layer 740. Software 732 or applications 742 may include web-based service software or applications, such as those offered by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. Framework layer 720 may be a type of free and open source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter “Spark”), that may use distributed file system 738 for large-scale data processing (e.g., “big data”). In at least one embodiment, job scheduler 732 may include a Spark driver to facilitate scheduling of workloads supported by various tiers of data center 700. The configuration manager 734 may be capable of configuring different layers, for example, the software layer 730 and the framework layer 720 including Spark and distributed file system 738 to support large scale data processing. The resource manager 736 may be capable of managing the clustered or grouped computing resources that are mapped or allocated to support the distributed file system 738 and the job scheduler 732. In at least one embodiment, the clustered or grouped computing resources may include the computing resources 714 grouped in the data center infrastructure layer 710. The resource manager 1036 may coordinate with the resource orchestrator 712 to manage these mapped or allocated computing resources.

[0057] In at least one embodiment, software 732 included in software layer 730 may include software used by at least a portion of nodes CR 716(1)-716(N), grouped computational resources 714, and / or distributed file system 738 of framework layer 720. The one or more types of software may include, but are not limited to, Internet web page searching software, email virus scanning software, database software, and streaming video content software.

[0058] In at least one embodiment, the applications 742 included in the application layer 740 may include one or more types of applications used by at least a portion of the nodes CRs 716(1)-716(N), the grouped computational resources 714, and / or the distributed file system 738 of the framework layer 720. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0059] In at least one embodiment, any of configuration manager 734, resource manager 736, and resource orchestrator 712 can implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. The self-rewriting actions can free data center operators of data center 700 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming portions of the data center.

[0060] Data center 700 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculation of weight parameters via a neural network architecture using the software and / or computing resources described above with respect to data center 700. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 700, for example, by using the weight parameters calculated via one or more training techniques, including but not limited to those described herein.

[0061] In at least one embodiment, data center 700 may use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or corresponding virtual computing resources) for training and / or performing inference using the aforementioned resources. Additionally, one or more of the aforementioned software and / or hardware resources may be configured as services, such as image recognition, speech recognition, or other artificial intelligence services, to enable a user to train or perform inference on information.

[0062] Example Network Environment A network environment suitable for use in implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other back-end devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented with one or more instances of computing device 600 of FIG. 6 - e.g., each device may include similar components, features, and / or functionality of computing device 600. In addition, if a back-end device (e.g., server, NAS, etc.) is implemented, the back-end device may be included as part of a data center 700, an example of which is further detailed herein with respect to FIG.

[0063] Components of a network environment may communicate with each other via a network, which may be wired, wireless, or both. A network may include multiple networks, or a network of networks. Illustratively, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. When a network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (as well as other components) may provide wireless connectivity.

[0064] Compatible network environments may include one or more peer-to-peer network environments, where no server may be included in the network environment, and one or more client-server network environments, where one or more servers may be included in the network environment. In a peer-to-peer network environment, the functionality described herein with respect to a server may be implemented in any number of client devices.

[0065] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support software in the software layer and / or one or more applications in the application layer. The software or applications may include web-based service software or applications, respectively. In an embodiment, one or more of the client devices may use the web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open source software web application framework that may use a distributed file system, for example, for large-scale data processing (e.g., "big data").

[0066] A cloud-based network environment may provide cloud computing and / or cloud storage performing any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across a state, region, country, or world). When a connection to a user (e.g., a client device) is relatively close to an edge server, the core server may delegate at least a portion of the functionality to the edge server. A cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0067] A client device may include at least some of the components, features, and functionality of the exemplary computing device 600 described herein with respect to Figure 6. By way of example, and not limitation, a client device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a ship, an airship, a virtual machine, a drone, a robot, a handheld communications device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these depicted devices, or any other suitable device.

[0068] The present disclosure may be described in the general context of computer code or machine usable instructions, including computer executable instructions, such as program modules, being executed by a computer or other machine, e.g., a personal data assistant or other handheld device. Generally, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs particular tasks or implements particular abstract data types. The present disclosure may be implemented in a variety of system configurations, including handheld devices, consumer electronics, general purpose computers, more specialized computing devices, etc. The present disclosure may also be implemented in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0069] As used herein, the statement "and / or" with respect to two or more elements should be interpreted to mean only one element or combination of elements. For example, "element A, element B, and / or element C" may include element A only, element B only, element C only, elements A and element B, elements A and element C, elements B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Furthermore, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0070] The subject matter of the present disclosure has been described with the specificity set forth herein to meet legal requirements. However, the description itself is not intended to limit the scope of the present disclosure. Instead, it is contemplated by the inventors that the subject matter of the present claims may be implemented in other ways, including different steps or combinations of steps similar to those described in this document, in conjunction with other current or future technologies. Furthermore, although the terms "step" and / or "block" may be used herein to connote different elements of the method used, these terms should not be construed as implying any particular order between the various steps described herein, unless and when the order of the individual steps is expressly described.

Claims

1. determining, by a computing device, receipt of a data packet from an input device communicatively coupled to said computing device; obtaining, by the computing device, data from the data packet indicative of a latency value associated with an input received at the input device, the latency value corresponding to an amount of time between the input being received at the input device and input data representing the input being received by the computing device; A method comprising: the latency value is a first latency value, and the method further comprises: determining a second latency value corresponding to another amount of time between the input data being received by the computing device and a display reflecting the input data; calculating an end-to-end latency value based at least in part on the first latency value and the second latency value; The method further comprising:

2. The method of claim 1 , further comprising displaying data representative of at least one of the first latency value, the second latency value, or the end-to-end latency value.

3. 2. The method of claim 1, wherein obtaining the data comprises intercepting the data packet in response to determining the receipt of the data packet.

4. The method of claim 1 , wherein the computing device includes a display device, the display device including a pass-through functionality for a connection type of the input device.

5. 5. The method of claim 4, wherein obtaining the data comprises calculating the latency values ​​using a field programmable gate array (FPGA) of the display device.

6. 2. The method of claim 1, wherein each data packet received from the input device includes a data field corresponding to the latency value, and the data is obtained from an instance of the data field in the data packet.

7. 7. The method of claim 6, wherein during configuration of the input device, data is received representing a report descriptor corresponding to the input device, the report descriptor identifying a location of the data field within each data packet.

8. in response to receiving an input, generating a first timestamp corresponding to a time that the input was received; transmitting input data representing said input to a computing device; generating a second timestamp based at least in part on the transmission, the second timestamp corresponding to a time the input data was received by the computing device; calculating a peripheral latency value based at least in part on the first timestamp and the second timestamp; generating a data packet to include the peripheral latency value; transmitting the data packet to the computing device; A method comprising: The method, wherein the method is performed by a human interface device (HID) communicatively coupled to the computing device.

9. 9. The method of claim 8, wherein the step of calculating the peripheral latency value comprises determining a difference between the first timestamp and the second timestamp, the peripheral latency value being equal to the difference.

10. The method of claim 1, further comprising the steps of: transmitting data representing a report descriptor during configuration of an input device, the report descriptor including an indication that each data packet includes a data field designated for a peripheral latency value. The method of claim 8 , further comprising:

11. The method of claim 10 , wherein generating the data packet comprises adding the peripheral latency value to the data field.

12. 9. The method of claim 8, wherein the method is performed by a human interface device (HID) that is part of a computing system, and the peripheral latency value is used as one latency component of a plurality of latency components to determine an end-to-end latency of the computing system.

13. 1. A system comprising: one or more processors; one or more memory devices storing instructions that, when executed by the one or more processors, cause the one or more processors to: determining receipt of a data packet from an input device communicatively coupled to the computing device; obtaining data from the data packet indicative of a peripheral latency value associated with an input received at the input device, the peripheral latency value corresponding to an amount of time between the input being received at the input device and input data representing the input being received by the computing device; and determining an end-to-end latency value for the system based at least in part on the peripheral latency value; one or more memory devices, A system comprising:

14. Display devices, Control systems for autonomous or semi-autonomous machines; A system for performing a simulation operation; A system for performing deep learning operations, A system implemented using edge devices; Systems implemented using robots, A system incorporating one or more virtual machines (VMs); A system implemented at least in part in a data center; or The system of claim 13 , wherein the system is included in at least one of the systems implemented at least in part using cloud computing resources.

15. The operation, determining a latency value corresponding to another amount of time between the input data being received by the computing device and a display reflecting the input data; Further comprising: The system of claim 13 , wherein determining the end-to-end latency value is further based at least in part on the latency value.

16. 16. The system of claim 15, wherein the operations further comprise displaying data representative of at least one of the peripheral latency value, the latency value, or the end-to-end latency value.

17. 14. The system of claim 13, wherein obtaining the data comprises intercepting the data packet in response to determining the receipt of the data packet.

18. 14. The system of claim 13, wherein each data packet received from the input device includes a data field corresponding to the peripheral latency value, the data being obtained from an instance of the data field in the data packet.

Images