Augmented reality for internet connectivity installation

Abstract

To provide a device (for example, a system and device) and a method to view otherwise hidden connectivity of networking devices.SOLUTION: Electronic devices such as networking devices connected to a variety of different devices are observed through a real-time image in which information about connectivity and / or connection can be displayed, to show identities, connection states, and other information associated with ports of the networking devices. These methods can be implemented on a mobile device used to capture images of the networking devices and present an overlay of virtual objects on the captured images to a user in real time. The virtual objects can be dynamically moved or changed in shape in real time depending on movement of the mobile device. One or more filtering techniques can be used to stabilize the virtual objects with respect to the captured images.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This patent application claims priority to U.S. Provisional Patent Application No. 62 / 900,403, entitled “AUGMENTED REALITY FOR INTERNET CONNECTIVITY INSTALLATION,” filed September 13, 2019, which is incorporated by reference in its entirety.

[0002] Incorporation by Reference All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.

[0003] The present disclosure relates generally to augmented reality applications, and in particular to rendering augmented reality objects using images of network devices and associated components, including servers, switches, access points, routers, and any of many other network-related devices. [Background technology]

[0004] Computer network equipment generally includes hardware devices that enable communication and interaction between devices on a computer network. Examples of network devices generally include switches, routers, gateways, and wireless access points that mediate data transmissions in a computer network. When used in a business environment, network devices are typically used to connect various users' computers and office equipment, such as printers, to a local computer network and the Internet. Network devices are typically mounted in racks and organized in stacks with various other network devices. The racks may be housed in a temperature-controlled room separate from other racks of network devices. An information technology or technical professional will connect cables to ports of the network devices to various other devices within the rack. Sometimes, a network administrator needs to access one or more network devices to repair or troubleshoot a problem. Typically, multiple ports and cables look the same or similar to each other. Thus, when looking at the various cables and ports of network devices in a rack, it may be difficult to identify which ports are used to connect which network devices.

[0005] When used in the home setting, network devices are typically used to wirelessly connect various personal computers and other electronic devices to the Internet. Consumers often set up their own Internet connection by connecting various cables to the network devices, or in more complicated cases, hire a professional to properly connect the network devices. While user manuals can be helpful, this process can be confusing, especially for non-professionals. If the Internet connection is down, it can be difficult for a user to determine which devices are properly connected and working without time-consuming troubleshooting. Furthermore, today's advances and trends in home automation are expanding the interconnectivity in the home environment. For example, real-time analytics, machine learning, sensor technology, and embedded systems are converging to provide further monitoring and control of various home appliances and systems, providing a high level of home automation. For example, lighting, HVAC (heating, ventilation, and air conditioning), appliances, security systems, and other systems can be connected and centrally controlled, improving convenience, comfort, energy efficiency, and security (e.g., smart homes). As network connections increase and become more complex, the configuration and maintenance of such networks can become more complex.

[0006] In general, it would be beneficial to provide software, systems, and / or devices that enable users, whether in a business or home environment, to more easily configure, maintain, and troubleshoot network devices and associated components. Summary of the Invention

[0007] Described herein are augmented reality (AR) systems and methods, including software, that can be used to visualize network connectivity and other non-visible aspects of electronic devices. The systems and methods use virtual objects as aids in the installation, maintenance, management, and / or repair of various components of the network. The systems and methods can be used by technical support staff or information technology (IT) administrators in maintaining an organization's wireless and / or wired network systems. In some cases, the systems and methods are used by individual consumers of network devices while setting up or troubleshooting a home or home office network environment.

[0008] Any of the methods, systems, and devices described herein can be used in combination with a mobile device, such as a mobile phone, a tablet computer, a laptop computer, or a headset (e.g., a virtual reality headset). For example, a user can use the camera of the mobile device to capture one or more images (e.g., real-time images) of one or more network devices and view the images of the network devices on the display of the mobile device. In some cases, the method is encoded in an AR software application that is configured on the mobile device and / or otherwise accessible by the mobile device (e.g., via the Internet). In some cases, the software can be configured to operate in conjunction with other software (e.g., applications) accessible by the mobile device, such as commercially available AR software tools. Any AR software tool based on any operating system can be used. In some embodiments, the AR tools and features of the ARKit development platform developed by Apple Inc., headquartered in Cupertino, California, USA, are used.

[0009] Although augmented reality technology is known, the methods and systems described herein include functionality that allows a user to quickly and visually determine the connection status of a network device in real time, and possibly various other components of the network. A network device may include one or more identification codes, such as optical codes, RF codes, etc. An optical code may include a visible marking that can be detected and correlated to a network device (e.g., a switch), examples of optical codes include, but are not limited to, QR codes, alphanumeric codes, symbols, etc. An optical code may correspond to a two-dimensional (2D) pattern code (e.g., a matrix barcode) that includes encoded information associated with the network device. An optical code may be displayed on the network device, such as on the exterior surface of the network device, or other portion that is easily accessible to a user. In some embodiments, the optical code is on a changeable display, such as a touch screen display, that a user can change to access different optical codes or other information.

[0010] Alternatively or additionally, the identification code on the network device may be a radio-based identification code corresponding to the network device. In some examples, the network device may transmit a radio signal, such as a Bluetooth signal, or may interact with the mobile device by a radio frequency identification (RFID) code (tag) or a near field communication (NFC) code (tag). Radio-based identification may be used in combination with or in place of an optical code for identification of the network device. Those skilled in the art will appreciate that Bluetooth signals may have frequencies in the range of 2.402 gigahertz to 2.480 gigahertz. Radio frequency signals may have frequencies in the range of 20 kilohertz to 300 gigahertz.

[0011] The identification code may be uniquely associated with a particular network device. However, in some variations, the identification code may instead be associated with a particular subgroup of network devices, and the unique identity of the device may be determined based on secondary indicators, such as the identity of a scanning device (e.g., smartphone, tablet, etc.) that may be associated with a particular user. For example, the identification code may identify a particular category, class, or subset of network devices, and the unique identity of a particular device within this category, class, or subset of network devices may be uniquely identified by associated secondary information, such as the user identity of the user performing the scan, the scanning device (e.g., smartphone), one or more devices determined to be coupled or connected to the network device, etc.

[0012] The identification codes (e.g., optical and / or radio-based identification codes) can be used to retrieve real-time information related to the connection status of each of the ports of the network device. The identification codes (e.g., optical and / or radio-based identification codes) can also be used to access a library of virtual objects based on the connection status of the ports and the types of devices connected to the network. One or more virtual objects can then be combined (e.g., overlaid) with a captured camera image of the network device to provide a visual representation of the ports, the various devices connected to the network, and other information related to the various devices connected to the network.

[0013] In some embodiments, the virtual object includes an illustration of a communication port (a virtual port) aligned with a captured image of the corresponding port. For example, a user can use a camera of a mobile device to view a port of a network device, and the virtual port can be displayed over at least a portion of the image of the port. The position of the virtual port can be automatically and dynamically adjusted based on the movement of the mobile device. For example, as a user moves the mobile device to capture images of the network device from different perspectives, the position of the virtual port can be automatically adjusted to maintain its position over the image of the port, such that the user experiences a seamless transition as they move and view different parts of the network device. In some embodiments, the position of the virtual object is stabilized by removing one or more degrees of freedom used to determine the spatial relationship of the mobile device with respect to the port.

[0014] The virtual object may include one or more virtual labels with text, numbers, and / or symbols that convey information related to the port. For example, a virtual label on or near the image of the port may include an identification of the port (e.g., port number) and / or an identification of the device connected to the port. The virtual label may include text, numbers, or symbols and a line extending between the image of the port that indicates the relationship of the virtual label to the port.

[0015] The virtual object may include an illustration of one or more devices that are or have been previously connected to the port. The illustrations (icons) can provide the user with a quick visual reference as to which devices are associated with which ports. For example, the icons may be illustrations with enough detail for the user to determine whether the icon is a personal computer, a printer, a router, an access point, a router, or other device. In some cases, information about the connected device is used to illustrate a particular model or brand of the device. For example, the icon may be an illustration of a particular brand or model of a phone, television, laptop computer, headset, desktop computer, access point, media player, or tablet computer.

[0016] In some cases, a virtual object may have a different appearance depending on the connection state of the port. For example, a virtual port may have a different color or opacity when a device is connected to the port compared to when a device is not connected to the port. In some cases, an icon may have a different color or opacity when a device is connected to the port compared to when a device is not connected to the port. For example, an icon may have a grayed out or ghosted appearance when a device is not currently connected to the port but was connected to the port within a previous specified time.

[0017] According to some embodiments, the virtual object may convey information regarding whether a device connected to a port of the network device has other devices connected to it. For example, if the network device is connected to a wireless access point, the virtual label and / or illustration may indicate whether the access point device is connected to one or more other devices, such as a phone, laptop, tablet, or headset. The virtual object may also convey performance information of various devices connected to the network device, such as throughput and uptime. The virtual object may convey such data in the form of a virtual graph or chart.

[0018] For example, the connection information may also or alternatively include port statistics for both connected and / or unconnected ports on the network device. For example, the virtual display may include information regarding port statistics for all or a portion of the connected ports, and the user may toggle display / hide any (all or a subset) of the port statistics. The port statistics may include, for example, throughput information (e.g., current throughput, average throughput over the hour, day, week, etc., MAC address, IP address, etc.). In some variations, the port statistics may include historical information (e.g., indicating one or more connections made in the past, when connected / disconnected, etc.). In some variations, port statistics for ports that are not currently connected may be displayed. The port statistics may be displayed as text, icons, or some combination of both. This information may be displayed at the user's choice (e.g., select from a touch screen to expand or collapse additional information). Thus, the port connection information may be displayed as part of a virtual object, in addition to or separate from the virtual object.

[0019] These and other features and advantages of the AR methods, systems, and devices are described in detail herein.

[0020] Any of the methods described herein may be performed by an apparatus (e.g., a system, a device, etc.) configured to perform the method, including any optional steps described herein. For example, described herein is a system for displaying a network device with augmented reality using a mobile device. Such a system may include receiving an identification code from the network device by the mobile device, displaying a captured camera image of the network device on a display of the mobile device, the image including a plurality of ports of the network device, using the identification code to retrieve information related to a connection state of each of the plurality of ports, using the identification code to determine a spatial relationship of the mobile device to the one or more ports, and overlaying one or more virtual ports over the network device on the image, the virtual ports including information related to the connectivity of the network device within the one or more virtual ports.

[0021] The identification code may be an optical code (e.g., a barcode, a QR code, etc.) or an RF code (e.g., a Bluetooth code, NFC, etc.). Any of these methods may include accessing a library of virtual objects based on information related to the connection state of the port, and the virtual objects may include one or more icons corresponding to the types of devices connected to one or more ports of the network device (e.g., a camera, a phone, a computer, an access point, etc.). The virtual ports may include one or more icons. These icons, when displayed, may be used to indicate information about the connected devices.

[0022] For example, a method of displaying a network device with augmented reality using a mobile device may include displaying a captured camera image of the network device on a display of the mobile device, the image including one or more ports of the network device and an optical code on the network device including network connectivity information associated with the network device; using the optical code to retrieve information related to a connectivity status of the one or more ports; accessing a library of virtual objects based on the information related to the connectivity status of the one or more ports, the virtual objects including icons corresponding to types of devices connected to each of the one or more ports of the network device; using the optical code to determine a spatial relationship of the mobile device to the one or more ports; and overlaying one or more virtual ports, the virtual ports each including icons corresponding to types of devices connected to the one or more ports, over the image of the network device, a characteristic of the virtual port indicating a connectivity status of the port based on the retrieval of the information.

[0023] A system for displaying network devices with augmented reality using a mobile device may include one or more processors and a memory coupled to the one or more processors, the memory configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method including receiving, by the mobile device, an identification code from the network device; displaying a captured camera image of the network device on a display of the mobile device, the image including a plurality of ports of the network device; using the identification code to retrieve information related to a connection status of each of the plurality of ports; determining a spatial relationship of the mobile device to the one or more ports using the identification code; and overlaying one or more virtual ports over the network device on the image, the virtual ports including information regarding connectivity of the network device within the one or more virtual ports.

[0024] The one or more processors may be part of a mobile device (e.g., a smartphone, a tablet, etc.).

[0025] Any of these methods and devices for carrying them out can be configured to operate in real-time or near real-time.

[0026] For example, a method for displaying network devices with augmented reality using a mobile device includes: capturing a video image of the network device using the mobile communications device, the video image including a plurality of ports; receiving at the mobile communications device from the network device an identification code unique to the network device, determining from the identification code information regarding connectivity of the plurality of ports of the network device; and displaying an overlay on the plurality of ports of the network device in real time on the video image, the overlay including network connectivity indicators unique to each port, the network connectivity indicators including one or more of connection status, connection speed, data traffic, connection identity, connection duration, and Power over Ethernet (POE) usage.

[0027] A method for displaying a network device with augmented reality using a mobile device may include capturing a video image of the network device using a mobile communications device, the video image including a code unique to the network device, determining from the code information regarding connectivity of one or more ports of the network device, and displaying an overlay on the one or more ports of the network device in real time on the video image, the overlay including network connectivity indicators unique to each port, the network connectivity indicators including one or more of connection status, connection speed, data traffic, connection identity, connection duration, and Power over Ethernet (POE) usage.

[0028] The network device may be any suitable network device having one or more ports, such as, for example, a switch, a router, an access point, etc. The mobile communication device may include a smartphone or a tablet. The code may include a QR code, such as a digital QR code. The digital QR code may be updated or modified to optically transmit information to a handheld device (e.g., smartphone, tablet, etc.).

[0029] Any of these methods may include determining a location of each of one or more ports of the network device from the code and the image. For example, any of these methods may include determining a location of each of one or more ports of the network device from the code and the image, as well as an orientation of the mobile communications device.

[0030] The determining may include determining information regarding the connectivity of one or more ports of the network device by accessing a remote database using the mobile communication device and using a code that identifies information regarding the one or more ports. In any of these ways, the one or more ports can each be uniquely identified.

[0031] A method of displaying network devices with augmented reality using a mobile device may include capturing a video image of a plurality of network devices using the mobile communications device, each of the network devices comprising a plurality of ports and a unique identifier code; receiving at the mobile communications device from a network device of the plurality of network devices a unique identification code unique to each of the network devices; determining information regarding connectivity of one or more ports of each of the network devices from the unique identification code unique to each of the network devices; displaying an overlay on each of the one or more ports of the network devices in real time on the video image, the overlay including a network connectivity indicator unique to each port, the network connectivity indicator including one or more of a connection status, a connection speed, data traffic, a connection identity, a connection duration, and Power over Ethernet (POE) usage; and updating the display in real time as the network connectivity indicator changes.

[0032] For example, a method of displaying network devices with augmented reality using a mobile device may include capturing a video image of a plurality of network devices using the mobile communications device, the video image including a digital QR code unique to each of the network devices of the plurality of network devices; determining from the digital QR code information regarding connectivity of one or more ports of each of the network devices; displaying an overlay on each of the one or more ports of the network devices in real time on the video image, the overlay including network connectivity indicators unique to each port, the network connectivity indicators including one or more of connection status, connection speed, data traffic, connection identity, connection duration, and Power over Ethernet (POE) usage; and updating the display in real time as the network connectivity indicators change.

[0033] Also described herein is a system for displaying network devices with augmented reality using a mobile device. For example, the system may include one or more processors and a memory coupled to the one or more processors, the memory configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method, the computer-implemented method including: capturing a video image of a network device using a mobile communication device, the video image including a plurality of ports; receiving, at the mobile communication device, from the network device, an identification code unique to the network device, determining information regarding connectivity of the plurality of ports of the network device from the identification code; and displaying an overlay on the plurality of ports of the network device in real time on the video image, the overlay including a network connectivity indicator unique to each port, the network connectivity indicator including one or more of a connection status, a connection speed, a data traffic, a connection identity, a connection duration, and a power over Ethernet (POE) usage.

[0034] A system for displaying network devices with augmented reality using a mobile device may include one or more processors and a memory coupled to the one or more processors, the memory configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method including: capturing a video image of a plurality of network devices using a mobile communications device, each of the network devices comprising a plurality of ports and a unique identifier code; receiving at the mobile communications device from a network device of the plurality of network devices a unique identification code unique to each of the network devices; determining information regarding connectivity of the one or more ports of each of the network devices from the unique identification code unique to each of the network devices; displaying an overlay on each of the one or more ports of the network devices in real time on the video image, the overlay including a network connectivity indicator unique to each port, the network connectivity indicator including one or more of a connection status, a connection speed, a data traffic, a connection identity, a connection duration, and Power over Ethernet (POE) usage; and updating the display in real time as the network connectivity indicator changes.

[0035] The methods and apparatus described herein may be configured for use with multiple network devices (e.g., multiple switches, routers, etc.), each including multiple ports. For example, in some variations, the method may include receiving an identification code (e.g., a unique identification code) for each of the multiple network devices, such as a stack of devices. The identification codes may be all optical codes, all RF codes, or a combination of optical and RF codes. The methods and apparatus may allow a user to toggle the display of virtual ports for each of the multiple devices. For example, a user may move a mobile communication device up and down a stack of network devices to change the view that displays the virtual ports corresponding to each, alternatively or additionally, a user may select a control on the mobile communication device to toggle an augmented reality view between different network devices. In some variations, multiple different network devices may be displayed together at the same time. [Brief description of the drawings]

[0036] The novel features of the invention are set forth with particularity in the following claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings.

[0037] [Figure 1A-1B] FIG. 1 illustrates an example of a user using the devices and methods described herein. [Figure 2A] FIG. 1 illustrates an example of a stack of network devices having optical code. [Figure 2B] FIG. 1 illustrates an example of a display screen displaying one or more optical codes. [Figure 3A-3C] FIG. 2 illustrates an example of an optical code generated using a code generator. [Figure 4A-4F] 1 illustrates an exemplary user interface for setting up and using a mobile device in AR mode. [Figure 5A-5C] FIG. 13 illustrates another exemplary user interface for setting up a mobile device in AR mode. [Figure 6] FIG. 13 illustrates an example of an AR overlay for use in conjunction with a captured image to view a network device. [Figure 7A-7B] FIG. 13 illustrates another example of a user interface for using a mobile device in AR mode. [Figure 8] FIG. 1 illustrates an example of a user interface that includes various virtual objects related to the performance of one or more network devices. [Figure 9A-9E] FIG. 1 illustrates an embodiment of an example of a 3D coordinate system used to render an AR object. [Figure 10A-10D] FIG. 1 illustrates a flowchart and architecture map depicting an example process for setting up and using an AR application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] Generally, described herein are augmented reality (AR) systems, devices, and methods for viewing unseen features of electronic devices. In a particular application, the AR systems, devices, and methods are used to render virtual objects that convey information related to the network connectivity of the electronic devices. The virtual objects may provide a visual representation to a user regarding the network connectivity status of the network devices, thereby facilitating the installation and management of computer networks. A user can use a camera of a mobile device, such as a mobile phone or tablet computer, to capture a stream of images of the network devices in real time. The virtual objects can be rendered on a display of the mobile device along with the captured images to provide the user with an augmented reality experience.

[0039] 1A and 1B illustrate an example of a user 101 using the AR devices and methods described herein. The user 101 may be any person, such as a technical support person or IT administrator installing or managing a network at work, or an individual consumer installing or managing a home network. The user 101 may use the mobile device 100 to visually observe unseen aspects associated with the network device 102 and to install, troubleshoot, and / or check the status of the network device 102.

[0040] In general, the network device 102 may be any electronic device that can be wired and / or wirelessly connected to one or more computer networks. The computer network may include one or more networks that range in size anywhere from a wide area computer network to a nanoscale computer network. The computer network may be a local network, a wide area network such as a cloud-based network (e.g., the Internet), or a combination of local and wide area networks. For example, the computer network may be a local computer network that interconnects computers in a business, residence, or school, which may be connected to the Internet. In some cases, the network device 102 is configured to mediate data transmission in one or more computer networks. The network device 102 may be a network switch (also called a switching hub) that uses packet switching to connect devices on the computer network and receives, processes, and forwards data to a destination device. The network device 102 may be a gateway device that allows data flow between different networks. The network device 102 may be a router that forwards data packets between computers. Network device 102 may be a digital hub that receives information from various sensors and controllers (nodes) that may form a "smart" wirelessly connected habitable space (e.g., home, office, workplace, etc.) network. Network device 102 may be a network bridge that creates an aggregate network from multiple communication networks or network segments. Network device 102 may be a repeater that receives and retransmits signals. Network device 102 may be an Ethernet hub for connecting multiple Ethernet devices together. Network device 102 may be a hybrid network device such as a multi-layer switch, a protocol converter, and / or a bridge router (brouter).In some embodiments, the network device is a UniFi Dream Machine Pro (UDMP) or other network device manufactured by Ubiquiti Network, based in New York, NY, USA. The network device 102 may be a boundary network device that exists at the connection point of different networks, such as a proxy server, a firewall, or a network address translator (NAT). The network device 102 may be an end station device used to establish a network or dial-up connection, such as a network interface controller (NIC), a wireless network interface controller, a modem, an ISDN terminal adapter, or a line driver. The network device 102 may be a server that provides functionality to one or more clients. The network device 102 may be one of several network devices arranged in a stack 110 of network devices, in this case in a rack.

[0041] The devices and methods described herein use virtual objects to enable the user 101 to see unseen aspects of the network device 102, or of many network devices in the stack 110. To initiate the AR tool, the user 101 may use a scanning device, such as the mobile device 100, to scan the optical code 104 on the network device 102, as shown in FIG. 1A. For example, the user may use the camera of the mobile device 100 to capture one or more images of the optical code 104. The optical code 104 may be any machine-readable code that includes information about the network device 102. The optical code 104 may be a two-dimensional (2D) pattern that encodes information related to the network device 102. For example, the optical code 104 may be a quick response (QR) code, an ARTag code, a barcode (e.g., a linear barcode), an alphanumeric code, a numeric code, written characters, or any combination thereof. The optical code 104 may be on a surface of the network device 102, such as an exterior surface that is easily accessible to the user 101. The optical code 104 may be attached to a surface using any means. For example, the optical code 104 may be painted on the surface of the network device or on a sticker affixed to the surface of the network device. In some cases, the optical code 104 is generated by a display on the network device 102.

[0042] The optical code 104 may include one or more identifiers (e.g., unique identifiers) associated with the network device 102 to allow access to information associated with the network device 102. For example, when a user captures one or more images of the optical code 104, the mobile device 100 may transmit the identifier to a network management system (e.g., one or more remote servers). Upon receiving the identifier from the mobile device 100, the network management system may transmit the mobile device 100 and / or network device 102 information related to the network device 102. For example, the mobile device 100 may utilize a database of information related to the network device 102. The database may include a library of virtual objects based on the identity of the network device 102, the connection state of the network device 102, and the identity of any devices connected to the network device 102. The information may include information regarding the type (e.g., make and model) of the network device 102 and one or more communication interfaces, such as hardware or software interfaces. The information may include information related to one or more communication ports 108 of the network device 102. The ports 108 may be any type of interface between the device and other computers or peripheral devices. The port 108 may be an input or output interface of the device. The port 108 may include a hardware interface, including, but not limited to, a Universal Serial Bus (USB) port, a serial port, a video port, an audio port, a high-definition multimedia interface (HDMI) port, or a parallel port. In some cases, the communication interface is a wireless interface that wirelessly connects to one or more computers.

[0043] When the mobile device 100 receives information related to the network device 102, this information can be used to render virtual objects in real-time on the display of the mobile device 100 along with the captured camera image. In some embodiments, the virtual objects overlay the captured camera image. The virtual objects may include illustrations and / or text that convey information to the user 101 that is not normally readily apparent. The virtual objects may include virtual ports 108, cables 106, or other illustrations of parts of the network device 102. The virtual objects may include text, numbers, and labels that convey information about the identity and / or status of any devices connected to the network device 102 via the ports 108 and / or wireless communications.

[0044] The virtual objects may include icons corresponding to illustrations of various devices connected to the network device 102. In some cases, the icons may be displayed adjacent to a captured image of the ports 108, allowing the user 101 to quickly identify the types and connectivity of the various devices. The icons may depict any type of device connected to the network, such as a computer or other device (e.g., a sensor). For example, the computer may be a personal computer, such as a desktop computer or a mobile device (e.g., a laptop, tablet, phone, or headset). The computer may be a server computer configured to be shared by one or more users of the network. The computer may be a server or a client. The computer may be a router, such as a wireless router. The computer may be a modem. The computer may be a printer or other office environment computing device. The computer may be an access point device configured to allow Wi-Fi devices to connect to the network. The computer may be a smart home device, such as a thermostat, a smoke or carbon monoxide detector, a security system, a doorbell, a smart lock, a kitchen appliance, etc. The computer may be a camera, a speaker, or a microphone device. The computer may be a Bluetooth device.

[0045] In some embodiments, the mobile device 100 can dynamically display virtual objects in real-time. That is, the virtual objects may be rendered on the display of the mobile device 100 during the time, or close enough to the time, that images captured by the mobile device's camera are being rendered on the display, such that the user can experience a substantially continuous progression of images, such as a video with embedded virtual objects. As the camera of the mobile device 100 captures a stream of images over time, the virtual objects can be rendered on the display along with the stream of images. Thus, as the user 101 moves the mobile device 100 relative to the network device 102, such as when scanning various ports 108 of the network device 102, the virtual objects may dynamically update themselves based on the progression of the captured images. For example, the virtual objects may maintain their position on the display relative to the corresponding ports 108 as the mobile device 100 is moved. This spatial capability can be enabled using a spatial coordinate system, which can rely on one or more motion sensors of the mobile device 100 to detect the movement of the mobile device 100. For example, the mobile device 100 may include a motion sensing system that may include an accelerometer and / or a gyroscope.

[0046] The virtual objects may update information about the connection status of the various ports 108 in real time. In this manner, the user 101 may use the information provided by the virtual objects to troubleshoot the network device 102, for example, by connecting and disconnecting one or more cables 106 to and from the ports 108, or by turning on and off various devices connected to the network device 102. In some embodiments where data is transmitted in network packets (e.g., packet switching), the virtual objects may enable the user 101 to perform feed packet inspection for security control. The user 101 may also check the connectivity status of each of the ports 108 in real time to determine, for example, whether the network device 102 is properly connected to the various devices in the network. The virtual objects may convey information about the status of the various devices in the network, including secondary, tertiary, etc. devices connected to those devices. Other network devices 102 in the stack 110 may be similarly visible by AR, such that the user 101 may scan all network devices in the stack 110. In this manner, the AR tools described herein can enable a user 101 to quickly and easily determine the status of an entire network system, including its subsystems.

[0047] FIG. 2A illustrates an example of network devices 202a-202f having different optical codes 204a-204f that encode information related to the corresponding network devices 202a-202f. As illustrated, the optical codes 204a-204f can have any of a number of different patterns. For example, the optical codes 204a-204f can be a Quick Response (QR) code, an ARTag code, a barcode (e.g., a linear barcode), an alphanumeric code, a numeric code, written characters, or any combination thereof. The different patterns can encode different types of information based on the identity and status of the corresponding network device. In some cases, the optical codes 204a-204f are displayed on a display (e.g., a flat panel display) of the network devices 202a-202f. FIG. 2B illustrates an example of a display component that can be configured to display one or more optical codes. The display can be of any type, such as a liquid crystal display (LCD) and / or a light emitting diode (LED) display (e.g., an organic light emitting diode (OLED) display). In some embodiments, the display is a touchscreen display that can display different images based on touch input For example, the touchscreen display can be configured to change what is displayed in response to a user swiping the touchscreen.

[0048] In some cases, a network device may not have an LCD or other type of visually active screen, but instead may use a static optical code associated with the network device (e.g., a sticker or other printed identifier, etc.).

[0049] As mentioned above, the identification code may be a radio frequency (e.g., RF) identification code, such as a radio-based ID (e.g., Bluetooth, NFC, RFID) code, that may be used by the handheld device to identify (e.g., uniquely identify) the network device. The RF identification code may be dynamic (similar to an optical code on a screen) or it may be static (e.g., a printed optical code attached to the exterior of the network device).

[0050] 3A-3C show three different examples of optical codes that include different code patterns. These examples show how the different patterns can encode unique information based on the network device, including the connection state of the network device. Information from the optical code can then be sent, for example, to a cloud-based management system that transmits the mobile device and / or network device information associated with the network device and is used to generate virtual objects. In some examples, the optical code is generated using a pattern generator program that generates a pattern (e.g., randomly) to provide a unique optical code.

[0051] 4A-4F show examples of user interfaces for setting up and using a mobile device 400 in AR mode, according to some embodiments. FIG. 4A shows a user using the camera of the mobile device 400 to view multiple network devices 402, each with a corresponding optical code (e.g., 404). The mobile device 400 can be considered an example of a mobile device 100. The user interface 420 can be displayed on a display screen of the mobile device and can guide the user through several interface screens for initial setup for the mobile device 400 to enter the AR mode. The user interface 420 can include a geographic location of the network device, a type of network device (e.g., make and model), and / or an illustration of the network device. The user interface 420 can video record an image of the network device 402 captured by the camera of the mobile device 400. The user interface 420 can include a button for entering the AR mode.

[0052] 4B1 and 4B2 show two views of an exemplary user interface 421 prior to scanning the optical codes of the network devices. The user interface 421 may prompt the user to scan each of the network devices so that information about each of the network devices can be accessed and loaded for use in the AR mode. The user interface 421 may include an optical code alignment frame 422 for aligning a captured image of each network device's optical code (e.g., 404).

[0053] 4C1-4C4 show four views of an exemplary user interface 422 during scanning of the optical codes of multiple network devices of a stack ("rack"). After scanning the first optical code of a first network device, the user interface may prompt the user to scan the second optical code of a second network device. A visual indicator may be used to indicate locking of the optical code in the optical code alignment frame, such as presenting the optical code alignment frame 422 in a different color or shade. Alternatively or additionally, a different visual indicator, such as a checkmark symbol 427, may become visible to indicate that the optical code has been successfully scanned. In some embodiments, a tactile and / or audible indicator, such as a vibration and / or sound generated by the mobile device, may be used to indicate successful locking and / or scanning of the optical code. The user interface may include a list 424 or table of network devices that is progressively updated as each of the optical codes of the network devices is successfully scanned. For example, an icon 425 depicting a corresponding network device may be populated in the list 424 after the optical code of the network device is successfully scanned and / or recognized (e.g., by a network management system). The list 424 may subsequently populate with additional icons 426 after the optical codes of additional network devices are successfully scanned and / or recognized (e.g., by a network management system). This scanning of different optical codes of different network devices may continue, for example, until all network devices in the stack have been scanned.

[0054] 4D1-4D4 show four views of an exemplary user interface 430 illustrating how AR settings may be applied after the optical code of a network device is successfully scanned. According to one aspect, the user interface 430, in addition to showing a list 432 of added scanned network devices, may show a prompt 433 (e.g., a button) for the user to respond whether to scan more network devices. The user interface 430 may provide a prompt 434 (e.g., a button) for the user to proceed to the next display screen of the user interface. After all desired network devices are scanned, the user interface may provide a prompt 435 (e.g., a button) for the user to optionally enter a name for a stack (e.g., a "rack") of network devices, and may provide a prompt 436 (e.g., a button) for the user to proceed to the next user interface display screen. Any display screen of the user interface 430 may include a back prompt 438 to return to the previous display screen. The AR settings of the network device may then be applied to the network stack based on the information received by the optical code and the information provided by the user. After the settings are applied and completed, the complete AR experience can now be loaded, allowing the mobile device to utilize the camera in AR mode on the newly created stack of network devices.

[0055] 4E illustrates an example of a mobile device immediately after entering AR mode. As shown, the display of the mobile device 400 can display a user interface 450 that can include one or more virtual objects 401 that overlay a captured image of a network device, thereby conveying information related to the network device that is not normally visible to the user.

[0056] FIG. 4F shows a close-up view of the user interface 450 in AR mode, according to some embodiments. The virtual objects may include text (e.g., numbers, letters, and / or symbols) and / or illustrations. For example, a virtual identification text 452 of the network device, such as the name and / or Internet Protocol (IP) address of the network device, may be positioned adjacent to or on (e.g., partially on) the image of the network device. The virtual objects may include a port area 454 that at least partially frames various virtual objects associated with a particular network device. The virtual objects may include an optical code alignment frame 456 for aligning a captured image of the optical code. The optical code alignment frame 456 may be used to assist a user in aligning the optical code within a lock area A of the optical code alignment frame 456 for locking the optical code. The lock area A may be in a central area of ​​the optical code alignment frame 456. In some embodiments, the port area 454 and / or the optical code alignment frame 456 may have a transparent or semi-transparent quality that allows the underlying image of the network device to be seen. In some cases, virtual port B overlays a corresponding port of the network device. Virtual port B may have the shape and size of the corresponding physical port. Virtual port B may include a number that corresponds to the port number as identified by the manufacturer of the network device, for example. Virtual port B may be filled with color C and / or may be at least partially transparent to allow the underlying image of the network device to be visible. In some cases, the color and / or translucent nature may change depending on the state of the corresponding port.For example, a virtual port B that is actively connected to a computer may be filled with a first (e.g., dark) color and / or a translucent characteristic, while a virtual port B that is not actively connected to a computer may be hollow (i.e., transparent) or filled with a second (e.g., light) color and / or a translucent characteristic.

[0057] The virtual objects may include icons F (also called virtual devices or virtual computers) that match the devices connected to the corresponding ports. The icons F may be illustrations with sufficient detail to allow the user to identify the type of device (e.g., phone, tablet, laptop, desktop, television, access point, virtual assistant device, home appliance, or security system device), allowing the user to easily and quickly distinguish different types of computers from each other. The icons F may be selected from a database (also called a library) of illustrations of virtual objects that are automatically assigned or selected by the user. The relative position of the computer icon F may be contained within the icon area E.

[0058] The virtual object may include a virtual device identification text H that describes the identity of the computer. The virtual device identification text H may include information that helps to further identify a device that is connected or was previously connected to the corresponding port. For example, the user's name and / or the make and model of the device may be identified in the virtual device identification text H. The virtual device identification text H may be automatically registered based on information provided by, for example, a cloud-based network management system or selected by the user. The relative location of the virtual device identification text H may be contained within the device identification text area G.

[0059] In some embodiments, the icon F and / or the virtual device identification text H may have different characteristics based on whether a device is actively connected to the corresponding port. For example, the icon F and / or the virtual device identification text H may have a fully colored or opaque appearance if the device is actively connected to the port, a ghosted or grayed appearance if the device is not currently connected but was previously connected to the port within the predetermined period, and is not displayed (i.e., no illustration or text in areas E and G) if the device is not required to be connected to the port within the predetermined period. The predetermined period may vary. For example, the predetermined period may be approximately 30 minutes, 1 hour, 2 hours, 3 hours, 6 hours, 12 hours, 24 hours, 3 days, or 1 week or less. In some embodiments, the predetermined period may be selected by the user. In some cases, the icon F and / or the virtual device identification text H includes information regarding the operational status of the device. The operational status may include performance characteristics (e.g., real-time or historical) of the device.

[0060] In some cases, the icon F and / or the virtual device identification text H may include an indication as to whether the device is connected to another device (e.g., a secondary, tertiary, etc. device). For example, an access point device may be operatively connected to one or more phones, tablets, laptops, and / or headsets. The icon F may have a different appearance indicating connectivity to one or more additional devices, such as being presented in a different color, surrounded by a halo or shadow, or given other attributes. Alternatively or additionally, the virtual device identification text H may include text indicating connectivity to one or more additional devices. In some cases, the icon F and / or the virtual device identification text H includes information regarding the operational status of one or more additional devices.

[0061] Aspects of the user interface may vary depending on, for example, the display dimensions, form factor, and / or operating system of the mobile device. For example, a mobile phone may be narrower in height or width than a tablet computer. The user interface may be configured to address these differences.

[0062] 5A-5C show other exemplary user interfaces for setting up a mobile device in AR mode. FIG. 5A shows a user interface 521 before scanning an optical code similar to user interface 421 (FIGS. 4B1-4B2), but the user interface 521 includes a list 523, which may be registered network devices and positioned laterally relative to the optical code alignment frame 522. FIG. 5B shows a user interface 523 during scanning an optical code similar to user interface 423 (FIGS. 4C1-4C4), but the user interface 523 includes a list 526, which may be registered network devices and positioned laterally relative to the optical code alignment frame 522. FIG. 5C shows a user interface 530 for applying AR settings similar to user interface 430 (FIGS. 4D1-4D4), but a prompt 535 (e.g., a button) for the user to select to complete the AR settings is presented on the bottom side (e.g., right side) of the user interface 530.

[0063] As described herein, the optical code may include encoded information that provides access to a library of virtual objects. The optical code may act as a reference marker to inform the appearance and placement of the virtual object. The relative location of the virtual object may be placed in an overlay that overlays an image captured by, for example, a mobile device's camera. FIG. 6 illustrates an example of an overlay 600 according to some embodiments. At least a portion of the overlay 600 may be substantially transparent so that the underlying image can be seen when rendered on a mobile device's display. The optical code alignment frame 630 may define an area of ​​the overlay 600 that is configured to scan and receive an image of the optical code. In some cases, the optical code alignment frame 630 may be outlined with a border or frame and / or have a central marker (e.g., a circle). In other cases, the optical code alignment frame 630 may not have a border or frame.

[0064] Once the optical code is scanned and verified by a network management system (e.g., local or cloud-based) as being associated with a particular network device, a library of virtual objects associated with the network device may be accessed and the library may be viewed on a mobile device display. For example, the optical code may inform a virtual object administrator of the make and model of the network device and any associated physical characteristics such as port type, location, size, etc. This information may be used to create one or more virtual ports 634 from a database of various types of virtual ports that correspond to the real ports of the accessed network device. Additionally, the network management system may access information related to the connectivity status of the ports. This information may be used to determine various aspects of the virtual object that convey information related to the connectivity status of one or more ports 634.

[0065] Overlay 600 may include a port region 632 that defines an area of ​​overlay 600 that includes virtual ports 634. Port region 632 may or may not include a visible outline or frame. In some cases, port region 632 is a different color and / or semi-transparent compared to other portions of overlay 600. Virtual ports 634 may include text, such as a number to identify the number of the port. In some cases, virtual ports 634 include one or more status indicators (e.g., 638 and 639) that indicate a connection state of the port. The status indicators may have a different appearance (e.g., different color or shape) based on whether the port is actively connected to a computer, has previously been connected to a computer within a predetermined period of time, or has not been connected to a computer within a predetermined period of time. Such status indicators (e.g., 638 and 639) may be used in addition to or as an alternative to the filled in trait described above with reference to FIG. 4E.

[0066] AR can enable video tracking capabilities that calculate the position and orientation of the camera in real time relative to a physical marker, such as an optical code on a network device. For example, once the camera's location is known, a virtual camera can be positioned at the same point, revealing a virtual object (e.g., optical code alignment frame 630) at the location of the optical code. Information from the network management system about a particular network device can be used to provide precise placement and alignment of the virtual ports 634 relative to the optical code. This information can also be used to determine the relative distance between the virtual ports 634, as well as the shape and size of the virtual ports 634. Information from the network management system can further be used to change aspects of the virtual object in real time based on the connection status of various ports, as described herein.

[0067] The spatial relationship between the mobile device and the network device may change in real time, as the user may move the mobile device relative to the network device, thereby projecting an image of the network device at different depths. To address the changes in depth, the generation of virtual objects typically relies on tracking movement based on rotational degrees of freedom around fixed orthogonal axes (x, y, z axes), similar to a gimbal system. Movement can be tracked using traditional three-dimensional rotational motions, such as roll (rotation around the x axis), pitch (rotation around the y axis), and yaw (rotation around the z axis), which define six degrees of freedom. While this model is useful, the data associated with continuously monitoring and detecting movement in real time can be tangential, which can result in delays before the virtual object is generated on the display.

[0068] In any of the embodiments described herein, the spatial relationship can be simplified by eliminating one or more degrees of freedom. For example, the position of the network device is substantially stable, and the ports on the network device are generally oriented along a plane (e.g., with the back of the network device perpendicular to the floor). Thus, one of the three degrees of freedom is considered fixed and can be eliminated from the calculations for determining the spatial relationship between the mobile device and the network device. In some embodiments, this is accomplished by excluding (e.g., ignoring) a portion of the raw data related to the three-dimensional movement of the mobile device that is not necessary to calculate the relative movement of the mobile device with respect to the port. For example, in some embodiments, the movement related to roll (rotation around the x-axis) can be eliminated from the calculation, thereby reducing the degrees of freedom and information required for the calculation by one-third. Filtering the data in this manner can speed up the calculations, thereby allowing the virtual objects (e.g., virtual ports) on the captured image to be updated more quickly. In addition, this can reduce the amount of corrections required to update each image frame, resulting in the virtual objects having a more stable (e.g., less wobbly) location and appearance in real time.

[0069] Additionally, the AR functionality may use optical feature recognition to determine the location of the virtual port 634. In some examples, the optical feature recognition may be used in conjunction with or in lieu of the spatial relationship between the mobile device and the network device. In some examples, the optical feature recognition may include optical character recognition (OCR).

[0070] More specifically, OCR can be used to allow the camera to identify any relevant text on the hardware (e.g., port numbers, actual ports, etc.) to aid in the placement of the AR feature in the overlay. This can mitigate sensor drift or other alignment issues that may occur with the overlay of dead reckoning or other motion-based features. Additionally, OCR can be used to view and verify the specific identity of a network device before locating a virtual port.

[0071] For example, and referring briefly back to Figure 1, if the network device 102 is a 24 point network switch, the text on the label of the network device 102 can be captured via OCR and identified as a 24 point switch. The mobile device 100 can then display the identifying information, i.e., "This is a 24 point switch," and can then look up the virtual ports of the network device with an AR overlay, as previously described.

[0072] In some embodiments, after OCR is performed, the mobile device 100 may display details of the expected configuration of the network device. For example, the mobile device 100 may display a list of ports to which the network device is expected to be connected and how they should be connected by flashing a message (title) "Expected" or "Expected Configuration" before starting to look for the ports. In various embodiments, the above steps may be performed sequentially or in parallel.

[0073] 7A and 7B show another example of a user interface 720 of a mobile device 700 in an AR mode, according to some embodiments. FIGS. 7A and 7B show an image of a network device 720 rendered on a display of the mobile device 700 in real time in two different perspective views. The user interface 720 may include one or more virtual objects overlaying a portion of the displayed image of the network device 702. The virtual objects may include virtual ports 740 and 741, virtual cables 742, icons 750, and virtual labels 752 (e.g., text). In some cases, a port area 732 surrounding a virtual port is outlined with a virtual line 745. The position of the virtual object may be configured to change according to a change in depth by the mobile device using a spatial coordinate model described herein. In this manner, the virtual object may substantially maintain alignment with a corresponding object in the captured image. Thus, the virtual ports 740 and 741, the virtual cable 742, the icon 750, and the virtual label 752 may appear to move when the mobile device 700 is moved relative to the network device. One or more virtual objects may change based on the connection state of the corresponding one or more ports. For example, the virtual port 740 may have an outline of a first color (e.g., white) to indicate that the corresponding port is connected to a device, and the virtual port 741 may have an outline of a second color (e.g., blue) to indicate that the corresponding port is not connected to a device.

[0074] Any of the AR devices and methods described herein may include virtual objects that indicate the performance of the network device and / or devices connected to the network device. FIG. 8 shows a close-up view of a user interface of a mobile device display while in AR mode, illustrating examples of virtual objects conveying performance data. The virtual objects may include various selectable performance metrics, such as throughput 862 and uptime 864 of the network device. Once a selectable performance metric is selected, one or more virtual charts or graphs 860 may be displayed. In FIG. 8, graph 860 illustrates throughput values ​​associated with devices connected to the network device. Throughput metrics may include data related to the throughput performance of the devices, such as Transmission Control Protocol (TCP) throughput or file transfer time. Uptime metrics may include the percentage of time the device is operational. Other metrics may include Internet traffic usage and / or power usage of the network device and / or devices connected to the network device. In some cases, the metrics may include the length of time one or more devices have been connected to the network device. In some embodiments, a tree graph showing various (e.g., primary, secondary, tertiary, etc.) devices that are connected or have been connected to the network device. In some cases, the metrics may include usage data related to a particular user of a device. For example, a graph or chart can be displayed showing the time a user used a social media application or website.

[0075] The user interface may include selectable virtual icons that a user may select by touch (e.g., using a touch display of a mobile device) and / or by other selectable methods such as an electronic pencil. The virtual icons may include a port icon 870, a power icon 872, a performance icon 874, and a settings icon 876. When the port icon 870 is selected, various virtual ports and virtual port labels may become visible, as described herein. When the power icon 872 is selected, virtual objects related to power and battery status and usage may become visible. When the performance icon 874 is selected, virtual objects related to performance, such as throughput 862 and continuous uptime 864, may become visible. When the settings icon 876 is selected, virtual objects related to network setup and configuration may become visible.

[0076] As described herein, AR techniques can be based on using a three-dimensional coordinate system (3D XYZ). Figures 9A-9E show aspects of a 3D XYZ axis system used as a basis for the AR methods described herein according to some embodiments, where X represents width, Y represents height, and Z represents depth. The 3D XYZ system can ensure that the spatial data used to form the AR object is understandable (e.g., consistent). In some embodiments, a cross-platform software engine is used to establish the 3D XYZ system. For example, the Unity3D game engine developed by Unity Technologies, based in San Francisco, California, USA, can be used. A point positioned in 3D with width 1 (on the X axis), height 4 (on the Y axis), and depth 3 (on the Z axis) can be presented as Vector3{x:1, y:4, z;3} using the Unity3D system.

[0077] The 3D spatial record can be captured during scanning of the optical code (e.g., see Figs. 4C1-4C4 or Fig. 5B). For example, an AR application on a mobile device (e.g., phone, tablet, or headset) can use the mobile device's camera and / or orientation system (e.g., accelerometer, gyroscope, and / or compass) to "capture" a 3D spatial record of the network devices and "remember" the user's rack configuration. This 3D spatial capture can be a one-time setup. Fig. 9B shows an example of a 3D spatial record of a rack 910 of network devices 902. The spatial record of each of the network devices 902 can be established as a set of vectors 909 in a 3D XYZ coordinate system. The 3D spatial record corresponds to a group of normalized positions (e.g., Vector3) that preserves its local position based on the tracker. The 3D spatial record can be filtered as described herein. In one implementation of filtering, the orientation of the network device is typically fixed (e.g., has a bottom surface parallel to the ground) and therefore the local location can be considered to be on a "panel" (e.g., a 2D surface) such that there is substantially no rotation about the X-axis. One of the trackers can be designated as the origin of the spatial recording, and the relative locations of the other trackers can be recorded relative to this origin. For example, the first optical code 904a (e.g., scanned first) can be designated as having the origin, and the relative locations of the remaining optical codes 904 (e.g., scanned subsequently) can be recorded relative to the first optical code 904a. In some cases, the top optical code 904a is designated as the origin. The coordinate system can be based on any unit of measurement. In some embodiments, the coordinate system is based on the metric system (e.g., meters). Note that in some embodiments, Vector1 or Vector2 structures can be used instead of Vector3. However, in some cases, Vector3 may provide more functionality to accommodate more complex network device layouts.

[0078] Data can flow between the 3D XYZ Platform and the mobile device's native code, such as Swift or Java. FIG. 9C shows an example of JavaScript Object Notation (JSON) code for setting up a network device. FIG. 9D shows an example using a HyperText Transfer Protocol (HTTP) request made in native code to provide additional data (rack_name for racks, name and model number for network devices) to the user interface during the spatial recording capture process. FIG. 9E shows an example JSON code for setting up multiple network devices.

[0079] 10A-10D show a flowchart 1000 and corresponding architecture map 1100 depicting an example process for setting up and using an AR application, according to some embodiments. Each of the steps of the flowchart 1000 includes a representation of a corresponding architecture component (e.g., Swift, Unity AR, network controller, or USW / LCM) in the corresponding map 1100. The map 1100 shows the interaction of the network controller, Swift, Unity AR, USW, and LCM components.

[0080] FIG. 10A illustrates a portion of a flowchart 1000 and a map 1100 corresponding to initializing an application with network information. Referring to the flowchart 1000, at 1002, a user can use a mobile device to pull a network application programming interface (API) for an AR application from a network controller to a Swift component. At 1004, the network JSON is filtered, for example, with one or more filtering methods described herein. At 1006, the network JSON data is parsed by the Unity AR component. At 1008, the spatial record is pulled by the Swift component. At 1010, if the spatial record was pulled successfully (yes), the process continues at FIG. 10D, and if the spatial record was not pulled successfully (no), the process continues at FIG. 10B.

[0081] FIG 10B illustrates a portion of the flowchart 1000 and map 1100 corresponding to the distribution of tracker seeds. Referring to the flowchart 1000, if it is determined in FIG 10A that the spatial record was not successfully pulled (No), at 1012, a Mac-tracker pair is created by the Unity AR component and sent to the Swift component. At 1014, the Mac-tracker pair is posted by the Swift component to the network controller. At 1016, the Mac-tracker pair is distributed to the LCM and USW, and at 1018, trackers are generated from the seeds (e.g., to generate QR codes).

[0082] 10C shows a portion of a flowchart 1000 and a map 1100 corresponding to a setup (e.g., one-time setup) for storing spatial records. With reference to the flowchart 1000, at 1020, a tracker is scanned and used to write a spatial record and sent to the Swift component. At 1022, the spatial record is input to a network controller, and at 1024, the spatial record is received and stored by the network controller.

[0083] 10D shows a portion of the flowchart 1000 and map 1100 corresponding to resuming normal use of the mobile device and using an AR application based on the spatial record. Referring to the flowchart 1000, at 1026, the spatial record is pulled by the Swift component so that the trackers on the LCM from the spatial record (USW / LCM) can be resumed. Further, the network controller responds to the spatial record at 1030. At 1032, the spatial record is parsed by the Unity AR component, which can scan any of the displayed trackers at 1034. At 1036, the AR overlay of the rack is resumed in the Unity AR component, which builds an AR user interface (1038) with automatic toggle interaction (1040) and renders the port details (1042).

[0084] Although many of the examples described herein have involved network devices such as switches, routers, and wireless access points, the AR technology described herein is not limited to these types of devices. For example, the AR technology described herein may be used to view any type of computer or other device that can connect to one or more computer networks. In some cases, the AR technology described herein is used to render invisible connectivity aspects of a mobile device such as a mobile phone, tablet, laptop, or headset. The AR technology described herein can be used to render invisible connectivity aspects (i.e., not limited to hardware ports) of wireless communication capabilities. For example, connectivity aspects related to one or more antennas (e.g., radio frequency or Bluetooth chips) can be rendered using AR objects.

[0085] In this specification, when a feature or element is referred to as being "on" another feature or element, it may be directly on the other feature or element, or there may be intervening features and / or elements. In contrast, when a feature or element is referred to as being "directly on" another feature or element, there are no intervening features or elements. It will also be understood that when a feature or element is referred to as being "connected," "attached," or "coupled" to another feature or element, it may be directly connected, attached, or coupled to the other feature or element, or there may be intervening features or elements. In contrast, when a feature or element is referred to as being "directly connected," "directly attached," or "directly coupled" to another feature or element, there are no intervening features or elements. Features and elements so described or illustrated are described or illustrated with respect to one embodiment, but may be applicable to other embodiments. It will also be understood by those skilled in the art that references to structures or features disposed "adjacent" to another feature may have portions that overlap or underlie the adjacent feature.

[0086] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the invention. For example, as used herein, the singular forms "a", "an" and "the" are intended to include the plural unless the context clearly indicates otherwise. It will be further understood that as used herein, the terms "comprises" and / or "comprising" specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and can be abbreviated as " / ".

[0087] Spatially relative terms such as "under," "below," "lower," "over," "upper," and the like may be used herein for ease of description to describe the relationship of one element or feature to another element or feature as depicted in the drawings. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the drawings. For example, if the device in the drawings is inverted, an element described as "under" or "beneath" the other element or feature would then be oriented "over" the other element or feature. Thus, the exemplary term "under" can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or to other orientations) and the spatially relative descriptors used herein may be interpreted accordingly. Similarly, terms such as "upward," "downward," "vertical," "horizontal," and the like are used herein for descriptive purposes only, unless otherwise indicated.

[0088] The terms "first" and "second" may be used herein to describe various features / elements (including steps), but these features / elements should not be limited by these terms unless the context indicates otherwise. These terms may be used to distinguish one feature / element from another. Thus, a first feature / element discussed below may be referred to as a second feature / element, and similarly, a second feature / element discussed below may be referred to as a first feature / element without departing from the teachings of the present invention.

[0089] As used herein, including in the examples, and in the specification and claims, unless otherwise specified, all numbers may be read as if preceded by the words "about" or "approximately", even if the term is not explicitly stated. The phrases "about" or "approximately" may be used when describing a size and / or location to indicate that the described value and / or location is within a reasonable expected range of values ​​and / or locations. For example, a numerical value may have values ​​of + / -0.1% of the stated value (or range of values), + / -1% of the stated value (or range of values), + / -2% of the stated value (or range of values), + / -5% of the stated value (or range of values), + / -10% of the stated value (or range of values), etc. Any numerical range recited herein is intended to include all subranges incorporated therein.

[0090] Although various exemplary embodiments have been described above, any of a number of modifications may be made to the various embodiments without departing from the scope of the invention as described by the claims. For example, the order in which the various method steps described are performed may often be changed in alternative embodiments, and in other alternative embodiments, one or more method steps may be skipped entirely. Optional features of the various device and system embodiments may be included in some embodiments and not included in other embodiments. Thus, the foregoing description has been provided primarily for illustrative purposes and should not be construed to limit the scope of the invention as described in the claims.

[0091] The examples and illustrations contained herein show, by way of example and not limitation, specific embodiments in which the subject matter may be practiced. As mentioned, other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Such embodiments of the subject matter of the present invention, when one or more are in fact disclosed, may be individually or collectively referred to herein by the term "invention" for convenience only, without intending to autonomously limit the scope of the present application to any single invention or inventive concept. Thus, although specific embodiments have been illustrated and described herein, any arrangement made to achieve the same purpose may be substituted for the specific embodiment shown. The present disclosure is intended to cover any and all adaptations or variations of the various embodiments. Combinations of the above embodiments, as well as other embodiments not specifically described herein, will be apparent to those skilled in the art upon reviewing the above description.

Claims

1. 1. A method for displaying network devices with augmented reality using a mobile device, comprising: receiving, by the mobile device, an identification code from the network device; displaying a captured camera image of the network device on a display of the mobile device, the image including a plurality of ports of the network device; using said identification code to retrieve information relating to a connection status of each of said plurality of ports; determining a spatial relationship of the mobile device to the one or more ports using the identification code; and overlaying one or more virtual ports over the network devices on the image, the virtual ports including within the one or more virtual ports information regarding connectivity of the network devices.

2. The method of claim 1 , wherein an alignment of the one or more virtual ports with respect to the camera image is dynamically adjusted based on the determined spatial relationship.

3. The method of claim 1 , further comprising: each of said virtual ports indicating said connection state of said port based on said retrieved information.

4. 2. The method of claim 1, further comprising overlaying a virtual label on the image proximate each of the virtual ports, the virtual label comprising text indicating one or more characteristics of a device connected to each of the ports corresponding to each of the virtual ports.

5. The method of claim 1 , wherein each of the virtual ports has a size and shape that substantially matches that of the image of the one or more ports.

6. The method of claim 1 , wherein the location of the virtual port changes dynamically according to changes in a depth of the camera of the mobile device relative to the network device.

7. 2. The method of claim 1, wherein each of the virtual ports has a different appearance when a corresponding port of the network device is connected compared to when the corresponding port is not connected to a device.

8. 2. The method of claim 1, further comprising: accessing a library of virtual objects based on the information related to a connection state of the ports, the virtual objects including one or more icons corresponding to types of devices connected to one or more ports of the network device, and the virtual ports including the one or more icons.

9. 9. The method of claim 8, wherein the type of device is a mobile phone, a tablet computer, a laptop computer, a desktop computer, a wireless access point device, a virtual assistant device, a television, a home appliance, or a security system device.

10. 9. The method of claim 8, wherein the one or more icons have a different appearance when the port is connected to the device compared to when the port was previously connected to the device and is no longer connected.

11. 10. The method of claim 1, further comprising stabilizing the one or more virtual ports for the network device by excluding one or more degrees of freedom used to determine the spatial relationship of the mobile device to the network device.

12. The method of claim 11 , wherein excluding one or more degrees of freedom includes excluding data associated with rotation about an axis in a coordinate space.

13. The method of claim 1 , wherein the identification code is an optical code on a touch display of the network device.

14. The method of claim 1 , wherein the identification code is an RF code.

15. The method of claim 1 , wherein the connection state of the port includes whether the port is connected to the device.

16. 2. The method of claim 1, wherein the connection state of the port includes information related to an operational status of a computer that is connected to the port or that has been connected to the port.

17. 2. The method of claim 1, wherein the connection state of the port includes information regarding whether a device that is or was connected to the port is connected to a second device.

18. 20. The method of claim 17, wherein the connection state of the port comprises an operational status of the device and an operational status of the second device.

19. The method of claim 1 , wherein the mobile device is a mobile phone, a tablet computer, a laptop computer, or a headset.

20. The method of claim 1 , wherein the network device is a network switch, a router, or an access point.

21. The method of claim 1 , further comprising dynamically adjusting the overlay of the one or more virtual ports.

22. 1. A method for displaying network devices with augmented reality using a mobile device, comprising: displaying a captured camera image of the network device on a display of the mobile device, the image including one or more ports of the network device and optical code on the network device that includes network connectivity information associated with the network device; using the optical code to retrieve information relating to a connection status of the one or more ports; accessing a library of virtual objects based on the information related to a connection state of the one or more ports, the virtual objects including icons corresponding to types of devices connected to each of the one or more ports of the network device; determining a spatial relationship of the mobile device to the one or more ports using the optical code; overlaying one or more virtual ports, each of the virtual ports including an icon corresponding to the type of device connected to the one or more ports over the image of the network device, and a characteristic of the virtual port indicating the connection status of the port based on retrieving the information.

23. 1. A system for displaying network devices with augmented reality using a mobile device, comprising: one or more processors; and a memory coupled to the one or more processors, the memory configured to store the computer program instructions, the instructions, when executed by the one or more processors, performing a computer-implemented method, the computer-implemented method comprising: receiving, by the mobile device, an identification code from the network device; displaying a captured camera image of the network device on a display of the mobile device, the image including a plurality of ports of the network device; using said identification code to retrieve information relating to a connection status of each of said plurality of ports; determining a spatial relationship of the mobile device to the one or more ports using the identification code; and overlaying one or more virtual ports over the network devices on the image, the virtual ports including within the one or more virtual ports information regarding connectivity of the network devices.

24. 1. A method for displaying network devices with augmented reality using a mobile device, comprising: Capturing a video image of a network device using a mobile communication device, the video image including a plurality of ports; receiving, at the mobile communications device, from the network device, an identification code unique to the network device; determining from the identification code information regarding connectivity of the plurality of ports of the network device; and displaying an overlay on the video image in real time on the multiple ports of the network device, the overlay including network connectivity indicators specific to each port, the network connectivity indicators including one or more of connection status, connection speed, data traffic, connection identity, connection duration, and Power over Ethernet (POE) usage.

25. The method of claim 24 , wherein the network device comprises a switch.

26. The method of claim 24 , wherein the mobile communication device comprises a smartphone or a tablet.

27. The method of claim 24 , wherein the identification code comprises an optical code.

28. The method of claim 24 , wherein the identification code comprises a digital QR code.

29. The method of claim 24 , wherein the identification code comprises an RF code.

30. 25. The method of claim 24, further comprising determining a location of each of the one or more ports of the network device from the identification code and the image.

31. 25. The method of claim 24, further comprising determining the location of each of the ports of the plurality of ports from the identification code and the image, as well as an orientation of the mobile communication device.

32. 25. The method of claim 24, wherein the determining comprises determining information regarding the connectivity of the plurality of ports of the network device by accessing a remote database using the mobile communications device.

33. 1. A method for displaying network devices with augmented reality using a mobile device, comprising: Capturing video images of a plurality of network devices using a mobile communications device, each of the network devices comprising a plurality of ports and a unique identifier code; receiving, at the mobile communications device, from a network device of the plurality of network devices, the unique identification code specific to each of the network devices; determining information regarding connectivity of one or more ports of each of the network devices from the unique identification code specific to each of the network devices; displaying an overlay on the video image in real time over each of the one or more ports of the network device, the overlay including network connectivity indicators specific to each port, the network connectivity indicators including one or more of connection status, connection speed, data traffic, connection identity, connection duration, and Power over Ethernet (POE) usage; and updating the display in real time as the network connectivity indicator changes.

34. 1. A system for displaying network devices with augmented reality using a mobile device, comprising: one or more processors; and a memory coupled to the one or more processors, the memory configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method, the computer-implemented method comprising: Capturing a video image of a network device using a mobile communication device, the video image including a plurality of ports; receiving, at the mobile communications device, from the network device, an identification code unique to the network device; determining from the identification code information regarding connectivity of the plurality of ports of the network device; and displaying an overlay on the video image in real time on the multiple ports of the network device, the overlay including network connectivity indicators specific to each port, the network connectivity indicators including one or more of connection status, connection speed, data traffic, connection identity, connection duration, and Power over Ethernet (POE) usage.

35. 1. A system for displaying network devices with augmented reality using a mobile device, comprising: one or more processors; and a memory coupled to the one or more processors, the memory configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method, the computer-implemented method comprising: Capturing video images of a plurality of network devices using a mobile communications device, each of the network devices comprising a plurality of ports and a unique identifier code; receiving, at the mobile communications device, from a network device of the plurality of network devices, the unique identification code specific to each of the network devices; determining information regarding connectivity of one or more ports of each of the network devices from the unique identification code specific to each of the network devices; displaying an overlay on the video image in real time over each of the one or more ports of the network device, the overlay including network connectivity indicators specific to each port, the network connectivity indicators including one or more of connection status, connection speed, data traffic, connection identity, connection duration, and Power over Ethernet (POE) usage; and updating the display in real time as the network connectivity indicator changes.

Images