Augmented reality for internet connection setup

Abstract

To provide a device (for example, a system and device) and a method to view otherwise hidden connectivity of networking devices.SOLUTION: Electronic devices such as networking devices connected to a variety of different devices are observed through a real-time image in which information about connectivity and / or connection can be displayed, to show identities, connection states, and other information associated with ports of the networking devices. These methods can be implemented on a mobile device used to capture images of the networking devices and present an overlay of virtual objects on the captured images to a user in real time. The virtual objects can be dynamically moved or changed in shape in real time depending on movement of the mobile device. One or more filtering techniques can be used to stabilize the virtual objects with respect to the captured images.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] Cross - Reference to Related Applications This patent application claims the priority of U.S. Provisional Patent Application No. 62 / 900,403, filed on September 13, 2019, entitled "AUGMENTED REALITY FOR INTERNET CONNECTIVITY INSTALLATION", which is hereby incorporated by reference in its entirety.

[0002] Incorporation by Reference All publications and patent applications mentioned in this specification are hereby incorporated by reference in their entirety, as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.

[0003] The present disclosure generally relates to augmented reality applications, and particularly to those when rendering augmented reality objects using images of related components including network devices, as well as any of servers, switches, access points, routers, and many other network - related devices.

Background Art

[0004] Computer network equipment generally includes hardware devices that enable communication and interaction between devices on a computer network. Examples of network devices typically include switches, routers, gateways, and wireless access points that mediate data transmission on a computer network. When used in a business environment, network devices are typically used to connect various users' computers and office equipment such as printers to a local computer network and the internet. Network devices are usually mounted in racks and organized in stacks with various other network devices. Racks may be housed in temperature-controlled rooms separate from other racks of network devices. Information technology technicians or technical experts will connect cables to the ports of network devices to various other devices within the rack. Sometimes, network administrators need to access one or more network devices to repair or troubleshoot problems. Typically, multiple ports and cables look the same or similar to one another. Therefore, when looking at the various cables and ports of network devices in a rack, it can be difficult to identify which port is used to connect which network device.

[0005] In home settings, network devices are typically used to wirelessly connect various personal computers and other electronic devices to the internet. Consumers often set up their own internet connections by connecting various cables to network devices, or, in more complex cases, hire a professional to properly connect the network devices. User manuals can be helpful, but this process can be confusing, especially for non-experts. If the internet connection is down, it can be difficult for users to determine which devices are properly connected and working without time-consuming troubleshooting. Furthermore, today's advancements and trends in home automation are expanding interoperability in the home environment. For example, real-time analytics, machine learning, sensor technology, and embedded systems are converging to provide further monitoring and control of various appliances and systems, offering a high level of home automation. For example, lighting, HVAC (heating, ventilation, and air conditioning), appliances, security systems, and other systems can be connected and centrally controlled, improving convenience, comfort, energy efficiency, and security (e.g., smart homes). As network connectivity increases and becomes more complex, setting up and maintaining such networks can also become more complex.

[0006] In general, whether in a business or home environment, it can be beneficial to provide software, systems, and / or devices that enable users to more easily configure, maintain, and troubleshoot network devices and related components. [Overview of the Initiative]

[0007] Described herein are augmented reality (AR) systems and methods, including software, that can be used to visualize network connectivity and other invisible aspects of electronic devices. The systems and methods utilize virtual objects as an aid in the installation, maintenance, management, and / or repair of various network components. The systems and methods may be used by technical support staff or information technology (IT) administrators when maintaining an organization's wireless and / or wired network systems. In some cases, the systems and methods may be used by individual consumers of network devices during the setup or troubleshooting of a home or home office network environment.

[0008] Any of the methods, systems, and devices described herein can be used in conjunction with a mobile device such as a mobile phone, tablet computer, laptop computer, or headset (e.g., a virtual reality headset). For example, a user can use the camera of their mobile device to capture one or more images (e.g., real-time images) of one or more network devices and view the images of the network devices on the display of their mobile device. In some cases, the method is encoded into an AR software application installed on the mobile device and / or other things accessible by the mobile device (e.g., via the internet). In some cases, the software may be configured to work in conjunction with other software (e.g., applications) accessible by the mobile device, such as commercially available AR software tools. Any AR software tool based on any operating system can be used. In some embodiments, AR tools and features of the ARKit development platform developed by Apple Inc., headquartered in Cupertino, California, USA, are used.

[0009] While augmented reality technologies are known, the methods and systems described herein include the ability to enable a user to quickly and visually determine the connectivity status of a network device in real time, and possibly various other components of the network. A network device may include one or more identification codes, such as optical codes and RF codes. An optical code may include a visible marking that can be detected and correlated with a network device (e.g., a switch), and examples of optical codes include, but are not limited to, QR codes, alphanumeric codes, symbols, etc. An optical code may correspond to a two-dimensional (2D) pattern code (e.g., a matrix barcode) containing encoded information associated with the network device. An optical code may be displayed on the network device, such as on the exterior of the network device or on another part that is easily accessible to the user. In some embodiments, the optical code is on a changeable display, such as a touchscreen display, which the user can change to access different optical codes or other information.

[0010] Alternatively or additionally, the identification code on a network device may be a radio-based identification code corresponding to the network device. In some examples, a network device may transmit a radio signal, such as a Bluetooth signal, or interact with a mobile device by a radio frequency identification (RFID) code (tag) or a near-field communication (NFC) code (tag). Radio-based identification can be used in combination with, or instead of, an optical code for identifying a network device. Those skilled in the art will understand that a Bluetooth signal may have frequencies in the range of 2.402 GHz to 2.480 GHz. A radio frequency signal may have frequencies in the range of 20 kHz to 300 GHz.

[0011] An identification code may be uniquely associated with a particular network device. However, in some variations, the identification code may instead be associated with a particular subgroup of network devices, and the unique identity of a device may be determined based on secondary indicators, such as the identity of a scan device (e.g., a smartphone, tablet, etc.) that may be associated with a particular user. For example, an identification code may identify a particular category, class, or subset of network devices, and the unique identity of a particular device within this category, class, or subset of network devices may be uniquely identified by relevant secondary information, such as the user identity of the user performing the scan, the scan device (e.g., a smartphone), or one or more devices that are determined to be associated with or connected to the network device.

[0012] Identification codes (e.g., optical codes and / or wireless-based identification codes) can be used to retrieve real-time information related to the connection status of each port of a network device. These identification codes can also be used to access a library of virtual objects based on the port connection status and the type of device connected to the network. One or more virtual objects can then be combined (e.g., overlaid) with captured camera images of the network device to provide a visual representation of the ports, the various devices connected to the network, and other information related to those devices.

[0013] In some embodiments, a virtual object includes an illustration of a communication port (virtual port) aligned with a captured image of the corresponding port. For example, a user can view a port on a network device using the camera of a mobile device, and the virtual port can be displayed on at least a portion of the port's image. The position of the virtual port may be automatically and dynamically adjusted based on the movement of the mobile device. For example, if a user moves the mobile device to capture an image of the network device from a different viewpoint, the position of the virtual port can be automatically adjusted to maintain its position on the port's image, resulting in a seamless transition for the user as they move and view different parts of the network device. In some embodiments, the position of the virtual object is stabilized by excluding one or more degrees of freedom used to determine the mobile device's spatial relationship to the port.

[0014] A virtual object may include one or more virtual labels containing text, numbers, and / or symbols that convey information related to a port. For example, a virtual label on or near an image of a port may include the port's identification (e.g., port number) and / or the identification of a device connected to the port. A virtual label may include text, numbers, or symbols and lines extending between the virtual label and the image of the port indicating the relationship between the virtual label and the port.

[0015] A virtual object may include an illustration of one or more devices connected to or previously connected to a port. The illustration (icon) can provide the user with a quick visual reference of which device is associated with which port. For example, an icon might be an illustration with enough detail for the user to determine whether the icon is a personal computer, printer, router, access point, or other device. In some cases, information about the connected device may be used to illustrate a specific model or brand of the device. For example, an icon might be an illustration of a specific brand or model of a telephone, television, laptop computer, headset, desktop computer, access point, media player, or tablet computer.

[0016] In some cases, virtual objects may have different appearances depending on the port's connection status. For example, the color or opacity of a virtual port may differ depending on whether the device is connected to the port or not. In some cases, icons may have different color or opacity depending on whether the device is connected to the port or not. For example, an icon may have a grayed-out or ghosted appearance if the device is not currently connected to the port but was connected to the port within a specified time period previously.

[0017] According to some embodiments, a virtual object may convey information about whether a device connected to a port of a network device has other devices connected to it. For example, if a network device is connected to a wireless access point, a virtual label and / or illustration may indicate whether the access point device is connected to one or more other devices such as a telephone, laptop, tablet, or headset. The virtual object may also convey performance information about various devices connected to the network device, such as throughput and uptime. The virtual object may convey such data in the form of a virtual graph or chart.

[0018] For example, connection information may also, or alternatively, include port statistics for both connected and / or unconnected ports on a network device. For example, a virtual display may include information about port statistics for all or some connected ports, and the user can toggle the display of any (all or a subset) of the port statistics information. Port statistics information may include, for example, throughput information (e.g., current throughput, average throughput such as time, day, week, MAC address, IP address, etc.). In some variations, port statistics information may include historical information (e.g., showing one or more connections made in the past, when they were connected / disconnected, etc.). In some variations, port statistics information for ports that are not currently connected may be displayed. Port statistics information may be displayed as text, icons, or a combination of both. This information may be displayed at the user's discretion (e.g., by selecting from a touchscreen to expand or collapse additional information). Thus, port connection information may be displayed in addition to the virtual object, or separately from the virtual object, or as part of the virtual object.

[0019] These and other features and advantages of AR methods, systems, and devices are described in detail herein.

[0020] Any of the methods described herein can be performed by an apparatus (e.g., a system, device, etc.) configured to perform the method, including any optional steps described herein. For example, described herein is a system for displaying a network device with augmented reality using a mobile device. Such a system may include: receiving an identification code from a network device by the mobile device; displaying a captured camera image of the network device on the mobile device's display, wherein the image includes multiple ports of the network device; using the identification code to retrieve information relating to the connectivity status of each of the multiple ports; using the identification code to determine the spatial relationship of the mobile device to one or more ports; and overlaying one or more virtual ports on the network device on the image, wherein the virtual ports include information relating to the connectivity of the network device.

[0021] The identification code may be an optical code (e.g., a barcode, QR code, etc.) or an RF code (e.g., a Bluetooth code, NFC, etc.). Either of these methods may involve accessing a library of virtual objects based on information related to the port's connection status, and the virtual objects may contain one or more icons corresponding to the types of devices connected to one or more ports of a network device (e.g., cameras, telephones, computers, access points, etc.). A virtual port may contain one or more icons. When these icons are displayed, they may be used to indicate information about the connected device.

[0022] For example, a method of displaying a network device with augmented reality using a mobile device may include displaying a captured camera image of the network device on the mobile device's display, wherein the image includes one or more ports of the network device and an optical code on the network device containing network connectivity information associated with the network device; using the optical code to retrieve information relating to the connectivity status of one or more ports; accessing a library of virtual objects based on the information relating to the connectivity status of one or more ports, wherein the virtual objects include icons corresponding to the types of devices connected to each of the one or more ports of the network device; using the optical code to determine the spatial relationship of the mobile device to one or more ports; and overlaying one or more virtual ports, wherein each virtual port includes an icon on the image of the network device corresponding to the types of devices connected to one or more ports, and the characteristics of the virtual ports indicate the connectivity status of the ports based on the retrieved information.

[0023] A system for displaying a network device with augmented reality using a mobile device may include one or more processors and memory coupled to one or more processors, wherein the memory is configured to store computer program instructions, and when an instruction is executed by one or more processors, a computer implementation method is performed, the computer implementation method including receiving an identification code from the network device by the mobile device, displaying a captured camera image of the network device on the mobile device's display, the image including multiple ports of the network device, using the identification code to retrieve information relating to the connectivity status of each of the multiple ports, using the identification code to determine the spatial relationship of the mobile device to one or more ports, and overlaying one or more virtual ports on the network device on the image, the virtual ports including information relating to the connectivity of the network device within one or more virtual ports.

[0024] One or more processors may be part of a mobile device (e.g., a smartphone, tablet, etc.).

[0025] Any of these methods, and the devices used to implement them, can be configured to operate in real time or near real time.

[0026] For example, a method of displaying a network device together with augmented reality using a mobile device, the method comprising: capturing, using a mobile communication device, a video image of the network device, the video image including a plurality of ports; receiving, in the mobile communication device, an identification code unique to the network device from the network device; determining, from the identification code, information regarding the connectivity of the plurality of ports of the network device; and displaying, in real time on the video image, an overlay on the plurality of ports of the network device, the overlay including an indicator of network connectivity unique to each port, the indicator of network connectivity including one or more of connection status, connection speed, data traffic, identity of the connection, connection duration, and power over Ethernet (POE) usage.

[0027] A method of displaying a network device together with augmented reality using a mobile device may include: capturing, using a mobile communication device, a video image of the network device, the video image including a code unique to the network device; determining, from the code, information regarding the connectivity of one or more ports of the network device; and displaying, in real time on the video image, an overlay on the one or more ports of the network device, the overlay including an indicator of network connectivity unique to each port, the indicator of network connectivity including one or more of connection status, connection speed, data traffic, identity of the connection, connection duration, and power over Ethernet (POE) usage.

[0028] The network device can be any suitable network device having one or more ports, such as, for example, a switch, a router, an access point, etc. The mobile communication device can include a smartphone or a tablet. The code can include a QR code, such as a digital QR code. The digital QR code can be updated or modified to optically transmit information to a handheld device (e.g., a smartphone, a tablet, etc.).

[0029] Any of these methods can include determining the location of each of the one or more ports of the network device from the code and the image. For example, any of these methods can include determining the location of each of the one or more ports of the network device from the code and the image, as well as the orientation of the mobile communication device.

[0030] Determining can include accessing a remote database using the mobile communication device and using a code that identifies information about the one or more ports to determine information regarding the connectivity of the one or more ports of the network device. In any of these methods, each of the one or more ports can be uniquely identified.

[0031] A method for displaying network devices with augmented reality using a mobile device may include capturing video images of multiple network devices using a mobile communication device, each of which has multiple ports and a unique identifier code; receiving a unique identifier code specific to each network device from the multiple network devices in the mobile communication device; determining information regarding the connectivity of one or more ports of each network device from the unique identifier code specific to each network device; and displaying an overlay on each of the one or more ports of the network devices in real time on the video image, the overlay including a network connectivity indicator specific to each port, the network connectivity indicator including one or more of connection status, connection speed, data traffic, connection identity, connection duration, and power over Ethernet (PoE) usage; and updating the display in real time as the network connectivity indicator changes.

[0032] For example, a method for displaying network devices with augmented reality using a mobile device may include capturing video images of multiple network devices using a mobile communication device, wherein the video images include a digital QR code unique to each of the multiple network devices; determining from the digital QR codes information about the connectivity of one or more ports of each of the network devices; and displaying an overlay on each of the one or more ports of the network devices on the video images in real time, wherein the overlay includes a network connectivity indicator unique to each port, and the network connectivity indicator includes one or more of the following: connection status, connection speed, data traffic, connection identity, connection duration, and power over Ethernet (PoE) usage; and updating the display in real time as the network connectivity indicator changes.

[0033] This specification also describes a system for displaying a network device in conjunction with augmented reality using a mobile device. For example, the system may include one or more processors and memory coupled to one or more processors, the memory being configured to store computer program instructions, and when an instruction is executed by one or more processors, a computer implementation method is performed, the computer implementation method being to capture a video image of a network device using a mobile communication device, wherein the video image includes multiple ports; the mobile communication device receiving an identification code unique to the network device from the network device; determining from the identification code information regarding the connectivity of multiple ports of the network device; and displaying an overlay on the multiple ports of the network device in real time on the video image, wherein the overlay includes a network connectivity indicator unique to each port, and the network connectivity indicator includes one or more of the following: connection status, connection speed, data traffic, connection identity, connection duration, and power over Ethernet (PoE) usage.

[0034] A system for displaying network devices with augmented reality using a mobile device may include one or more processors and memory coupled to one or more processors, wherein the memory is configured to store computer program instructions, and when an instruction is executed by one or more processors, a computer implementation method is performed, the computer implementation method being to capture video images of multiple network devices using a mobile communication device, each of which network devices has multiple ports and unique identifier codes; to receive a unique identifier code specific to each network device from the network devices of the multiple network devices; to determine information regarding the connectivity of one or more ports of each network device from the unique identifier code specific to each network device; and to display an overlay on each of the one or more ports of the network devices on the video image in real time, the overlay including a network connectivity indicator specific to each port, the network connectivity indicator including one or more of connection status, connection speed, data traffic, connection identity, connection duration, and power over Ethernet (PoE) usage; and to update the display in real time as the network connectivity indicator changes.

[0035] The methods and apparatus described herein may be configured for use with multiple network devices (e.g., multiple switches, routers, etc.), each containing multiple ports. For example, in some modifications, the method may include receiving an identification code (e.g., a unique identification code) for each of the multiple network devices, such as a stack of devices. The identification code may be all optical code, all RF code, or a combination of optical and RF code. These methods and apparatus may allow a user to switch the display of virtual ports for each of the multiple devices. For example, a user may move a mobile communication device up or down a stack of network devices to change the view displaying the virtual ports corresponding to each, and alternatively or additionally, a user may select controls on the mobile communication device to switch augmented reality views between different network devices. In some modifications, multiple different network devices may be displayed together simultaneously. [Brief explanation of the drawing]

[0036] The novel features of the present invention are described in detail in the following claims. A better understanding of the features and advantages of the present invention will be obtained by referring to the following detailed description and accompanying drawings which describe exemplary embodiments in which the principles of the present invention are utilized.

[0037] [Figure 1A-1B] This figure shows an example of a user using the devices and methods described herein. [Figure 2A] This figure shows an example of a stack of network devices with optical codes. [Figure 2B] This figure shows an example of a display screen that shows one or more optical codes. [Figure 3A-3C] This figure shows an example of an optical code generated using a code generator. [Figures 4A-4F] This figure shows an exemplary user interface for setting up and using a mobile device in AR mode. [Figure 5A-5C] This figure shows another exemplary user interface for setting up a mobile device in AR mode. [Figure 6] This figure shows an example of an AR overlay used to view network devices in combination with captured images. [Figures 7A-7B] This figure shows another example of a user interface for using a mobile device in AR mode. [Figure 8] This figure shows an example of a user interface that includes various virtual objects related to the performance of one or more network devices. [Figures 9A-9E] This figure shows an example of a 3D coordinate system used for rendering AR objects. [Figure 10A-10D] This diagram shows a flowchart and architecture map illustrating an example of the process for setting up and using an AR application. [Modes for carrying out the invention]

[0038] Generally, this specification describes augmented reality (AR) systems, devices, and methods for viewing invisible features of electronic devices. In specific applications, AR systems, devices, and methods are used to render virtual objects that convey information related to the network connectivity of electronic devices. These virtual objects may provide users with a visual representation of the network connectivity status of network devices, thereby facilitating the installation and management of computer networks. Users can capture a stream of images of network devices in real time using the camera of a mobile device, such as a cell phone or tablet computer. The virtual objects can be rendered on the mobile device's display along with the captured images, providing users with an augmented reality experience.

[0039] Figures 1A and 1B show an example of a user 101 using the AR device and method described herein. User 101 may be any person, such as a technical support person or IT administrator who installs or manages a workplace network, or an individual consumer who installs or manages a home network. User 101 may use the mobile device 100 to visually observe invisible aspects related to the network device 102 and to install, troubleshoot, and / or check the status of the network device 102.

[0040] Generally, network device 102 may be any electronic device that can connect to one or more computer networks by wire and / or wirelessly. A computer network can include one or more networks ranging in size from wide-area computer networks to nanoscale computer networks. A computer network may be a local network, a wide-area network such as a cloud-based network (e.g., the Internet), or a combination of a local network and a wide-area network. For example, a computer network may be a local computer network interconnecting computers in a business, home, or school that may be connected to the Internet. In some cases, network device 102 is configured to mediate data transmission in one or more computer networks. Network device 102 may be a network switch (also called a switching hub) that uses packet switching to connect devices on a computer network, receive and process data, and forward it to a destination device. Network device 102 may be a gateway device that enables data flow between different networks. Network device 102 may be a router that forwards data packets between computers. Network device 102 may be a digital hub that receives information from various sensors and controllers (nodes) that can form a “smart” wirelessly connected, habitable space network (e.g., home, office, workplace, etc.). Network device 102 may be a network bridge that creates a aggregate network from multiple communication networks or network segments. Network device 102 may be a repeater that receives and retransmits signals. Network device 102 may be an Ethernet hub for connecting multiple Ethernet devices together. Network device 102 may be a hybrid network device such as a multilayer switch, protocol converter, and / or bridge router (browser).In some embodiments, the network device is a UniFi Dream Machine Pro (UDMP) or other network device manufactured by Ubiquiti Network, based in New York City, New York, USA. Network device 102 may be a perimeter network device located at a connection point between different networks, such as a proxy server, firewall, or network address converter (NAT). Network device 102 may be an end station device used to establish a network or dial-up connection, such as a network interface controller (NIC), wireless network interface controller, modem, ISDN terminal adapter, or line driver. Network device 102 may be a server that provides functionality to one or more clients. Network device 102 may be one of several network devices located in a stack 110 of network devices, in this case in a rack.

[0041] The devices and methods described herein use virtual objects to enable a user 101 to see invisible aspects of a network device 102, or of many network devices in a stack 110. To initiate the AR tool, user 101 may scan an optical code 104 on the network device 102 using a scanning device, such as a mobile device 100, as shown in Figure 1A. For example, the user may use the camera of the mobile device 100 to capture one or more images of the optical code 104. The optical code 104 may be any machine-readable code containing information about the network device 102. The optical code 104 may be a two-dimensional (2D) pattern encoding information related to the network device 102. For example, the optical code 104 may be a quick response (QR) code, an ARTag code, a barcode (e.g., a linear barcode), an alphanumeric code, a numeric code, written characters, or any combination thereof. The optical code 104 may be on a surface of the network device 102, such as the outer surface that is easily accessible to user 101. The optical code 104 can be attached to a surface using any means. For example, the optical code 104 may be printed on the surface of the network device or on a sticker attached to the surface of the network device. In some cases, the optical code 104 is generated by a display on the network device 102.

[0042] The optical code 104 may include one or more identifiers (e.g., unique identifiers) associated with the network device 102 so that information associated with the network device 102 can be accessed. For example, when a user captures one or more images of the optical code 104, the mobile device 100 can send the identifiers to a network management system (e.g., one or more remote servers). Upon receiving the identifiers from the mobile device 100, the network management system can send information about the mobile device 100 and / or the network device 102. For example, the mobile device 100 may utilize a database of information related to the network device 102. The database may include a library of virtual objects based on the identity of the network device 102, the connection status of the network device 102, and the identity of any devices connected to the network device 102. The information may include the type (e.g., model and type) of the network device 102, as well as information about one or more communication interfaces, such as hardware or software interfaces. The information may include information related to one or more communication ports 108 of the network device 102. The ports 108 may be any type of interface between the device and other computers or peripheral devices. Port 108 may be an input or output interface for a device. Port 108 may include, but is not limited to, a hardware interface including a Universal Serial Bus (USB) port, a serial port, a video port, an audio port, a High Definition Multimedia Interface (HDMI) port, or a parallel port. In some cases, the communication interface may be a wireless interface that connects wirelessly to one or more computers.

[0043] When the mobile device 100 receives information related to the network device 102, this information can be used to render a virtual object in real time on the mobile device 100's display along with the captured camera image. In some embodiments, the virtual object overlays the captured camera image. The virtual object may include illustrations and / or text that convey information to the user 101 that is not normally readily apparent. The virtual object may include illustrations of a virtual port 108, a cable 106, or other parts of the network device 102. The virtual object may also include text, numbers, and labels that convey information about the identity and / or status of any device connected to the network device 102 via port 108 and / or wireless communication.

[0044] The virtual object may include icons corresponding to illustrations of various devices connected to the network device 102. In some cases, the icons may be displayed adjacent to the captured image of port 108, thereby enabling user 101 to quickly identify the types and connectivity of various devices. The icons can depict any type of device connected to the network, such as a computer or other device (e.g., a sensor). For example, a computer could be a personal computer such as a desktop computer or a mobile device (e.g., a laptop, tablet, phone, or headset). A computer may be a server computer configured to be shared by one or more users of the network. A computer may be a server or a client. A computer may be a router, such as a wireless router. A computer may be a modem. A computer may be a printer or other office environment computer device. A computer may be an access point device configured to allow Wi-Fi devices to connect to the network. A computer may be a smart home device, such as a thermostat, smoke or carbon monoxide detector, security system, doorbell, smart lock, or kitchen appliance. A computer may be a camera, speaker, or microphone device. A computer may be a Bluetooth device.

[0045] In some embodiments, the mobile device 100 can dynamically display virtual objects in real time. That is, virtual objects may be rendered on the mobile device 100's display during, or very close to, the time it takes for images captured by the mobile device's camera to be rendered on the display, so that the user can experience a substantially continuous progression of images, such as a video in which the virtual objects are incorporated. As the mobile device 100's camera captures a stream of images over time, virtual objects can be rendered on the display along with the stream of images. Therefore, when the user 101 moves the mobile device 100 relative to a network device 102, such as when scanning various ports 108 of the network device 102, the virtual objects can dynamically update themselves based on the progression of the captured images. For example, the virtual objects may maintain their position on the display relative to the corresponding port 108 as the mobile device 100 is moved. This spatial capability can be enabled using a spatial coordinate system that can rely on one or more motion sensors in the mobile device 100 to detect the movement of the mobile device 100. For example, the mobile device 100 may include a motion detection system which may include an accelerometer and / or a gyroscope.

[0046] The virtual object may update information about the connectivity status of various ports 108 in real time. In this way, user 101 can troubleshoot the network device 102 using the information provided by the virtual object, for example, by connecting and disconnecting one or more cables 106 to and from port 108, or by turning on and off various devices connected to the network device 102. In some embodiments where data is transmitted in network packets (e.g., packet switching), the virtual object may enable user 101 to perform feed packet inspection for security control. User 101 can also check the connectivity status of each port 108 in real time to determine, for example, whether the network device 102 is properly connected to various devices in the network. The virtual object may also transmit information about the status of various devices in the network, including secondary, tertiary, and other devices connected to those devices. Other network devices 102 in the stack 110 can also be viewed by AR so that user 101 can scan all network devices in the stack 110. In this way, the AR tools described herein enable user 101 to quickly and easily determine the status of the entire network system, including network subsystems.

[0047] Figure 2A shows an example of network devices 202a-202f, each having different optical codes 204a-204f that encode information related to the corresponding network devices 202a-202f. As shown, the optical codes 204a-204f can have any of several different patterns. For example, the optical codes 204a-204f may be a quick response (QR) code, an ARTag code, a barcode (e.g., a linear barcode), an alphanumeric code, a numeric code, written characters, or any combination thereof. Different patterns can encode different types of information based on the identification and status of the corresponding network devices. In some cases, the optical codes 204a-204f are displayed on the display (e.g., a flat panel display) of the network devices 202a-202f. Figure 2B shows an example of a display component that can be configured to display one or more optical codes. The display may be any type, such as a liquid crystal display (LCD) and / or a light-emitting diode (LED) display (e.g., an organic light-emitting diode (OLED) display). In some embodiments, the display is a touchscreen display that can display different images based on touch input. For example, the touchscreen display may be configured to change what is displayed in response to the user swiping the touchscreen.

[0048] In some cases, network devices may not have an LCD or other type of visually active screen; instead, a static optical code associated with the network device (e.g., a sticker or other printed identifier) ​​may be used.

[0049] As described above, the identification code may be a radio frequency (e.g., RF) identification code, such as a wireless-based ID (e.g., Bluetooth, NFC, RFID) code, that can be used by a handheld device to identify (e.g., uniquely identify) a network device. The RF identification code may be dynamic (similar to an optical code on a screen) or static (e.g., a printed optical code attached to the outside of the network device).

[0050] Figures 3A–3C show three different optical code examples containing different code patterns. These examples demonstrate how different patterns can encode unique information based on a network device, including the connectivity status of the network device. The information from the optical code can then be sent to a cloud-based management system, for example, to transmit mobile device and / or network device information associated with a network device and to generate virtual objects. In some examples, the optical code is generated using a pattern generator program that generates patterns (e.g., randomly) to provide a unique optical code.

[0051] Figures 4A to 4F show examples of user interfaces for setting up and using a mobile device 400 in AR mode, according to several embodiments. Figure 4A shows a user viewing multiple network devices 402, each having a corresponding optical code (e.g., 404), using the camera of the mobile device 400. The mobile device 400 can be considered an example of the mobile device 100. The user interface 420 may be displayed on the display screen of the mobile device and may guide the user through several interface screens for the initial setup of the mobile device 400 to enter AR mode. The user interface 420 may include the geographical location of the network devices, the type of network devices (e.g., type and model), and / or illustrations of the network devices. The user interface 420 may video record images of the network devices 402 captured by the camera of the mobile device 400. The user interface 420 may include a button to enter AR mode.

[0052] Figures 4B1 and 4B2 show two exemplary user interface 421 before scanning the optical codes of network devices. The user interface 421 may prompt the user to scan each of the network devices, thereby allowing access to and loading of information about each of the network devices for use in AR mode. The user interface 421 may include an optical code alignment frame 422 for aligning captured images of the optical codes (e.g., 404) of each network device.

[0053] Figures 4C1–4C4 show four illustrations of an exemplary user interface 422 during scanning of optical codes of multiple network devices in a stack ("rack"). After scanning a first optical code of a first network device, the user interface may prompt the user to scan a second optical code of a second network device. Visual indicators may be used to indicate the locking of an optical code within the optical code alignment frame, such as presenting the optical code alignment frame 422 in a different color or tone. Alternatively or additionally, different visual indicators, such as a checkmark symbol 427, may be visible to indicate that the optical code has been successfully scanned. In some embodiments, tactile and / or audible indicators, such as vibration and / or sound generated by a mobile device, may be used to indicate that the locking and / or scanning of the optical code has been successful. The user interface may include a list 424 or table of network devices, which is progressively updated as each of the optical codes of the network devices is successfully scanned. For example, icon 425, which depicts a corresponding network device, can be registered in list 424 after the optical code of the network device has been successfully scanned and / or recognized (e.g., by a network management system). List 424 can then register additional icons 426 after the optical codes of additional network devices have been successfully scanned and / or recognized (e.g., by a network management system). This scanning of different optical codes of different network devices may continue, for example, until all network devices in the stack have been scanned.

[0054] Figures 4D1 to 4D4 show four diagrams of an exemplary user interface 430 illustrating how AR settings may be applied after the optical codes of network devices have been successfully scanned. In one view, the user interface 430 may, in addition to displaying a list 432 of the additional scanned network devices, display a prompt 433 (e.g., a button) for the user to respond whether to scan for further network devices. The user interface 430 may provide a prompt 434 (e.g., a button) for the user to proceed to the next display screen of the user interface. After all desired network devices have been scanned, the user interface may provide a prompt 435 (e.g., a button) for the user to optionally enter a name for a stack (e.g., a "rack") of network devices, and a prompt 436 (e.g., a button) for the user to proceed to the next display screen of the user interface. Any display screen of the user interface 430 may include a back prompt 438 for returning to the previous display screen. The AR settings of the network devices can then be applied to the network stack based on the information received by the optical codes and the information provided by the user. Once the settings are applied and complete, the full AR experience can be loaded at this point, allowing the mobile device to utilize its camera in AR mode within the newly created stack of network devices.

[0055] Figure 4E shows an example of a mobile device immediately after entering AR mode. As illustrated, the display of the mobile device 400 can display a user interface 450 which may include one or more virtual objects 401 that overlay captured images of the network device, thereby conveying information related to the network device that is not normally visible to the user.

[0056] Figure 4F shows an enlarged view of the user interface 450 in AR mode according to several embodiments. The virtual objects may include text (e.g., numbers, letters, and / or symbols) and / or illustrations. For example, virtual identification text 452 of a network device, such as the name of the network device and / or its Internet Protocol (IP) address, may be positioned adjacent to or on top of (e.g., partially on) an image of the network device. The virtual objects may include a port area 454 that at least partially frames various virtual objects associated with a particular network device. The virtual objects may include an optical code alignment frame 456 for aligning a captured image of an optical code. The optical code alignment frame 456 can be used to assist a user in aligning an optical code within a lock area A of the optical code alignment frame 456 for locking the optical code. The lock area A may be in the central area of ​​the optical code alignment frame 456. In some embodiments, the port area 454 and / or the optical code alignment frame 456 may have transparent or translucent properties, allowing the underlying image of the network device to be seen. In some cases, virtual port B overlays the corresponding port of the network device. Virtual port B may have the shape and size of the corresponding physical port. Virtual port B may include a number corresponding to the port number as identified by the network device manufacturer, for example. Virtual port B may be filled with color C and / or be at least partially transparent to allow the underlying image of the network device to be seen. In some cases, the color and / or transparency properties may be changed depending on the state of the corresponding port.For example, a virtual port B that is actively connected to a computer can be filled with a first (e.g., dark) color and / or translucent properties, while a virtual port B that is not actively connected to a computer can be hollow (i.e., transparent) or filled with a second (e.g., light) color and / or translucent properties.

[0057] A virtual object may include an icon F (also called a virtual device or virtual computer) that matches the device connected to the corresponding port. Icon F may be an illustration with sufficient detail to allow the user to identify the type of device (e.g., a phone, tablet, laptop, desktop, television, access point, virtual assistant device, home appliance, or security system device), thereby enabling the user to easily and quickly distinguish between different types of computers. Icon F may be automatically assigned or selected from a database (also called a library) of virtual object illustrations chosen by the user. The relative position of the computer icon F may be contained within the icon area E.

[0058] A virtual object may include virtual device identifier text H that describes the identity of the computer. The virtual device identifier text H may include information that helps further identify devices connected to or previously connected to the corresponding port. For example, the user's name and / or the device type and model may be identified in the virtual device identifier text H. The virtual device identifier text H may be automatically registered, for example, based on information provided by a cloud-based network management system or selected by the user. The relative position of the virtual device identifier text H may be contained within a device identifier text area G.

[0059] In some embodiments, icon F and / or virtual device identifier text H may have different characteristics based on whether the device is actively connected to the corresponding port. For example, icon F and / or virtual device identifier text H may have a fully colored or opaque appearance if the device is actively connected to the port, a ghosted or grayed-out appearance if the device is not currently connected but was previously connected to the port within a predetermined period, and no indication (i.e., no illustration or text in areas E and G) if the device does not need to be connected to the port within a predetermined period. The predetermined period may vary. For example, the predetermined period may be about 30 minutes, 1 hour, 2 hours, 3 hours, 6 hours, 12 hours, 24 hours, 3 days, or less than a week. In some embodiments, the predetermined period may be selected by the user. In some cases, icon F and / or virtual device identifier text H may include information about the operational status of the device. The operational status may include the device's performance characteristics (e.g., real-time or historical).

[0060] In some cases, icon F and / or virtual device identifier text H may include an indication of whether the device is connected to another device (e.g., a secondary, tertiary, etc.). For example, an access point device may be operablely connected to one or more phones, tablets, laptops, and / or headsets. Icon F may have different appearances indicating connectivity to one or more additional devices, such as being presented in different colors, surrounded by a halo or shadow, or being given other attributes. Alternatively or additionally, virtual device identifier text H may include text indicating connectivity to one or more additional devices. In some cases, icon F and / or virtual device identifier text H may include information about the operational status of one or more additional devices.

[0061] The characteristics of a user interface can vary depending, for example, on the dimensions, form factor, and / or operating system of a mobile device's display. For instance, a mobile phone may be narrower in height or width than a tablet computer. The user interface can be configured to address these differences.

[0062] Figures 5A to 5C show other exemplary user interfaces for setting up a mobile device in AR mode. Figure 5A shows user interface 521 before scanning an optical code, similar to user interface 421 (Figures 4B1 to 4B2), but user interface 521 includes a list 523 which may be registered network devices and are positioned laterally relative to the optical code alignment frame 522. Figure 5B shows user interface 523 during scanning an optical code, similar to user interface 423 (Figures 4C1 to 4C4), but user interface 523 includes a list 526 which may be registered network devices and are positioned laterally relative to the optical code alignment frame 522. Figure 5C shows user interface 530 for applying AR settings, similar to user interface 430 (Figures 4D1 to 4D4), but a prompt 535 (e.g., a button) for the user to choose to complete the AR settings is presented on the bottom side (e.g., the right side) of user interface 530.

[0063] As described herein, the optical code may contain encoded information that provides access to a library of virtual objects. The optical code may function as a reference marker for indicating the appearance and position of virtual objects. The relative positions of virtual objects may be placed within an overlay, for example, overlaying an image captured by a mobile device's camera. Figure 6 shows an example of an overlay 600 according to several embodiments. At least a portion of the overlay 600 may be substantially transparent so that the underlying image can be seen when rendered on the mobile device's display. An optical code alignment frame 630 may define an area of ​​the overlay 600 that is configured to scan and receive an image of the optical code. In some cases, the optical code alignment frame 630 may be contoured with a boundary or frame and / or have a central marker (e.g., a circle). In other cases, the optical code alignment frame 630 may not have a boundary or frame.

[0064] When an optical code is scanned and verified by a network management system (e.g., local or cloud-based) as being associated with a specific network device, it can access a library of virtual objects associated with the network device, which can then be viewed on the mobile device's display. For example, the optical code can inform the virtual object administrator of the type and model of the network device, as well as any relevant physical characteristics such as the type, location, and size of the port. This information can be used to form one or more virtual ports 634 corresponding to the actual ports of the accessed network device from a database of various types of virtual ports. Furthermore, the network management system may access information related to the port's connectivity status. This information can be used to determine various aspects of the virtual object that convey information related to the connectivity status of one or more ports 634.

[0065] The overlay 600 may include a port area 632 that defines the area of ​​the overlay 600 containing the virtual port 634. The port area 632 may or may not include a visible outline or frame. In some cases, the port area 632 may be a different color and / or semi-transparent compared to the rest of the overlay 600. The virtual port 634 may include text, such as a number, to identify the port number. In some cases, the virtual port 634 may include one or more status indicators (e.g., 638 and 639) that indicate the port's connectivity status. The status indicators may have different appearances (e.g., different colors or shapes) based on whether the port is actively connected to a computer, previously connected to a computer within a given period of time, or not connected to a computer within a given period of time. Such status indicators (e.g., 638 and 639) may be used in addition to, or as an alternative to, the filled-in features described above with reference to Figure 4E.

[0066] AR enables video tracking capabilities that calculate the camera's position and orientation in real time relative to physical markers such as optical codes on network devices. For example, once the camera's position is known, a virtual camera can be positioned at the same point, revealing a virtual object (e.g., an optical code alignment frame 630) at the location of the optical code. Information from a network management system regarding a specific network device can be used to provide precise placement and alignment of virtual ports 634 relative to the optical code. This information can also be used to determine the relative distance between virtual ports 634, as well as the shape and size of the virtual ports 634. Information from the network management system can be further used to modify the appearance of virtual objects in real time based on the connection status of various ports, as described herein.

[0067] Because users can move their mobile devices relative to network devices, thereby projecting images of the network devices at different depths, the spatial relationship between the mobile device and the network device can change in real time. To address changes in depth, the generation of virtual objects typically relies on tracking motion based on rotational degrees of freedom around fixed orthogonal axes (x, y, z axes), similar to a gimbal system. Motion can be tracked using conventional three-dimensional rotational motion, such as roll (rotation around the x-axis), pitch (rotation around the y-axis), and yaw (rotation around the z-axis), which define six degrees of freedom. While this model is useful, the data associated with continuously monitoring and detecting motion in real time can be significant, potentially resulting in a delay before the virtual object is generated on the display.

[0068] In any of the embodiments described herein, spatial relationships can be simplified by eliminating one or more degrees of freedom. For example, the position of a network device is substantially stable, and ports on a network device are generally oriented along a plane (e.g., to the back of the network device perpendicular to the floor). Thus, one of the three-dimensional degrees of freedom can be considered fixed and excluded from calculations for determining the spatial relationship between the mobile device and the network device. In some embodiments, this is achieved by excluding (e.g., ignoring) some of the raw data related to the three-dimensional movement of the mobile device that is not necessary to calculate the relative movement of the mobile device to the ports. For example, in some embodiments, movement related to roll (rotation around the x-axis) can be excluded from calculations, thereby reducing the degrees of freedom and information required for calculations by one-third. Filtering the data in this manner can speed up calculations, allowing virtual objects (e.g., virtual ports) on captured images to be updated more quickly. Furthermore, this reduces the amount of correction required to update each image frame, resulting in virtual objects having a more stable (e.g., less wobbling) location and appearance in real time.

[0069] Furthermore, the AR functionality may use optical feature recognition to determine the location of virtual port 634. In some examples, optical feature recognition may be used in conjunction with, or instead of, the spatial relationship between the mobile device and the network device. In some examples, optical feature recognition may include optical character recognition (OCR).

[0070] More specifically, OCR can be used to identify any relevant text on hardware (e.g., port numbers, actual ports) with a camera to assist in the placement of AR features in the overlay. This can mitigate sensor drift or other alignment issues that may occur with overlays of dead reckoning or other motion-based features. Furthermore, OCR can be used to display and verify specific identification information of network devices before locating virtual ports.

[0071] For example, returning to Figure 1 for a brief reference, if network device 102 is a 24-point network switch, the text on the label of network device 102 can be captured via OCR and identified as a 24-point switch. The mobile device 100 can then display the identification information, namely "This is a 24-point switch," and then, as mentioned above, the virtual ports of the network device can be searched using the AR overlay.

[0072] In some embodiments, after OCR is performed, the mobile device 100 may display details of the expected configuration of the network device. For example, the mobile device 100 may display a list of ports to which the network device is expected to connect, and before starting to search for ports, it may display how to connect them by flashing a message (title) "Expected" or "Expected Configuration". In various embodiments, the above steps can be performed sequentially or in parallel.

[0073] Figures 7A and 7B show another example of a user interface 720 for a mobile device 700 in AR mode, according to several embodiments. Figures 7A and 7B show images of the network device 720 rendered in real time on the display of the mobile device 700 in two different perspective views. The user interface 720 may include one or more virtual objects that overlay a portion of the displayed image of the network device 702. The virtual objects may include virtual ports 740 and 741, a virtual cable 742, an icon 750, and a virtual label 752 (e.g., text). In some cases, a port area 732 surrounding a virtual port is outlined with a virtual line 745. The position of the virtual objects can be configured to change in accordance with the change in depth by the mobile device using the spatial coordinate model described herein. In this way, the virtual objects can substantially maintain alignment with the corresponding objects in the captured image. Therefore, virtual ports 740 and 741, virtual cable 742, icon 750, and virtual label 752 may appear to move when the mobile device 700 is moved relative to the network device. One or more virtual objects may change based on the connection status of one or more corresponding ports. For example, virtual port 740 may have an outline of a first color (e.g., white) indicating that the corresponding port is connected to the device, and virtual port 741 may have an outline of a second color (e.g., blue) indicating that the corresponding port is not connected to the device.

[0074] Any AR device or method described herein may include a virtual object that shows the performance of a network device and / or a device connected to a network device. Figure 8 shows a magnified view of the user interface of a mobile device display while in AR mode, illustrating an example of a virtual object that conveys performance data. The virtual object may include a variety of selectable performance metrics, such as network device throughput862 and uptime864. When selectable performance metrics are selected, one or more virtual charts or graphs860 may be displayed. In Figure 8, graph 860 shows throughput values ​​related to a device connected to a network device. Throughput metrics may include data related to the device's throughput performance, such as Transmission Control Protocol (TCP) throughput or file transfer time. Uptime metrics may include the percentage of time the device is operational. Other metrics may include internet traffic usage and / or power usage of the network device and / or a device connected to a network device. In some cases, metrics may include the length of time one or more devices have been connected to the network device. In some embodiments, a tree graph shows various devices (e.g., primary, secondary, tertiary, etc.) that are connected to or have been connected to a network device. In some cases, the metrics may include usage data related to a specific user of the device. For example, a graph or chart showing the time a user spent using a social media application or website could be displayed.

[0075] The user interface may include selectable virtual icons that can be selected by the user by touch (e.g., using the touch display of a mobile device) and / or by other selectable methods such as an electronic pencil. The virtual icons may include port icons 870, power icons 872, performance icons 874, and settings icons 876. When port icon 870 is selected, various virtual ports and virtual port labels may become visible, as described herein. When power icon 872 is selected, virtual objects related to power and battery status and usage may become visible. When performance icon 874 is selected, virtual objects related to performance, such as throughput 862 and uptime 864, may become visible. When settings icon 876 is selected, virtual objects related to network setup and configuration may become visible.

[0076] As described herein, AR technology can be based on the use of a three-dimensional coordinate system (3D XYZ). Figures 9A–9E show embodiments of a 3D XYZ axis system used as the basis for the AR methods described herein according to several embodiments, where X represents width, Y represents height, and Z represents depth. The 3D XYZ system can ensure that the spatial data used to form AR objects is understandable (e.g., consistent). In some embodiments, a cross-platform software engine is used to establish the 3D XYZ system. For example, the Unity3D game engine developed by Unity Technologies, based in San Francisco, California, USA, can be used. A 3D-positioned point with width 1 (on the X axis), height 4 (on the Y axis), and depth 3 (on the Z axis) can be presented as Vector3{x:1,y:4,z;3} using the Unity3D system.

[0077] 3D spatial recordings can be captured during optical code scanning (see, for example, Figures 4C1–4C4 or Figure 5B). For example, an AR application on a mobile device (e.g., a phone, tablet, or headset) can use the mobile device's camera and / or compass system (e.g., accelerometer, gyroscope, and / or compass) to "capture" a 3D spatial recording of a network device and "remember" the user's rack configuration. This 3D spatial capture may be a one-time setup. Figure 9B shows an example of a 3D spatial recording of rack 910 of network device 902. Each spatial recording of network device 902 can be established as a set of vectors 909 in a 3D XYZ coordinate system. The 3D spatial recording corresponds to a group of normalized positions (e.g., Vector3) that hold its local position based on the tracker. The 3D spatial recording can be filtered as described herein. In one implementation of filtering, the orientation of network devices is typically fixed (e.g., having a base parallel to the ground), so the local position can be considered to be on a “panel” (e.g., a 2D surface) with virtually no rotation around the X-axis. One of the trackers can be designated as the origin of spatial recording, and the relative locations of the other trackers can be recorded relative to this origin. For example, the first optical code 904a (e.g., the first scanned) may be designated as having the origin, and the relative locations of the remaining optical codes 904 (e.g., those scanned later) can be recorded relative to the first optical code 904a. In some cases, the topmost optical code 904a may be designated as the origin. The coordinate system can be based on any unit of measurement. In some embodiments, the coordinate system is based on the metric system (e.g., meters). Note that in some embodiments, a Vector1 or Vector2 structure may be used instead of Vector3. However, in some cases, Vector3 may provide more functionality to accommodate more complex network device layouts.

[0078] Data can flow between the 3D XYZ platform and native code on mobile devices, such as Swift or Java. Figure 9C shows an example of JavaScript Object Notation (JSON) code for setting up a network device. Figure 9D shows an example of using a Hypertext Transfer Protocol (HTTP) request written in native code to use additional data (rack_name for racks, name and model number for network devices) in the user interface during the spatial recording capture process. Figure 9E shows exemplary JSON code for setting up multiple network devices.

[0079] Figures 10A–10D show flowcharts 1000 and corresponding architecture maps 1100 representing exemplary processes for setting up and using an AR application according to several embodiments. Each step in flowchart 1000 includes a display of the corresponding architecture component (e.g., Swift, Unity AR, network controller, or USW / LCM) in the corresponding map 1100. Map 1100 illustrates the interaction of the network controller, Swift, Unity AR, USW, and LCM components.

[0080] Figure 10A shows portions of flowchart 1000 and map 1100 corresponding to the initialization of an application using network information. Referring to flowchart 1000, at 1002, the user can use a mobile device to pull a network application programming interface (API) for the AR application from the network controller to the Swift component. At 1004, the network JSON is filtered using one or more filtering methods described herein, for example. At 1006, the network JSON data is parsed by the Unity AR component. At 1008, the spatial record is pulled by the Swift component. At 1010, if the spatial record is successfully pulled (yes), the process continues in Figure 10D; if the spatial record is not successfully pulled (no), the process continues in Figure 10B.

[0081] Figure 10B shows a portion of flowchart 1000 and map 1100 corresponding to the distribution of tracker seeds. Referring to flowchart 1000, if it is determined in Figure 10A that the spatial record has not been successfully pulled (no), then at 1012, a Mac-tracker pair is created by the Unity AR component and sent to the Swift component. At 1014, the Mac-tracker pair is posted to the network controller by the Swift component. At 1016, the Mac-tracker pair is distributed to the LCM and USW, and at 1018, the tracker is generated from the seed (for example, to generate a QR code).

[0082] Figure 10C shows portions of flowchart 1000 and map 1100 corresponding to a setup for storing spatial records (e.g., a one-time setup). Referring to flowchart 1000, at 1020, a tracker is scanned and used to write spatial records, which are then sent to a Swift component. At 1022, the spatial records are input to the network controller, and at 1024, the spatial records are received and stored by the network controller.

[0083] Figure 10D shows portions of flowchart 1000 and map 1100 corresponding to resuming normal use of mobile devices and using AR applications based on spatial recordings. Referring to flowchart 1000, at 1026, the spatial recording is pulled by a Swift component so that trackers on the LCM from the spatial recording (USW / LCM) can resume. Furthermore, the network controller responds to the spatial recording at 1030. At 1032, the spatial recording is parsed by the Unity AR component and at 1034 can scan any of the displayed trackers. At 1036, the AR overlay of the rack is resumed by the Unity AR component, building an AR user interface (1038) with auto-toggle interaction (1040) and rendering port details (1042).

[0084] While many of the examples described herein relate to network devices such as switches, routers, and wireless access points, the AR technologies described herein are not limited to these types of devices. For example, the AR technologies described herein can be used to view any type of computer or other device that can connect to one or more computer networks. In some cases, the AR technologies described herein can be used to render invisible connectivity aspects of mobile devices such as cell phones, tablets, laptops, or headsets. The AR technologies described herein can be used to render invisible connectivity aspects of wireless communication capabilities (i.e., not limited to hardware ports). For example, connectivity aspects related to one or more antennas (e.g., radio frequency or Bluetooth chips) can be rendered using AR objects.

[0085] In this specification, when a feature or element is referred to as being "on top of" another feature or element, it may be directly on the other feature or element, or there may be intervening features and / or elements. In contrast, when a feature or element is referred to as being "directly on top of" another feature or element, there are no intervening features or elements. Also, when a feature or element is referred to as being "connected," "attached," or "joined" to another feature or element, it will be understood that it may be directly connected, attached, or joined to the other feature or element, or there may be intervening features or elements. In contrast, when a feature or element is referred to as being "directly connected," "directly attached," or "directly joined" to another feature or element, there are no intervening features or elements. Features and elements described or illustrated in this manner are described or illustrated in relation to one embodiment, but may be applicable to other embodiments. Also, it will be understood by those skilled in the art that a reference to a structure or feature positioned "adjacent" to another feature may have a portion that overlaps with or is beneath the adjacent feature.

[0086] The terms used herein are intended solely to describe specific embodiments and are not intended to limit the invention. For example, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural form unless the context clearly indicates otherwise. As used herein, the terms “includes / equips” and / or “includes / equips” specify the presence of the described features, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, actions, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any combination of one or more of the related enumerated items and may be abbreviated as “ / ”.

[0087] Spatially relative terms such as “under,” “below,” “lower,” “over,” and “upper” are used herein to facilitate explanation and to describe the relationship of one element or characteristic to another, as shown in the drawings. It will be understood that spatially relative terms are intended to encompass different orientations of the device in use or operation, in addition to the orientations depicted in the drawings. For example, if the device in the drawing is inverted, an element described as “under” or “beneath” another element or characteristic will then be oriented “over” that other element or characteristic. Thus, the exemplary term “under” can encompass both upward and downward orientations. The device may be oriented in other ways (rotated 90 degrees or to other orientations), and the spatially relative descriptors used herein may be interpreted accordingly. Similarly, terms such as “upward,” “downward,” “vertical,” and “horizontal” are used herein for illustrative purposes only, unless otherwise specified.

[0088] The terms “first” and “second” may be used herein to describe various characteristics / elements (including steps), but unless the context otherwise indicates, these characteristics / elements should not be limited by these terms. These terms may be used to distinguish one characteristic / element from another. Thus, the first characteristic / element considered below may be called the second characteristic / element, and similarly, the second characteristic / element considered below may be called the first characteristic / element without departing from the teachings of the present invention.

[0089] In this specification, including those used in the examples, and where used in the specification and claims, all numbers, unless otherwise specified, can be read as if preceded by the words “approximately” or “about” even when the term is not explicitly stated. The words “approximately” or “about” can be used when describing magnitude and / or location to indicate that the described value and / or location is within a reasonable expected range of the value and / or location. For example, a number may have values ​​such as + / -0.1% of the stated value (or range of value), + / -1% of the stated value (or range of value), + / -2% of the stated value (or range of value), + / -5% of the stated value (or range of value), + / -10% of the stated value (or range of value), and so on. Any numerical range enumerated herein is intended to include all subranges incorporated therein.

[0090] While various exemplary embodiments have been described above, any of several modifications can be made to various embodiments without departing from the scope of the invention as described in the claims. For example, the order in which the various method steps described are performed may often be changed in alternative embodiments, and in other alternative embodiments, one or more method steps may be skipped entirely. Optional characteristics of various device and system embodiments may be included in some embodiments and not in others. Therefore, the foregoing description is provided primarily for illustrative purposes and should not be construed as limiting the scope of the invention as described in the claims.

[0091] The examples and illustrations contained herein illustrate, not as limitations but as examples, specific embodiments in which the subject matter may be implemented. As noted, other embodiments may be utilized and derived therefrom so that structural and logical substitutions and modifications can be made without departing from the scope of this disclosure. Such embodiments of the subject matter of the present invention, if one or more are disclosed, may be referred to individually or collectively in this specification by the term “invention” simply for convenience, without the intention of voluntarily limiting the scope of this application to any single invention or concept of invention. Thus, while specific embodiments have been illustrated and described herein, any arrangement made to achieve the same objective may be substituted for the specific embodiments shown. This disclosure is intended to cover all possible adaptations or variations of various embodiments. Combinations of the embodiments described above, and other embodiments not specifically described herein, will be apparent to those skilled in the art upon consideration of the above description.

Claims

1. A method for displaying network devices along with augmented reality using a mobile device, The mobile device scans for the network device identifier of each of the multiple network devices, and the mobile device visually indicates that the scan of the network device identifier was successful for each of the multiple network devices. Based on the scanned network device identifier, retrieve information relating to the connection status of at least one port among a plurality of ports of the plurality of network devices, wherein the retrieved information includes the connection status of at least one port among the plurality of ports. Based on the retrieved information, the connection status of a given port is determined. Determining the spatial relationship of the mobile device to at least one of the ports, A method for displaying the connection status of a given port on the display of a mobile device, wherein the display includes an image captured by the camera of the mobile device, and the connection status is displayed in the image as an overlay of a virtual port.

2. The method according to claim 1, wherein the connection state includes the identity of a device coupled to at least one of the plurality of ports of the network device.

3. The method according to claim 1, wherein searching for information relating to the connection status includes searching a database of devices coupled to at least one of the plurality of ports of the network device.

4. The method according to claim 1, wherein displaying the connection status includes displaying an icon of a device connected to at least one of the plurality of ports.

5. The method according to claim 1, wherein the connection status includes information of a device coupled to at least one of the plurality of ports.

6. The method according to claim 1, wherein displaying the connection status includes displaying a virtual label containing text indicating one or more characteristics of a device connected to at least one of the ports.

7. The method according to claim 1, wherein the overlay of the virtual port has a size and shape that substantially matches that of the image captured by the camera.

8. The method according to claim 1, wherein the retrieved information includes a library of virtual objects based on the scanned network device identifier.

9. The method according to claim 8, wherein the scanned network device identifier includes a radio frequency (RF) code, an optical code, a barcode, an ARTag code, an alphanumeric code, a numeric code, or a combination thereof.

10. The method according to claim 1, wherein the connection state includes whether at least one of the plurality of ports is connected to another device.

11. The method according to claim 1, wherein the connection state includes the operating state of a computer connected to at least one of the plurality of ports.

12. The method according to claim 1, wherein the mobile device includes a mobile phone, a tablet computer, a laptop computer, or a headset.

13. A method for displaying network devices along with augmented reality using a mobile device, The mobile device scans for the network device identifier of each of the multiple network devices, and the mobile device visually indicates that the scan of the network device identifier was successful for each of the multiple network devices. Based on the scanned network device identifier, retrieve information relating to the connection status of at least one port among a plurality of ports of the plurality of network devices, wherein the retrieved information includes the connection status of at least one port among the plurality of ports. Based on the retrieved information, the connection status of a given port is determined. Determining the spatial relationship of the mobile device to at least one of the ports, A method for displaying the connection status of a given port on the display of a mobile device, wherein the display includes an image captured by the camera of the mobile device, the connection status is displayed in the image as an overlay of a virtual port, and an icon of a device coupled to at least one of the plurality of ports is displayed adjacent to at least one of the plurality of ports.

14. A system for displaying network devices along with augmented reality using a mobile device, One or more processors, The computer implementation method is performed when the computer program instructions are executed by the one or more processors, and the computer implementation method is performed The mobile device scans for the network device identifier of each of the multiple network devices, and the mobile device visually indicates that the scan of the network device identifier was successful for each of the multiple network devices. Based on the scanned network device identifier, retrieve information relating to the connection status of at least one port among a plurality of ports of the plurality of network devices, wherein the retrieved information includes the connection status of at least one port among the plurality of ports. Based on the retrieved information, the connection status of a given port is determined. Determining the spatial relationship of the mobile device to at least one of the ports, A system that displays the connection status of a given port on the display of a mobile device, wherein the display includes an image captured by the camera of the mobile device, and the connection status is displayed in the image as an overlay of a virtual port.

15. The system according to claim 14, wherein the connection state includes the identity of a device coupled to at least one of the plurality of ports of the network device.

16. The system according to claim 14, wherein searching for information related to the connection status includes searching a database of devices coupled to at least one of the plurality of ports of the network device.

17. The system according to claim 14, wherein displaying the connection status includes displaying an icon of a device connected to at least one of the plurality of ports.

18. The system according to claim 14, wherein the connection status includes information of a device coupled to at least one of the plurality of ports.

19. The system according to claim 14, wherein the overlay of the virtual port has a size and shape that substantially matches that of the image captured by the camera.

20. The system according to claim 14, wherein the retrieved information includes a library of virtual objects based on the scanned network device identifier.

Images