Visual light-based instructions to a robot system

Abstract

According to one embodiment, a method, a computer system, and a computer program product for optical-based navigation of a robotic device. Embodiments may include detecting an optical beam. Embodiments may include identifying a light source location and an end point location of the optical beam. The end point location includes where the optical beam intersects a surface. Embodiments may include receiving an audio command to proceed to the end point location. Embodiments may include instructing a mobile robotic device to move directly to the end point location. In response to the mobile robotic device reaching the end point location, embodiments may include instructing the mobile robotic device to perform an activity there.

Description

Technical Field

[0001] The present invention generally relates to the field of computing, and more particularly to robotics.

[0002] Robotics is an interdisciplinary field of computer science and engineering that includes the design, construction, operation, application of robots, and the computer systems for their control, sensory feedback, and information processing. Robotics aims to design machines (i.e., robots) that autonomously or semi-autonomously perform physical tasks on behalf of humans. These machines can replace humans by replicating human actions and can be used for many purposes in many situations. Generally, robots perform repetitive tasks or tasks that are too dangerous for humans to perform safely. For example, robots can be used in environments where humans are vulnerable or unable to survive. Robots can be constructed in many forms, but these machines interact with the physical world using a variety of sensors, actuators, and data processing techniques. Robots may be guided by external inputs or control devices, may have built-in guidance control, or may utilize a combination of external and internal inputs for guidance.

Summary of the Invention

[0003] According to one embodiment, a method, a computer system, and a computer program product for optical-based navigation of a robotic device. The embodiment may include detecting an optical beam. The embodiment may include identifying a light source location and an end point location of the optical beam. The end point location includes the location where the optical beam intersects a surface. The embodiment may include receiving an audio command to proceed to the end point location. The embodiment may include instructing a mobile robotic device to move directly to the end point location. In response to the mobile robotic device reaching the end point location, the embodiment may include instructing the mobile robotic device to perform an activity there.

[0004] Next, preferred embodiments of the present invention will be described by way of mere example with reference to the following drawings.

Brief Description of the Drawings

[0005]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0006] The drawings are provided to clarify and facilitate understanding of the present invention by those skilled in the art in conjunction with the detailed description, and the various features of the drawings are not to scale.

[0007] Although detailed embodiments of the claimed structures and methods are disclosed herein, it is to be understood that the disclosed embodiments merely exemplify the claimed structures and methods that may be embodied in various forms. However, the present invention may be implemented in many different forms and should not be construed as limited to the exemplary embodiments described herein. Well-known features and details of techniques may be omitted herein to avoid unnecessarily obscuring the presented embodiments.

[0008] Unless the context clearly dictates otherwise, it should be understood that the singular forms "a", "an", and "the" include plural referents. Thus, for example, a reference to "a component surface" includes a reference to one or more such surfaces unless the context clearly dictates otherwise.

[0009] Embodiments of the present invention generally relate to the field of computing, and more particularly to robotics. The exemplary embodiments described below provide, in particular, a system, method, and program product for instructing one or more robotic devices to travel along a path identified by a light beam or to proceed to a goal position. Accordingly, the present embodiments have the ability to improve the technical field of robotics by dynamically identifying a direction of movement (e.g., a path) or a destination location or both using a light beam and instructing a robotic device to move according to the identified direction of movement or the identified destination location or both.

[0010] As described above, robotics is an interdisciplinary field of computer science and engineering that includes computer systems for the design, construction, operation, application, and control, sensory feedback, and information processing of robots. Robotics aims to design machines (i.e., robots) that perform physical tasks autonomously or semi-autonomously on behalf of humans. These machines can replace humans by replicating human actions and can potentially be used for many purposes in many situations. Generally, robots perform repetitive tasks or tasks that are too dangerous for humans to perform safely. For example, robots can be used in environments where humans are vulnerable or unable to survive. Robots can be constructed in many forms, but these machines utilize a variety of sensors, actuators, and data processing techniques to interact with the physical world. Robots may be guided by an external input or control device, may have an embedded guidance control, or may utilize a combination of external and internal inputs for guidance.

[0011] For a mobile robot device, since it may be necessary to avoid dangerous situations such as collisions and non-safe states, the ability to navigate its own environment is important. Similarly, when the mobile robot device has an objective (e.g., a task to be performed) related to one or more specific locations in the surrounding environment, it is important for the robot device to identify those locations and move towards them. In a human-robot ecosystem, a human user may need to indicate / instruct the mobile robot device about a moving direction, a route, or a target location or a combination thereof, whereby the robot device can navigate according to the indicated direction, move back and forth along the route, or navigate to the target location, or a combination thereof, and execute some activity. Consider a scenario where disaster area mitigation activities, construction activities, or rescue operations are required to be performed by a mobile robot device during dark hours. In such a scenario, the human user may need to indicate from a distance a passable route, a moving direction, or a destination that the robot device should follow or move towards. However, due to geographical conditions including distance and darkness, it may be difficult for the user to indicate to the robot device a passable route, a moving direction, or a destination. Therefore, it may be essential to provide a system for providing visual light-based navigation instructions to one or more robot devices. Accordingly, embodiments of the present invention may advantageously, among other things, utilize a light beam to identify a moving direction, a route, or a target location or a combination thereof, and accordingly instruct the navigation and activities of one or more robot devices. The present invention does not require that all advantages must be incorporated into all embodiments of the present invention.

[0012] According to at least one embodiment, one or more activities to be performed by one or more mobile robot devices in a human-robot ecosystem can be identified based on a context analysis of the ecosystem's environment. The movement path, activity area, or both can be identified based on their illumination via at least one light beam from an external light source. One or more mobile robot devices can traverse the identified movement path (i.e., follow the direction of the light beam) or proceed to the identified activity area (i.e., proceed to where the light beam intersects / contact / lands on a surface) to perform one or more activities. According to at least one embodiment, the light beam may be transmitted from a user or from a mobile robot device in a human-robot ecosystem. According to at least one other embodiment, the type of activity performed within the activity area may be based on the color of the light beam illuminating the activity area. According to at least one further embodiment, the boundary of the activity area may be based on a closed-loop contour drawn using the light beam.

[0013] The present invention can be a system, method, or computer program product, or a combination thereof, at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple computer-readable storage media) having computer-readable program instructions for causing a processor to implement aspects of the present invention.

[0014] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction-executing device. The computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, but is not limited thereto. A non-exhaustive listing of more specific examples of computer-readable storage media includes portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick (registered trademark), floppy (registered trademark) disk, mechanically encoded devices such as punch cards or raised structures engraved in grooves in which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be construed as being a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through an electrical wire.

[0015] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers or combinations thereof. The network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within the individual computing / processing device.

[0016] The computer-readable program instructions for carrying out the operations of the present invention may be source code or object code written in any combination of one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or any combination of object-oriented programming languages such as Smalltalk®, C++, and procedural programming languages such as the “C” programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer as a stand-alone software package, partly on the user's computer, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may utilize the state information of the computer-readable program instructions to execute the computer-readable program instructions to individually configure the electronic circuit in order to carry out aspects of the present invention.

[0017] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0018] These computer-readable program instructions, when executed via a processor of a computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of a flowchart, a block diagram, or both, and may thus provide to the processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus to create a machine. These computer-readable program instructions may also be stored in a computer-readable storage medium that includes a manufactured article including instructions for implementing the aspects of the functions / acts specified in one or more blocks of a flowchart, a block diagram, or both, and can direct a computer, a programmable data processing apparatus, or other device or combinations thereof to function in a particular manner.

[0019] The computer-readable program instructions may also be loaded onto a computer, other programmable apparatus, or other device to create a computer-implemented process such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of a flowchart, a block diagram, or both, and cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device.

[0020] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, segment, or portion of instructions that include one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions represented by the blocks may occur in a different order than shown in the drawings. For example, two blocks shown in succession may actually be executed simultaneously, or substantially simultaneously, or the blocks may sometimes be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagram or flowchart diagram, or both, and combinations of blocks of the block diagram or flowchart diagram, or both, can be implemented by a dedicated hardware-based system that performs the specified function or action, or that executes a combination of dedicated hardware and computer instructions.

[0021] The exemplary embodiments described below provide a system, method, and program product for identifying a direction of movement or a destination location via a light beam and commanding one or more mobile robot devices to navigate according to the identified direction of movement or move to the identified destination location.

[0022] Referring to FIG. 1, an exemplary network computer environment 100 according to at least one embodiment is illustrated. The network computer environment 100 may include client computing devices 102, servers 112, and mobile robot devices 118 interconnected via a communication network 114. According to at least one implementation, the network computer environment 100 may include a plurality of client computing devices 102, servers 112, and mobile robot devices 118, and only one of each is shown for simplicity of illustration. Additionally, in one or more embodiments, the client computing devices 102, servers 112, and mobile robot devices 118 may each host an optical-based navigation (LBN) program 110A, 110B, 110C. In one or more other embodiments, the LBN programs 110A, 110B, 110C may be partially hosted on the client computing devices 102, servers 112, and mobile robot devices 118 so as to enable functionality separation between the devices.

[0023] The communication network 114 may include various types of communication networks such as a personal area network (PAN), wide area network (WAN), local area network (LAN), telecommunications network, wireless network, wireless ad hoc network (i.e., wireless mesh network), public switched network, or satellite network, or combinations thereof. The communication network 114 may include wired or wireless communication links, or connections such as optical fiber cables. It will be understood that FIG. 1 merely provides an example of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many changes may be made to the depicted environment based on design and implementation requirements.

[0024] According to an embodiment of the present invention, the client computing device 102 can include a processor 104 and a data storage device 106, host and execute a software program 108 and an LBN 110A, and communicate with the server 112 and the mobile robot device 118 via a communication network 114. The client computing device 102 can be, for example, a mobile device, a smartphone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device that can execute a program and access a network. As will be discussed with reference to FIG. 3, the client computing device 102 can include internal components 402a and external components 404a, respectively.

[0025] According to an embodiment of the present invention, the server computer 112 can be a laptop computer, a netbook computer, a personal computer (PC), a desktop computer, or any programmable electronic device or any network of programmable electronic devices that can host and execute an LBN program 110B and a database 116 and communicate with the client computing device 102 and the mobile robot device 118 via a communication network 114. As will be discussed with reference to FIG. 3, the server computer 112 can include internal components 402b and external components 404b, respectively. The server 112 can also operate in a cloud computing service model such as SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). The server 112 may be deployed in a cloud computing deployment model such as a private cloud, a community cloud, a public cloud, or a hybrid cloud.

[0026] The mobile robot device 118 can execute one or more user-defined tasks, is connected to the communication network 114, can transmit and receive data with the client computing device 102 and the server 112, and can host and execute the LBN program 110C, and may be an IoT (Internet of Things)-compatible robot device. In one or more embodiments, examples of the mobile robot device 118 may include an autonomous flight platform (e.g., a drone, an unmanned aerial vehicle), an autonomous marine platform, a mobile industrial robot, a walking robot, a Spot robot, or other ground mobile robots (e.g., an autonomous truck vehicle, an autonomous wheeled vehicle). Additionally, in embodiments of the present invention, the mobile robot device 118 may include one or more cameras, an electro-optical sensor for detecting light (i.e., electromagnetic radiation from infrared to ultraviolet wavelengths), a laser-based rangefinder, a microphone, and an electrical light source (e.g., a light-emitting diode (LED), a laser, an ultraviolet lamp). Further, the robot device 118 may be capable of utilizing image processing, robotic mapping, three-dimensional (3D) mapping, and a satellite navigation system (e.g., a global positioning system (GPS)). Although only one mobile robot device is shown in FIG. 1, in at least one other embodiment, the environment 100 of FIG. 1 may include a plurality of mobile robot devices 118 that can function individually or collaboratively to execute one or more user-defined tasks and can transmit and receive data between each device via the communication network 114. As will be discussed with reference to FIG. 3, the mobile robot device 118 can each include internal components 402c and external components 404c.

[0027] According to this embodiment, the LBN programs 110A, 110B, 110C can detect a light beam and identify its light source location, identify the movement and direction of the end point of the light beam or the stationary position where the light beam stays (i.e., the place where the light beam intersects / contact / lands on a certain surface) or both, identify the movement path or activity area or both based on the characteristics of the light beam, instruct a mobile robot device (e.g., the mobile robot device 118) to execute some activity by following the identified movement path or proceeding to the identified activity area, specify the type of activity executed by the robot device based on the detected color of the light beam, capture the user's voice commands, perform image analysis of the environment of the robot device, robotic mapping, and 3D mapping, and deploy additional robot devices to the identified activity area. In at least one embodiment, the LBN programs 110A, 110B, 110C can request the user for opt-in to system usage at the startup or installation of the LBN programs 110A, 110B, 110C. The LBN method will be described in more detail below with reference to FIG. 2.

[0028] Next, referring to FIG. 2, an operational flowchart is depicted for instructing the navigation of a robotic device via a light source in an optical-based navigation process 200 according to at least one embodiment. At 202, the LBN programs 110A, 110B, 110C detect a light beam and identify its light source location. In a human-robot ecosystem, a human user can use a light beam from an electrical light source to instruct a robotic device, such as a mobile robotic device 118 within the ecosystem, of a travel path or a destination location. The electrical light source can create light beams of various colors. The LBN programs 110A, 110B, 110C can detect a light beam via input from one or more cameras or other optical sensors disposed on the robotic device and identify its color. Moreover, the LBN programs 110A, 110B, 110C can perform image analysis of feeds from one or more cameras or other optical sensors in combination with 3D mapping technology of the robotic device's environment to identify the physical location of the light source of the light beam, the travel path of the light beam, the physical location of the place where the light beam stops (i.e., the place where it intersects on the surface where the end point of the light beam is), and the direction of the light beam from its light source location. Additionally, the LBN programs 110A, 110B, 110C may identify the physical location of the robotic device. The identified physical location can be described using the 3D coordinates of a 3D map of the robotic device's environment. According to at least one embodiment, the LBN programs 110A, 110B, 110C can access a 3D mapping software application (e.g., software program 108) when performing 3D mapping of the robotic device's environment. Further, the LBN programs 110A, 110B, 110C can receive context information of the environment (e.g., terrain, topology, floor plan) from another software program (e.g., software program 108) or from the user for reference when applying 3D mapping technology.According to at least one other embodiment, the LBN programs 110A, 110B, 110C may display a created 3D map of the environment of the robotic device and the 3D coordinates of the identified physical location via the screen of the user's computing device.

[0029] According to at least one embodiment, an electric light source may be disposed on a robotic device (e.g., mobile robotic device 118), and a light beam from the electric light source may be used to indicate a travel path or a destination location to the robotic device or to another robotic device within the ecosystem. The electric light source can create light beams of various colors. In such an embodiment, the LBN programs 110A, 110B, 110C may identify the physical location of the robotic device as the physical location of the light source of the light beam.

[0030] In at least one other embodiment where a human user uses a light beam from an electric light source co-located with the user to indicate a travel path or a destination location to a robotic device, the LBN programs 110A, 110B, 110C may utilize the GPS coordinates of a computing device also co-located with the user to identify the physical location of the light source of the light beam. For example, the user's smartphone may be paired with the robotic device (e.g., mobile robotic device 118) via Bluetooth (Bluetooth and all trademarks and logos based on Bluetooth are trademarks or registered trademarks of Bluetooth SIG, Inc. or its related companies or both), and the LBN programs 110A, 110B, 110C may utilize the GPS coordinates of the smartphone to identify the physical location of the light source of the light beam. The GPS function of the robotic device may also be utilized by the LBN programs 110A, 110B, 110C to identify the physical location (e.g., coordinates) of the robotic device.

[0031] Next, at 204, the LBN programs 110A, 110B, 110C receive voice commands from a human user that describe the actions required of the robotic device in response to the detection of the light beam. The LBN programs 110A, 110B, 110C can, in part, control the robotic device based on the user's voice commands captured by the user's computing device. As described above, the user's smartphone may be paired with the robotic device (e.g., the mobile robotic device 118) (e.g., via the network 114), such that the LBN programs 110A, 110B, 110C can capture the voice commands and relay them to the robotic device. Alternatively, the LBN programs 110A, 110B, 110C may utilize a microphone of the robotic device (e.g., a long-range microphone) to capture the user's voice commands. The LBN programs 110A, 110B, 110C can perform speech-to-text processing or natural language processing (NLP) when analyzing the received voice commands. The user's voice commands can include commands for the robotic device to follow the identified travel path of the detected light beam or to move to the identified physical location where the light beam stops.

[0032] In 206, the LBN programs 110A, 110B, and 110C determine whether the received user voice command contains a command for the robot device to follow the identified movement path of the detected light beam. When making this determination, the LBN programs 110A, 110B, and 110C use NLP technology to analyze the extracted text of the received voice command and identify one or more keywords / phrases such as "Follow", "Follow the movement of the light beam", "Go to the end point of the light beam", or "Proceed to the place where the light beam intersects the surface". In response to determining that the received user voice command contains a command for the robot device to follow the identified movement path of the detected light beam (step 206, "Yes" branch), the light-based navigation creation process 200 can proceed to step 210. In response to determining that the received user voice command does not contain a command for the robot device to follow the identified movement path of the detected light beam (step 206, "No" branch), the light-based navigation process 200 can proceed to step 208.

[0033] According to at least one embodiment, in 206, instead of or in addition to receiving a voice command from the user, the LBN programs 110A, 110B, and 110C may identify whether the end point of the detected light beam is moving in order to determine the action the robot device should take as a response. For example, if the LBN programs 110A, 110B, and 110C determine that the end point of the detected light beam is moving (i.e., the physical location where the light beam stops is changing) (step 206, "Yes" branch), the light-based navigation creation process 200 can proceed to step 210. However, if the LBN programs 110A, 110B, and 110C determine that the end point of the detected light beam is not moving (i.e., the physical location where the light beam stops is stationary) (step 206, "No" branch), the light-based navigation creation process 200 can proceed to step 208.

[0034] According to at least one other embodiment, in 206, instead of receiving a voice command from the user or in addition to receiving a voice command from the user, the LBN programs 110A, 110B, 110C may identify the color of the detected light beam to determine the action that the robotic device should take as a response. For example, if the LBN programs 110A, 110B, 110C determine that the color of the detected light beam is a first specified color (e.g., a color associated with an action of following the detected light beam to its moving end point) (step 206, "yes" branch), the light-based navigation creation process 200 can proceed to step 210. However, if the LBN programs 110A, 110B, 110C determine that the color of the detected light beam is a second specified color (e.g., a color associated with an action of proceeding directly to the stationary position where the detected light beam stops) (step 206, "no" branch), the light-based navigation creation process 200 can proceed to step 208.

[0035] Next, at 208, the LBN programs 110A, 110B, 110C command a robotic device (e.g., mobile robotic device 118) to proceed directly to the identified location where the detected light beam stops (i.e., the location where the light beam intersects a surface) and perform some activity. As described above, the identified location can be described using the 3D coordinates of a 3D map of the robotic device's environment by the LBN programs 110A, 110B, 110C. The activity performed by the robotic device can be specified based on the identified color of the light beam. Different light beam colors can be coded / associated with different activities / tasks to be performed by the robotic device. The association of color and activity is set by the user for access by the LBN programs 110A, 110B, 110C via the network 114 when analyzing the color of the light beam and may be stored in the data storage device 106 or the database 116. Further, according to at least one embodiment, the LBN programs 110A, 110B, 110C may identify the boundaries of the activity area at the identified location based on an image analysis of a closed-loop contour drawn using the light beam. For example, the user may narrow or widen the focus of the light beam and draw the contour of the boundary of the activity area using the light beam, or may define the boundary of the activity area based on the diameter of the light beam at the position where the light beam stops. As another example, the contour of the boundary of the activity area can be drawn or defined using a light beam from an electrical light source disposed on the robotic device. In any example, a voice command may be received by the LBN programs 110A, 110B, 110C to command the robotic device to enter the activity area boundary identification mode using the image analysis of the light beam. Alternatively, the LBN programs 110A, 110B, 110C may command the robotic device to enter the activity area boundary identification mode based on the identification of a specific light beam color.

[0036] According to at least one other embodiment, the LBN programs 110A, 110B, 110C may identify an activity area based on the convergence of a plurality of light beams. For example, when multiple users are each using a respective light beam to instruct a robotic device of a target location (i.e., the activity area), the LBN programs 110A, 110B, 110C can identify the activity area as the physical location where the end points of the respective light beams converge. As another example, when a light beam from an electrical light source located on a robotic device is used, together with at least one other light beam from a user or another robotic device within the ecosystem, to indicate a target location, the LBN programs 110A, 110B, 110C can identify the activity area as the physical location where the end points of the light beams converge.

[0037] According to at least one embodiment, the speed at which the LBN programs 110A, 110B, 110C instruct one or more robotic devices to proceed directly to an identified location where a detected light beam remains may be based on the intensity (e.g., luminance level / value) of the detected light beam identified by the LBN programs 110A, 110B, 110C. For example, the LBN programs 110A, 110B, 110C can instruct a robotic device to proceed at a faster speed to the identified activity area in response to the identified intensity (e.g., luminance value) of the detected light beam being above a threshold value. Similarly, when the identified intensity is below the threshold value, it can be instructed to slow down. Moreover, when multiple light beams are being used to identify multiple activity areas, the LBN programs 110A, 110B, 110C can prioritize (i.e., assign a priority value to) the identified activity areas based on the intensity of each identified light beam, and instruct one or more robotic devices within the ecosystem to proceed directly to the identified activity area having the highest priority value, or in the case of multiple robotic devices, to proceed to the identified activity areas having the top k priority values.

[0038] At 210, the LBN programs 110A, 110B, 110C command the robot device to follow the identified movement path of the detected light beam and to perform some activity while following the path. The LBN programs 110A, 110B, 110C may calculate the 3D coordinates of the identified movement path by repeatedly calculating the 3D coordinates of the end point of the light beam moving along the movement path. The activity that the robot device performs while following the path can be specified based on the identified color of the detected light beam. Different light beam colors can be coded / associated with different activities / tasks to be performed by the robot device. The association between color and activity can be set by the user for access by the LBN programs 110A, 110B, 110C via the network 114 when analyzing the color of the light beam and stored in the data storage device 106 or the database 116. According to at least one embodiment, the speed at which the LBN programs 110A, 110B, 110C command one or more robot devices to follow the identified movement path of the detected light beam may be based on the intensity (e.g., luminance value) of the detected light beam identified by the LBN programs 110A, 110B, 110C. For example, the LBN programs 110A, 110B, 110C can command the robot device to proceed to follow the identified movement path at a faster speed in response to the identified intensity of the detected light beam being above a threshold value. Similarly, when the identified intensity is below the threshold value, it can be commanded to slow down the speed.

[0039] According to at least one embodiment, based on the context or type of activity being performed by a robotic device, the LBN programs 110A, 110B, 110C can command the robotic device to generate a light beam via an integrated electric light source and to direct a movement path or destination location to one or more other robotic devices in the ecosystem. Additionally, the LBN programs 110A, 110B, 110C can generate a light beam via the integrated electric light source and command the robotic device to direct a movement path or destination location to one or more other robotic devices based on the identified light beam direction received from a user or other robotic devices in the ecosystem.

[0040] According to at least one embodiment, the LBN programs 110A, 110B, 110C store in the data storage device 106 or the database 116 not only the context information of the types of activities being executed within the ecosystem for later reference as past learning, but also the context information of the environment of the human-robot ecosystem. The context information of the executed activities may also include the profile information of each mobile robot device within the ecosystem, including the type and technical capabilities of the robot. Such profile information may be created or updated by the LBN programs 110A, 110B, 110C and stored in the data storage device 106 or the database 116. The profile information of the robot device can also be initially input by the system administrator. When the current human-robot ecosystem and the current activities are presented, the LBN programs 110A, 110B, 110C can identify whether additional robot devices of the same or different types, or with the same or different technical capabilities, are needed to perform some activity within the ecosystem based on past learning from past ecosystems and activities. If it is determined that additional robot devices are needed, the LBN programs 110A, 110B, 110C can deploy one or more additional robot devices to the current ecosystem, instruct one or more robot devices already in the ecosystem to generate a light beam via an integrated electric light source, and instruct the one or more additional robot devices to a travel path or destination location.

[0041] FIG. 2 merely provides an illustration of one implementation form and is not meant to impose any limitation on how various implementation forms can be implemented. Many changes can be made to the depicted environment based on design and implementation requirements.

[0042] FIG. 3 is a block diagram 400 of internal and external components of the client computing device 102, server 112, and mobile robot device 118 depicted in FIG. 1, according to an embodiment of the present invention. It should be understood that FIG. 3 merely provides an example of one implementation form and does not imply any limitation with respect to the environment in which various implementation forms can be implemented. Many changes can be made to the depicted environment based on design and implementation requirements.

[0043] Data processing systems 402, 404 represent any electronic device capable of executing machine-readable program instructions. The data processing systems 402, 404 can represent smartphones, computer systems, PDAs, or other electronic devices. Examples of computing systems, environments, or configurations or combinations thereof that can be represented by the data processing systems 402, 404 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, IoT devices, edge devices, and distributed cloud computing environments including any of the above systems or devices.

[0044] The client computing device 102, the server 112, and the mobile robot device 118 may each include a respective set of internal components 402a, b, c and external components 404a, b, c as shown in FIG. 3. Each set of internal components 402 includes, on one or more buses 426, one or more processors 420, one or more computer-readable RAMs 422, and one or more computer-readable ROMs 424, as well as one or more operating systems 428 and one or more computer-readable tangible storage devices 430. The one or more operating systems 428, the software program 108 and the LBN program 110A within the client computing device 102, the LBN program 110B within the server 112, and the LBN program 110C are stored in one or more of the respective computer-readable tangible storage devices 430 for execution by one or more of the respective processors 420 via one or more of the respective RAMs 422 (typically including cache memory). In the embodiment shown in FIG. 3, each of the computer-readable tangible storage devices 430 is a magnetic disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices 430 is a semiconductor storage device such as a ROM 424, an EPROM, a flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0045] Each set of internal components 402a, b, c also includes an R / W drive or interface 432 for reading from and writing to one or more portable computer-readable tangible storage devices 438, such as CD-ROMs, DVDs, memory sticks (registered trademark), magnetic tapes, magnetic disks, optical disks, or semiconductor memory devices. Software programs, such as LBN programs 110A, 110B, 110C, are stored on one or more of the respective portable computer-readable tangible storage devices 438, read via the respective R / W drives or interfaces 432, and can be loaded onto the respective hard drives 430.

[0046] Each set of internal components 402a, b, c also includes a network adapter or interface 436 such as a TCP / IP adapter card, a wireless Wi-Fi interface card, a wireless Bluetooth® interface card, or a 3G or 4G wireless interface card, or other wired or wireless communication links. The software program 108 and the LBN program 110A within the client computing device 102, the LBN program 110B within the server 112, and the LBN program 110C within the mobile robot device 118 can be downloaded from an external computer to the client computing device 102, the server 112, and the mobile robot device 118 via a network (e.g., the Internet, a local area network, or other wide area network) and their respective network adapters or interfaces 436. From the network adapter or interface 436, the software program 108 and the LBN program 110A within the client computing device 102, the LBN program 110B within the server 112, and the LBN program 110C within the mobile robot device 118 are loaded onto their respective hard drives 430. The network can include copper wire, optical fiber, wireless transmission, routers, firewalls, switches, gateway computers or edge servers, or combinations thereof.

[0047] Each of the sets of external components 404a, b, c can include a computer display monitor 444, a keyboard 442, and a computer mouse 434. The external components 404a, b, c can also include a touch screen, a virtual keyboard, a touch pad, a pointing device, and other human interface devices. Each of the sets of internal components 402a, b, c also includes a device driver 440 for interfacing with the computer display monitor 444, the keyboard 442, and the computer mouse 434. The device driver 440, the R / W drive or interface 432, and the network adapter or interface 436 include hardware and software (stored in the storage device 430 or the ROM 424 or both).

[0048] Although the present disclosure includes a detailed description of cloud computing, it should be understood in advance that the implementation of the teachings presented herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0049] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), and these resources can be rapidly provisioned and released with minimal management effort or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0050] The characteristics are as follows. On-demand self-service: Cloud users can automatically and unilaterally provision computing capabilities such as server time and network storage as needed, without the need for human interaction with the service provider. Broad network access: The capabilities are available over the network and can be accessed using standard mechanisms, facilitating use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs). Resource pooling: The provider's computing resources are pooled and provided to multiple users using a multi-tenant model, and various physical and virtual resources are dynamically assigned and re-assigned according to demand. Although users generally have a sense of location independence in that they are usually unaware of and do not manage the exact location of the resources provided, at a higher level of abstraction, the location (e.g., country, state, or data center) can be specified. Rapid elasticity: The capabilities can be provisioned quickly and flexibly, sometimes automatically, scale out rapidly, and be released quickly and scale in rapidly. The capabilities available for provisioning often appear to users to be purchasable in any quantity at any time without limit. Measured service: The cloud system automatically controls and optimizes resource use at an appropriate level of abstraction for the type of service (e.g., storage, processing, bandwidth, and active user accounts). The amount of resource use can be monitored, controlled, and reported, providing transparency to both the provider and the user of the service being utilized.

[0051] The service model is as follows. SaaS (Software as a Service): The capabilities provided to users are the use of the provider's applications running on cloud infrastructure. Those applications can be accessed from various client devices via a thin-client interface such as a web browser (e.g., web-based email). Users do not manage or control the underlying cloud infrastructure, which includes the network, servers, operating systems, storage, or individual application features, except for limited user-specific application configuration settings. PaaS (Platform as a Service): The capabilities provided to users are to deploy the applications created or obtained by the users, which are created using programming languages and tools supported by the provider, onto the cloud infrastructure. Users do not manage or control the underlying cloud infrastructure, which includes the network, servers, operating systems, or storage, but can control the deployed applications and, in some cases, the configuration of the application hosting environment. IaaS (Infrastructure as a Service): The capabilities provided to users are the provisioning of processing, storage, network, and other basic computing resources, and users can deploy and run any software that can include operating systems and applications. Users do not manage or control the underlying cloud infrastructure, but can control the operating systems, storage, deployed applications, and, in some cases, have limited control over selected network components (e.g., host firewalls).

[0052] The deployment models are as follows. Private Cloud: The cloud infrastructure is operated only for an organization. It can be managed by that organization or a third party and can exist on-premises or off-premises. Community Cloud: The cloud infrastructure is shared by multiple organizations and supports a specific community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-premises or off-premises. Public Cloud: The cloud infrastructure is made available to the general public or a large industry group and is owned by an organization that sells cloud services. Hybrid Cloud: The cloud infrastructure remains a unique entity but is a composite of two or more clouds (private, community, or public) joined by standardized or proprietary technologies (e.g., cloud bursting for load distribution between clouds) that enable data and application portability.

[0053] The cloud computing environment is a service-oriented environment that emphasizes statelessness, loose coupling, modularity, and semantic interoperability. At the center of cloud computing is an infrastructure with a network of interconnected nodes.

[0054] Referring now to FIG. 4, an exemplary cloud computing environment 50 is shown. As illustrated, cloud computing environment 50 includes one or more cloud computing nodes 100 that can communicate with a local computing device (e.g., a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N, or combinations thereof) used by a cloud user. The nodes 100 can communicate with one another. These can be physically or virtually grouped (not shown) within one or more networks such as the private cloud, community cloud, public cloud, or hybrid cloud, or combinations thereof, described above. Thereby, cloud computing environment 50 can provide an infrastructure, platform, or SaaS, or combinations thereof, for which cloud users need not maintain resources on a local computing device. The types of computing devices 54A - 54N shown in FIG. 4 are for illustration only, and it is understood that cloud computing node 100 and cloud computing environment 50 can communicate with any type of computer controlled device via any type of network or network addressable connection (e.g., a connection using a web browser) or both.

[0055] Next, referring to FIG. 5, a set of functional abstraction layers 600 provided by cloud computing environment 50 is shown. It should be understood in advance that the components, layers, and functions shown in FIG. 6 are for illustration only and that embodiments of the present invention are not limited thereto. As illustrated, the following layers and corresponding functions are provided.

[0056] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include mainframe 61, RISC (Reduced Instruction Set Computer) architecture-based server 62, server 63, blade server 64, storage device 65, and network and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0057] The virtualization layer 70 provides an abstraction layer that can provide virtual entities, such as virtual server 71, virtual storage 72, virtual network 73 including a virtual private network, virtual applications and operating systems 74, and virtual clients 75.

[0058] In one example, the management layer 80 can provide the functions described below. Resource provisioning 81 performs dynamic procurement of computing resources and other resources utilized to execute tasks within a cloud computing environment. Measurement and pricing 82 performs cost tracking when resources are utilized within a cloud computing environment, and billing or invoicing for the utilization of those resources. In one example, these resources can include application software licenses. Security performs identity verification of cloud users and tasks, and protects data and other resources. The user portal 83 provides access to the cloud computing environment to users and system administrators. Service level management 84 performs allocation and management of cloud computing resources to meet the required service levels. Service level agreement (SLA) planning and execution 85 performs advance preparation and procurement of cloud computing resources for which future demands are anticipated, in accordance with the SLA.

[0059] The workload layer 90 provides examples of functionality that can be utilized in a cloud computing environment. Examples of workloads and functions that can result from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analysis processing 94, transaction processing 95, and optical-based navigation 96. Optical-based navigation 96 may relate to instructing a robotic device of a travel route or destination location using a light beam from an electrical light source.

[0060] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive and is not limited to the disclosed embodiments. Many changes and modifications will become apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, the practical application, or a technical improvement found in the marketplace, or to enable other skilled artisans to understand the embodiments disclosed herein.

Claims

1. Detecting an optical beam; Identifying a light source location and an end point location of the optical beam, wherein the identifying includes the end point location including a location where the optical beam intersects a surface where the optical beam is located; Receiving an audio command to proceed to the end point location; Instructing a mobile robot device to move directly to the end point location; In response to the mobile robot device reaching the end point location, instructing the mobile robot device to perform an activity there A computer-implemented method comprising.

2. Identifying a travel path of the optical beam; Receiving an audio command to follow the travel path; Instructing the mobile robot device to follow the travel path and performing the activity while following the travel path The method according to claim 1, further comprising.

3. Identifying a boundary of an activity area based on image analysis of a closed-loop contour drawn using the optical beam, wherein the identifying includes the activity area including the end point location of the optical beam The method according to claim 1 or 2, further comprising.

4. Identifying a plurality of optical beams; Identifying a boundary of an activity area based on focusing of at least two of the plurality of optical beams, wherein the identifying includes the activity area including the end point location of the optical beam The method according to any one of claims 1 to 3, further comprising.

5. The method according to any one of claims 1 to 4, wherein the type of the activity is specified based on the color of the detected optical beam.

6. The method according to claim 2, wherein a speed at which the mobile robot device moves directly to the end point location or a speed at which the mobile robot device follows the travel path of the optical beam is based on a luminance value of the detected optical beam.

7. The method according to any one of claims 1 to 6, wherein the mobile robot device is selected from the group consisting of an autonomous flight platform, an autonomous marine platform, and a ground mobile robot.

8. A computer system comprising one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more tangible storage media for execution by at least one of the one or more processors via at least one of the one or more memories, detecting a light beam; identifying a light source location and an end point location of the light beam, the identifying including the end point location being a location where the light beam intersects a surface; receiving an audio command to proceed to the end point location; instructing a mobile robot device to proceed directly to the end point location; in response to the mobile robot device reaching the end point location, instructing the mobile robot device to perform an activity there A computer system capable of executing a method including.

9. identifying a travel path of the light beam; receiving an audio command to follow the travel path; instructing the mobile robot device to follow the travel path and performing the activity while following the travel path The computer system according to claim 8, further comprising.

10. identifying a boundary of an activity area based on image analysis of a closed-loop contour drawn using the light beam, the identifying including the activity area including the end point location of the light beam The computer system according to claim 8 or 9, further comprising.

11. identifying a plurality of light beams; identifying a boundary of an activity area based on the convergence of at least two of the plurality of light beams, the identifying including the activity area including the end point location of the light beam The computer system according to any one of claims 8 to 10, further comprising.

12. The computer system according to any one of claims 8 to 11, wherein the type of the activity is specified based on the color of the detected light beam.

13. The computer system according to claim 9, wherein a speed at which the mobile robot device directly advances to the end point location or a speed at which the mobile robot device follows the travel path of the light beam is based on the detected luminance value of the light beam.

14. The computer system according to any one of claims 8 to 13, wherein the mobile robot device is selected from the group consisting of an autonomous flight platform, an autonomous marine platform, and a ground mobile robot.

15. A computer program product including one or more computer-readable tangible storage media and program instructions stored in at least one of the one or more computer-readable tangible storage media, the program instructions being executable by a processor capable of executing a method, the method comprising: detecting a light beam; identifying a light source location and an end point location of the light beam, the end point location including a location where the light beam intersects a surface on which the light beam is located; receiving an audio command to advance to the end point location; instructing a mobile robot device to directly advance to the end point location; in response to the mobile robot device reaching the end point location, instructing the mobile robot device to perform an activity there The computer program product including the above.

16. identifying a travel path of the light beam; receiving an audio command to follow the travel path; instructing the mobile robot device to follow the travel path and performing the activity while following the travel path The computer program product according to claim 15, further including the above.

17. identifying a boundary of an activity area based on image analysis of a closed-loop contour drawn using the light beam, the activity area including the end point location of the light beam The computer program product according to claim 15 or 16, further including the above.

18. identifying a plurality of light beams; identifying a boundary of an activity area based on focusing of at least two of the plurality of light beams, the activity area including the end point location of the light beam The computer program product according to any one of claims 15 to 17, further including the above.

19. The computer program product according to any one of claims 15 to 18, wherein the type of the activity is specified based on the color of the detected light beam.

20. The computer program product according to claim 16, wherein the speed at which the mobile robot device directly proceeds to the end location or the speed at which the mobile robot device follows the travel path of the light beam is based on the luminance value of the detected light beam.

21. A computer program, comprising program code means adapted to execute the method according to any one of claims 1 to 7 when the program is executed on a computer.

Images