Autonomous mobile robot with security depth camera

Abstract

The invention relates to an autonomous mobile robot with a secure depth camera. Various aspects of techniques, systems, and cases may be used to control an autonomous mobile robot using a safe depth camera. An example technique may include receiving infrared data from at least two infrared receivers secured to a security depth camera of a robotic system, determining a security state of the robotic system related to a detected object in an environment based on the infrared data, and send an indication to at least one of emergency braking circuitry of the robotic system or adjustable braking circuitry of the robotic system based on the safety status.

Description

Background Art

[0001] Robots and other autonomous agents can be programmed to perform complex real-world tasks. The field of robotics has advanced to using artificial intelligence (AI) technologies to perform tasks in industrial and many other environments. For example, robotics encompasses a wide range of industrial applications such as smart manufacturing assembly lines, multi-robot automotive component assembly, computer and consumer electronics manufacturing, smart retail and warehouse logistics, robotic data centers, and so on. Robots typically interact with humans to complete tasks. Brief Description of the Drawings

[0002] In the drawings, which are not necessarily to scale, like reference numerals may describe like components in different views. Identical reference numerals with different alphabetic suffixes may represent different instances of like components. These drawings generally illustrate, by way of example and not limitation, the various embodiments discussed in this document.

[0003] Figure 1 Illustrates an environmental view according to an example, the environmental view including a robotic system, a safety zone, and a person.

[0004] Figure 2 Illustrates a comparison diagram of light detection and ranging (LIDAR) and infrared for object detection according to an example.

[0005] Figure 3 Illustrates a simplified data flow diagram according to an example including a camera that shares insights with both a vision system and a safety system.

[0006] Figure 4 Illustrates a robotic system according to an example showing a safety data flow.

[0007] Figure 5 Illustrates a block diagram of safety detection according to an example.

[0008] Figure 6 Illustrates a flowchart of a technique for determining a safety state of a robotic system according to an example.

[0009] Figure 7 Illustrates a flowchart of a technique for generating a depth image according to an example.

[0010] Figure 8A Provides an overview of example components of computing deployed at a computing node.

[0011] Figure 8B Provides a further overview of example components within a computing device. Detailed Description

[0012] The systems and techniques described herein provide security information (e.g., for a robotic system) and depth images using a depth camera, for example, in a single housing. The security information can include infrared data (e.g., images) captured using the depth camera. The depth camera can include two or more infrared sensors, an infrared projector, and a camera (e.g., a color camera (such as an RGB camera), a black and white camera, etc.) in a single housing. The alignment of the infrared sensors, the infrared projector, and the camera can be static relative to each other, e.g., predetermined. The predetermination allows for the processing of images or data captured by the infrared sensors or the camera without recalibration, which may be necessary if the sensors or the camera were located in separate housings or devices. The infrared sensors can be used to capture depth data, which can be used for the security of a robotic device (e.g., object detection and emergency or regular braking), and for adding depth information to images captured by the camera. The infrared data can be routed along two or more paths, e.g., routed to a security control circuitry to determine whether to brake the robotic system, and routed to a processing circuitry to generate, for example, a depth image. The depth image can be used for localization or mapping (e.g., the orientation or position of a robotic system within a map of an environment).

[0013] In an example, one or more security depth cameras (e.g., compliant with standards such as the International Electrotechnical Commission (IEC) 61496-4-3 Safety of machinery – Electro-sensitive protective equipment – Part 4-3: Particular requirements for equipment using vision based protective devices (VBPD) – Equipment using stereoscopic vision technology (VBPDST) or additional requirements when complying with the International Organization for Standardization (ISO) 3691-4) can be used with an autonomous mobile robot (AMR) (e.g., incorporated into the AMR, fixed to the AMR, communicatively coupled with the AMR, etc.) for security. The depth camera can be used for both a vision system and a security system. Using the depth camera can allow the AMR to not use LIDAR or other security systems. The depth camera provides a more efficient vision system than a vision system that relies on LIDAR, which also requires a visible light camera.

[0014] One example technical benefit of using a depth camera instead of LIDAR is improved vertical detection range of obstacles. For example, LIDAR is typically a planar solution and can only detect the ankles of a human. A depth camera can be used to detect at multiple heights and can thus detect ankles, feet, etc. Multiple heights can be captured using a single depth camera, while attempting to detect multiple heights using LIDAR may require multiple LIDAR systems, which is typically cost-prohibitive.

[0015] Figure 1 FIG. illustrates an environmental view 100 according to an example, the environmental view 100 including a robotic system 102, safety zones 104 and 106, and a person 108. The robotic system 102 may include an AMR having a depth camera 103, such as including two or more infrared sensors and a visible light camera in a single housing. The infrared sensors may scan the safety zones 104 and 106 to detect whether an obstacle (e.g., the person 108) has entered one of these zones. The safety zones may include a stop zone 104 and a deceleration zone 106. The infrared sensors may capture infrared data indicating the presence or absence of an obstacle in the stop zone 104 or the deceleration zone 106. The infrared data may be sent to a safety circuitry to evaluate whether to apply an emergency brake in the robotic system 102 (e.g., when an obstacle is detected in the stop zone 104), whether to decelerate the robotic system 102 (e.g., when an obstacle is detected in the deceleration zone 106), or whether to continue normal operation (e.g., when no obstacle is present in either safety zone 104).

[0016] In an example, the robotic system 102 may be an AMR operating with humans in a warehouse or factory environment. In some examples, a robot working on the floor with humans is referred to as a collaborative robot (cobot), i.e., a collaborative robot. The robotic system 102 may be self-navigating, e.g., configured to follow a path. The path may be pre-programmed or determined by the robotic system 102 or elsewhere, such as based on a goal, such as picking up an object at a first location and moving it to a second location. In some examples, the robotic system 102 may move only in a direction aligned with the field of view of the depth camera 103. In other examples, multiple depth cameras may be used for different fields of view (e.g., one in the front, one in the rear, and one on each side).

[0017] The infrared sensors of the robot system 102 can be used to detect three-dimensional objects. For example, the infrared sensors can capture 3D infrared data at two or more heights, for instance. This can increase the confidence of object detection and avoid false positives, etc. The data captured by the infrared sensors can be used together with the visible light images captured by the cameras of the robot system 102, for example, to create a depth image. In an example, the cameras can be used for object identification. In some examples, the data captured by the cameras can be used for Simultaneous Localization and Mapping (SLAM) techniques. The SLAM techniques can be used to map the environment in the environmental view 100 while also determining the position of the robot system 102 within the environment (e.g., within the map).

[0018] In an example, the images captured by the visible light cameras can be used to navigate the robot system 102. For example, an object of interest can be identified or a path can be determined based on the images and optionally the infrared data captured by the infrared sensors.

[0019] Figure 2 The figure shows a comparative diagram illustrating LIDAR and infrared object detection according to an example. Figure 2 The side view in shows a typical LIDAR side view 202 and an infrared (e.g., depth camera) side view 204 for object detection according to the systems and techniques described herein. The side view 204 illustrates how the use of infrared provides vertical coverage (e.g., 3D coverage) at different heights for object detection. In contrast, the LIDAR side view 202 only has a single planar object detection height coverage. Figure 2 The top view in shows a LIDAR top view 206 and an infrared top view 208. The two top views appear to have substantially similar coverage, meaning that the infrared coverage does not lose any side field of view compared to LIDAR, but as seen in the side view 204, provides a greatly improved height coverage.

[0020] Figure 3 The figure shows a simplified data flow diagram 300 according to an example. The data flow diagram illustrates a depth camera 302 (in Figure 3In the example, for illustrative purposes, four depth cameras are shown). The depth camera 302 can send the captured data to a controller (such as a safety programmable logic controller (PLC) 304) and a computing motherboard 306 (e.g., processing circuitry). The safety PLC 304 can determine whether braking is required based on the objects detected in the safety zone. The computing motherboard 306 can use the data captured by the depth camera 302 for typical robotic system purposes, such as SLAM, depth image generation, object identification, etc.

[0021] The depth camera 302 can be connected to the safety PLC 304 through specific dedicated safety hardware (e.g., EtherCat, output signal switching device (OSSD), etc.). In some examples, the depth camera 302 can send data (e.g., raw data or fused data) to the computing motherboard 306 for non-safety algorithms or purposes. The data can be transmitted to the computing motherboard 306 via a wired connection (e.g., USB, Ethernet, dedicated signal bus, etc.). The depth camera 302 can include one or more presets for detecting one or more AMR movement options.

[0022] Using the data from the depth camera 302, the AMR can perform one or more operations. For example, the AMR can monitor its own computing performance and software processes. The AMR can execute autonomous algorithms, navigate, collect and send telemetry data, record video for streaming, gain insights into the environment (e.g., a warehouse), collect 3D insights of the environment, use object or face detection, etc.

[0023] Figure 4 FIG. illustrates a robotic system 400 showing the safety data flow according to an example. The robotic system 400 illustrates the internal system of an AMR that uses depth cameras for safety and image capture technologies. In Figure 4 the example shown, the depth camera captures infrared and visible light data (e.g., using an infrared sensor and a color camera). The infrared data is processed by the safety control circuitry and is sent to the emergency braking system or the safety PLC system when appropriate, which may slow down the AMR. According to some examples, the safety control circuitry determines whether an object is present in a stop zone or a deceleration zone. The visible light data can be (optionally together with the infrared data) sent to the processing circuitry for SLAM, navigation, collision avoidance (e.g., outside the safety zone), trajectory planning, path planning, etc.

[0024] In an example, an infrared sensor can capture a 3D point cloud (e.g., reflected from the environment based on a pattern projected by an infrared projector, where the infrared projector, infrared sensor, and visible light camera can all be housed in a single unit). According to an example criterion, a minimum detection of a leg can include a dimension of at least 70 millimeters (mm). In some examples, the 3D point cloud can be projected onto a 2D matrix at a height of 15 to 30 centimeters (cm). Each cell can be filtered against a threshold, e.g., to reduce noise. After thresholding, clusters with a size of at least 70 mm can be selected (e.g., considered to identify a human). The process can be iterated through different height ranges, e.g., to detect the presence of a foot, where the previous height indicates the presence of an ankle. When a foot cluster close to a leg or ankle cluster is identified, these clusters can be merged and considered to be identified as a human.

[0025] In an example, detecting a human foot may occur earlier than detecting a leg or ankle, which gives us more time to use other types of safety brakes, such as SS1 (Safety Stop 1), rather than STO (e.g., full-power braking). In these examples, the deceleration is smoother and the risk of loss of the cargo carried by the AMR is reduced.

[0026] When several cameras on the AMR are used, some or all of these cameras can have overlapping fields of view. For each overlapping region, a comparison of the clusters found above can be used. In some examples, the distance from a particular camera to a cluster can be weighted. When the cluster is far from the camera (e.g., 2 - 3 meters), the cluster can be magnified because in a stereo camera, the noise level is greater according to the distance. The comparison can be performed by checking the field of view and the centroid of each cluster. When a cluster is found in only one camera in an area where it should be detected by a second camera, an emergency message can be sent to perform an emergency stop because there is a high likelihood that one of these cameras is faulty.

[0027] Figure 5The figure shows a block diagram 500 of a security detection according to an example. The block diagram 500 includes blocks that describe techniques for securely detecting a human leg or foot according to an example. The block diagram 500 includes a block for obtaining a 3D point cloud. The 3D point cloud can be projected onto a 2D matrix, such as for a specific height or height range (e.g., within a few millimeters). Each pixel in the 2D matrix that has more than a threshold number of hits (e.g., five hits) can be thresholded. The matrix can be segmented after thresholding. Clusters having a minimum width (e.g., a minimum value of 50 mm, 70 mm, 100 mm, etc.) according to security criteria can be identified in the segmented matrix. When a cluster is identified as not meeting the security criteria (e.g., greater than the minimum value), the cluster can be marked as an obstacle. One or more iterations can occur, such as to detect whether a cluster exists at another height value or range. When two or more clusters are identified as obstacles at two or more heights, they can be marked as a human. In some examples, the nearest obstacle (e.g., human, cluster, etc.) can be selected as the first priority for avoidance, such as for emergency braking or deceleration.

[0028] Figure 6 The figure shows a flowchart of a technique 600 for determining a security state of a robotic system according to an example. The technique 600 can be performed by a computing device (e.g., including processing circuitry), by a robotic device (e.g., including processing circuitry), etc.

[0029] The technique 600 includes an operation 602 for receiving infrared data from at least two infrared receivers of a security depth camera fixed to the robotic system. The at least two infrared receivers can be housed in a single housing of the security depth camera. In an example, the single housing can also include an infrared projector.

[0030] The technique 600 includes an operation 604 for determining a security state of the robotic system related to an object detected in the environment based on the infrared data. The detected object can include parts of a human, such as a leg, ankle, or foot. In an example, the infrared data can include a three-dimensional point cloud. In this example, the operation 604 can include projecting the three-dimensional point cloud onto a first 2D matrix at a first specified height from the ground position, determining whether the first 2D matrix includes a first set of clusters having a minimum width, and in response, marking the detected object corresponding to the first set of clusters in a map of the environment. This example can be iterated at a second height. A determination can be made as to whether the first set of clusters and the iterated second set of clusters are adjacent (e.g., within a minimum distance). The operation 604 can include determining the nearest identifiable obstacle.

[0031] Technique 600 includes operation 606 for sending an indication to at least one of an emergency braking circuitry of a robotic system or an adjustable braking circuitry of the robotic system based on a safety state. Operation 606 may include sending an indication to the emergency braking circuitry when the safety state indicates that a detected object is within a first threshold, and sending an indication to the adjustable braking circuitry when the detected object is outside the first threshold and within a second threshold.

[0032] Technique 600 includes operation 608 for receiving infrared data and a color image from a camera of a safety depth camera. Technique 600 includes operation 610 for generating a depth image using the infrared data and the color image.

[0033] Technique 600 includes operation 612 for using the depth image to identify a change in at least one of a planned trajectory, a map, or a position or orientation of the robotic system within the map. Operation 612 may include using a visual simultaneous localization and mapping (SLAM) algorithm to identify a change in the position and orientation of the robotic system within the map.

[0034] Figure 7 The figure shows a flowchart of a technique 700 for generating a depth image according to an example. Technique 700 may be executed by a computing device (e.g., including processing circuitry), by a robotic device (e.g., including processing circuitry), etc.

[0035] Technique 700 includes operation 702 for receiving a three-dimensional point cloud of an environment from two or more infrared sensors of a robotic system.

[0036] Technique 700 includes operation 704 for projecting the three-dimensional point cloud onto a first two-dimensional matrix at a first specified height from a ground position.

[0037] Technique 700 includes operation 706 for determining whether the first two-dimensional matrix includes a first set of clusters having a minimum width. In an example, operation 706 includes thresholding each pixel in the first two-dimensional matrix using more than a minimum number of points. In this example, operation 706 may include segmenting the first two-dimensional matrix after thresholding.

[0038] Technique 700 includes operation 708 for marking a first obstacle corresponding to the first set of clusters in a map of the environment in response to determining that the first two-dimensional matrix includes the first set of clusters having a minimum width.

[0039] Technique 700 includes operation 710 for projecting the three-dimensional point cloud onto a second two-dimensional matrix at a second specified height from the ground position. In an example, the second specified height is closer to the ground position than the first specified height.

[0040] Technique 700 includes operation 712 for determining whether a second two-dimensional matrix includes a second set of clusters having a minimum width.

[0041] Technique 700 includes operation 714 for marking a second obstacle corresponding to the second set of clusters in a map of the environment in response to determining that the second two-dimensional matrix includes the second set of clusters having a minimum width.

[0042] Technique 700 includes operation 716 for determining a safety state based on whether the first set of clusters and the second set of clusters are adjacent. Operation 716 may include determining the nearest identified obstacle.

[0043] Technique 700 includes operation 718 for sending an indication to at least one of an emergency braking circuitry of a robotic system or an adjustable braking circuitry of the robotic system based on the safety state. In an example, when the safety state indicates that a first obstacle is within a first threshold, the indication is sent to the emergency braking circuitry, and when the first obstacle is outside the first threshold and within a second threshold, the indication is sent to the adjustable braking circuitry.

[0044] Technique 700 may include using a safety depth camera that includes two or more infrared sensors, an infrared projector, and an image capture sensor. Technique 700 may include generating a depth image from a three-dimensional point cloud and a color image captured by the image capture sensor, and using the depth image to identify at least one of the following changes: a planned trajectory, a map, a position or orientation of the robotic system within the map, etc.

[0045] In a further example, any one of the computing nodes or devices discussed with reference to a current computing system (e.g., a robotic device) and the environment may be implemented based on Figure 8A and Figure 8B the components depicted in. The corresponding computing node may be embodied as some type of device, apparatus, computer, or other "thing" capable of communicating with other edge components, networking components, or endpoint components. For example, an edge computing device may be embodied as a personal computer, a server, a smart phone, a mobile computing device, a smart device, an on-board computing system (e.g., a navigation system), a self-contained device having an enclosure, a housing, etc., a robotic system, or other device or system capable of performing the described functions.

[0046] In Figure 8AIn the simplified example depicted, edge computing node 800 includes a computing engine (also referred to herein as "computing circuitry") 802, an input / output (I / O) subsystem 808, a data storage device 810, a communication circuitry subsystem 812, and optionally, one or more peripheral devices 814. In other examples, the corresponding computing device may include other or additional components, such as those commonly found in a computer (e.g., a display, peripheral devices, etc.). Additionally, in some examples, one or more of the illustrative components may be incorporated into another component or otherwise form part of another component.

[0047] Computing node 800 may be embodied as any type of engine, device, or collection of devices capable of performing various computing functions. In some examples, computing node 800 may be embodied as a single device, such as an integrated circuit, an embedded system, a field-programmable gate array (FPGA), a system-on-a-chip (SOC), or other integrated system or device. In an illustrative example, computing node 800 includes or is embodied as a processor 804 and a memory 806. Processor 804 may be embodied as any type of processor capable of performing the functions described herein (e.g., executing an application). For example, processor 804 may be embodied as a multi-core processor, a microcontroller, a processing unit, a specialized or dedicated processing unit, or other processor or processing / control circuitry.

[0048] In some examples, the processor 804 may be embodied as, include, or be coupled to an FPGA, an application specific integrated circuit (ASIC), reconfigurable hardware, or hardware circuitry, or other specialized hardware for facilitating the performance of the operations described herein. Moreover, in some examples, the processor 804 may be embodied as a specialized x-processing unit (xPU), also referred to as a data processing unit (DPU), an infrastructure processing unit (IPU), or a network processing unit (NPU). Such an xPU may be embodied as a stand-alone circuit or circuit package, integrated within a SOC, or integrated with networking circuitry (e.g., in a smart NIC or enhanced smart NIC), an acceleration circuit, a storage device, or AI hardware (e.g., a GPU or programmed FPGA). Outside of a CPU or general-purpose processing hardware, such an xPU may be designed to receive programming to process one or more data streams and perform specific tasks and actions for the data streams (such as orchestrating microservices, performing service management or orchestration, organizing or managing server or data center hardware, managing a service mesh, or collecting and distributing telemetry). However, it will be understood that the xPU, SOC, CPU, and other variants of the processor 804 may work in concert with each other to perform many types of operations and instructions within and on behalf of the computing node 800.

[0049] The memory 806 may be embodied as any type of volatile memory or data storage device (e.g., dynamic random access memory (DRAM), etc.) or non-volatile memory or data storage device capable of performing the functions described herein. Volatile memory may be a storage medium that requires power to maintain the state of the data stored by the medium. Non-limiting examples of volatile memory may include various types of random access memory (RAM), such as DRAM or static random access memory (SRAM). A particular type of DRAM that may be used in a memory module is synchronous dynamic random access memory (SDRAM).

[0050] In an example, the memory device is a block-addressable memory device, such as those based on NAND or NOR technology. The memory device may also include a three-dimensional cross-point memory device (e.g., 3D XPoint TM memory) or other byte-addressable in-situ write non-volatile memory devices. The memory device may refer to the die itself and / or to a packaged memory product. In some examples, the 3D cross-point memory (e.g., 3D XPoint TM memory) may include a transistor-less stackable cross-point architecture where memory cells are located at the intersections of word lines and bit lines and are individually addressable, and where bit storage is based on a change in bulk resistance. In some examples, all or part of the memory 806 may be integrated into the processor 804. The memory 806 may store various software and data used during operation, such as one or more applications, data operated on by the (one or more) applications, libraries, and drivers.

[0051] The computing circuitry 802 is communicatively coupled via the I / O subsystem 808 to other components of the computing node 800, and the I / O subsystem 808 may be embodied as circuitry and / or components configured to facilitate input / output operations with the computing circuitry 802 (e.g., having the processor 804 and / or the main memory 806) and other components of the computing circuitry 802. For example, the I / O subsystem 808 may be embodied as or otherwise include a memory controller hub, an input / output control hub, an integrated sensor hub, a firmware device, a communication link (e.g., a point-to-point link, a bus link, a line, a cable, an optical waveguide, a printed circuit board trace, etc.), and / or other components and subsystems for facilitating input / output operations. In some examples, the I / O subsystem 808 may form part of a system-on-a-chip (SoC) and may be incorporated into the computing circuitry 802 together with one or more of the processor 804, the memory 806, and other components of the computing circuitry 802.

[0052] One or more illustrative data storage devices 810 may be embodied as any type of device configured for short-term or long-term storage of data, such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. Each data storage device 810 may include a system partition storing data and firmware code for the data storage device 810. Each data storage device 810 may also include one or more operating system partitions storing data files and executable files for an operating system, depending on, for example, the type of the computing node 800.

[0053] The communication circuitry 812 may be embodied as any communication circuit, device, or collection thereof capable of enabling communication between the computing circuitry 802 and another computing device (e.g., a gateway of an implemented computing system) over a network. The communication circuitry 812 may be configured to use any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., cellular networking protocols (such as 3GPP 4G or 5G standards), wireless local area network protocols (such as IEEE802.11 / Wi- Fi), wireless wide area network protocols, Ethernet, Bluetooth Low Energy, IoT protocols (such as IEEE802.15.4 or ), low-power wide-area network (LPWAN) or low-power wide-area (LPWA) protocols, etc.) to effectuate such communication.

[0054] An illustrative communication circuitry 812 includes a network interface controller (NIC) 820, which may also be referred to as a host fabric interface (HFI). The NIC 820 may be embodied as one or more plug-in boards, daughter cards, network interface cards, controller chips, chip sets, or other devices that may be used by the computing node 800 to connect to another computing device (e.g., a gateway node). In some examples, the NIC 820 may be embodied as part of a system-on-chip (SoC) that includes one or more processors, or the NIC 820 may be included on a multi-chip package that also contains one or more processors. In some examples, the NIC 820 may include a local processor (not shown) and / or local memory (not shown), both of which are local to the NIC 820. In such examples, the local processor of the NIC 820 may be capable of performing one or more of the functions of the computing circuitry 802 described herein. Additionally or alternatively, in such examples, the local memory of the NIC 820 may be integrated into one or more components of the client computing node at the board level, socket level, chip level, or other level.

[0055] Additionally, in some examples, the respective computing node 800 can include one or more peripheral devices 814. Depending on the specific type of computing node 800, such peripheral devices 814 can include any type of peripheral device found in a computing device or server, such as audio input devices, displays, other input / output devices, interface devices, and / or other peripheral devices. In further examples, the computing node 800 can be embodied by a corresponding computing node in a computing system (whether a client, gateway, or aggregation node), or by a similar form of apparatus, computer, subsystem, circuitry, or other components.

[0056] In a more detailed example, Figure 8B FIG. illustrates a block diagram of an example of components that can be present in a computing node 850 for implementing the techniques described herein (e.g., operations, processes, methods, and methodologies). The computing node 850 provides a closer view of the corresponding components of node 800 when implemented as a computing device (e.g., a mobile device, base station, server, gateway, etc.) or as part of a computing device (e.g., a mobile device, base station, server, gateway, etc.). The computing node 850 can include any combination of the hardware or logic components cited herein, and the computing node 850 can include any device or be coupled to a device that can be used with a communication network or a combination of such networks. These components can be implemented as an integrated circuit (IC), a portion of an IC, discrete electronic devices, or other modules, instruction sets, programmable logic, or algorithms, hardware, hardware accelerators, software, firmware, or a combination thereof suitable for use in the computing node 850, or be implemented as components otherwise incorporated within the chassis of a larger system.

[0057] The computing device 850 can include processing circuitry in the form of a processor 852, which can be a microprocessor, multi-core processor, multi-threaded processor, ultra-low voltage processor, embedded processor, xPU / DPU / IPU / NPU, specialized processing unit, dedicated processing unit, or other known processing element. The processor 852 can be part of a system on a chip (SoC), in which the processor 852 and other components are formed as a single integrated circuit or a single package, such as the Edison TM (Edison TM ) or Galileo TM (Galileo TM ) SoC board. As an example, the processor 852 can include a CPU processor based on architecture Core TM (Core TM ) (such as QuarkTM (Quark TM ), Atom TM (Atom TM ), i3, i5, i7, i9, or an MCU-class processor), or another such processor available from . However, any number of other processors may be used, such as processors available from Advanced Micro Devices, of Sunnyvale, California, designs based on from MIPS Technologies, Inc. of Sunnyvale, California, designs based on licensed from ARM Holdings plc, or processors available from customers, licensees, or adopters of the foregoing companies. The processor may include units such as the A5-A13 processors from company, the Snapdragon (Snapdragon TM ) processors from TM Technologies, Inc., or the OMAP TM processors from Texas Instruments Incorporated. Processor 852 and the accompanying circuitry may be provided in a single-socket form factor, a multi-socket form factor, or various other formats, including in a limited hardware configuration or in a configuration that includes fewer than all of the elements shown in Figure 8B .

[0058] The processor 852 may communicate with the system memory 854 via an interconnect 856 (e.g., a bus). Any number of memory devices may be used to provide a given amount of system memory. As an example, the memory 854 may be a random access memory (RAM) designed according to the Joint Electron Devices Engineering Council (JEDEC), such as DDR or Mobile DDR standards (e.g., LPDDR, LPDDR2, LPDDR3, or LPDDR4). In a particular example, the memory component may conform to a DRAM standard promulgated by JEDEC, such as JESD79F for DDR SDRAM, JESD79-2F for DDR2 SDRAM, JESD79-3F for DDR3 SDRAM, JESD79-4A for DDR4 SDRAM, JESD209 for Low Power DDR (LPDDR), JESD209-2 for LPDDR2, JESD209-3 for LPDDR3, and JESD209-4 for LPDDR4. Such standards (and similar standards) may be referred to as DDR-based standards, and the communication interfaces of storage devices implementing such standards may be referred to as DDR-based interfaces. In various implementations, each memory device may be any number of different package types, such as a single die package (SDP), a dual die package (DDP), or a quad die package (QDP). In some examples, these devices may be directly soldered to the motherboard to provide a thin solution, while in other examples, the devices are configured as one or more memory modules, which are in turn coupled to the motherboard via a given connector. Any number of other memory implementations may be used, such as other types of memory modules, e.g., different kinds of dual inline memory modules (DIMMs), including but not limited to microDIMM or MiniDIMM.

[0059] To provide persistent storage of information (such as data, applications, operating systems, etc.), storage device 858 may also be coupled to processor 852 via interconnect 856. In an example, storage device 858 may be implemented via a solid-state disk drive (SSDD). Other devices that can be used for storage device 858 include flash memory cards (such as Secure Digital (SD) cards, microSD cards, extreme digital (eXtreme Digital, XD) picture cards, etc.) and Universal Serial Bus (USB) flash drives. In an example, the memory device may be or may include a memory device using chalcogenide glass, a multi-threshold level NAND flash memory, a NOR flash memory, a single-level or multi-level phase change memory (PCM), a resistive memory, a nanowire memory, a ferroelectric transistor random access memory (FeTRAM), an antiferroelectric memory, a magnetoresistive random access memory (MRAM) including memristor technology, a resistive memory including a metal oxide substrate, an oxygen vacancy substrate and a conductive bridge random access memory (CB-RAM), or a spin transfer torque (STT)-MRAM, a device based on a spin electronic magnetic junction memory, a device based on a magnetic tunneling junction (MTJ), a device based on a DW (Domain Wall) and a SOT (Spin Orbit Transfer), a thyristor-based memory device, or a combination of any of the above or other memories.

[0060] In a low-power implementation, the storage device 858 may be an on-die memory or register associated with the processor 852. However, in some examples, the storage device 858 may be implemented using a micro hard disk drive (HDD). Furthermore, in addition to or in lieu of the described techniques, any number of new technologies may be used for the storage device 858, such as resistive memory, phase change memory, holographic memory, or chemical memory, among others.

[0061] Components may communicate via interconnect 856. Interconnect 856 may include any number of technologies, including industry standard architecture (ISA), extended ISA (EISA), peripheral component interconnect (PCI), peripheral component interconnect extended (PCIx), PCI express (PCIe), or any number of other technologies. Interconnect 856 may be a proprietary bus used, for example, in a SoC-based system. Other bus systems may be included, such as an Inter-Integrated Circuit (I2C) interface, a Serial Peripheral Interface (SPI) interface, a point-to-point interface, and a power bus, among others.

[0062] The interconnect 856 may couple the processor 852 to a transceiver 866 for communication with the connected device 862. The transceiver 866 may use any number of frequencies and protocols, such as 2.4 Gigahertz (GHz) transmissions under the IEEE 802.15.4 standard, using a 100 MHz RF interface such as a 400 MHz RF interface. Special Interest Groups Defined Low Energy (BLE) standard, or standards, etc. Any number of radios configured for a particular wireless communication protocol may be used for connection to device 862. For example, a wireless local area network (WLAN) unit may be used to implement wireless communications according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard. Additionally, wireless wide area communications, such as according to cellular or other wireless wide area protocols, may occur via wireless wide area network (WWAN) units.

[0063] The wireless network transceiver 866 (or multiple transceivers) can communicate using a variety of standards or radios for communication at different ranges. For example, the computing node 850 can use a local transceiver based on Bluetooth Low Energy (BLE) or another low-power radio to communicate with nearby (e.g., within about 10 meters) devices to save power. A more distant (e.g., within about 50 meters) connected device 862 can communicate via or other intermediate power radios. The two communication technologies can occur over a single radio at different power levels, or can occur over separate transceivers, such as a local transceiver using BLE and a wireless transceiver using Bluetooth LE. A separate mesh transceiver.

[0064] A wireless network transceiver 866 (e.g., a radio transceiver) may be included to communicate with devices or services in the cloud 895 via a local area network protocol or a wide area network protocol. The wireless network transceiver 866 may be a low-power wide-area (LPWA) transceiver that complies with the IEEE 802.15.4 or IEEE 802.15.4g standard, etc. The computing node 850 may use the LoRaWAN developed by Semtech and the LoRa Alliance. TM (Long Range Wide Area Network) to communicate over a wide area. The techniques described herein are not limited to these techniques, but can be used with any number of other cloud transceivers that enable long-range, low-bandwidth communications, such as Sigfox and other technologies. Further, other communication techniques may be used, such as time-division channel hopping as described in the IEEE 802.15.4e specification.

[0065] In addition to the systems mentioned for the wireless network transceiver 866 as described herein, any number of other radio communications and protocols may be used. For example, the transceiver 866 may include a cellular transceiver that uses spread spectrum (SPA / SAS) communications to achieve high-speed communications. Further, any number of other protocols may be used, such as for medium-speed communications and supply network communications. Network. The transceiver 866 may include a radio compatible with any number of 3GPP (Third Generation Partnership Project) specifications, such as Long Term Evolution (LTE) and Fifth Generation (5G) communication systems discussed in further detail at the end of this disclosure. A network interface controller (NIC) 868 may be included to provide wired communication to nodes of the cloud 895 or to other devices, such as (e.g., connected devices 862 operating in a grid). The wired communication may provide an Ethernet connection, or may be based on other types of networks, such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a DeviceNet, a ControlNet, a Data Highway+, PROFIBUS, or PROFINET, etc. An additional NIC 868 may be included to enable connection to a second network, for example, a first NIC 868 provides communication to the cloud via Ethernet, and a second NIC 868 provides communication to other devices via another type of network.

[0066] In view of the diversity of applicable communication types from a device to another component or network, applicable communication circuitry used by the device may include or be embodied by any one or more of components 864, 866, 868, or 870. Thus, in various examples, applicable means for communicating (e.g., receiving, transmitting, etc.) may be embodied by such communication circuitry.

[0067] The computing node 850 may include or be coupled to an acceleration circuit system 864, which may be embodied by one or more artificial intelligence (AI) accelerators, neural computing sticks, neuromorphic hardware, FPGAs, arrangements of GPUs, arrangements of xPU / DPU / IPU / NPUs, one or more SoCs, one or more CPUs, one or more digital signal processors, dedicated ASICs, or other forms of specialized processors or circuit systems designed to perform one or more specialized tasks. These tasks may include AI processing (including machine learning, training, reasoning, and classification operations), visual data processing, network data processing, object detection, rule analysis, etc. These tasks may also include specific computing tasks for service management and service operations discussed elsewhere in this document.

[0068] The interconnect 856 may couple the processor 852 to a sensor hub or external interface 870 for connecting additional devices or subsystems. The devices may include sensors 872 such as accelerometers, level sensors, flow sensors, optical light sensors, camera sensors, temperature sensors, global positioning system (i.e., global navigation system, GPS) sensors, pressure sensors, barometric pressure sensors, etc. The hub or interface 870 may further be used to connect the computing node 850 to actuators 874 (such as power switches, valve actuators, audible sound generators, visual warning devices, etc.).

[0069] In some optional examples, various input / output (I / O) devices may be present within or connected to the computing node 850. For example, a display or other output device 884 may be included to display information, such as sensor readings or actuator positions. An input device 886 (such as a touch screen or keypad) may be included to accept input. The output device 884 may include any number of audio or visual display forms, including: simple visual output, such as a binary status indicator (e.g., a light-emitting diode (LED)); multi-character visual output; or more complex output, such as a display screen (e.g., a liquid crystal display (LCD) screen) with output of characters, graphics, multimedia objects, etc. generated or produced from the operation of the computing node 850. In the context of the present system, the display or console hardware may be used to provide output to and receive input to the computing system; to manage components or services of the computing system; to identify the status of computing components or services; or to perform any other number of management or administrative functions or service use cases.

[0070] The battery 876 can power the computing node 850, but in examples where the computing node 850 is installed in a fixed location, the computing node 850 can have a power source coupled to the grid, or the battery can be used as a backup or for temporary functions. The battery 876 can be a lithium-ion battery or a metal-air battery (such as a zinc-air battery, an aluminum-air battery, a lithium-air battery), etc.

[0071] A battery monitor / charger 878 may be included in the computing node 850 to track the state of charge (SoCh) of the battery 876 (if included). The battery monitor / charger 878 may be used to monitor other parameters of the battery 876 to provide failure prediction, such as the state of health (SoH) and state of function (SoF) of the battery 876. The battery monitor / charger 878 may include a battery monitoring integrated circuit, such as the LTC4020 or LTC2990 from Linear Technologies, the ADT7488A from ON Semiconductor of Phoenix, Arizona, or the UCD90xxx family of ICs from Texas Instruments of Dallas, Texas. The battery monitor / charger 878 may transmit information about the battery 876 to the processor 852 via the interconnect 856. Battery monitor / charger 878 may also include an analog-to-digital (ADC) converter that enables processor 852 to directly monitor the voltage of battery 876 or the current from battery 876. Battery parameters may be used to determine actions that computing node 850 may perform, such as transmission frequency, mesh network operation, sensing frequency, and the like.

[0072] A power block 880 or other power source coupled to the grid may be coupled to a battery monitor / charger 878 to charge the battery 876. In some examples, the power block 880 may be replaced with a wireless power receiver to obtain power wirelessly, for example, through a loop antenna in the computing node 850. A wireless battery charging circuit (such as an LTC4020 chip from Linear Technology, Inc. of Milpitas, California, etc.) may be included in the battery monitor / charger 878. A specific charging circuit may be selected based on the size of the battery 876 and therefore the current required. Charging may be performed using the Airfuel standard promulgated by the Airfuel Alliance, the Qi wireless charging standard promulgated by the Wireless Power Consortium, or the Rezence charging standard promulgated by the Alliance for Wireless Power, etc.

[0073] The storage 858 may include instructions 882 in the form of software, firmware, or hardware commands for implementing the techniques disclosed herein. Although such instructions 882 are illustrated as code blocks included in the memory 854 and the storage 858, it is understood that any of the code blocks may be replaced with hardwired circuits, such as built into an application specific integrated circuit (ASIC).

[0074] In an example, the instructions 882 provided via the memory 854, the storage device 858, or the processor 852 may be embodied as a non-transitory machine-readable medium 860, which includes code for directing the processor 852 to perform electronic operations in the computing node 850. The processor 852 can access the non-transitory machine-readable medium 860 through the interconnect 856. For example, the non-transitory machine-readable medium 860 may be embodied by the device described for the storage device 858, or may include a specific storage unit, such as an optical disk, a flash drive, or any number of other hardware devices. The non-transitory machine-readable medium 860 may include instructions for directing the processor 852 to perform, for example, a specific sequence of actions or flow of actions described with reference to the (one or more) flow charts and (one or more) block diagrams of the operations and functions depicted above. As used herein, the terms "machine-readable medium" and "computer-readable medium" are interchangeable.

[0075] Additionally, in certain examples, instructions 882 on processor 852 (alone or in combination with instructions 882 of machine-readable medium 860) may configure execution or operation of a trusted execution environment (TEE) 890. In an example, TEE 890 operates as a protected area accessible to processor 852 for secure execution of instructions and secure access to data. For example, a secure environment may be created using Software Guard Extensions (SGX) or Hardware security extensions, Management Engine (ME) or The Converged Security Manageability Engine (CSME) provides various implementations of TEE 890 and accompanying security zones in processor 852 or memory 854. Security hardening, hardware root of trust, and other aspects of trusted or protected operations can be implemented in device 850 through TEE 890 and processor 852.

[0076] In a further example, a machine-readable medium also includes any tangible medium that can store, encode or carry instructions for execution by a machine and cause the machine to perform any one or more of the methods of the present disclosure, or that can store, encode or carry data structures utilized by or associated with such instructions. "Machine-readable media" may therefore include, but are not limited to, solid-state memory, optical media, and magnetic media. Specific examples of machine-readable media include non-volatile memory, including, by way of example, but not limited to: semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Instructions embodied by machine-readable media may be transmitted or received further over a communication network using a transmission medium, via a network interface device, using any of several transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)).

[0077] The machine-readable medium may be provided by a storage device or other device capable of mastering data in a non-transient format. In an example, information stored on a machine-readable medium or otherwise provided on a machine-readable medium may represent instructions, such as the instructions themselves or a format from which instructions can be derived. The format from which instructions can be derived may include source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., divided into multiple packages), etc. The information representing the instructions in the machine-readable medium may be processed into instructions by a processing circuit system to implement any operation discussed herein. For example, deriving instructions from the information (e.g., processing by a processing circuit system) may include: (e.g., from source code, object code, etc.) compiling, interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, decoding, encrypting, decrypting, packaging, unpacking, or otherwise manipulating information into instructions.

[0078] In an example, the derivation of instructions may include (e.g., by a processing circuit system) assembling, compiling, or interpreting information to create instructions from an intermediate or pre-processed format provided by a machine-readable medium. When the information is provided in multiple parts, it can be combined, unpacked, and modified to create instructions. For example, the information may be in multiple compressed source code packages (or object codes, or binary executable codes, etc.) on one or more remote servers. The source code package may be encrypted when transmitted over a network, and may be decrypted, decompressed, (if necessary) assembled (e.g., linked) at a local machine, and compiled or interpreted (e.g., compiled or interpreted into a library, a stand-alone executable file, etc.), and executed by the local machine.

[0079] It should be understood that the functional units or capabilities described in this specification may have been referred to or labeled as components or modules, thereby particularly emphasizing the independence of their implementation. Such components can be embodied in any number of software or hardware forms. For example, a component or module can be implemented as a hardware circuit, which includes a customized very-large-scale integration (VLSI) circuit or gate array, a ready-made semiconductor such as a logic chip, a transistor, or other discrete components. A component or module may also be implemented in a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc. A component or module may also be implemented in software for execution by various types of processors. The identified components or modules of an executable code may, for example, include one or more physical or logical boxes of a computer instruction, which may be organized into, for example, an object, a process, or a function. However, the executable files of an identified component or module need not be physically located together, but may include different instructions stored in different locations, which when logically joined together (e.g., including by wires, by a network, using one or more platforms, wirelessly, via software components, etc.) include the component or module and achieve the intended purpose for the component or module.

[0080] In fact, a component or module of an executable code can be a single instruction or many instructions, and can even be distributed over several different code segments, between different programs, and across several memory devices or processing systems. Specifically, some aspects of the described process (such as code rewriting and code analysis) may be performed on a processing system (e.g., a computer in a data center) that is different from the processing system in which the code is deployed (e.g., a computer embedded in a sensor or robot). Similarly, operational data can be identified and shown within a component or module here, and can be embodied in any suitable form and can be organized in any suitable type of data structure. The operational data can be collected as a single data set, or can be distributed over different locations (including on different storage devices), and can exist at least in part only as electronic signals on a system or network. A component or module can be passive or active, including an agent that can be operated to perform a desired function.

[0081] Additional examples of the presently described method, system, and device embodiments include the following non-limiting implementations. Each of the following non-limiting examples may exist independently or may be combined with any one or more of the other examples provided below or throughout the present disclosure in any arrangement or combination.

[0082] Each of these non-limiting examples may stand alone or may be combined with one or more of the other examples in various permutations or combinations.

[0083] Example 1 is a device comprising: a safety control circuit system, wherein the safety control circuit system is used to: receive infrared data from at least two infrared receivers of a safety depth camera fixed to a robotic system; determine a safety status of the robotic system related to an object detected in an environment based on the infrared data; and send an indication to at least one of an emergency braking circuit system of the robotic system or an adjustable braking circuit system of the robotic system based on the safety status; and a processing circuit system, which is used to: receive infrared data and a color image from a camera of the safety depth camera; generate a depth image using the infrared data and the color image; and use the depth image to identify a change in at least one of: a planned trajectory, a map, or a position or orientation of the robotic system within the map.

[0084] In Example 2, the subject matter of Example 1 includes wherein when the safety status indicates that the detected object is within a first threshold, the indication is sent to the emergency brake circuit system, and when the detected object is outside the first threshold and within a second threshold, the indication is sent to the adjustable brake circuit system.

[0085] In Example 3, the subject matter of Examples 1-2 includes, wherein the detected object is a part of a human being.

[0086] In Example 4, the subject matter of Examples 1-3 includes, wherein identifying the change using the depth image includes: using a visual simultaneous localization and mapping (SLAM) algorithm to identify the change in the position and orientation of the robotic system within the map.

[0087] Example 5 is a robotic system comprising: a safety depth camera comprising: an infrared projector; at least two infrared receivers; and an image capture sensor; an emergency braking circuit system; an adjustable braking circuit system; a safety control circuit system for: receiving infrared data from at least two infrared receivers; determining a safety status associated with a detected object based on the infrared data; and sending an indication to at least one of the emergency braking circuit system or the adjustable braking circuit system based on the safety status; and a processing circuit system for: receiving infrared data and a color image from the image capture sensor; generating a depth image using the infrared data and the color image; and using the depth image to identify a change in at least one of: a planned trajectory, a map, or a position or orientation of the robotic system within the map.

[0088] In Example 6, the subject matter of Example 5 includes, wherein the infrared projector, the at least two infrared receivers, and the image capture sensor are located within a single housing of the security depth camera.

[0089] In Example 7, the subject matter of Examples 5-6 includes, wherein the infrared data includes a three-dimensional point cloud, and wherein determining the safety status includes the following operations: projecting the three-dimensional point cloud to a first two-dimensional matrix at a first specified height from a ground position; determining whether the first two-dimensional matrix includes a first cluster set having a minimum width; and in response to determining that the first two-dimensional matrix includes a first cluster set having a minimum width, marking a detected object corresponding to the first cluster set in a map of the environment.

[0090] In Example 8, the subject matter of Example 7 includes, wherein determining the safety status includes the following operations: projecting the three-dimensional point cloud to a second two-dimensional matrix at a second specified height from the ground position; determining whether the second two-dimensional matrix includes a second cluster set with a minimum width; and in response to determining that the second two-dimensional matrix includes a second cluster set with a minimum width, marking a second detected object corresponding to the second cluster set in a map of the environment.

[0091] In Example 9, the subject matter of Example 8 includes, wherein determining the security status comprises determining the security status based on whether the first set of clusters and the second set of clusters are adjacent.

[0092] In Example 10, the subject matter of Examples 5-9 includes wherein when the safety status indicates that the detected object is within a first threshold, the indication is sent to the emergency braking circuit system, and when the detected object is outside the first threshold and within a second threshold, the indication is sent to the adjustable braking circuit system.

[0093] In Example 11, the subject matter of Examples 5-10 includes, wherein the detected object is a part of a human being.

[0094] In Example 12, the subject matter of Examples 5-11 includes, wherein identifying the change using the depth image includes identifying a change in the position and orientation of the robotic system within the map using a visual simultaneous localization and mapping (SLAM) algorithm.

[0095] In Example 13, the subject matter of Example 5 includes, wherein determining the safety status includes determining a nearest identified obstacle.

[0096] Example 14 is at least one machine-readable medium, comprising instructions that, when executed by a safety processing circuit system of a robotic system, cause the safety processing circuit system to perform operations including: receiving a three-dimensional point cloud of an environment from two or more infrared sensors of the robotic system; projecting the three-dimensional point cloud to a first two-dimensional matrix at a first specified height from a ground position; determining whether the first two-dimensional matrix includes a first cluster set with a minimum width; in response to determining that the first two-dimensional matrix includes the first cluster set with a minimum width, marking a first obstacle corresponding to the first cluster set in a map of the environment; projecting the three-dimensional point cloud to a second two-dimensional matrix at a second specified height from the ground position; determining whether the second two-dimensional matrix includes a second cluster set with a minimum width; in response to determining that the second two-dimensional matrix includes a second cluster set with a minimum width, marking a second obstacle corresponding to the second cluster set in a map of the environment; determining a safety status based on whether the first cluster set and the second cluster set are adjacent; and sending an indication to at least one of an emergency braking circuit system of the robotic system or an adjustable braking circuit of the robotic system based on the safety status.

[0097] In Example 15, the subject matter of Example 14 includes, wherein determining whether the first two-dimensional matrix includes a first set of clusters having a minimum width includes thresholding each pixel in the first two-dimensional matrix with more than a minimum number of points.

[0098] In Example 16, the subject matter of Example 15 includes, wherein determining whether the first two-dimensional matrix includes a first set of clusters having a minimum width includes segmenting the first two-dimensional matrix after thresholding.

[0099] In Example 17, the subject matter of Examples 14-16 includes, wherein the second designated altitude is closer to the ground location than the first designated altitude.

[0100] In Example 18, the subject matter of Examples 14-17 includes, wherein determining the safety status includes determining a nearest identified obstacle.

[0101] In Example 19, the subject matter of Examples 14-18 includes wherein when the safety status indicates that the first obstacle is within a first threshold, an indication is sent to the emergency braking circuit system, and when the first obstacle is outside the first threshold and within a second threshold, an indication is sent to the adjustable braking circuit system.

[0102] In Example 20, the subject matter of Examples 14-19 includes, wherein two or more infrared sensors of the robotic system are part of a safety depth camera, the safety depth camera includes an image capture sensor, and also includes operations including: generating a depth image from a three-dimensional point cloud and a color image captured by the image capture sensor; and using the depth image to identify a change in at least one of: a planned trajectory, a map, or a position or orientation of the robotic system within the map.

[0103] Example 21 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1-20.

[0104] Example 22 is an apparatus comprising means for implementing any of Examples 1-20.

[0105] Example 23 is a system for implementing any of Examples 1-20.

[0106] Example 24 is a method for implementing any of Examples 1-20.

[0107] Although these implementations have been described with reference to specific exemplary aspects, it will be apparent that various modifications and changes can be made to these aspects without departing from the broader scope of the present invention. Many of the arrangements and processes described herein can be combined or used in parallel with implementations for providing greater bandwidth / throughput and for supporting edge service selections that can make them available to the edge system being served. Accordingly, the specification and the accompanying drawings should be considered illustrative rather than restrictive. The attached drawings forming part of this article illustrate specific aspects in which the subject matter can be implemented in an illustrative rather than restrictive manner. The illustrated aspects are described in sufficient detail to enable those skilled in the art to implement the teachings disclosed herein. Other aspects can be utilized and derived therefrom so that structural and logical substitutions and changes can be made without departing from the scope of the present disclosure. Therefore, this specific embodiment is not carried out in a restrictive sense, and the scope of each aspect is limited only by the attached claims and the full scope of equivalent schemes authorized by such claims.

[0108] Such aspects of the inventive subject matter may be referenced herein individually and / or collectively, and if more than one aspect or inventive concept is actually disclosed, this is merely for convenience and is not intended to actively limit the scope of the present application to any single aspect or inventive concept. Thus, although specific aspects have been illustrated and described herein, it should be appreciated that any arrangement that is expected to achieve the same purpose may replace the specific aspects shown. The present disclosure is intended to cover any and all modifications or variations of various aspects. Upon reviewing the above description, the combination of the above aspects and other aspects not specifically described herein will be apparent to those skilled in the art.

[0109] The methods described herein may be at least partially machine or computer-implemented. Some examples may include a computer-readable medium or machine-readable medium encoded with instructions, which may be operable to configure an electronic device to perform the methods described in the above examples. The implementation of such methods may include code, such as microcode, assembly language code, higher-level language code, etc. Such code may include computer-readable instructions for performing various methods. The code may form multiple parts of a computer program product. In addition, in an example, the code may be tangibly stored on one or more volatile, non-transient or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable disks, removable optical disks (e.g., compact disks), memory cards or sticks, random access memory (RAM), read-only memory (ROM), etc.

Claims

1. A device, comprising: Safety control circuit system for: receiving infrared data from at least two infrared receivers affixed to a safety depth camera of the robotic system; determining a safety status of the robotic system relative to a detected object in an environment based on the infrared data; and sending an indication to at least one of an emergency brake circuitry of the robotic system or an adjustable brake circuitry of the robotic system based on the safety status; and Processing circuit system for: receiving the infrared data and the color image from a camera of the safety depth camera; generating a depth image using the infrared data and the color image; and The depth image is used to identify a change in at least one of: a planned trajectory, a map, or a position or orientation of the robotic system within the map.

2. The device according to claim 1, wherein: When the safety status indicates that the detected object is within a first threshold, the indication is sent to the emergency brake circuitry, and when the detected object is outside the first threshold and within a second threshold, the indication is sent to the adjustable brake circuitry.

3. The device of claim 1, wherein: The detected object is a part of a human being.

4. The apparatus of any one of claims 1-3, wherein identifying the change using the depth image comprises using a visual simultaneous localization and mapping (SLAM) algorithm to identify changes in the position and the orientation of the robotic system within the map.

5. A robot system comprising: Security Depth Camera, including: Infrared projector; at least two infrared receivers; and Image capture sensor; Emergency brake circuit system; Adjustable brake circuit system; Safety control circuit system for: receiving infrared data from the at least two infrared receivers; determining a safety status associated with the detected object based on the infrared data; and and sending an indication to at least one of the emergency brake circuitry or the adjustable brake circuitry based on the safety status; and Processing circuit system for: receiving the infrared data and color image from the image capture sensor; generating a depth image using the infrared data and the color image; and The depth image is used to identify a change in at least one of: a planned trajectory, a map, or a position or orientation of the robotic system within the map.

6. The robot system according to claim 5, wherein: The infrared projector, the at least two infrared receivers, and the image capture sensor are located within a single housing of the security depth camera.

7. The robot system according to claim 5, wherein: The infrared data includes a three-dimensional point cloud, and wherein determining the safety status includes the following operations: Projecting the three-dimensional point cloud to a first two-dimensional matrix at a first specified height from a ground position; determining whether the first two-dimensional matrix includes a first set of clusters having a minimum width; and In response to determining that the first two-dimensional matrix includes the first set of clusters having the smallest width, marking the detected objects corresponding to the first set of clusters in the map of an environment.

8. The robot system according to claim 7, wherein: Determining the security status includes the following operations: projecting the three-dimensional point cloud to a second two-dimensional matrix at a second specified height from the ground position; determining whether the second two-dimensional matrix includes a second set of clusters having the minimum width; as well as In response to determining that the second two-dimensional matrix includes the second set of clusters having the minimum width, marking a second detected object corresponding to the second set of clusters in the map of the environment.

9. The robot system of claim 8, wherein: Determining the security status includes determining the security status based on whether the first set of clusters and the second set of clusters are adjacent.

10. The robot system according to claim 5, wherein: When the safety status indicates that the detected object is within a first threshold, the indication is sent to the emergency brake circuitry, and when the detected object is outside the first threshold and within a second threshold, the indication is sent to the adjustable brake circuitry.

11. The robot system according to claim 5, wherein: The detected object is a part of a human being.

12. The robotic system of claim 5, wherein identifying the change using the depth image comprises using a visual simultaneous localization and mapping (SLAM) algorithm to identify changes in the position and the orientation of the robotic system within the map.

13. The robot system according to any one of claims 5 to 12, wherein: Determining the safety status includes determining a nearest identified obstacle.

14. At least one machine-readable medium comprising instructions that, when executed by safety processing circuitry of a robotic system, cause the safety processing circuitry to perform operations comprising: receiving a three-dimensional point cloud of an environment from two or more infrared sensors of the robotic system; Projecting the three-dimensional point cloud to a first two-dimensional matrix at a first specified height from a ground position; determining whether the first two-dimensional matrix includes a first set of clusters having a minimum width; In response to determining that the first two-dimensional matrix includes the first set of clusters having the minimum width, marking a first obstacle corresponding to the first set of clusters in a map of the environment; projecting the three-dimensional point cloud to a second two-dimensional matrix at a second specified height from the ground position; determining whether the second two-dimensional matrix includes a second set of clusters having the minimum width; In response to determining that the second two-dimensional matrix includes the second set of clusters having the minimum width, marking a second obstacle corresponding to the second set of clusters in the map of the environment. determining a security status based on whether the first set of clusters and the second set of clusters are adjacent; as well as An indication is sent to at least one of an emergency brake circuitry of the robotic system or an adjustable brake circuitry of the robotic system based on the safety status.

15. The at least one machine-readable medium of claim 14, wherein: Determining whether the first two-dimensional matrix includes the first set of clusters having the minimum width includes thresholding each pixel in the first two-dimensional matrix with more than a minimum number of points.

16. The at least one machine-readable medium of claim 15, wherein: Determining whether the first two-dimensional matrix includes the first set of clusters having the minimum width includes segmenting the first two-dimensional matrix after the thresholding.

17. The at least one machine-readable medium of claim 14, wherein: The second designated height is closer to the ground position than the first designated height.

18. The at least one machine-readable medium of claim 14, wherein: Determining the safety status includes determining a nearest identified obstacle.

19. The at least one machine-readable medium of claim 14, wherein: When the safety status indicates that the first obstacle is within a first threshold, the indication is sent to the emergency brake circuitry, and when the first obstacle is outside the first threshold and within a second threshold, the indication is sent to the adjustable brake circuitry.

20. At least one machine-readable medium as claimed in any one of claims 14 to 19, wherein: The two or more infrared sensors of the robotic system are part of a safety depth camera, which includes an image capture sensor and further includes operations including: generating a depth image from the three-dimensional point cloud and a color image captured by the image capture sensor; and The depth image is used to identify a change in at least one of: a planned trajectory, a map, or a position or orientation of the robotic system within the map.

Images