Thin object detection and avoidance in flying robots

By using an image sensor and machine learning to analyze pixel changes, flying robots can effectively detect and avoid thin objects, improving navigation and reducing collision risks.

JP2024542034APending Publication Date: 2024-11-13BROOKHURST GARAGE INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2024525711
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-01
Filing Date
2022-11-01
Publication Date
2024-11-13

AI Technical Summary

Technical Problem

Flying robots, such as drones, struggle to detect thin objects like wires and cables due to their size and shape, leading to potential collisions and dangerous conditions.

Method used

Equipping flying robots with an image sensor and a machine learning model, such as a convolutional neural network, to identify thin objects by analyzing pixel changes across images and estimating distance based on movement data from inertial measurement units.

Benefits of technology

Enables accurate detection and avoidance of thin objects, enhancing the robot's navigation and reducing collision risks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542034000001_ABST
    Figure 2024542034000001_ABST
Patent Text Reader

Abstract

The flying robot includes an image sensor for capturing images of an environment. The robot receives a first image captured at a first location. The robot identifies one or more first pixels in the first image. The first pixels correspond to one or more target features of an object identified in the first image. The robot receives a second image captured at a second location. The robot receives distance data estimating movement of the robot from the first location to the second location. The robot identifies a second pixel in the second image. The second pixel corresponds to a target feature of the object appearing in the second image. The robot determines an estimated distance between the robot and the object based on a change in location of the second pixel from the first pixel relative to the movement of the robot as provided by the distance data.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 274,450, filed November 1, 2021, which is incorporated herein by reference in its entirety.

[0002] (Technical field) The present disclosure relates generally to object detection by a robot, and more specifically, to a robot that uses neural networks to detect and avoid thin objects. [Background technology]

[0003] For a flying robot, such as a drone, to be autonomous, the flying robot must navigate through an environment without colliding with objects. Some objects are more difficult for the robot's sensors to detect due to the size and shape of the objects. For example, even the most advanced robots cannot detect power wires and cables because the wires are often too thin for the robot to generate point cloud data with depth measurements of the wires. Without manual control, flying robots often collide with power lines, causing property damage and potentially dangerous conditions. Summary of the Invention [Means for solving the problem]

[0004] An embodiment relates to a flying robot including an image sensor that captures an image of an environment. The flying robot receives a first image of the environment captured at a first location. The flying robot identifies an object in the first image. The object may be a thin object. The flying robot identifies one or more first pixels in the first image that correspond to one or more target features of the identified object. The flying robot receives a second image of the environment captured at a second location. The flying robot receives distance data that estimates a movement of the flying robot from the first location to the second location. The flying robot identifies one or more second pixels in the second image that correspond to a target feature of the object that appears in the second image. The flying robot determines an estimated distance between the flying robot and the object based on a change in position of the second pixel from the first pixel with respect to the movement of the flying robot provided by the distance data. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram illustrating a system environment of an example custodial site, according to some embodiments. [Diagram 2] FIG. 2 is a block diagram illustrating components of an example robot and an example base station, according to some embodiments. [Diagram 3] 1 is a flowchart illustrating an example method of managing inventory at a storage site, according to some embodiments. [Figure 4] FIG. 1 is a conceptual diagram of an example layout of a storage site with a robot, according to some embodiments. [Diagram 5] 1 is a flowchart illustrating an example method of navigation for a robot, according to some embodiments. [Figure 6] FIG. 1 is a conceptual diagram illustrating a robot detecting thin objects in accordance with some embodiments. [Figure 7] 1 is a flowchart illustrating an example method of a robot for detecting thin objects in an environment, according to some embodiments. [Figure 8A] FIG. 1 is a block diagram illustrating an example machine learning model, according to some embodiments. [Figure 8B] FIG. 1 is a block diagram illustrating another example machine learning model, according to some embodiments. [Figure 9] FIG. 1 is a block diagram illustrating components of an example computing machine, according to some embodiments.

[0006] For purposes of illustration only, the figures depict and the detailed description sets forth various non-limiting embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The figures and the following description relate to preferred embodiments by way of example only, and those skilled in the art may recognize alternative embodiments of the structures and methods disclosed herein as practical alternatives that may be employed without departing from the essence of what is disclosed.

[0008] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. It should be noted that wherever feasible similar or similar reference numerals are used in the figures, the similar or similar reference numerals may indicate similar or similar functionality. The figures represent embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the essence described herein.

[0009] An embodiment relates to a flying robot that navigates through an environment by using a machine learning model to identify thin objects in the environment and estimate a distance between the robot and the identified thin objects. The robot may include a thin object detector that uses a convolutional neural network to distinguish pixels corresponding to the thin objects from the remaining pixels. The robot may detect identifiable features of the thin objects and track the movement of the identified features across different image frames captured by the robot's image sensor. Based on the robot's movement and a pose estimate that may be generated by a state estimator and an inertial measurement unit, the robot analyzes the movement of the identified features that appear in the images and determines an estimated distance between the robot and the thin objects.

[0010] System Overview FIG. 1 is a block diagram illustrating an example robot-assisted or fully autonomous storage site system environment 100, according to some embodiments. By way of example, the system environment 100 includes a storage site 110, a robot 120, a base station 130, an inventory management system 140, a computer server 150, a data store 160, and user devices 170. The entities and components of the system environment 100 communicate with each other via a network 180. In various embodiments, the system environment 100 may include different, fewer, or additional components. Also, even when each component of the system environment 100 is described in the singular, the system environment 100 may include one or more of each component. For example, the storage site 110 may include one or more robots 120 and one or more base stations 130. Each robot 120 may have a corresponding base station 130, or multiple robots 120 may share a base station 130.

[0011] The storage site 110 may be any suitable facility for storing, selling, or displaying inventory, such as goods, merchandise, groceries, items, and collections. Exemplary storage sites 110 may include warehouses, inventory lots, bookstores, shoe stores, outlets, other retail stores, libraries, museums, and the like. The storage site 110 may include any number of regularly shaped structures. A regularly shaped structure may be a structure, fixture, equipment, furniture, frame, shell, rack, or other suitable object within a storage site that has a readily identifiable regular shape or contour, whether the object is permanent or temporary, fixed or movable, load-bearing or not. Regularly shaped structures are often used for storing inventory at the storage site 110. For example, racks (including metal racks or shells or frames or other similar structures) are often used for storing goods and merchandise in warehouses. However, a regularly shaped structure need not be used for inventory storage. A storage site 110 may include a particular layout in which various items are placed and systematically stored. For example, in a warehouse, racks may be organized into sections and separated by aisles. Each rack may include multiple pallet locations that may be identified using row and column numbers. A storage site may include high and low racks and, in some cases, may carry most inventory items primarily near ground level.

[0012] Storage site 110 may include one or more robots 120 used to track inventory and manage inventory within storage site 110. For ease of reference, robot 120 may be described in the singular even if more than one robot 120 is used. Also, in some embodiments, more than one type of robot 120 may be present at storage site 110. For example, some robots 120 may be specialized to scan inventory at storage site 110 while other robots 120 may be specialized to move items. Robot 120 may also be referred to as an autonomous robot, or an inventory cycle count robot, or an inventory survey robot, or an inventory detection robot, or an inventory management robot. Inventory robots may be used to track inventory items, move inventory items, and perform other inventory management tasks. The degree of autonomy may vary between embodiments. For example, in some embodiments, robot 120 may be fully autonomous such that robot 120 automatically performs assigned tasks. In other embodiments, the robot 120 may be semi-autonomous such that the robot 120 can move around the storage site 110 with minimal human command or control. In some embodiments, regardless of the degree of autonomy the robot 120 has, the robot 120 may be remotely operated or may be switched to a manual mode. The robot 120 may take a variety of forms, such as an aerial drone, a ground robot, a vehicle, a forklift, or a mobile picking robot.

[0013] The base station 130 may be a device for the robot 120 to return to and for the aerial robot to land. The base station 130 may include one or more return locations. The base station 130 may be used to repower the robot 120. Various methods of repowering the robot 120 may be used in various embodiments. For example, in some embodiments, the base station 130 serves as a battery exchange station to exchange the batteries of the robot 120 when the robot arrives at the base station so that the robot 120 can quickly resume its duties. The exchanged batteries may be charged at the base station 130, either wired or wirelessly. In other embodiments, the base station 130 serves as a charging station with one or more charging terminals that mate with the charging terminals of the robot 120 to recharge the batteries of the robot 120. In still other embodiments, the robot 120 may use fuel for power and the base station 130 may repower the robot 120 by filling its fuel tank.

[0014] The base station 130 may also serve as a communication station for the robots 120. For example, in certain types of storage sites 110, such as warehouses, network coverage may not exist or may only exist in certain locations. The base station 130 may communicate with other components in the system environment 100 using wireless or wired communication channels, such as Wi-Fi or Ethernet cables. When the robot 120 returns to the base station 130, the robot 120 may communicate with the base station 130. The base station 130 may send input, such as commands, to the robot 120 and may download data captured by the robot 120. In embodiments where multiple robots 120 are used, the base station 130 may include a swarm controller or algorithms for coordinating movements between the robots. The base station 130 and the robots 120 may communicate in any suitable manner, such as radio frequency, Bluetooth, near-field communication (NFC), or wired communication. In some embodiments, the robot 120 communicates primarily with the base station, while in other embodiments, the robot 120 may have the ability to communicate directly with other components in the system environment 100. In some embodiments, the base station 130 may serve as a wireless signal amplifier for the robot 120 that communicates directly with the network 180.

[0015] The inventory management system 140 may be a computer system operated by an administrator (e.g., a company that owns the inventory, a warehouse management administrator, a retailer selling the inventory) using the storage site 110. The inventory management system 140 may be a system used to manage the location of inventory items. The inventory management system 140 may include a database that stores data about inventory items and item related information such as quantity and metadata tags, asset type tags, barcode labels, and item location coordinates within the storage site 110. The inventory management system 140 may provide both front-end and back-end software for the administrator to access the central database and inventory metrics to analyze data, generate reports, forecast future demand, and manage the location of inventory items to ensure that items are correctly located. The administrator may rely on the item coordinate data in the inventory management system 140 to ensure that items are correctly located at the storage site 110 so that they can be quickly retrieved from the storage location. This prevents misplaced items from taking up space reserved for upcoming inventory and also reduces the time it takes to locate missing items during the outbound process.

[0016] The computer server 150 may be a server tasked with analyzing data provided by the robot 120 and providing instructions for the robot 120 to perform various inventory recognition and management tasks. The robot 120 may be controlled by the computer server 150 or the user device 170 or the inventory management system 140. For example, the computer server 150 may instruct the robot 120 to scan and capture images of inventory stored at various locations in the storage site 110. Based on the data provided by the inventory management system 140 and the ground truth data captured by the robot 120, the computer server 150 checks for discrepancies between the two sets of data and determines whether an item should be flagged for various reasons, such as misplaced, lost, or damaged. The computer server 150 may then instruct the robot 120 to resolve potential issues, such as moving the misplaced item to its correct location. In some embodiments, the computer server 150 may generate a report of flagged items so that site personnel can manually resolve the issues.

[0017] The computer server 150 may include one or more computers operating at various locations. For example, a portion of the computer server 150 may be a local server that resides at the storage site 110. Computer hardware, such as a processor, may be associated with a computer at the site or may be included in the base station 130. Another portion of the computer server 150 may be a geographically distributed cloud server. The computer server 150 may serve as a ground control station (GCS) and may provide data processing and maintain end-user software used by the user devices 170. The GCS may be responsible for controlling, monitoring, and maintaining the robot 120. In some embodiments, the GCS may reside at the site as part of the base station 130. The data processing pipeline and end-user software servers may reside remotely or on-site.

[0018] The computer server 150 may maintain software applications that allow a user to manage inventory and the base stations 130 and the robots 120. The computer server 150 and the inventory management system 140 may or may not be operated by the same entity. In some embodiments, the computer server 150 may be operated by an entity separate from the storage site administrator. For example, the computer server 150 may be operated by a robotics service provider that supplies the robots 120 and associated systems to modernize and automate the storage site 110. The software applications provided by the computer server 150 may take a variety of forms. In some embodiments, the software applications may be integral to the inventory management system 140 or may be integrated as add-ons to the inventory management system 140. In other embodiments, the software applications may be separate applications that supplement or replace the inventory management system 140. In some embodiments, the software application may be provided as software as a service (SaaS) to administrators of the storage sites 110 by a robot service provider supplying the robots 120.

[0019] The data store 160 includes one or more storage devices, such as memory, in the form of a non-transitory, non-volatile computer storage medium that stores various data uploaded by the robot 120 and the inventory management system 140. For example, the data stored in the data store 160 may include images, sensor data, and other data captured by the robot 120. The data may also include inventory data maintained by the inventory management system 140. A computer-readable storage medium is a medium that does not include a transitory medium such as a propagating signal or carrier wave. The data store 160 may take various forms. In some embodiments, the data store 160 communicates with other components over a network 180. This type of data store 160 may be referred to as a cloud storage server. Exemplary cloud storage service providers may include AWS, AZURE STORAGE, GOOGLE CLOUD STORAGE, etc. In other embodiments, the data store 160 is a storage device controlled and connected to the computer server 150 instead of a cloud storage server. For example, data store 160 may take the form of memory (e.g., a hard drive, flash memory, disk, ROM, etc.) used by computer server 150, such as storage in a storage server room operated by computer server 150.

[0020] The user device 170 may be used by an administrator of the storage site 110 to provide commands to the robot 120 and manage the inventory of the storage site 110. For example, the administrator may use the user device 170 to provide task commands to the robot for the robot to automatically complete a task. In one case, the administrator may specify a specific target location or a range of storage locations for the robot 120 to scan. The administrator may specify specific items for the robot 120 to locate or check placement. Examples of user devices 170 include personal computers (PCs) or desktop computers or laptop computers or tablet computers or wearable electronic devices such as smartphones or smart watches or other suitable electronic devices.

[0021] The user device 170 may include a user interface 175 that takes the form of a graphical user interface (GUI). A software application provided by the computer server 150 or the inventory management system 140 may be displayed as the user interface 175. The user interface 175 may take different forms. In some embodiments, the user interface 175 is part of a front-end software application that includes a GUI that is displayed at the user device 170. In one case, the front-end software application is a software application that may be downloaded and installed at the user device 170 via, for example, an application store (e.g., an App Store) of the user device 170. In other cases, the user interface 175 takes the form of a web interface of the computer server 150 or the inventory management system 140 where a client may perform actions via a web browser. In other embodiments, the user interface 175 does not include graphical elements and communicates with the computer server 150 or the inventory control system 140 through other suitable methods, such as, for example, a command window or application program interfaces (APIs).

[0022] Communications between the robot 120, the base station 130, the inventory management system 140, the computer server 150, the data store 160, and the user devices 170 may be transmitted over a network 180, such as the Internet. In some embodiments, the network 180 uses standard communication technologies and / or protocols. As such, the network 180 may include links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 3G, 4G, LTE, 5G, digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express, etc. Similarly, network protocols used on network 180 may include multiprotocol label switching (MPLS), transmission control protocol / Internet protocol (TCP / IP), user datagram protocol (UDP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), file transfer protocol (FTP), etc. Data exchanged on network 180 may be represented using technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc.Moreover, all or part of the link may be encrypted using conventional encryption techniques such as secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet protocol security (IPsec), etc. Network 180 may also include links and packet switching networks such as the Internet. In some embodiments, two computer servers, such as computer server 150 and inventory management system 140, may communicate via an API. For example, computer server 150 may retrieve inventory data from inventory management system 140 via the API.

[0023] Example robot and base station FIG. 2 is a block diagram illustrating components of an example robot 120 and an example base station 130, according to some embodiments. The robot 120 may include an image sensor 210, a processor 215, a memory 220, a flight control unit (FCU) 225 including an inertial measurement unit (IMU) 230, a state estimator 235, a visual reference engine 240, a planner 250, a communication engine 255, an I / O interface 260, and a power source 265. The functionality of the robot 120 may be distributed among various components in different ways as described below. In various embodiments, the robot 120 may include different and / or fewer and / or additional components. Also, although each of the components in FIG. 2 is described in the singular, a plurality of components may be included. For example, the robot 120 may include one or more image sensors 210 and one or more processors 215.

[0024] The image sensor 210 captures images of the storage environment for purposes of navigation, localization, collision avoidance, object recognition and identification, and inventory recognition. The robot 120 may include one or more image sensors 210 and one or more types of such image sensors 210. For example, the robot 120 may include a digital camera that captures optical images of the environment for the state estimator 235. For example, data captured by the image sensor 210 may be provided to a VIO unit 236 included in the state estimator 235 for localization purposes, such as to determine the position and orientation of the robot 120 with respect to an inertial system, such as a global frame, whose position is known and fixed. The robot 120 may also include a stereo camera including two or more lenses so that the image sensor 210 can capture three-dimensional images through stereo photography. The stereo camera may generate point cloud data including, for example, red, green, and blue (RGB) pixel values ​​and depth information for each image frame. Images captured by the stereo camera may be provided to the visual reference engine 240 for object recognition purposes. The image sensor 210 may be other types of image sensors, such as light detection and ranging (LIDAR) sensors, infrared cameras, and 360-degree depth cameras. The image sensor 210 may also capture images of labels (e.g., barcodes) on items for inventory cycle count purposes. In some embodiments, a single stereo camera may be used for various purposes. For example, the stereo camera may provide image data to the visual reference engine 240 for object recognition. The stereo camera may be used to capture images of labels (e.g., barcodes). In some embodiments, the robot 120 includes a rotating platform, such as a gimbal, that may rotate the image sensor 210 to various angles and stabilize the images captured by the image sensor 210. In some embodiments, the image sensor 210 may capture data along a path for the purpose of mapping a storage site.

[0025] The robot 120 includes one or more processors 215 and one or more memories 220 that store one or more sets of instructions. The one or more sets of instructions, when executed by the one or more processors, cause the one or more processors to execute a method implemented as one or more software engines. Various components of the robot 120, such as the FCU 255 and the state estimator 235, may be implemented as a combination of software and hardware (e.g., sensors). The robot 120 may use a single general processor to execute the various software engines or may use separate, more specialized processors for different functions. In some embodiments, the robot 120 may use a general purpose computer (e.g., CPU) capable of executing various sets of instructions for the various components (e.g., the FCU 225, the visual referencing engine 240, the state estimator 235, the planner 250). The general purpose computer may run on a suitable operating system such as LINUX, ANDROID, etc. For example, in some embodiments, the robot 120 may carry a smartphone that includes an application used to control the robot. In other embodiments, the robot 120 includes multiple processors dedicated to different functions. For example, some of the functional components, such as the FCU 225, the visual referencing engine 240, the state estimator 235, and the planner 250, may be modularized and each includes its own processor, memory, and set of instructions. The robot 120 may include a central processor unit (CPU) to coordinate and communicate with each modularized component. Thus, depending on the embodiment, the robot 120 may include a single processor or multiple processors 215 to perform various tasks. The memory 220 may store images and videos captured by the image sensor 210. The images may include images of the surrounding environment and images of inventory items, such as bar codes and labels.

[0026] The flight control unit (FCU) 225 may control the movement of the robot 120 through a combination of software and hardware such as an inertial measurement unit (IMU) 230 and other sensors. For a ground robot 120, the flight control unit 225 may be referred to as a microcontroller unit (MCU). The FCU 225 controls the movement of the robot 120 based on information provided by other components. For example, a planner 250 determines the path of the robot 120 from a start point to a destination and provides commands to the FCU 225. Based on the commands, the FCU 225 generates electrical signals to various mechanical components (e.g., actuators, motors, engines, wheels) of the robot 120 to regulate the movement of the robot 120. The precise mechanical components of the robot 120 may vary depending on the embodiment and the type of robot 120.

[0027] The IMU 230 may be part of the FCU 225 or may be an independent component. The IMU 230 may include one or more accelerometers, gyroscopes, and other suitable sensors to generate measurements of forces, linear acceleration, and rotation of the robot 120. For example, the accelerometers measure forces acting on the robot 120 and sense linear acceleration. The multiple accelerometers work together to sense the acceleration of the robot 120 in three-dimensional space. For example, a first accelerometer senses acceleration in the x-direction, a second accelerometer senses acceleration in the y-direction, and a third accelerometer senses acceleration in the z-direction. The gyroscopes sense the rotation and angular acceleration of the robot 120. Based on the measurements, the processor 215 may obtain an estimated position of the robot 120 by accumulating the movement and rotation data of the IMU 230 over time. The IMU 230 may measure the orientation of the robot 120. For example, the gyroscopes of the IMU 230 may provide readings of the pitch, roll, and yaw angles of the robot 120.

[0028] The state estimator 235 may represent a set of software instructions stored in the memory 220 that may be executed by the processor 215. The state estimator 235 may be used to generate localization information for the robot 120 and may include various sub-components for estimating the state of the robot 120. For example, in some embodiments, the state estimator 235 may include a visual-inertial odometry (VIO) unit 236 and an altitude estimator 238. In other embodiments, other modules, sensors, and algorithms may be used by the state estimator 238 to localize the robot 120.

[0029] The VIO unit 236 receives image data from the image sensor 210 (e.g., stereo camera) and measurements from the IMU 230 and generates localization information such as the position and orientation of the robot 120. The position data obtained from double integration of acceleration measurements from the IMU 230 is often prone to drift errors. The VIO unit 236 extracts image features and tracks the feature points in the image sequence to generate optical flow vectors that represent the movement of edges, boundaries, and surfaces of objects in the environment captured by the image sensor 210. Various signal processing techniques such as filtering (e.g., Wiener filter, Kalman filter, bandpass filter, particle filter) and optimization, and data / image transformations may be used to reduce various errors in the determination of localization information. The position data generated by the VIO unit 236 may include an estimate of the pose of the robot 120, which may be expressed in terms of the roll, pitch, and yaw angles of the robot 120.

[0030] The altitude estimator 238 may be a combination of software and hardware used to measure the absolute and relative altitude of the robot 120 (e.g., distance from an object on the floor). The altitude estimator 238 may include a distance sensor 239 that measures the altitude of the robot 120 relative to the ground or relative to an object below the robot 120. The distance sensor 239 may be based on electromagnetic, laser, optical, acoustic, ultrasonic, or other suitable signals. For example, the distance sensor 239 may be a laser range finder, a lidar range finder, an acoustic range finder, an ultrasonic range finder, or a radar. The range finder may also include one or more emitters that emit a signal (e.g., infrared, laser, acoustic, etc.) and one or more sensors that sense the round trip time of the signal reflecting off an object. In some embodiments, the robot 120 may be equipped with a single emitter range finder. The altitude estimator 238 may receive data from the VIO unit 236 that estimates the altitude of the robot 120, but typically in a less accurate manner than the distance sensor 239. If the robot 120 flies over various objects and inventory placed on the floor or other horizontal surface, the altitude estimator 238 may include software algorithms to combine data generated by the distance sensor 239 and data generated by the VIO unit 236. The data generated by the altitude estimator 238 may be used for collision avoidance and target location. The altitude estimator 238 may set a global maximum altitude to prevent the robot 120 from hitting the ceiling. The altitude estimator 238 also provides information about how many rows of racks are below the robot 120 so that the robot 120 can locate the target location. The altitude data may be used along with the number of rows the robot 120 has passed to determine the vertical level of the robot 120.

[0031] The visual referencing engine 240 may represent a set of software instructions stored in the memory 220 that may be executed by the processor 215. The visual referencing engine 240 may include various image processing and position algorithms to determine the current position of the robot 120, identify objects, edges and surfaces in the environment near the robot 120, and determine an estimated distance and orientation (e.g., yaw) of the robot 120 relative to nearby surfaces of objects. The visual referencing engine 240 may receive pixel data of successive images and point cloud data from the image sensor 210. The position information generated by the visual referencing engine 240 may include the distance and yaw from the object and the offset of the center from a target point (e.g., the center point of the target object).

[0032] The visual reference engine 240 may include one or more algorithms and machine learning models for creating an image segmentation from an image captured by the image sensor 210. The image segmentation may include one or more segments separating frames (e.g., vertical or horizontal bars of a rack) or contours of regularly shaped structures appearing in the captured image from other objects and the environment. Algorithms used for image segmentation may include a convolutional neural network (CNN). Other image segmentation algorithms may be used in performing the segmentation, such as edge detection algorithms (e.g., Canny operator, Laplace operator, Sobel operator, Prewitt operator), corner detection algorithms, Hough transform, and other suitable feature detection algorithms.

[0033] The visual reference engine 240 also performs object recognition (e.g., object detection and further analysis) and keeps track of the relative motion of objects across successive images. The visual reference engine 240 may track the number of regularly shaped structures of the storage site 110 that the robot 120 has passed over. For example, the visual reference engine 240 may identify a reference point (e.g., center of gravity) on the frame of a rack and determine across successive images whether the reference point passes through a particular location of the image (e.g., whether the reference point passes through the center of the image). If so, the visual reference engine 240 increments the number of regularly shaped structures that the robot 120 has passed over.

[0034] The robot 120 may use various components to generate various position information, including position information and localization information relative to nearby objects. For example, in some embodiments, the state estimator 235 may process data from the VIO unit 236 and the altitude estimator 238 to provide localization information to the planner 250. The visual referencing engine 240 may count the number of regularly shaped structures that the robot 120 has passed to determine the current position. The visual referencing engine 240 may generate position information relative to nearby objects. For example, when the robot 120 reaches a target position on a rack, the visual referencing engine 240 may use point cloud data to reconstruct the surface of the rack and may use depth data from the point cloud to determine more accurate yaw and distance between the robot 120 and the rack. The visual referencing engine 240 may determine a center offset that corresponds to the distance between the robot 120 and the center of the target position (e.g., the center point of the target position on the rack). The planner 250 uses the centroid offset information to control the robot 120 to move to a target location and take a picture of the inventory at the target location. When the robot 120 changes direction (e.g., turns, transitions from horizontal to vertical motion, transitions from vertical to horizontal motion, etc.), the centroid offset information may be used to determine the exact position of the robot 120 relative to the object.

[0035] The planner 250 may correspond to a set of software instructions stored in the memory 220 that may be executed by the processor 215. The planner 250 may include various path planning algorithms for planning a path for the robot 120 as the robot moves from a first location (e.g., a start location, a current location of the robot 120 after completing a previous leg) to a second location (e.g., a target location). The robot 120 may receive input, such as a user command, to perform a specific action (e.g., scanning inventory, moving items, etc.) at a specific location. The planner 250 may include two types of routes, corresponding to spot checks and range scans. In spot checks, the planner 250 may receive inputs including coordinates of one or more specific target locations. In response, the planner 250 plans a path for the robot 120 to move to the target locations and perform an action. In range scans, the inputs may include a range of coordinates corresponding to a range of target locations. In response, the planner 250 plans a path for the robot 120 to execute actions for a full scan or range of target locations.

[0036] The planner 250 may plan a route for the robot 120 based on the data provided by the visual reference engine 240 and the data provided by the state estimator 235. For example, the visual reference engine 240 may estimate the current location of the robot 120 by tracking the number of regularly shaped structures in the storage site 110 that the robot 120 has passed by. Based on the position information provided by the visual reference engine 240, the planner 250 may determine a route for the robot 120 and adjust the movement of the robot 120 as it moves along the route.

[0037] Planner 250 may include fail-safe mechanisms in case the motion of robot 120 deviates from the plan. For example, if planner 250 determines that robot 120 has passed a target path and is traveling too far from the target path, planner 250 may send a signal to FCU 225 to attempt to correct the path. If the error is not corrected after a timeout or within a reasonable distance, or if planner 250 cannot accurately determine the current location, planner 250 may instruct the FCU to land or stop robot 120.

[0038] Relying on various position information, the planner 250 may also include algorithms for collision avoidance purposes. In some embodiments, the planner 250 relies on distance information, yaw angle, and center offset information relative to nearby objects to plan the movements of the robot 120 to provide sufficient clearance between the robot 120 and nearby objects. Alternatively or additionally, the robot 120 may include one or more depth cameras, such as a set of 360-degree depth cameras, that generate distance data between the robot 120 and nearby objects. The planner 250 uses the position information from the depth cameras to perform collision avoidance.

[0039] The communication engine 255 and I / O interface 260 are communication components that allow the robot 120 to communicate with other components in the system environment 100. The robot 120 may communicate with external components, such as the base station 130, wirelessly or wired using different communication protocols. Exemplary communication protocols may include Wi-Fi, Bluetooth, NFC, USB, etc., that connect the robot 120 to the base station 130. The robot 120 may send various types of data, such as image data, flight logs, position data, inventory data, and robot status information. The robot 120 may receive input from external sources to identify actions that the robot 120 needs to perform. Commands may be generated automatically or manually by an administrator. The communication engine 255 may include algorithms for various communication protocols and standards, encoding, decoding, multiplexing, traffic control, data encryption, etc., for various communication methods. The I / O interface 260 may include software and hardware components, such as hardware interfaces, antennas, etc., for communication.

[0040] The robot 120 also includes a power source 265 that is used to power the various components and movements of the robot 120. The power source 265 may be one or more batteries or a fuel tank. Exemplary batteries may include lithium ion batteries, lithium polymer (LiPo) batteries, fuel cells, and other suitable types of batteries. The batteries may be permanently in place or may be easily replaced. For example, the batteries may be removable so that they can be replaced when the robot 120 returns to the base station 130.

[0041] Although Figure 2 shows various example components, the robot 120 may include additional components. For example, mechanical features and components of the robot 120 are not shown in Figure 2. The robot 120 may include various types of motors, actuators, robotic arms, lifts, other moving components, and other sensors, depending on the type of robot, to perform various tasks.

[0042] 2, the example base station 130 includes a processor 270, a memory 275, an I / O interface 280, and a repowering unit 285. In various embodiments, base station 130 may include different and / or fewer and / or additional components.

[0043] The base station 130 includes a processor 270 and one or more memories 275 that include one or more sets of instructions that cause the processor 270 to perform various methods implemented as one or more software modules. The base station 130 may provide input and commands to the robot 120 to perform various inventory control tasks. The base station 130 may include sets of instructions to perform group control among multiple robots 120. Group control may include task allocation, routing and planning, coordinating movements between the robots to avoid collisions, etc. The base station 130 may be used as a central control unit to coordinate the robots 120. The memory 275 may also include various sets of instructions to perform analysis of data and images downloaded from the robot 120. The base station 130 may provide various degrees of data processing, from format conversion of raw data to complete data processing to generate information useful for inventory management. Alternatively or additionally, base station 130 may directly upload data downloaded from robot 120 to a data store, such as data store 160. Base station 130 may provide operational and management commands to robot 120. In some embodiments, base station 130 may be remotely controlled by user device 170 or computer server 150 or inventory management system 140.

[0044] The base station 130 may include various types of I / O interfaces 280 for communication with the robot 120 and for the Internet. The base station 130 may continuously communicate with the robot 120 using a wireless protocol such as Wi-Fi or Bluetooth. In some embodiments, one or more components of the robot 120 of FIG. 2 may be in a base station, which may provide commands to the robot 120 for movement and movement. Alternatively or additionally, when the robot 120 lands or stops at the base station 130, the base station 130 may communicate with the robot 120 via a short-range communication protocol such as NFC or a wired connection. The base station 130 may connect to a network 180, such as the Internet. The wireless network (e.g., LAN) of the storage site 110 may not have sufficient coverage. The base station 130 may connect to the network 180 via an Ethernet cable.

[0045] The repowering unit 285 includes components used to detect the power level of the robot 120 and repower the robot 120. Repowering can be done by replacing batteries, recharging batteries, refilling fuel tanks, etc. In some embodiments, the base station 130 includes a mechanical actuator, such as a robot arm, to replace batteries in the robot 120. In other embodiments, the base station 130 may be used as a charging station for the robot 120, either by wired charging or inductive charging. For example, the base station 130 may include a landing or resting pad with an induction coil on the bottom to wirelessly charge the robot 120 via an induction coil within the robot. Other suitable methods of repowering the robot 120 are also possible.

[0046] Example inventory management method 3 is a flow chart depicting an example method of managing inventory at a storage site, according to some embodiments. The method may be performed by a computer, which may be a single operating unit in the traditional sense (e.g., a single personal computer) or a set of distributed computers working together to execute a set of instructions (e.g., virtual machines, distributed computer systems, cloud computing, etc.). Also, although a computer is described in the singular, a computer performing the method of FIG. 3 may include one or more computers in conjunction with a computer server 150, or an inventory management system 140, or a robot 120, or a base station 130, or a user device 170.

[0047] According to some embodiments, a computer receives 310 a configuration of the storage site 110. The storage site 110 may be a warehouse or a retail store or other suitable location. The configuration information of the storage site 110 may be uploaded to the robot 120 for the robot to navigate the storage site 110. The configuration information may include the total number of regularly shaped structures in the storage site 110 and dimensional information of the regularly shaped structures. The configuration information provided may be in the form of a computer-aided design (CAD) drawing or other type of file format. The configuration may include a layout of the storage site 110, such as the layout of racks and the arrangement of other regularly shaped structures. The layout may be a two-dimensional layout. The computer extracts the number of sections, aisles, and racks, and the number of rows and columns of each rack from the CAD drawing by counting the number of them that appear in the CAD drawing. The computer may extract the height and width of the rack cells from the CAD drawing or other source. In some embodiments, the computer does not need to extract the exact distance between a given set of racks or the width of each aisle or the total length of the racks. Instead, the robot 120 may measure the aisle, rack and cell dimensions from depth sensor data or may use a counting method implemented by the planner 250 in conjunction with the visual referencing engine 240 to traverse the storage sites 110 by counting the number of rows and columns that the robot 120 traverses. Therefore, in some embodiments, the exact dimensions of the racks may not be required.

[0048] Some configuration information may be manually entered by an administrator of the storage site 110. For example, the administrator may provide the number of sections, the number of aisles and racks in each section, and the cell dimensions of the racks. The administrator may enter the number of rows and columns for each rack.

[0049] Alternatively or additionally, configuration information may also be obtained from a mapping method, such as a pre-flight mapping or a mapping method performed when the robot 120 performs inventory control operations. For example, for a storage site 110 that is newly performing an automated management method, an administrator may provide the size of the navigable space of the storage site to one or more mapping robots to count the number of sections, aisles, and rows and columns of regularly shaped structures within the storage site 110. Also, in some embodiments, the mapping or configuration information does not need to measure precise distances between racks or other structures within the storage site 110. Instead, the robot 120 may navigate the storage site 110 with only a rough layout of the storage site 110 by counting regularly shaped structures along a path to identify target locations. As the robot 120 continues to perform various inventory control operations, the robotic system may gradually perform mapping or estimations of the size and location of various structures.

[0050] The computer receives (320) inventory control data for inventory control operations at the storage site 110. Certain inventory control data may be manually entered by an administrator while other data may be downloaded from the inventory management system 140. The inventory control data may include scheduling and planning for inventory control activities, including frequency of operations, time windows, etc. For example, the management data may specify that each location of the racks at the storage site 110 should be scanned at certain intervals (e.g., daily) and that the inventory scanning method should be performed by the robot 120 at night after the storage site is closed. The inventory management system 140 data may provide bar codes and labels of items, exact coordinates of the inventory items, information about the racks and other storage space that needs to be cleared for the next inventory items, etc. The inventory control data may also include items that need to be retrieved from the storage site 110 each day (e.g., purchase order items that need to be shipped) and the robot 120 may need to focus on those items.

[0051] The computer creates 330 a plan to perform the inventory control. For example, the computer may create an automated plan that includes various commands to instruct the robot 120 to perform various scans. The commands may specify a range of locations the robot 120 needs to scan or one or more specific locations the robot 120 needs to go to. The computer may estimate the time per scanning trip and design the plan for each work interval based on the time available for the robotic inventory control. For example, at a particular storage site 110, robotic inventory control is not performed during business hours.

[0052] The computer generates 340 various commands to operate one or more robots 120 to move around the storage site 110 according to the plan and information derived from the configuration of the storage site 110. The robot 120 may move around the storage site 110 by at least visually recognizing the regularly shaped structures in the storage site and counting the number of regularly shaped structures. In some embodiments, in addition to the localization techniques such as those used by VIO, the robot 120 counts the number of racks, rows, and columns that it has passed in order to determine its current location along a path from a start location to a target location without knowing the exact distance and direction that the robot 120 has traveled.

[0053] Inventory scanning or other inventory control operations may be performed autonomously by the robot 120. In some embodiments, the scanning operation begins at a base station where the robot 120 receives input including coordinates of a target location or a range of target locations within the storage site 110 (342). The robot 120 departs from the base station 130 (344). The robot 120 navigates the storage site 110 by visually recognizing regularly shaped structures (346). For example, the robot 120 tracks the number of regularly shaped structures that the robot 120 has passed. The robot 120 pivots and translates based on the regularly shaped structures that it recognizes as captured by the image sensor 210. Upon arriving at the target location, the robot 120 may align itself with a reference point (e.g., a center position) of the target location. At the target location, the robot 120 captures 348 target location data (e.g., measurements, photos, etc.) including the inventory item, the barcode, and the label on the inventory item's box. If the initial command before the robot 120 departs includes multiple target locations or a range of target locations, the robot 120 continues to the next target location by moving up, down, or sideways to the next location to continue the scanning operation.

[0054] After completing the scanning trip, the robot 120 returns to the base station 130 by counting (350) the number of regularly shaped structures that the robot has passed in the reverse direction. The robot 120 may potentially recognize the structures that the robot has passed as the robot 120 moves to the target location. Alternatively, the robot 120 may return to the base station 120 by reversing the path without counting. The base station 130 repowers the robot 120. For example, the base station 130 provides the robot 120 with the next command and replaces the robot 120's batteries (352) so that the robot 120 can return quickly to service another scanning trip. Used batteries may be charged at the base station 130. The base station 130 may download the data and images captured by the robot 120 and upload the data and images to the data store 160 for further processing. Alternatively, the robot 120 may include wireless communication components for transmitting its data and images to the base station 130 or directly to the network 180 .

[0055] The computer performs an analysis of the data and images captured by the robot 120 (360). For example, the computer may compare bar codes (including serial numbers) in the images captured by the robot 120 to data stored in the inventory management system 140 to determine if an item has been misplaced or lost within the storage site 110. The computer may determine other conditions of the inventory. The computer may generate reports for display on the administrator user interface 175 to take corrective action for misplaced or lost inventory. For example, reports may be generated daily so that personnel at the storage site 110 manually locate and move misplaced items. Alternatively or additionally, the computer may generate an automation plan for the robot 120 to move misplaced inventory. The data and images captured by the robot 120 may be used to verify the removal or arrival of inventory items.

[0056] Example of movement method FIG. 4 is a schematic diagram of an example layout of a storage site 110 with a robot 120, according to some embodiments. FIG. 4 shows a two-dimensional layout of the storage site 110 with a close-up of an example rack shown in inset 405. The storage site 110 may be divided into various regions based on regularly shaped structures. In this example, the regularly shaped structures are racks 410. The storage site 110 may be divided by sections 415, aisles 420, rows 430, and columns 440. For example, a section 415 is a collection of racks. Each aisle may have racks on both sides. Each rack 410 may include one or more columns 440 and multiple rows 430. A storage unit of racks 410 may be referred to as a cell 450. Each cell 450 may carry one or more pallets 460. In this particular example, two pallets 460 are placed in each cell 450. Inventory at storage site 110 is transported on pallets 460. The division and designations shown in Figure 4 are used as examples only; storage site 110 in other embodiments may be divided in different ways.

[0057] Each inventory item at the storage site 110 may be located on a pallet 460. The target location of the inventory item (e.g., pallet location) may be identified using a coordinate system. For example, an item placed on a pallet 460 may have an aisle number (A), a rack number (K), a row number (R), and a column number (C). For example, a pallet location coordinate of [A3, K1, R4, C5] means that the pallet 460 is located at rack 410 in the north rack of the third aisle. The location of pallet 460 in rack 410 is the fourth row and fifth column (counting from the ground). In some cases, such as the particular layout shown in FIG. 4, an aisle 420 may have racks 410 on both sides. Further coordinate information may be used to identify the racks 410 on the north side of the aisle 420 and the racks 410 on the south side of the aisle 420. Alternatively, the upper and lower racks may have different aisle numbers. For spot checks, the robot 120 may be provided with a single coordinate if there is only one location, or multiple coordinates if there is more than one location. For a range scan to check for pallets 460 within a range, the robot 120 may be provided with a range of coordinates, such as an aisle number, a rack number, a start row, a start column, an end row, and an end column. In some embodiments, the location coordinates of a pallet may be referred to in different ways. For example, in one case, the coordinate system may take the form "aisle-rack-shelf-location." The shelf number may correspond to the row number, and the location number may correspond to the column number.

[0058] 5 in conjunction with FIG. 4, FIG. 5 is a flow chart illustrating an example method of movement of the robot 120, according to some embodiments. The robot 120 receives 510 a target location 474 at the storage site 110. The target location 474 may be expressed in a coordinate system, as described above with respect to FIG. 4. The target location 474 may be received as an input command from the base station 130. The input command may also include an action that the robot 120 needs to take, such as taking a picture at the target location 474 to capture the barcode and label of an inventory item. The robot 120 may generate location information based on the VIO unit 236 and the altitude estimator 238. In some cases, the start location of the route is the base station 130. In some cases, the start location of the route may be anywhere at the storage site 110. For example, the robot 120 may have recently completed a task and may begin another task without returning to the base station 130.

[0059] A processor of the robot 120, such as executing the planner 250, controls 520 the robot 120 along a path 470 to a target location 474. The path 470 may be determined based on the coordinates of the target location 474. The robot 120 may rotate so that the image sensor 210 faces a regularly shaped structure (e.g., a rack). Movement of the robot 120 to the target location 474 may include moving to a particular aisle, rotating to enter the aisle, moving horizontally to a row of targets, moving vertically to a row of targets, and rotating to the correct angle to face the target location 474 to take a picture of the inventory items on the pallet 460.

[0060] As the robot 120 moves to the target location 474, the robot 120 captures (530) images of the storage site 110 using the image sensor 210. The captured images may be sequential images. The robot 120 receives images captured by the image sensor 210 as the robot 120 moves along the path 470. The images may capture objects in the environment, including regularly shaped structures such as racks. For example, the robot 120 may use algorithms in the visual reference engine 240 to visually recognize regularly shaped structures.

[0061] The robot 120 determines (540) its current location on the path 470 by analyzing the images captured by the image sensor 210 and tracking the number of regularly shaped structures in the storage site that the robot 120 has passed. The robot 120 may use various image processing and object recognition techniques to identify the regularly shaped structures and track the number of structures that the robot 120 has passed. With respect to the path 470 shown in FIG. 4, the robot 120 may move to a turning point 476 while facing the racks 410. The robot 120 determines that it has reached the target aisle because it has passed two racks 410. In response, the robot 120 rotates counterclockwise to enter the target aisle facing the target racks. The robot 120 counts the number of rows it has passed until it reaches the target row. According to the target rows, the robot 120 may move vertically up or down to reach the target location. Upon reaching the target location, the robot 120 performs the action specified in the input command, such as taking a picture of the inventory item at the target location.

[0062] Exemplary Thin Object Detection Method FIG. 6 is a conceptual diagram illustrating a robot 600 for detecting thin objects, according to some embodiments. The robot 600 may be an aerial robot and may be exemplary of the robot 120 used in the storage site 110. Although FIGS. 1-5 focus on a robot moving within a storage site, the robot 600 may be used in other settings, such as outdoor or urban environments, where thin objects are common. For example, a drone designed for urban areas may be equipped with the ability to detect thin objects to avoid collisions with power lines, signs, and other thin objects. The robot 600 may include some or all of the components shown in FIG. 2. For example, in some embodiments, the robot 600 includes at least the image sensor 210, the processor 215, the memory 220, the state estimator 235, and the IMU 230. Other suitable configurations of the robot 600 are possible.

[0063] A thin object may refer to an object that is thin in one or more dimensions. Examples of thin objects may include electrical wires, horizontal and vertical poles, other cables, and the like. In the storage site 110, thin objects such as wires, chains, connecting cables, and the like may exist. In different settings, various thin objects exist. Whether an object is thin may depend on how far away the object is from the robot 600. For example, a first object that is farther from the robot 600 appears thinner than a second object that is closer to the robot 600. In some embodiments, whether an object is thin may be defined based on the number of pixels occupied by the object in the thin dimension when the object is captured in an image. For example, in some embodiments, an object may be classified as thin if the thickness of the object in the thin dimension is less than 10 pixels. In some embodiments, an object may be classified as thin if the thickness of the object in the thin dimension is less than 5 pixels. In some embodiments, whether an object is thin may be defined based on the angular resolution of the image sensor. For example, a typical camera installed on a flying robot may have a certain angular resolution. In some embodiments, an object may be classified as a thin object if the angle it occupies is less than 2 degrees, 1.5 degrees, 1 degree, etc. In Figure 6, the thin object 610 is represented by a horizontal bar 610 supported by its stand 612. Note that the setup in Figure 6 is for illustrative purposes only. Also, although the thin object 610 is shown as a horizontal thin object, the thin object may have different orientations, such as vertical or diagonal.

[0064] The robot 600 includes one or more image sensors for capturing images in the environment. The image sensor used by the robot 600 may be a mono camera with a single lens or a stereo camera with multiple lenses. In some embodiments, a technique for detecting thin objects allows the robot 600 with a mono camera to detect various thin objects. The robot 600 may capture images continuously using the image sensor. For example, at a first location 620, the image sensor captures a first image 630. For simplicity, only the thin object 610 is shown in the first image 630, but the first image 630 may include various objects captured by the image sensor.

[0065] The robot 600 may include a thin object detector for identifying thin objects in images captured by the image sensor. The thin object detector may be a software algorithm that may be stored in a memory, such as the memory 220, and executed by one or more processors, such as the processor 215, to analyze the images. The thin object detector may include one or more machine learning models used to analyze the images. The structure and training of the machine learning models are shown in FIG. 8A and FIG. 8B. For example, FIG. 8B shows an example structure of a convolutional neural network that may be specifically trained to identify thin objects in images. The thin object detector may use the machine learning model to mark pixels of the image that are identified as corresponding to thin objects. The thin object detector may segment the thin object from the rest of the scene.

[0066] The robot 600 may also include a state estimator used to determine the pose of the robot 600 relative to the thin object 610. The pose of the robot 600 may be represented by the pose of the image sensor. The state estimator may be a combination of hardware sensors and software algorithms used to track the position and localization information of the robot 600. An example of a state estimator may be the state estimator 235 described in FIG. 2. The state estimator may also include the IMU 230, which generates acceleration and orientation data. The acceleration data may be converted to distance data. The robot 600 uses data from the IMU 230 and potentially the VIO 236 to determine the pose of the robot 600 relative to the thin object 610.

[0067] The robot 600 includes a depth estimator that estimates the distance between the robot 600 and the thin object 610 at a given location. The depth estimator may be a software algorithm stored in memory and executed by one or more processors that uses results from the thin object detector and state estimator to determine an estimated distance between the robot 600 and the thin object 610. The depth estimator determines the estimated distance based on two or more images captured at different locations by comparing pixels of the thin object 610 captured in the images. The robot 600 captures the thin object 610 at different locations in the images as the robot 600 moves from a first location to a second location. The depth estimator compares pixel locations and IMU distance data to estimate the distance between the robot 600 and the thin object 610.

[0068] As an example, at the first location 620, the thin object detector marks pixels corresponding to the thin object 610 in the first image 630. The depth estimator receives the pixel locations of the thin object in the first image 630. The depth estimator identifies pixel locations 632 of one or more specific features of the thin object 610. The features may be easily identifiable parts or regions of the thin object 610 that are continuously and identifiably imaged when the image sensor captures the thin object 610 from different perspectives. For example, the features may be a centroid, a center, a far end, a quantile, an identifiable subdivision, or any other suitable part associated with the thin object 610. In the first image 630, the depth estimator identifies both ends of the thin object 610 as features and marks two pixel locations 632 corresponding to the far ends. For example, the depth estimator may identify pixel locations having the highest and lowest values ​​in a particular dimension based on the pixel coordinates to determine the two far ends.

[0069] The depth estimator compares the pixel locations of features after the robot 600 moves from the first location 620 to the second location 640. At the second location 640, the image sensor of the robot 600 captures a second image 650. In the second image 650, the thin object 610 appears longer because the robot 600 is located closer to the thin object 610 at the second location 640. The level of the thin object 610 may also change because the robot 600 may have moved vertically. The thin object detector analyzes the second image 650 and marks the pixels corresponding to the thin object 610. The depth estimator receives the pixel locations from the object detector and identifies the features of the thin object 610. For example, the depth estimator again identifies the two far ends of the thin object 610 and determines pixel locations 652 corresponding to the two far ends. The pixel locations 652 are changed relative to pixel locations 632 as the robot 600 moves to the second location 640.

[0070] The depth estimator determines an estimated distance between the robot 600 and the thin object 610 based on the difference between pixel locations 632 and 652 and based on the distance data and pose provided by the state estimator. For example, the distance data generated by the IMU may be used to estimate the distance the robot 600 has traveled from the first location 620 to the second location 640. The state estimator may also provide the pose of the robot 600 at the two locations. The distance between the robot 600 and the thin object 610 may be calculated based on mathematical operations such as projection, linear transformation, and geometric relationships.

[0071] FIG. 7 is a flow chart illustrating an example of a robotic method for detecting thin objects in an environment, according to some embodiments. The method illustrated in FIG. 7 may be a detailed example of the method illustrated in FIG. 6. The method illustrated in FIG. 7 may be performed through a software algorithm stored as computer instructions executable by one or more processors (e.g., CPU) of the robot. The instructions, when executed by the processor, cause the processor to perform various steps described in the method. In various embodiments, one or more steps in the method may be skipped or modified. The robot performing the method may be robot 600.

[0072] The robot receives (710) a first image of the environment. The first image may be captured at a first location. The robot includes an image sensor to capture the image. In some implementations, the image may be captured by a mono camera and may not need to include a point cloud or any three-dimensional data. The first location may be an instantaneous location of the robot as it moves along a path. The method illustrated in FIG. 7 may be a continuous method that is repeatedly performed as the robot moves to different locations. The image may capture various objects in the environment. Some of the objects may be thin objects that are difficult to detect using traditional image sensors and point cloud data because the objects correspond to only a very small number of pixels in one or more dimensions. The robot often cannot generate traditional point cloud data that includes depth data for thin objects because depth changes in the location of the thin objects are often indistinguishable from noise.

[0073] The robot identifies an object in the first image (720). The object may be a thin object. The robot may include a thin object detector running a machine learning model trained to identify a specific target object. For example, the machine learning model may be a convolutional neural network (CNN) including one or more dilated convolution layers. An example structure of a CNN is shown in FIG. 8B. The machine learning model receives the first image as an input and tags pixels corresponding to the identified object as an output. For example, the CNN may be trained to specifically identify thin objects. For example, the CNN may be trained using a training set of images of various environments containing different target objects. Training the CNN may include iteratively reducing an error in identifying the location of the thin object relative to the training set of images. Detailed steps in training the machine learning model are described in FIG. 8A.

[0074] Although identifying thin objects is described as an example, the method depicted in FIG. 7 may also be used to identify other types of objects and to determine the distance between a robot and an object.

[0075] The robot identifies (730) one or more first pixels in the first image. The one or more first pixels may correspond to one or more target features of the identified object. Identifying the first pixels may include various sub-steps. As an example, the robot may tag pixels in the projected first image that correspond to the identified object. The robot may cluster the pixels to form multiple contours. The robot may merge the contours to form a merged contour. The robot may identify the target feature from pixels in the merged contour.

[0076] The method may include various image segmentation and object identification techniques to separate pixels corresponding to identified objects from the background. Image segmentation may be performed by any suitable algorithm. In some embodiments, the robot may have a machine learning model perform image segmentation and assign tags to pixels. In some embodiments, the robot may also input a series of images to a machine learning model, which may output an image segmentation from the series of images, taking into account objects that appear consecutively in the series of images. Instead of or in addition to CNN, other types of machine learning models, such as another type of neural network, clustering, Markov Random Fields (MRF), etc., may also be used in the image segmentation method. Instead of or in addition to using machine learning techniques, other image segmentation algorithms may also be used, such as edge detection algorithms (e.g., Canny operator, Laplacian operator, Sobel operator, Prewitt operator), corner detection algorithms, Hough transform, and other suitable feature detection algorithms.

[0077] From the pixels tagged with symbols representing the identified objects, the robot may create contours for the regions of those pixels. Each contour may be referred to as a pixel cluster. These clustering of pixels may be based on distance between pixels, pixel color, pixel intensity, and / or other suitable characteristics of the pixels. The robot may cluster pixels that are similar (e.g., in terms of distance, color, and / or intensity) to create contours that are likely to correspond to regions of the identified objects. Multiple clusters corresponding to different sub-regions may be created for each identified object (e.g., each wire or another type of thin object). The robot may determine a reference point for each contour. The reference point may be the center of gravity, the most extreme point in one direction, or any relevant reference point. For example, the robot may determine the average of the pixel locations of the pixels that are within the contour to determine the center of gravity.

[0078] The robot may perform noise filtering and contour merging, which merges contours based on their respective fiducial points. For noise filtering, the robot may filter contours based on size before merging. For example, the robot may consider contours whose area is less than a threshold (e.g., contours that are too small) as noise. The robot keeps contours that are large enough. For contour merging, in some cases, the clustering algorithm results in pixels corresponding to an identified object being classified into multiple contours (e.g., multiple regions of a wire). The robot merges contours that are likely to represent the same identified object. Merging may be based on the location of the fiducial points of the contours and the boundaries of the contours. For example, for a horizontal wire, the robot may identify contours that have fiducial points at a similar vertical level and merge those contours. In some cases, when two contours are merged, pixels between the two contours that may belong to a smaller cluster or not be identified in any cluster may also be classified as the same structure. Merging may be based on the distance between two fiducial points of the two contours. If the distance is less than a threshold, the robot may merge the two contours.

[0079] Upon isolating the pixels corresponding to the identified object from the remaining pixels, the robot identifies one or more target features within the object. The target feature may be a specific location of the quantile relative to the length of the object. For example, for a thin object, the target features may include the far ends, center, and quantiles. For other types of objects, in addition to or instead of a specific location, the target feature may be an identifiable part of the object. For example, in a storage site setting, if a structure is the identified object, the structure may include identifiable parts such as corner locations, screw locations, pattern locations, sign locations, etc. The robot may use object recognition techniques to identify these parts. If the target feature is a specific location relative to the length of the object, the robot may use the coordinate values ​​of the segmented pixels to determine the target feature location. For example, in some embodiments, the robot tracks the two far ends of the identified thin object and stores the coordinates in the first image.

[0080] The robot receives (740) a second image of the scene. The second image is captured at a second location, different from the first location. As the robot moves, its image sensor continues to capture additional images of the scene. In some cases, the second image may be the frame immediately following the first image. In other cases, the second image is a subsequent frame that shows a sufficiently large change in image content compared to the first image. Although FIG. 7 is illustrated with two images, in some embodiments the robot may also analyze a series of multiple images to refine the estimation of the distance between the robot and the identified object. This method may also be performed repeatedly to dynamically determine distance as the robot moves to different locations.

[0081] The robot receives range data (750). The range data estimates the movement of the flying robot from a first location to a second location. The range data may be generated by an IMU. The robot uses a state estimator to determine the distance traveled by the robot from the first location to the second location. The state estimator may also estimate a relative pose of the image sensor with respect to objects identified at the first location and the second location, respectively.

[0082] The robot identifies one or more second pixels in the second image (760). The one or more second pixels correspond to a target feature of the object. The identification of the second pixels is similar to the identification of the first pixel described in step 730. In some embodiments, the robot also relies on the distance data and pose data generated by the state estimator to identify the second pixels. Based on the results of the state estimator, the robot may estimate the travel distance and the change in orientation of the robot. The robot may project a neighborhood of the target feature (the location of the one or more first pixels) in the second image based on the distance data and pose data. The robot may use the projection to identify the target feature. For example, the robot may search for the target feature in the projected neighborhood. The result may be used in combination with the object recognition result generated by the CNN.

[0083] The robot determines (770) an estimated distance between the robot and the object based on the change in position of one or more second pixels from one or more first pixels relative to the robot's movement as provided by the distance data. This estimate may be determined by a depth estimator of the robot. The distance between the robot and the identified object may be calculated based on a mathematical operation such as projection, linear transformation, or geometric relationship. For example, if the robot's movement is b between a first pixel and a second pixel, and the disparity of the corresponding pixel locations is d, then the distance z between the robot and the object may be calculated by z=f*b / d, where f is the focal length of the camera. The depth estimator may send the distance to a flight control unit to instruct the robot to avoid a collision with the identified object. For example, the flight control unit may change the route of the robot to avoid the object.

[0084] Example machine learning model In various embodiments, a wide variety of machine learning techniques may be used. Examples include various forms of supervised, unsupervised, and semi-supervised learning, such as decision trees, support vector machines (SVMs), regression, Bayesian networks, and genetic algorithms. Deep learning techniques such as neural networks, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory networks (LSTMs), may also be used. For example, various object recognition, localization, object recognition, particularly of thin objects, and other processes performed by visual reference engine 240 may apply one or more machine learning and deep learning techniques.

[0085] In various embodiments, training techniques for machine learning models may be supervised, semi-supervised, or unsupervised. In supervised learning, the machine learning model may be trained on a set of labeled training examples. For example, for a machine learning model trained to classify objects, the training examples may be various pictures of objects with a label for the type of object. The label for each training example may be binary or multi-class. In training a machine learning model for image segmentation, the training examples may be pictures of regularly shaped objects at various repository sites with portions of the images manually identified. In some cases, unsupervised learning techniques may be used. The samples used for training are unlabeled. Various unsupervised learning techniques, such as clustering, may be used. In some cases, training may be semi-supervised, with a training set having a mixture of labeled and unlabeled examples.

[0086] A machine learning model may be associated with an objective function that generates a metric that describes the desired goal of the training method. For example, training may be intended to reduce the model's error rate in generating predictions. In this case, the objective function may monitor the error rate of the machine learning model. In object recognition (e.g., object detection and classification), the objective function of a machine learning algorithm may be the training error rate in classifying the objects in the training set. Such an objective function may be called a loss function. Other forms of objective functions may be used, especially for unsupervised learning models where the error rate is not easily determined because the objects are not labeled. In image segmentation, the objective function may correspond to the difference between the segments predicted by the model and the segments manually identified in the training set. In various embodiments, the error rate may be measured as a cross-entropy loss, an L1 loss (e.g., the sum of absolute differences between the predicted and actual values), or an L2 loss (e.g., the sum of squared distances).

[0087] Referring to FIG. 8A, an example CNN structure is shown according to some embodiments. The CNN 800 may receive an input 810 and generate an output 820. The CNN 800 may include various types of layers, such as a convolutional layer 830, a pooling layer 840, a recurrent layer 850, a fully connected layer 860, and a custom layer 870. The convolutional layer 830 convolves the input of the layer (e.g., an image) with one or more kernels to generate various types of images filtered by the kernels and generate feature maps. Each convolution result may be associated with an activation function. The convolutional layer 830 may be followed by a pooling layer 840 that selects the maximum value (max pooling) or the average value (average pooling) from the portion of the input covered by the kernel size. The pooling layer 840 reduces the spatial size of the extracted features. In some embodiments, the pair of convolutional layers 830 and pooling layer 840 may be followed by a recurrent layer 850 that includes one or more feedback loops 855. The feedback 855 may be used to estimate spatial relationships of image features or temporal relationships of objects in the image. Layers 830, 840, and 850 may be followed by a number of fully connected layers 860 with interconnected nodes (shown as squares in FIG. 8A ). The fully connected layers 860 may be used for classification and object detection. In some embodiments, one or more custom layers 870 may be present for generation of the output 820 in a specific format. For example, a custom layer may be used in image segmentation for labeling pixels of the image input with various segment labels.

[0088] The order of layers and number of layers of the CNN 800 in FIG. 8A are for illustration only. In various embodiments, the CNN 800 includes one or more convolutional layers 830, but may or may not include a pooling layer 840 or a recurrent layer 850. If a pooling layer 840 is present, not all convolutional layers 830 are always followed by a pooling layer 840. Recurrent layers may be arranged differently in other positions of the CNN. In each convolutional layer 830, the size of the kernels (e.g., 3×3, 5×5, 7×7, etc.) and the number of kernels allowed to be trained may be different from the other convolutional layers 830.

[0089] A machine learning model may include specific layers and / or nodes and / or kernels and / or coefficients. Training a neural network such as the CNN800 may include forward propagation and backpropagation. Each layer of a neural network may include one or more nodes that are fully or partially connected to other nodes in adjacent layers. In forward propagation, the neural network performs a forward calculation based on the output of the previous layer. The work of a node may be defined by one or more functions. The functions that define the work of a node may include various computational tasks such as convolution of data with one or more kernels, pooling, recurrent loops in RNNs, various gates in LSTMs, etc. The functions may also include activation functions that adjust the weights of the node's output. Nodes in different layers may be associated with different functions.

[0090] Each of the functions of the neural network may be associated with various coefficients (e.g., weights and kernel coefficients) that can be adjusted during training. Additionally, some of the nodes of the neural network may be associated with activation functions that determine the weights of the node's output in forward propagation. Common activation functions may include step functions, linear functions, sigmoid functions, hyperbolic tangent (tanh) functions, and rectified linear unit (ReLU) functions. After an input is provided to the neural network and passed forward through the neural network, the results may be compared with the training labels or other values ​​of the training set to determine the performance of the neural network. The prediction method may be repeated for other images in the training set to calculate the value of the objective function for a particular training round. In turn, the neural network performs backpropagation by a gradient descent method, such as stocgastic gradient descent (SGD), to adjust the coefficients of the various functions to improve the value of the objective function.

[0091] Multiple rounds of forward and backpropagation may be performed. Training may be terminated when the objective function is sufficiently stable (e.g., the machine learning model has converged), or after a predefined number of rounds on a particular set of training samples. The trained machine learning model may be used for prediction or object detection or image segmentation or any other suitable task for which the model was trained.

[0092] FIG. 8B is a conceptual diagram illustrating an example CNN 880 structured and trained to identify thin objects, according to some embodiments. The CNN 880 may be an example of the CNN 800, and the training techniques discussed in FIG. 8A may also be used for the CNN 880. The CNN 880 may include a front-end layer 882 and a context layer 884. The front-end layer 882 includes a convolutional layer 830 and a pooling layer 840 similar to the CNN 800. The size of the kernel, the number of convolutional layers 830, and the pooling parameters may be customized depending on the embodiment. The front-end layer may be used to detect edges in the input image and identify certain low-level patterns in the image.

[0093] The context layer 884 includes one or more dilated convolution layers 890 that are used to generate the output 895. Each dilated convolution layer 890 may be associated with a dilation coefficient. A kernel with a dilation coefficient is dilated in size and filled with zeros in the dilated space. For example, a dilation coefficient of 2 inserts zeros between two values ​​in a row and inserts rows with zeros between the original rows. A higher dilation coefficient inserts more zeros and further expands the size of the kernel. Common dilation coefficients may be 2, 4, 8, 16, etc. The dilated convolutions may enable the CNN 880 to distinguish larger patterns from more local patterns. The context layer 884 includes one or more dilated convolution layers 890 that may improve the performance of the CNN 880 in detecting thin objects such as wires and cables. In some implementations, some of the dilated convolution layers 890 may have large dilation coefficients. For example, a series of dilated convolution layers 890 may have dilation coefficients d1-d2-d4-d8.

[0094] The CNN880 may be trained with a set of training samples, which are images containing various thin objects. The training samples may be generated by real images of various scenes in different environments, such as indoors, outdoors, or with different backgrounds. To generate more training samples, some of the images may be further manipulated, such as by rotating, scaling, tilting, adjusting contrast, adjusting the color tone of the images, etc. Thin objects in the images may also be adjusted to make some of the images simulate different conditions, such as lighting and weather.

[0095] Computer Architecture 9 is a block diagram illustrating components of an exemplary computing machine capable of reading instructions from a computer-readable medium and executing the instructions on a processor (or controller). The computers described herein may include a single computing machine as shown in FIG. 9, or a virtual machine, or a distributed computing system including multiple nodes of the computing machine as shown in FIG. 9, or any other suitable arrangement of computing devices.

[0096] 9 shows a diagrammatic representation of a computing machine in the exemplary form of a computer system 900 in which instructions 924 (e.g., software or program code or machine code) stored on a computer-readable medium may be executed to cause the machine to perform one or more methods described herein. In some embodiments, the computing machine may operate as a stand-alone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or may operate as a peer machine in a peer-to-peer (or distributed) network environment.

[0097] The computing machine configuration described in Figure 9 may correspond to any software or hardware components or combinations of components shown in Figures 1 and 2, including, but not limited to, inventory management system 140, computer server 150, data store 160, user device 170, and the various engines, modules, interfaces, terminals, and machines shown in Figure 2. Although Figure 9 shows various hardware and software elements, each of the components described in Figures 1 and 2 may include additional or fewer elements.

[0098] By way of example, the computing machine may be a personal computer (PC), or a tablet PC, or a set-top box, or a personal digital assistant (PDA), or a mobile phone, or a smartphone, or a web appliance, or a network router, or an internet of things (IoT) appliance, or a switch or bridge, or any machine capable of executing instructions 924 that specify actions to be taken by the machine. Additionally, although only one machine is illustrated, the term "machine" may be taken to include a collection of machines that individually or collectively execute instructions 924 to perform one or more methodologies described herein.

[0099] The exemplary computer system 900 includes one or more processors (generally, processor 902) (e.g., a central processing unit (CPU), or a graphics processing unit (GPU), or a digital signal processor (DSP), or one or more application-specific integrated circuits (ASICs), or one or more radio-frequency integrated circuits (RFICs), or a combination thereof), a main memory 904, and a non-volatile memory 906, which are configured to communicate with each other via a bus 908. The computer system 900 may further include a graphics display device 910 (e.g., a plasma display panel (PDP) or a liquid crystal display (LCD) or a projector or a cathode ray tube (CRT)). The computer system 900 may also include an alphanumeric input device 912 (e.g., a keyboard), a cursor control device 914 (e.g., a mouse or a trackball or a joystick or a motion sensor or other pointing means), a storage device 916, a signal generating device 918 (e.g., a speaker), and a network interface device 920, which are also configured to communicate via the bus 908.

[0100] The storage device 916 includes a computer-readable medium 922 having stored thereon instructions 924 embodying one or more of the methodologies or functions described herein. The instructions 924 may reside, completely or at least partially, within the main memory 904 or within the processor 902 (e.g., within a processor cache memory) during execution of the instructions by the computer system 900, with the main memory 904 and the processor 902 also constituting computer-readable media. The instructions 924 may be sent or received over a network 926 via a network interface device 920.

[0101] Although computer readable medium 922 is shown to be a single medium in the illustrated embodiment, the term "computer readable medium" should be interpreted to include a medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) capable of storing instructions (e.g., instructions 924). Computer readable media may include any medium capable of storing instructions (e.g., instructions 924) for execution by a machine, causing the machine to perform one or more of the methodologies described herein. Computer readable media includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media. Computer readable media does not include transitory media such as signals or carrier waves.

[0102] Further configuration considerations Particular embodiments are described herein as including logic circuits, or a number of components or engines or modules or mechanisms. An engine may comprise a software module (e.g., code embodied on a computer-readable medium) or a hardware module. A hardware engine is a tangible device capable of performing certain operations, and may be configured or arranged in a particular manner. In example embodiments, one or more hardware engines of one or more computer systems (e.g., standalone, client, or server computer systems) or computer systems (e.g., processors or groups of processors) may be configured with software (e.g., applications or application portions) as hardware engines that operate to perform certain operations as described herein.

[0103] In various embodiments, the hardware engine may be implemented mechanically or electronically. For example, the hardware engine may include dedicated circuitry or logic circuitry that is permanently configured (e.g., as a dedicated processor such as a field programmable gate array (FPGA) or application-specific integrated circuit (ASIC)) to perform specific tasks. The hardware engine may also include programmable logic circuitry or circuitry that is temporarily configured by software to perform specific tasks (e.g., contained within a general-purpose processor or other programmable processor). The decision to implement a hardware engine of mechanical, dedicated, permanently configured circuitry, or temporarily configured circuitry (e.g., configured in software) may be based on cost and time considerations.

[0104] Various operations of the example methods described herein may be performed, at least in part, by one or more processors, such as processor 902, that are temporarily or permanently configured (e.g., by software) to perform the associated operations. Whether temporarily or permanently configured, such processors may constitute a processor-implemented engine that operates to perform one or more operations or functions. An engine as referred to herein may, in some example embodiments, include a processor-implemented engine.

[0105] Performance of a particular task may be distributed across one or more processors located across several machines, rather than just residing within one machine. In some example embodiments, one or more processors or processor-implemented modules may be located in one geographic location (e.g., in a home environment or in an office environment or in a server farm). In other example embodiments, one or more processors or processor-implemented modules may be distributed across several geographic locations.

[0106] Upon reading this disclosure, those skilled in the art will recognize still further alternative structural and functional designs for similar systems or methods through the principles disclosed herein. Thus, while specific embodiments and applications have been shown and described, it should be understood that the disclosed embodiments are not limited to the exact structures and components disclosed herein. It will be apparent to those skilled in the art that various modifications, changes and variations may be made in the details of the arrangements and operations and methods disclosed herein without departing from the spirit and scope as defined by the appended claims.

[0107] (Additional Note) (Appendix 1) 1. A method of operating an aerial robot, comprising: receiving a first image of an environment captured by the flying robot at a first location; Identifying an object in the first image; identifying one or more first pixels in the first image that correspond to one or more target features of the object; receiving a second image of an environment captured by the flying robot at a second location; receiving distance data of the flying robot that estimates a movement of the flying robot from the first location to the second location; identifying one or more second pixels in the second image that correspond to the one or more target features of the object; and determining an estimated distance between the flying robot and the object based on a change in location of one or more of the second pixels from one or more of the first pixels with respect to a movement of the flying robot provided by the distance data; A method comprising:

[0108] (Appendix 2) 2. The method of claim 1, wherein the object is identified by a convolutional neural network.

[0109] (Appendix 3) 3. The method of claim 2, wherein the convolutional neural network includes a dilated convolutional layer.

[0110] (Appendix 4) Identifying one or more first pixels in the first image includes: tagging pixels in the first image that are projected to correspond to the object; clustering the pixels to form a plurality of contours; merging the contours to form a merged contour; and identifying the one or more target features from pixels within the merged contour; The method of claim 1, comprising:

[0111] (Appendix 5) 5. The method of claim 4, wherein one or more of the contours are filtered based on size before merging.

[0112] (Appendix 6) 2. The method of claim 1, wherein the one or more targeted features are one or more at a far end of the object.

[0113] (Appendix 7) Identifying one or more of the second pixels in the second image includes: projecting one or more locations of the first pixel in the second image based on the distance data; and identifying one or more of the second pixels based on the projected locations of one or more of the first pixels; The method of claim 1, comprising:

[0114] (Appendix 8) 2. The method of claim 1, wherein the object is a thin object having a width of less than 10 pixels.

[0115] (Appendix 9) 2. The method of claim 1, wherein the object is a wire.

[0116] (Appendix 10) 2. The method of claim 1, wherein the distance data is generated from an inertial measurement unit (IMU).

[0117] (Appendix 11) one or more processors; a memory configured to store instructions; A flying robot comprising: The instructions, when executed by the one or more processors, receiving a first image of an environment captured at a first location; Identifying an object in the first image; identifying one or more first pixels in the first image that correspond to one or more target features of the object; receiving a second image of the environment captured at a second location; receiving distance data of the flying robot that estimates a movement of the flying robot from the first location to the second location; identifying one or more second pixels in the second image that correspond to the one or more target features of the object; and determining an estimated distance between the flying robot and the object based on a change in location of one or more of the second pixels from one or more of the first pixels with respect to a movement of the flying robot provided by the distance data; causing said one or more processors to execute Flying robot.

[0118] (Appendix 12) 12. The flying robot of claim 11, wherein the object is identified by a convolutional neural network.

[0119] (Appendix 13) 13. The flying robot of claim 12, wherein the convolutional neural network includes a dilated convolutional layer.

[0120] (Appendix 14) The instructions for identifying one or more first pixels in the first image include: tagging pixels in the first image that are projected to correspond to the object; clustering said pixels to form a plurality of contours; merging the contours to form a merged contour; and identifying the one or more target features from pixels within the merged contour; comprising instructions for causing the processor to execute 12. The flying robot of claim 11.

[0121] (Appendix 15) 15. The flying robot of claim 14, wherein one or more of the contours are filtered based on size before merging.

[0122] (Appendix 16) 12. The flying robot of claim 11, wherein the one or more targeted features are one or more at a far end of the object.

[0123] (Appendix 17) The instructions for identifying one or more second pixels in the second image include: projecting one or more locations of the first pixel in the second image based on the distance data; and identifying one or more of the second pixels based on the projected locations of one or more of the first pixels; comprising instructions for causing the processor to execute 12. The flying robot of claim 11.

[0124] (Appendix 18) 12. The flying robot of claim 11, wherein the object is a thin object having a width of less than 10 pixels.

[0125] (Appendix 19) 12. The flying robot of claim 11, wherein the object is a wire.

[0126] (Appendix 20) 12. The flying robot of claim 11, further comprising an inertial measurement unit (IMU), wherein the distance data is generated from the IMU.

Claims

1. 1. A method of operating an aerial robot, comprising: receiving a first image of an environment captured by the flying robot at a first location; Identifying an object in the first image; identifying one or more first pixels in the first image that correspond to one or more targeted features of the object; receiving a second image of an environment captured by the flying robot at a second location; receiving distance data of the flying robot that estimates a movement of the flying robot from the first location to the second location; identifying one or more second pixels in the second image that correspond to the one or more target features of the object; and determining an estimated distance between the flying robot and the object based on a change in location of one or more of the second pixels from one or more of the first pixels with respect to a movement of the flying robot provided by the distance data; A method comprising:

2. The method of claim 1 , wherein the objects are identified by a convolutional neural network.

3. The method of claim 2 , wherein the convolutional neural network includes a dilated convolutional layer.

4. Identifying one or more of the first pixels in the first image includes: tagging pixels in the first image that are projected to correspond to the object; clustering the pixels to form a plurality of contours; merging the contours to form a merged contour; and identifying the one or more target features from pixels within the merged contour; The method of claim 1 , comprising:

5. The method of claim 4 , wherein one or more of the contours are filtered based on size before merging.

6. The method of claim 1 , wherein the one or more targeted features are one or more at a far end of the object.

7. Identifying one or more of the second pixels in the second image includes: projecting one or more locations of the first pixel in the second image based on the distance data; and identifying one or more of the second pixels based on the projected locations of one or more of the first pixels; The method of claim 1 , comprising:

8. The method of claim 1 , wherein the object is a thin object having a width of less than 10 pixels.

9. The method of claim 1 , wherein the object is a wire.

10. The method of claim 1 , wherein the distance data is generated from an inertial measurement unit (IMU).

11. one or more processors; a memory configured to store instructions; A flying robot comprising: The instructions, when executed by the one or more processors, receiving a first image of an environment captured at a first location; Identifying an object in the first image; identifying one or more first pixels in the first image that correspond to one or more targeted features of the object; receiving a second image of the environment captured at a second location; receiving distance data of the flying robot that estimates a movement of the flying robot from the first location to the second location; identifying one or more second pixels in the second image that correspond to the one or more target features of the object; and determining an estimated distance between the flying robot and the object based on a change in location of one or more of the second pixels from one or more of the first pixels with respect to a movement of the flying robot provided by the distance data; causing the one or more processors to execute Flying robot.

12. The flying robot of claim 11 , wherein the object is identified by a convolutional neural network.

13. The flying robot of claim 12 , wherein the convolutional neural network includes a dilated convolutional layer.

14. The instructions for identifying one or more first pixels in the first image include: tagging pixels in the first image that are projected to correspond to the object; clustering said pixels to form a plurality of contours; merging the contours to form a merged contour; and identifying the one or more target features from pixels within the merged contour; comprising instructions for causing the processor to execute The flying robot according to claim 11.

15. The flying robot of claim 14 , wherein one or more of the contours are filtered based on size before merging.

16. The flying robot of claim 11 , wherein the one or more targeted features are one or more at a far end of the object.

17. The instructions for identifying one or more second pixels in the second image include: projecting one or more locations of the first pixel in the second image based on the distance data; and identifying one or more of the second pixels based on the projected locations of one or more of the first pixels; comprising instructions for causing the processor to execute The flying robot according to claim 11.

18. The flying robot of claim 11 , wherein the object is a thin object having a width of less than 10 pixels.

19. The flying robot of claim 11 , wherein the object is a wire.

20. The flying robot of claim 11 , further comprising an inertial measurement unit (IMU), wherein the distance data is generated from the IMU.

Citation Information

Patent Citations

  • Autonomous work vehicles, various work vehicles, drones, or the like, and method for controlling autonomous work vehicles or the like

    JP2019204500A

  • Distance measurement method, program, distance measurement system and movable object

    JP2020030204A

  • Computer vision-based thin object detection

    US20200226392A1