Robot sensing apparatus and sensor planning method
Through the computer-implemented sensor planning method combined with deep learning algorithm, the sensor sampling process is optimized, the problems of sampling errors and inefficiency in sensor planning are solved, and efficient and accurate capture of the region of interest is achieved.
Patent Information
- Application Number
- CN202511110066.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2016-11-03
- Filing Date
- 2017-10-11
- Publication Date
- 2025-09-19
AI Technical Summary
Existing sensor planning methods are prone to sampling errors, omissions, or failures when capturing unknown or changing areas of interest, and manual programming and large-scale sampling methods are inefficient.
A computer-implemented sensor planning method is used to define the area of interest through computing equipment, identify sensor sensing parameters, determine the sampling combination, and control the sensor to obtain samples, and combine deep learning algorithms to optimize the sampling process.
This enables efficient capture of samples in regions of interest at the desired level of detail and completeness, reducing redundancy and time consumption.
Smart Images
Figure CN120663329A_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with international application number "PCT / US2017 / 056072" filed with the International Bureau on October 11, 2017, and national phase application number "201780080502.1" which entered the Chinese national phase on June 25, 2019, and the invention name is "Robot Sensing Device and Sensor Planning Method". Technical Field
[0002] The present disclosure generally relates to robotic sensing devices and sensor planning methods. Background Art
[0003] Sensor planning is a general requirement in inspection, measurement, and robotic positioning, navigation, or mapping relative to an area of interest. The area of interest can include a component, part, detail, assembly, or spatial region, such as a geographic area or other 2D or 3D space. Additionally, a general requirement for inspection and measurement methods, as well as autonomous robots, is to employ sensors that capture samples (e.g., images or measurements) of the area of interest in sufficient detail and with the desired level of integrity.
[0004] A known solution for sensor planning is to utilize manually programmed sensor plans, such as coordinate systems, routes, or paths, to capture a desired region of interest. However, manually programmed sensor plans typically require an unchanging and / or essentially fixed region of interest. Consequently, deviations from the pre-programmed region of interest often result in sampling errors, omissions, or other failures.
[0005] Another known solution for sensor planning can involve programming a robot to capture a large number of samples to ensure that the region of interest is captured with sufficient detail and the desired level of completeness. However, capturing a large number of samples is also expensive, time-consuming, and results in an inefficient number of redundant samples. Additionally, an unknown nominal region of interest or variations from the nominal region of interest can similarly lead to errors, omissions, or failures to capture the desired region of interest.
[0006] Therefore, a need exists for robotic sensing systems and sensor planning methods that can capture samples of desired and / or potentially unknown or changing areas of interest with sufficient detail and completeness while minimizing redundancy and time. Summary of the Invention
[0007] Aspects and advantages of the invention will be set forth in part in the following description, or may be obvious from the description, or may be learned through practice of the invention.
[0008] The present disclosure is directed to a computer-implemented method for sensor planning for acquiring samples via an apparatus comprising one or more sensors. The computer-implemented method includes: defining an area of interest by one or more computing devices; identifying one or more sensing parameters of one or more sensors by the one or more computing devices; determining, by the one or more computing devices, a sampling combination for acquiring a plurality of samples by the one or more sensors based at least in part on the one or more sensing parameters; and providing, by the one or more computing devices, one or more command control signals to the apparatus comprising the one or more sensors to acquire a plurality of samples of the area of interest using the one or more sensors based at least on the sampling combination.
[0009] Another aspect of the present disclosure is directed to a robotic sensing apparatus for sensor planning. The apparatus includes one or more sensors and a computing device, wherein the computing device includes one or more processors and one or more memory devices. The one or more memory devices store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations include: receiving an area of interest; receiving one or more sensing parameters from one or more sensors; determining a sampling combination for acquiring a plurality of samples from the one or more sensors; and acquiring the plurality of samples using the one or more sensors based at least on the sampling combination.
[0010] Yet another aspect of the present disclosure is directed to an apparatus for sensor planning that includes a translational robotic apparatus, one or more sensors mounted to the translational robotic apparatus, and one or more computing devices configured to operate the translational robotic apparatus and the one or more sensors.
[0011] These and other features, aspects and advantages of the present invention will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] A full and enabling disclosure of the invention, including the best mode thereof, is set forth in this specification, which refers to the accompanying drawings, in which:
[0013] Figure 1 is an exemplary embodiment of a robotic sensing device;
[0014] Figure 2 is an exemplary embodiment of another robotic sensing device;
[0015] Figure 3 is a flow chart outlining an exemplary sensor planning method; and
[0016] Figure 4is yet another exemplary embodiment of a robotic sensing device.
[0017] Repeat use of reference characters in the present specification and drawings is intended to represent same or analogous features or elements of the invention. DETAILED DESCRIPTION
[0018] Reference will now be made in detail to embodiments of the present invention, one or more examples of which are illustrated in the accompanying drawings. Each embodiment is provided to explain the present invention, not to limit the present invention. In fact, it will be apparent to those skilled in the art that various modifications and variations may be made in the present invention without departing from the scope or spirit of the present invention. For example, features shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present invention is intended to cover such modifications and variations as fall within the scope of the appended claims and their equivalents.
[0019] As used herein, the terms “first,” “second,” and “third” may be used interchangeably to distinguish one component from another and are not intended to indicate the position or importance of the respective components.
[0020] Generally, a robotic sensing apparatus and sensor planning method are provided that can capture samples of an area of interest while minimizing redundancy and time. The methods and systems described herein can include steps or operations that can capture samples of an area of interest (e.g., a component, assembly, or geographic area) at a desired resolution and level of completeness while minimizing the number of samples (e.g., images or measurements) used to capture the area of interest. Various embodiments of the robotic sensing apparatus and methods described herein can utilize deep learning methods in conjunction with sensor planning. In addition, the systems and methods described herein can generally autonomously plan and capture the minimum number of samples to capture the area of interest at a desired level of completeness.
[0021] Now refer to Figure 1 and Figure 2A robotic sensing apparatus 90 for sensor planning (herein referred to as "apparatus 90") includes one or more sensors 110 for acquiring samples of an area of interest 130. The one or more sensors 110 may include an imaging device, a proximity sensor, or a combination thereof. In one embodiment, the imaging device may generally include a camera. In another embodiment, the imaging device may specifically include an interferometer, such as, but not limited to, optical coherence tomography (e.g., a white light scanner or a blue light scanner). In other embodiments, the one or more sensors 110 include proximity sensors, which may generally include sensors that can emit and / or retrieve electromagnetic signals and process changes in the electromagnetic signals. For example, proximity sensors may include, but are not limited to, capacitive, infrared, inductive, magnetic, sonic or ultrasonic proximity sensors, radar, LIDAR, or laser rangefinders. In various embodiments, the one or more sensors 110 may include a combination of imaging devices and / or proximity sensors. In various embodiments, the one or more sensors 110 acquire samples, including images or measurements, at various resolutions, angles, distances, orientations, sampling or measurement rates, frequencies, and the like.
[0022] In one embodiment, the apparatus 90 includes a translatable robotic apparatus 100 (referred to herein as "robot 100"). The robot 100 may include a movable fixture, such as Figure 1 and Figure 2 The robotic arm shown in , or an autonomous mobile vehicle, e.g. Figure 4 The robot 100, the one or more sensors 110 and / or the area of interest 130 may include, but is not limited to, six-axis motion (e.g., up / down, left / right, forward / backward, etc.), pivoting, turning, rotation and / or displacement at a constant or variable rate of motion. Figure 1 and Figure 2 In the illustrated embodiment, the sensor 110 may be mounted to the robot 100 , wherein the robot 100 translates the sensor 110 to various portions 131 of the area of interest 130 at various angles or distances relative to the area of interest 130 .
[0023] In other embodiments of the apparatus 90, the robot 100 can translate the area of interest 130 relative to the one or more sensors 110. For example, the robot 100 (e.g., a robotic arm) can translate the area of interest 130 relative to one or more fixed sensors 110. The robot 100 can translate the area of interest 130 to various distances, angles, and / or orientations relative to the one or more sensors 110.
[0024] Now refer to Figure 3 , generally provides a flowchart outlining the steps of an exemplary embodiment of a sensor planning method 300 (referred to herein as “method 300 ”). Figure 3 The method 300 shown in FIG. Figure 1 and Figure 2 The method 300 may also be performed by one or more computing devices (such as Figure 4 The described computing device 120) is implemented. Figure 3 The steps are depicted as being performed in a specific order for the purposes of illustration and discussion. Using the disclosure provided herein, one of ordinary skill in the art will understand that the steps of any method disclosed herein may be modified, adjusted, expanded, rearranged, and / or omitted in various ways without departing from the scope of the present disclosure.
[0025] Method 300 may include, at (310), defining an area of interest by one or more computing devices, identifying, at (320), one or more sensing parameters of one or more sensors by one or more computing devices, determining, at (330), a sampling combination for acquiring a plurality of samples by the one or more sensors based at least in part on the one or more sensing parameters by the one or more computing devices, and, at (340), providing, by the one or more computing devices, one or more command control signals to an apparatus including the one or more sensors to acquire a plurality of samples of the area of interest using the one or more sensors based at least on the sampling combination.
[0026] At 310, method 300 may include defining an area of interest. In one embodiment, defining the area of interest includes receiving a point cloud. Receiving the point cloud may include receiving an image file, such as a computer-aided design (CAD) file, of the area of interest. The image file may include a nominal file of the area of interest relative to which samples from the sensor may be measured.
[0027] In another embodiment, defining the area of interest may include defining a confined space in which the robot and / or one or more sensors may operate, e.g. Figure 1 , 2 and 4 , and / or the one or more sensors 110. For example, the extent to which the robot 100 can translate can be spatially limited. In one example, the robot 100, as a robotic arm, can be limited in its range of motion, reach, etc. In another example, the robot 100, as a drone, can be geographically limited by coordinates, operating range, or operating envelope (e.g., altitude, speed, maneuverability, etc.). Thus, defining an area of interest can include defining a 2D or 3D space in which samples can be collected.
[0028] In other embodiments, defining the area of interest may include acquiring samples of the area of interest. For example, acquiring samples of the area of interest may include acquiring samples that broadly capture the area of interest, including the edges of the area of interest. Broadly capturing the area of interest may include sampling at low resolution, or at a significant distance from the area of interest, or with minimal detail, to obtain and define the perimeter of the area of interest. In various embodiments, broadly capturing the area of interest may include capturing a defined spatial region limited by coordinates, operating range, operating envelope, etc. In other embodiments, broadly capturing the area of interest may depend on a maximum sampling area of one or more sensors, such that the area of interest may be defined by one or more sensing parameters.
[0029] At (320), method 300 includes identifying one or more sensing parameters of one or more sensors. In one embodiment, identifying the one or more sensing parameters may include defining one or more of a measurement resolution, a field of view, and / or a depth of field. Defining the measurement resolution may include defining a lateral resolution, a lens resolution, an image resolution, and / or a sensor resolution. Defining the sensor resolution may include defining a spatial and / or temporal sensor resolution.
[0030] In another embodiment, identifying one or more sensing parameters may include defining a total area covered by one or more sensors. For example, defining the total area covered by one or more sensors may be a function of one or more of the aforementioned resolutions defined. As another non-limiting example, for example, with respect to Figure 1 , 2 and 4, the total area of coverage can be approximately equal to or less than the portion 131 of the area of interest 130 captured by the one or more sensors 110. In one embodiment, the total area covered by the one or more sensors can be a function of one or more of the aforementioned resolutions defined based at least on the desired sample quality and additional user-defined limitations. For example, the total area of coverage can include the hardware capabilities of the one or more sensors and a subset of the hardware capabilities defined based at least on the user-defined limitations. The user-defined limitations can generally be based on user-defined good visibility criteria. The good visibility criteria can be based at least on the measurement resolution, field of view and / or depth of field of the one or more sensors.
[0031] In other embodiments at (320), identifying one or more sensing parameters may include calculating a curvature and / or a normal vector of at least a portion of the region of interest. Calculating the curvature and / or the normal vector may be based at least on the surface of the point cloud or image file defined in (310). The normal vector may define one or more sensor centers. For example, referring to Figure 1 and Figure 2, the normal vector may define one or more centerlines 111 of the one or more sensors 110 relative to the portion 131 of the region of interest 130. As another non-limiting example, the normal vector may define a viewing angle 112 of the sensor 110 relative to the region of interest 130 or portion 131 thereof.
[0032] At (330), method 300 includes determining a sampling combination for acquiring a plurality of samples by one or more sensors based at least in part on one or more sensing parameters. The sampling combination can be a combination of samples of the region of interest that captures the region of interest. The combination of samples of the region of interest can include a translation of the sensor and / or the region of interest relative to each other. The sampling combination can also be a combination of sensing parameters relative to the translation of the sensor and / or the region of interest. In various embodiments, the sampling combination can include a combination of samples of various portions of the region of interest that capture the region of interest. For example, referring to Figure 1 , 2, or 4, the sampling combination may include a specific sequence of translation and / or sensing parameters at various portions 131 of region of interest 130 until region of interest 130 is captured. The specific sequence of translation and / or sensing parameters may include a distance, angle, and / or resolution of one or more sensors 110 relative to region of interest 130 for each sample captured from portion 131 of region of interest 130.
[0033] Determining the sampling combination to be acquired by the one or more sensors may also include determining a minimum number of samples to be acquired to capture the region of interest. In one embodiment, determining the sampling combination to be acquired by the one or more sensors may include selecting the sampling combination based at least on a scoring function,
[0034]
[0035] For one or more sampling combinations (c0, c1, ..., c r ). The total area covered is based on at least one or more sensing parameters or portions of the region of interest. The overlap margin is the amount of a sample that is redundant with (e.g., overlaps with) a previous sample. Lambda λ is the overlap index. The overlap index is a factor that favors overlap between a sample and a previous sample. For example, λ=0 may not favor overlap and favor sampling combinations including samples with a large total area covered. However, λ=0 may result in sampling combinations in which portions of the region of interest are not captured between samples. As another example, λ>0 may favor overlap to ensure that portions of the region of interest between samples (e.g., gaps) are captured. However, for a given sampling combination, λ>0 may result in a large number of samples, or translation of the sensor or region of interest, to capture the region of interest.
[0036] In another embodiment, determining a sampling combination acquired by one or more sensors may include determining a combination of overlap indices based at least on a reinforcement learning (RL) algorithm at (332), calculating a scoring function for the one or more sampling combinations based at least on a total area covered by the one or more sensors, an overlap margin, and the one or more overlap indices at (334), and selecting a sampling combination corresponding to a maximum scoring function at (336).
[0037] At (332), method 300 may include determining a combination of overlap indices using at least one of a state-action-result-state-action (SARSA), Q-learning, and a policy gradient RL algorithm, the combination of overlap indices may output a sampling combination that minimizes the number of samples taken for the region of interest based on capturing the region of interest to a desired level of completeness. In various embodiments, at least one of the Q-learning and policy gradient RL algorithms may be used in conjunction with a deep learning method to determine a minimum number of samples for capturing the region of interest at a desired level of completeness. Determining the combination of overlap indices may include determining a combination of zero and non-zero overlap indices that may result in a maximum scoring function while capturing the region of interest in a minimum number of samples to a desired level of completeness.
[0038] At (340), method 300 includes acquiring a plurality of samples using one or more sensors based at least on the sampling combination. In one embodiment, acquiring the plurality of samples may include translating one or more sensors and / or the region of interest relative to each other. For example, referring to Figure 1 , 2 or 4, one or more sensors 110 and / or the region of interest 130 can be mounted to the robot 100 and translated to capture samples at a plurality of portions 131 of the region of interest 130 until the region of interest 130 is captured with a desired level of detail and completeness. In another embodiment, the method 300 can be implemented to determine the position, placement, setup, orientation, distance, etc. of one or more sensors relative to the region of interest using the determined sampling combination. For example, within a defined region of interest, such as a 2D or 3D space, the determined sampling combination can provide the position, placement, and orientation of the sensors such that the region of interest within the 2D or 3D space is captured to a desired level of detail and completeness using a minimum number of sensors.
[0039] Figure 4 An example apparatus 90 is depicted according to an exemplary embodiment of the present disclosure. The apparatus 90 may include one or more sensors 110, a robot 100, and one or more computing devices 120. In one embodiment, the robot 100 defines a robotic arm, such as a Figure 1 and 2 In another embodiment, for example Figure 4 As shown, the robot 100 defines an autonomous mobile vehicle, such as a drone. As described herein, one or more sensors 120, the robot 100, and / or the computing device 120 can be configured to communicate via one or more networks 410, which can include any suitable wired and / or wireless communication links for transmitting communications and / or data. For example, the network 410 can include a SATCOM network, an ACARS network, an ARINC network, a SITA network, an AVICOM network, a VHF network, an HF network, a Wi-Fi network, a WiMAX network, a gatelink network, and the like.
[0040] The computing device 120 may include one or more processors 121 and one or more memory devices 122. The one or more processors 121 may include any suitable processing device, such as a microprocessor, a microcontroller, an integrated circuit, a logic device, and / or other suitable processing device. The one or more memory devices 122 may include one or more computer-readable media, including but not limited to non-transitory computer-readable media, RAM, ROM, a hard drive, a flash drive, and / or other memory devices.
[0041] One or more memory devices 122 may store information accessible by one or more processors 121, including computer-readable instructions 123 executable by one or more processors 121. Instructions 123 may be any set of instructions that, when executed by one or more processors 121, cause one or more processors 121 to perform operations. In some embodiments, instructions 123 may be executed by one or more processors 121 to cause one or more processors 121 to perform operations, such as any of the operations and functions for which computing device 120 is configured, such as operations for sensor planning (e.g., method 300) described herein, operations for defining or receiving a region of interest, operations for identifying or receiving one or more sensing parameters of one or more sensors, operations for determining a sampling combination for acquiring a plurality of samples from one or more sensors based at least in part on the one or more sensing parameters, operations for acquiring a plurality of samples of a region of interest using one or more sensors based at least on the sampling combination, and / or any other operations or functions of one or more computing devices 120. Instructions 123 may be software written in any suitable programming language or implemented in hardware. Additionally and / or alternatively, instructions 123 may be executed in logically and / or virtually separate threads on processor 121. The memory device 122 may also store data 124 that may be accessed by the processor 121. For example, the data 124 may include one or more of samples, sampling combinations, sensing parameters, defined regions of interest, scoring functions, RL algorithms, overlap indices, and / or any other data and / or information described herein.
[0042] For example, computing device 120 may also include a network interface 125 for communicating with other components of apparatus 90 (e.g., via network 410). Network interface 125 may include any suitable components for interfacing with one or more networks, including, for example, transmitters, receivers, ports, controllers, antennas, and / or other suitable components.
[0043] The techniques discussed herein refer to computer-based systems, actions taken by computer-based systems, and information sent to and from computer-based systems. Those skilled in the art will recognize that the inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and divisions of tasks and functions between components. For example, the processes discussed herein can be implemented using a single computing device or multiple computing devices working in combination. Databases, memory, instructions, and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.
[0044] This written description uses examples to disclose the invention, including the best mode, and also to enable any person skilled in the art to practice the invention, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the invention is defined by the claims and may include other examples that occur to those skilled in the art. If such other examples include structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal language of the claims, such other examples are intended to fall within the scope of the claims.
Claims
1. A computer-implemented method for sensor planning for acquiring samples via an apparatus comprising one or more sensors, characterized in that The computer-implemented method comprises: defining, by one or more computing devices, a region of interest; identifying, by the one or more computing devices, one or more sensed parameters of the one or more sensors; determining, by the one or more computing devices, a sampling combination for acquiring a plurality of samples by the one or more sensors based at least in part on the one or more sensed parameters; and One or more command control signals are provided, via the one or more computing devices, to the apparatus including the one or more sensors to acquire the plurality of samples of the region of interest using the one or more sensors based at least on the sampling combination.
2. The computer-implemented method of claim 1, wherein: Determining the sampling combination acquired by the one or more sensors includes: determining, by one or more computing devices, a combination of overlap indices based at least on a reinforcement learning algorithm; computing, by one or more computing devices, a scoring function for one or more sampling combinations based on at least the total area covered by the one or more sensors, the overlap margin, and the one or more overlap indices; and The sampling combination corresponding to the maximum scoring function is selected by one or more computing devices.
3. The computer-implemented method of claim 2, wherein: The reinforcement learning algorithm includes using at least one of SARSA, Q-learning and policy gradient reinforcement learning algorithms.
4. The computer-implemented method of claim 2, wherein: Wherein determining the combination of overlap indices based at least on the reinforcement learning algorithm comprises determining a combination of zero and non-zero overlap indices.
5. The computer-implemented method of claim 1 , wherein: Wherein determining a sampling combination for acquiring a plurality of samples by the one or more sensors is based on at least a scoring function, wherein the scoring function is a function of at least one of a total area covered, an overlap margin, and an overlap index.
6. The computer-implemented method of claim 1 , wherein: Wherein identifying one or more sensing parameters comprises calculating a curvature and / or a normal vector of at least a portion of the region of interest.
7. The computer-implemented method of claim 1 , wherein: Wherein identifying one or more sensed parameters comprises defining a total area covered by the one or more sensors.
8. The computer-implemented method of claim 1, wherein: Wherein identifying the one or more sensing parameters includes defining one or more of a measurement resolution, a field of view, and / or a depth of field.
9. The computer-implemented method of claim 1 , wherein: Defining the region of interest includes receiving a point cloud.
10. The computer-implemented method of claim 1, wherein: Wherein defining the region of interest comprises defining a confined space in which the one or more sensors operate.