Affordance-aware, multi-resolution, free-form object manipulation planning

Abstract

The present disclosure relates to affordance-aware, multi-resolution, free-form object manipulation planning. Systems, devices, and methods may provide for controlling one or more end effectors by: generating a semantically annotated image based on image data, wherein the semantically annotated image is used to identify the shape of an object and a semantic label of the object; associating a first set of actions with the object; and generating a plan based on the intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions is associated with the command.

Description

[0001] This application is a divisional application of the invention patent application with application date of December 25, 2020, application number 202011560272.9, and name “Functional Availability Awareness, Multi-Resolution, Free-Form Object Manipulation Planning”. Technical Field

[0002] Embodiments generally relate to end effectors. Specifically, embodiments relate to control of an end effector of a robot in a dynamic environment. Background Art

[0003] Robots are able to perform tasks autonomously to accomplish certain goals. For example, a person may instruct a robot to perform a task, and the robot may then perform the task without supervision. As the complexity of the task increases and the variability of the environment increases, the robot may face increasing difficulties in performing the task with reliability and trustworthiness. In addition, dynamic environments may present difficult analysis for the robot. For example, objects may move and have irregular shapes, making interaction difficult. Summary of the invention

[0004] In one aspect, the present disclosure provides at least one memory including machine-readable instructions for causing one or more processors to perform at least the following operations: calculating a three-dimensional (3D) point cloud of an object based on input image data; calling a neural network based on the 3D point cloud to determine one or more grasps of the object associated with a planned trajectory; and causing a robot to perform an action based on the one or more grasps.

[0005] On the other hand, the present disclosure provides a device, including: a memory; instructions; and one or more processors, for operating based on the instructions to: calculate a 3D point cloud of an object based on input image data; call a neural network based on the 3D point cloud to determine one or more grasps of the object associated with a planned trajectory; and enable a robot to perform an action based on the one or more grasps.

[0006] On the other hand, the present disclosure provides a method, including: using one or more processor circuits to calculate a 3D point cloud of an object based on input image data; calling a neural network based on the 3D point cloud to determine one or more grasps of the object associated with a planned trajectory; and causing a robot to perform an action based on the one or more grasps. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Various advantages of the embodiments will become apparent to those skilled in the art upon reading the following description and appended claims, and by referring to the following drawings, in which:

[0008] Figure 1is a process flow chart of an example of an environmental analysis and robot end effector control process according to an embodiment;

[0009] Figure 2 is a flowchart of an example of a method of controlling an end effector according to an embodiment;

[0010] Figure 3A and 3B is a process flow diagram of an example of a scene segmentation process according to an embodiment;

[0011] Figure 4 is a process flow diagram of an example of modeling a kinematic end effector according to an embodiment;

[0012] Figure 5 is a process flow chart of an example of a planning task according to an embodiment;

[0013] Figure 6 is a flow chart of an example of a method of analyzing and representing an environment according to an embodiment;

[0014] Figure 7 is a flow chart of an example of a method of classifying an end effector according to an embodiment;

[0015] Figure 8 is a flowchart of an example of a task planning method according to an embodiment;

[0016] Fig. 9 is a diagram of an example of an octree according to an embodiment;

[0017] Fig.10 is an illustration of an example of a structured point cloud according to an embodiment;

[0018] Fig.11 is an illustration of an example of a semantically segmented and annotated image according to an embodiment;

[0019] Fig.12 is a diagram of an example of semantically segmenting an image by parts according to an embodiment;

[0020] Fig.13 is a process flow diagram of an example of a surface and occupancy subsampling process according to an embodiment;

[0021] Fig.14 is a flowchart of an example of a method for training a neural network according to an embodiment;

[0022] Fig.15 is a block diagram of an example of a performance-enhanced computing system according to an embodiment;

[0023] Fig.16 is a diagram of an example of a semiconductor device according to an embodiment;

[0024] Fig.17 is a block diagram of an example of a processor according to an embodiment; and

[0025] Fig.18 is a block diagram of an example of a multi-processor based computing system according to an embodiment. DETAILED DESCRIPTION

[0026] Now go to Figure 1 , an enhanced environmental analysis and robotic end effector control process 100 is shown. The robot may include a sensor array 102, a graph and semantic generator 104, a robot modeler 106, a task planner 108, an end effector controller 128, and a robotic end effector 132 (e.g., a multi-fingered robotic end effector). The process 100 may be an overall perception planning solution to enable the robot to grasp free-form objects in a reliable, flexible, efficient, and affordance-compelling manner (e.g., based on specific characteristics of objects with variable sizes, objects with different material properties, objects with different surface properties, etc.). Affordance-compelling may include generating robotic grasping configurations with 6D poses that are consistent with expected action flows and partially mimic social and cultural behaviors implicit during job execution. The robot may operate in a semi-structured human-centric space (e.g., home, health, retail, etc.) to complete a job. For example, the robot may grasp elements that are in free form in a human-centric space. In detail, the task planner 108 can efficiently, quickly, and adaptively determine an appropriate grasping configuration for a given robotic manipulator to grasp (e.g., physically manipulate) a free-form object. The free-form object can be previously unseen such that the robot has no prior interaction with the object (e.g., the object is "new" to the robot).

[0027] The sensor array 102 may include imaging sensors (e.g., 2D cameras, 3D depth cameras, and 6D inertial measurement units), auditory sensors, distance sensors, position sensors, etc. The sensor array 102 may provide data to the graph and semantics generator 104, the robot modeler 106, and the mission planner 108. For example, the sensor array 102 may provide image data (e.g., red, green, blue, and depth (RGB-D) image data, 3D camera orientation, 3D point cloud, etc.) and / or distance data 110 to the graph and semantics generator 104.

[0028] The map and semantics generator 104 may generate one or more maps based on the image and / or distance data. For example, the map and semantics generator 104 may generate an occupancy map to represent the robot's environment, such as an occupancy map (continuous or discrete) that plots occupied spaces. In some embodiments, the map and semantics generator 104 may further plot unoccupied spaces and / or unknown spaces (space that cannot be identified as occupied or unoccupied) and store the unoccupied spaces and / or unknown spaces in an occupancy map or another map.

[0029] The graph and semantics generator 104 can also generate a surface graph that identifies surfaces based on the sensor data and the occupancy map (e.g., classifies the occupied space into various surfaces). For example, the surface graph can be a structured point cloud that includes a collection of 3D vertices on the surface linked by edges.

[0030] The graph and semantics generator 104 may also generate a semantically annotated graph based on the surface graph and the sensor data (e.g., connecting labels to surfaces in the surface graph). For example, the graph and semantics generator 104 may include a deep neural network that recognizes each object in the surface graph, identifies the boundaries of the objects, applies labels (e.g., cup, cube, bottle, table, etc.) to the objects (e.g., surface fragments), and assigns unique values ​​(e.g., instance identifiers) to the objects for future reference.

[0031] The graph and semantics generator 104 can also generate a partially annotated semantic graph based on the semantically annotated graph and sensor data (e.g., generating surface areas that are assigned semantics, which can be referred to as semantic patches or surface patches). For example, the partially annotated semantic graph can identify the parts of each object. As a more detailed example, if a motorcycle is identified, the parts can include handlebars, frame, seat, fuel tank, and wheels. Each part can be labeled in the partially annotated semantic graph.

[0032] In some embodiments, if parts of the object are smaller than a predetermined size (e.g., smaller than the contact area of ​​the end effector), the graph and semantics generator 104 can omit these parts from further analysis. In some embodiments, the resolution of the surface patches can be reduced to more efficiently store and access the corresponding identification data (e.g., the corresponding image data can be stored as an octree), and further based on the annotated parts, actions that can be taken (e.g., physical manipulation of the object) are associated with each recognized object (e.g., move the cup, refill the cup, clean the cup, etc.). Thus, the graph and semantics generator 104 can link verbs (e.g., actions) and noun names in the segmented surfaces and / or parts.

[0033] The robot modeler 106 may receive the position data 112 from the sensor array 102 . However, in some embodiments, the robot modeler 106 may perform the following functions without the position data 112 .

[0034] The robot modeler 106 may identify the current position of the robot, for example, based on the position data 112. The robot modeler 106 may generate a model of the static and dynamic geometry (e.g., kinematics) of the robot to enable the task planner 108 to plan motion. For example, the robot modeler 106 may define the robot actuator as a collection of links (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler 106 may further generate a grasping capability map. The grasping capability map may be a discretization of the workspace in which the robot can apply contact with a minimum nominal force of the robot end effector 132. The robot modeler 106 may further penalize grasps based on force and kinematic feasibility, and quickly reject inappropriate grasps. The robot modeler 106 may also rank actions based on feasibility and force to identify actions with the highest probability of success based on a particular metric (e.g., actions with the highest probability of success given a particular object in a graph and / or image and a particular available space around the object).

[0035] In some embodiments, the position data 112 may provide sensor information that the robot modeler 106 uses in conjunction with direct and inverse kinematics to pre-compute a 3D body-dependent reachability map. Using the reachability map and the current state of the scene, the robot modeler 106 may identify a set of possible actions (e.g., actions that physically manipulate an object). In some embodiments, the map and semantics generator 104 may provide the scene to the robot modeler 106.

[0036] In some embodiments, the robot modeler 106 may receive one or more graphs of the graph and semantic generator 104 and determine appropriate grips for various objects based on the grip capability graphs. For example, the robot modeler 106 may identify that certain grips will be invalid for an object (e.g., will not be able to manipulate the object, will not be able to hold a slide object, etc.), and therefore exclude such grips from being transmitted to the task planner 108 as ranked actions.

[0037] The graph and semantics generator 104 may provide the graph and semantics data 118 to the mission planner 108, and the robot modeler 106 may also provide the ranked actions 120 to the mission planner 108. The mission planner 108 may receive command data 114 from the sensor array 102. The command data may take the form of various sensor data, such as audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends a command to a second device directly or through a cloud).

[0038] The task planner 108 can identify a job from a high-level instruction (e.g., clean the kitchen). For example, the high-level instruction can be broken down into a series of granular atomic actions that can be referred to as a macroplan. The macroplan can not only provide the actions taken to complete the job (e.g., physical actions to physically manipulate an object), but can also provide operational constraints corresponding to a specific domain and job.

[0039] For example, the mission planner 108 may set a maximum speed for the robot end effector 132 along a manipulation trajectory (e.g., for social space sharing) or maintain a container with orientation constraints to avoid failure of the mission (e.g., liquid spillage in the container). The case-by-case operational constraints may change at each step of the planning. Thus, identifying operational constraints may filter affordances by attributes and prioritize affordances by range matching.

[0040] In some embodiments, the task planner 108 may receive one or more graphs of the graph and semantics generator 104 and / or identification of ranked actions from the robot modeler 106. The task planner 108 may determine a resulting symbolic plan having an active subset of attributes and actions based on the one or more graphs of the graph and semantics generator 104. The active subset of actions may be an intersection of actions from the one or more graphs of the graph and semantics generator 104 (e.g., verbs linked by noun names in segmented portions) with actions from the ranked actions from the robot modeler 106 that implement the plan and actions (e.g., granular atomic actions) identified from the command data 114.

[0041] The mission planner 108 may provide the decomposed commands and plans 126 to an end effector controller 128 (e.g., a processor on an end effector that controls actions). In addition, the robot modeler 106 may provide ranked actions to the end effector controller 128, and the graph and semantics generator 104 may provide graph and semantic data to the end effector controller 128. The end effector controller 128 controls the robot end effectors 130, 132 to implement the decomposed commands and plans, which include actions (e.g., intersections of actions) identified by the mission planner 108. The sensor array 102 may also provide sensor data 134 to the end effector controller 128, so that the end effector controller 128 may control the end effector 130 based on updated sensor data (e.g., position data).

[0042] Some embodiments may rely on learned kinematic specific grasping functions. Some embodiments may include a multi-resolution basis that allows for the integration of semantic and linguistic knowledge cues while extracting appropriate grasping configurations based on: i) spatial context (environment occupancy and reconstructed object surface), ii) semantic instance segmentation by parts, and iii) kinematic description of the end effector. Inputs may be merged via multiple processes orchestrated for reconstruction, filtering, and inference jointly driven by high-level job-specific instructions.

[0043] In some embodiments, process 100 can implement a spatial programming paradigm that delivers enhanced parallel computing capabilities through hardware accelerators such as FPGAs. For example, any element such as graph and semantic generator 104, robot modeler 106, task planner 108, sensor array 102, and end effector controller 128 can be constructed in the form of a hardware accelerator. In addition, the low energy consumption required by such computing devices can improve power efficiency, which may be beneficial when deploying battery-powered mobile robots. In summary, enhanced multi-resolution analysis at any time, language-based functional affordances, and the advantageous use of low-power parallel devices can produce reliable components for sensor-driven grasping planning. The above process 100 can enable an autonomous service robot to perform real-world physical interaction tasks that generate and capture values ​​in a semi-structured environment.

[0044] Figure 2 A method 800 for controlling an end effector is shown. The method 800 may generally be implemented in a robotic process, such as the process 100 already discussed ( Figure 1). In an embodiment, method 800 is implemented in one or more modules as a set of logic instructions stored in a machine or computer readable storage medium such as a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), firmware, flash memory, etc.; in configurable logic such as a programmable logic array (PLA), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), etc.; in fixed function logic hardware using circuit technology such as an application specific integrated circuit (ASIC), complementary metal oxide semiconductor (CMOS), or transistor-transistor logic (TTL) technology; or in any combination thereof.

[0045] For example, the computer program code for performing the operations shown in method 800 may be written in any combination of one or more programming languages, including object-oriented programming languages, such as JAVA, SMALLTALK, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. In addition, the logic instructions may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for integrated circuits, state information that personalizes hardware-local electronic circuits and / or other structural components (e.g., main processors, central processing units / CPUs, microcontrollers, etc.).

[0046] The illustrated processing block 802 generates a semantically annotated image based on the image data, wherein the semantically annotated image is used to identify the shape of the object and the semantic label of the object. The illustrated processing block 804 associates a first set of actions with the object. The illustrated processing block 806 generates a plan based on the intersection of the first set of actions and the second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions will be associated with the command.

[0047] In some embodiments, method 800 may include applying a first label to a first portion of the object, and applying a second label to a second portion of the object, wherein the second label is different from the first label. In some embodiments, method 800 may also include generating a surface patch from the semantically annotated image, the surface patch representing the object, reducing a resolution of the surface patch, and generating a plan based on the surface patch having the reduced resolution.

[0048] In some embodiments, method 800 may further include identifying a contact force map representing contact force outputs mapped to locations of one or more end effectors, and generating a plan based on the contact force map. In some embodiments, method 800 may further include generating an occupancy map based on the image data, wherein the occupancy map is used to identify locations of the image to be occupied, and generating a surface map based on the occupancy map, the surface map being used to identify surfaces of the object. In some embodiments, method 800 may further include connecting surfaces in the surface map to tags to generate a semantically annotated image.

[0049] The method 800 can enable a robot to grasp free-form objects in a reliable, flexible, efficient, and affordance-rich manner (e.g., based on specific characteristics of objects with variable sizes) so that the robot can operate in dynamic and rapidly changing environments. The method 800 can enable a robot to operate in a semi-structured human-centric space (e.g., home, health, retail, etc.) to complete a job. In detail, the method 800 can efficiently, quickly, and adaptively determine an appropriate grasping configuration for a given robotic manipulator to grasp a free-form object. The free-form object can be previously unseen, so that the robot has no previous interaction with the object (e.g., the object is "new" to the robot).

[0050] Figure 3A and 3B A scene segmentation process 300 is shown. The process 300 may include a scene semantic space context generator 304. The scene semantic space context generator 304 may be easily substituted for Figure 1 Graph and semantic generator 104 .

[0051] The imaging and / or distance sensor 302 can provide sensor data 336 to the scene semantic space context generator 304. The sensor data can include imaging data (e.g., RGB-D data) and / or distance data. The imaging sensor of the imaging and / or distance sensor 302 can be a device contained in a combined sensor (e.g., an RGB-D camera or camera module). For example, the imaging and / or distance sensor 302 can provide three data streams that capture information about the content in the field of view and the time-varying 6D pose of one or more objects.

[0052] For example, the field of view can be captured in the form of a structured point cloud, either by a discrete depth image or another image from which depth can be derived (e.g., a 3D projection calibrated via an intrinsic sensor). The image can be a collection of 3D points (e.g., a noisy front wave composite of the scene surface) similar to an infrared-enhanced (e.g., associated infrared, reflectance intensity, or absorption values) pattern.

[0053] The imaging and / or distance sensor 302 can also provide a digital RGB color image of the scene with high resolution. Such a passive camera captures the colored appearance of objects with fine details. Such an image may lack clear depth information. Therefore, based on the external calibration between the depth camera and the color camera, the scene semantic space context generator 304 can map the 3D points from the structured point cloud to the RGB image plane to associate the color value of each point. Some embodiments can also associate depth with most pixels of the digital image. In some embodiments, the imaging and / or distance sensor 302 may include LIDAR or other 3D distance sensors.

[0054] The imaging and / or distance sensor 302 may also include a built-in multi-axis inertial measurement unit (IMU), which may provide IMU measurements in the sensor data 336. Thus, the scene semantics spatial context generator 304 may derive fused state assertions from the measured acceleration, linear velocity, and angular velocity of the imaging and / or distance sensor 302 (e.g., a camera module).

[0055] Therefore, the scene semantic space context generator 304 can generate a 6D kinematic frame based on the fused assertions. The 6D kinematic frame can be 6D due to the 3 position degrees of freedom and the 3 orientation degrees of freedom. The set of 6 degrees of freedom can clearly define the pose in space. In some embodiments, the motion can also be related to the velocity in each dimension, i.e., V x 、V y 、V z and V 滚转 、V 俯仰 and V 偏航 , these speeds will correspond to another set of 6 degrees of freedom. The degrees of freedom of the robot end effector 132 may not be limited to 6 degrees of freedom, but may depend on the robot structure, motors, and joint types associated with the robot end effector 132. In some embodiments, the robot may need to have at least 6 degrees of freedom to grasp objects in general locations.

[0056] The 6D kinematic framework may include additional features such as gravity orientation. In doing so, the relative orientation of the floor may be discerned based on the constant gravity acceleration and may be used to provide context to surface and object dynamics. The collection of this signal may be an exteroceptive sensor input, which is processed by the following geometric, semantic, and fusion processes as described below.

[0057] The scene semantics spatial context generator 304 may include an vacancy and occupancy map generator 316 that generates vacancy and occupancy spatial maps 306, 308, 324. The vacancy and occupancy map generator 316 may generate spatial maps and surface reconstructions on the spatial maps.

[0058] The free and occupied map generator 316 can generate a sparse bi-space map that can capture and separate occupied and unfilled (free) space. This map generation can allow: i) to register various 3D images while exploring various interaction (e.g., grasping) scenarios for kinematic end effectors, ii) to determine possible collision-free manipulator 6D poses in the environment, and iii) to serve as an efficient scaffolding data structure to store multi-resolution local surface descriptors, such as volumetric (e.g., relative to voxels) semantic labels and other attributes.

[0059] The sparse bi-space map is a high-resolution octree of the working space, which can further allow a clear separation of graspable space from non-graspable space, and specifically classifies space into three categories: 1) unseen space, 2) occupied space, and 3) empty space. The octree can be considered as fully observed (e.g., so external voxels can be marked as occupied) or empty space (e.g., large internal voxels are marked as empty), respectively. The initial map that can be generated based on the sensor data can be formally expressed as a continuous occupancy mapping function Γ, whose logarithmic evaluation complexity is

[0060]

[0061] In addition, for each point The idle and occupied map generator 316 may determine the corresponding voxel θ of the point via a discretization function i Containers and implicit graph paths containing subspaces and / or subgraphs. The discretization function can be a voxelization process that can be performed while generating an octree. The voxelization process can include sequential insertion into a spatial partitioning data structure. The result can be a regular tree with an order corresponding to the subdivision of a spatial octant in 3D space (e.g., 8 degrees). For example, the following Equation 2 can represent β that can be used as an octree:

[0062]

[0063] In Formula 2, θ 0 The root voxel at may represent the entire captured scene subspace. Depending on the implementation, both external and internal nodes of the octree may contain different attributes. Attributes may be expressed by "access functions", as indicated below in Equation 3:

[0064]

[0065] Equation 3 may be a value-attribute mapping on the custom attribute set Ω. The attribute set and / or Equation 3 may be used during fusion and scaffolding of free space map 306 and occupied space map 308.

[0066] The free space and occupancy map generator 316 may extract a single-category version of the initial map to generate a free space map 306 and an occupied space map 308. The occupied space map 308 may be a set of points that satisfy the constraint Γ(x)=1.

[0067]

[0068] Therefore, based on Equation 4, the union of occupied voxels can produce an occupancy map 308, which can be based on the following

[0069] Equation 5 is used to determine the occupied space map 308:

[0070]

[0071] In addition, the idle and occupied map generator 316 may invert the occupied space map 308 or modify one or more of equations 1-5 (e.g., replace the occupied space Γ(x i )=1 is modified to empty Γ(x i )=0) to generate the free space map 306. For example, equation 3 and / or equation 5 may be modified so that the point-by-point empty space map N is defined by equation 4: p or a discrete version of the blank voxel map, thereby generating N stored as free space map 306. v Thus, free space map 306 may be an inverted octree scene relative to occupied space map 308 .

[0072] The idle and occupancy map generator 316 can provide the idle and occupancy maps 306, 308, 326 to the surface reconstruction and representation generator 318. In some embodiments, only the occupancy map 308 is provided to the surface reconstruction and representation generator 318, and / or can be filtered based on criteria. The surface reconstruction and representation generator 318 can also identify image data (e.g., RGB data) from the sensor data 336.

[0073] The structured point cloud (which may be the same as the point cloud described above and may be based on a 2D RGB image) may include a collection of 3D vertices. The 3D vertices may be provided by:

[0074]

[0075] Vertices can be linked by edges in the 8-connectivity class pixel function defined by:

[0076]

[0077]

[0013] In an enhancement, generating graphical structures on point clouds may be the ability to extract a set of implicit simplexes that define surface primitives (eg, 3D triangles).

[0078] In some embodiments, due to sensing limitations, the depth image may omit values ​​that may produce undesirable surface holes on the graphical structure (e.g., the graphical structure has holes instead of vertices on the surface of the object). Some embodiments may also include additional metrics when extracting the underlying primitives. For example, if the above equation 7 satisfies L(x i ,x j )=1, where the length Its upper limit is a function of the depth d from the camera to the vertex m ~∈·max(x i ,x j ), then there can be an edge E i,j The connectivity criterion can be shown in various ways, such as a valid edge with a first characteristic (e.g., yellow) or an invalid edge with a second characteristic (e.g., red) in the structured point cloud. i,j,k For each vertex of , the surface reconstruction and representation generator 318 may determine the associated normal vector according to the following formula:

[0079]

[0080] Equation 8 may be based on a normalized set of connected triangle normals. Equation 8 may be a way to obtain the normal vector for each point on the surface. Doing so may help plan contact points and margin distances along the approximation of the robot end effector 132. For example, Equation 8 may indicate that T{i,j,k} refers to the triangle passing through surface points i, j, and k, so for any point, the normal may be determined by averaging the normals of the attached triangles.

[0081] In some embodiments, the surface map 310 of the scene, represented by φ in the following equation, is the union of all primitives in the scene:

[0082]

[0083] Additionally, connected subsets of triangles can define surface components The extraction of this surface component simultaneously links each triangle to its occupancy discrete map M from Eq. 5 v The subjective triangle to voxel attribute function can also allow the subsequent fusion and semantic transfer process. The triangle to voxel attribute function can be provided as follows:

[0084] λ(T i,j,k ∈φ)→β(x i )∪β(x j )∪β(x k )

[0085] Formula 10

[0086] Additionally, the surface map 310 may include attributes from the subjective triangle to voxel attribute function.The surface reconstruction and representation generator 318 may thus generate the surface maps 310, 328 and store the surface map 310.

[0087] The surface reconstruction and representation generator 318 may provide the surface maps 310, 338 to the semantic object segmentor 320. The semantic object segmentor 320 may identify objects, segment objects, and label objects as described below.

[0088] The semantic object segmenter 320 can identify RGB images from the sensor data 336 The semantic object segmenter 320 may utilize pixel-wise semantic instance segmentation with a deep neural network DNN (e.g., represented as a single function ) to generate instance-recognized, semantically annotated images

[0089] In a semantically annotated image, each pixel can contain: i) The probability distribution on the set have predefined prototype classes, and ii) corresponding instance identifiers (denoted as "id", which can be non-zero unique integers), where the network distinctively associates pixels to instances. For example, a labeled dataset of semantically annotated images may include the following annotated dataset Σ: = [cup, ε 0 ,h i ],[bottle,ε 1 ,h j ],[plate,ε 2 ,h k ],[table,ε 3 ,h l ],…}. In some embodiments, a special “unknown” class may also be included for all unclassified pixels, such as [unknown, ε n ,0]]. The data set can also obey the following formula:

[0090] and 0≤ε i ≤1

[0091] Formula 11

[0092] In the dataset, h i represents a numerical identifier associating each pixel with an object instance, or “0” for those pixels in the unknown class. The total extraction process α over all pixel locations can be given by

[0093] Formula 12 provides:

[0094]

[0095] The semantic object segmenter 320 can also integrate the derived semantic information, labels, confidences, and instance IDs through consistency in 3D occupancy while connecting labels to surface segments as follows. For example, while simultaneously utilizing mutual external calibration between imaging and / or distance sensors 302 (e.g., depth cameras and color cameras), the semantic object segmenter 320 can project spatial points into a camera image plane (e.g., an RGB camera plane). Based on this mapping, a 2D semantic label can be transferred to each vertex of the surface. A semantic surface map with labels transferred to each vertex can be stored as a semantic surface map 332.

[0096] In some embodiments, the DNN may generate the results described in Equation 12. To mitigate errors or differences at the boundaries of objects represented in the image, each point may be connected to a surface component, as determined from Equation 9, and provided as follows:

[0097]

[0098] We can only target high confidence classification probabilities ε i Using the object id (e.g., h in Eq. 12) i ) segmentation and merging process is performed by conditional region growing. In doing so, the 3D surface connected components can be transformed (e.g., one or more of erosion and segmentation) into short semantically annotated surface regions or patches.

[0099] In some embodiments, extracted semantic patches of an object (e.g., surfaces annotated with a semantic label, such as the back of a chair, the armrests of a chair, the seat of a chair, the legs of a chair, etc.) that each have a size smaller than the contact surface of the robotic manipulator can be excluded from further consideration or interaction with the robotic manipulator. Furthermore, such excluded semantic patches can be excluded from subsequent stages without concern for conflicts during motion due to the latest occupancy map. At this point, the surface segments The content of the semantic surface graph 312 may be described.

[0100] The semantic object segmenter 320 may further perform partial classification on each semantic patch associated with the object instance. For example, the semantic object segmenter 320 may perform partial classification r refinement processing, where Describes the set of possible part names derived on object categories. The part classification can be stored in a semantic surface map 312. The semantic object segmenter 320 can provide the semantic surface map 312, 340 to the perceptual part generator 322, which generates the perceptual grasp components 314, 334.

[0101] The perceptual part generator 322 may also identify 2D RGB image data from the sensor data 336. The perceptual part generator 322 may generate perceptual grasp components (“PGCs”). Each PGC may be composed into a 4-tuple having i) a semantic surface patch ii) The class of the object to which the patch belongs iii) Partial name associated with the patch and iv) the set of verbs that can be defined by:

[0102]

[0103] Describes the associated likelihood σ of possible actions applied to such object parts i It can be provided by:

[0104]

[0105] The associated set of actions can be extracted from object-action co-occurrence The above object-to-action co-occurrence comes from the frequency analysis of a large text corpus describing application processes structurally organized by domain ontology. Equation 15 can define the 4-tuple of PGC.

[0106] Furthermore, these PGCs can be transformed according to the geometric resolution (e.g., decimated so that the boundaries of objects have lower resolution). Figure 3B , for each tuple τ of Eq. 15, the surface patch Subsampling (e.g., sequentially decimating by half) is performed to reduce the resolution of the corresponding image. In some embodiments, decimation may occur at least five times so that the corresponding resolution is reduced by at least five levels in a manner such that the remaining signal content at each level is 2^(-L) 0≤L≤4, where L refers to the detail level. In doing so, the occupancy alignment of the signal to the geometry in the octree diagram may be maintained (M from Equation 5) v ). The following formulas may represent various degraded tuples that may be stored in PGC 314:

[0107]

[0108] Function l can be in the occupancy map attribute M v The dual-purpose operation of L subsampling and surface registration in .

[0109] For example, the original ultra-high PGC 314a may be reduced. The process 300 may reduce the resolution 342 of the ultra-high PGC 314a to generate a high-resolution PGC 314b. The process 300 may also reduce the resolution 344 of the high-resolution PGC 314b to generate a medium-resolution PGC 314c. The process 300 may also reduce the resolution 346 of the medium-resolution PGC 314c to generate a low-resolution PGC 314d. The process 300 may also reduce the resolution 348 of the low-resolution PGC 314d to generate an ultra-low-resolution PGC 314e. The PGC 314e may be stored as the PGC 314.

[0110] PGC 314 may be provided by, for example, Figure 1 The low-resolution PGC 314e may be accessed by a task planner such as the task planner 108 of the task planner to generate a plan for the end effector control used to manipulate the object. In some embodiments, the low-resolution PGC 314e may be accessed by the task planner to generate a plan.

[0111] Figure 4 A process 350 for modeling a kinematic end effector is shown. For example, a robot modeler 354 may be connected to a semantic surface graph 358 and a PGC 360, as described herein. The robot modeler 354 may include a volume description 356 and end effector kinematics 362. The robot modeler 354 may easily replace Figure 1 The robot modeler 106 .

[0112] The volume description 356 and the end effector kinematics 362 may be at least partially pre-programmed. The end effector kinematics 362 may represent the robot actuator as a collection of links and joints (e.g., axis and joint range limits, torque capability graphs, link and joint trees, and axis limits). Such end effector kinematics 362 may facilitate planning motion while avoiding undesired collisions. The end effector kinematics 362 may represent the forward kinematics ξ and inverse kinematics ξ of the actuator. -1 is represented as a mapping expressed by:

[0113]

[0114] In Eq. 17, the n motorized degrees of freedom for the rigid frame can be expressed as This notation allows taking into account the non-bijective nature of the transformation by adding the solution-free configuration “\0” in Eq. 17 on both the domain and the codomain.

[0115] Volume description 356 may include the volume of a segment of a kinematic end effector. Different segments may include different volumes.

[0116] The robot modeler 354 can generate a grasping capability map 368. The grasping capability map 368 can be a discretization of the workspace where the kinematic end effector can apply contact with a minimum nominal force. The grasping capability map 368 can be a property container voxel map based on the semantic surface map 358 and / or the PGC 360. For example, the grasping capability map 368 contains not only occupancy values, but also a range of strengths and orientations of forces that can be achieved at that location. For example, each grasping capability map 368 can be a color map reflecting a cross-section that encodes the dexterity of each voxel (e.g., blue represents maximum grasping capability). The discrete function or grasping index can be represented by the following formula:

[0117]

[0118] The robot modeler 354 may penalize grasps based on volume feasibility, force feasibility, and motion feasibility. For example, grasps that exceed contact force as identified from the grasp capability map 368 may be rejected to avoid damaging the object.

[0119] The robot modeler 354 can thus effectively and quickly reject inappropriate grasps while providing mathematical analysis to drive gradient estimation for optimal grasps during, for example, neural network training. The robot modeler 354 can further select grasps from the grasp capability map 368 and generate grasp actions and rankings 366 based on forward and inverse kinematics and grasp indices. Thus, the robot modeler 354 can generate executable grasp models and rank the models based on metrics 364.

[0120] Figure 5 A process 380 for planning a mission is shown. The process 380 may include a mission planner 390. The mission planner 390 may be easily substituted for Figure 1 The mission planner 390 may be connected to the semantic surface graph 384, the executable grasping model and ranking based on the metric 382, ​​the sensor array 386, and the PGC 398.

[0121] Task planner 390 may capture and expand high-level instructions (e.g., "clean the kitchen") from sensor data provided by sensor array 386. Task planner 390 may decompose instructions into a fine-grained sequence of physical atomic actions or jobs (e.g., main jobs, auxiliary jobs, target object partial assertions, function affordance lists, etc.) to complete the high-level instructions. The jobs may be stored in job information 392.

[0122] Additionally, actions can be thought of as macroplans that provide operational constraints 394 that correspond to specific domains and job and action goals. Task planner 390 can set a maximum velocity for the end effector along a manipulation trajectory (for social space sharing) or hold a container with orientation constraints (e.g., smoothness or responsiveness) to avoid spilling liquid. Contextual operational constraints can be changed at each step of the plan and can be stored in operational constraints 388. Thus, providing this information in the grasp plan enables filtering affordances by attributes and prioritizing affordances through range matching.

[0123] The resulting symbolic plan with attributes and valid subset of actions may be stored as part of the online composition and description of the atomic job 396. The valid subset of actions may be the resulting intersection of actions derivable from the PGC 398 (verbs linked by noun names in segment parts or patches) and verbs from the decomposition of the high-level instructions when creating the atomic job.

[0124] Furthermore, the language-based formal representation of a contextualized atomic task can be an action tuple consisting of five elements: i) a location label describing the general context or location in which the action is taking place; c , such as "kitchen", "living room", etc., ii) object label describing the target object ι o , such as “mug”, “pillow”, etc., iii) partial labels describing the region derived from the target object ι p , i.e. "handle", "armrest", "bracket", iv) verb segmentation labels describing segmented actions ι s , which may contain one or more atomic actions, such as "arrange", "classify", "arrange", etc., and finally v) an atomic verb label representing an indivisible operation ι a , such as “pick”, “slide”, “push”, “pull”, etc. For learning and inference purposes, these labels are embedded into the vector In the representation, the vector The representation enables mathematical vector operations that exhibit semantic properties such as implicit inference through vector subtraction or addition. The following expression can represent the above:

[0125]

[0126] The dimensionality (e.g., 5g) can depend on the specific embedding (e.g., bag of words or skip-gram), or one that is more compact when using the most recent independent unsupervised methods that do not require text windows.

[0127] Figure 6 A method 400 for analyzing and representing an environment is shown. The method 400 may generally be implemented in a robotic process, such as the process 100 already discussed ( Figure 1 ) and / or process 300( Figure 3A and 3B ). Method 400 may be implemented in one or more modules as a set of logic instructions stored in a machine or computer readable storage medium such as RAM, ROM, PROM, firmware, flash memory; in configurable logic such as PLA, FPGA, CPLD; in fixed function logic hardware using circuit technology such as ASIC, CMOS or TTL technology; or in any combination thereof.

[0128] Processing block 402 may generate one or more of an occupied space map, an unoccupied space map, and an unknown space map. Processing block 404 may generate a surface map based on one or more of the occupied space map, the unoccupied space map, and the unknown space map. Processing block 406 may annotate objects in the surface map. Processing block 408 may annotate portions of the objects in the surface map. Method 400 may also include performing PGC processing on the surface map.

[0129] Figure 7 A method 420 for classifying an end effector is shown. The method 420 may generally be implemented in a robotic process, such as the process 100 ( Figure 1 ) and / or process 350( Figure 4 ). Method 420 may be implemented in one or more modules as a set of logic instructions stored in a machine or computer readable storage medium such as RAM, ROM, PROM, firmware, flash memory; in configurable logic such as PLA, FPGA, CPLD; in fixed function logic hardware using circuit technology such as ASIC, CMOS or TTL technology; or in any combination thereof.

[0130] Illustrated processing block 422 identifies characteristics of the end effector. Illustrated processing block 424 identifies capabilities of the end effector. Illustrated processing block 426 determines forward and inverse kinematics associated with the end effector. Illustrated processing block 428 generates an end effector index (e.g., a grasping capability map). Illustrated processing block 430 ranks the actions.

[0131] Figure 8 A method 440 for task planning is shown. The method 440 may generally be implemented in a robotic process, such as the process 100 already discussed ( Figure 1 ) and / or process 380( Figure 5). Method 440 may be implemented in one or more modules as a set of logic instructions stored in a machine or computer readable storage medium such as RAM, ROM, PROM, firmware, flash memory; in configurable logic such as PLA, FPGA, CPLD; in fixed function logic hardware using circuit technology such as ASIC, CMOS or TTL technology; or in any combination thereof.

[0132] Illustrated processing block 442 identifies the operational constraints of the task. Illustrated processing block 444 identifies the job information. Illustrated processing block 446 determines the operational boundaries. Illustrated processing block 448 determines the objectives of the task. Illustrated processing block 450 generates a plan based on the information provided in the above identified processing blocks.

[0133] Fig. 9 An octree 480 is shown as a spatial map that separates occupied space from empty space based on a multi-scan point cloud. As shown, the octree 480 represents a ladder. The octree 480 can be an occupancy map.

[0134] Fig.10 A structured point cloud with vertices and edges is shown. As shown, a surface 470 may be tessellated with primitives, which are triangles 472. Valid triangles 472 are shown with a first characteristic (e.g., solid lines). In contrast, triangles 474 may be considered invalid primitives (based on criteria as described herein) and are shown as dashed lines. Invalid triangles 474 may be discarded for future processing.

[0135] Fig.11 A semantically segmented and annotated image 498 is shown. As shown, the first object may be annotated as a phone 486, while the remaining objects may be annotated as cubes 482, 484, 496. The labels may be generated as described herein. The labels may be stored in a tuple for each object.

[0136] Fig.12 A semantically segmented image 500 of a chair by parts is shown. The various parts of the chair can be labeled. For example, the backrest 488, armrests 490, seat 492, and base 494 can be identified and labeled. The labels can be stored in a tuple of the chair.

[0137] Fig.13 A surface and occupancy subsampling process 510 is shown that reduces the resolution of a high quality image and generates a PGC. The process 510 may reduce the resolution 512, 514 to exponentially decay the level of detail (eg, decimation), thereby allowing efficient summarization while preserving conflict cues.

[0138] Fig.14A training method 550 for training a DNN for any of the embodiments described herein is shown. The method 550 may generally be implemented in a robotic process, such as the process 100 ( Figure 1 ), Method 800( Figure 2 ), process 300( Figure 3A-3B ), process 350( Figure 4 ), process 380( Figure 5 )、Method 400( Figure 6 )、Method 420( Figure 7 )、Method 440( Figure 8 ). Method 440 may be implemented in one or more modules as a set of logic instructions stored in a machine or computer readable storage medium such as RAM, ROM, PROM, firmware, flash memory; in configurable logic such as PLA, FPGA, CPLD; in fixed function logic hardware using circuit technology such as ASIC, CMOS or TTL technology; or in any combination thereof.

[0139] Illustrated processing block 552 identifies the resolution (e.g., five) at which to train the DNN. Illustrated processing block 554 may train the DNN at the identified resolution.

[0140] To train each of these networks by supervised corrective learning (e.g., stochastic gradient descent), the utilization of a simulation engine can enable the regeneration of sensor signals comparable to those from real sensors (equivalent to the effects of noise). Using these virtual signals from the 3D scene (e.g., a watertight model with semantic and functional knowledge), the execution of the sensing pipeline can extract view-dependent geometric primitives while obtaining semantic and part labels directly from the model. This allows for the efficient generation of quasi-realistic annotated perception datasets.

[0141] Furthermore, utilizing discrete closed form (DCF) methods for model-based grasping with fine-grained sampling, it is possible to obtain a large set of valid grasp configurations that may only lack semantic understanding. Therefore, the process of associating semantic parts (such as PGCs) with DCF grasp configurations that have sufficient (with respect to dynamic stability) contact points in a single annotated part can define small batches during training. Furthermore, by growing this association via language-based affordances in the form of vectors generated in the language embedding, it is then possible to formulate the training of each of the neural networks at their native resolution using all geometric, semantic part, affordance cues. The cost function used as the optimization objective during training can be expressed as:

[0142]

[0143] In Equation 20, “O” represents the object in the training set, and the alignment cost function Map grasps T with minimum distance to the set of grasps generated by training i and T j The increment in rotation, translation, and joint configuration between , has . The grasping ability can be used as a weighting factor for this distance, in this way, that lower support errors mean a larger cost for the network optimizer.

[0144] Finally, once the network is trained, illustrated processing block 556 may compose the weights and kernel values ​​into a bitstream for a programmable accelerator.

[0145] Now go to Fig.15 , an enhanced object manipulation planning computing system 150 is shown. The system 150 may generally be part of an electronic device / platform having the following functions: computing functions (e.g., personal digital assistant / PDA, notebook computer, tablet computer, convertible tablet computer, server), communication functions (e.g., smart phone), imaging functions (e.g., camera, camcorder), media playback functions (e.g., smart TV / TV), wearable functions (e.g., watch, glasses, headwear, footwear, jewelry), vehicle functions (e.g., car, truck, motorcycle), robotic functions (e.g., autonomous robot), etc., or any combination thereof. In the example shown, the system 150 includes a main processor 152 (e.g., CPU) having an integrated memory controller (IMC) 154 that may be coupled to a system memory 156.

[0146] The illustrated system 150 also includes an input-output (IO) module 158 implemented as a system-on-chip (SoC) on a semiconductor die 162 together with a main processor 152 and a graphics processor 160 (e.g., a GPU). The illustrated IO module 158 communicates, for example, with a display 164 (e.g., a touch screen, a liquid crystal display / LCD, a light-emitting diode / LED display), a network controller 166 (e.g., wired and / or wireless), and a mass storage device 168 (e.g., a hard disk drive / HDD, an optical disk, a solid-state drive / SSD, a flash memory). In some embodiments, the system 150 may also include a processor and / or an AI accelerator 148 dedicated to artificial intelligence (AI) and / or neural network (NN) processing. For example, the system SoC 162 may include a visual processing unit (VPU) and / or other AI / NN dedicated processors, such as an AI accelerator 148, etc. In some embodiments, any aspects of the embodiments described herein may be implemented in one or more of a processor and / or accelerator such as an AI accelerator 148 dedicated to AI and / or NN processing, a graphics processor 160, and / or a main processor 152.

[0147] The main processor 152, graphics processor 160 and / or IO module 158 may execute instructions 170 retrieved from system memory 156 and / or mass storage device 168. In an embodiment, the computing system 150 operates in an application development level, and the instructions 170 include executable program instructions to perform the process 100 ( Figure 1 ), Method 800( Figure 2 ), process 300( Figure 3A-3B ), process 350( Figure 4 ), process 380( Figure 5 )、Method 400( Figure 6 )、Method 420( Figure 7 ) and method 440( Figure 8 ). Thus, execution of the illustrated instructions 170 may cause the computing system 150 to generate a semantically annotated image based on image data from the sensor data, wherein the semantically annotated image is used to identify the shape of an object and a semantic label of the object, associate a first set of actions with the object, decompose a command from a user into a second set of actions associated with the object, and generate a plan based on the intersection of the first set of actions and the second set of actions to satisfy the command through actuation of one or more end effectors. The kinematic end effector 144 may implement the plan to manipulate the object.

[0148] The system 150 may also include an imaging sensor 142 and a microphone 140 to receive sensor data. For example, a user may issue verbal commands to the system 150 through the microphone 140. In some embodiments, the network controller 166 may record commands issued from another device coupled to and remote from the system 150. The imaging sensor 142 may capture images that are analyzed to determine image data.

[0149] Thus, the illustrated computing system 150 is considered to be performance enhanced at least insofar as it enables computing system 150 to utilize environmental data to generate efficient plans for manipulating kinematic end effector 144 to manipulate objects. Objects may not have been previously seen by system 150, and system 150 may execute in dynamic and changing environments.

[0150] Fig.16 A semiconductor device 172 (e.g., a chip, die, package) is shown. The device 172 shown includes one or more substrates 174 (e.g., silicon, sapphire, gallium arsenide) and logic 176 (e.g., transistor arrays and other integrated circuit / IC components) coupled to the substrate(s) 174. In an embodiment, the device 172 operates in an application development level, and the logic 176 performs the process 100 ( Figure 1 ), Method 800( Figure 2 ), process 300( Figure 3A-3B), process 350( Figure 4 ), process 380( Figure 5 )、Method 400( Figure 6 )、Method 420( Figure 7 ) and method 440( Figure 8 ), thus, the logic 176 can generate a semantically annotated image based on the image data from the sensor data, wherein the semantically annotated image is used to identify the shape of the object and the semantic label of the object, associate a first set of actions with the object, decompose a command from the user into a second set of actions associated with the object, and generate a plan based on the intersection of the first set of actions and the second set of actions to satisfy the command through actuation of one or more end effectors. Thus, the logic 176 can allow the kinematic end effector to dynamically adjust based on the environment to grasp the object.

[0151] Logic 176 may be implemented at least in part in configurable logic or fixed function hardware logic. In one example, logic 176 includes a transistor channel region located within (e.g., embedded in) substrate(s) 174. Thus, the interface between logic 176 and substrate(s) 174 may not be an abrupt junction. Logic 176 may also be considered to include an epitaxial layer grown on an initial wafer of substrate(s) 174.

[0152] In some embodiments, logic 176 may also include a processor (not shown) and / or an accelerator (not shown) dedicated to AI and / or NN processing. For example, logic 176 may include a VPU and / or other AI / NN dedicated processor, etc. In some embodiments, any aspect of the embodiments described herein may be implemented in a processor and / or accelerator dedicated to AI and / or NN processing.

[0153] Fig.17 A processor core 200 is shown according to one embodiment. The processor core 200 may be a core for any type of processor, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, or other device that executes code. Fig.17 Only one processor core 200 is shown, but the processing element may alternatively include more than one Fig.17 Processor core 200 is shown in FIG. Processor core 200 may be a single-threaded core, or for at least one embodiment, processor core 200 may be multi-threaded in that it may include more than one hardware thread context (or "logical processor") per core.

[0154] Fig.17Also shown is a memory 270 coupled to the processor core 200. The memory 270 may be any of a variety of memories (including various layers of a memory hierarchy) known or otherwise available to those skilled in the art. The memory 270 may include one or more code 213 instructions to be executed by the processor core 200, wherein the code 213 may implement the process 100 ( Figure 1 ), Method 800( Figure 2 ), process 300( Figure 3A-3B 、Process 350( Figure 4 ), process 380( Figure 5 )、Method 400( Figure 6 )、Method 420( Figure 7 ) and method 440( Figure 8 ). The processor core 200 follows the program sequence of instructions indicated by the code 213. Each instruction may enter the front end portion 210 and be processed by one or more decoders 220. The decoders 220 may generate micro-operations as their outputs, such as fixed-width micro-operations of a predetermined format, or may generate other instructions, micro-instructions, or control signals that reflect the original code instructions. The front end portion 210 shown also includes register renaming logic 225 and scheduling logic 230, which generally allocates resources and queues operations corresponding to the converted instructions for execution.

[0155] Processor core 200 is shown as including execution logic 250 having a set of execution units 255-1 to 255-N. Some embodiments may include multiple execution units dedicated to a specific function or set of functions. Other embodiments may include only one execution unit or one execution unit that can perform a specific function. The execution logic 250 shown performs the operations specified by the code instructions.

[0156] After completing execution of the operations specified by the code instructions, the back-end logic 260 retires the instructions of the code 213. In one embodiment, the processor core 200 allows out-of-order execution, but requires orderly retirement of instructions. The retirement logic 256 can take various forms known to those skilled in the art (e.g., a reorder buffer, etc.). In this way, the processor core 200 transforms during execution of the code 213 at least in terms of the output generated by the decoder, the hardware registers and tables utilized by the register renaming logic 225, and any registers modified by the execution logic 250 (not shown).

[0157] Despite Fig.172, but the processing element may include other elements on the chip along with the processor core 200. For example, the processing element may include memory control logic as well as the processor core 200. The processing element may include I / O control logic and / or may include I / O control logic integrated with memory control logic. The processing element may also include one or more caches.

[0158] Reference now Fig.18 , shows a block diagram of an embodiment of a computing system 1000 according to an embodiment. Fig.18 1 is a multiprocessor system 1000 that includes a first processing element 1070 and a second processing element 1080. Although two processing elements 1070 and 1080 are shown, it should be understood that embodiments of system 1000 may also include only one such processing element.

[0159] System 1000 is shown as a point-to-point interconnect system, where a first processing element 1070 and a second processing element 1080 are coupled via a point-to-point interconnect 1050. It should be understood that Fig.18 Any or all of the interconnects shown in may be implemented as multi-drop buses rather than point-to-point interconnects.

[0160] like Fig.18 As shown, each of the processing elements 1070 and 1080 may be a multi-core processor including first and second processor cores (i.e., processor cores 1074a and 1074b and processor cores 1084a and 1084b). Such cores 1074a, 1074b, 1084a, 1084b may be configured to be combined with the above Fig.17 The instruction code is executed in a similar manner as discussed above.

[0161] Each processing element 1070, 1080 may include at least one shared cache 1896a, 1896b. Shared caches 1896a, 1896b may store data (e.g., instructions) utilized by one or more components of a processor, such as cores 1074a, 1074b and 1084a, 1084b, respectively. For example, shared caches 1896a, 1896b may locally cache data stored in memories 1032, 1034 for faster access by components of the processor. In one or more embodiments, shared caches 1896a, 1896b may include one or more intermediate level caches, such as level 2 (L2), level 3 (L3), level 4 (L4), or other level caches, last level caches (LLC), and / or combinations thereof.

[0162] Although only two processing elements 1070, 1080 are shown, it should be understood that the scope of the embodiments is not limited thereto. In other embodiments, one or more additional processing elements may be present in a given processor. Alternatively, one or more of the processing elements 1070, 1080 may be an element other than a processor, such as an accelerator or a field programmable gate array. For example, (one or more) additional processing elements may include an additional processor identical to the first processor 1070, an additional processor heterogeneous or asymmetric to the processor of the first processor 1070, an accelerator (e.g., a graphics accelerator or a digital signal processing (DSP) unit), a field programmable gate array, or any other processing element. There may be various differences between the processing elements 1070, 1080 in terms of a series of quality metrics including architecture, microarchitecture, heat, power consumption characteristics, etc. These differences can effectively present themselves as asymmetry and heterogeneity between the processing elements 1070, 1080. For at least one embodiment, various processing elements 1070, 1080 may reside in the same die package.

[0163] The first processing element 1070 may also include a memory controller logic (MC) 1072 and point-to-point (PP) interfaces 1076 and 1078. Similarly, the second processing element 1080 may include a MC 1082 and PP interfaces 1086 and 1088. Fig.10 As shown, MCs 1072 and 1082 couple the processors to respective memories, namely memory 1032 and memory 1034, which may be portions of main memory locally attached to the respective processors. Although MCs 1072 and 1082 are shown as being integrated into processing elements 1070, 1080, for alternative embodiments, the MC logic may be discrete logic external to processing elements 1070, 1080 rather than integrated therein.

[0164] First processing element 1070 and second processing element 1080 may be coupled to I / O subsystem 1090 via PP interconnects 1076 and 1086, respectively. Fig.10 As shown, I / O subsystem 1090 includes PP interfaces 1094 and 1098. In addition, I / O subsystem 1090 includes interface 1092 to couple I / O subsystem 1090 with high performance graphics engine 1038. In one embodiment, bus 1049 may be used to couple graphics engine 1038 to I / O subsystem 1090. Alternatively, a point-to-point interconnect may couple these components.

[0165] I / O subsystem 1090, in turn, may be coupled to first bus 1016 via interface 1096. In one embodiment, first bus 1016 may be a peripheral component interconnect (PCI) bus, or a bus such as a PCI Express bus or other third generation I / O interconnect bus, but the scope of the embodiments is not limited in this regard.

[0166] like Fig.18 As shown, various I / O devices 1014 (e.g., biometric scanners, speakers, cameras, sensors) can be coupled to first bus 1016 along with bus bridge 1018, which can couple first bus 1016 to second bus 1020. In one embodiment, second bus 1020 can be a low pin count (LPC) bus. In one embodiment, various devices can be coupled to second bus 1020, including, for example, keyboard / mouse 1012, communication device(s) 1026, and data storage unit 1019, such as a disk drive or other mass storage device, which can include code 1030. The code 1030 shown can implement the process 100 discussed ( Figure 1 ), Method 800( Figure 2 ), process 300( Figure 3A-3B ), process 350( Figure 4 ), process 380( Figure 5 )、Method 400( Figure 6 )、Method 420( Figure 7 )、Method 440( Figure 8 ) and may be similar to the code 213 ( Fig.17 ). In addition, an audio I / O 1024 may be coupled to the second bus 1020 , and a battery 1010 may provide power to the computing system 1000 .

[0167] Note that other embodiments are contemplated. For example, instead of Fig.18 Instead of a point-to-point architecture, the system can implement a multi-drop bus or another such communication topology. Fig.18 More or less integrated chips are shown to divide Fig.18 components.

[0168] Additional notes and examples:

[0169] Example 1 includes a computing system comprising one or more sensors for generating sensor data, the sensor data including image data, a processor coupled to the one or more sensors, and a memory including a set of executable program instructions that, when executed by the processor, cause the computing system to: generate a semantically annotated image based on image data from the sensor data, wherein the semantically annotated image is used to identify a shape of an object and a semantic label of the object; associate a first set of actions with the object; and generate a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions is associated with the command.

[0170] Example 2 includes the computing system of Example 1, wherein the instructions, when executed, further cause the computing system to: apply a first label to a first portion of the object, and apply a second label to a second portion of the object, wherein the second label is different from the first label.

[0171] Example 3 includes the computing system of Example 1, wherein the instructions, when executed, further cause the computing system to: generate a surface patch from the semantically annotated image, the surface patch being used to represent the object; reduce a resolution of the surface patch; and generate a plan based on the surface patch having the reduced resolution.

[0172] Example 4 includes the computing system of Example 1, wherein the instructions, when executed, further cause the computing system to: identify a contact force map representing contact force outputs mapped to portions of one or more end effectors; and generate a plan based on the contact force map.

[0173] Example 5 includes the computing system of any one of Examples 1 to 4, wherein the instructions, when executed, further cause the computing system to: generate an occupancy map based on the image data, wherein the occupancy map is used to identify a portion of the image to be occupied, and generate a surface map based on the occupancy map, the surface map being used to identify a surface of the object.

[0174] Example 6 includes the computing system of Example 5, wherein the instructions, when executed, further cause the computing system to: connect surfaces in the surface map to labels to generate a semantically annotated image; and identify a first set of actions based on the labels, wherein the first set of actions includes a first action that physically manipulates the object, wherein the second set of actions includes the first action that physically manipulates the object; and wherein planning includes physical manipulation of the object using the first action.

[0175] Example 7 includes a semiconductor device comprising: one or more substrates, and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more of configurable logic or fixed-function hardware logic, and the logic coupled to the one or more substrates is used to: generate a semantically annotated image based on image data, wherein the semantically annotated image is used to identify a shape of an object and a semantic label of the object; associate a first action set with the object; and generate a plan based on an intersection of the first action set and a second action set to satisfy a command from a user through actuation of one or more end effectors, wherein the second action set is associated with the command.

[0176] Example 8 includes the apparatus of Example 7, wherein the logic coupled to the one or more substrates is to: apply a first tag to a first portion of the object; and apply a second tag to a second portion of the object, wherein the second tag is different from the first tag.

[0177] Example 9 includes the apparatus of Example 7, wherein logic coupled to the one or more substrates is used to generate a surface patch from a semantically annotated image, the surface patch being used to represent the object; reduce a resolution of the surface patch; and generate a plan based on the surface patch having the reduced resolution.

[0178] Example 10 includes the apparatus of Example 7, wherein the logic coupled to the one or more substrates is to: identify a contact force map representing contact force outputs mapped to portions of the one or more end effectors; and generate a plan based on the contact force map.

[0179] Example 11 includes the apparatus of any of Examples 7 to 10, wherein logic coupled to one or more substrates is used to: generate an occupancy map based on the image data, wherein the occupancy map is used to identify a portion of the image to be occupied; and generate a surface map based on the occupancy map, the surface map being used to identify a surface of an object.

[0180] Example 12 includes the apparatus of Example 11, wherein logic coupled to one or more substrates is used to: connect surfaces in a surface map to tags to generate a semantically annotated image; and identify a first set of actions based on the tags; wherein the first set of actions includes a first action that physically manipulates an object; wherein the second set of actions includes a first action that physically manipulates an object; and wherein planning includes physical manipulation of the object using the first action.

[0181] Example 13 includes the apparatus of Example 7, wherein the logic coupled to the one or more substrates includes a transistor channel region located within the one or more substrates.

[0182] Example 14 includes at least one computer-readable storage medium, which includes a set of executable program instructions, which, when executed by a computing system, causes the computing system to: generate a semantically annotated image based on image data, wherein the semantically annotated image is used to identify a shape of an object and a semantic label of the object; associate a first set of actions with the object; and generate a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions is associated with the command.

[0183] Example 15 includes at least one computer-readable storage medium of Example 14, wherein the instructions, when executed, further cause the computing system to: apply a first label to a first portion of the object; and apply a second label to a second portion of the object, wherein the second label is different from the first label.

[0184] Example 16 includes at least one computer-readable storage medium of Example 14, wherein the instructions, when executed, further cause the computing system to: generate a surface patch from the semantically annotated image, the surface patch being used to represent the object; reduce the resolution of the surface patch; and generate a plan based on the surface patch having the reduced resolution.

[0185] Example 17 includes at least one computer-readable storage medium of Example 14, wherein the instructions, when executed, further cause the computing system to: identify a contact force map representing contact force outputs mapped to portions of one or more end effectors; and generate a plan based on the contact force map.

[0186] Example 18 includes at least one computer-readable storage medium of any one of Examples 14 to 17, wherein the instructions, when executed, further cause the computing system to: generate an occupancy map based on the image data, wherein the occupancy map is used to identify a portion of the image to be occupied; and generate a surface map based on the occupancy map, the surface map being used to identify a surface of the object.

[0187] Example 19 includes at least one computer-readable storage medium of Example 18, wherein the instructions, when executed, further cause the computing system to: connect surfaces in the surface map to labels to generate a semantically annotated image; and identify a first set of actions based on the labels, wherein the first set of actions includes a first action that physically manipulates the object; wherein the second set of actions includes the first action that physically manipulates the object; and wherein planning includes physical manipulation of the object using the first action.

[0188] Example 20 includes a method of operating a computing system, the method comprising generating a semantically annotated image based on image data, wherein the semantically annotated image is used to identify a shape of an object and a semantic label of the object; associating a first set of actions with the object; and generating a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions is associated with the command.

[0189] Example 21 includes the method of Example 20, further comprising: applying a first label to a first portion of the object; and applying a second label to a second portion of the object, wherein the second label is different from the first label.

[0190] Example 22 includes the method of Example 20, further comprising: generating a surface patch from the semantically annotated image, the surface patch for representing the object; reducing a resolution of the surface patch; and generating a plan based on the surface patch with the reduced resolution.

[0191] Example 23 includes the method of Example 20, further comprising: identifying a contact force map representing contact force outputs mapped to portions of one or more end effectors; and generating a plan based on the contact force map.

[0192] Example 24 includes the method of any one of Examples 20 to 23, further comprising: generating an occupancy map based on the image data, wherein the occupancy map is used to identify a portion of the image to be occupied; and generating a surface map based on the occupancy map, the surface map is used to identify a surface of the object.

[0193] Example 25 includes the method of Example 24, further comprising: connecting surfaces in the surface map to labels to generate a semantically annotated image; and identifying a first set of actions based on the labels; wherein the first set of actions includes a first action that physically manipulates the object; wherein the second set of actions includes a first action that physically manipulates the object; and wherein planning includes physical manipulation of the object using the first action.

[0194] Example 26 includes a semiconductor device comprising: a device for generating a semantically annotated image based on image data, wherein the semantically annotated image is used to identify the shape of an object and a semantic label of the object; a device for associating a first set of actions with the object; and a device for generating a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions is associated with the command.

[0195] Example 27 includes the apparatus of Example 26, further comprising: means for applying a first label to a first portion of the object; and means for applying a second label to a second portion of the object, wherein the second label is different from the first label.

[0196] Example 28 includes the apparatus of Example 26, further comprising: a device for generating a surface patch from a semantically annotated image, the surface patch being used to represent an object; a device for reducing the resolution of the surface patch; and a device for generating a plan based on the surface patch having the reduced resolution.

[0197] Example 29 includes the apparatus of Example 26, further comprising: means for identifying a contact force map representing contact force outputs mapped to portions of one or more end effectors; and means for generating a plan based on the contact force map.

[0198] Example 30 includes the apparatus of any one of Examples 26 to 29, further comprising: a device for generating an occupancy map based on the image data, wherein the occupancy map is used to identify a portion of the image to be occupied; and a device for generating a surface map based on the occupancy map, the surface map being used to identify a surface of an object.

[0199] Example 31 includes the apparatus of Example 26, further comprising: a device for connecting surfaces in a surface map to labels to generate a semantically annotated image; and a device for identifying a first set of actions based on the labels; wherein the first set of actions includes a first action that physically manipulates the object; wherein the second set of actions includes the first action that physically manipulates the object; and wherein planning includes physical manipulation of the object using the first action.

[0200] Thus, the techniques described herein can provide an autonomous robot that dynamically adjusts a kinematic end effector to manipulate (e.g., grasp) an object in a rapidly changing environment. The autonomous robot can also reduce memory footprint and latency by utilizing reduced resolution semantic patches, and early review and rejection of inappropriate grasps and semantic patches.

[0201] Embodiments are applicable to use with all types of semiconductor integrated circuit ("IC") chips. Examples of these IC chips include, but are not limited to, processors, controllers, chipset components, programmable logic arrays (PLA), memory chips, network chips, systems on chips (SoC), SSD / NAND controller ASICs, etc. In addition, in some of the drawings, signal conductors are represented by lines. Some may be different to indicate more constituent signal paths, with digital markings to indicate the number of constituent signal paths, and / or with arrows at one or more ends to indicate the main information flow direction. However, this should not be interpreted in a restrictive manner. Instead, these added details can be used in conjunction with one or more exemplary embodiments to facilitate easier understanding of the circuit. Any signal line represented, whether or not with additional information, may actually include one or more signals that can propagate in multiple directions, and may be implemented using any appropriate type of signal scheme, for example, digital or analog lines, fiber optic lines, and / or single-ended lines implemented using differential pairs.

[0202] Exemplary sizes / models / values / ranges have been given, but the embodiments are not limited thereto. As manufacturing techniques (e.g., lithography) mature over time, it is expected that devices of smaller sizes can be manufactured. In addition, for simplicity of explanation and discussion, and in order not to make certain aspects of the embodiments difficult to understand, known power / ground connections to IC chips and other components may or may not be shown in the drawings. In addition, the arrangement may be shown in block diagram form to avoid making the embodiments difficult to understand, and in view of the fact that the details of the implementation of such block diagram arrangements are highly dependent on the computing system in which the embodiments will be implemented, that is, such details should be within the knowledge of those skilled in the art. In the case of elaborating specific details (e.g., circuits) for describing example embodiments, it should be clear to those skilled in the art that the embodiments may be practiced without these specific details or with variations of these specific details. Therefore, this description should be considered illustrative rather than restrictive.

[0203] The term "coupled" may be used herein to refer to any type of direct or indirect relationship between the components being discussed, and may apply to electrical, mechanical, fluid, optical, electromagnetic, electromechanical or other connections. In addition, the terms "first," "second," etc. are used herein only to facilitate discussion, and unless otherwise indicated, do not carry a specific temporal or sequential meaning.

[0204] As used in this application and in the claims, a list of items connected by the term "one or more of" may mean any combination of the listed items. For example, the phrase "one or more of A, B, or C" may mean A; B; C; A and B; A and C; B and C; or A, B, and C.

[0205] Those skilled in the art will appreciate from the foregoing description that the broad techniques of the embodiments can be implemented in a variety of forms. Therefore, while the embodiments have been described in conjunction with specific examples thereof, the true scope of the embodiments should not be so limited, as other modifications will become apparent to those skilled in the art after studying the drawings, the specification, and the appended claims.

Claims

1. At least one memory including machine-readable instructions for causing one or more processors to perform at least the following operations: Calculating a three-dimensional (3D) point cloud of an object based on input image data; Invoking a neural network based on the 3D point cloud to determine one or more grasps of an object associated with the planned trajectory; and The robot is caused to perform an action based on the one or more grasps.

2. The at least one memory of claim 1, wherein the instructions are configured to cause the one or more processors to determine the action based on a current state.

3. At least one memory according to claim 1, wherein the motion is associated with six degrees of freedom.

4. At least one memory according to claim 3, wherein the six degrees of freedom correspond to three positional degrees of freedom and three orientational degrees of freedom.

5. The at least one memory of claim 1, wherein the instructions are configured to cause the one or more processors to obtain the input image data from a sensor.

6. The at least one memory of claim 1, wherein the action is a first action and the instructions are for causing the one or more processors to determine the first action based on a second action.

7. The at least one memory of claim 1, wherein the instructions are configured to cause the one or more processors to perform image segmentation to identify the object.

8. An apparatus comprising: Memory; instruction; as well as One or more processors configured to operate based on the instructions to: Calculating a three-dimensional (3D) point cloud of an object based on input image data; Invoking a neural network based on the 3D point cloud to determine one or more grasps of an object associated with the planned trajectory; and The robot is caused to perform an action based on the one or more grasps.

9. The apparatus of claim 8, wherein the one or more processors are configured to determine the action based on a current state.

10. The apparatus of claim 8, wherein the motion is associated with six degrees of freedom.

11. The apparatus of claim 10, wherein the six degrees of freedom correspond to three positional degrees of freedom and three orientational degrees of freedom.

12. The apparatus of claim 8, wherein the one or more processors are configured to obtain the input image data from a sensor.

13. The apparatus of claim 8, wherein the action is a first action, and the one or more processors are configured to determine the first action based on a second action.

14. The apparatus of claim 8, wherein the one or more processors are configured to perform image segmentation to identify the object.

15. A method comprising: computing, using one or more processor circuits, a three-dimensional (3D) point cloud of the object based on the input image data; Invoking a neural network based on the 3D point cloud to determine one or more grasps of an object associated with the planned trajectory; as well as The robot is caused to perform an action based on the one or more grasps.

16. The method according to claim 15, wherein the method comprises: The action is determined based on the current state.

17. The method of claim 15, wherein the motion is associated with six degrees of freedom.

18. The method of claim 17, wherein the six degrees of freedom correspond to three positional degrees of freedom and three orientational degrees of freedom.

19. The method according to claim 15, wherein the method comprises: The input image data is obtained from a sensor.

20. The method of claim 15, wherein the action is a first action, and the method comprises: The first action is determined based on a second action.

21. The method of claim 15, wherein the method comprises: Image segmentation is performed to identify the objects.

Images