Probabilistic method for unified representation of robotic mapping

Abstract

The present disclosure provides a method, an apparatus, and a computer-readable medium for evaluating reliability of three-dimensional shape prediction. In an embodiment, a method includes obtaining a scene representation comprising one or more images, estimating semantic information of a partially observed object in the scene representation based on geometric information extracted from the one or more images, determining a segmented point cloud of the partially observed object based on the semantic information and the geometric information, and displaying the segmented point cloud of the partially observed object in the scene representation. A metric embedding of segmented point clouds corresponding to the partially observed object is created, a complemented shape point cloud of the partially observed object is predicted, a confidence score for each point of the complemented shape point cloud is generated using a probabilistic machine learning model based on a correlation between geometric information and semantic information, and a confidence score for each point of the complemented shape point cloud is generated based on a correlation between geometric information and semantic information. And controlling motion of the robot based on the confidence score for each point of the complemented shape point cloud.

Description

Technical Field

[0001] The present disclosure relates generally to robotic mapping and control, and more particularly, to methods, apparatus, and computer-readable media for evaluating the reliability of three-dimensional (3D) shape predictions. Background Art

[0002] In light of the high demand for performing more complex robotic tasks, the importance of the ability to map a robot's environment using both semantic and geometric information may have increased significantly. That is, without a semantic and / or geometric understanding of its surroundings, a robot may not be able to perform tasks such as, but not limited to, autonomous driving, home assistants, navigating through complex environments (e.g., warehouses), etc. For example, a home assistant robot with manipulation capabilities and / or a warehouse robot performing pick and place operations may need to sense the entire environment around the robot in order to work seamlessly even in the presence of occlusions and / or visual obstructions.

[0003] When only a partial and / or obstructed view of an object is available in such an environment, shape completion can be used to predict the complete shape of the partially observable object. Using the predicted completed shape of the object, the robot can plan the motion required to perform tasks (e.g., grasping, pushing, pulling, etc.). However, in order to reliably perform these tasks, the robot may need to evaluate the uncertainty of each predicted point based on the completed shape of the object. That is, the robot may need to decide which points of the predicted shape are more likely to be accurate and, therefore, useful for reliably planning the motion required to perform the task.

[0004] However, related techniques in robotic mapping may be optimized for reconstruction quality and, therefore, may not provide uncertainty information, may only provide uncertainty information for the scene as a whole, and / or may only provide uncertainty information for the object as a whole. Therefore, reliance on mapping information produced by these related techniques may be problematic because it may be difficult to obtain an estimate of the uncertainty of a point in the predicted shape.

[0005] Therefore, there is a need for further improvements in robotic mapping, as the need to perform more complex robotic tasks may be constrained by the uncertainty of the surrounding environment where the robot's mapping cannot be assessed. This paper proposes improvements. These improvements may also be applicable to other computer vision and / or mapping techniques. Summary of the invention

[0006] Technical Solution

[0007] A simplified overview of one or more embodiments of the present disclosure is presented below in order to provide a basic understanding of these embodiments. This overview is not an extensive overview of all contemplated embodiments, and is neither intended to identify key or important elements of all embodiments nor to delineate the scope of any or all embodiments. Its sole purpose is to present some concepts of one or more embodiments of the present disclosure in a simplified form as a prelude to a more detailed description presented later.

[0008] The present disclosure discloses methods, apparatuses, and computer readable media for evaluating the reliability of three-dimensional (3D) shape predictions. Various aspects of the present disclosure provide for performing robotic mapping and / or controlling a robot based on the reliability of 3D shape predictions.

[0009] According to one aspect of the present disclosure, a method for controlling robot motion performed by a processor is provided. The method may include obtaining a scene representation including one or more images. Each of the one or more images may include depth information. The method may also include estimating semantic information of a partially observed object in the scene representation based on geometric information extracted from the one or more images. The method may also include determining a segmented point cloud of a partially observed object in the scene representation based on semantic information and geometric information. The method may also include creating a metric embedding of a segmented point cloud corresponding to a partially observed object of the scene representation. The method may also include predicting a completed shape point cloud of a partially observed object of the scene representation. The method may also include using a probabilistic machine learning model to generate a confidence score for each point of the completed shape point cloud based on the correlation between the geometric information and the semantic information. The probabilistic machine learning model may have been trained using training data including a metric embedding of a segmented point cloud. The method may also include controlling the motion of the robot based on the confidence score of each point of the completed shape point cloud.

[0010] In an embodiment, the processor may be included in the robot, and the partially observed objects of the scene representation may include the target object. In an embodiment, controlling the movement of the robot may include controlling at least one of an arm and an end effector of the robot so that the robot can grasp the target object.

[0011] In an embodiment, the processor may be included in an external device that may be provided separately from the robot, and the partially observed objects represented by the scene may include a target object. In an embodiment, controlling the motion of the robot may include sending motion control parameters of at least one of an arm and an end effector of the robot to the robot, which may enable the robot to grasp the target object based on the motion control parameters.

[0012] In an embodiment, the probabilistic machine learning model may be a Gaussian process model, and the partially observed objects of the scene representation may include a target object. In an embodiment, the generating of the confidence score may include generating the confidence score based on at least one of an object property indicating whether the target object is fragile and past robot experience indicating whether the robot has previously handled the target object.

[0013] In an embodiment, the partially observed objects of the scene representation may include a target object. In an embodiment, the method may further include identifying an area of ​​the target object having a number of data points in the completed shape point cloud having corresponding confidence scores below a predetermined threshold, and controlling the robot to capture at least one other image of the area of ​​the target object.

[0014] In an embodiment, obtaining the scene representation may include obtaining one or more images from a first camera position. In an embodiment, the method may further include filtering out points having corresponding confidence scores that may fail to meet a threshold confidence score from the completed shape point cloud, resulting in the filtered points of the completed shape point cloud, and determining a second camera position different from the first camera position based on the filtered points of the completed shape point cloud, obtaining at least one other image of the scene representation from the second camera position, the at least one other image may include at least one partially observed portion of a partially observed object of the scene representation.

[0015] In an embodiment, the method may further include adjusting the threshold confidence score based on an updated confidence score of at least one other image.

[0016] In an embodiment, determining a segmented point cloud may include extracting geometric information from image data and depth information of one or more images, estimating semantic information for each pixel of the one or more images, determining an estimated confidence score for the semantic information for each pixel of the one or more images using the geometric information, and combining the semantic information with the depth information, resulting in a segmented point cloud.

[0017] In an embodiment, the determination of the segmented point cloud may include maintaining a semantic map of the segmented point cloud, which may include semantic information of partially observed objects of the scene representation, and tracking the partially observed objects between a first image in the one or more images and a second image in the one or more images using the semantic information. The semantic information may include a semantic class assigned to each of the partially observed objects.

[0018] In an embodiment, the creation of a metric embedding of a segmented point cloud may include providing the segmented point cloud to a transformer model to obtain an embedding of the segmented point cloud, and providing the embedding of the segmented point cloud to a perceptron model to obtain a metric embedding of the segmented point cloud. The embedding of the segmented point cloud may include semantic information and geometric information of the segmented point cloud. The metric embedding of the segmented point cloud may have been forced into a unit-sphere metric space.

[0019] In an embodiment, the creation of the metric embedding of the segmented point cloud may further include assigning zero embeddings to free spaces in the metric embedding of the segmented point cloud.

[0020] In an embodiment, the prediction of the completed shape point cloud may include providing one or more images including image data and depth information to a shape completion model, obtaining a shape reconstruction of a partially observed object of the scene representation from the shape completion model, and transforming the shape reconstruction from a coordinate system of the image sensor to a world coordinate system based on physical configuration information of the image sensor. The shape reconstruction may be mapped to the coordinate system of the image sensor to obtain the one or more images.

[0021] According to one aspect of the present disclosure, a means for controlling robot motion is provided. The means may include a memory storing instructions, and a processor communicatively coupled to the memory. The processor may be configured to execute instructions to obtain a scene representation including one or more images. Each of the one or more images may include depth information. The processor may also be configured to execute instructions to estimate semantic information of a partially observed object in the scene representation based on geometric information extracted from the one or more images. The processor may also be configured to execute instructions to determine a segmented point cloud of a partially observed object in the scene representation based on semantic information and geometric information. The processor may also be configured to execute instructions to create a metric embedding of a segmented point cloud corresponding to a partially observed object of the scene representation. The processor may also be configured to execute instructions to predict a completed shape point cloud of a partially observed object of the scene representation. The processor may also be configured to execute instructions to generate a confidence score for each point of the completed shape point cloud using a probabilistic machine learning model based on the correlation between the geometric information and the semantic information. The probabilistic machine learning model may have been trained using training data including a metric embedding of a segmented point cloud. The processor may be further configured to execute instructions to control the movement of the robot based on the confidence score of each point of the completed shape point cloud.

[0022] In an embodiment, the apparatus may be included in a robot, and the partially observed objects of the scene representation may include a target object. In an embodiment, the processor may also be configured to execute further instructions to control at least one of an arm and an end effector of the robot to enable the robot to grasp the target object.

[0023] In one embodiment, the apparatus may be included in an external device that may be provided separately from the robot, and the partially observed objects represented by the scene may include a target object. In one embodiment, the processor may be further configured to execute further instructions to send motion control parameters of at least one of the robot's arm and end effector to the robot, so that the robot grasps the target object based on the motion control parameters.

[0024] In an embodiment, the probabilistic machine learning model may be a Gaussian process model, and the portion of the observed objects of the scene representation may include a target object. In an embodiment, the processor may be further configured to execute further instructions to generate a confidence score based on at least one of an object property indicating whether the target object is fragile and past robot experience indicating whether the robot has previously handled the target object.

[0025] In an embodiment, the partially observed objects of the scene representation may include a target object. In an embodiment, the processor may be further configured to execute further instructions to identify a region of the target object having a plurality of data points in the completed shape point cloud having corresponding confidence scores below a predetermined threshold, and control the robot to capture at least one other image of the region of the target object.

[0026] In an embodiment, the processor may be further configured to execute further instructions to extract geometric information from image data and depth information of one or more images, estimate semantic information for each pixel of the one or more images using the geometric information, determine an estimated confidence score for the semantic information for each pixel of the one or more images, and combine the semantic information with the depth information, resulting in a segmented point cloud.

[0027] In an embodiment, the processor may be further configured to execute further instructions to provide the segmented point cloud to the transformer model, obtain an embedding of the segmented point cloud, and provide the embedding of the segmented point cloud to the perceptron model to obtain a metric embedding of the segmented point cloud. The embedding of the segmented point cloud may include semantic information and geometric information of the segmented point cloud. The metric embedding of the segmented point cloud may have been forced into a unit sphere metric space.

[0028] According to one aspect of the present disclosure, a device including means for controlling the motion of a robot is provided. The device may include means for obtaining a scene representation including one or more images. Each of the one or more images may include depth information. The device may also include means for estimating semantic information of a partially observed object in the scene representation based on geometric information extracted from the one or more images. The device may also include means for determining a segmented point cloud of a partially observed object in the scene representation based on semantic information and geometric information. The device may also include means for creating a metric embedding of the segmented point cloud corresponding to the partially observed object of the scene representation. The device may also include means for predicting a completed shape point cloud of the partially observed object of the scene representation. The device may also include means for generating a confidence score for each point of the completed shape point cloud using a probabilistic machine learning model based on the correlation between the geometric information and the semantic information. The probabilistic machine learning model may have been trained using training data including the metric embedding of the segmented point cloud. The device may also include means for controlling the motion of the robot based on the confidence score of each point of the completed shape point cloud.

[0029] According to one aspect of the present disclosure, a computer-readable storage medium storing computer executable instructions for controlling the motion of a robot is provided. The computer executable instructions, when executed by at least one processor of a device, may cause the device to obtain a scene representation including one or more images. Each of the one or more images may include depth information. The computer executable instructions may also cause the device to estimate semantic information of a partially observed object in the scene representation based on geometric information extracted from the one or more images. The computer executable instructions may also cause the device to determine a segmented point cloud of a partially observed object in the scene representation based on the semantic information and the geometric information. The computer executable instructions may also cause the device to create a metric embedding of the segmented point cloud corresponding to the partially observed object of the scene representation. The computer executable instructions may also cause the device to predict a completed shape point cloud of the partially observed object of the scene representation. The computer executable instructions may also cause the device to generate a confidence score for each point of the completed shape point cloud using a probabilistic machine learning model based on the correlation between the geometric information and the semantic information. The probabilistic machine learning model may have been trained using training data including the metric embedding of the segmented point cloud. The computer executable instructions may also cause the device to control the motion of the robot based on the confidence score of each point of the completed shape point cloud.

[0030] According to one aspect of the present disclosure, a method for evaluating the reliability of a 3D shape prediction performed by a processor is provided. The method may include creating a metric embedding of a segmented point cloud corresponding to a partially observed object represented by a scene based on estimated semantic information of the partially observed object and geometric information extracted from a scene representation. The method may also include predicting a completed shape point cloud of the partially observed object represented by the scene. The method may also include generating a confidence score for each point of the completed shape point cloud using a probabilistic machine learning model based on a correlation between the geometric information and the semantic information.

[0031] Additional aspects are set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other aspects, features and advantages of certain embodiments of the present disclosure may be more apparent from the following description in conjunction with the accompanying drawings, in which:

[0033] Figure 1 Depicts examples of devices that may be used to implement one or more aspects of the present disclosure;

[0034] Figure 2 An example of a process flow for evaluating the reliability of a three-dimensional (3D) shape prediction according to various aspects of the present disclosure is shown;

[0035] Figure 3 depicts an example of a robotic control system according to various aspects of the present disclosure;

[0036] Figure 4 An example of a view planning process according to various aspects of the present disclosure is shown;

[0037] Figure 5 depicts an example of a data pipeline for evaluating the reliability of 3D shape predictions in accordance with various aspects of the present disclosure;

[0038] Figure 6 An example of a metric embedding component according to various aspects of the present disclosure is shown;

[0039] Figure 7 depicts an example of a Gaussian process (GP) inference component according to various aspects of the present disclosure;

[0040] Figure 8 A block diagram illustrating an example apparatus for evaluating the reliability of a 3D shape prediction according to various aspects of the present disclosure;

[0041] Fig. 9 A flowchart depicting an example method of controlling robot motion according to various aspects of the present disclosure; and

[0042] Fig.10 A flow chart illustrating another example method of evaluating the reliability of a 3D shape prediction according to various aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0043] The detailed description set forth below in conjunction with the accompanying drawings is intended to be a description of various configurations, and is not intended to represent the only configuration that can practice the concepts described herein. For the purpose of providing a thorough understanding of various concepts, the specific implementation includes specific details. However, it is clear to those skilled in the art that these concepts can be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid blurring such concepts. In the following description, throughout the specification and the accompanying drawings, the same components are marked with the same numbers, respectively.

[0044] The following description provides examples and does not limit the scope, applicability or embodiments set forth in the claims. Without departing from the scope of the present disclosure, the functions and / or arrangements of the elements discussed may be changed. Various examples may appropriately omit, replace or add various processes or components. For example, the described methods may be performed in an order different from the described order, and various steps may be added, omitted and / or combined. Alternatively or additionally, the features described with reference to some examples may be combined in other examples.

[0045] Various aspects and / or features may be presented in terms of systems that may include multiple devices, components, modules, etc. It should be understood and appreciated that the various systems may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Combinations of these aspects may also be used.

[0046] As a general introduction to the subject matter described in more detail below, various aspects described herein relate to apparatus, methods, and computer-readable media for evaluating the reliability of three-dimensional (3D) shape predictions. Various aspects described herein can be used to perform robotic mapping and / or control a robot based on the reliability of 3D shape predictions.

[0047] Various aspects presented herein can provide a probabilistic mapping method for scenes where only partial views of at least some objects in the scene are visible. For example, various aspects presented herein can provide semantic information for predicting partially observed objects in a scene representation, obtaining a segmented point cloud of the partially observed objects based on the semantic information and geometric information, and creating a metric embedding of the segmented point cloud corresponding to the partially observed objects. In addition, various aspects described herein can provide a shape point cloud for predicting the completion of the partially observed objects. Other aspects presented herein can evaluate the uncertainty of the completed shape point cloud using its corresponding semantic information to provide a confidence score for each point of the completed shape point cloud of the partially observed object. For example, the evaluation can be performed in a relatively high-dimensional space by a probabilistic machine learning model, which can generate a confidence metric with a relatively high degree of quality (e.g., accuracy). Therefore, various aspects presented herein can be used to filter the predicted completed shapes using confidence scores, and thus generate an environment map with a known level of certainty. In turn, robots can use these environment maps to plan movements to perform tasks (e.g., grasping, pushing, pulling, etc.).

[0048] Various aspects described herein can provide advantages over related robotic mapping techniques by providing confidence scores, which can be used to assess the uncertainty of the points of the predicted shape completion across the entire scene. Various aspects described herein can also provide merging semantic and geometric information obtained from an image representing a scene with a predicted completed shape to generate a confidence score. In this way, various aspects described herein can be free from the constraints (bound) of object semantic classes, the complexity of object geometry, and / or the shape completion network. Therefore, various aspects described herein can generally be applied to a variety of environments, scenes, and embodiments.

[0049] As mentioned above, this document discusses certain embodiments related to evaluating the reliability of 3D shape predictions. However, before discussing these concepts in further detail, Figure 1 Examples of computing devices that may be used to implement and / or otherwise provide various aspects of the present disclosure are discussed.

[0050] Figure 1 An example of a device 100 that can be used to implement one or more aspects of the present invention according to one or more illustrative aspects discussed herein is depicted. For example, in some instances, the device 100 can implement one or more aspects of the present disclosure by reading and / or executing instructions and performing one or more actions accordingly. In one or more arrangements, the device 100 can represent, be incorporated into, and / or include a robotic device, a robotic controller, a desktop computer, a computer server, a virtual machine, a network device, a mobile device (e.g., a laptop computer, a tablet computer, a personal digital assistant (PDA), a smart phone, any other type of mobile computing device, etc.), a wearable device (e.g., a smart watch, a headset, a headset, etc.), a smart device (e.g., a voice-controlled virtual assistant, a set-top box (STB), a refrigerator, an air conditioner, a microwave oven, a television, etc.), an Internet of Things (IoT) device, and / or any other type of data processing device.

[0051] For example, device 100 may include a processor, a personal computer (PC), a printed circuit board (PCB) including a computing device, a minicomputer, a mainframe computer, a microcomputer, a telephone computing device, a wired / wireless computing device (e.g., a smart phone, a PDA), a laptop computer, a tablet computer, a smart device, a wearable device, or any other similarly functional device.

[0052] In an embodiment, Figure 1 As shown, device 100 may include a set of components such as processor 120, memory 130, storage component 140, input component 150, output component 160, communication interface 170, and shape prediction evaluation component 180. The set of components of device 100 may be communicatively coupled via bus 110.

[0053] The bus 110 may include one or more components that may permit communication between the set of components of the device 100. For example, the bus 110 may be a communication bus, a crossbar, a network, etc. Although the bus 110 may be Figure 1 Although depicted as a single line in FIG. 1 , bus 110 may be implemented using multiple (eg, two or more) connections between the set of components of device 100. The present disclosure is not limited in this regard.

[0054] The device 100 may include one or more processors, such as a processor 120. The processor 120 may be implemented in hardware, firmware, and / or a combination of hardware and software. For example, the processor 120 may include a central processing unit (CPU), an application processor (AP), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an image signal processor (ISP), a neural processing unit (NPU), a sensor hub processor, a communication processor (CP), an artificial intelligence (AI) dedicated processor designed to have a hardware structure specific to processing AI models, a general-purpose single-chip and / or multi-chip processor, or other programmable logic devices designed to perform the functions described herein, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may include a microprocessor or any conventional processor, controller, microcontroller, or state machine.

[0055] The processor 120 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of a main processor and an auxiliary processor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. In an embodiment, specific processes and methods may be performed by circuits specific to a given function. In an embodiment, the auxiliary processor may be configured to consume less power than the main processor. Alternatively or additionally, one or more processors may be implemented separately (e.g., as several different chips) and / or may be combined into a single form.

[0056] Processor 120 may control the overall operation of device 100 and / or the set of components of device 100 (eg, memory 130, storage component 140, input component 150, output component 160, communication interface 170, and shape prediction evaluation component 180).

[0057] The device 100 may also include a memory 130. In an embodiment, the memory 130 may include a volatile memory, such as, but not limited to, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), etc. In an embodiment, the memory 130 may include a non-volatile memory, such as, but not limited to, a read-only memory (ROM), an electrically erasable programmable ROM (EPROM), a NAND flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), a magnetic memory, an optical memory, etc. However, the present disclosure is not limited in this regard, and the memory 130 may include other types of dynamic and / or static memory storage. In an embodiment, the memory 130 may store information and / or instructions used (e.g., executed) by the processor 120.

[0058] The storage component 140 of the device 100 may store information and / or computer-readable instructions and / or code related to the operation and use of the device 100. For example, the storage component 140 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a universal serial bus (USB) flash drive, a Personal Computer Memory Card International Association (PCMCIA) card, a floppy disk, a cassette, a magnetic tape, and / or another type of computer-readable medium, and a corresponding drive.

[0059] Device 100 may also include input component 150. Input component 150 may include one or more components that may allow device 100 to receive information, such as via user input (e.g., a touch screen, keyboard, keypad, mouse, stylus, buttons, switches, microphone, camera, virtual reality (VR) headset, haptic gloves, etc.). Alternatively or additionally, input component 150 may include one or more sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, a transducer, a contact sensor, a proximity sensor, a range-finding device, a camera, a video camera, a depth camera, a time-of-flight (TOF) camera, a stereo camera, etc.). In one embodiment, input component 150 may include more than one of the same sensor type (e.g., multiple cameras).

[0060] The output component 160 of the device 100 may include one or more components that can provide output information from the device 100 (e.g., a display, a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), a tactile feedback device, a speaker, a buzzer, an alarm, etc.).

[0061] The device 100 may also include a communication interface 170. The communication interface 170 may include a receiver component, a transmitter component and / or a transceiver component. The communication interface 170 may enable the device 100 to establish a connection and / or transmit communication with other devices (e.g., a server, another device). The communication may be implemented via a wired connection, a wireless connection, or a combination of a wired and wireless connection. The communication interface 170 may allow the device 100 to receive information from another device and / or provide information to another device. In an embodiment, the communication interface 170 may provide communication with another device via a network, such as a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a private network, a self-organizing network, an intranet, the Internet, a fiber-based network, a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a telephone network (e.g., a public switched telephone network (PSTN)), etc. and / or a combination of these or other types of networks. Alternatively or additionally, the communication interface 170 may provide communication with another device via a device-to-device (D2D) communication link, such as FlashLinQ, WiMedia, Bluetooth TM 、Bluetooth TM Low Energy (BLE), ZigBee, Institute of Electrical and Electronics Engineers (IEEE) 802.11x (Wi-Fi), LTE, 5G, etc. In an embodiment, the communication interface 170 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a USB interface, an IEEE1094 (FireWire) interface, etc.

[0062] In one embodiment, the device 100 may include a shape prediction evaluation component 180, which may be configured to evaluate the reliability of the 3D shape prediction. For example, the shape prediction evaluation component 180 may be configured to obtain a scene representation, estimate semantic information of a partially observed object, determine a segmented point cloud of the partially observed object, create a metric embedding of the segmented point cloud, predict a completed shape point cloud, generate a confidence score for the completed shape point cloud, and control the motion of the robot.

[0063] Device 100 may perform one or more processes described herein. Device 100 may perform operations based on processor 120 executing computer-readable instructions and / or code that may be stored by a computer-readable medium, such as memory 130 and / or storage component 140. The computer-readable medium may refer to a memory device. The memory device may include memory space within a single physical storage device and / or memory space distributed across multiple physical storage devices.

[0064] The computer-readable instructions and / or code may be read into the memory 130 and / or storage component 140 from another computer-readable medium or from another device via the communication interface 170. The computer-readable instructions and / or code stored in the memory 130 and / or storage component 140, if or when executed by the processor 120, may cause the device 100 to perform one or more processes described herein.

[0065] Alternatively or additionally, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein.Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0066] Figure 1 The number and arrangement of components shown in the figure are provided as examples. In practice, Figure 1 There may be additional components, fewer components, different components, or differently arranged components than those shown in . Figure 1 Two or more components shown in may be implemented in a single component, or Figure 1 The single component shown in can be implemented as multiple distributed components. Alternatively or additionally, Figure 1 A set (one or more) of components shown in the figure may perform the operations described as being performed by Figure 1 One or more functions performed by another set of components shown in FIG.

[0067] Having discussed examples of apparatus that may be used to provide and / or implement various aspects of the disclosure, various embodiments are now discussed in further detail. In particular, and as introduced above, some aspects of the disclosure generally relate to evaluating the reliability of 3D shape predictions.

[0068] Figure 2 An example of a process flow for evaluating the reliability of a 3D shape prediction according to various aspects of the present disclosure is shown. Figure 2 The shape prediction evaluation process 200 may include an image capture operation 210 , a segmentation and point cloud embedding operation 220 , a shape completion operation 230 , and an uncertainty estimation operation 240 .

[0069] In an embodiment, at least a portion of the shape prediction evaluation process 200 may be performed by Figure 1 The shape prediction evaluation process 200 is performed by the device 100, which may include a shape prediction evaluation component 180. Alternatively or additionally, another computing device (e.g., a robotic device, a robotic controller, a server, a laptop, a smart phone, a user equipment (UE), a wearable device, a smart device, an IoT device, etc.) including the shape prediction evaluation component 180 can perform at least a portion of the shape prediction evaluation process 200.

[0070] refer to Figure 2 In the image capture operation 210, the shape prediction evaluation process 200 may include a processor (e.g., Figure 1 The processor 120 of the device 100 obtains a scene representation including one or more images. In an embodiment, the one or more images may include a red-green-blue (RGB) image. Alternatively or additionally, the one or more images may include depth information. For example, the one or more images may include a two-dimensional (2D) RGB image associated with and / or including the depth information. In an embodiment, the one or more images may be composed of Figure 1 The input component 150 of the device 100 is captured. For example, the input component 150 may include at least one of a camera, a video camera, a depth camera, a TOF camera, a stereo camera, etc., capable of capturing and / or providing one or more images representing a scene. That is, the present disclosure is not limited in this regard, and the various aspects presented herein can be applied with other types of image sensors. It is worth noting that the various aspects presented herein can be used with any image sensor capable of providing image data and geometric information associated with the image data. As another example, the input component 150 may include multiple (e.g., more than one) sensors of the same type (e.g., multiple cameras). That is, in such an example, multiple cameras can be used to capture one or more images.

[0071] like Figure 2 As shown, one or more images representing the scene obtained by the image capture operation 210 can be provided to the segmentation and point cloud embedding operation 220 and the shape completion operation 230.

[0072] In the segmentation and point cloud embedding operation 220, the shape prediction evaluation process 200 may include determining the segmented objects from the image data (e.g., one or more images) and creating a segmented point cloud. The segmented point cloud may correspond to one or more objects observed in the scene representation. Alternatively or additionally, at least one of the objects observed in the scene representation may be referred to as a partially observed object. As used herein, a partially observed object may include an object included in the scene representation that may not be fully visible to the image sensor. For example, a partially observed object may include an object in which at least a portion of the object is obscured by the presence of one or more visual obstacles (obstructions) that may prevent the image sensor from capturing the entire object. Alternatively or additionally, a partially observed object may include an object in which at least a portion of the object is obscured (e.g., invisible) by the object itself. That is, the image sensor may not be able to capture the back (or far) side of a partially observed object that may be obscured and / or blocked by the front (or near) side of the object (e.g., when the object is opaque).

[0073] The partially observed objects of the scene representation may include a target object. As used herein, a target object may include an object included in a scene, which may be an intended target of a robot, on which an action (e.g., grasping, pushing, pulling, etc.) may be required.

[0074] The segmentation and point cloud embedding operation 220 may also include outputting a metric embedding of the segmented point cloud. For example, the segmentation and point cloud embedding operation 220 may include creating a metric embedding of the segmented point cloud corresponding to the partially observed object of the scene representation based on a combination of estimated semantic information of the partially observed object and geometric information extracted from the scene representation. Figure 5 and Figure 6 The segmentation and point cloud embedding operation 220 is further described.

[0075] The shape prediction evaluation process 200 may include predicting and outputting a shape completion (e.g., a point cloud) of a portion of an observed object in a scene representation in a shape completion operation 230. For example, the shape completion operation 230 may include predicting a shape point cloud of a completion of a portion of an observed object in a scene representation. Figure 5 The shape completion operation 230 is further described.

[0076] The uncertainty estimation operation 240 may be provided as input the metric embedding of the segmented point cloud created in the segmentation and point cloud embedding operation 220 and the shape-completed point cloud predicted in the shape completion operation 230, such as Figure 2 As shown. The uncertainty estimation operation 240 may include evaluating the uncertainty of the shape completion prediction and outputting an uncertainty (and / or confidence) score for each point in the shape completion point cloud. For example, the uncertainty estimation operation 240 may include using a probabilistic machine learning model to generate a confidence score for each point of the completed shape point cloud based on the correlation between the geometric information extracted from the scene representation and the semantic information of the partially observed object. Figure 5 and Figure 7 The uncertainty estimation operation 240 is further described.

[0077] As reference Figure 2As described, various aspects of the present disclosure can provide a probabilistic mapping method for scenes where only a partial view of an object is visible. As described herein, semantic information and shape completion information can be inferred from an image including a partial view, and therefore, semantic understanding and the completed shape of the observed object can be obtained from the partial view. In addition, a classification model with a metric embedding defined in a unit sphere metric space can be used to evaluate the reliability of these inferences. As a result, the semantic information and shape completion information can be unified using an evaluation value (e.g., a confidence score). Therefore, when compared to related methods, the various aspects provided herein can potentially achieve more accurate predictions and therefore provide better estimates of scene mapping, which can allow tasks to be performed in complex environments, even in the presence of occlusion and / or visual impairments.

[0078] Figure 3 Examples of robotic control systems according to various aspects of the present disclosure are depicted.

[0079] refer to Figure 3 , shows an example of a robot control system 300 implementing one or more aspects of the present disclosure. In an embodiment, at least a portion of the operation and / or functionality of the robot control system 300 may be implemented by Figure 1 The device 100 may include a shape prediction evaluation component 180. Alternatively or additionally, another computing device (e.g., a robotic device, a robotic controller, a server, a laptop, a smart phone, a UE, a wearable device, a smart device, an IoT device, etc.) including the shape prediction evaluation component 180 may perform at least a portion of the operations and / or functions of the robotic control system 300.

[0080] like Figure 3 As shown, the robot 310 can use the image sensor 320 to observe objects (e.g., the first object 330A, the second object 330B, the third object 330C, and the fourth object 330D, hereinafter generally referred to as "330"). For example, the robot 310 can capture one or more images representing a scene including the object 330. Figure 2 As described, the one or more images may include a red-green-blue (RGB) image, which may include depth information (eg, an RGBD image).

[0081] In an embodiment, an image sensor 320 (eg, a camera, a video camera, a depth camera, a TOF camera, a stereo camera, etc.) may be provided on the robot 310. For example, the robot 310 may be and / or may include Figure 1In such an example, the image sensor 320 may be mounted on a movable portion of the robot 310, such as, but not limited to, a robot arm (e.g., a robot arm having seven (7) degrees of freedom (7DOF)), a joint of the robot arm (e.g., a wrist of the robot arm), an end-effector of the robot 310, etc. In one embodiment, the device 100 may control the movement of at least one of the robot 310, the robot arm, and the end-effector of the robot 310 so that the robot 310 can perform an action (e.g., grab, push, pull, etc.) on at least one target object of the objects 330.

[0082] In an embodiment, the image sensor 320 may be provided on the robot 310, and the device 100 may be provided on an external device (e.g., a robot controller, a desktop computer, a computer server, a virtual machine, a network device, a mobile device, a wearable device, a smart device, an IoT device, and / or any other type of data processing device) provided separately from the robot 310. Figure 2 The image capture operation 210 may include receiving one or more images of a scene representation by an external device. Alternatively or additionally, the external device may send motion control parameters of at least one of the arm and the end effector of the robot 310 to the robot 310, which may cause the robot 310 to perform an action (e.g., grasping, pushing, pulling, etc.) based on the motion control parameters. However, the present disclosure is not limited to the location and / or mounting configuration of the image sensor and / or device 100. For example, in an embodiment, the image sensor 320 and the device 100 may be provided separately from the robot 310.

[0083] In an embodiment, a first portion of one or more images of a scene representation may be obtained at a first position, and a second portion of one or more images may be obtained at a second position. That is, in an embodiment, when the image sensor 320 is positioned at a first position (e.g., a first position and / or a first orientation), at least one first image of the one or more images may be obtained, and when the image sensor 320 is positioned at a second position (e.g., a second position and / or a second orientation), at least one second image of the one or more images may be obtained. For example, one or more images may be obtained from different geographic locations. Alternatively or additionally, one or more images may be obtained from different orientations of the image sensor 320 (e.g., different alignments of the image sensor relative to a particular direction). As another example, one or more images may be obtained from the same geographic location and / or orientation.

[0084] Continue to refer Figure 3, the robot 310 may observe an object 330 in its environment through an image sensor 320. In an embodiment, the robot 310 may infer a semantic segmentation of the object 330 based on geometric information extracted from one or more images. Alternatively or additionally, the robot 310 may determine a segmented point cloud of a partially observed object from the object 330 in the scene representation based on the semantic information and the geometric information, and / or the robot 310 may create a metric embedding of the segmented point cloud corresponding to the partially observed object. In an embodiment, the robot 310 may predict a completed shape point cloud of the partially observed object. The robot 310 may generate a confidence score for each point of the completed shape point cloud based on a correlation between the geometric information and the semantic information.

[0085] In an embodiment, the robot 310 may filter the shape predictions based on the confidence scores and / or use the filtered predictions to map the surroundings and / or environment of the robot 310. Alternatively or additionally, the robot 310 may use the filtered predictions to perform an action (e.g., grab, push, pull, etc.) on at least one target object in the objects 330.

[0086] In an embodiment, the robot 310 may filter the shape predictions based on different thresholds based on the importance (e.g., priority) and / or risk associated with the task to be performed. For example, the robot 310 may filter the shape predictions using a relatively high confidence threshold when performing a high priority and / or high risk task. Alternatively or additionally, the robot 310 may filter the shape predictions using a relatively low confidence threshold when performing a low priority and / or low risk task. As another example, the robot 310 may filter the shape predictions after having adjusted the confidence score based on at least one of an object attribute indicating whether the object is fragile and past robot experience indicating whether the robot 310 has previously handled the object.

[0087] For example, the robot 310 can perform a view planning process for its surrounding environment based on the filtered predictions, as shown in FIG. Figure 4 described.

[0088] Figure 4 An example of a view planning process according to various aspects of the present disclosure is shown.

[0089] refer to Figure 4 , showing an example of a view planning process 400 implementing one or more aspects of the present disclosure. In an embodiment, at least a portion of the operations and / or functions of the view planning process 400 may be performed by Figure 1 The device 100 and / or Figure 3The robot 310 may include the shape prediction evaluation component 180. Alternatively or additionally, another computing device (e.g., a robotic device, a robotic controller, a server, a laptop computer, a smart phone, a UE, a wearable device, a smart device, an IoT device, etc.) including the shape prediction evaluation component 180 may perform at least a portion of the operations and / or functions of the view planning process 400.

[0090] like Figure 4 As shown, the view planning process 400 may include filtering the completed shape point cloud of the partially observed object represented by the scene based on the confidence score 410 in operation 420. For example, the filtering may include filtering out points having corresponding confidence scores that fail to meet a threshold confidence score from the completed shape point cloud, resulting in filtered points of the completed shape point cloud.

[0091] In operation 430, the view planning process 400 may include finding a new camera position based on the filtered points of the completed shape point cloud. For example, the view planning process 400 may include determining a new camera position based on the filtered points of the completed shape point cloud, obtaining at least one other image of the scene representation from the new camera position, the at least one other image including at least one partially observed portion of a partially observed object of the scene representation. As another example, operation 430 may include identifying a region of the target object having a plurality of data points in the completed shape point cloud having corresponding confidence scores below a predetermined threshold, and determining a new camera position from which at least one other image of the region was captured. In an embodiment, operation 430 may include finding a new robot arm position based on geometric information and inverse kinematic information of the robot arm (e.g., when the image sensor 320 is provided in the robot arm of the robot 310).

[0092] In an embodiment, the view planning process 400 may include adjusting the threshold confidence score based on an updated confidence score of at least one other image.

[0093] The view planning process 400 may include, in operation 440, controlling the robot 310 to move the robot 310 (or a mechanical arm of the robot 310 or an end effector of the robot 310) to a new camera position when the image sensor 320 is provided on the robot 310. Alternatively or additionally, operation 440 may include moving the image sensor 320 to a new camera position when the image sensor 320 is provided separately from the robot 310. In an embodiment, the robot 310 may capture at least one other image of the scene from the new camera position. Thus, the robot 310 may increase its understanding of the scene (e.g., a higher confidence score) based on the at least one other image captured at the new camera position. In an embodiment, the robot 310 may perform an action (e.g., grasp, push, pull, etc.) on at least one target object at the new camera position.

[0094] In one or more aspects of the present disclosure, Figure 1 The device 100 and / or Figure 3 The robot 310 can perform the view planning process 400 one or more times to obtain a semantic and geometric understanding of its surrounding environment, which can be used to perform tasks and / or actions in complex environments even in the presence of occlusion and / or visual obstructions. For example, suppose the target application consists of a camera mounted on the wrist of a home assistant robot, and the robot can move its arm to observe a given environment. As the robot moves, the robot can build a map that provides semantic information of observed objects, shape completion predictions of the objects, and uncertainty in the predictions, as described above. When compared to related mapping methods, the robot can use such completion information of the scene to make improved planning decisions.

[0095] Having described in general terms the process flow and possible use cases for evaluating the reliability of 3D shape predictions, we now refer to Figures 5 to 7 Various embodiments are discussed in greater detail.

[0096] Figure 5 Depicted are examples of data pipelines for evaluating the reliability of 3D shape predictions in accordance with various aspects of the present disclosure.

[0097] refer to Figure 5 , shows a data pipeline 500 for evaluating the reliability of 3D shape predictions implementing one or more aspects of the present disclosure. In an embodiment, at least a portion of the operations and / or functions depicted by the data pipeline 500 may be implemented by Figure 1 The device 100 and Figure 3The data pipeline 500 may be executed by at least one of the robots 310, which may include the shape prediction evaluation component 180. Alternatively or additionally, another computing device (e.g., a robotic device, a robotic controller, a server, a laptop, a smart phone, a UE, a wearable device, a smart device, an IoT device, etc.) including the shape prediction evaluation component 180 may execute at least a portion of the data pipeline 500.

[0098] In an embodiment, Figure 5 The data pipeline 500 depicted in FIG. 5 can be used to implement the reference Figures 2 to 4 At least a portion of at least one of the shape prediction evaluation process 200, the robotic control system 300, and the view planning process 400 are described, and may include additional features not mentioned above.

[0099] like Figure 5 As shown, the data pipeline 500 may include a segmentation component 520 , a metric embedding component 530 , a shape completion component 540 , and a Gaussian process (GP) inference component 550 .

[0100] In an embodiment, the data pipeline 500 may take as input one or more images 510 representing a scene. The one or more images 510 may include image data 512 (e.g., an RGB image) and depth data 514 corresponding to the image data 512. For example, the one or more images 510 may include a 2D RGB image associated with and / or including the depth data 514. In an embodiment, the one or more images 510 may be composed of Figure 1 The input component 150 of the device 100 can capture. For example, in one embodiment, the input component 150 can include at least one of a camera, a video camera, a depth camera, a TOF camera, a stereo camera, etc. that can capture and / or provide one or more images of a scene representation. That is, the present disclosure is not limited in this regard, and the various aspects presented herein can be applied with other types of image sensors. It is worth noting that the various aspects presented herein can be used with any image sensor that can provide image data and geometric information associated with the image data.

[0101] The segmentation component 520 may be configured to extract geometric information from the image data 512 and the depth data 514 of the one or more images 510. In an embodiment, the geometric information may include 3D shape geometric information of an object observed in the scene representation. Alternatively or additionally, at least one of the objects observed in the scene representation may be referred to as a partially observed object. As used herein, a partially observed object may include an object included in the scene representation that may not be fully visible to the image sensor. For example, a partially observed object may include an object in which at least a portion of the object is obscured by the presence of one or more visual obstacles that may prevent the image sensor from capturing the entire object. Alternatively or additionally, a partially observed object may include an object in which at least a portion of the object is obscured (e.g., invisible) by the object itself. That is, the image sensor may not be able to capture the rear (or far) side of a partially observed object that may be obscured and / or blocked by the front (or near) side of the object (e.g., when the object is opaque).

[0102] The partially observed objects of the scene representation may include a target object. As used herein, a target object may include an object included in a scene, which may be an intended target of a robot, on which an action (e.g., grasping, pushing, pulling, etc.) may be required.

[0103] In an embodiment, the segmentation component 520 may estimate semantic information of partially observed objects in a scene represented by the one or more images 510 based on geometric information extracted from the one or more images 510. For example, the segmentation component 520 may use the geometric information to estimate semantic information of each pixel of the one or more images 510. That is, the segmentation component 520 may estimate the semantic information of objects (e.g., Figure 3 Semantic information 522 of the object 330).

[0104] In one embodiment, one or more images 510 (e.g., image data 512) may be provided to a segmentation network (or model), which may be configured to provide spatial information (e.g., a bounding box) for each object and / or a segmentation class label for each object, as well as a corresponding confidence score for the predicted segmentation class. That is, in addition to the semantic information 522, the segmentation network and / or model may determine an estimated confidence score for the semantic information 522 for each pixel of the one or more images 510. In an embodiment, the segmentation network (or model) may be and / or may include a mask region-based convolutional neural network (Mask R-CNN). However, the present disclosure is not limited in this regard, and other networks and / or models may be used without departing from the scope of the present disclosure. It is noteworthy that the various aspects presented herein may be used in conjunction with any segmentation network and / or model capable of providing segmentation information from image data.

[0105] The segmentation component 520 can be configured to combine the semantic information 522 with the depth data 514 to obtain a segmented point cloud 524 of the object 330 in the scene represented by the one or more images 510. That is, the segmentation component 520 can determine the segmented point cloud 524 of the partially observed object in the scene representation by combining the semantic information 522 and geometric information (such as, but not limited to, the depth data 514). In one embodiment, the segmentation component 520 can generate a point cloud from the image data 512 and the corresponding depth data 514. In an embodiment, obtaining the segmented point cloud 524 can include combining the point cloud with the semantic information 522 for each pixel in the one or more images 510 and the confidence score of the predicted segmentation class.

[0106] like Figure 5 As shown, the segmentation component 520 may include a semantic map 526. For example, the segmentation component 520 may be configured to maintain a semantic map 526 of the segmented point cloud 524, which includes semantic information 522 of the partially observed objects represented by the scene. The semantic map 526 may be configured to maintain a semantic class assigned to each of the partially observed objects. Alternatively or additionally, the semantic map 526 may be configured to maintain relevant semantic information, such as an object class, an instance identification (ID), and a confidence score for each voxel belonging to an observed object (e.g., any one of the objects 330 including the partially observed object), as shown in Table 1.

[0107]

Table 1

[0108] serial number property 1 Position (e.g., x, y, z coordinates) 2 Color (e.g., red (R), green (G), blue (B)) 3 Semantic class (mode of the histogram) 4 Credibility Score 5 Object ID

[0109] Table 1: Attributes stored in each voxel of the semantic map.

[0110] In an embodiment, the semantic graph 526 can be implemented using an OctoMap library, which can use an OctTree data structure. However, the present disclosure is not limited in this regard. That is, the semantic graph 526 can be implemented using other data structures and / or libraries.

[0111] In one embodiment, the segmentation component 520 may include a tracker 528 that can help retrieve semantic information. For example, the segmentation component 520 may use the tracker 528 to use the semantic information 522 to track a partially observed object between a first image of one or more images 510 and a second image of one or more images 510. That is, the tracker 528 may be configured to maintain a consistent object ID on the observed object unless the object is severely cluttered. For example, if there is a significant camera view change between one or more images 510, the tracker 528 can help recover the semantic information of the object. As another example, when the movement of the robot (e.g., robot 310) and / or the image sensor (e.g., image sensor 320) causes no overlapping observations between two sequential views of one or more images 510, the tracker 528 can help recover the semantic information. Related image-based object trackers can rely on overlapping information between different views, and therefore, if the views do not share any visual cues, tracking information may be lost. Therefore, according to various aspects of the present invention, the tracker 528 can be and / or can include a 3D geometry-based tracker that can overcome such limitations.

[0112] In one embodiment, the tracker 528 of the segmentation component 520 can calculate the corresponding centroids of the point cloud of the object 330 observed in the scene. The tracker 528 can be configured to determine the Euclidean distance between the new centroid and the existing centroid (e.g., attribute no. 1 of Table 1) in order to update the centroid set (e.g., delete, add, modify the centroid). Alternatively or additionally, the tracker 528 can assign an object ID to each centroid in the centroid set and store the information in the semantic map 526 together with the semantic information 522 in each voxel of the segmented point cloud 524.

[0113] In an embodiment, the tracker 528 may maintain a histogram for each semantic class in a voxel map to resolve temporary mispredictions that may be caused by misclassification and / or noise from the image sensor 320. For example, when semantic information is aggregated from different views, instabilities in segmentation predictions may result in conflicts in semantic information for each voxel. To resolve these conflicts, the tracker 528 may maintain a histogram for each semantic class at the voxel level, and / or use the mode of the histogram as the semantic class of the voxel. Thus, the semantic map may be resilient to temporary mispredictions from the segmentation model and / or noise from the image sensor 320.

[0114] Continue to refer Figure 5 The segmented point cloud 524 of the partially observed object in the scene representation determined by the segmentation component 520 can be provided to the metric embedding component 530. The metric embedding component 530 can be configured to create a metric embedding 535 of the segmented point cloud 524 corresponding to the partially observed object in the scene representation.

[0115] For example, the metric embedding component 530 may provide the segmented point cloud 524 to the transformer model to obtain an embedding of the segmented point cloud 524. In an embodiment, the embedding of the segmented point cloud 524 may include semantic information 522 and geometric information of the segmented point cloud 524. In an embodiment, the metric embedding component 530 may provide the embedding of the segmented point cloud 524 to the perceptron model to obtain a metric embedding 535 of the segmented point cloud 524 that may have been forced into the unit sphere metric space. Alternatively or additionally, the metric embedding component 530 may assign zero embeddings to free space in the metric embedding 535 of the segmented point cloud 524. Reference Figure 6 The metric embedding component 530 is further described.

[0116] like Figure 5 As shown, data pipeline 500 can provide image data 512 and / or depth data 514 of one or more images 510 to shape completion component 540. Shape completion component 540 can be configured to predict a completed shape point cloud of a partially observed object of a scene represented by one or more images 510.

[0117] As used herein, shape completion may refer to predicting the shape of an object when only a partial observation of the object is available. Related shape completion methods may focus on predicting the shape of an object based on an object frame. However, such object frame-based shape completion methods may require additional pose information of the object. That is, it may be necessary to capture multiple poses of the object (e.g., from different positions and / or orientations) in order to obtain sufficient pose information to predict the complete shape of the object. According to various aspects of the present disclosure, a shape completion component 540 may provide one or more images 510 including image data 512 and depth data 514 to a shape completion model that is capable of overcoming such limitations and capable of predicting a completed shape point cloud 545 from a single view (or pose).

[0118] In an embodiment, the shape completion model may be and / or may include a CenterSnap model that may complete the shape of an object in a world frame. However, the present disclosure is not limited in this regard, and other networks and / or models may be used without departing from the scope of the present disclosure. It is noteworthy that the various aspects presented herein may be employed with any shape completion network and / or model capable of providing a completed shape point cloud from an image and / or depth data containing a single view.

[0119] The shape completion model may receive image data 512 and depth data 514 as input. The shape completion model may have been trained using a publicly available dataset (such as, but not limited to, a Normalized Object Coordinate Space (NOCS) dataset). However, the present disclosure is not limited in this regard, and the shape completion model may be trained using other datasets containing multiple object classes (e.g., six (6)). The shape completion model may be configured to output a 3D shape reconstruction (e.g., a point cloud) of an object (e.g., object 330) in a scene represented by one or more images 510. The shape completion model may also be configured to output a classified six (6) degrees of freedom (6DOF) pose (e.g., 3D orientation and position) and / or size estimate of the object in the scene. For example, the shape completion model may output a point cloud and bounding box of the shape completion in a camera frame. That is, the shape reconstruction may be mapped to the coordinate system of the image sensor 320, which may have been used to obtain the one or more images 510.

[0120] In an embodiment, the shape completion component 540 may be configured to transform the shape reconstruction from the coordinate system of the image sensor 320 to the world coordinate system based on the physical configuration information of the image sensor 320. For example, the physical configuration information of the image sensor 320 may include position information of the image sensor relative to the world coordinate system (e.g., the base of the robot arm of the robot 310). Although the completed shape point cloud 545 provided by the shape completion component 540 can be used to predict unobserved portions of objects in the scene, the completed shape point cloud 545 may not provide a method for evaluating the reliability (e.g., confidence) of each predicted point in the completed shape point cloud 545. To this end, the shape completion component 540 may provide the completed shape point cloud 545 that has been transformed into the world coordinate system to the Gaussian process (GP) inference component 550.

[0121] The Gaussian process (GP) inference component 550 can be configured to generate a confidence score 555 for each point of the completed shape point cloud 545 based on the correlation between the geometric information and the semantic information 522 extracted from the one or more images 510 using a probabilistic machine learning model such as a GP model. The GP model can define or use a distribution on a function that explains the observed data, wherein the distribution can be characterized by a mean function and a covariance function of the distribution. In one embodiment, the GP model may have been conditioned (trained) using training data 537, which may include a 3D metric embedding 535 of the segmented point cloud 524 created by the metric embedding component 530 and semantic information 529 contained in the semantic map 526 (e.g., a semantic class label, confidence score, and object ID for each voxel). That is, the GP model can be trained in a high-dimensional embedding space and conditioned with a partial point cloud of an object in the scene and its semantic information. Therefore, unlike other related robotic mapping methods, the GP model can be generalized to input modalities and / or may not be constrained by the complexity of the object semantic class or the object geometry. As a result, the GP model can provide an improved confidence score for the completed shape point cloud when compared to related robotic mapping methods. Although the Gaussian process (GP) inference component 550 using the GP model is described as an example, various embodiments of the present disclosure are not limited thereto, and another probabilistic machine learning model such as a Bayesian neural network model can be used instead of the GP model. Figure 7 The GP inference component 550 is further described.

[0122] Figure 6 Examples of metric embedding components according to various aspects of the present disclosure are shown.

[0123] refer to Figure 6, FIG. 600 illustrates an embodiment of one or more aspects of the present disclosure. Figure 5 In an embodiment, at least a portion of the operations and / or functions described as being performed by the metric embedding component 530 may be performed by Figure 1 Device 100 or Figure 3 The robot 310 may include the shape prediction evaluation component 180. Alternatively or additionally, another computing device (e.g., a robotic device, a robotic controller, a server, a laptop, a smart phone, a UE, a wearable device, a smart device, an IoT device, etc.) including the shape prediction evaluation component 180 may perform at least a portion of the operations and / or functions of the metric embedding component 530.

[0124] In an embodiment, Figure 6 The metric embedding component 530 depicted in FIG. 5 can be used to implement the reference Figures 2 to 5 At least a portion of at least one of the shape prediction evaluation process 200, the robotic control system 300, the view planning process 400, and the data pipeline 500 are described, and may include additional features not mentioned above.

[0125] like Figure 6 As shown in , the metric embedding component 530 can be provided as input with a segmented point cloud 524 representing a portion of an observed object in one or more images 510 of a scene. Figure 5 As depicted, segmented point cloud 524 may have been obtained by combining semantic information 522 and geometric information (such as, but not limited to, depth data 514 ).

[0126] The metric embedding component 530 can provide the segmented point cloud 524 to the transformer model 532 to obtain an initial embedding 533 of the segmented point cloud 524. In an embodiment, the transformer model 532 can be and / or can include a point cloud segmenter model, such as, but not limited to, a dynamic graph convolutional neural network (DGCNN). However, the present disclosure is not limited in this regard, and other networks and / or models can be used without departing from the scope of the present disclosure. It is worth noting that the various aspects presented herein can be employed with any point cloud segmentation network and / or model capable of providing an embedding from a segmented point cloud.

[0127] In an embodiment, the metric embedding component 530 can be configured to provide the segmented point cloud 524 to a transformer model 532 (e.g., DGCNN) and obtain a seven (7) dimensional (7D) initial embedding 533 from the last dense layer of the transformer model 532. As used herein, an embedding may include an intermediate result obtained from a neural network before the last layer of the network makes a prediction. In an embodiment, the intermediate result may be referred to as an embedding because the result embeds information, such as, but not limited to, semantic information and geometric information.

[0128] The metric embedding component 530 can provide the initial embedding 533 to the perceptron model 534 to obtain a 3D metric embedding 535 of the segmented point cloud 524. The perceptron model 534 can be configured to enforce the embedding to the unit sphere metric space. In an embodiment, the perceptron model 534 can be and / or can include a multi-layer perceptron (MLP) model. However, the present disclosure is not limited in this regard, and other networks and / or models can be used without departing from the scope of the present disclosure. It is worth noting that the various aspects presented herein can be used with any network and / or model capable of enforcing an embedding to the unit sphere metric space.

[0129] In an embodiment, the perceptron model 534 may be composed of four dense layers (e.g., (7×250), (250×100), (100×50), and (50×3)). Alternatively or additionally, the perceptron model 534 may be trained with an angular softmax (A-Softmax) loss, which may force the perceptron model 534 to generate an embedding on a unit sphere. As a result, the metric embedding component 530 may yield a 3D metric embedding 535 of the point cloud 524 for segmentation. In an embodiment, the free space may not have any geometric shape, and therefore, its embedding may not be learned. Therefore, the perceptron model 534 may explicitly assign a zero embedding to the free space. Alternatively or additionally, in order to make the object embedding equidistant to the free space embedding, the perceptron model 534 may constrain the object embedding to be on the surface of the unit sphere.

[0130] As referenced above Figure 5 As described, the 3D metric embedding 535 and the semantic information 529 can be used to train the GP inference component 550 before the GP inference component 550 is used to generate a confidence score 555 for each point of the completed shape point cloud 545 generated by the shape completion component 540 .

[0131] Figure 7 Depicted are examples of GP inference components in accordance with various aspects of the present disclosure.

[0132] refer to Figure 7 , FIG. 700 depicts an embodiment of one or more aspects of the present disclosure. Figure 5 In an embodiment, at least a portion of the operations and / or functions described as being performed by the GP inference component 550 may be performed by Figure 1 Device 100 or Figure 3 The robot 310 may include the shape prediction evaluation component 180. Alternatively or additionally, another computing device (e.g., a robotic device, a robotic controller, a server, a laptop computer, a smart phone, a UE, a wearable device, a smart device, an IoT device, etc.) including the shape prediction evaluation component 180 may perform at least a portion of the operations and / or functions of the GP inference component 550.

[0133] In an embodiment, Figure 7 The GP inference component 550 depicted in FIG. 5 can be used to implement the reference Figures 2 to 5 At least a portion of at least one of the shape prediction evaluation process 200, the robotic control system 300, the view planning process 400, and the data pipeline 500 are described, and may include additional features not mentioned above.

[0134] The GP inference component 550 can be and / or can include a GP model. The GP model can capture the underlying structure from the training data points (e.g., the 3D metric embedding 535 and semantic information 529 of the segmented point cloud 524), and use the captured structure to predict the test data points (e.g., the completed shape point cloud 545). However, the computational cost of performing these predictions may be too high, especially when the size of the data is relatively large (e.g., exceeding a certain threshold). In order to potentially address these limitations, a GP acceleration library can be used to allow a processor (e.g., a GPU) to run GP inference at a relatively high speed. For example, the GP acceleration library can be and / or can include GPytorch, which is an open source library that uses Pytorch and the Compute Unified Device Architecture (CUDA) to enable GPUs to run relatively fast GP inferences. However, the present disclosure is not limited in this regard, and other GP acceleration libraries can be used without departing from the scope of the present disclosure.

[0135] In an embodiment, the GP model may be and / or may include a Dirichlet GP classification model, which may solve the classification problem by transforming the class labels using the Dirichlet distribution. However, the present disclosure is not limited in this regard, and other networks and / or models may be used without departing from the scope of the present disclosure. It is noteworthy that the various aspects presented herein may be employed with any GP-based network and / or model capable of generating confidence scores for a completed shape point cloud and conditioning on the metric embedding and semantic information of the segmented point cloud.

[0136] The GP model can be constructed using training points obtained in a learned metric embedding space, such as the 3D metric embedding 535 created by the metric embedding component 530. In an embodiment, the 3D metric embedding 535 can be appended to the relevant points, resulting in 6-dimensional (6D) training data (e.g., x, y, z, e 1 、e 2 and e 3 ). Alternatively or additionally, the covariance matrix K of the GP model can be expressed as an equation similar to Equation 1.

[0137] [Equation 1]

[0138]

[0139] Referring to Equation 1, X can represent a training data point, and X * For example, during the training phase of the GP model, the K(X,X) matrix can be embedded 535 using the 3D metric of the training points (e.g., e 1 、e 2 and e 3 Alternatively or additionally, during the inference phase of the GP model, K(X,X * ) matrix can be calculated using the 3D coordinates (e.g., x, y, and z) of both the training points and the test points. In one embodiment, at least some of the test points may be unobserved (e.g., occluded and / or blocked), and thus, a priori information (e.g., color, cluster shape, etc.) about the test points may not be available and / or known, except for their predicted geometric coordinates. Therefore, the 3D coordinates of the training points may be used to obtain the clustering information via K(X, X * ) matrix to find the correlation between the training test points and the test points. In a similar way, K(X * ,X * ) matrix can be calculated using the 3D coordinates (e.g., x, y, and z) of the test points.

[0140] During the inference phase, the GP model can provide the variance σ as well as the prediction. In one embodiment, the GP inference component 550 can be configured to calculate a raw confidence score s for each prediction by measuring the variance reduction. The raw confidence score s can be expressed as an equation similar to Equation 2.

[0141] [Equation 2]

[0142]

[0143] Alternatively or additionally, the GP inference component 550 may normalize the raw confidence score s to obtain a normalized score ranging between zero (0) and one (1) The normalized score can be calculated using an equation similar to Equation 3

[0144] [Equation 3]

[0145]

[0146] In an embodiment, the GP inference component 550 may be configured to normalize the scores by The confidence score given by the segmentation network of the segmentation component 520 is multiplied to calculate the final predicted confidence score 555 of the shape-completed point cloud and its associated semantic information. For example, the GP inference component 550 can obtain the segmentation confidence score from the semantic information 529 provided to the GP inference component 550. Advantageously, the above reference Figures 1 to 7 The described apparatus and process for evaluating the reliability of 3D shape predictions can provide a probabilistic method for robotic mapping that unifies semantic information and shape completion predictions obtained from partial view images and calculates confidence scores for the shape completion predictions. That is, the various aspects presented herein provide for transferring sensor measurements to a learned metric space for training a GP model. Alternatively or additionally, the trained GP model can be used to measure the uncertainty of the unified information and make more accurate predictions when compared to related GP models that may not use the unified information.

[0147] Figure 8 800 is a block diagram of an example apparatus for evaluating the reliability of 3D shape prediction and / or controlling robot motion according to various aspects of the present disclosure. Apparatus 800 may be a computing device (e.g., Figure 1The device 100) and / or the computing device may include the apparatus 800. In one embodiment, the apparatus 800 may include a receiving component 802 configured to receive communications (e.g., wired, wireless) from another apparatus (e.g., apparatus 808), a shape prediction evaluation component 180 configured to evaluate the reliability of the 3D shape prediction, and a sending component 806 configured to send communications (e.g., wired, wireless) to another apparatus (e.g., apparatus 808). The components of the apparatus 800 may communicate with each other (e.g., via one or more buses or electrical connections). Figure 8 As shown, the device 800 can communicate with another device 808 (such as, but not limited to, a robotic device, a robotic controller, a server, a laptop, a smart phone, a UE, a wearable device, a smart device, an IoT device, etc.) using a receiving component 802 and / or a sending component 806.

[0148] In one embodiment, the apparatus 800 may be configured to perform the Figures 1 to 7 Alternatively or additionally, the apparatus 800 may be configured to perform one or more processes described herein, such as Fig. 9 Method 900 and / or Fig.10 In one embodiment, the apparatus 800 may include a reference Figure 1 One or more components of device 100 are described.

[0149] The receiving component 802 can receive communications, such as control information, data communications, or a combination thereof, from an apparatus 808 (e.g., a robotic device, a robotic controller, a server, a laptop, a smartphone, a UE, a wearable device, a smart device, an IoT device, etc.). The receiving component 802 can provide the received communications to one or more other components of the apparatus 800, such as the shape prediction evaluation component 180. In one embodiment, the receiving component 802 can perform signal processing on the received communications and can provide the processed signals to one or more other components. In one embodiment, the receiving component 802 can include a reference Figure 1 One or more antennas, a receive processor, a controller / processor, a memory, or a combination thereof of the described device 100.

[0150] The sending component 806 can send communications, such as control information, data communications, or a combination thereof, to an apparatus 808 (e.g., a robotic device, a robotic controller, a server, a laptop, a smartphone, a UE, a wearable device, a smart device, an IoT device, etc.). In one embodiment, the shape prediction evaluation component 180 can generate communications and can send the generated communications to the sending component 806 for transmission to the apparatus 808. In one embodiment, the sending component 806 can perform signal processing on the generated communications and can send the processed signals to the apparatus 808. In one embodiment, the sending component 806 can include a reference Figure 1 One or more antennas, transmit processors, controllers / processors, memory, or a combination thereof of the described device 100. In one embodiment, the transmitting component 806 can be co-located with the receiving component 802, such as in a transceiver and / or transceiver component.

[0151] The shape prediction evaluation component 180 may be configured to evaluate the reliability of the 3D shape prediction. In an embodiment, the shape prediction evaluation component 180 may include a set of components, such as an acquisition component 810 configured to obtain a scene representation, an estimation component 820 configured to estimate semantic information of a partially observed object, a determination component 830 configured to determine a segmented point cloud of the partially observed object, a creation component 840 configured to create a metric embedding of the segmented point cloud, a prediction component 850 configured to predict a completed shape point cloud of the partially observed object, a generation component 860 configured to generate a confidence score of the completed shape point cloud, and a control component 870 configured to control the motion of the robot.

[0152] In one embodiment, the set of components may be separate and distinct from the shape prediction evaluation component 180. In one embodiment, one or more components in the set of components may include the above referenced Figure 1 The described device 100 may be implemented in or within a controller / processor (e.g., processor 120), memory (e.g., memory 130), or a combination thereof of the device 100. Alternatively or additionally, one or more of the set of components may be at least partially implemented as software stored in a memory (such as memory 130). For example, a component (or a portion of a component) may be implemented as computer-executable instructions or code stored in a computer-readable medium (e.g., a computer-readable medium) and executable by a controller or processor to perform the function or operation of the component.

[0153] Figure 8 The number and arrangement of components shown in FIG. 1 are provided as examples. In practice, Figure 8There may be additional components, fewer components, different components, or differently arranged components than those shown in . Figure 8 Two or more components shown in the figure may be implemented in a single component, or Figure 8 The single component shown in can be implemented as multiple distributed components. Alternatively or additionally, Figure 8 The set of (one or more) components shown in the figure may perform the operations described as being performed by Figures 1 to 7 One or more functions performed by another set of components shown in FIG.

[0154] refer to Fig. 9 , in operation, the apparatus 800 may perform a method 900 for controlling the motion of a robot. The method 900 may be performed by at least one of the device 100 (which may include a processor 120, a memory 130, and a storage component 140, and which may be the entire device 100 and / or include one or more components of the device 100, such as an input component 150, an output component 160, a communication interface 170, and / or a shape prediction evaluation component 180), the robot 310, and / or the apparatus 800. The method 900 may be performed by the apparatus 100, the robot 310, the apparatus 800, and / or a shape prediction evaluation component 180 in communication with a device 808 (e.g., a robot device, a robot controller, a server, a laptop, a smart phone, a UE, a wearable device, a smart device, an IoT device, etc.).

[0155] exist Fig. 9 At block 910 of , method 900 may include obtaining a scene representation including one or more images, each of the one or more images including depth information. For example, in one aspect, the device 100, the robot 310, the shape prediction evaluation component 180, and / or the obtaining component 810 may be configured as or may include means for obtaining a scene representation including one or more images 510, each of the one or more images including depth information 514.

[0156] For example, the obtaining at block 910 may include obtaining one or more images 510 from a first camera position. In an embodiment, the obtaining at block 910 may further include filtering out points having corresponding confidence scores 555 that fail to satisfy a threshold confidence score from the completed shape point cloud 545, resulting in filtered points of the completed shape point cloud 545. The obtaining at block 910 may further include determining a second camera position different from the first camera position based on the filtered points of the completed shape point cloud 545, obtaining at least one other image of the scene representation from the second camera position, the at least one other image including at least one partially observed portion of a partially observed object of the scene representation.

[0157] In one embodiment, the obtaining at block 910 may include adjusting the threshold confidence score based on an updated confidence score of at least one other image.

[0158] exist Fig. 9 At block 920 of , method 900 may include estimating semantic information of a partially observed object in the scene representation based on geometric information extracted from the one or more images. For example, in one aspect, the device 100, the robot 310, the shape prediction evaluation component 180, and / or the estimation component 820 may be configured as or may include means for estimating semantic information 522 of a partially observed object in the scene representation based on geometric information extracted from the one or more images 510.

[0159] For example, the estimation at block 920 may include providing one or more images 510 (e.g., image data 512) to a segmentation network (or model), which may be configured to provide spatial information (e.g., a bounding box) for each object and / or a segmentation class label for each object, as well as a corresponding confidence score for the predicted segmentation class, as described in reference Figure 5 as described.

[0160] exist Fig. 9 At block 930 of , method 900 may include determining a segmented point cloud of a partially observed object in the scene representation based on the semantic information and the geometric information. For example, in one aspect, the device 100, the robot 310, the shape prediction evaluation component 180, and / or the determination component 830 may be configured as or may include means for determining a segmented point cloud 524 of a partially observed object in the scene representation based on the semantic information 522 and the geometric information.

[0161] For example, the determination at block 930 may include extracting geometric information from the image data 512 and the depth information 514 of the one or more images 510 .

[0162] In an embodiment, the determination at block 930 may include estimating semantic information 522 for each pixel of the one or more images 510 using geometric information.

[0163] In an embodiment, the determination at block 930 may include determining an estimated confidence score for the semantic information 522 for each pixel of the one or more images 510 .

[0164] In an embodiment, the determination at block 930 may include combining the semantic information 522 and the depth information 514 , resulting in a segmented point cloud 524 .

[0165] In an embodiment, the determination at block 930 may include maintaining a semantic map 526 of the segmented point cloud 524 that includes semantic information 522 of partially observed objects of the scene representation. The semantic information may include a semantic class assigned to each partially observed object.

[0166] In an embodiment, the determination at block 930 may include tracking, with the tracker 528 , the partially observed object between a first image in the one or more images 510 and a second image in the one or more images 510 using the semantic information 522 .

[0167] exist Fig. 9 At block 940 of , method 900 may include creating a metric embedding of the segmented point cloud corresponding to the partially observed objects of the scene representation. For example, in one aspect, the device 100, the robot 310, the shape prediction evaluation component 180, and / or the creation component 840 may be configured as or may include means for creating a metric embedding 535 of the segmented point cloud 524 corresponding to the partially observed objects of the scene representation.

[0168] For example, the creating at block 940 may include providing the segmented point cloud to the transformer model to obtain an embedding of the segmented point cloud. The embedding of the segmented point cloud may include semantic information and geometric information of the segmented point cloud.

[0169] In an embodiment, the creating at block 940 may include providing the embedding of the segmented point cloud to a perceptron model to obtain a metric embedding of the segmented point cloud. The metric embedding of the segmented point cloud has been forced into a unit sphere metric space.

[0170] In an embodiment, the creating at block 940 may include assigning zero embeddings to free spaces in the metric embedding of the segmented point cloud.

[0171] exist Fig. 9 At block 950 of , method 900 may include predicting a shape point cloud of a completion of a partially observed object represented by the scene. For example, in one aspect, the device 100, the robot 310, the shape prediction evaluation component 180, and / or the prediction component 850 may be configured as or may include means for predicting a shape point cloud 545 of a completion of a partially observed object represented by the scene.

[0172] For example, predicting at block 950 may include providing one or more images including image data and depth information to a shape completion model.

[0173] In an embodiment, the prediction at block 950 may include obtaining a shape reconstruction of a partially observed object of the scene representation from the shape completion model.The shape reconstruction may be mapped to a coordinate system of an image sensor used to obtain the one or more images.

[0174] In an embodiment, the prediction at block 950 may include transforming the shape reconstruction from a coordinate system of the image sensor to a world coordinate system based on physical configuration information of the image sensor.

[0175] exist Fig. 9 At block 960 of , method 900 may include generating a confidence score for each point of the completed shape point cloud using a probabilistic machine learning model based on the correlation between the geometric information and the semantic information. For example, in one aspect, the device 100, the robot 310, the shape prediction evaluation component 180, and / or the generation component 860 may be configured as or may include means for generating a confidence score 555 for each point of the completed shape point cloud 545 using a probabilistic machine learning model 550 based on the correlation between the geometric information and the semantic information 522.

[0176] For example, the generation at box 960 can include training the probabilistic machine learning model 550 using training data including the metric embedding 535 of the segmented point cloud 524.

[0177] In an embodiment, the probabilistic machine learning model 550 may be a Gaussian process model, and the partially observed objects of the scene representation may include the target object. In an embodiment, the generating at block 960 may include generating a confidence score 555 based on at least one of an object property indicating whether the target object is fragile and past robot experience indicating whether the robot has previously handled the target object.

[0178] exist Fig. 9 At block 970 of , the method 900 may include controlling the motion of the robot based on the confidence score of each point of the completed shape point cloud. For example, in one aspect, the apparatus 100, the robot 310, the shape prediction evaluation component 180, and / or the control component 870 may be configured as or may include means for controlling the motion of the robot based on the confidence score of each point of the completed shape point cloud.

[0179] In an embodiment, the processor 120 may be included in the robot 310, and the partially observed objects of the scene representation include the target object. In an embodiment, the control at block 970 may include controlling at least one of an arm and an end effector of the robot 310 to enable the robot 310 to grasp the target object.

[0180] In an embodiment, the processor 120 may be included in an external device that may be provided separately from the robot 310, and the partially observed objects represented by the scene may include the target object. In an embodiment, the control at block 970 may include sending motion control parameters of at least one of an arm and an end effector of the robot 310 to the robot 310, so that the robot 310 grasps the target object based on the motion control parameters.

[0181] In an optional or additional aspect that may be combined with any other aspect, the partially observed objects of the scene representation may include a target object. In such an aspect, the method 900 may also include identifying a region of the target object having a plurality of data points in the completed shape point cloud having corresponding confidence scores below a predetermined threshold, and controlling the robot 310 to capture at least one other image of the region of the target object.

[0182] refer to Fig.10 In operation, the apparatus 800 may perform a method 1000 for evaluating the reliability of a 3D shape prediction. The method 1000 may be performed by at least one of the device 100 (which may include a processor 120, a memory 130, and a storage component 140, and which may be the entire device 100 and / or include one or more components of the device 100, such as an input component 150, an output component 160, a communication interface 170, and / or a shape prediction evaluation component 180), the robot 310, and / or the apparatus 800. The method 1000 may be performed by the apparatus 100, the robot 310, the apparatus 800, and / or a shape prediction evaluation component 180 in communication with a device 808 (e.g., a robotic device, a robotic controller, a server, a laptop, a smart phone, a UE, a wearable device, a smart device, an IoT device, etc.).

[0183] exist Fig.10 At block 1010 of , method 1000 may include creating a metric embedding of a segmented point cloud corresponding to a partially observed object of the scene representation based on estimated semantic information of the partially observed object and geometric information extracted from the scene representation. For example, in one aspect, the device 100, the robot 310, the shape prediction evaluation component 180, and / or the creation component 840 may be configured as or may include means for creating a metric embedding of a segmented point cloud corresponding to a partially observed object of the scene representation based on a combination of estimated semantic information of the partially observed object and geometric information extracted from the scene representation.

[0184] For example, the creating at block 1010 may include providing the segmented point cloud to the transformer model to obtain an embedding of the segmented point cloud. The embedding of the segmented point cloud may include semantic information and geometric information of the segmented point cloud.

[0185] In an embodiment, the creating at block 1010 may include providing the embedding of the segmented point cloud to a perceptron model to obtain a metric embedding of the segmented point cloud. The metric embedding of the segmented point cloud has been forced into a unit sphere metric space.

[0186] In an embodiment, the creating at block 1010 may include assigning zero embeddings to free spaces in the metric embedding of the segmented point cloud.

[0187] exist Fig.10 At block 1020 of , method 1000 may include predicting a shape point cloud of a completion of a partially observed object represented by the scene. For example, in one aspect, the device 100, the robot 310, the shape prediction evaluation component 180, and / or the prediction component 850 may be configured as or may include means for predicting a shape point cloud of a completion of a partially observed object represented by the scene.

[0188] For example, predicting at block 1020 may include providing one or more images including image data and depth information to a shape completion model.

[0189] In an embodiment, the prediction at block 1020 may include obtaining a shape reconstruction of a partially observed object of the scene representation from a shape completion model. The shape reconstruction may be mapped to a coordinate system of an image sensor used to obtain the one or more images.

[0190] In an embodiment, the prediction at block 1020 may include transforming the shape reconstruction from a coordinate system of the image sensor to a world coordinate system based on physical configuration information of the image sensor.

[0191] exist Fig.10 At block 1030 of , method 1000 may include generating a confidence score for each point of the completed shape point cloud using a probabilistic machine learning model based on the correlation between the geometric information and the semantic information. For example, in one aspect, the device 100, the robot 310, the shape prediction evaluation component 180, and / or the generation component 860 may be configured as or may include means for generating a confidence score for each point of the completed shape point cloud using a probabilistic machine learning model based on the correlation between the geometric information and the semantic information.

[0192] For example, the generation at box 1030 can include training the probabilistic machine learning model 550 using training data including the metric embedding 535 of the segmented point cloud 524.

[0193] In an embodiment, the probabilistic machine learning model 550 may be a Gaussian process model and the partially observed objects of the scene representation may include the target object. In such an embodiment, the generating at block 1030 may include generating a confidence score 555 based on at least one of an object property indicating whether the target object is fragile and past robot experience indicating whether the robot has previously handled the target object.

[0194] The following aspects are merely illustrative, and aspects thereof may be combined with aspects of other embodiments or teachings described herein without limitation thereto.

[0195] Aspect 1 is a method for evaluating the reliability of 3D shape prediction performed by a processor. The method includes obtaining a scene representation including one or more images. Each of the one or more images includes depth information. The method also includes estimating semantic information of a partially observed object in the scene representation based on geometric information extracted from the one or more images. The method also includes obtaining a segmented point cloud of the partially observed object in the scene representation based on the semantic information and the geometric information. The method also includes creating a metric embedding of the segmented point cloud corresponding to the partially observed object of the scene representation. The method also includes predicting a completed shape point cloud of the partially observed object of the scene representation. The method also includes using a probabilistic machine learning model to generate a confidence score for each point of the completed shape point cloud based on the correlation between the geometric information and the semantic information. The probabilistic machine learning model has been trained using training data including the metric embedding of the segmented point cloud. The method also includes controlling the movement of the robot based on the confidence score of each point of the completed shape point cloud.

[0196] In aspect 2, the processor as described in aspect 1 may be included in a robot, and the partially observed object of the scene representation may include a target object. In such an aspect, controlling the movement of the robot may include controlling at least one of an arm and an end effector of the robot to enable the robot to grasp the target object.

[0197] In aspect 3, the processor as described in aspect 1 may be included in an external device that may be provided separately from the robot, and the partially observed objects represented by the scene may include a target object. In such an aspect, controlling the motion of the robot may include sending motion control parameters of at least one of an arm and an end effector of the robot to the robot, which may enable the robot to grasp the target object based on the motion control parameters.

[0198] In aspect 4, the probabilistic machine learning model as described in any of aspects 1 to 3 may be a Gaussian process model, and the partially observed objects of the scene representation as described in any of aspects 1 to 3 may include a target object. In such an aspect, the generation of the confidence score may include generating the confidence score based on at least one of an object property indicating whether the target object is fragile and past robot experience indicating whether the robot has previously handled the target object.

[0199] In aspect 5, the partially observed objects of the scene representation as described in any one of aspects 1 to 4 may include a target object. In such an aspect, the method may also include identifying an area of ​​the target object having a plurality of data points having corresponding confidence scores in the completed shape point cloud below a predetermined threshold, and controlling the robot to capture at least one other image of the area of ​​the target object.

[0200] In aspect 6, the obtaining of the scene representation as described in any of aspects 1 to 5 may include obtaining one or more images from a first camera position. In such aspects, the method may also include filtering out points having corresponding confidence scores that may fail to meet a threshold confidence score from the completed shape point cloud, resulting in filtered points of the completed shape point cloud, and determining a second camera position different from the first camera position based on the filtered points of the completed shape point cloud, obtaining at least one other image of the scene representation from the second camera position, the at least one other image may include at least one partially observed portion of a partially observed object of the scene representation.

[0201] In aspect 7, the method as described in any one of aspects 1 to 6 may further include adjusting the threshold confidence score based on an updated confidence score of at least one other image.

[0202] In aspect 8, obtaining the segmented point cloud as described in any one of aspects 1 to 7 may include extracting geometric information from image data and depth information of one or more images, estimating semantic information of each pixel of one or more images, using the geometric information to determine an estimated confidence score of the semantic information of each pixel of one or more images, and combining the semantic information and depth information, resulting in a segmented point cloud.

[0203] In aspect 9, obtaining the segmented point cloud as described in any one of aspects 1 to 8 may include maintaining a semantic map of the segmented point cloud, which may include semantic information of partially observed objects represented by the scene, and tracking the partially observed objects between a first image in the one or more images and a second image in the one or more images using the semantic information. The semantic information may include a semantic class assigned to each partially observed object.

[0204] In aspect 10, the creation of the metric embedding of the segmented point cloud as described in any of aspects 1 to 9 may include providing the segmented point cloud to the transformer model to obtain an embedding of the segmented point cloud, and providing the embedding of the segmented point cloud to the perceptron model to obtain the metric embedding of the segmented point cloud. The embedding of the segmented point cloud may include semantic information and geometric information of the segmented point cloud. The metric embedding of the segmented point cloud may have been forced into a unit sphere metric space.

[0205] In aspect 11, the creating of the metric embedding of the segmented point cloud as described in any one of aspects 1 to 10 may further include assigning zero embeddings to free spaces in the metric embedding of the segmented point cloud.

[0206] In aspect 12, the prediction of the completed shape point cloud as described in any of aspects 1 to 11 may include providing one or more images including image data and depth information to a shape completion model, obtaining a shape reconstruction of a partially observed object of the scene representation from the shape completion model, and transforming the shape reconstruction from a coordinate system of the image sensor to a world coordinate system based on the physical configuration information of the image sensor. The shape reconstruction may be mapped to the coordinate system of the image sensor used to obtain the one or more images.

[0207] Aspect 13 is a device for evaluating the reliability of 3D shape prediction. The device includes a memory storing instructions and a processor communicatively coupled to the memory. The processor is configured to execute the instructions to perform one or more of the methods described in any one of aspects 1 to 12.

[0208] Aspect 14 is a device for evaluating the reliability of 3D shape prediction, comprising a device for performing one or more of the methods described in any one of aspects 1 to 12.

[0209] Aspect 15 is a computer-readable storage medium storing computer-executable instructions for evaluating the reliability of 3D shape prediction. The computer-executable instructions are configured to cause the device to perform one or more of the methods described in any one of aspects 1 to 12 when executed by one or more processors of the device.

[0210] Aspect 16 is a method for evaluating the reliability of a 3D shape prediction performed by a processor. The method includes creating a metric embedding of a segmented point cloud corresponding to a partially observed object represented by a scene based on estimated semantic information of the partially observed object and geometric information extracted from a scene representation. The method also includes predicting a completed shape point cloud of the partially observed object represented by the scene. The method also includes using a probabilistic machine learning model to generate a confidence score for each point of the completed shape point cloud based on a correlation between the geometric information and the semantic information.

[0211] In aspect 17, the method described in aspect 16 may include one or more of the methods described in any one of aspects 1 to 12.

[0212] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the embodiments.

[0213] As used herein, the terms "component", "module", "system", etc. are intended to include computer-related entities such as, but not limited to, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As an illustration, both an application running on a computing device and a computing device can be a component. One or more components can reside within a process and / or a thread of execution, and a component can be located on one computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures stored thereon. Components can communicate by means of local and / or remote processes, such as according to signals having one or more data packets, such as data from a component interacting with another component in a local system, a distributed system, and / or data from a component interacting with another system across a network (such as the Internet) by means of signals.

[0214] Some embodiments may be directed to systems, methods and / or computer readable media at any possible level of technical detail integration.A computer readable medium may include a computer readable storage medium (or medium) having thereon computer readable program instructions for causing a processor to perform operations.

[0215] A computer-readable storage medium may be a tangible device that can retain and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EEPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a DVD, a memory stick, a floppy disk, a mechanical encoding device (such as a punched card or a raised structure in a groove on which instructions are recorded), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0216] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0217] The computer readable program code / instruction for performing the operation can be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, a configuration data for an integrated circuit, or a source code or object code written in any combination of one or more programming languages, and the programming language includes object-oriented programming languages ​​such as Smalltalk, C++, etc., and a procedural programming language such as "C" programming language or similar programming language. The computer readable program instruction can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including LAN or WAN), or can be connected to an external computer (for example, by using the Internet of an Internet Service Provider (ISP)). In some embodiments, the electronic circuit including, for example, a programmable logic circuit, an FPGA or a programmable logic array (PLA) can execute the computer readable program instruction to personalize the electronic circuit by utilizing the state information of the computer readable program instruction, so as to perform various aspects or operations.

[0218] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, and / or other equipment to function in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0219] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0220] According to example embodiments, the drawings (eg, Figure 1 , Figure 3 and Figures 5 to 10) in the box represented by the components, elements, modules or units (collectively referred to as "components" in this paragraph) can be implemented as various numbers of hardware, software and / or firmware structures that perform the above-mentioned corresponding functions. According to example embodiments, at least one of these components can use a direct circuit structure, such as a memory, a processor, a logic circuit, a lookup table, etc., which can perform the corresponding functions by the control of one or more microprocessors or other control devices. In addition, at least one of these components can be embodied by a part of a module, a program or a code, which contains one or more executable instructions for performing a specified logical function and is executed by one or more microprocessors or other control devices. In addition, at least one of these components can include a processor, a microprocessor, etc. such as a CPU that performs the corresponding function, or can be implemented by a processor, a microprocessor, etc. such as a CPU that performs the corresponding function. Two or more of these components can be combined into a single component that performs all operations or functions of the two or more components combined. In addition, at least a part of the function of at least one of these components can be performed by another of these components. The functional aspects of the above-mentioned example embodiments can be implemented in an algorithm executed on one or more processors. In addition, the components represented by blocks or processing steps can use any number of related technologies for electronic configuration, signal processing and / or control, data processing, etc.

[0221] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer-readable medium according to various embodiments. In this regard, each frame in the flow chart or block diagram may represent a part of a module, segment or instruction, which includes one or more executable instructions for implementing the specified logical function. The method, computer system and computer-readable medium may include additional frames, fewer frames, different frames or frames of different arrangements compared to those frames depicted in the figure. In some alternative embodiments, the functions marked in the frame may not occur in the order marked in the figure. For example, the two frames shown in succession can actually be performed simultaneously or substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the functions involved. It can also be noted that each frame shown in the block diagram and / or flow chart and the combination of frames shown in the block diagram and / or flow chart can be implemented by a system based on dedicated hardware that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0222] It is clear that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0223] Unless explicitly described as such, the elements, actions or instructions used herein should not be interpreted as critical or necessary. In addition, as used herein, the articles "one" and "an" are intended to include one or more projects, and can be used interchangeably with "one or more". In addition, as used herein, the term "set" is intended to include one or more projects (e.g., related projects, unrelated projects, combinations of related projects and unrelated projects, etc.), and can be used interchangeably with "one or more". In the case of only one project, the term "one" or similar language is used. In addition, as used herein, the terms "has", "have", "include", "including", etc. are intended to be open terms. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "based at least in part". In addition, statements such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B, or both A and B.

[0224] References to "one embodiment", "embodiment" or similar language throughout this specification means that the specific features, structures or characteristics described in conjunction with the indicated embodiment are included in at least one embodiment of this solution. Therefore, phrases "in one embodiment", "in an embodiment" and similar language throughout this specification may but not necessarily all refer to the same embodiment. As used herein, terms such as "first" and "second" or "first" and "second" can be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (for example, importance or order). It should be understood that if an element (for example, a first element) is referred to as being coupled to another element (for example, a second element) "coupled to", "coupled to", "connected to" or "connected to" another element (for example, a second element) with or without the term "operably" or "communicatively", it means that the element can be coupled to another element directly (for example, wired), wirelessly or via a third element.

[0225] It should be understood that when an element or layer is referred to as being "over," "above," "on," "below," "under," "beneath," "connected to," or "coupled to" another element or layer, it can be directly on, above, above, below, below, connected to, or coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being "directly on," "directly on," "directly on," "directly under," "directly under," "directly under," "directly connected to," or "directly coupled to" another element or layer, there are no intervening elements or layers.

[0226] The description of various aspects and embodiments has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Even if combinations of features are described in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways that are not specifically described in the claims and / or disclosed in the specification. Although each dependent claim listed below may be directly subordinate to only one claim, the disclosure of possible implementations includes the combination of each dependent claim with each other claim in the claim set. Many modifications and variations are clear to those of ordinary skill in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements to technologies found in the market, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

[0227] It should be understood that the specific order or hierarchy of the blocks in the disclosed process / flowchart is an illustration of an exemplary approach. It should be understood that the specific order or hierarchy of the blocks in the process / flowchart can be rearranged based on design preferences. In addition, some blocks can be combined or omitted. The attached claims present the elements of the various blocks in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0228] In addition, the described features, advantages, and characteristics of the present disclosure may be combined in any suitable manner in one or more embodiments. Based on the description herein, those skilled in the relevant art may recognize that the present disclosure may be practiced without one or more specific features or advantages of a particular embodiment. In other cases, additional features and advantages that may not be present in all embodiments of the present disclosure may be recognized in certain embodiments.

Claims

1. A method for controlling the motion of a robot, the method comprising: obtaining a scene representation comprising one or more images, each of the one or more images comprising depth information; estimating semantic information of partially observed objects in the scene representation based on geometric information extracted from the one or more images; determining a segmented point cloud of a portion of an observed object in the scene representation based on the semantic information and the geometric information; creating a metric embedding of the segmented point cloud corresponding to a portion of an observed object represented by the scene; predicting a completed shape point cloud of a portion of an observed object represented by the scene; Based on the correlation between the geometric information and the semantic information, using a probabilistic machine learning model, generating a confidence score for each point of the completed shape point cloud; as well as The movement of the robot is controlled based on the confidence score of each point of the completed shape point cloud.

2. The method according to claim 1, wherein: A processor is included in the robot, The portion of the observed objects represented by the scene includes a target object, and Controlling the movement of the robot includes controlling at least one of an arm and an end effector of the robot so that the robot can grasp the target object.

3. The method according to any one of claims 1 to 2, wherein: The processor is included in an external device provided separately from the robot, The portion of the observed objects represented by the scene includes a target object, and Controlling the movement of the robot includes sending a motion control parameter of at least one of an arm and an end effector of the robot to the robot, so that the robot grasps the target object based on the motion control parameter.

4. The method according to any one of claims 1 to 3, wherein: The probabilistic machine learning model is a Gaussian process model, The portion of the observed objects represented by the scene includes a target object, and The generating of the confidence score includes generating the confidence score based on at least one of an object property indicating whether the target object is fragile and past robot experience indicating whether the robot has previously handled the target object.

5. The method according to any one of claims 1 to 4, wherein: The portion of the observed objects represented by the scene includes a target object, and The method further comprises: identifying a region of the target object having a plurality of data points having corresponding confidence scores in the completed shape point cloud below a predetermined threshold; and The robot is controlled to capture at least one other image of the area of ​​the target object.

6. The method according to any one of claims 1 to 5, wherein: The obtaining of the scene representation comprises obtaining the one or more images from a first camera position, and The method further comprises: filtering out points having corresponding confidence scores that fail to satisfy a threshold confidence score from the completed shape point cloud, resulting in filtered points of the completed shape point cloud; and Based on the filtered points of the completed shape point cloud, a second camera position different from the first camera position is determined, and at least one other image of the scene representation including at least one partially observed portion of the partially observed object of the scene representation is obtained from the second camera position.

7. The method according to any one of claims 1 to 6, further comprising: The threshold confidence score is adjusted based on the updated confidence score of the at least one other image.

8. The method according to any one of claims 1 to 7, wherein: The determination of the segmented point cloud includes: extracting geometric information from the image data and the depth information of the one or more images; estimating semantic information for each pixel of the one or more images using the geometric information; determining an estimated confidence score for the semantic information for each pixel of the one or more images; and The semantic information is combined with the depth information, resulting in a segmented point cloud.

9. The method according to any one of claims 1 to 8, wherein: The determination of the segmented point cloud includes: maintaining a semantic map of the segmented point cloud including semantic information of partially observed objects of the scene representation, the semantic information including a semantic class assigned to each of the partially observed objects; and A partially observed object between a first image of the one or more images and a second image of the one or more images is tracked using the semantic information.

10. The method according to any one of claims 1 to 9, wherein: The creation of a metric embedding of the segmented point cloud includes: providing the segmented point cloud to a transformer model to obtain an embedding of the segmented point cloud, the embedding of the segmented point cloud comprising semantic information and geometric information of the segmented point cloud; and The embedding of the segmented point cloud is provided to the perceptron model to obtain a metric embedding of the segmented point cloud that has been forced into the unit sphere metric space.

11. The method according to any one of claims 1 to 10, further comprising: Assign zero embeddings to the free space in the metric embedding of the segmented point cloud.

12. The method according to any one of claims 1 to 11, wherein: The prediction of the completed shape point cloud includes: providing one or more images including image data and depth information to a shape completion model; obtaining a shape reconstruction of a partially observed object of the scene representation from the shape completion model, the shape reconstruction being mapped to a coordinate system of an image sensor used to obtain the one or more images; and The shape reconstruction is transformed from a coordinate system of the image sensor to a world coordinate system based on physical configuration information of the image sensor.

13. The method according to any one of claims 1 to 12, wherein: The probabilistic machine learning model has been trained using training data comprising metric embeddings of the segmented point cloud.

14. A computer-readable storage medium storing computer-executable instructions, which, when executed by at least one processor of a device, cause the device to perform the method according to any one of claims 1 to 13.

15. A device (100) for controlling the movement of a robot, the device (100) comprising: A memory (130) for storing instructions; as well as A processor (120) is communicatively coupled to the memory (130), wherein the processor (120) is configured to execute the instructions to: obtaining a scene representation comprising one or more images, each of the one or more images comprising depth information; estimating semantic information of partially observed objects in the scene representation based on geometric information extracted from the one or more images; determining a segmented point cloud of a portion of an observed object in the scene representation based on the semantic information and the geometric information; creating a metric embedding of the segmented point cloud corresponding to a portion of an observed object represented by the scene; predicting a completed shape point cloud of a portion of an observed object represented by the scene; Based on the correlation between the geometric information and the semantic information, using a probabilistic machine learning model, generating a confidence score for each point of the completed shape point cloud; and The movement of the robot is controlled based on the confidence score of each point of the completed shape point cloud.

Images