Shape-configuration-based machine learning model for object localization

Abstract

A method and system for training a model includes performing 2D object detection on training images to identify objects (304). The training images are cropped around the objects (304). A category-level shape reconstruction is generated using a neural radiance field model (306). A normalized coordinate model is trained (402) using the training images and ground truth information from the category-level shape reconstruction.

Description

[Technical Field]

[0001] Related Application Information This application claims priority to U.S. Patent Application No. 63 / 421,607, filed November 2, 2022, U.S. Patent Application No. 63 / 463,356, filed May 2, 2023, and U.S. Patent Application No. 18 / 499,680, filed November 1, 2023, each of which is incorporated by reference in its entirety. This application is related to the application entitled "NEURAL SHAPE MACHINE LEARNING FOR OBJECT LOCALIZATION WITH MIXED TRAINING DOMAINS," having Attorney Docket No. 22111, which is incorporated by reference in its entirety.

[0002] The present invention relates to computer vision, and more particularly to object localization in a three-dimensional environment. [Background technology]

[0003] 2. Description of Related Art Monocular cameras are sometimes used for computer vision tasks, such as helping autonomous vehicles and robots navigate their environments. Because monocular vision lacks the depth perception inherent in binocular vision, such as the human visual system, locating objects in three-dimensional space is difficult. Summary of the Invention

[0004] A method for training the model includes performing 2D object detection on training images to identify the object. The training images are cropped around the object. A neural radiance field (NeRF) model is used to generate a category-level shape reconstruction. A normalized coordinate model is trained using the training images and ground truth information from the category-level shape reconstruction.

[0005] A system for training a model includes a hardware processor and a memory that stores a computer program that, when executed by the hardware processor, causes the hardware processor to perform 2D object detection on training images to identify objects, crop the training images around the objects, generate category-level shape reconstructions using a NeRF model, and train a normalized coordinate model using the training images and ground truth information from the category-level shape reconstructions. [Brief explanation of the drawings]

[0006] The present disclosure provides details in the following description of preferred embodiments with reference to the following figures.

[0007] [Figure 1] 1 is a diagram of an environment with different objects positioned around a camera in three-dimensional space, according to one embodiment of the present invention.

[0008] [Figure 2] FIG. 1 is a block diagram of a healthcare facility capable of performing three-dimensional object localization, according to one embodiment of the present invention.

[0009] [Figure 3] FIG. 1 is a block / flow diagram of a 3D object localization model that uses neural radiance field processing to provide ground truth information during training, according to one embodiment of the present invention.

[0010] [Figure 4] FIG. 2 is a block / flow diagram of a method for training and using a 3D object localization model according to one embodiment of the present invention.

[0011] [Figure 5] FIG. 1 is a block diagram of a computing device that can be used to train and implement a three-dimensional object localization model, according to one embodiment of the present invention.

[0012] [Figure 6] FIG. 2 is a diagram of a neural network architecture that can be used to form part of a three-dimensional object localization model, according to one embodiment of the present invention.

[0013] [Figure 7] FIG. 1 is a diagram of a deep neural network architecture that can be used to form part of a three-dimensional object localization model, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] Given an image from a single camera, the position and orientation of an object in the image can be estimated. Additional information, such as the object's 3D size, can also be determined. To do this, 2D object detection may be used to first identify objects of interest in the image. The object's 3D attributes may then be estimated. 3D object localization has applications in autonomous driving, advanced driver assistance systems, traffic scene analysis, and health monitoring.

[0015] Reconstruction-based approaches to 3D localization use appearance information and are independent of camera geometry. This allows them to generalize across various data sources with large variations in camera parameters and viewpoints. Given the noisy nature of the data, ground truth for object shapes may not be available. Therefore, neural radiance fields (NeRFs) can be used to learn category-level shape reconstruction. The category-level NeRF can be trained using 3D pose annotations and instance masks in the training dataset. While the category-level NeRF can convert an image of an object to its 3D shape, it does not directly aid in localization. For this purpose, a deep neural network model can be trained to regress normalized object coordinates. Ground truth values ​​for object coordinates can be rendered from the NeRF using the ground truth 3D pose. The object coordinate representation shows the correspondence between the 2D and 3D representations. The 3D pose can be estimated using Perspective-n-Point.

[0016] NeRF learns category-level common knowledge by representing shapes as linear combinations of a set of low-rank NeRF bases. The coefficients of the linear combinations can be learned by a convolutional neural network (CNN) conditioned on the input images. Because shape reconstruction does not yield 3D poses and directly optimizing the pose with differentiable renderings can be ambiguous, especially in highly occluded cases, a separate CNN can be used to regress normalized object coordinates from the input image of the object. Image pixels may be mapped to normalized object coordinates relative to the object's center. Ground truth for object coordinates during training can be obtained by rendering the learned category-level NeRF with the ground truth 3D pose. 2D-to-3D correspondences can be obtained from the normalized object coordinates, along with the estimated 3D size.

[0017] 1 , an exemplary road scene is shown. A vehicle 102 is traveling on a road 100. The vehicle 102 is equipped with sensors that collect information about the road 100. For example, the vehicle 102 may be equipped with multiple video cameras 104 positioned at different locations around the vehicle to obtain visual information about the road 100 from multiple different perspectives, providing a wide-range scene. The vehicle 102 may further be equipped with a 360-degree LiDAR sensor 106 positioned to collect geometric information about the road 100 all around the vehicle 102.

[0018] The vehicle 102 records information from its sensors. By driving as many roads as possible in the area, the vehicle 102 can gather information about the road 100. The collected information is used to identify defects on the road 100 and objects in the surrounding environment. This information can be used to assist in the navigation and control of the vehicle 102.

[0019] Exemplary defects include potholes 108, ruts, cracks 110, and faded road markings 112. Exemplary objects include stationary objects 113, including traffic control features such as barricades and environmental control features such as streetlights and signs, as well as stationary objects unrelated to the function of the road 100, such as mailboxes, convenience stores, and trash cans. Moving objects include other vehicles 114 and pedestrians 116, which behave according to their own unpredictable logic. Thus, these objects are interpreted as obstacles that may pose immediate navigation challenges or future hazards or challenges. Thus, as the vehicle 102 travels along the road 100, its sensors collect information about the road itself and stationary and moving objects visible to the vehicle. The vehicle can identify safe routes and autonomously navigate the environment to safely reach its destination.

[0020] In addition to operating autonomous vehicles on roads, the principles of the present invention can be applied to any suitable computer vision task involving identifying and locating objects in three-dimensional space. For example, a robot in a healthcare facility may have the ability to navigate hallways to deliver supplies to healthcare professionals and interact with patients. To accomplish such tasks, the healthcare robot must navigate a dynamic environment where medical equipment is regularly moved and left in hallways, and healthcare professionals may need to urgently reach their destination. In such an environment, the robot must be able to quickly identify and react to changes in the environment by locating objects in the three-dimensional space around it.

[0021] While onboard cameras 104 are specifically considered for both collecting training data and using trained object localization models, it should be understood that cameras may be mounted in other locations. For example, cameras may be attached to fixed locations, such as permanent or semi-permanent infrastructure. Data obtained from such fixed sources may also be used for object localization, such as security surveillance to identify hazardous materials or unauthorized persons.

[0022] 2, a diagram illustrating object localization in a healthcare facility 200 is shown. A healthcare robot 206 is capable of navigating the environment of the healthcare facility 200. The robot 206 not only assists the medical professional 202, but also interfaces with the treatment system 204 to assist in providing treatment to patients. This allows the robot 206 to be used to assist in the monitoring and treatment of multiple patients, for example, responding to changing environmental conditions, supply shortages, etc.

[0023] The healthcare facility may be staffed by one or more medical professionals 202 who provide information about events and measurements of system status. The medical professionals 202 represent both potential obstacles for the healthcare robot 206 as it moves through the healthcare facility and task-related destinations for the robot 206—for example, the robot 206 may be tasked with delivering supplies to a particular medical professional 202. The treatment system 204 may represent stationary objects within the healthcare facility 200 and may be designed to monitor patient status, compile medical records, and automatically administer and adjust treatment as needed.

[0024] The various components of healthcare facility 200 may communicate with one another over network 210, for example, using any suitable wired or wireless communication protocol and medium, thereby enabling healthcare robot 206 to receive information regarding the location of other objects within healthcare facility 200 using multiple sensors located external to the robot itself.

[0025] Referring now to FIG. 3, a block / flow diagram of a method / system for object localization in three-dimensional space is shown. Block 302 captures a monocular image of an environment containing at least one object. The captured image may include an array of red-green-blue (RGB) or grayscale pixels representing visual information. A backbone neural network model, implemented as a residual neural network (ResNet), for example, may regress the normalized coordinate map and an object mask to predict coefficients representing the shape and color of the object. Two-dimensional object detection 304 is performed on the image to identify objects within the image. Any suitable object detection model can be used, and the output of the object detection includes two-dimensional bounding boxes of detected objects, providing, for example, the coordinates where each object is located within the image.

[0026] The category-level NeRF 306 is run on the detected objects. The image may be cropped within a corresponding bounding box for processing by a NeRF machine learning model that jointly optimizes linear coefficients and a set of low-rank NeRF bases. The NeRF machine learning model may be implemented, for example, as a feature grid followed by a multilayer perceptron (MLP). The linear coefficients may be regressed by a CNN conditioned on the cropped image. Training of NeRF may be supervised by differentiable rendered instance masks and RGB values ​​at ground truth 3D poses, as described in more detail below. Notably, NeRF 306 only needs to be run during training because it generates pseudo-ground truth labels for training other parts of the model.

[0027] In block 308, a coordinate neural network model, such as a ResNet model, is used to regress normalized object coordinates based on the cropped input image. The coordinate neural network model can be trained using coordinates rendered from category-level NeRF 306. During inference, NeRF 306 can be omitted and normalized coordinates can be determined based on the trained coordinate neural network model and detected object information.

[0028] In block 310, the size of the detected object is determined by regressing the height, width, and length of the object using, for example, a machine learning model. Size estimation can be performed using an appropriate convolutional neural network (CNN) model. Block 312 takes the normalized object coordinates and the estimated size and builds a correspondence between the 2D and 3D information. This n-point perspective processing can be performed using linear algebra and nonlinear least-squares optimization. Block 314 can use the collected information to estimate the 3D location of the object.

[0029] Object localization involves determining the three-dimensional occupancy of an object based on its dimensions.

number

number

[0030] Normalized Camera Coordinates

number

number

number

number

number

number

number

number

[0031] NeRF can be used in conjunction with the localization branch to train a normalized object coordinate prediction model. Images around detected objects can be cropped and used to regress a normalized coordinate map and an object mask. This model then predicts two sets of coefficients representing the shape and color of object instances, which are used to deform a NeRF-based shape model represented by a latent grid. During training, ground truth information about object pose and size is used to train the NeRF model using the object mask, appearance, and optionally LiDAR depth. The normalized coordinate prediction branch can then be supervised using the normalized coordinates from the NeRF model. NeRF fuses categorical shape supervision from all training data to provide reliable and dense normalized coordinate supervision.

[0032] To benefit from accurate and dense normalized coordinate supervision, an image-conditioned shape representation (e.g., NeRF) is trained to fuse shape supervision from multiple sources to render dense normalized coordinate supervision for localization. Such sources can include object masks, LiDAR data, and depth maps. A 3D latent grid φ can be used as the shape representation implemented as a multi-resolution dense grid. Input normalized object coordinates

number

number

[0033] To model categorical shape variations, we construct a low-rank deformable shape representation using a set of learnable 3D latent grids. The mean shape grid φ μ and a set of deformed grid bases

number

number

number

[0034] The grid is compatible with volume rendering. For each object, the line of sight in the normalized object coordinate system

number

number

number

number

number

number

number

number

number

number

[0035] Training the shape representation can include multiple shape losses, including occupancy loss, RGB loss, and additional shape supervision from LiDAR and dense depth maps.

number

[0036] The occupancy loss can be supervised using a ground truth triplet mask containing three categories: foreground, background, and unknown (e.g., occlusion). The occupancy is set to 1 for foreground and 0 for background, and can be omitted for unknown regions. Other losses can be applied to foreground regions.

number

number

[0037] While the RGB loss is not required for shape training, having a supplementary RGB loss provides an additional photometric constraint that regularizes the shape and improves performance. The shape and color bases and coefficients can be used separately. High-quality examples (e.g., at least 40 pixels high and no occlusions) can be used to train the latent bases; otherwise, the latent bases can be fixed and optimized for coefficient prediction.

[0038] Despite using admissible rank spaces to force different object instances to share a deformation basis, deformations may not be sufficiently regularized, resulting in unnecessary deformations competing with each other and producing noisy shape results. Mask monitoring also suffers from hull intersection ambiguity, and sparse LiDAR information may not completely cover the object, leaving ambiguities in uncovered and unobserved regions. These imperfect shape monitoring can result in unpredictable, random deformations.

[0039] To mitigate this risk, the Kullback-Leibler (KL) divergence loss can be used to minimize the information gain from the deformation coefficients.

number

number

number

number

number

[0040] Without depth supervision, the occupancy loss is conceptually just shape reconstruction from silhouettes, which suffers from ambiguity due to hull intersections. To further regularize the shape randomness against ambiguity, a dense shape prior can be used to favor solids over empty space when both solutions are possible.

number

number

[0041] As mentioned above, cropping the image around a detected object omits the global context of the object. The global context, which would otherwise be obtained from the entire captured image, can negatively impact normalized coordinate learning, leading to poor performance for small, distant, or highly occluded objects. By removing the context from the input, the network learns normalized coordinates through a strict mapping from object appearance to coordinates, potentially resulting in better generalization performance.

[0042] Therefore, the identification of normalized coordinates can be separated from the object detection task and performed using a separate machine learning model. The normalized coordinates ensure consistency between the model's coordinate prediction and the coordinate rendering. The object latent coefficients z predicted by the model are used to generate the latent shape representation Φ. obj Get

number

[0046] , the normalized coordinate map can be rendered using the ground truth pose and size.

[0043] The consistency loss of normalized coordinates can be expressed as the L2 loss between predicted and rendered coordinates.

number

[0044] Furthermore, a self-supervised reprojection error loss can be further used to amplify the loss orthogonal to the gaze direction as follows:

number

number

number

number

number

[0045] Given the object size predictions from the detection branch and the NOC with uncertainty, the PnP problem can be solved using the Levenberg-Marquardt algorithm with a random sampling initialization scheme. To predict the pose confidence, the 3D bounding box IoU can be regressed from the feature map. Due to the lack of context, the network cannot estimate the object distance from the cropped object image, and therefore cannot estimate the pose uncertainty due to object distance. The Jacobian map of the reprojection error to the solved pose can be used as an input feature.

number

[0046] The scale of monocular localization is inherently ill-posed, and perspective-n-point based methods that rely solely on object size prediction can be unreliable. Therefore, direct depth prediction can be fused with n-point prediction. pred Using this, object size prediction

number

number

number

number

number

number

[0047] Cross-domain object localization tasks present additional challenges. Domain gaps can lead to performance degradation due to scene and sensor configurations. For example, if a localization model is trained in one city and tested in a second city, performance may degrade due to differences in the distribution of scene structure between the two cities. Even within the same city, performance may degrade if the sensor configuration changes between the sensors capturing training data and those used for inference. Such sensor configuration changes may include, for example, mounting position, orientation, and camera intrinsic parameters.

[0048] Training an object localization system can require large amounts of accurately annotated training data created in the same domain as the test data, which severely limits the scalability and flexibility of 3D object localization in real-world applications.

[0049] The normalized coordinate model can be trained on images from a first domain with complete 3D box annotations. In new domains, complete annotations may not be available, so it cannot be assumed that object sizes can be estimated. Object sizes can be annotated and trained at a lower cost than complete annotations. Unlike complete annotations, annotating only object sizes can be achieved without the need for LiDAR data. The size prediction network can then be adapted to new domains with more limited size annotations.

[0050] Referring now to FIG. 4, an overview of how an object localization model is trained and used is provided. Model training 402 uses an annotated training dataset to jointly train a category-level NeRF model and a regression network that predicts normalized coordinates with a foreground mask, e.g., using a deep learning approach. The NeRF is represented as a trainable latent grid that includes a standard mean grid and several low-rank grid bases that take deformations into account. These bases are linearly combined by image-conditioned learnable coefficients. The normalized coordinates rendered by the NeRF supervise the regression of the normalized coordinates from the image.

[0051] The training can include training a size estimation model. As mentioned above, the size estimation can be based on training data from different regions, and the training data from different regions can have different levels of annotation. Some training data can have relatively complete annotation information, while other training data can be annotated only with the sizes of objects in the image. Both sets can be used to train the size estimation model.

[0052] Changes between domains may also include changes in viewpoint. For example, training data captured by an autonomous vehicle or robot can be used to train models deployed in fixed infrastructure-based systems, such as cameras mounted on utility poles on the side of the road or security cameras in healthcare facilities. During training, the cameras can be assumed to be calibrated relative to the ground plane.

[0053] After training is complete, the trained model can be deployed (404). Such deployment can include copying the trained model's parameters to a target vehicle or robot so that the target can perform object localization (406). During inference, NeRF rendering is not required, but normalized coordinate regression can be used for 3D localization via the Perspective-n-Point method. Therefore, deployment (404) does not require copying the NeRF portion of the model, as the NeRF output is no longer needed as a pseudo-ground truth for training.

[0054] During object localization 406, object detection is performed on images from infrastructure-mounted cameras, and the detected objects can be processed with a normalized coordinate model. Given the pixel-by-pixel normalized coordinate predictions, the optimization objective can be formulated as a reprojection loss in a perspective-n-point problem, but with both object size and object pose simultaneously optimized. Object locations can be constrained to lie on a calibrated ground plane as a priori, which removes scale ambiguity in single-viewpoint 3D reconstruction and allows for the optimization of object size along with object pose.

[0055] 5, an exemplary computing device 500 is shown, according to one embodiment of the present invention. The computing device 500 is configured to train and use object localization models.

[0056] Computing device 500 may be embodied as any type of computing device or computer device capable of performing the functions described herein, including, but not limited to, a computer, a server, a rack-based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and / or a consumer electronics device. Additionally or alternatively, computing device 500 may be embodied as one or more computing sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically distributed computing device.

[0057] 5, computing device 500 includes a processor 510, an input / output subsystem 520, a memory 530, a data storage device 540, a communications subsystem 550, and / or other components and devices typically found in a server or similar computing device. Computing device 500, in other embodiments, may include other or additional components, such as those typically found in a server computer (e.g., various input / output devices). Furthermore, in some embodiments, one or more of the illustrated components may be incorporated into or form part of another component. For example, in some embodiments, memory 530, or portions thereof, may be incorporated into processor 510.

[0058] The processor 510 may be embodied as any type of processor capable of performing the functions described herein, and may be embodied as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.

[0059] The memory 530 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. During operation, the memory 530 may store various data and software used during operation of the computing device 500, such as an operating system, applications, programs, libraries, drivers, etc. The memory 530 is communicatively coupled to the processor 510 via an I / O subsystem 520, which may be embodied as circuits and / or components that facilitate input / output operations with the processor 510, the memory 530, and other components of the computing device 500. For example, the I / O subsystem 520 may be embodied as or include a memory controller hub, an input / output control hub, a platform controller hub, an integrated control circuit, a firmware device, a communication link (such as a point-to-point link, a bus link, a wire, a cable, an optical guide, a trace on a printed circuit board, etc.), and / or other components and subsystems that facilitate input / output operations. In some embodiments, the I / O subsystem 520 may form part of a system-on-chip (SOC) and may be integrated on a single integrated circuit chip along with the processor 510, memory 530, and other components of the computing device 500.

[0060] The data storage device 540 may be embodied as any type of device(s) configured for short-term or long-term data storage, such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. The data storage device 540 may store program code 540A for model training and program code 540B for three-dimensional object localization. Any or all of these program code blocks may be included in a given computing system. The communications subsystem 550 of the computing device 500 may be embodied as any network interface controller or other communications circuitry, device, or collection thereof that enables communications between the computing device 500 and other remote devices over a network. The communications subsystem 550 may be configured to use any one or more communications technologies (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand, Bluetooth, Wi-Fi, WiMAX, etc.) to achieve such communications.

[0061] As shown, computing device 500 may also include one or more peripheral devices 560. Peripheral devices 560 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 560 may include a display, a touch screen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0062] Of course, computing device 500 may include other elements (not shown) or omit certain elements, as would readily occur to one skilled in the art. For example, various other sensors, input devices, and / or output devices may be included in computing device 500, depending on the particular implementation, as would be readily understood by one skilled in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memory, etc. may be utilized in a variety of configurations. These and other variations of processing system 500 will be readily contemplated by one skilled in the art in view of the teachings of the present invention provided herein.

[0063] 6 and 7, exemplary neural network architectures are shown that can be used to implement portions of the present models, such as NeRF models 602 and 702. Neural networks are generalized systems whose functionality and accuracy improves with exposure to additional empirical data. Neural networks are trained by exposure to empirical data. During training, the neural network stores and adjusts weights that are applied to input empirical data. Applying the adjusted weights to the data can identify that data belongs to a specific predefined class from a set of classes and output the probability that the input data belongs to each class.

[0064] Empirical data (also called training data) from a set of examples can be formatted as a string of values ​​and fed to the input of a neural network. Each example can be associated with a known outcome or output. Each example can be represented as an (x,y) pair, where x represents the input data and y represents the known output. The input data can be of various data types and contain multiple different values. The network can have one input node for each value that makes up the example's input data, and a separate weight can be applied to each input value. The input data can be formatted, for example, as a vector, array, or string, depending on the architecture of the neural network being built and trained.

[0065] Neural networks "learn" by comparing neural network outputs generated from input data with known values ​​from examples and adjusting stored weights to minimize the difference between the output value and the known value. Adjustments can be made to the stored weights through backpropagation, and the weight's influence on the output value can be determined by calculating the mathematical gradient and adjusting the weight in a way that shifts the output toward the minimum difference. This optimization is called gradient descent and is a non-limiting example of how training occurs. A subset of examples with known values ​​that were not used in training can be used to test and validate the accuracy of a neural network.

[0066] In operation, a trained neural network can be generalized to new data not previously used for training or validation. The adjusted weights of the neural network can be applied to the new data, and the weights estimate a function developed from the training examples. The parameters of the estimated function captured by the weights are based on statistical inference.

[0067] In a hierarchical neural network, nodes are arranged in layers. An exemplary simple neural network has an input layer 620 of source nodes 622 and a single computational layer 630 with one or more computational nodes 632 that also function as output nodes, with a single computational node 632 for each possible category into which an input example can be classified. The input layer 620 may have a number of source nodes 622 equal to the number of data values ​​612 in the input data 610. The data values ​​612 in the input data 610 may be represented as a column vector. Each computational node 632 in the computational layer 630 generates a weighted linear combination of values ​​from the input data 610 provided to the input node 620 and applies a differentiable nonlinear activation function to the sum. An exemplary simple neural network can perform classification on linearly separable examples (e.g., patterns).

[0068] A deep neural network, such as a multilayer perceptron, may have an input layer 620 of source nodes 622, one or more computational layers 630 with one or more computational nodes 632, and an output layer 640 with a single output node 642 for each category into which an input example may be classified. The input layer 620 may have a number of source nodes 622 equal to the number of data values ​​612 in the input data 610. The computational nodes 632 in the computational layer 630 may also be referred to as hidden layers because they are located between the source nodes 622 and the output nodes 642 and are not directly observed. Each node 632, 642 in the computational layer generates a weighted linear combination of values ​​from the values ​​output by the nodes in the previous layer and applies a nonlinear activation function that is differentiable over the range of the linear combination. The weights applied to the values ​​from each previous node may be, for example, w1, w2, ... w n-1 ,w n The output layer provides the network's overall response to the input data. Deep neural networks can be fully connected, where each node in a computational layer is connected to every other node in the previous layer, or the connections between layers can have other configurations. If links between nodes are missing, the network is said to be partially connected.

[0069] Training a deep neural network may involve two phases: a forward phase, in which the weights of each node are fixed and inputs are propagated through the network, and a backward phase, in which error values ​​are propagated back through the network and the weight values ​​are updated.

[0070] Computational nodes 632 in one or more computational (hidden) layers 630 perform nonlinear transformations on the input data 612 that generate a feature space, where classes or categories are more easily separable in the feature space than in the original data space.

[0071] The embodiments described herein may be entirely hardware, entirely software, or contain both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0072] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer-readable medium may include any apparatus that stores, communicates, propagates, or transfers a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device), or a propagation medium. The medium may include computer-readable storage media such as semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks.

[0073] Each computer program can be tangibly stored on a machine-readable storage medium or device (e.g., program memory or magnetic disk) readable by a general-purpose or special-purpose programmable computer to configure and control the operation of the computer when the storage medium or device is read by the computer to perform the procedures described herein. The system of the present invention can also be considered to be embodied in a computer-readable storage medium configured with a computer program, where the configured storage medium causes the computer to operate in a particular, predetermined manner to perform the functions described herein.

[0074] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory employed during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0075] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0076] As employed herein, the terms “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.) that may be on-board or off-board or dedicated for use by the hardware processor subsystem.

[0077] In some embodiments, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code for achieving a specified result.

[0078] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0079] These and other variations of the hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0080] References herein to "one embodiment" or "one embodiment" of the present invention, as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "in one embodiment" or "in one embodiment," as well as any other variations thereof, in various places throughout this specification do not necessarily all refer to the same embodiment. However, it should be understood that features of one or more embodiments may be combined given the teachings of the present invention provided herein.

[0081] For example, in the case of "A / B," the use of any of the following " / ," "and / or," "at least one," such as "A and / or B" or "at least one of A and B" will be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), the selection of only the first and third listed alternatives (A and C), the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as there are listed items.

[0082] The foregoing is understood in all respects to be illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but from the claims which are interpreted in accordance with the full breadth permitted by the patent laws. It will be understood that the embodiments shown and described herein are merely exemplary of the invention, and that those skilled in the art could make various modifications without departing from the scope and spirit of the invention. Various other feature combinations could be implemented by those skilled in the art without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is set forth in the appended claims.

Claims

1. 1. A computer-implemented method for training a model, comprising: performing 304 two-dimensional object detection on the training images to identify objects; Cropping 304 the training images around the object; generating 306 a category-level shape reconstruction using a neural radiance field (NeRF) model; training 402 a normalized coordinate model using the training images and ground truth information from the category-level shape reconstruction.

2. The method of claim 1 , wherein the training images are of a navigable environment within a healthcare facility, and the objects are navigation obstacles.

3. The method of claim 1 , wherein training the normalized coordinate model comprises training a neural network model using a deep learning process.

4. The method of claim 1 , wherein training the coordinate model comprises optimizing a loss function including an occupancy term, a color information term, a LiDAR term, and a depth term.

5. The method of claim 1 , wherein cropping the image comprises excluding information from the training image that is outside a bounding box determined by the object detection.

6. The method of claim 1 , further comprising determining a 3D pose of the object based on normalized coordinates from the normalized coordinate model and an estimated object size.

7. The method of claim 6 , further comprising navigating an environment using the normalized coordinates and the three-dimensional pose of the object in an autonomous vehicle.

8. The method of claim 6 , further comprising training a size estimation model to generate the estimated object size in response to the training images.

9. The method of claim 8 , wherein training the size estimation model includes training datasets derived from a plurality of different regions with different degrees of annotation.

10. The method of claim 9 , wherein at least one region of the training dataset is free of position and orientation annotations but has object size annotations.

11. 1. A system for training a model, comprising: a hardware processor (510); and a memory (540) for storing a computer program, the computer program, when executed by the hardware processor, causing the hardware processor to: performing 304 two-dimensional object detection on the training images to identify objects; Cropping 304 the training images around the object; generating 306 a category-level shape reconstruction using a neural radiance field (NeRF) model; training (402) a normalized coordinate model using the training images and ground truth information from the category-level shape reconstruction.

12. The system of claim 11 , wherein the training images are of a navigable environment within a healthcare facility, and the objects are navigation obstacles.

13. 12. The system of claim 11, wherein the computer program further causes the hardware processor to train a neural network model using deep learning processes.

14. 12. The system of claim 11, wherein the computer program further causes the hardware processor to optimize a loss function including an occupancy term, a color information term, a LiDAR term, and a depth term.

15. 12. The system of claim 11, wherein the computer program further causes the hardware processor to crop the images to exclude information from the training images that is outside a bounding box determined by the object detection.

16. The system of claim 11 , wherein the computer program further causes the hardware processor to determine a three-dimensional pose of the object based on normalized coordinates from the normalized coordinate model and an estimated object size.

17. 17. The system of claim 16, wherein the computer program further causes the hardware processor to navigate an environment using the normalized coordinates and the three-dimensional pose of the object in an autonomous vehicle.

18. 17. The system of claim 16, wherein the computer program further causes the hardware processor to train a size estimation model to generate the estimated object size in response to the training images.

19. 20. The system of claim 18, wherein the computer program further causes the hardware processor to use training data sets derived from a plurality of different regions with different degrees of annotation.

20. 20. The system of claim 19, wherein at least one region of the training dataset is devoid of position and orientation annotations but has object size annotations.

Images