Metapre-training with amplification to adaptively generalize neural network processing for domains

Abstract

A method receives first data and second data generated from first and second domains including a first set of objects and a second set of objects; receiving a first category mark for each object in the first group of objects; and receiving a second category flag for each object in the second set of objects. The method generates a training data set by amplifying the first data and a corresponding first category tag; and locally updating neural network parameters of the model based on the training data set. The method generates a verification dataset by amplifying the second data and a corresponding second category tag; and globally updating the neural network parameters of the model based on the verification data set. The method also generates a plurality of target tags for target data generated from a target domain including a third set of objects after globally updating the neural network parameters of the model based on the verification data set.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 17 / 939,361, filed on September 7, 2022, and entitled “META-PRE-TRAINING WITH AUGMENTATIONS TO GENERALIZE NEURAL NETWORK PROCESSING FOR DOMAIN ADAPTATION,” the disclosure of which is expressly incorporated by reference in its entirety. Technical Field

[0003] Aspects of the present disclosure generally relate to three-dimensional (3D) object detectors. Background Art

[0004] An artificial neural network may include an interconnected group of artificial neurons (e.g., a neuron model). An artificial neural network may be a computing device or represented as a method to be performed by a computing device. A convolutional neural network is a feedforward artificial neural network. A convolutional neural network may include a collection of neurons, each of which has a receptive field and collectively spells out an input space. Convolutional neural networks (CNNs), such as deep convolutional neural networks (DCNs), have numerous applications. Specifically, these neural network architectures are used in various technologies, such as image recognition, speech recognition, acoustic scene classification, keyword detection, autonomous driving, and other classification tasks.

[0005] Three-dimensional (3D) object detection is an important task for autonomous driving. Many areas in autonomous driving, such as prediction, planning, and motion control, rely on a faithful representation of the 3D space around the ego vehicle. In recent years, researchers have exploited high-precision Light Detection and Ranging (LiDAR) point cloud data for accurate 3D object detection. A 3D object detector generates a 3D bounding box that locates each object in the environment. Object detection is challenging because the prediction should be highly accurate, reliable, and importantly, in real time. The current state of the art in 3D object detection is based on deep learning, trained on short driving segments with labeled bounding boxes, which yields high accuracy.

[0006] Prior art 3D object detectors have shown promising accuracy, but tend to overfit to the training domain, causing them to fail in new target environments. Overfitting may occur because other cars may be larger and fewer, roads may be snowy, and / or buildings may look different during training and may be different from buildings in the test environment. Therefore, when training a model with a different dataset, the bounding boxes may have different heights, widths, and lengths, as well as different data distributions. It has been shown that such differences may cause the accuracy of the system to drop by more than 35%. Closing this adaptation gap is a challenge for vehicles that drive themselves freely. It is desirable to generalize neural networks to adapt to different environments. Summary of the invention

[0007] In various aspects of the present disclosure, a computer-implemented method includes: receiving first data generated from a first domain including a plurality of objects, and receiving a first category label for each of the objects. The method also includes: receiving second data generated from a second domain including a second group of objects, and receiving a second category label for each of the second objects. The method also includes: generating a training data set by augmenting the first data and the corresponding first category labels, and locally updating the neural network parameters of the model based on the training data set. The method also includes: generating a validation data set by augmenting the second data and the corresponding second category labels, and globally updating the neural network parameters of the model based on the validation data set. The method also includes: generating a plurality of target labels for target data generated from a target domain including a third group of objects, the target labels being generated by the model after globally updating the neural network parameters of the model based on the validation data set.

[0008] Other aspects of the present disclosure relate to an apparatus having a memory and one or more processors coupled to the memory. The processor is configured to receive first data generated from a first domain including a first group of objects, and receive a first category label for each object in the first object. The processor is also configured to receive second data generated from a second domain including a second group of objects, and receive a second category label for each object in the second object. The processor is further configured to generate a training data set by augmenting the first data and the corresponding first category label, and locally update the neural network parameters of the model based on the training data set. The processor is further configured to generate a validation data set by augmenting the second data and the corresponding second category label, and globally update the neural network parameters of the model based on the validation data set. The processor is further configured to generate a plurality of target labels for target data generated from a target domain including a third group of objects, which are generated by the model after globally updating the neural network parameters of the model based on the validation data set.

[0009] In other aspects of the present disclosure, a non-transitory computer-readable medium having program code recorded thereon is disclosed. The program code is executed by a processor and includes program code for receiving first data generated from a first domain including a first group of objects, and program code for receiving a first category label for each object in the first group of objects. The program code also includes program code for receiving second data generated from a second domain including a second group of objects, and program code for receiving a second category label for each object in the second object. The program code also includes program code for generating a training data set by amplifying the first data and the corresponding first category label, and program code for locally updating the neural network parameters of the model based on the training data set. The program code also includes program code for generating a validation data set by amplifying the second data and the corresponding second category label, and program code for globally updating the neural network parameters of the model based on the validation data set. The program code also includes program code for generating a plurality of target labels for target data generated from a target domain including a third group of objects, which are generated by the model after the neural network parameters of the model are globally updated based on the validation data set.

[0010] Other aspects of the present disclosure relate to an apparatus for a computer-implemented method, comprising: a component for receiving first data generated from a first domain including a first group of objects; and a component for receiving a first category label for each object in the first group of objects. The apparatus also includes a component for receiving second data generated from a second domain including a second group of objects, and a component for receiving a second category label for each object in the second group of objects. The apparatus also includes a component for generating a training data set by augmenting the first data and the corresponding first category label, and a component for locally updating the neural network parameters of the model based on the training data set. The apparatus also includes a component for generating a validation data set by augmenting the second data and the corresponding second category label, and a component for globally updating the neural network parameters of the model based on the validation data set. The apparatus also includes a component for generating a plurality of target labels for target data generated from a target domain including a third group of objects, the target labels being generated by the model after globally updating the neural network parameters of the model based on the validation data set.

[0011] Additional features and advantages of the present disclosure will be described below. It will be appreciated by those skilled in the art that the present disclosure can be easily used as a basis for modifying or designing other structures for implementing the same purpose as the present disclosure. It will also be appreciated by those skilled in the art that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features considered to be characteristic of the present disclosure, both in terms of its organization and method of operation, together with further objects and advantages, will be better understood when the following description is considered in conjunction with the accompanying drawings. However, it is to be expressly understood that each of the accompanying drawings is provided for illustration and description purposes only and is not intended to be a definition of limitations of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The features, nature and advantages of the present disclosure will become more apparent when the detailed description set forth below is read in conjunction with the accompanying drawings, in which like reference characters are identified correspondingly throughout.

[0013] Figure 1 An example implementation of a neural network using a system on a chip (SOC) including a general purpose processor in accordance with certain aspects of the present disclosure is illustrated.

[0014] Figure 2A , Figure 2B and Figure 2C is a diagram illustrating a neural network according to aspects of the present disclosure.

[0015] Figure 2D is a diagram illustrating an exemplary deep convolutional network (DCN) according to aspects of the present disclosure.

[0016] Figure 3 is a block diagram illustrating an exemplary deep convolutional network (DCN) according to aspects of the present disclosure.

[0017] Figure 4 An exemplary representation of LIDAR point cloud data is illustrated.

[0018] Figure 5 Pseudocode for supervised meta-learning task generation for three-dimensional (3D) object detection according to aspects of the present disclosure is illustrated.

[0019] Figure 6 is a block diagram illustrating processing blocks for meta-pretraining with augmentation to generalize neural network processing for domain adaptation in accordance with aspects of the present disclosure.

[0020] Figure 7 is a flow chart illustrating an example process for operating a neural network in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0021] The specific embodiments described below in conjunction with the accompanying drawings are intended as descriptions of various configurations and are not intended to represent the only configurations in which the described concepts can be practiced. In order to provide a comprehensive understanding of the various concepts, the specific embodiments include specific details. However, it is obvious to those skilled in the art that these concepts can be practiced without these specific details. In some instances, in order to avoid obscuring such concepts, well-known structures and components are shown in block diagram form.

[0022] Based on the teachings, those skilled in the art will recognize that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether the aspect is implemented independently of any other aspect of the present disclosure or implemented in combination with any other aspect. For example, a device or a method may be implemented using any number of aspects described. In addition, the scope of the present disclosure is intended to cover such devices or methods that are practiced using other structures, functionality, or structures and functionality that are supplementary to or different from the various aspects of the present disclosure described. It should be understood that any aspect of the present disclosure disclosed may be embodied by one or more elements of the claims.

[0023] The word “exemplary” is used to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0024] Although specific aspects are described, numerous variations and permutations of these aspects fall within the scope of the present disclosure. Although some benefits and advantages of preferred aspects are mentioned, the scope of the present disclosure is not intended to be limited to specific benefits, uses or purposes. On the contrary, various aspects of the present disclosure are intended to be widely applicable to different technologies, system configurations, networks and protocols, some of which are illustrated by way of example in the accompanying drawings and the following description of preferred aspects. The specific embodiments and drawings are merely illustrative of the present disclosure and are not limiting, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0025] Three-dimensional (3D) object detection is an important task for autonomous driving. Many important areas in autonomous driving, such as prediction, planning, and motion control, rely on a faithful representation of the 3D space around the ego vehicle. In recent years, researchers have exploited high-precision light detection and ranging (LiDAR) point cloud data for accurate 3D object detection. Object detection is challenging because the prediction should be highly accurate and in real time.

[0026] Prior art 3D object detectors have shown promising accuracy, but tend to overfit to the training domain, causing them to fail in new target environments. Therefore, when training models with different datasets, bounding boxes may have different heights, widths, and lengths, as well as different data distributions. Closing this adaptation gap is a challenge for autonomously driving vehicles. Aspects of the present disclosure propose a meta-learning based approach to test the generalizability of augmentation.

[0027] Figure 1 An example implementation of a system on chip (SOC) 100 is illustrated, which may include a central processing unit (CPU) 102 or a multi-core CPU configured for meta-pre-training for domain adaptation. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a memory block 118, or may be distributed across multiple blocks. Instructions executed at CPU 102 may be loaded from a program memory associated with CPU 102, or may be loaded from memory block 118.

[0028] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 that may, for example, detect and recognize gestures. In one specific implementation, the NPU 108 is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120, which may include a global positioning system.

[0029] SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into the general processor 102 may include code for receiving first data generated from a first domain including a first group of objects and receiving a first category label for each object in the first object. The general processor 102 may also include code for receiving second data generated from a second domain including a second group of objects and receiving a second category label for each object in the second object. The general processor 102 may also include code for generating a training data set by amplifying the first data and the corresponding first category label and locally updating the neural network parameters of the model based on the training data set. The general processor 102 may also include code for generating a validation data set by amplifying the second data and the corresponding second category label and globally updating the neural network parameters of the model based on the validation data set. The general processor 102 may also include code for generating a plurality of target labels for target data generated from a target domain including a third group of objects, which are generated by the model after the neural network parameters of the model are globally updated based on the validation data set.

[0030] Deep learning architectures can perform object recognition tasks by learning to represent inputs at successively higher levels of abstraction in each layer, thereby building useful feature representations of the input data. In this way, deep learning solves the main bottleneck of traditional machine learning. Before the advent of deep learning, machine learning methods for object recognition problems may rely heavily on human-designed features, possibly in conjunction with shallow classifiers. A shallow classifier can be a two-class linear classifier, for example, in which the weighted sum of feature vector components can be compared to a threshold to predict which class the input belongs to. Human-designed features can be templates or kernels customized for a specific problem domain by engineers with domain expertise. In contrast, although deep learning architectures can learn to represent features similar to those that human engineers may design, they require training. In addition, deep networks can learn to represent and recognize new types of features that humans may not have considered.

[0031] Deep learning architectures can learn hierarchies of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0032] Deep learning architectures perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0033] Neural networks can be designed to have a variety of connection patterns. In a feedforward network, information is passed from a lower layer to a higher layer, where each neuron in a given layer communicates with a neuron in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The loop architecture can help identify patterns that span more than one input data block in the input data blocks that are sequentially delivered to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of a high-level concept can assist in discerning specific low-level features of the input, a network with many feedback connections may be helpful.

[0034] The connections between neural network layers can be fully connected or partially connected. Figure 2A An example of a fully connected neural network 202 is illustrated. In a fully connected neural network 202, a neuron in a first layer may communicate its output to each neuron in a second layer, such that each neuron in the second layer will receive input from each neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strengths may have different values ​​(e.g., 210, 212, 214, and 216). The locally connected connectivity patterns may produce spatially different receptive fields in higher layers because higher layer neurons in a given area may receive input that is tuned through training to properties of a limited portion of the total input to the network.

[0035] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is illustrated. The convolutional neural network 206 may be configured such that the connection strengths associated with the inputs of each neuron in the second layer are shared (e.g., 208). Convolutional neural networks may be well suited for problems where the spatial location of the inputs is meaningful.

[0036] One type of convolutional neural network is a deep convolutional network (DCN). Figure 2D A detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230 (such as a vehicle-mounted camera) is illustrated. The DCN 200 of the current example can be trained to identify traffic signs and numbers provided on traffic signs. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or identifying traffic lights.

[0037] The DCN 200 may be trained by supervised learning. During training, an image (such as an image 226 of a speed limit sign) may be presented to the DCN 200, and a forward pass may then be calculated to produce an output 222. The DCN 200 may include a feature extraction portion and a classification portion. Upon receiving the image 226, a convolutional layer 232 may apply a convolutional kernel (not shown) to the image 226 to generate a first set of feature maps 218. As an example, the convolutional kernel of the convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels are applied to the image 226 at the convolutional layer 232. A convolutional kernel may also be referred to as a filter or a convolutional filter.

[0038] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (such as 14x14) is smaller than the size of the first set of feature maps 218 (such as 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more sets of subsequent feature maps (not shown).

[0039] exist Figure 2D In the example of , the second set of feature maps 220 are convolved to generate a first feature vector 224. In addition, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as "sign", "60", and "100". A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Therefore, the output 222 of the DCN 200 is the probability that the image 226 includes one or more features.

[0040] In this example, the probability of "logo" and "60" in output 222 is higher than the probability of other numbers (such as "30", "40", "50", "70", "80", "90", and "100") in output 222. Before training, the output 222 generated by DCN 200 may be incorrect. Therefore, the error between the output 222 and the target output can be calculated. The target output is the ground truth of image 226 (e.g., "logo" and "60"). The weights of DCN 200 can then be adjusted so that the output 222 of DCN 200 is closer to the target output.

[0041] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease if the weights are adjusted. At the top layer, the gradient may correspond directly to the value of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradients of the higher layers. The weights may then be adjusted to reduce the error. This way of adjusting weights may be referred to as "backward propagation" because it involves a "backward pass" through the neural network.

[0042] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient is close to the true error gradient. This approximation may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, new images may be presented to the DCN, and a forward pass through the network may produce an output 222 that may be considered an inference or prediction of the DCN.

[0043] Deep belief network (DBN) is a probabilistic model including multiple layers of hidden nodes. DBN can be used to extract hierarchical representations of training data sets. DBN can be obtained by stacking the layers of restricted Boltzmann machine (RBM). RBM is a type of artificial neural network that can learn probability distributions through a set of inputs. Because RBM can learn probability distributions without information about the category to which each input should be classified, RBM is generally used for unsupervised learning. Using an unsupervised and supervised hybrid paradigm, the bottom RBM of DBN can be trained in an unsupervised manner and can be used as a feature extractor, while the top RBM can be trained in a supervised manner (on the joint distribution of the input and target class from the previous layer) and can be used as a classifier.

[0044] A deep convolutional network (DCN) is a network that is a convolutional network configured with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many examples and are used to modify the weights of the network by using a gradient descent method.

[0045] The DCN may be a feed-forward network. In addition, as described above, the connections from a neuron in the first layer of the DCN to a set of neurons in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of the DCN may be used for fast processing. For example, the computational burden of the DCN may be much smaller than that of a similarly sized neural network that includes loops or feedback connections.

[0046] The processing of each layer of the convolutional network can be considered as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green and blue channels of a color image, the convolutional network trained on the input can be considered to be three-dimensional, with two spatial dimensions along the axis of the image and the third dimension capturing color information. The output of the convolutional connection can be considered to form a feature map in a subsequent layer, where each element in the feature map (e.g., 220) receives input from a certain range of neurons in the previous layer (e.g., feature map 218) and from each channel in the multiple channels. The values ​​in the feature map can be further processed with nonlinearity (such as correction, max(0,x)). The values ​​from adjacent neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction. Normalization corresponding to whitening can also be applied by lateral inhibition between neurons in the feature map.

[0047] The performance of deep learning architectures can increase as more labeled data points become available or as computing power increases. Modern deep neural networks are often trained with computing resources that are thousands of times what was available to a typical researcher just fifteen years ago. New architectures and training paradigms can further improve the performance of deep learning. Rectified linear units can reduce the training problem known as vanishing gradients. New training techniques can reduce overfitting and therefore enable larger models to achieve better generalization. Encapsulation techniques can extract data within a given receptive field and further improve overall performance.

[0048] Figure 3 is a block diagram illustrating a deep convolutional network 350. Based on connectivity and weight sharing, the deep convolutional network 350 may include multiple different types of layers. Figure 3 As shown, the deep convolution network 350 includes convolution blocks 354A and 354B. Each of the convolution blocks 354A and 354B may be configured with a convolution layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360.

[0049] The convolution layer 356 may include one or more convolution filters that may be applied to the input data to generate a feature map. Although only two convolution blocks 354A, 354B are shown, the present disclosure is not limited thereto, but instead, any number of convolution blocks 354A, 354B may be included in the deep convolution network 350 according to design preferences. The normalization layer 358 may normalize the output of the convolution filter. For example, the normalization layer 358 may provide whitening or lateral suppression. The maximum pooling layer 360 may provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0050] For example, the parallel filter bank of the deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the deep convolutional network 350 may access other processing blocks that may be present on the SOC 100, such as the sensor processor 114 and the navigation module 120 dedicated to sensors and navigation, respectively.

[0051] The deep convolutional network 350 may also include one or more fully connected layers 362 (FC1 and FC2). The deep convolutional network 350 may also include a logistic regression (LR) layer 364. There are weights (not shown) to be updated between each layer 356, 358, 360, 362, 364 of the deep convolutional network 350. The output of each of these layers (e.g., 356, 358, 360, 362, 364) can be used as an input to a subsequent layer of these layers (e.g., 356, 358, 360, 362, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from the input data 352 (e.g., image, audio, video, sensor data, and / or other input data) supplied at the first convolutional block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 can be a set of probabilities, where each probability is a probability that the input data includes a feature from a set of features.

[0052] Three-dimensional (3D) object detection includes prediction, planning, and motion control, which rely on a faithful representation of the 3D space around the ego vehicle. In recent years, researchers have exploited high-precision light detection and ranging (LiDAR) point cloud data for accurate 3D object detection. A 3D object detector generates a 3D bounding box that locates each object in the environment that reflects the LiDAR light waves. The 3D bounding box can be a cubic bounding box. Object detection is challenging because the prediction should be highly accurate, reliable, and in real time.

[0053] Figure 4An exemplary representation of LIDAR point cloud data is illustrated. Figure 4 , side view 402 illustrates point cloud data and bounding boxes 404 generated by a 3D object detector when a vehicle moves in a direction from left to right in an environment. In side view 402, only two of bounding boxes 404 are indicated by leading lines. Bird's eye view 406 also illustrates point cloud data from the same environment of a vehicle moving from left to right. In bird's eye view 406, only two of bounding boxes 404 are indicated by leading lines.

[0054] State-of-the-art 3D object detectors tend to overfit to the training domain, causing them to fail in new target environments, resulting in bounding boxes with different heights, widths, and lengths and different data distributions when the model is trained with different datasets. It is desirable to generalize neural networks to adapt to different environments.

[0055] Scaling-based augmentation (such as resizing bounding boxes) has been used to pre-train robust 3D object detectors. However, the generalizability during pre-training has not been evaluated. Various aspects of the present disclosure propose a meta-learning-based approach to test the generalizability of augmentation.

[0056] Figure 5 Pseudocode 500 is illustrated for supervised meta-learning task generation for three-dimensional (3D) object detection according to aspects of the present disclosure. Pseudocode 500 illustrates one example of techniques for implementing the present disclosure with respect to 3D object detection. The process operates on any number of datasets using source domain labeled data S. Figure 5 In the examples of , the source datasets are referred to as the Lyft dataset, the Waymo dataset, and the Argoverse dataset, but any labeled dataset may be used. Argoverse is a registered trademark of Argo AI, LLC of Pittsburgh, Pennsylvania, Lyft is a registered trademark of Lyft Inc. of San Francisco, California, and Waymo is a registered trademark of Waymo, LLC of Mountain View, California.

[0057] The source domain labeled data S includes the input point cloud data from each dataset And the output mark The source domain labeled data S may be generated from the first domain and include multiple objects, such as cars, trucks, cyclists, etc. The class count N corresponds to the type of the object. For example, if the first source domain labeled data S only includes the car class and the cyclist class, then N=2. Parameter N MB Corresponding to the meta-batch size, the parameter N s corresponds to the number of updates, and the parameters α and β correspond to the step size hyperparameters. A random object scaling (ROS) strategy should be defined. For example, objects can be randomly scaled. An object can be 4x 3x 2 pixels and then scaled to 3x 4x 2 pixels. The number of samples K used for training (tr) and the number of samples used for validation K (val) can be defined for meta-learning.

[0058] Based on this information, the meta-learning task generates an output that is an updated object detection model to be used for the target domain. The updated object detection model includes the final set of parameters. Figure 5 At line 502, pseudo code 500 randomizes an initial parameter set θ for an object detection model. Next, at line 504, for a batch size N MB For each sample in , process sample N annotates a 3D bounding box with each of the different labels from the source domain labeled data S. That is, bounding boxes are sampled for each object category (cars and cyclists in this example, so that N=2). For example, each bounding box may have X, Y, and Z coordinates, yaw angle, width, length, and height. At line 506, task T i For each sample in the batch of source domain labeled data S, it can be defined using a sample corresponding to the input data P and the output label L.

[0059] For the batch size N MB After completing the sampling and task definition, at line 508, for each task T i Generate training dataset D i aug,(tr) The training dataset is augmented for each task T by using the random object scaling (ROS) technique based on the selected i That is, the process generates a training dataset D by augmenting the data sampled from the source domain labeled data S and the corresponding category labels. i aug,(tr) .

[0060] After the training dataset is generated, the training dataset is used to update the parameters of the model. More specifically, at line 510, local updates are performed by first defining a local neural network parameter set θ' based on the loss from the augmented training set. At line 512, for each task T iThe gradient of a local set of parameters is evaluated based on the loss function L. In this context, local means Figure 5 The inner loop of the pseudo code 500 shown. When receiving the point cloud P as input, the function f(θ') is the 3D bounding box output of the network. The parameters θ are the parameters of the network. The adaptive parameters can be calculated based on the gradient using the stochastic gradient descent (SGD) process at line 514. The foregoing process is repeated as an inner loop. The outer loop process will now be described.

[0061] The outer loop uses the validation dataset or the second dataset to perform a global update on the parameters of the neural network model. In this context, global refers to Figure 5 The pseudo code 500 is shown in the outer loop. More specifically, at line 516, the validation data set D i aug,(val) Based on the number of samples K used for validation (val) Generated by performing a random object scaling (ROS) technique on each task in the validation set. Each task may correspond to an object and class label from the validation data set. The random object scaling distribution used for the validation set may be different from the random object scaling technique used for the training set to improve generalization. After the validation data set is augmented, at line 518, the network parameters are globally updated based on the validation data set. More specifically, the loss is collected for the validation set, and the neural network model parameters are updated using the loss collected from meta-training.

[0062] After the model has been globally updated, the model can receive input from the target dataset to generate labels, such as bounding boxes. Due to the meta-learning during pre-training, the model generalizes well to the target environment to predict accurate bounding boxes.

[0063] Figure 6 is a block diagram illustrating processing blocks for meta-pretraining with augmentation to generalize neural network processing for domain adaptation in accordance with aspects of the present disclosure. Figure 6 The example of illustrates pre-training for improving generalizability through meta-training. Figure 6 In the example, the point cloud input data from the source dataset 602 undergoes some type of augmentation (Aug) such as random object scaling (ROS) at block 604 to generate augmented data

[0064] exist Figure 6 In the example, there are two categories. Therefore, each task T i includes two categories of augmented data. Therefore, task T i For the training dataset for the first task T1, the augmented data includes two samples and For the nth task Tn , the amplification data includes two samples and Each validation set includes three samples, called three-shot. For the first task T1, the expanded samples include and Represents three different augmentations for verification. Similarly, for the nth task T n , amplified samples include and In the meta-learning task distribution, different tasks can represent different augmentation strategies or scopes, so the network can learn to generalize to different augmentations.

[0065] In some aspects, the expanded validation dataset Based on the target dataset The noise ε is calculated by adding a certain amount of noise ε, as seen at 606. The noise ε can come from a normal distribution with mean 0 and a certain variance σ. The variance is usually fixed as a constant throughout the training process (it is a hyperparameter), but can be increased by a multiplication factor at certain time intervals. Noise can be added to each value of the target data, for example, the X, Y, Z, height, width, length, and angle of the bounding box.

[0066] The output from the meta-task processing is a network function F 608 that is based on the point cloud input data 602 to generate a bounding box. After training the model based on the training data and the validation data, the global loss L is determined. det .

[0067] A network function F 608 is provided to generate a bounding box. The network function F receives unlabeled target point cloud data P from the target environment. i target 610. Based on the received point cloud data P i target 610, network function F 608 generates a bounding box from the target dataset

[0068] Methods for pre-training according to aspects of the present disclosure generally generalize well to new scenarios better than traditional augmentation methods because generalizability during pre-training is evaluated. Because the method uses meta-learning during pre-training, the method can quickly adapt to few-shot fine-tuning. The method also generalizes better from a few samples than traditional augmentation-based pre-training.

[0069] Figure 7 7 is a flow chart illustrating an example process 700 for meta-pretraining with augmentation to generalize neural network processing for domain adaptation in accordance with aspects of the present disclosure. Figure 7 As shown, in some aspects, process 700 may include: receiving first data generated from a first domain including a first set of objects (block 702). In some aspects, process 700 may include: receiving a first category label for each object in the first set of objects (block 704). For example, the first domain may be a first source domain including first source domain label data. The first source domain label data may include input point cloud data and output labels.

[0070] In some aspects, process 700 may include receiving second data generated from a second domain including a second set of objects (block 706). Process 700 may include receiving a second category label for each object in the second plurality of objects (block 708). For example, the second domain may be a second source domain including second source domain label data. The second source domain label data may also include input point cloud data and output labels.

[0071] In some aspects, process 700 may include generating a training data set by augmenting the first data and the corresponding first category labels (block 710). For example, the augmentation may include a random object scaling strategy. Process 700 may include locally updating neural network parameters of a model based on the training data set (block 712). For example, an updated object detection model may be generated for use with a target domain.

[0072] In some aspects, process 700 may include generating a validation data set by augmenting the second data and the corresponding second class labels (block 714). For example, the validation data set may be generated by augmenting samples in the task based on a selected random object scaling technique. Augmentation may include scaling a set of bounding boxes.

[0073] In some aspects, process 700 may include globally updating the neural network parameters of the model based on the validation dataset (block 716). The global update may occur after augmenting the validation dataset. For example, the loss of the validation set may be collected, and the neural network model parameters may be updated using the loss collected from meta-training.

[0074] In some aspects, process 700 may include generating target labels for target data generated from a target domain including a third set of objects. The target labels are generated by the model after globally updating the neural network parameters of the model based on the validation dataset (block 718). Due to meta-learning during pre-training, the model generalizes well to the target environment to predict accurate bounding boxes.

[0075] Although the present disclosure is mainly described for 3D object detection, the technology is not limited thereto. The technology can be extended to any object recognition task that specifies augmentation during pre-training. That is, the technology is applicable to any general model that adapts to another environment so that the source data set and the destination data set have different distributions. For example, speaker verification can adopt a model of the source user, which is pre-trained to better adapt to the target user with enrollment data. The model classifies the target speaker and the imposter speaker, even if there is a large difference between the source speaking style and the target speaking style. The speaker verification model can also be applied to mobile and home personal assistant systems. The image verification model can be applied to biometrics, such as face, iris and handwriting style. The general model learned from the source data can use the augmentation technology of the present disclosure. The fingerprint anti-spoofing model can be used to distinguish between real fingerprints and spoofed fingerprints when introducing new spoofed fingerprints with large domain differences from existing fingerprints. Such a model will also benefit from the technology of the present disclosure. Personalized advanced driver assistance system (ADAS) pre-trains a model for classifying the degree of driver involvement based on a limited number of demonstrations. In this regard, the augmentation technology of the present disclosure can also assist in domain adaptation.

[0076] Example aspects

[0077] Aspect 1: A computer-implemented method comprising: receiving first data generated from a first domain including a first plurality of objects; receiving a first category label for each object in the first plurality of objects; receiving second data generated from a second domain including a second plurality of objects; receiving a second category label for each object in the second plurality of objects; generating a training data set by augmenting the first data and the corresponding first category labels; locally updating a neural network parameter of a model based on the training data set; generating a validation data set by augmenting the second data and the corresponding second category labels; globally updating the neural network parameter of the model based on the validation data set; and generating a plurality of target labels for target data generated from a target domain including a third plurality of objects, the target labels being generated by the model after globally updating the neural network parameter of the model based on the validation data set.

[0078] Aspect 2: A method according to Aspect 1, wherein the model comprises a three-dimensional (3D) object detection model, and the multiple target tags comprise multiple target bounding boxes, and the multiple target bounding boxes locate each object in the third multiple objects defined by the target data.

[0079] Aspect 3: The method according to aspect 1 or 2, wherein augmenting the first data and the corresponding first category labels comprises scaling a first set of bounding boxes, and augmenting the second data and the corresponding second category labels comprises scaling a second set of bounding boxes.

[0080] Aspect 4: The method according to any one of the preceding aspects, wherein a first distribution used to amplify the first data and the corresponding first category labels is different from a second distribution used to amplify the second data and the corresponding second category labels.

[0081] Aspect 5: The method according to any one of the preceding aspects, wherein the local update occurs within an inner loop based on each class type of the first class label, and the global update occurs within an outer loop based on each class type of the second class label.

[0082] Aspect 6: A method according to any one of the preceding aspects, wherein the first domain includes a first environment, the first data includes first point cloud data, the second domain includes a second environment, the second data includes second point cloud data, the target domain includes a target environment, and the target data includes target point cloud data.

[0083] Aspect 7: An apparatus comprising: a memory; and at least one processor coupled to the memory, the at least one processor being configured to: receive first data generated from a first domain comprising a first plurality of objects; receive a first category label for each object in the first plurality of objects; receive second data generated from a second domain comprising a second plurality of objects; receive a second category label for each object in the second plurality of objects; generate a training data set by augmenting the first data and the corresponding first category labels; locally update the neural network parameters of a model based on the training data set; generate a validation data set by augmenting the second data and the corresponding second category labels; globally update the neural network parameters of the model based on the validation data set; and generate a plurality of target labels for target data generated from a target domain comprising a third plurality of objects, the target labels being generated by the model after globally updating the neural network parameters of the model based on the validation data set.

[0084] Aspect 8: An apparatus according to Aspect 7, wherein the model comprises a three-dimensional (3D) object detection model, and the plurality of target tags comprises a plurality of target bounding boxes, and the plurality of target bounding boxes locate each object in the third plurality of objects defined by the target data.

[0085] Aspect 9: An apparatus according to Aspect 7 or 8, wherein the at least one processor is further configured to amplify the first data and the corresponding first category label by scaling the first set of bounding boxes, and the at least one processor is further configured to amplify the second data and the corresponding second category label by scaling the second set of bounding boxes.

[0086] Aspect 10: The apparatus according to any one of aspects 7-9, wherein a first distribution used to amplify the first data and the corresponding first category label is different from a second distribution used to amplify the second data and the corresponding second category label.

[0087] Aspect 11: An apparatus according to any one of Aspects 7-10, wherein the at least one processor is further configured to locally update the neural network parameters based on each category type of the first category label within an inner loop, and the at least one processor is further configured to globally update the neural network parameters based on each category type of the second category label within an outer loop.

[0088] Aspect 12: An apparatus according to any one of Aspects 7-11, wherein the first domain includes a first environment, the first data includes first point cloud data, the second domain includes a second environment, the second data includes second point cloud data, the target domain includes a target environment, and the target data includes target point cloud data.

[0089] Aspect 13: A non-transitory computer-readable medium having program code recorded thereon, the program code being executed by a processor and comprising: program code for receiving first data generated from a first domain comprising a first plurality of objects; program code for receiving a first category label for each of the first plurality of objects; program code for receiving second data generated from a second domain comprising a second plurality of objects; program code for receiving a second category label for each of the second plurality of objects; program code for generating a training data set by augmenting the first data and the corresponding first category labels; program code for locally updating the neural network parameters of a model based on the training data set; program code for generating a validation data set by augmenting the second data and the corresponding second category labels; program code for globally updating the neural network parameters of the model based on the validation data set; and program code for generating a plurality of target labels for target data generated from a target domain comprising a third plurality of objects, the target labels being generated by the model after globally updating the neural network parameters of the model based on the validation data set.

[0090] Aspect 14: A non-transitory computer-readable medium according to Aspect 13, wherein the model comprises a three-dimensional (3D) object detection model, and the plurality of target tags comprises a plurality of target bounding boxes, and the plurality of target bounding boxes locate each object in the third plurality of objects defined by the target data.

[0091] Aspect 15: A non-transitory computer-readable medium according to Aspect 13 or 14, wherein the program code for generating a training data set by amplifying the first data and the corresponding first category labels also includes program code for scaling a first set of bounding boxes, and the program code for generating a validation data set by amplifying the second data and the corresponding second category labels also includes program code for scaling a second set of bounding boxes.

[0092] Aspect 16: The non-transitory computer-readable medium of any one of aspects 13-15, wherein a first distribution used to amplify the first data and the corresponding first category labels is different from a second distribution used to amplify the second data and the corresponding second category labels.

[0093] Aspect 17: A non-transitory computer-readable medium according to any one of Aspects 13-16, wherein the program code for local update is based on each category type of the first category tag within an inner loop, and the program code for global update is based on each category type of the second category tag within an outer loop.

[0094] Aspect 18: A non-transitory computer-readable medium according to any one of Aspects 13-17, wherein the first domain includes a first environment, the first data includes first point cloud data, the second domain includes a second environment, the second data includes second point cloud data, the target domain includes a target environment, and the target data includes target point cloud data.

[0095] Aspect 19: An apparatus for a computer-implemented method, comprising: a component for receiving first data generated from a first domain including a first plurality of objects; a component for receiving a first category label for each object in the first plurality of objects; a component for receiving second data generated from a second domain including a second plurality of objects; a component for receiving a second category label for each object in the second plurality of objects; a component for generating a training data set by augmenting the first data and the corresponding first category label; a component for locally updating the neural network parameters of a model based on the training data set; a component for generating a validation data set by augmenting the second data and the corresponding second category label; a component for globally updating the neural network parameters of the model based on the validation data set; and a component for generating a plurality of target labels for target data generated from a target domain including a third plurality of objects, the target labels being generated by the model after globally updating the neural network parameters of the model based on the validation data set.

[0096] Aspect 20: An apparatus according to Aspect 19, wherein the model comprises a three-dimensional (3D) object detection model, and the plurality of target tags comprises a plurality of target bounding boxes, and the plurality of target bounding boxes locate each object in the third plurality of objects defined by the target data.

[0097] Aspect 21: An apparatus according to Aspect 19 or 20, wherein the component for generating a training dataset by augmenting the first data and the corresponding first category labels includes a component for scaling a first set of bounding boxes, and the component for generating a validation dataset by augmenting the second data and the corresponding second category labels includes a component for scaling a second set of bounding boxes.

[0098] Aspect 22: The apparatus according to any one of aspects 19-21, wherein a first distribution used to amplify the first data and the corresponding first category label is different from a second distribution used to amplify the second data and the corresponding second category label.

[0099] Aspect 23: An apparatus according to any one of aspects 19-22, wherein the component for local updating is based on each category type of the first category label in an inner loop, and the component for global updating is based on each category type of the second category label in an outer loop.

[0100] Aspect 24: An apparatus according to any one of Aspects 19-23, wherein the first domain includes a first environment, the first data includes first point cloud data, the second domain includes a second environment, the second data includes second point cloud data, the target domain includes a target environment, and the target data includes target point cloud data.

[0101] The various operations of the above methods may be performed by any suitable components capable of performing the corresponding functions. These components may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. In general, where operations are illustrated in the accompanying drawings, these operations may have corresponding paired components plus function components with similar numbers.

[0102] As used, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, etc. Additionally, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Furthermore, "determining" may include resolving, selecting, choosing, establishing, etc.

[0103] As used, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.

[0104] The various illustrative logical blocks, modules, and circuits described in conjunction with the present disclosure may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0105] The steps or algorithms of the methods described in conjunction with the present disclosure may be directly embodied in hardware, software modules executed by a processor, or a combination of the two. The software module may reside in any form of storage medium known in the art. Some examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs, and the like. The software module may include a single instruction, perhaps multiple instructions, and may be distributed over several different code segments, distributed between different programs, and distributed across multiple storage media. The storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integral with the processor.

[0106] The disclosed methods include one or more steps or actions for implementing the described methods. The steps and / or actions of the methods may be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0107] The functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system in a device. The processing system may be implemented using a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnecting buses and bridges. The bus may link various circuits together, including a processor, a machine-readable medium, and a bus interface. The bus interface may be used to connect a network adapter, etc., to the processing system via the bus. The network adapter may be used to implement signal processing functions. For some aspects, a user interface (e.g., a keypad, a display, a mouse, a joystick, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be described further.

[0108] The processor may be responsible for managing the bus and general processing, including executing software stored on the machine-readable medium. The processor may be implemented using one or more general-purpose processors and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuits that can execute software. Software should be broadly interpreted as meaning instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or other. By way of example, a machine-readable medium may include a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a disk, an optical disk, a hard disk drive, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be embodied in a computer program product. The computer program product may include packaging materials.

[0109] In hardware implementation, machine-readable media can be part of a processing system separate from a processor. However, as will be readily appreciated by those skilled in the art, machine-readable media or any part thereof can be outside a processing system. By way of example, machine-readable media can include a transmission line, a carrier modulated by data, and / or a computer product separate from a device, all of which can be accessed by a processor through a bus interface. Alternatively or in addition, machine-readable media or any part thereof can be integrated into a processor, such as with a cache and / or a general register stack. Although the various components discussed can be described as having specific locations, such as local components, they can also be configured in various ways, such as certain components being configured as a part of a distributed computing system.

[0110] The processing system can be configured as a general processing system having one or more microprocessors providing processor functionality and an external memory providing at least a portion of a machine-readable medium, all of which are linked together with other supporting circuit systems through an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the described neuron model and neural system model. As another alternative, the processing system may be implemented with an application specific integrated circuit (ASIC) having a processor, a bus interface, a user interface, support circuits, and at least a portion of a machine-readable medium integrated in a single chip, or with one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, discrete hardware components, or any other suitable circuits, or circuits capable of performing the various functionalities described throughout this disclosure. Those skilled in the art will recognize how to best implement the functionality of the processing system depending on the specific application and the overall design constraints imposed on the entire system.

[0111] The machine-readable medium may include multiple software modules. These software modules include instructions that cause the processing system to perform various functions when executed by the processor. The software module may include a sending module and a receiving module. Each software module may reside in a single storage device or be distributed across multiple storage devices. By way of example, when a triggering event occurs, the software module may be loaded from a hard drive into a RAM. During the execution of the software module, the processor may load some of the instructions into a cache to increase access speed. Then one or more cache lines may be loaded into a general register stack for execution by the processor. When the functionality of a software module is mentioned below, it will be understood that such functionality is implemented by the processor when executing instructions from the software module. In addition, it should be understood that various aspects of the present disclosure produce improvements in the functionality of processors, computers, machines, or other systems that implement such aspects.

[0112] If implemented in software, each function may be stored as one or more instructions or codes on or sent through a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any media that facilitates the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is also appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, optical cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwaves, the coaxial cable, optical cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. As used, disks and optical disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and optical disks. Optical disks, where magnetic disks typically reproduce data magnetically, and optical disks reproduce data optically with lasers. Thus, in some aspects, computer-readable media may include non-transitory computer-readable media (e.g., tangible media). Additionally, for other aspects, computer-readable media may include transient computer-readable media (e.g., signals). Combinations of the above should also be included within the scope of computer-readable media.

[0113] Thus, some aspects may include a computer program product for performing the operations presented. For example, such a computer program product may include a computer-readable medium having stored (and / or encoded) thereon instructions that can be executed by one or more processors to perform the described operations. For some aspects, the computer program product may include packaging materials.

[0114] In addition, it should be understood that the modules and / or other appropriate components for performing the described methods and techniques can be downloaded and / or otherwise obtained by the user terminal and / or base station where applicable. For example, such a device can be coupled to a server to facilitate the transmission of the components for performing the described methods. Alternatively, the various methods described can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a compact disc (CD) or a floppy disk) so that once the storage component is coupled to or provided to the device, the user terminal and / or base station can obtain the various methods. In addition, any other suitable technology suitable for providing the described methods and techniques to the device can be used.

[0115] It is to be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. A computer-implemented method comprising: receiving first data generated from a first domain including a first plurality of objects; receiving a first category label for each object in the first plurality of objects; receiving second data generated from a second domain including a second plurality of objects; receiving a second category label for each object in the second plurality of objects; generating a training data set by amplifying the first data and corresponding first category labels; Locally updating the neural network parameters of the model based on the training data set; generating a validation data set by amplifying the second data and corresponding second category labels; Globally updating the neural network parameters of the model based on the validation dataset; as well as A plurality of target labels are generated for target data generated from a target domain including a third plurality of objects, the target labels being generated by the model after globally updating the neural network parameters of the model based on the validation dataset.

2. The method of claim 1 , wherein the model comprises a three-dimensional (3D) object detection model, the plurality of target tags comprising a plurality of target bounding boxes that locate each object in the third plurality of objects defined by the target data. 3 . The method of claim 2 , wherein augmenting the first data and the corresponding first category labels comprises scaling a first set of bounding boxes, and augmenting the second data and the corresponding second category labels comprises scaling a second set of bounding boxes. 4 . The method of claim 1 , wherein a first distribution used to amplify the first data and the corresponding first category labels is different from a second distribution used to amplify the second data and the corresponding second category labels. 5 . The method of claim 1 , wherein the local updating occurs within an inner loop based on each class type of the first class label, and the global updating occurs within an outer loop based on each class type of the second class label.

6. The method of claim 1, wherein the first domain comprises a first environment, the first data comprises first point cloud data, the second domain comprises a second environment, the second data comprises second point cloud data, the target domain comprises a target environment, and the target data comprises target point cloud data.

7. A device comprising: Memory; and at least one processor coupled to the memory, the at least one processor configured to: receiving first data generated from a first domain including a first plurality of objects; receiving a first category label for each object in the first plurality of objects; receiving second data generated from a second domain including a second plurality of objects; receiving a second category label for each object in the second plurality of objects; generating a training data set by amplifying the first data and corresponding first category labels; Locally updating the neural network parameters of the model based on the training data set; generating a validation data set by amplifying the second data and corresponding second category labels; Globally updating the neural network parameters of the model based on the validation dataset; as well as A plurality of target labels are generated for target data generated from a target domain including a third plurality of objects, the target labels being generated by the model after globally updating the neural network parameters of the model based on the validation dataset.

8. The apparatus of claim 7, wherein the model comprises a three-dimensional (3D) object detection model, the plurality of target tags comprising a plurality of target bounding boxes locating each object in the third plurality of objects defined by the target data.

9. The apparatus of claim 8, wherein the at least one processor is further configured to augment the first data and the corresponding first category label by scaling a first set of bounding boxes, and the at least one processor is further configured to augment the second data and the corresponding second category label by scaling a second set of bounding boxes.

10. The apparatus of claim 7, wherein a first distribution used to amplify the first data and the corresponding first category label is different from a second distribution used to amplify the second data and the corresponding second category label.

11. The apparatus of claim 7, wherein the at least one processor is further configured to locally update the neural network parameters based on each class type of the first class label within an inner loop, and the at least one processor is configured to globally update based on each class type of the second class label within an outer loop.

12. The apparatus of claim 7, wherein the first domain comprises a first environment, the first data comprises first point cloud data, the second domain comprises a second environment, the second data comprises second point cloud data, the target domain comprises a target environment, and the target data comprises target point cloud data.

13. A non-transitory computer readable medium having program code recorded thereon, the program code being executed by a processor and comprising: program code for receiving first data generated from a first domain comprising a first plurality of objects; program code for receiving a first category label for each object in the first plurality of objects; program code for receiving second data generated from a second domain comprising a second plurality of objects; program code for receiving a second category label for each object in the second plurality of objects; program code for generating a training data set by augmenting the first data and corresponding first category labels; program code for locally updating neural network parameters of a model based on the training data set; program code for generating a validation data set by amplifying the second data and corresponding second class labels; program code for globally updating the neural network parameters of the model based on the validation data set; and Program code for generating a plurality of target labels for target data generated from a target domain comprising a third plurality of objects, the target labels generated by the model after globally updating the neural network parameters of the model based on the validation data set.

14. The non-transitory computer-readable medium of claim 13, wherein the model comprises a three-dimensional (3D) object detection model, the plurality of target tags comprising a plurality of target bounding boxes locating each object in the third plurality of objects defined by the target data.

15. The non-transitory computer-readable medium of claim 14, wherein the program code for generating the training dataset by augmenting the first data and the corresponding first category labels further comprises program code for scaling a first set of bounding boxes, and the program code for generating the validation dataset by augmenting the second data and the corresponding second category labels further comprises program code for scaling a second set of bounding boxes.

16. The non-transitory computer readable medium of claim 13, wherein a first distribution used to amplify the first data and the corresponding first category labels is different from a second distribution used to amplify the second data and the corresponding second category labels.

17. The non-transitory computer readable medium of claim 13, wherein the program code for locally updating updates within an inner loop based on each category type of the first category tag, and the program code for globally updating updates within an outer loop based on each category type of the second category tag.

18. The non-transitory computer-readable medium of claim 13, wherein the first domain comprises a first environment, the first data comprises first point cloud data, the second domain comprises a second environment, the second data comprises second point cloud data, the target domain comprises a target environment, and the target data comprises target point cloud data.

19. An apparatus for a computer-implemented method, comprising: means for receiving first data generated from a first domain comprising a first plurality of objects; means for receiving a first category label for each object in the first plurality of objects; means for receiving second data generated from a second domain comprising a second plurality of objects; means for receiving a second category label for each object in the second plurality of objects; means for generating a training data set by augmenting said first data and corresponding first category labels; means for locally updating neural network parameters of a model based on the training data set; means for generating a validation data set by amplifying said second data and corresponding second class labels; means for globally updating the neural network parameters of the model based on the validation dataset; and means for generating a plurality of target labels for target data generated from a target domain comprising a third plurality of objects, the target labels being generated by the model after globally updating the neural network parameters of the model based on the validation data set.

20. The apparatus of claim 19, wherein the model comprises a three-dimensional (3D) object detection model, the plurality of target tags comprising a plurality of target bounding boxes locating each object in the third plurality of objects defined by the target data.

21. The apparatus of claim 20, wherein the component for generating the training dataset by augmenting the first data and the corresponding first category labels comprises a component for scaling a first set of bounding boxes, and the component for generating the validation dataset by augmenting the second data and the corresponding second category labels comprises a component for scaling a second set of bounding boxes.

22. The apparatus of claim 19, wherein a first distribution used to amplify the first data and the corresponding first category labels is different from a second distribution used to amplify the second data and the corresponding second category labels.

23. The apparatus of claim 19, wherein the means for local updating is within an inner loop based on each class type of the first class label, and the means for global updating is within an outer loop based on each class type of the second class label.

24. The apparatus of claim 19, wherein the first domain comprises a first environment, the first data comprises first point cloud data, the second domain comprises a second environment, the second data comprises second point cloud data, the target domain comprises a target environment, and the target data comprises target point cloud data.

Images