Computer-implemented method for 3D localization of objects based on image and depth data
By fusing image and depth data through convolutional neural networks, the problem of the inability to effectively determine the reference point of 3D objects in existing technologies is solved, and efficient improvement of 3D positioning accuracy is achieved on small datasets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2021-04-29
- Publication Date
- 2026-06-02
AI Technical Summary
Existing 2D object detection algorithms cannot effectively utilize depth data to determine the 3D reference point of an object in three-dimensional space, and large-scale real-world training datasets are scarce and expensive, resulting in poor performance of detection models on small datasets.
A convolutional neural network is used to extract and fuse features from image and depth data, and the model is trained with 3D localization data. Synthetic data augmentation and transfer learning are used to adjust the ratio of image and depth data to maintain the object's metric ratio. 3D localization is performed using a pre-trained YOLO v3 model.
It enables efficient training on small datasets and can accurately determine the 3D reference points of objects, such as the centroid or centroid, thereby improving the 3D object localization accuracy in robotic applications.
Smart Images
Figure CN113569896B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to image processing, and more particularly to the field of 3D object localization based on image and depth data. Background Technology
[0002] For service, household, and industrial robots that need to interact with their environment, computer-generated detection of individuals and objects in 3D space is essential. Recent advances have largely solved the problem of single-modal 2D object recognition based on 2D images. Focusing on 2D detection, different strategies for fusing multiple sensor modalities, such as image and depth data, have also been investigated.
[0003] However, a specific problem with these detection models is that if an individual / object has a non-negligible spatial size in 3D space, it is necessary to locate the 3D object reference point, such as the 3D centroid, represented by the image and depth data. Furthermore, large-scale real-world training datasets using 3D ground truth annotations for 3D detection are scarce and expensive, which necessitates that detection models be specifically configured to achieve excellent detection quality when trained on smaller training datasets.
[0004] For image-based object recognition, high-performance algorithms such as YOLO v3 Level 1 detection are well-known. The YOLO v3 algorithm allows for the acquisition of object contour data, such as bounding boxes representing the 2D regions in which the detected individual / object resides. However, since the individual / object to be detected may also extend significantly in the depth dimension, many applications, such as in robotics, require determining 3D object reference points within the object's boundaries, such as the centroid or 3D centroid of the individual / object. Conventional algorithms for 2D object detection do not provide such 3D object reference points. They do not fully utilize the additional depth data that can be acquired simultaneously using existing sensors. Summary of the Invention
[0005] According to the present invention, a computer-implemented method for 3D localization of an individual or object using a detection model based on image data and depth data is provided, a computer-implemented method for training such a detection model is provided, and a system and a moving subject according to the present invention are provided.
[0006] According to a first aspect, a computer-implemented method for 3D object localization based on image data and depth data indicating distance information of pixels in an image represented by the image data is provided. The method applies a convolutional neural network having a first set of consecutive layers, a second set of consecutive layers, and a third set of consecutive layers, each layer being configured to have one or more filters. The convolutional neural network is trained to associate object recognition / classification and corresponding 3D localization data for the recognized / classified object with image data items and depth data items. The method includes the following steps:
[0007] - The first set of layers of the convolutional neural network is used to extract one or more image data features from the image data;
[0008] - The second set of layers of the convolutional neural network is used to extract one or more depth data features from the depth data;
[0009] - Fuse one or more image data features and one or more depth data features to obtain at least one fused feature map;
[0010] - The at least one fused feature map is processed by applying the third set of layers of the convolutional neural network to identify / classify objects and provide 3D localization data for the identified / classified objects, wherein the 3D localization data includes object reference point data.
[0011] The method described above uses a convolutional neural network to extract 3D localization data from image and depth data. The image and depth data can correspond to pixel matrices, where, specifically, the image data can be set as RGB data. For image data, pixels typically indicate brightness and / or color, while depth data indicates distance information for each pixel.
[0012] Convolutional neural networks are configured to perform feature extraction separately from image data and depth data. This is accomplished by a series of consecutive (convolutional) layers, each with one or more associated filters, enabling the separate extraction of image data features and depth data features from the provided input data.
[0013] In the fusion stage, image data features and depth data features are combined to obtain a fused feature map. This fused feature map is then further processed by applying a third set of consecutive layers. The first, second, and third sets of layers in the convolutional neural network are trained using training data, which includes image data, corresponding depth data, and 3D localization data of the object to be identified / classified. The 3D localization data includes object reference point data indicating representative 3D locations (3D pixels). Even if the object is partially occluded, the object reference point data is suitable for representing relevant points of the object.
[0014] The architecture of convolutional neural networks described above is flexible and has proven effective in training them using synthetic images and depth data to learn 3D localization, including object reference point data.
[0015] Specifically, to train a convolutional neural network, sample images based on image and depth data can be cropped and expanded by scaling up and down, utilizing corresponding adjustments to object contour data. The resulting image can then be resized to a fixed-size rectangular input image to obtain the input data for training the convolutional neural network. However, directly applying cropping / expansion enhancements and resizing image and depth data can disrupt the understanding of object proportions in the perceptual environment, which is crucial for accurate 3D localization. Therefore, to generate training data, it can be configured to adapt the depth data according to the scaling during the cropping or expansion process, preserving the original image's proportions and aspect ratio.
[0016] Furthermore, for training, a pre-trained RGB YOLO v3 detector model can be used as a starting point, which has been trained on a large-scale 2D object detection dataset. For the layers in the first and third groups, weights can be copied from the pre-trained model. Weights from the layers in the first group can also be transferred to the layers in the second group, while the first layer of the second group, which acquires a single-channel depth image, is randomly initialized from scratch.
[0017] In addition, the fusion step may include concatenating image data features and depth data features, followed by applying a 1x1 convolutional layer with an adjusted number of filters to preserve the original number of channels.
[0018] This can be configured to reduce the size of the linked feature map to the size of one or more image data features.
[0019] In addition, object reference point data may include the 3D centroid or the centroid of the object, which indicates the 3D coordinates of the object's geometric center.
[0020] A step can be performed to process at least one fused feature map by applying a third set of layers of a convolutional neural network to obtain boundary data indicating the contours of the identified / classified object.
[0021] The layers in the first, second, and / or third groups can be configured such that they include one or more active layers and / or one or more residual blocks with skip connections.
[0022] According to another aspect, a convolutional neural network is provided, the convolutional neural network having a first set of consecutive layers, a second set of consecutive layers, and a third set of consecutive layers, each layer being configured to have one or more filters, wherein a fusion layer is configured to fuse one or more image data features obtained from the first set of layers and one or more depth data features obtained from the second set of layers, wherein the convolutional neural network is trained to associate image data items and depth data items with object recognition / classification and corresponding 3D localization data for the recognized / classified object, wherein the 3D localization data includes object reference point data.
[0023] According to another aspect, a method for training the above-mentioned convolutional neural network is provided, wherein training data is used to train the convolutional neural network, wherein each training data item includes image data, depth data, and 3D positioning data, and wherein at least a portion of the training data item is synthesized.
[0024] Specifically, the synthesized training data is rendered using a 3D graphics engine with foreground and background randomization, where one or more foreground objects are included in each training data item.
[0025] According to one embodiment, additional training data items are generated from the image data items and corresponding depth data items of the training data items through the following steps:
[0026] - Randomly crop or expand image data from training data items while maintaining aspect ratio;
[0027] - By applying a scaling factor based on the size of the image data of the training data item and the size of the cropped image data, the size of the cropped / expanded image data item is adjusted to the size of the image data of the training data item;
[0028] - The additional training data items are obtained by adapting the depth data items and the associated 3D localization data of the identified / classified objects based on the scaling factor.
[0029] According to another aspect, an apparatus is provided for 3D localization of an object / individual based on image data and depth data indicating distance information of pixels in the image represented by the image data. The convolutional neural network is configured to have a first set of consecutive layers, a second set of consecutive layers, and a third set of consecutive layers, each layer being configured to have one or more filters. A fusion layer is configured to fuse one or more image data features obtained from the first set of layers and one or more depth data features obtained from the second set of layers to obtain at least one fused feature map for further processing by the third set of layers. The convolutional neural network is trained to associate object recognition / classification and corresponding 3D localization data for the recognized object with image data items and depth data items. The apparatus is further configured to perform the following steps:
[0030] - The first set of layers of the convolutional neural network is used to extract one or more image data features from the image data;
[0031] - The second set of layers of the convolutional neural network is used to extract one or more depth data features from the depth data;
[0032] - Fuse one or more image data features and one or more depth data features to obtain at least one fused feature map;
[0033] - The at least one fused feature map is processed by applying the third set of layers of the convolutional neural network to identify objects and provide 3D localization data for the identified objects, wherein the 3D localization data includes object reference point data. Attached Figure Description
[0034] The embodiments are described in more detail with reference to the accompanying drawings, wherein:
[0035] Figure 1 This is a schematic diagram of the architecture of a convolutional neural network trained to determine 3D localization data from image data items and corresponding depth data items. Detailed Implementation
[0036] Figure 1 The architecture of a convolutional neural network1 is schematically illustrated for processing input data including image data such as RGB data and corresponding depth data. Each image data item is associated with a depth data item so that they all refer to the same scene. Image data can be acquired using, for example, a camera. Image data may include one or more pixel matrix channels, such as color channels of RGB data, etc.
[0037] Depth data can be obtained using appropriate depth sensors based on LiDAR, radar, ultrasonic sensors, or other sensors that capture a pixel matrix, where each pixel is associated with a distance value.
[0038] The network architecture can be similar to the YOLO v3 network, which has been modified to also consider depth data and estimate 3D positioning data.
[0039] Generally, the convolutional neural network 1 comprises a first group 2 of consecutive layers 21. The first group 2 of layers 21 may include consecutive convolutional layers, activation layers, normalization layers, and residual blocks, the corresponding configurations of which are known. The number of consecutive convolutional layers in the first group 2 can be between 4 and 60, preferably between 5 and 30. Image data item I is fed into the input of the first group of layers 21. The first group 2 of layers 21 serves as a feature extractor for image data item I to obtain one or more image data features.
[0040] In addition, a second group of three consecutive layers 31 is provided. The second group of three layers 31 may include consecutive convolutional layers, activation layers, corresponding normalization layers, and residual blocks, the configuration of which is well known. The number of consecutive convolutional layers 31 in the second group of three can be between 5 and 30. Depth data term D is fed into the input of the second group of three layers 31. The second group of three layers 31 serves as a feature extractor for the depth data term D to obtain one or more depth data features.
[0041] Image data features and depth data features are fused using fusion layer 4. Specifically, a connection layer 41 is applied to connect the feature maps along the channel dimension. A 1x1 convolutional layer 42 is used to reduce the increase in size of the connected feature maps by halving the number of channels to the original channel size.
[0042] The output of fusion layer 4 is further processed in a third group of 5 consecutive layers 51. The third group of 5 layers 51 may include consecutive convolutional layers, activation layers, associated normalization layers, and residual blocks, the configuration of which is well-known. The number of consecutive convolutional layers in the third group of 5 can be between 5 and 30. The third group of 5 layers 51 serves as a feature extractor for the combined image and depth data features.
[0043] The output of layer 51 in the third group 5 is fed to output block 6.
[0044] The output block 6 can be comprised of one or more stages 61 of consecutive blocks of convolutional layers 62. Each stage 61 specifically includes an activation layer and / or associated normalization layer for the corresponding convolutional layer, followed by an upsampling layer 63 and a connection layer 64 that connects the result of the upsampling layer 63 to the output of the residual block of the third group of layers 51. The configuration of the residual blocks is well known in the art. The number of consecutive convolutional layers 62 in each block of output block 6 can be between 5 and 30. The output of the convolutional layers 62 in each block is input to a 1x1 output layer 65.
[0045] One or more connection layers 64 of output block 6 can use skip connections to combine feature maps from layer 51 of the third group 5. Additionally, an upsampling layer 63 can be configured; specifically, the upsampling layer 63 can be placed directly before the connection layers 64.
[0046] Upsampling layer 63 increases the size of the feature map in both width and height dimensions, thus acting as a counterpart to the original convolution in the feature extractor that reduces the feature map size. Upsampling can be achieved via deconvolution. The entire construction (first reducing the resolution, then increasing it) resembles a "functional pyramid".
[0047] As in the original YOLO v3, the upsampling layer allows for predictions at three different scales. The functionality of the upsampling layer 63 is part of the well-known YOLO v3 architecture.
[0048] Feature maps from one or more layers 61 of the output block 6 are deconcatenated and fed to a 1x1 convolutional layer that provides outputs representing boundary data and 3D localization data. The 1x1 output layer 65 is configured to compute the regressed target tcu, tcv, tcz and the existing target for the 2D bounding box. This can be learned entirely from the data. The following equation describes how the output of the 1x1 layer (of which tcu, tcv, and tcz are part) is used to compute cu, cv, cz, and then compute CX, CY, cz. This occurs in the post-processing step after the network pre-pass is complete.
[0049] Boundary data B can correspond to bounding box data representing the region where objects have been identified in the provided image and depth data. For each identified / classified object, 3D localization data representing an object reference point L has been determined. The object reference point represents a significant location within the region defined by the boundary data (bounding box). Therefore, the object reference point can represent 3D centroid, centroid, geometric center, human joints, etc.
[0050] To estimate the metric 3D coordinates of the object's reference point c x c y c zIt can be derived from pixel coordinates c u c v Infer c x and c y Coordinates. Pixel coordinates c u c v Compared to having a top-left corner b u b v And width and height b w b h The predicted 2D bounding box is used for representation. A sigmoid function is used to ensure that the object reference point lies within the 2D bounding box. The regression target tc u , tc v It is the variable of this sigmoid function, not the direct regression c. u c v Conversely, c z via regression target tc z Direct regression by measuring proportion:
[0051]
[0052] After calculating c u c v and c z After that, you can use c z And the inverse mapping c of the given camera matrix K u and c v To obtain c x and c y Coordinates. In computer vision, a camera matrix, or (camera) projection matrix, is a matrix that describes the mapping from a pinhole camera to 3D points in the real world to 2D points in an image. This is a well-known concept in computer vision and is typically obtained through camera calibration.
[0053] The output parameter structure essentially corresponds to the standard YOLO v3 configuration, where the final 1x1 convolution of the output stage is expanded to predict the regression target tc for each bounding box. u , tc v , tc z Training is based on the loss that is minimized during training (i.e., the loss required for backpropagation to compute gradients). For example, tc z L1 loss can be used, tc u and tc v S-type binary cross-entropy loss can be used.
[0054] To train a convolutional neural network1, a substantially stochastic gradient descent can be applied, optionally with different learning rates. Essentially, training is performed using a training dataset that includes image data, depth data, and associated objects represented by boundary data, such as bounding boxes, and by 3D localization data, such as object reference points.
[0055] The training dataset can be obtained from a real-world environment setting or can be synthetically generated, for example, based on a CAD model or photogrammetric 3D scan. To obtain a sufficiently diverse training dataset, domain randomization can be performed, for example, by changing object pose, number of objects, and lighting conditions. In particular, a semi-realistic computer graphics engine can be used to generate synthetic images and depth data items to at least partially render the training dataset. Some sensor effects, such as noise patterns, can be simulated in the RGB and / or depth data. Boundary data and 3D localization data of associated objects can be obtained directly from the CAD model.
[0056] The training process may involve a 2D data augmentation step, which includes cropping and expanding randomly selected image portions using appropriate adjustments to boundary data (bounding boxes) to allow for scaling up and down of image portions containing objects. The resulting image portions are then resized to a fixed pixel matrix size to normalize the image and depth data sizes used for the convolutional neural network. Directly applying this existing form of cropping and expanding augmentation, along with the resizing to the pixel matrix size of the training data, can distort the "understanding" of the object matrix proportions in the perceptual environment. Therefore, this augmentation is enhanced to become depth-aware, adapting the boundary data and 3D localization data of associated objects to the current scaling factor.
[0057] The scaling factor is based on a factor used to adjust the size of the pixel matrix required for training the convolutional neural network. For d n xd n The input resolution of the square of pixels, with a size of d. c xd c Random square cropping. Under the assumption of a single image sensor with a known camera matrix K, a scaling factor can be applied. Adjust the cutting size to the input size d. n xd n Since scaling is typically attributed to changes in focal length and keeps intrinsic parameters constant, applying proportional scaling to the depth value results in...
[0058]
[0059] Where x, y, and z are 3D points resolved in the image sensor frame. It is the new scaled depth at the input image pixel (u, v).
[0060] In addition to performing depth scaling to generate a training dataset with image data items and corresponding depth data items, the 3D localization data of associated objects is also scaled.
[0061] Furthermore, a transfer learning strategy can be applied that benefits from existing large-scale 2D object detection datasets. Thus, layers 21 and 51 of the first and third groups can be initialized using weights from an existing YOLO v3 RGB detector pre-trained on existing 2D object detection datasets. This means that weights can be initially set for layer 21 of the first group 2 and layer 5 of the third group, while for layer 31 of the second group 3, the weights of layer 21 of the first group 2 are copied. However, the first layer of layer 31 in the second group 3 can be initialized from scratch.
[0062] At the start of training, fusion layer 4 is initialized so that the image data features at the output of layer 21 of group 2 are forwarded unchanged to the input of layer 51 of group 5. The weights of newly added neurons in the output block are randomly initialized, while the original 2D detection weights remain unchanged.
[0063] With this initialization strategy, the pre-trained 2D performance can be maintained regardless of changes in the network architecture.
Claims
1. A method for training a convolutional neural network (1), wherein, The convolutional neural network (1) has a first group (2) of consecutive layers (21), a second group (3) of consecutive layers (31), and a third group (5) of consecutive layers (51). Each layer is configured to have one or more filters. The fusion layer (4) is configured to fuse one or more image data features obtained from the first group (21) of layers (21) and one or more depth data features obtained from the second group (31) of layers (31). The convolutional neural network (1) is trained to associate image data items and depth data items with object recognition / classification and corresponding 3D localization data for the recognized / classified object. The 3D localization data includes object reference point data. The convolutional neural network (1) is trained using training data, wherein each training data item includes image data, depth data, and 3D localization data, and at least a portion of the training data item is synthetically generated. The additional training data items are generated from the image data items and corresponding depth data items of the training data items through the following steps: While maintaining the aspect ratio, crop or expand randomly selected image portions from the image data of the training data items; By applying a scaling factor based on the size of the image data item in the training data item and the size of the cropped / expanded image portion, the size of the cropped / expanded image portion is adjusted to match the size of the image data item in the training data item; and The additional training data items are obtained by adapting the depth data items and the associated 3D localization data of the identified / classified objects based on the scaling factor.
2. The method according to claim 1, wherein, The synthesized training data is rendered using a 3D graphics engine with foreground and background randomization, where one or more foreground objects are included in each training data item.
3. A computer program product comprising a computer-readable medium having computer program code tools thereon, which, when loaded, causes a computer to run the program to perform all the steps of the method according to any one of claims 1 to 2.
4. A machine-readable medium having a program recorded thereon, wherein, The program is used to cause a computer to perform the method according to any one of claims 1 to 2.