A method for depth and 3D key point estimation for intelligent accompanying patrol devices

Abstract

This invention proposes a depth and 3D keypoint estimation method for intelligent accompanying patrol vehicles: depth estimation is achieved end-to-end from binocular infrared images, while simultaneously identifying the detected region from the depth map and performing 3D keypoint recognition of the target. The corrected left and right viewpoint images are jointly processed by a parameter-shared feature extraction module, then respectively enter the depth estimation branch for depth estimation and the keypoint branch for 3D keypoint recognition. This invention enables end-to-end estimation from binocular infrared images to depth maps and 3D keypoints, avoiding cumbersome intermediate processes and improving the computational and operational complexity of depth map estimation and 3D keypoint recognition. Through feature sharing, joint constraints are achieved in the parameter learning process between branches, improving the inference accuracy of each task. This invention also improves the efficiency of matching cost aggregation in the depth estimation task and the coordinate calculation efficiency in the keypoint estimation task.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for depth estimation and 3D key point estimation that can be used in intelligent accompanying patrol vehicles, while simultaneously achieving automated recognition of depth and key point coordinates. Background Technology

[0002] In the deep space exploration plans of various countries around the world, manned deep space exploration is an inevitable trend. Just as technological changes in Earth applications demonstrate, human-computer interaction is evolving from traditional button-based to touchscreen-based, and then to contactless methods. In manned deep space exploration, intelligent companion rovers also face the need for contactless human-computer interaction, such as intelligently accompanying astronauts and monitoring their behavior and safety without interfering with their work. Considering the rich environmental information that depth-sensing payloads can acquire, reusing the depth-sensing payloads on the rover to achieve intelligent astronaut companionship is the foreseeable best choice. Gestures and human posture are important ways to understand human behavior; therefore, this invention will focus on the study of astronaut behavior monitoring by intelligent companion rovers using a depth estimation module, based on key points of the human body and hands.

[0003] In terms of task fusion, in recent years, more and more scholars have noticed the promoting effect of multi-task fusion on model learning. Although a series of research results have been used to handle the fusion of tasks such as segmentation, detection, classification, and recognition with depth estimation, there are still research gaps in some important areas. Among them, keypoint recognition is a worthy area of ​​fusion research. Human body and gesture recognition are important representatives of keypoint recognition tasks and are important ways to realize human-computer interaction in modern artificial intelligence devices. They play a key role in AR / VR glasses, intelligent driving control, smart homes, motion-sensing games, and other fields, and are also of great significance for monitoring astronaut behavior. Existing 3D keypoint recognition tasks usually directly use depth maps acquired by active depth cameras as input, which not only fails to effectively fuse recognition and depth estimation, but also makes active depth acquisition methods unsuitable for deep space exploration. By comparing the network structure of depth estimation methods with the image and depth information processing processes of keypoint recognition methods, it can be found that the processing of depth information can be regarded as the inverse process of encoding and decoding. Therefore, based on the idea of ​​transfer learning, a multi-task joint step-by-step training method can be used to realize fused 3D keypoint recognition depth estimation that can be used for deep space exploration rovers. Summary of the Invention

[0004] The purpose of this invention is to solve the problem of computational redundancy in depth estimation and 3D keypoint estimation, while simplifying the estimation process and achieving automated depth and 3D keypoint estimation. This addresses the technical challenges faced by intelligent accompanying rover vehicles in terrain exploration, human-computer interaction, and intelligent companion functions. To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0005] A depth and 3D keypoint estimation method for intelligent accompanying patrol vehicles is characterized by: end-to-end depth estimation from binocular infrared images, while automatically identifying the detection area from the depth map and completing 3D keypoint recognition of the target. The overall process of the method is as follows: Figure 1 As shown, the corrected left and right viewpoint images are jointly processed by a parameter-shared feature extraction module, then respectively enter the depth estimation branch for depth estimation and the keypoint branch for 3D keypoint recognition. The method includes the following steps:

[0006] Step 1: Calibrate the binocular camera system, obtain the camera's baseline distance b and lateral focal length f, and correct the left and right viewpoint images according to the camera's intrinsic and extrinsic parameters to ensure that the images meet the conditions of optical axis parallelism and pole infinity.

[0007] Step 2: Extract image features using a neural network. The specific steps are as follows:

[0008] Step 2a: Use one such Figure 2 (a) shows the refining unit and 3 as shown in the image. Figure 2 (b) shows a maintenance unit that constitutes a downsampling module. Image features are then extracted through the downsampling unit, and the feature space resolution is compressed to half the input space resolution. The convolutional layer in the figure is represented by: kernel size, convolution type, and post-processing operation. BN represents BatchNormalization, and LReLU represents LeakyReLU filtering applied to the output. Figure 2 (b) The SE module in the maintenance unit consists of cascaded average pooling and two 1×1 convolutional layers, where the number of channels in the convolutional layers is first reduced and then increased compared to the number of input channels. The feature extraction process uses three sets of downsampling modules to compress the features to 1 / 8 of the original spatial resolution.

[0009] Step 2b: Using the features from the previous step as input, extract the feature matrices after downsampling to 1 / 8, 1 / 16, and 1 / 24 of the input space using the feature pyramid.

[0010] Step 3: Within the maximum disparity range, calculate the similarity between features at the left and right viewpoints along the epipolar line. Feature similarity is measured by the dot product of feature vectors. For the pixel at position p = (x, y) in the left viewpoint, the feature F... l(p) and candidate disparity d (the candidate disparity corresponds one-to-one with the channels in the matching cost, and the numbering values ​​are the same), then it is necessary to calculate the feature F corresponding to the pixel with right viewpoint coordinates q=(xd,y). r (q) and F l The similarity between (p). Then, for resolution scale s, the corresponding matching cost C s The formula for calculating (p,d) is:

[0011]

[0012] C in the formula f The number of feature channels is denoted by <*>, and <*> represents the dot product operation between vectors.

[0013] Step 4: Aggregate the matching cost across multiple scales using a neural network. The specific steps are as follows:

[0014] Step 4a: Given the cost matrix C(d,p) at position p at a certain scale, calculate the matching cost jointly contributed by the K neighboring positions adjacent to position p within both fixed and variable neighborhoods:

[0015]

[0016] in, The value of Δp represents the matching cost at position p on the feature channel corresponding to candidate disparity d, after processing by the pre-aggregation module. k and m k p represents the offset of the k-th variable neighbor pixel relative to the center position and the corresponding weight of the convolution operation, respectively. k and w k Let b be the offset of the k-th fixed neighbor pixel relative to the center position and the corresponding weight of the convolution operation. d K is the bias corresponding to channel d. 2 This represents the total number of neighboring regions.

[0017] Step 4b: Using the output of the previous step as input, aggregate the matching costs between scales according to the following rules:

[0018] a) When the scales corresponding to the input and output are the same, the cost matrix is ​​directly passed forward.

[0019] b) When the input scale is smaller than the output scale, the matching cost aggregation structure first performs bilinear upsampling of the input channels spatially through bilinear interpolation, and then applies a fully connected layer constructed from 2D convolutional layers with a kernel size of 1×1 to the upsampling result. The upsampling and fully connected operations are repeated until the input is transformed to the target spatial resolution corresponding to the output.

[0020] c) When the input scale is larger than the output scale, apply a 2D convolution operation with a stride of 2 to the input until the input is transformed to the target spatial resolution corresponding to the output.

[0021] Step 4c: For the output of Step 4b, aggregate the matching costs between different disparities along the matching cost channel dimension. The specific calculation formula is as follows:

[0022]

[0023] in, This represents the cost matrix after aggregation by the multi-scale aggregation module. This is the cost matrix after processing by the post-aggregation module. d t w represents the disparity corresponding to the t-th disparity neighborhood within the symmetric disparity range centered on the candidate disparity d. t This represents the weight corresponding to the disparity neighborhood channel. For any candidate disparity d, T disparity neighborhoods are sampled. If the disparity neighborhood range exceeds the disparity range, it is set to 0 or d based on the nearest boundary. max .

[0024] Step 4d: via, for example Figure 2 The expansion unit shown in (c) simultaneously expands the matching cost matrix at each scale by a factor of 8 at the spatial resolution, restoring the spatial resolution of the highest scale to the same as the input spatial resolution. After a convolutional layer with a kernel size of 1×1, all channels are aggregated to obtain the disparity map estimate D. The convolutional layer in the figure is represented by: kernel size, convolution type, and post-processing operation. BN represents applying Batch Normalization, and LReLU represents applying LeakyReLU filtering to the output.

[0025] Step 5: Using the disparity map with the highest spatial resolution as input, calculate the depth information at each position p from the disparity map estimate D based on the baseline distance b and focal length f using the following formula: All depth information together constitutes the depth map D.

[0026] Step 6: Filter the target region from the depth map estimation using the following steps:

[0027]

[0028]

[0029] Step 7: Based on the cropping window, extract the target region from the left viewpoint image, left viewpoint features, matching cost matrix, and depth map, and then, at the channel level, form texture features composed of the left viewpoint image and left viewpoint features, and depth features composed of the matching cost matrix and depth map. Input both into... Figure 3The texture / depth preprocessing unit shown in the figure integrates the results by channel to form the input of the keypoint estimation branch. The convolutional layer in the figure is represented by: kernel size, convolution type, and post-processing operation. BN represents applying Batch Normalization, and LReLU represents applying LeakyReLU filtering to the output.

[0030] Step 8: Send the features to, for example Figure 4 The 3D keypoint estimation network structure shown extracts the direction and weight information of each anchor point. Each block in the diagram represents the function of that part of the network structure. This indicates digit-wise multiplication. This indicates phase-by-phase concatenation. The feature integration part first uses a 1×1 convolutional layer to map the number of feature channels to 256, and then uses ResNet to further extract deep feature information. Each branch in the keypoint estimation network is as follows: Figure 5 As shown in the figure, the input is the integrated anchor point feature, with C channels and a size of h×w. The text circled in squares describes the function of the network, and the cube represents the output format after this processing. The description format of the convolutional layer is: kernel size, convolution type, and post-processing operation. BN represents applying Batch Normalization, and LReLU represents applying LeakyReLU filtering to the output.

[0031] Step 9: Integrate the anchor point information and calculate the coordinate estimate of each key point using the following formula:

[0032]

[0033] Among them, KP k (i) represents the coordinate estimate of the i-th keypoint in dimension l. This indicates element-wise multiplication. Ultimately, the 3D keypoint estimation KP has a dimension of K×3, representing the XYZ three-dimensional coordinates of each keypoint. H represents the anchor weight. k This is a heatmap for the l-th dimension. W A (j) and H l (j) represents the anchor weight and popularity on the j-th channel, respectively.

[0034] Compared with existing depth estimation and keypoint estimation methods, this invention has the following advantages:

[0035] 1. This invention can simultaneously achieve end-to-end estimation from binocular infrared images to depth maps and 3D key points, avoiding cumbersome intermediate processes and improving the computational load and operational complexity of depth map estimation and 3D key point recognition.

[0036] 2. This invention achieves joint constraints on the parameter learning process among branches through feature sharing, effectively improving the inference accuracy of each task; and in the key point recognition task, this invention supports 3D key point estimation of the hand and human joints, and can obtain the absolute depth of each joint, which is beneficial for applications in real-world scenarios.

[0037] 3. Based on the idea of ​​dimensional splitting, this invention improves the efficiency of matching cost aggregation in depth estimation tasks and also improves the efficiency of coordinate calculation in key point estimation tasks. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the method for fusion depth and 3D key point estimation proposed in this invention.

[0039] Figure 2 (a)-(c) are schematic diagrams of the neural network unit structure used in this invention.

[0040] Figure 3 This is a schematic diagram of the texture and depth preprocessing unit structure constructed in this invention.

[0041] Figure 4 This is a schematic diagram of the 3D key point recognition network structure proposed in this invention.

[0042] Figure 5 This is a schematic diagram of the coordinate heatmap calculation structure in the key point recognition network proposed in this invention.

[0043] Figure 6 (a)-(d) represent the left viewpoint image in the KITTI dataset, the disparity estimation results of the GA-Net method, the disparity estimation results of the EdgeStereo method, and the disparity estimation results of the present invention, respectively.

[0044] Figure 7 (a)-(e) represent the left and right viewpoint image inputs in the MADS dataset, the depth map estimation results of the present invention, the 2D projection of the 3D keypoint estimation of the present invention onto the left viewpoint image, and the 3D keypoint estimation results and corresponding depth map clipping of the present invention, respectively.

[0045] Figure 8 (a)-(f) are the left viewpoint image, the right viewpoint image, the depth map captured by the depth camera, the depth map estimated by the present invention, the 2D projection of the key point estimation of the present invention on the left viewpoint, and the 3D key point estimation result of the present invention, respectively, taken under the verification experimental platform of the present invention.

[0046] Figure 9(a)-(f) are the left viewpoint image, the right viewpoint image, the depth map captured by the depth camera, the depth map estimated by the present invention, the 2D projection of the key point estimation of the present invention on the left viewpoint, and the 3D key point estimation result of the present invention, respectively, taken under the verification experimental platform of the present invention. Detailed Implementation

[0047] The present invention will now be described in further detail with reference to the accompanying drawings and examples.

[0048] Using PyTorch as the deep learning framework and a RealSense D435 depth camera as the acquisition device, this invention includes the following steps:

[0049] Step 1: Calibrate the RealSense D435 camera to obtain the baseline distance b = 50.0 mm and the lateral focal length f = 382.995 mm for the binocular camera. Read and store the calibrated left viewpoint image, right viewpoint image, and depth image in the order they were captured. During shooting, the distance from your hand to the camera should be within the range of [10, 50] cm, and the distance from your body to the camera should be within the range of [0.5, 4.5] m.

[0050] Step 2: Extract image features using a neural network, specifically including:

[0051] Step 2a: Construct a feature extraction network structure within the PyTorch deep learning framework. This includes: reading the stereo image according to the image storage path; building a feature extraction structure consisting of three downsampling modules; and sequentially feeding the input image into the feature extraction structure to obtain image features. The refinement unit structure within the downsampling unit is as follows: Figure 2 As shown in (a), the number of channels in the refinement units of the three downsampling modules, from top to bottom, are 64 / 64 / 128, 128 / 128 / 256, and 256 / 256 / 512, respectively. The maintenance unit structure is as follows: Figure 2 As shown in (b), the number of channels from top to bottom are 128 / 64 / 128, 256 / 128 / 256, and 512 / 256 / 512, respectively. The spatial resolution of the captured image is 480×640. After feature extraction, a feature matrix with a spatial resolution of 60×80 is obtained. The number of channels in the feature matrix is ​​512.

[0052] Step 2b: Construct a feature pyramid extraction structure using pooling layers and convolutional layers with a kernel size of 1×1. The pooling window sizes are 1, 2, and 3, and the number of channels in the convolutional layers is 512. The spatial resolution of the features in the feature pyramid is downscaled to 1 / 8, 1 / 16, and 1 / 24 of the original spatial resolution, respectively, with 512 feature channels. When the spatial resolution is not divisible, a rounding down strategy is used, resulting in feature matrix sizes of:

[0053] 512×60×80, 512×30×80 and 512×20×26.

[0054] Step 3: Set the maximum disparity to 192. Within the maximum disparity range, calculate the similarity between features at the left and right viewpoints along the epipolar line. Feature similarity is measured by the dot product of feature vectors. Specifically, for the feature at coordinate (x, y) in the left viewpoint, calculate the similarity between the features of pixels with x-coordinates between [x-192, x] along the horizontal line with the y-coordinate of the right viewpoint, and the features at coordinate (x, y) in the left viewpoint. If the minimum x-coordinate is less than 0, only the feature similarity from 0 to x is calculated. The number of channels in the similarity calculation formula is 512, used to normalize the matching cost. The final dimensions are:

[0055] Three matching cost matrices: 192×60×80, 192×30×40, and 192×20×26.

[0056] Step 4: Construct a matching cost aggregation network within the PyTorch deep learning framework to aggregate matching costs across multiple scales. The specific steps are as follows:

[0057] Step 4a: Construct a spatial matching cost aggregation network using fixed convolution and deformable convolution network definitions. The total number K of both fixed and deformable neighborhoods is 3. The coordinate offset p of the fixed neighborhood is... k Uniformly distributed within the interval [-1, 1] on the x and y axes; the position offset of the deformable neighborhood is Δp. k The cost matrix is ​​learned by individual convolutional layers, specifically constructed according to the network definition of deformable convolution. Biases and weights are learned using fixed and deformable convolutions as defined in the PyTorch deep learning framework. The matching cost dimensions after scale aggregation at each resolution are as follows:

[0058] 192×60×80, 192×30×40 and 192×20×26.

[0059] Step 4b: Construct the network structure. Using the aggregated matching cost matrix from the previous step as input, aggregate the matching costs across different scales. The aggregation rules are: 1) When the scales corresponding to the input and output are the same, the cost matrix is ​​directly passed forward. 2) When the input scale is smaller than the output scale, the matching cost aggregation structure first performs bilinear upsampling on the spatial dimensions of each input channel using bilinear interpolation, and then applies a fully connected layer constructed from 2D convolutional layers with a kernel size of 1×1 to the upsampling result. Repeat the upsampling and fully connected operations until the input is transformed to the target spatial resolution corresponding to the output. 3) When the input scale is larger than the output scale, apply a 2D convolutional operation with a stride of 2 to the input until the input is transformed to the target spatial resolution corresponding to the output. Specifically, the size of the matching cost matrix in this example is:

[0060] 1) When the spatial resolution changes from 60×80 to 60×80, from 30×40 to 30×40, and from 20×26 to 20×26, the matching cost matrix is ​​directly passed backward.

[0061] 2) When the spatial resolution changes from 30×40 to 60×80, from 20×26 to 30×40, and from 20×26 to 60×80, interpolation is first used to upsample the spatial resolution to the target space, and then the information is integrated through convolutional layers.

[0062] 3) When the spatial resolution changes from 60×80 to 30×40, from 60×80 to 20×26, and from 30×40 to 20×26, a convolutional layer with a stride of 2 is first used to downsample the input in terms of spatial resolution. The number of downsampling convolutions applied in these cases are 1, 2, and 1, respectively. After aggregation, the sizes of the matching cost matrix are 192×60×80, 192×30×40, and 192×20×26, respectively.

[0063] Step 4c: Using the PyTorch deep learning framework, construct a matching cost aggregation structure between channels to achieve matching cost aggregation around candidate disparities $d$. The disparity neighborhood T = 5 is used for aggregation, and the maximum disparity is 192.

[0064] Step 4d: According to Figure 2 (c) Dilation units are constructed, with the number of channels in the three groups of dilation units from top to bottom being 512 / 512 / 256, 256 / 256 / 128, and 128 / 128 / 64, respectively. Subsequently, a convolutional layer with a kernel size of 1 × 1 and an output channel count of 1 is passed through to integrate the matching cost matrix and obtain the disparity map estimate. The spatial resolutions of the disparity maps at each scale are 480×640, 240×320, and 160×208, respectively.

[0065] Step 5: Using a disparity map with a spatial resolution of 480×640 as input, calculate a depth map with a spatial resolution of 480×640 by substituting b=50.0 and f=382.995 into the formula z=bf / d. Each pixel value in the map represents the actual depth at that location. The unit of the depth map at this point is millimeters.

[0066] Step 6: Filter the target region from the depth map estimation, specifically including:

[0067] Step 6a: Based on the actual shooting situation, use a mask of [10,50] cm (hand) or [0.5,4.5] m (human body) to filter out depth values ​​that are outside the range in the depth map and set them to 0.

[0068] Step 6b: Obtain the centroid of the cleaned depth point cloud according to the centroid calculation formula, and project it onto the world coordinate system. The projection formula is x world =(x pixel -ux)×z / fx、y world =(y pixel -uy)×z / fy, where fx and fy are the normalized focal lengths corresponding to the horizontal and vertical coordinates, respectively, both with a value of 382.995; ux and uy are the center coordinate offsets, with values ​​of 319.930 and 237.091, respectively; z is the depth corresponding to the pixel. pixel and x world These correspond to the x-coordinates in the image plane and the world coordinate system, respectively. pixel and y world These correspond to the ordinates in the image plane and the world coordinate system, respectively.

[0069] Step 6c: Based on the clipping window [250,250,250] (hand, unit: mm) and [0.7,2.5,1] (human body, unit: m), clip around the centroid in the world coordinate system to obtain the maximum and minimum values ​​of the clipping window on the XYZ axes respectively.

[0070] Step 6d: Project the clipping boundary from the world coordinate system back to the image coordinate system. The projection formula is the inverse transformation of the formula in step 6b.

[0071] Step 6e: Clean up the cropping window according to the actual size of the image, removing parts that exceed the image resolution.

[0072] Step 6f: Crop the tile in the depth map according to the clipping window to obtain the depth map of the target.

[0073] Step 7: Based on the cropping window, extract the target region from the left viewpoint image, left viewpoint features, matching cost matrix, and depth map, and then, at the channel level, form texture features composed of the left viewpoint image and left viewpoint features, and depth features composed of the matching cost matrix and depth map. First, scale the features to the target resolution of 176×176 (hand) or 228×228 (human body), then... Figure 3 A preprocessing unit is constructed to integrate the results by channel to form the input for the keypoint estimation branch. The texture preprocessing unit has 513 input channels, and the depth preprocessing unit has 193 input channels. The number of output channels for the texture preprocessing unit, from left to right, is 512 / 256 / 512 / 512 / 256 / 256; the number of output channels for the depth preprocessing unit, from left to right, is 256 / 512 / 256 / 256 / 128 / 256. Finally, the integrated input spatial resolution is 176×176 (hand) or 228×228 (human body), with 512 channels.

[0074] Step 8: Within the PyTorch deep learning framework, according to Figure 4 The following steps are taken to construct a 3D keypoint estimation network structure, perform feature integration, and estimate keypoints:

[0075] Step 8a: Construct the feature integration part, with a kernel size of 1×1 and 256 output feature channels for the convolutional layer.

[0076] Step 8b: Construct the anchor point orientation estimation branch. First, using a network structure similar to ResNet-50, further extract deep feature information, compressing the spatial resolution to 1 / 8 of the input, becoming 1024×22×22 (hand) and 1024×28×28 (human body) respectively. The number of input channels in the first convolutional layer of ResNet was modified according to the actual number of input channels. Subsequently, based on... Figure 5 A 3D split network structure is constructed in the anchor point orientation estimation branch. The input is weighted according to the Y-axis and X-axis directions to obtain heatmap estimates on the X and Y axes respectively; the input is weighted according to channels to obtain the heatmap estimate on the Z-axis. The heatmap size for the X and Y axes is 16K×22×1 (hand) or 16K×28×1 (human body), and the heatmap size for the Z-axis is 16K×22×22 (hand) or 16K×28×28 (human body). Then, the heatmaps on the X and Y axes are integrated sequentially according to dimensions to form a heatmap with a size of 16K×22×22 (hand) or 16K×28×28 (human body). The total number of keypoints K is 21 for the hand task and 15 for the human task.

[0077] Step 8c: Construct the anchor weight estimation branch. The network structure of the anchor weight estimation branch consists of two 3×3 convolutional layers (containing BN and activation functions) and one 1×1 convolutional layer integrating weight features. Then, the mean value in the channel plane is calculated for each channel to obtain the heatmap of each keypoint in different dimensions and the corresponding anchor weights. The weight space resolution is the same as the heatmap, either 16K×22×22 (hand) or 16K×28×28 (human body).

[0078] Step 9: Integrate the heatmap and anchor point weights according to the formula to obtain the three-dimensional coordinate estimate of each key point.

[0079] The technical effects of this invention will be further explained below with reference to experimental simulation:

[0080] 1. Simulation conditions

[0081] The PyTorch framework was used, with four NVIDIA P100 graphics cards. The pre-training database for the depth estimation branch was SceneFlow, and the fine-tuning database was KITTI2012. The hand 3D keypoint estimation branch used the STB dataset, and the human 3D keypoints used the MADS dataset. During the experiments, 90% of the SceneFlow training set and 85% of the KITTI training set were used to train the stereo matching network structure, with the remaining 10% and 15% of the training sets serving as validation sets. 80% of the data from STB and MADS was used as the training set, 15% as the validation set, and the remaining 5% as the test set. The network was trained with a learning rate of 0.0001, using the Adam parameter update strategy, with a batch size of 8. The depth estimation branch was trained for 60 epochs, and the keypoint estimation branch for 20 epochs.

[0082] 2. Simulation Content and Result Analysis

[0083] Simulation 1 tested the present invention on the SceneFlow and KITTI datasets and compared the results with other related research. The experimental results are shown in the table below. EPE is the MAE loss of the disparity estimation true value among all measured pixels, in pixels; >3px indicates the proportion of pixels with an EPE error greater than 3 pixels out of all measured pixels; Out-Noc and Out-All represent the results of >3px in the non-occluded region and all regions, respectively. Among all evaluation metrics, the smaller the error value, the higher the estimation accuracy of the method. The disparity estimation results on the KITTI2012 dataset are shown below. Figure 6 As shown in (a)-(d), Figure 6 (a) is the image from the left viewpoint. Figure 6 (b) and Figure 6 (c) is a comparison method. Figure 6(d) represents the result of this invention.

[0084]

[0085] Simulation 2 tested the invention on the MADS dataset, and the experimental results are as follows: Figure 7 As shown in (a)-(e). Figure 7 (a) and Figure 7 (b) is the input of the corresponding left and right viewpoint images in the dataset. Figure 7 (c) is the depth map estimation of the present invention. Figure 7 (d) is the projection of the 3D keypoint estimation result of the present invention onto the 2D image plane. Figure 7 (e) shows the 3D keypoint estimation results.

[0086] In simulation 3, videos of hands and human bodies were captured using a RealSense D435 camera in a real-world scenario to test the recognition performance of 3D key points on hands and human bodies, respectively. Figure 8 (a)-(f) present the 3D key point recognition results of the hand on the test platform. Figure 9 (a)-(f) present the results of 3D key point recognition of the human body on the test platform. Figure 8 (a) and Figure 8 (b) Figure 9 (a) and Figure 9 (b) Images from the left and right viewpoints, respectively. Figure 8 (c) and Figure 9 (c) Depth map acquired by RealSense D435 camera. Figure 8 (d) and Figure 9 (d) is the depth map estimation result of the present invention. Figure 8 (e) and Figure 9 (e) is the 2D projection of the 3D keypoint estimation. Figure 8 (f) and Figure 9 (f) shows the 3D keypoint visualization results.

Claims

1. A method for depth and 3D key point estimation for intelligent accompanying patrol vehicles, characterized in that: Includes the following steps: Step 1: Calibrate the binocular camera system, obtain the camera's baseline distance b and lateral focal length f, and correct the left and right viewpoint images according to the camera's intrinsic and extrinsic parameters to ensure that the images satisfy the conditions of optical axis parallelism and pole infinity. Step 2: Extract image features using a neural network; Step 3: Within the maximum disparity range, calculate the similarity between features at the left and right viewpoints along the epipolar line. The feature similarity is measured by the dot product of the feature vectors. Step 4: Aggregate matching costs across multiple scales using neural networks; Step 5: Using the disparity map with the highest spatial resolution as input, estimate the distance from the baseline b and focal length f from the disparity map. The depth information at each position p is calculated: All depth information together constitutes the depth map D; Step 6: Filter the target region from the depth map estimation: Step 7: Based on the cropping window, extract the target region from the left viewpoint image, left viewpoint features, matching cost matrix and depth map, and form texture features composed of the left viewpoint image and left viewpoint features and depth features composed of the matching cost matrix and depth map respectively at the channel level. Both are fed into the texture / depth preprocessing unit, and the results are integrated by channel to form the input of the key point estimation branch; Step 8: Feed the features into the 3D keypoint estimation network structure to extract the direction and weight information of each anchor point; Step 9: Integrate anchor point information and calculate the coordinate estimate of each key point.

2. The depth and 3D key point estimation method for intelligent accompanying patrol devices according to claim 1, characterized in that: In step 2, a set of downsampling modules consisting of one refining unit and three maintenance units is used. Image features are then extracted through the downsampling units, and the feature space resolution is compressed to 1 / 2 of the input space resolution. The convolutional layer is represented by kernel size, convolution type, and post-processing operation. BN represents applying Batch Normalization, and LReLU represents applying LeakyReLU filtering to the output. The SE module in the maintenance unit consists of cascaded average pooling and two convolutional layers with a kernel size of 1×1. The number of channels in the convolutional layer is first reduced and then increased compared to the number of input channels. A total of three sets of downsampling modules are used in the feature extraction process to compress the features to 1 / 8 of the original spatial resolution. Feature matrices after downsampling to 1 / 8, 1 / 16, and 1 / 24 of the input spatial resolution are extracted using the feature pyramid.

3. The depth and 3D key point estimation method for intelligent accompanying patrol devices according to claim 1, characterized in that: In step 3, the feature F corresponding to the pixel at position p = (x, y) in the left viewpoint is... l (p) and candidate disparity d, then it is necessary to calculate the feature F corresponding to the pixel with right viewpoint coordinates q=(xd,y). r (q) and F l The similarity between (p); at this point, for resolution scale s, the corresponding matching cost C s The formula for calculating (p,d) is: Among them, C f The number of feature channels is denoted by <*>, and <*> represents the dot product operation between vectors.

4. The depth and 3D key point estimation method for intelligent accompanying patrol devices according to claim 1, characterized in that: In step 4, given the cost matrix C(d,p) at position p at a certain scale, the matching cost jointly contributed by the K neighboring positions adjacent to position p is calculated within both fixed and variable neighborhoods: in, The value of Δp represents the matching cost at position p on the feature channel corresponding to candidate disparity d, after processing by the pre-aggregation module. k and m k p represents the offset of the k-th variable neighbor pixel relative to the center position and the corresponding weight of the convolution operation, respectively. k and w k Let b be the offset of the k-th fixed neighbor pixel relative to the center position and the corresponding weight of the convolution operation. d K is the bias corresponding to channel d. 2 This represents the total number of neighboring regions.

5. The depth and 3D key point estimation method for intelligent accompanying patrol devices according to claim 1, characterized in that: In step 4, the matching costs between scales are aggregated according to the following rules: a) When the scales corresponding to the input and output are the same, the cost matrix is ​​directly passed forward; b) When the input scale is smaller than the output scale, the matching cost aggregation structure first performs bilinear upsampling on the spatial interior of each input channel through bilinear interpolation, and then applies a fully connected layer constructed from a 2D convolutional layer with a kernel size of 1×1 to the upsampling result; the upsampling and fully connected operations are repeated until the input is transformed to the target spatial resolution corresponding to the output. c) When the input scale is larger than the output scale, apply a 2D convolution operation with a stride of 2 to the input until the input is transformed to the target spatial resolution corresponding to the output.

6. The method for depth and 3D key point estimation for intelligent accompanying patrol devices according to claim 1, characterized in that: In step 4, the matching costs between different disparities are aggregated along the matching cost channel dimension; the specific calculation formula is as follows: in, This represents the cost matrix after aggregation by the multi-scale aggregation module. This is the cost matrix after processing by the post-aggregation module; d t w represents the disparity corresponding to the t-th disparity neighborhood within the symmetric disparity range centered on the candidate disparity d. t The weights are defined for the corresponding disparity neighborhood channels. For any candidate disparity d, T disparity neighborhoods are sampled. If the range of a disparity neighborhood exceeds the disparity range, the value is set to 0 or d based on the nearest boundary. max .

7. The depth and 3D key point estimation method for intelligent accompanying patrol devices according to claim 1, characterized in that: In step 4, by using the expansion unit, the matching cost matrix at each scale is simultaneously expanded by 8 times at the spatial resolution, and the spatial resolution of the highest scale is restored to be the same as the input spatial resolution. A disparity map estimate is obtained by aggregating all channels through a single 1×1 convolutional layer.

8. The depth and 3D key point estimation method for intelligent accompanying patrol devices according to claim 1, characterized in that: In step 6, the process is as follows: 6.1 Generate a depth filter mask And based on the masking depth map: Calculate the centroid of the point cloud 6.2 Projecting the centroid coordinates to the world coordinate system yields p com,world ; 6.3 Calculate the clipping boundary B in the world coordinate system based on the clipping conditions; 6.4 Project the clipping boundary onto the UVD coordinate system to obtain the corresponding clipping boundary B. UVD ; 6.5 Clean up the boundaries and obtain the final clipping region R obj ; 6.6 Crop the depth map to obtain the depth map patch D of the target area. obj .

9. The depth and 3D key point estimation method for intelligent accompanying patrol devices according to claim 1, characterized in that: In step 8, the features are fed into the 3D keypoint estimation network structure to extract the direction and weight information of each anchor point; the feature integration part first uses a 1×1 convolutional layer to map the number of feature channels to 256, and then uses ResNet to further extract deep feature information; the input is the integrated anchor point features, with C channels and h×w size.

10. The depth and 3D key point estimation method for intelligent accompanying patrol devices according to claim 1, characterized in that: In step 9, the calculation formula is: Among them, KP k (i) represents the coordinate estimate of the i-th keypoint in dimension l. This indicates element-wise multiplication; ultimately, the 3D keypoint estimation KP has a dimension of K×3, representing the XYZ three-dimensional coordinates of each keypoint. H represents the anchor weight. k For the heatmap on the l-th dimension; W A (j) and H l (j) represents the anchor weight and popularity on the j-th channel, respectively.

Images