A detection and description model and method for key points of depth images

Abstract

The present application relates to the field of image processing technology, and provides a detection and description model and method for key points of a depth image, wherein the detection and description model for key points of a depth image includes a backbone feature extraction model, an upsampling module, a key point detector, and a key point descriptor. In the actual application process, on the one hand, in order to enhance the network's positioning accuracy for key points, the upsampling module reorganizes multiple levels of low-resolution feature maps to the original image resolution in a bilinear upsampling manner, and performs key point detection at the original image resolution, thereby improving the model's key point positioning accuracy. On the other hand, in order to enhance the network's perception of local shape details, the key point descriptor aggregates multiple layers of features together, and utilizes the dynamic receptive field of a deformable convolutional layer to achieve content perception of the input depth image, while filtering out noise on shallow feature maps. In addition, a two-layer fully connected network is used to screen out robust key point descriptors.

Description

Technical Field

[0001] The present application relates to the technical field of depth image processing, and in particular to a detection and description model and method for depth image key points. Background Art

[0002] Image key point detection and description is a popular method for finding pixel-level connections between images. For example, image matching and 3D reconstruction both use image key point detection and description. For example, in the process of anomaly detection of high-speed trains, historical images of the same train are matched with the current image, and then the image content of the two corresponding images is compared to detect missing parts and foreign objects, etc., to ensure the safe operation of high-speed trains. However, due to the challenges brought by the complexity of the operating environment of the high-speed rail system, such as changes in image appearance and style caused by changes in lighting and weather, the existing image key point detection and description methods are not effective enough.

[0003] Existing methods generally use linear array cameras installed near the rails to obtain images, and the key point detection and description algorithms cannot be robust in complex scenarios such as situations where the image appearance and visible area change greatly. In addition, existing deep learning-based methods often only use the deepest feature maps that are several times smaller (usually 4 or 8 times) than the original image for key point detection and description. In this method, locating the coordinates of the key points at the original image size in the deep feature map will inevitably reduce the detection accuracy of the key points. At the same time, in the deep feature map, key points of different local structures may share the same high-level semantic information, which makes it difficult to extract discriminative key point descriptors that retain rich local structure information.

[0004] In 3D computer vision, the pixel value of a depth image represents the distance information from the surface of a scene object to the viewpoint, reflecting the geometric shape of the object in the image. Unlike ordinary images, which are easily affected by changes in weather and lighting during imaging, causing changes in image content and style, the pixel value of a depth image is only related to the distance from the camera to each point in the scene. Therefore, it is less affected by the complex working environment of the high-speed rail system.

[0005] It can be seen that providing an image key point detection and description method that enhances the network's positioning accuracy of key points and enhances the network's perception of local shape details is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0006] In order to provide a method for enhancing the network's positioning accuracy of key points and enhancing the network's perception of local shape details, the present application provides a detection and description model and method for depth image key points.

[0007] The first aspect of the present application provides a depth image key point detection and description model, including:

[0008] A backbone feature extraction model, the backbone feature extraction model comprising a depth image input terminal, and a first feature extraction level, a second feature extraction level, a third feature extraction level, and a fourth feature extraction level connected in sequence;

[0009] The depth image input terminal is used to obtain a depth image to be detected, and the size of the depth image to be detected is H×W×1 pixel;

[0010] The first feature extraction level is used to convolve the depth image to be detected into a first feature map with a size of H×W×64 pixels;

[0011] The second feature extraction level performs pooling and convolution on the first feature map to obtain a second feature map with a size of H / 2×W / 2×64 pixels;

[0012] The third feature extraction level performs pooling and convolution on the second feature map to obtain a third feature map with a size of H / 4×W / 4×128 pixels;

[0013] The fourth feature extraction level performs pooling and deformable convolution on the third feature map to obtain a fourth feature map with a size of H / 8×W / 8×128 pixels;

[0014] An upsampling module, the upsampling module comprising four upsampling units and sampling convolution layers respectively connected to the first feature extraction level, the second feature extraction level, the third feature extraction level and the fourth feature extraction level, the upsampling unit is used to upsample the first feature map, the second feature map, the third feature map and the fourth feature map, and the sampling convolution layer is used to eliminate the aliasing effect that may occur in the upsampling to obtain feature maps of four original image resolutions, the sizes of which are: H×W×64 pixels, H×W×64 pixels, H×W×128 pixels and H×W×128 pixels respectively;

[0015] The key point detector is used to perform a convolution of 1 layer of 3×3×1 pixels and a step size of 1 pixel on the feature maps of the four original image resolutions to obtain four feature maps of size H×W×1 pixels. The feature maps of the four sizes are multiplied by coefficients of 0.1, 0.2, 0.3 and 0.4 in order from shallow to deep, and then accumulated to obtain a key point detection map of H×W×1 pixels.

[0016] The key point descriptor includes two fully connected layers, which are used to screen the feature maps of four original image resolutions stacked according to the feature scale to obtain a description feature map with a size of H×W×128 pixels. Finally, the description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length. The descriptor is composed of a 128-dimensional vector located at the key point coordinates in the description feature map.

[0017] In one implementation, the first feature extraction level includes two convolutional layers with a size of 3×3×64 pixels and a stride of 1 pixel;

[0018] The second feature extraction level includes a pooling layer with a stride of 2 pixels and two convolutional layers with a size of 3×3×64 pixels and a stride of 1 pixel;

[0019] The third feature extraction level includes a pooling layer with a stride of 2 pixels and two convolutional layers with a size of 3×3×128 pixels and a stride of 1 pixel;

[0020] The fourth feature extraction level consists of a pooling layer with a stride of 2 pixels and two deformable convolutional layers of size 3×3×128 pixels and a stride of 1 pixel.

[0021] In one implementation, the key point detector includes a convolution layer with a size of 3×3×1 pixels and a step size of 1 pixel, as well as a first weight module and an accumulation module.

[0022] In one implementation, the key point detector also includes a key point judgment module, which is used to judge whether a coordinate point on the key point detection map is a key point. If the value at the coordinate point on the key point detection map is greater than a preset detection threshold, the coordinate point is marked as a key point.

[0023] In one implementation, the key point descriptor also includes a second weight module to multiply the feature maps of the four original image resolutions by weight coefficients in order from shallow to deep and stack them together according to the channel dimension to obtain a description map with a size of H×W×384.

[0024] In one implementation, a key point matching module is further included, and the key point matching module is used to:

[0025] Extract key point detection images and descriptors corresponding to the two depth images to be detected;

[0026] For the key point descriptors corresponding to the two depth images to be detected, the nearest neighbor matching algorithm or deep learning matching algorithm is used to match them in the vector space.

[0027] In one implementation, the key point detector is trained by the following steps:

[0028] Constructing a synthetic data set, wherein an image in the synthetic data set includes at least points, lines or polygons;

[0029] Constructing an initial model, wherein the initial model includes a backbone feature extraction model, an upsampling module, an initial key point detector, and an initial key point descriptor;

[0030] Inputting images in the synthetic data set into the initial model, training the initial key point detector, and generating a training data set with pseudo-true value key point coordinate labels;

[0031] Train the initial keypoint detector using the training dataset with pseudo-true keypoint coordinate labels, and generate new pseudo-true keypoint coordinate labels again, and repeat this step multiple times until the keypoint feature detection loss of the initial keypoint detector no longer decreases;

[0032] And save the initial keypoint detector whose feature detection loss no longer decreases as the new keypoint detector.

[0033] In one implementation, the method further includes:

[0034] Replacing an initial key point detector in the initial model with a new key point detector;

[0035] An image of the training set is input, and the parameter values ​​of the initial model are adjusted through the back-propagation mechanism according to the key point detection error and the key point description loss between the label value and the predicted value, so as to finally obtain a deep image key point detection and description model, thereby obtaining the deep image key point detection and description model.

[0036] A second aspect of the present application provides a method for detecting and describing key points of a depth image, the method comprising:

[0037] Acquire a depth image to be detected, where the size of the depth image to be detected is H×W×1 pixel;

[0038] Performing pooling, convolution, or deformable convolution on the depth image to be detected to obtain a first feature map with a size of H×W×64 pixels, a second feature map with a size of H / 2×W / 2×64 pixels, a third feature map with a size of H / 4×W / 4×128 pixels, and a fourth feature map with a size of H / 8×W / 8×128 pixels;

[0039] Upsampling the first feature map, the second feature map, the third feature map, and the fourth feature map and performing convolution to eliminate aliasing effects to obtain four feature maps with original image resolutions, whose sizes are: H×W×64 pixels, H×W×64 pixels, H×W×128 pixels, and H×W×128 pixels, respectively;

[0040] Convolve the feature maps of the four original image resolutions to obtain four feature maps of size H×W×1 pixels. Multiply the feature maps of the four sizes by coefficients of 0.1, 0.2, 0.3, and 0.4 in order from shallow to deep, and then accumulate them to obtain a key point detection map of H×W×1 pixels.

[0041] The feature maps of the four original image resolutions are stacked in the feature dimension to obtain a feature map of H×W×384 pixels, which is then passed through two fully connected layers for feature map screening to obtain a description feature map of size H×W×128 pixels. Finally, the description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length, which is composed of a 128-dimensional vector located at the coordinates of the key points in the description feature map.

[0042] In one implementation, the method further includes:

[0043] Extract key points and descriptors of two depth images to be detected;

[0044] For the two depth image descriptors to be detected, the nearest neighbor matching algorithm or deep learning matching algorithm is used for matching in the vector space.

[0045] It can be seen from the above technical solutions that the present application provides a detection and description model and method for key points of depth images, wherein the detection and description model for key points of depth images includes a backbone feature extraction model, an upsampling module, a key point detector and a key point descriptor. In actual application, the depth image shields the complex effects of weather and illumination changes. On the one hand, in order to enhance the network's positioning accuracy for key points, the upsampling module reorganizes multiple levels of low-resolution feature maps to the original image resolution in a bilinear upsampling manner, and performs key point detection at the original image resolution, thereby improving the model's key point positioning accuracy. On the other hand, in order to enhance the network's perception of local shape details, the key point descriptor aggregates multiple layers of features together, and uses the dynamic receptive field of the deformable convolutional layer to achieve content perception of the input depth image, while filtering out the noise on the shallow feature map. In addition, a two-layer fully connected network is used to screen out robust key point descriptors. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 A schematic diagram of the structure of a detection and description model for key points of a depth image provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the training process of a detection and description model for key points of a depth image provided in an embodiment of the present application;

[0049] Figure 3 A flowchart of a method for detecting and describing key points of a depth image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to provide a method for enhancing the network's positioning accuracy for key points, and enhancing the network's perception of local shape details, an embodiment of the present application provides a detection and description model and method for depth image key points.

[0051] See also Figure 1 , which is a structural diagram of a depth image key point detection and description model provided in an embodiment of the present application.

[0052] The detection and description model of a depth image key point includes:

[0053] A backbone feature extraction model, the backbone feature extraction model includes a depth image input end, and a first feature extraction level, a second feature extraction level, a third feature extraction level and a fourth feature extraction level connected in sequence.

[0054] The depth image input end is used to obtain a depth image to be detected, and the size of the depth image to be detected is H×W×1 pixel.

[0055] The first feature extraction level includes two convolution layers with a size of 3×3×64 pixels and a step size of 1 pixel, which are used to convolve the depth image to be detected into a first feature map with a size of H×W×64 pixels.

[0056] The second feature extraction level includes a pooling layer with a step size of 2 pixels and two convolution layers with a size of 3×3×64 pixels and a step size of 1 pixel. The first feature map first passes through a pooling layer with a step size of 2 pixels and is downsampled to obtain a feature map with a size of H×W×64. The use of the pooling layer for downsampling can effectively prevent overfitting of the feature extraction process, thereby reducing the number of parameters of the feature extraction network. Then, the first feature map passes through convolution layers with a size of 3×3×64 pixels and a step size of 1 pixel on both sides to obtain a second feature map with a size of H / 2×W / 2×64 pixels.

[0057] The third feature extraction level includes a pooling layer with a step size of 2 pixels and two convolution layers with a size of 3×3×128 pixels and a step size of 1 pixel. The second feature map first passes through a pooling layer with a step size of 2 pixels and is downsampled to obtain a feature map of H / 4×W / 4×128 pixels to prevent overfitting in the feature extraction process, thereby reducing the number of parameters of the feature extraction network. It then passes through two convolution layers with a size of 3×3×128 pixels and a step size of 1 pixel to obtain a third feature map with a size of H / 4×W / 4×128 pixels.

[0058] The fourth feature extraction level includes a pooling layer with a step size of 2 pixels and two deformable convolution layers with a size of 3×3×128 pixels and a step size of 1 pixel, so the third feature map first passes through a pooling layer with a step size of 2 pixels and is downsampled to obtain a feature map of H / 4×W / 4×128 pixels to prevent overfitting of the feature extraction process and reduce the number of parameters of the feature extraction network, and then passes through two deformable convolution layers with a size of 3×3×128 pixels and a step size of 1 pixel to obtain a fourth feature map with a size of H / 8×W / 8×128 pixels.

[0059] It should be noted that ordinary convolution units sample fixed positions of the input feature map, that is, for a convolution layer, the receptive fields of all its activation units are the same, but in actual applications, the depth images that need to be processed often include objects with complex geometric shapes, and the 2D geometric shapes and sizes of the same object obtained from taking images at different angles and distances are often not completely consistent. Therefore, in order to adapt the receptive field of the convolutional neural network, the convolution layer used in the fourth feature extraction level of the embodiment of the present application is a deformable convolution layer, thereby enhancing the feature extraction of the network.

[0060] An upsampling module includes four upsampling units and sampling convolution layers respectively connected to the first feature extraction level, the second feature extraction level, the third feature extraction level and the fourth feature extraction level. The upsampling unit is used to upsample the first feature map, the second feature map, the third feature map and the fourth feature map, that is, the second feature map, the third feature map and the fourth feature map are upsampled by 2 times, 4 times and 8 times respectively. The sampling convolution layer is used to eliminate the aliasing effect that may occur in the sampling to obtain feature maps of four original image resolutions, with sizes of H×W×64 pixels, H×W×64 pixels, H×W×128 pixels and H×W×128 pixels respectively.

[0061] The key point detector includes a convolution layer with a size of 3×3×1 pixels and a step size of 1 pixel, a first weight module and an accumulation module. The feature maps of the four original image resolutions are convolved by the convolution layer with a size of 3×3×1 pixels and a step size of 1 pixel to obtain four feature maps with a size of H×W×1 pixels, and the feature maps of the four sizes are multiplied by coefficients of 0.1, 0.2, 0.3 and 0.4 in order from shallow to deep, and then accumulated to obtain a key point detection map of H×W×1. Among them, the feature maps from shallow to deep refer to the order of the first layer features, the second layer features, the third layer features and the fourth layer features.

[0062] The key point descriptor includes a second weight module and two fully connected layers (FC) to multiply the feature maps of the four original image resolutions by weight coefficients (0.1, 0.2, 0.3 and 0.4) in order from shallow to deep, and stack them together according to the channel dimension to obtain a description map with a size of H×W×384, and then pass through two fully connected layers with sizes of 384×128 pixels and 128×128 pixels respectively to perform feature map screening to obtain a description feature map with a size of H×W×128 pixels. Finally, the description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length, and the descriptor is composed of a 128-dimensional vector located at the coordinates of the key point in the description feature map.

[0063] It can be seen from the above technical scheme that a detection and description model for key points of a depth image provided by an embodiment of the present application includes a backbone feature extraction model, an upsampling module, a key point detector, and a key point descriptor. In actual application, on the one hand, in order to enhance the network's positioning accuracy for key points, the upsampling module reorganizes multiple levels of low-resolution feature maps to the original image resolution in a bilinear upsampling manner, thereby improving the model's key point positioning accuracy. On the other hand, in order to enhance the network's perception of local shape details, the key point descriptor aggregates multiple layers of features together and utilizes the dynamic receptive field of the deformable convolutional layer to achieve content perception of the input depth image while filtering out noise on the shallow feature map. In addition, a two-layer fully connected network is used to screen out robust key point descriptors.

[0064] Furthermore, in determining whether a coordinate point on the key point detection map is a key point, the key point detector also includes a key point determination module, which is used to determine whether the value at the coordinate point on the key point detection map is greater than a preset detection threshold. If the value at the coordinate point on the key point detection map is greater than the preset detection threshold, the point corresponding to the coordinate point on the original image is marked as a key point.

[0065] Among them, the detection threshold α = 0.5. Considering that too dense key points are harmful to the performance of the model, in the key point judgment module, non-maximum suppression (NMS) of size 3 is used to delete key points that are too close in spatial distribution, and the coordinates of the key points are recorded, wherein the origin of the coordinate system of the coordinates in the embodiment of the present application is the upper left corner of the feature map.

[0066] Furthermore, a detection and description model for key points of a depth image provided in an embodiment of the present application realizes image matching based on key point detection and description of the depth image, thereby realizing other processes such as anomaly detection or three-dimensional reconstruction. Specifically, the detection and description model for key points of a depth image also includes a key point matching module; used to extract key point detection images and descriptors corresponding to two depth images to be detected; and the coordinates of the key points in the two key point detection images and their corresponding descriptors are matched in the vector space using a nearest neighbor matching algorithm or a deep learning matching algorithm. In an embodiment of the present application, key point detection is first performed, key point description is performed, and then the descriptors of the key points are matched. Each descriptor has a corresponding key point coordinate. The descriptor matching result can be used to obtain the coordinate correspondence of the key points in the two depth images, and then the accurate (pixel level) correspondence between the two depth images is obtained.

[0067] Figure 2 A schematic diagram of the training process of a detection and description model for key points in a depth image provided in an embodiment of the present application; in order to more clearly introduce a detection and description model for key points in a depth image provided in an embodiment of the present application, the establishment and training process of the detection and description model for key points in a depth image are described below.

[0068] S201, constructing a synthetic data set, wherein the image in the synthetic data set includes at least points, lines or polygons. The depth image data set is formed by a 3D scanner installed near the rails, taking pictures when a train passes by to obtain depth images.

[0069] First, a synthetic data set containing images of basic shapes such as points, lines, triangles, etc. is constructed. In an embodiment of the present application, the synthetic data set is used to pre-train the key point detector.

[0070] In the embodiment of the present application, a 3D scanner installed near the rails is used to take photos when a train passes by to obtain a depth image data set, that is, the depth image used in the present application is a depth image of a high-speed train, and the pixel value of the image is only related to the distance between the scanner imaging point and the train surface, thereby shielding the influence of the complex operating environment of the high-speed train. Training on this can enhance the accuracy of key point detection. As a result, the abnormal detection task of the high-speed train is less affected by the complex working environment of the high-speed rail system.

[0071] Specifically, for the synthetic dataset, each synthetic image contains several basic shapes, such as points, lines, triangles or quadrilaterals, and a corresponding annotation file. The annotation file records the coordinates of the endpoints and corner points of the basic shapes, such as the endpoints of lines, the corner points of triangles and quadrilaterals, and the origin of the coordinates is the upper left corner of the depth image. These coordinates are used as labels for pre-training of the key point detector.

[0072] S202, constructing an initial model, wherein the initial model includes a backbone feature extraction model, an upsampling module, an initial key point detector, and an initial key point descriptor.

[0073] S203, inputting the images in the synthetic data set into the initial model, training the initial key point detector, and generating a training data set with pseudo-true value key point coordinate labels.

[0074] On the synthetic dataset, an initial keypoint detector is pre-trained, and then the pre-trained initial keypoint detector is used to generate pseudo-true keypoint coordinate labels for the training dataset of depth images.

[0075] S204, train the initial key point detector using the training data set with the pseudo-true value key point coordinate labels, and generate new pseudo-true value key point coordinate labels again, and repeat this step multiple times until the key point feature detection loss of the initial key point detector no longer decreases.

[0076] The purpose of repeatedly generating new pseudo-true value coordinate labels is to optimize the performance of the initial key point detector and improve the detection accuracy of the key points until the key point feature detection loss of the initial key point detector no longer decreases. The initial key point detector whose feature detection loss no longer decreases is saved as a new key point detector.

[0077] S205, replacing the initial key point detector in the initial model with a new key point detector, using the latest key point coordinate labels on the depth image dataset, and jointly training the initial key point detector and the initial key point descriptor.

[0078] Specifically, an image of the synthetic dataset is input to the backbone feature extraction network with a size of H×W×1. The backbone feature extraction model obtains feature maps with a size of H×W×64 pixels, a feature map with a size of H / 2×W / 2×64 pixels, a feature map with a size of H / 4×W / 4×128 pixels, and a feature map with a size of H / 8×W / 8×128 pixels after being processed by the deformable convolutional network.

[0079] In the upsampling module, all the feature maps of these four layers are upsampled to the original image resolution by bilinear interpolation. That is, the feature maps of the last three levels are upsampled by 2 times, 4 times, and 8 times respectively, and finally four levels of feature maps are obtained: H×W×64, H×W×64, H×W×128, and H×W×128.

[0080] For the feature detection part, the feature maps of these four levels are respectively passed through a convolution layer with a size of 3×3×1 pixels and a step size of 1 pixel to obtain four feature maps of size H×W×1 pixels. Then, considering that shallow features are easily affected by noise, the feature maps of these four layers from shallow to deep are multiplied by coefficients: 0.1, 0.2, 0.3 and 0.4, and then accumulated to obtain a key point detection map of H×W×1. The condition for a point on the detection map to be judged as a key point is that the value at the coordinate of the point is greater than the detection threshold α=0.5. Considering that too dense key points are harmful to the performance of the network, non-maximum suppression (NMS) of size 3 is used to delete key points that are too close in space. The coordinates of these key points are recorded, and the origin of the coordinates is the upper left corner of the feature map.

[0081] Then the detection can be regarded as an unbalanced binary classification problem: that is, whether the point located at the label point on the predicted key point depth image is judged as a key point. Obviously, the number of negative samples is much larger than the number of positive samples. Therefore, the loss of the detector can be constructed by a cross entropy loss function until the detection loss no longer decreases, and the key point detector is saved.

[0082] Specifically, load the initial key point detector, input the depth image in the depth image dataset, and label each depth image in the depth image dataset with a pseudo-true value key point label. Then use the label and the training set image to retrain the initial key point detector. When the detection loss no longer decreases, save the initial key point detector again, generate a pseudo-true value label, and then use the new key point coordinate label and dataset to retrain the saved initial key point detector. After repeating this process several times, the detection loss no longer decreases, save the initial key point detector, and complete the separate training of the initial key point detector.

[0083] Furthermore, the initial key point detector and the initial key point descriptor are jointly trained on the separately trained initial key point detector.

[0084] Specifically, the initial key point detector in the initial model is replaced with a new key point detector.

[0085] A depth image dataset is input, and the parameter values ​​of the initial model are adjusted through a back-propagation mechanism based on the key point detection error and key point description loss between the label value and the predicted value, so as to finally obtain a depth image key point detection and description model, thereby obtaining the depth image key point detection and description model.

[0086] Specifically, for each image I, a projection matrix H is randomly generated, and then the matrix is ​​used to randomly change the perspective of this depth image to obtain I'. I, I' and H are saved as a sequence. Input an original image I in sequence, with the key point set as K, calculate its key point detection loss, and then input its corresponding transformed depth image I', with K' as the key point set of I', and calculate its key point detection loss.

[0087] For the feature description part, the four levels of feature maps from shallow to deep obtained by the backbone multi-scale feature extraction network are multiplied by weight coefficients: 0.1, 0.2, 0.3 and 0.4 respectively, and then stacked together according to the channel dimension to obtain a description map of size H×W×384. Then, two layers of fully connected layers with sizes of 384×128 and 128×128 are passed to filter the feature maps, and the final description feature map size is H×W×128. Finally, L2 regularization is performed on the description feature map in the channel dimension to construct a descriptor of unit length. The final descriptor is composed of a 128-dimensional vector located at the coordinates of the key points in the description feature map.

[0088] For a pair of key points k and k', k comes from image I and k' comes from I'. The condition for k and k' to be paired is that their coordinate distance is less than 3 pixels. The corresponding descriptors are G and G'. Then the descriptor loss can be described by the ternary loss between G and G', and the total loss of the network is the sum of the key point detection loss and the key point description loss.

[0089] In this way, the initial key point detector and the initial key point descriptor are jointly trained. After the loss gradually stabilizes and no longer decreases, the trained initial key point detector and initial key point descriptor are saved to obtain the final initial key point detector and key point descriptor based on deep learning for downstream tasks such as deep image matching or three-dimensional reconstruction.

[0090] Corresponding to the aforementioned embodiment of a detection and description model for depth image key points, the present application also provides an embodiment of a detection and description method for depth image key points. The detection and description method for depth image key points is implemented using the detection and description model for depth image key points.

[0091] See also Figure 3 , which is a flow chart of a method for detecting and describing key points of a depth image provided in an embodiment of the present application.

[0092] The method for detecting and describing key points of a depth image comprises:

[0093] S301 , obtaining a depth image to be detected, where the size of the depth image to be detected is H×W×1 pixel.

[0094] S302, performing pooling, convolution or deformable convolution on the depth image to be detected to obtain a first feature map with a size of H×W×64 pixels, a second feature map with a size of H / 2×W / 2×64 pixels, a third feature map with a size of H / 4×W / 4×128 pixels, and a fourth feature map with a size of H / 8×W / 8×128 pixels.

[0095] S303, up-sample the first feature map, the second feature map, the third feature map and the fourth feature map to obtain four feature maps with original image resolutions, whose sizes are: H×W×64 pixels, H×W×64 pixels, H×W×128 pixels and H×W×128 pixels, respectively.

[0096] S304, convolve the feature maps of the four original image resolutions respectively to obtain four feature maps of size H×W×1 pixels, and multiply the feature maps of the four sizes by coefficients of 0.1, 0.2, 0.3 and 0.4 in order from shallow to deep, and then accumulate them to obtain a key point detection map of H×W×1.

[0097] S305, stack the feature maps of the four original image resolutions together according to the channel dimension to obtain a description map with a size of H×W×384, and then pass them through two fully connected layers to perform feature map screening to obtain a description feature map with a size of H×W×128 pixels. Finally, L2 regularization is performed on the description feature map in the channel dimension to construct a descriptor of unit length, and the descriptor is composed of a 128-dimensional vector located at the coordinates of the key point in the description feature map.

[0098] Furthermore, the method for detecting and describing key points of a depth image further includes:

[0099] Extract key point detection images and descriptors corresponding to the two high-speed train depth images to be detected.

[0100] For the descriptors of the corresponding coordinate points in the two key point detection images, the nearest neighbor matching algorithm or deep learning matching algorithm is used to match them in the vector space.

[0101] It can be seen from the above technical solutions that the embodiment of the present application provides a detection and description model and method for key points of a depth image, wherein the detection and description model for key points of a depth image includes a backbone feature extraction model, an upsampling module, a key point detector and a key point descriptor. In actual application, the depth image shields the complex effects of weather and lighting changes. On the one hand, in order to enhance the network's positioning accuracy for key points, the upsampling module reorganizes multiple levels of low-resolution feature maps to the original image resolution in a bilinear upsampling manner, thereby improving the model's key point positioning accuracy. On the other hand, in order to enhance the network's perception of local shape details, the key point descriptor aggregates multiple layers of features together, and uses the dynamic receptive field of the deformable convolutional layer to achieve content perception of the input depth image, while filtering out the noise on the shallow feature map. In addition, a two-layer fully connected network is used to screen out robust key point descriptors.

[0102] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above are only specific implementation methods of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. A detection and description model for key points of a depth image, characterized in that: include: A backbone feature extraction model, the backbone feature extraction model comprising a depth image input terminal, and a first feature extraction level, a second feature extraction level, a third feature extraction level, and a fourth feature extraction level connected in sequence; The depth image input terminal is used to obtain a depth image to be detected, and the size of the depth image to be detected is H×W×1 pixel; The first feature extraction level is used to convolve the depth image to be detected into a first feature map with a size of H×W×64 pixels; The second feature extraction level performs pooling and convolution on the first feature map to obtain a second feature map with a size of H / 2×W / 2×64 pixels; The third feature extraction level performs pooling and convolution on the second feature map to obtain a third feature map with a size of H / 4×W / 4×128 pixels; The fourth feature extraction level performs pooling and deformable convolution on the third feature map to obtain a fourth feature map with a size of H / 8×W / 8×128 pixels; An upsampling module, the upsampling module comprising four upsampling units and sampling convolution layers respectively connected to the first feature extraction level, the second feature extraction level, the third feature extraction level and the fourth feature extraction level, the upsampling unit is used to upsample the first feature map, the second feature map, the third feature map and the fourth feature map, the sampling convolution layer is used to eliminate the aliasing effect that may occur in the upsampling, so as to obtain feature maps of four original image resolutions, the feature map sizes of the four original image resolutions are: H×W×64 pixels, H×W×64 pixels, H×W×128 pixels and H×W×128 pixels respectively; The key point detector is used to perform a convolution of 1 layer of 3×3×1 pixels and a step size of 1 pixel on the feature maps of the four original image resolutions to obtain four feature maps of size H×W×1 pixels. The feature maps of the four sizes are multiplied by coefficients of 0.1, 0.2, 0.3 and 0.4 in order from shallow to deep, and then accumulated to obtain a key point detection map of H×W×1 pixels. The key point descriptor includes two fully connected layers, which are used to screen the feature maps of four original image resolutions stacked according to the feature scale to obtain a description feature map with a size of H×W×128 pixels. Finally, the description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length. The descriptor is composed of a 128-dimensional vector located at the key point coordinates in the description feature map.

2. A detection and description model for key points of a depth image according to claim 1, characterized in that: The first feature extraction level includes two convolutional layers with a size of 3×3×64 pixels and a stride of 1 pixel; The second feature extraction level includes a pooling layer with a stride of 2 pixels and two convolutional layers with a size of 3×3×64 pixels and a stride of 1 pixel; The third feature extraction level includes a pooling layer with a stride of 2 pixels and two convolutional layers with a size of 3×3×128 pixels and a stride of 1 pixel; The fourth feature extraction level consists of a pooling layer with a stride of 2 pixels and two deformable convolutional layers of size 3×3×128 pixels and a stride of 1 pixel.

3. The detection and description model of depth image key points according to claim 1, characterized in that: The key point detector includes a convolution layer with a size of 3×3×1 pixels and a step size of 1 pixel, a first weight module and an accumulation module.

4. A detection and description model for depth image key points according to claim 3, characterized in that: The key point detector also includes a key point judgment module, which is used to judge whether a coordinate point on the key point detection map is a key point. If the value at the coordinate point on the key point detection map is greater than a preset detection threshold, the coordinate point is marked as a key point.

5. The detection and description model of depth image key points according to claim 1, characterized in that: The key point descriptor also includes a second weight module to multiply the feature maps of the four original image resolutions by weight coefficients in order from shallow to deep and stack them together according to the channel dimension to obtain a description map of size H×W×384.

6. The detection and description model of depth image key points according to claim 1, characterized in that: It also includes a key point matching module, which is used to: Extract key point detection images and descriptors corresponding to the two depth images to be detected; For the key point descriptors corresponding to the two depth images to be detected, the nearest neighbor matching algorithm or deep learning matching algorithm is used to match them in the vector space.

7. The detection and description model of depth image key points according to claim 1, characterized in that: The key point detector is trained by the following steps: Constructing a synthetic data set, wherein an image in the synthetic data set includes at least points, lines or polygons; Constructing an initial model, wherein the initial model includes a backbone feature extraction model, an upsampling module, an initial key point detector, and an initial key point descriptor; Inputting images in the synthetic data set into the initial model, training the initial key point detector, and generating a training data set with pseudo-true value key point coordinate labels; Train the initial keypoint detector using the training dataset with pseudo-true keypoint coordinate labels, and generate new pseudo-true keypoint coordinate labels again, and repeat this step multiple times until the keypoint feature detection loss of the initial keypoint detector no longer decreases; And save the initial keypoint detector whose feature detection loss no longer decreases as the new keypoint detector.

8. A depth image key point detection and description model according to claim 7, characterized in that: Also includes: Replacing an initial key point detector in the initial model with a new key point detector; An image of the training set is input, and the parameter values ​​of the initial model are adjusted through the back-propagation mechanism according to the key point detection error and the key point description loss between the label value and the predicted value, so as to finally obtain a deep image key point detection and description model, thereby obtaining the deep image key point detection and description model.

9. A method for detecting and describing key points of a depth image, characterized in that: The method for detecting and describing key points of a depth image comprises: Acquire a depth image to be detected, where the size of the depth image to be detected is H×W×1 pixel; Performing pooling, convolution, or deformable convolution on the depth image to be detected to obtain a first feature map with a size of H×W×64 pixels, a second feature map with a size of H / 2×W / 2×64 pixels, a third feature map with a size of H / 4×W / 4×128 pixels, and a fourth feature map with a size of H / 8×W / 8×128 pixels; Upsampling the first feature map, the second feature map, the third feature map, and the fourth feature map and performing convolution to eliminate aliasing effects to obtain feature maps of four original image resolutions, where the sizes of the feature maps of the four original image resolutions are: H×W×64 pixels, H×W×64 pixels, H×W×128 pixels, and H×W×128 pixels, respectively; Convolve the feature maps of the four original image resolutions to obtain four feature maps of size H×W×1 pixels. Multiply the feature maps of the four sizes by coefficients of 0.1, 0.2, 0.3, and 0.4 in order from shallow to deep, and then accumulate them to obtain a key point detection map of H×W×1 pixels. The feature maps of the four original image resolutions are stacked in the feature dimension to obtain a feature map of H×W×384 pixels, which is then passed through two fully connected layers for feature map screening to obtain a description feature map of size H×W×128 pixels. Finally, the description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length, which is composed of a 128-dimensional vector located at the coordinates of the key points in the description feature map.

10. The method for detecting and describing key points of a depth image according to claim 9, characterized in that: Also includes: Extract key points and descriptors of two depth images to be detected; For the two depth image descriptors to be detected, the nearest neighbor matching algorithm or deep learning matching algorithm is used for matching in the vector space.

Images