A system and method for key point detection and description of high-speed train images
The joint key point detection and description of high-speed rail train images is solved through deep learning algorithms, which solves the problems of large amount of calculation and insufficient robustness of traditional methods, and achieves high-precision and robust image registration.
Patent Information
- Application Number
- CN202210414568.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-15
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2042-04-15
AI Technical Summary
Among the existing high-speed train image registration methods, the grayscale-based method has a large amount of calculation and takes a long time, while the feature-based method is not robust to complex scenes, making it difficult to meet the dynamic measurement, lighting, weather changes and other factors. Requirements for image appearance change.
Using deep learning algorithms, joint key point detection and description are carried out through the backbone feature extraction module, upsampling module, key point detector and key point descriptor, and robust key point descriptors are generated, which overcomes the large amount of calculations of traditional methods. The problem of insufficient robustness.
The positioning accuracy and robustness of key points are improved, and the generated descriptors can be effectively matched in complex scenarios, realizing accurate registration of high-speed train images.
Smart Images
Figure CN114820732B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a system and method for detecting and describing key points of a high-speed train image. Background Art
[0002] Image registration can be regarded as a basic task in the high-speed train inspection process. For example, the train chassis is an important part connecting the train and the track, and its safety inspection is an important part of ensuring the safety of high-speed train operation.
[0003] At present, the detection schemes for most high-speed trains in my country are improved and developed based on the Trouble of moving EMU Detection System (TEDS). The method is to first use a computer to match the images collected by the linear array camera installed near the rails with the stored historical images, and then use automatic recognition technology or manual identification of faults or foreign objects. If an abnormality is detected, it will remind humans to check again. Compared with the traditional purely manual method, this combination of man and machine saves time for train fault detection and reduces the demand for staff.
[0004] Traditional high-speed train image registration methods are mainly divided into grayscale-based registration and feature-based registration. Grayscale-based image registration can be regarded as an iterative optimization process that directly compares all the contents of two images. Although this method is simple to implement, it is computationally intensive and time-consuming, and it is difficult to meet the requirements of dynamic measurement in practice. In view of the characteristics of high-speed train images, many researchers have improved popular key point detection and description algorithms such as SIFT, SURF, and ORB. Feature-based image registration methods only need to calculate feature descriptors for key points, rather than for the entire image content, so the calculation amount is small and the calculation speed is fast, but traditional feature extraction algorithms can usually only extract shallow features, and it is difficult to obtain deeper and more expressive features. For situations where the appearance of images changes too much due to changes in lighting and weather, these methods have problems such as low registration accuracy and lack of generalization ability.
[0005] In summary, in order to provide a high-speed train image registration method that can overcome the shortcomings of grayscale-based registration algorithms, such as large computational complexity and long time consumption, and the shortcomings of traditional feature-based image registration algorithms that are not robust to more complex scenes, it has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention
[0006] In order to provide a method for enhancing the network's positioning accuracy for key points, an embodiment of the present application provides a system and method for key point detection and description of high-speed train images.
[0007] The first aspect of the present application provides a system for key point detection and description of a high-speed train image, comprising:
[0008] A backbone feature extraction module, wherein the backbone feature extraction model includes an image input terminal, and a first feature extraction level, a second feature extraction level, and a third feature extraction level connected in sequence;
[0009] The image input end is used to obtain the image to be registered, and the size of the image to be registered is H×W×1 pixel;
[0010] The first feature extraction level is used to convolve the image to be registered into a first feature map with a size of H×W×32 pixels;
[0011] The second feature extraction level convolves the first feature map to obtain a second feature map of size H / 2×W / 2×64 pixels;
[0012] The third feature extraction level convolves the second feature map to obtain a third feature map having a size of H / 4×W / 4×128 pixels;
[0013] An upsampling module, the upsampling module comprising three layers of upsampling units and sampling convolution layers respectively connected to the first feature extraction level, the second feature extraction level and the third feature extraction level, the upsampling unit is used to upsample the first feature map, the second feature map and the third feature map, the sampling convolution layer is used to eliminate the aliasing effect that may occur in the sampling, so as to obtain feature maps of three original image resolutions, the feature map sizes of the three original image resolutions are: H×W×32 pixels, H×W×64 pixels and H×W×128 pixels;
[0014] The key point detector is used to perform a convolution of 1 layer of 3×3×1 pixels and a step size of 1 pixel on the feature maps of the three original image resolutions to obtain three feature maps of size H×W×1 pixels. The feature maps of the three sizes are multiplied by coefficients of 0.17, 0.33, and 0.5 in order from shallow to deep, and then accumulated to obtain a key point detection map of H×W×1.
[0015] The key point descriptor multiplies the feature maps of the three sizes by coefficients 1, 2, and 3 in order from shallow to deep, and then stacks them in the feature dimension to obtain a description feature map of H×W×224. Finally, the description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length. The descriptor is composed of a 224-dimensional vector located at the key point coordinates in the description feature map;
[0016] A feature matching module is used to perform nearest neighbor matching on the descriptors of the key points of the two images to be registered in the vector space to obtain the feature matching results of the two images to be registered;
[0017] The transformation estimation module is used to calculate the perspective transformation matrix between the two images to be registered according to the feature matching results of the two images to be registered.
[0018] In one implementation, the first feature extraction level includes three convolutional layers with a size of 3×3×32 pixels and a stride of 1 pixel;
[0019] The second feature extraction level includes a convolutional layer with a size of 3×3×64 pixels and a stride of 2 pixels and two convolutional layers with a size of 3×3×64 pixels and a stride of 1 pixel;
[0020] The third feature extraction level includes one convolutional layer with a size of 3 × 3 × 128 pixels and a stride of 2 pixels and two convolutional layers with a size of 3 × 3 × 128 pixels and a stride of 1 pixel.
[0021] In one implementation, the key point detector includes a convolution layer with a size of 3×3×1 pixels and a step size of 1 pixel, as well as a first weight module and a feature accumulation module.
[0022] In one implementation, the key point detector also includes a key point judgment module, which is used to determine whether the value at the coordinate point on the key point detection map is greater than the preset detection threshold. If the value at the coordinate point on the key point detection map is greater than the preset detection threshold, the point corresponding to the coordinate point on the original image is marked as a key point.
[0023] In one implementation, the key point descriptor also includes a second weight module to multiply the feature maps of the three original image resolutions by weight coefficients in order from shallow to deep and stack them together according to the channel dimension to obtain a description map with a size of H×W×224.
[0024] In one implementation, the key point detector is trained by the following steps:
[0025] Constructing a synthetic data set, wherein an image in the synthetic data set includes at least points, lines or polygons;
[0026] Constructing an initial model, wherein the initial model includes a backbone extraction model, an upsampling module, an initial key point detector, and an initial key point descriptor;
[0027] Inputting images in the synthetic data set into the initial model, training the initial key point detector, saving the initial key point detector model, and using the saved initial key point detector model to detect key points of the training set images to generate a training data set with pseudo-true value key point coordinate labels;
[0028] Train the initial keypoint detector using the training dataset with pseudo-true keypoint coordinate labels, and generate new pseudo-true keypoint coordinate labels again, and repeat this step multiple times until the keypoint feature detection loss of the initial keypoint detector no longer decreases;
[0029] And save the initial keypoint detector whose feature detection loss no longer decreases as the new keypoint detector.
[0030] In one implementation, the method further includes:
[0031] Replacing an initial key point detector in the initial model with a new key point detector;
[0032] An image of the training set is input, and the parameter values of the initial model are adjusted through the back-propagation mechanism according to the key point detection error and the key point description loss between the label value and the predicted value, so as to finally obtain the image key point detection and description model, and obtain the key point detection and description system of the high-speed train image.
[0033] A second aspect of the present application provides a method for detecting and describing key points of a high-speed train image, the method comprising:
[0034] Acquire an image to be registered, where the size of the image to be registered is H×W×1 pixel;
[0035] Convolving the image to be registered to obtain a first feature map with a size of H×W×32 pixels, a second feature map with a size of H / 2×W / 2×64 pixels, and a third feature map with a size of H / 4×W / 4×128 pixels;
[0036] Upsampling and convolving the first feature map, the second feature map, and the third feature map to eliminate aliasing effects, to obtain feature maps of three original image resolutions, where the sizes of the feature maps of the three original image resolutions are: H×W×32 pixels, H / 2×W / 2×64 pixels, and H / 4×W / 4×128 pixels, respectively;
[0037] Convolve the feature maps of the three original image resolutions to obtain three feature maps of size H×W×1 pixels. Multiply the feature maps of the three sizes by coefficients 0.17, 0.33, and 0.5 in order from shallow to deep, and then accumulate them to obtain a key point detection map of H×W×1.
[0038] The feature maps of the three original image resolutions are multiplied by coefficients 1, 2, and 3 in order from shallow to deep, and then stacked in the feature dimension to obtain a description feature map of size H×W×224 pixels. Finally, the description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length, which is composed of a 224-dimensional vector located at the coordinates of the key point in the description feature map;
[0039] Obtaining the descriptors of the key points of the two images to be registered, and performing nearest neighbor matching on the descriptors of the key points of the two images to be registered in the vector space to obtain feature matching results of the two images to be registered;
[0040] According to the feature matching results of the two images to be registered, the perspective transformation matrix between the two images to be registered is calculated.
[0041] The present application relates to the field of image processing technology, and provides a system and method for detecting and describing key points of a high-speed train deep image. The system for detecting and describing key points of a high-speed train image includes a backbone feature extraction module, an upsampling module, a key point detector, and a key point descriptor. The system for detecting and describing key points of a high-speed train image is based on a deep learning algorithm for self-supervised training, and adopts a classic image registration process: key point detection, key point description, feature matching, and geometric transformation estimation. The use of a deep learning algorithm for joint key point detection and description overcomes the shortcomings of grayscale-based registration algorithms, which are large in computational complexity and time-consuming, and the shortcomings of traditional feature-based image registration algorithms, which are not robust to more complex scenes. The image to be registered in the present application uses multi-level features for joint key point detection and description in a network forward transmission, which greatly improves the positioning accuracy of key points while generating robust key point descriptors. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0043] Figure 1 A schematic diagram of the structure of a high-speed train image registration system provided in an embodiment of the present application;
[0044] Figure 2 A schematic diagram of a key point detector training process provided in an embodiment of the present application;
[0045] Figure 3 A flowchart of a high-speed train image registration method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to provide a method for enhancing the network's positioning accuracy for key points, an embodiment of the present application provides a model and method for high-speed train image registration.
[0047] See also Figure 1 , which is a structural diagram of a system for key point detection and description of high-speed train images provided in an embodiment of the present application.
[0048] The system for detecting and describing key points of a high-speed train image comprises a backbone feature extraction model, wherein the backbone feature extraction model comprises an image input end, and a first feature extraction level, a second feature extraction level and a third feature extraction level connected in sequence.
[0049] The image input end is used to obtain the image to be registered, and the size of the image to be registered is H×W×1 pixel. The first feature extraction level includes three convolution layers with a size of 3×3×32 pixels and a step size of 1 pixel, which are used to convolve the image to be registered into a first feature map with a size of H×W×32 pixels; the second feature extraction level includes a convolution layer with a size of 3×3×64 pixels and a step size of 2 pixels and two convolution layers with a size of 3×3×64 pixels and a step size of 1 pixel. The first feature map first passes through a convolution layer with a size of 3×3×64 pixels and a step size of 2 pixels, and then passes through two convolution layers with a size of 3×3×64 pixels and a step size of 1 pixel to obtain a second feature map with a size of H / 2×W / 2×64 pixels. The third feature extraction level includes a convolutional layer with a size of 3×3×128 pixels and a stride of 2 pixels and two convolutional layers with a size of 3×3×128 pixels and a stride of 1 pixel. The second feature map first passes through a convolutional layer with a size of 3×3×128 pixels and a stride of 2, and then passes through two convolutional layers with a size of 3×3×128 pixels and a stride of 1 pixel to obtain the third feature map with a size of H / 4×W / 4×128.
[0050] It should be noted that H and W in the embodiment of the present application are both 1000, but are not limited to 1000 and can also be set to other values according to the images collected by the design.
[0051] An upsampling module, wherein the upsampling module includes three layers of upsampling units and sampling convolution layers respectively connected to the first feature extraction level, the second feature extraction level and the third feature extraction level, wherein the upsampling unit is used to upsample the first feature map, the second feature map and the third feature map, that is, the second feature map and the third feature map are upsampled by 2 times and 4 times, respectively, and the sampling convolution layer is used to eliminate the aliasing effect that may occur in the sampling to obtain feature maps of three original image resolutions, and the feature map sizes of the three original image resolutions are: H×W×64 pixels, H×W×64 pixels and H×W×128 pixels, respectively.
[0052] The key point detector includes a convolution layer with a size of 3×3×1 pixels and a step size of 1 pixel, a first weight module and an accumulation module. The feature maps of the three original image resolutions are convolved by the convolution layer with a size of 3×3×1 pixels and a step size of 1 pixel to obtain three feature maps with a size of H×W×1 pixels, and the feature maps of the three sizes are multiplied by coefficients of 0.17, 0.33 and 0.5 in order from shallow to deep, and then accumulated to obtain a key point detection map of H×W×1. Among them, the feature map from shallow to deep refers to the order of the first layer feature, the second layer feature and the third layer feature.
[0053] Key point descriptor, the key point descriptor also includes a second weight module to multiply the feature maps of the three original image resolutions by weight coefficients (1, 2 and 3) in order from shallow to deep and stack them together according to the channel dimension to obtain a description map of size H×W×224. Finally, the description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length, and the descriptor is composed of a 224-dimensional vector located at the key point coordinates in the description feature map.
[0054] A feature matching module is used to obtain the descriptors of the key points of the two images to be registered, and perform nearest neighbor matching on the descriptors of the key points of the two images to be registered in the vector space to obtain the feature matching results of the two images to be registered;
[0055] The transformation estimation module is used to calculate the perspective transformation matrix between the two images to be registered according to the feature matching results of the two images to be registered.
[0056] Specifically, in the actual application process, the previously collected image I at the same position of the same train is input s and the currently acquired image I f The feature matching module extracts the descriptors of the key points of the two images to be registered, and then calculates the perspective transformation matrix H between the two images based on the extracted key point descriptors. fs , and output the currently captured image in the perspective transformation matrix H fs The output registration image I f , that is, image I′ f (I′ f =wsrp(H fs ,I f ). Thus, the image I collected previously is obtained s The image I′ under the coordinate f , that is, to achieve the current and previous high-speed train image registration. Among them, feature matching is the previously collected image I s and the currently acquired image I fAll key point descriptors in the image are implemented using the KNN algorithm (K-Nearest Neighbor) and are required to be adjacent to each other to find matching key point pairs in the two images. Then, the (find Fundamental Mat()) function in OpenCV is used to find the coordinate transformation relationship between the two images, that is, the homography transformation matrix M, and the RANSAC algorithm is used to filter outliers.
[0057] It can be seen from the above technical solutions that a high-speed train depth image key point detection and description system provided by an embodiment of the present application includes a backbone feature extraction model, an upsampling module, a key point detector, a key point descriptor, a feature matching module and a transformation estimation module. The high-speed train depth image key point detection and description system is based on a deep learning algorithm for self-supervised training, and adopts a classic image registration process: key point detection, key point description, feature matching and geometric transformation estimation. The use of a deep learning algorithm for joint key point detection and description overcomes the shortcomings of grayscale-based registration algorithms, which are large in computational complexity and time-consuming, as well as the shortcomings of traditional feature-based image registration algorithms that are not robust to more complex scenes. The image uses multi-level features for joint key point detection and description in a network forward transmission, which greatly improves the positioning accuracy of key points while generating robust key point descriptors.
[0058] Furthermore, in determining whether a coordinate point on the key point detection map is a key point, the key point detector also includes a key point determination module, which is used to determine whether the value at the coordinate point on the key point detection map is greater than a preset detection threshold. If the value at the coordinate point on the key point detection map is greater than the preset detection threshold, the point corresponding to the coordinate point on the original image is marked as a key point.
[0059] Among them, the detection threshold α = 0.5. Considering that too dense key points are harmful to the performance of the model, in the key point judgment module, non-maximum suppression (NMS) with a size of 4 is used to delete key points that are too close in spatial distribution, and the coordinates of these key points are recorded. In the embodiment of the present application, the origin of the coordinate system of the coordinates is the upper left corner of the feature map.
[0060] Figure 2 A schematic diagram of a key point detector training process provided in an embodiment of the present application; in order to more clearly introduce a system for key point detection and description of high-speed train images provided in an embodiment of the present application, the system establishment and training process of key point detection and description of high-speed train images are explained below.
[0061] S201, constructing a synthetic data set, wherein the image in the synthetic data set includes at least points, lines or polygons. Constructing a synthetic data set of images containing basic shapes such as points, lines, triangles, etc. In the embodiment of the present application, the synthetic data set is used to pre-train a key point detector.
[0062] In an embodiment of the present application, specifically, for a synthetic data set, each image contains several basic shapes, such as points, lines, triangles, quadrilaterals and other basic graphics, and a corresponding annotation file, which records the endpoints and coordinates of the corner points of the basic shapes, such as the endpoints of lines, the corner points of triangles and quadrilaterals, and the coordinate origin is the upper left corner of the image. These coordinates are used as labels for pre-training of the key point detector.
[0063] S202, constructing an initial model, wherein the initial model includes a backbone feature extraction model, an upsampling module, an initial key point detector, and an initial key point descriptor.
[0064] S203, inputting the images in the synthetic data set into the initial model, training the initial key point detector, and generating a training data set with pseudo-true value key point coordinate labels.
[0065] S204, train the initial key point detector using the training data set with the pseudo-true value key point coordinate labels, and generate new pseudo-true value key point coordinate labels again, and repeat this step multiple times until the key point feature detection loss of the initial key point detector no longer decreases.
[0066] First, pre-train the initial key point detector on the synthetic dataset, and then use the pre-trained initial key point detector to generate pseudo-true value coordinate labels for the data training set of the depth image. Then use the labeled training set to train the initial key point detector, and then use the newly trained initial key point detector to regenerate the labels. This step is repeated several times until the final feature detection loss no longer decreases.
[0067] The purpose of repeatedly generating new pseudo-true value coordinate labels is to optimize the performance of the initial key point detector and improve the detection accuracy of the key points.
[0068] S205, and save the initial key point detector whose feature detection loss no longer decreases as a new key point detector.
[0069] Specifically, an image of the synthetic dataset is input to the backbone feature extraction network with a size of H×W×1, and a feature map with a size of H×W×32 pixels, a feature map with a size of H / 2×W / 2×64 pixels, and a feature map with a size of H / 4×W / 4×128 pixels are obtained respectively.
[0070] The upsampling module performs bilinear interpolation upsampling on all four layers of feature maps to the original image resolution, that is, the feature maps of the last two levels are upsampled by 2 and 4 times respectively, and finally three levels of feature maps are obtained: feature maps of H×W×32 pixels, feature maps of H×W×64 pixels, and feature maps of H×W×128 pixels.
[0071] The three levels of feature maps are respectively passed through a convolution layer with a size of 3×3×1 pixels and a stride of 1, and three feature maps of size H×W×1 pixels are obtained. Then, considering that shallow features are easily affected by noise, in order to reduce the impact of noise, the three layers of feature maps from shallow to deep are multiplied by coefficients: 0.17, 0.33 and 0.5, and then accumulated to obtain a H×W×1 key point detection map. The condition for a point (i, j) on the detection map to be judged as a key point is that the value at the coordinate of the point is greater than the detection threshold α=0.5. Considering that too dense key points are harmful to the performance of the network, non-maximum suppression (NMS) with a size of 4 is used to delete key points that are too close in space. The coordinates of the remaining key points after deletion are recorded, and the origin of the coordinates is the upper left corner of the feature map.
[0072] Then key point detection can be regarded as an unbalanced binary classification problem: that is, whether the point located at the label point on the predicted key point image is judged as a key point. Obviously, the number of negative samples is much larger than the number of positive samples. Therefore, the loss of the initial key point detector can be constructed by a cross entropy loss function:
[0073]
[0074] l d =-λY ij log(K ij )-(1-Y ij )log(1-K ij );
[0075] Among them, λ is the case where the number of positive samples is far less than that of negative samples, H and W are the height and width of the input image, respectively, both of which are 1000 in this paper. K represents the key point detection map, Y is the key point label of the input image, and L d (K,Y) is the key point loss of image i, l d is the detection loss of a key point. After the detection loss stops decreasing for a long time, save the initial key point detector.
[0076] After the keypoint detection loss stops decreasing, save the pre-trained keypoint detector model.
[0077] Since it is very difficult to collect true value labels for key point detection, there is currently a lack of corresponding data sets for high-speed train image registration. For this reason, the embodiment of the present application adopts a transfer learning method to generate pseudo true value labels for key point detection.
[0078] Load the initial key point detector saved above, input the high-speed train training set image, and label each image of the training set with a pseudo-true value key point. Then use the label and the high-speed train training set image to retrain the initial key point detector. After the detection loss does not decrease for a long time, resave the initial key point detector, generate a pseudo-true value label, and then retrain the initial key point detector using the new key point coordinate label and training set. Repeat this process several times until the detection loss no longer decreases. Save the model, complete the separate self-supervised training of the initial key point detector, and obtain a new key point detector.
[0079] The purpose of repeatedly generating new pseudo-true value coordinate labels is to optimize the performance of the initial key point detector and improve the detection accuracy of the key points.
[0080] And save the initial keypoint detector whose feature detection loss no longer decreases as the new keypoint detector.
[0081] Furthermore, the initial key point detector and the initial key point descriptor are jointly trained on the separately trained initial key point detector.
[0082] During the joint training process, the loss consists of two parts: key point detection loss and key point description loss. For each image I, a projection matrix M (i.e., homography transformation matrix M) is generated, and then the matrix is used to randomly change the perspective of the image to obtain I'. I, I' and M are saved as a sequence.
[0083] An image of the training set is input, and the parameter values of the initial model are adjusted through a back-propagation mechanism according to the key point detection error and key point description loss between the label value and the predicted value, so as to finally obtain an image key point detection and description model, thereby obtaining a system for key point detection and description of high-speed train images.
[0084] Specifically, the initial key point detector in the initial model is replaced with a new key point detector, and the labeled high-speed train training set images are used to carry out joint training of the new key point detector and the initial key point descriptor. According to the detection error between the label value and the predicted value and the key point description loss, the parameter value of the model is adjusted through the back propagation mechanism, and finally a system for key point detection and description of high-speed train images as described above is obtained.
[0085] For the feature description part, the three levels of feature maps obtained by the backbone feature extraction model: H×W×32 pixels, H×W×64 pixels and H×W×128 pixels, these three layers of feature maps from shallow to deep are multiplied by weight coefficients: 1, 2, 3 respectively, and then stacked together according to the channel dimension to obtain the final multi-scale aggregated feature map. The size of the final description feature map is H×W×224. The description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length. The final descriptor is composed of a 224-dimensional vector located at the coordinates of the key point in the description feature map.
[0086] The descriptor contains the rich local shape details contained in the shallow features and the deep features that are robust to noise caused by lighting, seasonal changes, etc., so as to obtain a key point descriptor with high discrimination and robustness. In this way, the key points are jointly extracted and described in one forward pass. The multi-scale feature map is used to collect rich local feature information and select the most discriminative descriptor. In this way, dense key points can be extracted with high precision and are robust to the complex situations faced by high-speed railways.
[0087] The specific steps are as follows: load the new key point detector obtained in the previous step of separate training, input the training set of high-speed trains, input the original image of high-speed trains I in sequence, the key point set is K, calculate its key point detection loss, and then input its corresponding transformed image I', K' is the key point set of I', and its key point detection loss is:
[0088]
[0089] For paired keypoint sets K and K′, the corresponding descriptor sets are G and G′. Then the descriptor loss L f It can be described using the ternary loss between G and G′:
[0090]
[0091] in:
[0092] l f (gi)=max(0,p(g i )-n(g i ),g i ∈G;
[0093] Consider two points k∈K and k'∈K'. The condition for pairing these two points is:
[0094] ||warp(k,H)-k′||≤ε;
[0095] For g′ i ∈g′, p(gi) and n(gi) represent pairs of g i and g′i The positive sample distance between the unpaired g i and g′ k The negative sample distance between:
[0096] p(g i )=||g i -g′ i ||;
[0097]
[0098] In order to make the above formula calculation meaningful, k* represents the points within the image boundary, and the total loss of the model is:
[0099] L all (K,K′,Y,Y′,G,G′)=L d (K,Y)+L d (K′,Y′)+L f (G, G′);
[0100] In this way, a new key point detector and an initial key point descriptor are jointly trained. After a preset training period, the loss gradually stabilizes and no longer decreases. The model is saved, and a system for key point detection and description of high-speed train images is obtained.
[0101] Corresponding to the embodiment of the system for key point detection and description of high-speed train images, the present application also provides an embodiment of high-speed train image registration. The high-speed train image registration method is implemented using the system for key point detection and description of high-speed train images.
[0102] See also Figure 3 , which is a flow chart of a method for high-speed train image registration provided in an embodiment of the present application.
[0103] The method for detecting and describing key points of a high-speed train image comprises:
[0104] S301, obtaining a depth image to be registered, wherein the size of the image to be registered is H×W×1 pixel.
[0105] S302, convolve the image to be registered to obtain a first feature map with a size of H×W×32 pixels, a second feature map with a size of H / 2×W / 2×64 pixels, and a third feature map with a size of H / 4×W / 4×128 pixels.
[0106] S303 performs upsampling and convolution to eliminate aliasing effects on the first feature map, the second feature map, and the third feature map to obtain three feature maps with original image resolutions, whose sizes are: H×W×32 pixels, H / 2×W / 2×64 pixels, and H / 4×W / 4×128 pixels, respectively.
[0107] S304, convolve the feature maps of the three original image resolutions respectively to obtain three feature maps of size H×W×1 pixels, and multiply the feature maps of the three sizes by coefficients of 0.17, 0.33 and 0.5 in order from shallow to deep, and then accumulate them to obtain a key point detection map of H×W×1.
[0108] S305, multiply the feature maps of the three original image resolutions by coefficients 1, 2 and 3 in order from shallow to deep, and then stack them in the feature dimension to obtain a description feature map with a size of H×W×224 pixels. Finally, L2 regularization is performed on the description feature map in the channel dimension to construct a descriptor of unit length, and the descriptor is composed of a 224-dimensional vector located at the coordinates of the key point in the description feature map.
[0109] S306, performing nearest neighbor matching on the descriptors of the key points of the two images to be registered in the vector space to obtain feature matching results of the two images to be registered.
[0110] S307, calculating the perspective transformation matrix between the two images to be registered according to the feature matching results of the two images to be registered.
[0111] It can be seen from the above technical solutions that the present application relates to the field of image processing technology, and provides a system and method for detecting and describing key points of a high-speed train deep image. The system for detecting and describing key points of a high-speed train image includes a backbone feature extraction module, an upsampling module, a key point detector, and a key point descriptor. The system for detecting and describing key points of a high-speed train image is based on a deep learning algorithm for self-supervised training, and adopts a classic image registration process: key point detection, key point description, feature matching, and geometric transformation estimation. The use of a deep learning algorithm for joint key point detection and description overcomes the shortcomings of grayscale-based registration algorithms, which are large in computational complexity and time-consuming, and the shortcomings of traditional feature-based image registration algorithms, which are not robust to more complex scenes. The image to be registered in the present application uses multi-level features for joint key point detection and description in a network forward transmission, which greatly improves the positioning accuracy of key points while generating robust key point descriptors.
[0112] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above are only specific implementation methods of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.
Claims
1. A system for key point detection and description of high-speed train images, characterized in that: include: A backbone feature extraction module, wherein the backbone feature extraction model includes an image input terminal, and a first feature extraction level, a second feature extraction level, and a third feature extraction level connected in sequence; The image input end is used to obtain the image to be registered, and the size of the image to be registered is H×W×1 pixel; The first feature extraction level is used to convolve the image to be registered into a first feature map with a size of H×W×32 pixels; The second feature extraction level convolves the first feature map to obtain a second feature map with a size of H / 2×W / 2×64 pixels; The third feature extraction level convolves the second feature map to obtain a third feature map having a size of H / 4×W / 4×128 pixels; An upsampling module, the upsampling module comprising three layers of upsampling units and sampling convolution layers respectively connected to the first feature extraction level, the second feature extraction level and the third feature extraction level, the upsampling unit is used to upsample the first feature map, the second feature map and the third feature map, the sampling convolution layer is used to eliminate the aliasing effect that may occur in the sampling, so as to obtain feature maps of three original image resolutions, the feature map sizes of the three original image resolutions are: H×W×32 pixels, H×W×64 pixels and H×W×128 pixels; The key point detector is used to perform a convolution of 1 layer of 3×3×1 pixels and a step size of 1 pixel on the feature maps of the three original image resolutions to obtain three feature maps of size H×W×1 pixels. The feature maps of the three sizes are multiplied by coefficients of 0.17, 0.33, and 0.5 in order from shallow to deep, and then accumulated to obtain a key point detection map of H×W×1. The key point descriptor multiplies the feature maps of the three sizes by coefficients 1, 2, and 3 in order from shallow to deep, and then stacks them in the feature dimension to obtain a description feature map of H×W×224. Finally, the description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length. The descriptor is composed of a 224-dimensional vector located at the key point coordinates in the description feature map; A feature matching module is used to perform nearest neighbor matching on the descriptors of the key points of the two images to be registered in the vector space to obtain the feature matching results of the two images to be registered; The transformation estimation module is used to calculate the perspective transformation matrix between the two images to be registered according to the feature matching results of the two images to be registered.
2. The system for key point detection and description of high-speed train images according to claim 1, characterized in that: The first feature extraction level includes three convolutional layers with a size of 3×3×32 pixels and a stride of 1 pixel; The second feature extraction level includes a convolutional layer with a size of 3×3×64 pixels and a stride of 2 pixels and two convolutional layers with a size of 3×3×64 pixels and a stride of 1 pixel; The third feature extraction level includes one convolutional layer with a size of 3 × 3 × 128 pixels and a stride of 2 pixels and two convolutional layers with a size of 3 × 3 × 128 pixels and a stride of 1 pixel.
3. The system for key point detection and description of high-speed train images according to claim 1 is characterized in that: The key point detector includes a convolution layer with a size of 3×3×1 pixels and a step size of 1 pixel, a first weight module and a feature accumulation module.
4. The system for key point detection and description of high-speed train images according to claim 3 is characterized in that: The key point detector also includes a key point judgment module, which is used to judge whether the value of the coordinate point on the key point detection map is greater than the preset detection threshold. If the value of the coordinate point on the key point detection map is greater than the preset detection threshold, the point corresponding to the coordinate point on the original image is marked as a key point.
5. The system for key point detection and description of high-speed train images according to claim 1, characterized in that: The key point descriptor also includes a second weight module to multiply the feature maps of the three original image resolutions by weight coefficients in order from shallow to deep and stack them together according to the channel dimension to obtain a description map of size H×W×224.
6. The system for key point detection and description of high-speed train images according to claim 1, characterized in that: The key point detector is trained by the following steps: Constructing a synthetic data set, wherein an image in the synthetic data set includes at least points, lines or polygons; Constructing an initial model, wherein the initial model includes a backbone extraction model, an upsampling module, an initial key point detector, and an initial key point descriptor; Inputting images in the synthetic data set into the initial model, training the initial key point detector, saving the initial key point detector model, and using the saved initial key point detector model to detect key points of the training set images to generate a training data set with pseudo-true value key point coordinate labels; Train the initial keypoint detector using the training dataset with pseudo-true keypoint coordinate labels, and generate new pseudo-true keypoint coordinate labels again, and repeat this step multiple times until the keypoint feature detection loss of the initial keypoint detector no longer decreases; And save the initial keypoint detector whose feature detection loss no longer decreases as the new keypoint detector.
7. The system for key point detection and description of high-speed train images according to claim 6, characterized in that: Also includes: Replacing an initial key point detector in the initial model with a new key point detector; An image of the training set is input, and the parameter values of the initial model are adjusted through a back-propagation mechanism according to the key point detection error and key point description loss between the label value and the predicted value, so as to finally obtain an image key point detection and description model, thereby obtaining a system for key point detection and description of high-speed train images.
8. A method for key point detection and description of a high-speed train image, characterized in that: The method for detecting and describing key points of a high-speed train image comprises: Acquire an image to be registered, where the size of the image to be registered is H×W×1 pixel; Convolving the image to be registered to obtain a first feature map with a size of H×W×32 pixels, a second feature map with a size of H / 2×W / 2×64 pixels, and a third feature map with a size of H / 4×W / 4×128 pixels; Upsampling and convolving the first feature map, the second feature map, and the third feature map to eliminate aliasing effects, to obtain feature maps of three original image resolutions, where the sizes of the feature maps of the three original image resolutions are: H×W×32 pixels, H / 2×W / 2×64 pixels, and H / 4×W / 4×128 pixels, respectively; Convolve the feature maps of the three original image resolutions to obtain three feature maps of size H×W×1 pixels. Multiply the feature maps of the three sizes by coefficients 0.17, 0.33, and 0.5 in order from shallow to deep, and then accumulate them to obtain a key point detection map of H×W×1. The feature maps of the three original image resolutions are multiplied by coefficients 1, 2, and 3 in order from shallow to deep, and then stacked in the feature dimension to obtain a description feature map of size H×W×224 pixels. Finally, the description feature map is L2 regularized in the channel dimension to construct a descriptor of unit length, which is composed of a 224-dimensional vector located at the coordinates of the key point in the description feature map; Obtaining the descriptors of the key points of the two images to be registered, and performing nearest neighbor matching on the descriptors of the key points of the two images to be registered in the vector space to obtain feature matching results of the two images to be registered; According to the feature matching results of the two images to be registered, the perspective transformation matrix between the two images to be registered is calculated.