An image feature extraction method based on unsupervised deep learning neural networks

Abstract

This invention provides an image feature extraction method based on an unsupervised deep neural network, specifically including the following steps: acquiring images from different viewpoints, adjusting all images to a uniform size, and converting them to grayscale; performing homography transformation on the grayscale images, and reconstructing image pairs with the original images using the newly generated images; finally, forming an image set from all image pairs; randomly selecting one image from each image pair to form image group A, and the other image to form image group B; dividing each image in image group A into blocks according to its aspect ratio; using image group B as the input to the unsupervised deep learning neural network; and backpropagating the calculated error to the unsupervised deep learning neural network. The technical solution of this invention overcomes the problems of slow extraction speed and the need for a large amount of labeled data for network training in existing technologies based on convolutional neural networks.

Description

Technical Field

[0001] This invention relates to the field of visual SLAM technology, specifically to an image feature extraction method based on unsupervised deep neural networks. Background Technology

[0002] With the rapid development of computer science and automation technology, robots are increasingly being applied in various production and daily life scenarios. Visual SLAM can provide localization and navigation services for robots in indoor scenes, and loop closure detection in SLAM can significantly reduce errors and provide more accurate localization and navigation results. However, visual observations of the same location at different times can vary greatly due to factors such as different viewpoints, lighting, weather, and dynamic objects, posing a challenge to loop closure detection. Although many methods are now available that can effectively perform these tasks, such as scene flow algorithms and semantic segmentation algorithms, which can segment dynamic scenes, their processing speed is slow or their robustness is weak, making real-time execution difficult in challenging scenarios.

[0003] Convolutional neural networks (CNNs) are widely used in the field of visual recognition due to their powerful feature learning capabilities. However, existing CNN-based feature extraction methods are slow and require large amounts of labeled data for network training. Therefore, a simple and fast image feature extraction method is needed. Summary of the Invention

[0004] The main objective of this invention is to provide an image feature extraction method based on unsupervised deep neural networks, in order to solve the problems of slow extraction speed and the need for a large amount of labeled data for network training in the existing technology of convolutional neural networks.

[0005] To achieve the above objectives, this invention provides an image feature extraction method based on unsupervised deep neural networks, specifically including the following steps:

[0006] S1, acquire images {X1, X2, X3, ..., X...} from different viewpoints. n Adjust all images to a uniform size, and generate grayscale images {Y1, Y2, Y3, ..., Y} using grayscale conversion. n};

[0007] S2, for the grayscale image {Y1, Y2, Y3, ..., Y...} n The homography transformation is performed on image Y1 to generate a new image Y1', Y2 to generate Y2', and so on, finally generating n image pairs {(Y1, Y1'), (Y2, Y2'), (Y3, Y3'), ..., (Y... n Y n `)};

[0008] S3, randomly select one image from each image pair to form image group A, and the other image to form image group B;

[0009] S4. Divide each image in image group A into blocks according to the aspect ratio, and finally divide it into M blocks. Perform scale-invariant feature transformation on each block to extract the information of a key point in each block. Finally, compile the key point information extracted from the M blocks into an M×128 dimensional vector.

[0010] S5, take image group B as input to unsupervised deep learning neural network, use unsupervised deep learning neural network to extract features from images, generate N×4×14×19 feature map, where N represents the number of input images, and then use three fully connected layers to generate feature vector of M×128 dimensional image.

[0011] S6. The L2 loss function is used to calculate the error of the two M×128 one-dimensional vectors generated for each image in image group A and image group B. The calculated error E is then used to backpropagate the unsupervised deep learning neural network and adjust the neural network parameters layer by layer.

[0012] Furthermore, step S1 specifically includes:

[0013] Environmental images {X1, X2, X3, ..., X...} are acquired from different perspectives of the robot using a monocular camera. n After obtaining the relevant images, formula (1) is used.

[0014]

[0015] Convert all images to grayscale and rename the grayscale images to {Y1, Y2, Y3, ..., Y...}. n}; where Gray(i,j) represents the pixel value at horizontal distance i and vertical distance j after the image is grayscaled, and R(i,j), G(i,j), and B(i,j) represent the pixel values ​​of the (i,j) coordinate in the red, green, and blue channels of the color image, respectively.

[0016] Furthermore, step S2 specifically includes:

[0017] S2.1, convert the original grayscale image {Y1, Y2, Y3, ..., Y...} into a single image. n Four points Pr1(x1, y1), Pr2(x2, y2), Pr3(x3, y3), and Pr4(x4, y4) are randomly selected from the corner bounding boxes of each image in the image and correspond to the four corner points Pc1(0, 0), Pc2(0, h), Pc3(w, 0), and Pc4(w, h) of the new grayscale image, respectively.

[0018] Where, x iy1 (i = 1, 2, 3, 4) are the x and y coordinates of 4 random points, and w and h are the length and width of the original grayscale image, which are also the length and width of the new grayscale image.

[0019] S2.2, Calculate the parameters of the projection transformation matrix in groups.

[0020] The projection transformation matrix H is a 3×3 matrix, but it only has 8 degrees of freedom, so it is generally written as... From matrix equations have to Where x′1 and y′1 are the x and y coordinates of the point (x1, y1) after projection transformation;

[0021] Substituting the coordinates of the four sets of matching points respectively, we have

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] Thus, all parameters can be obtained, and the projection transformation matrix H can be derived.

[0031] S2.3, transform the coordinates of each pixel in the original grayscale image using matrix H to obtain the corresponding coordinates in the new grayscale image, thus obtaining new grayscale images {Y1`, Y2`, Y3`, ..., Y... from different viewpoints. n `};

[0032] S2.4, convert the grayscale images {Y1`, Y2`, Y3`, ..., Y... obtained from different viewpoints in step 2.3 into... n `} and the input grayscale image {Y1, Y2, Y3, ..., Y} n} are put together to form n image pairs {(Y1, Y1`), (Y2, Y2`), (Y3, Y3`), ..., (Y n Y n `)}.

[0033] Further, step S3 specifically involves: for image pairs Y1 and Y1', if Y1 is assigned to image group A, then Y1' is assigned to image group B; for the remaining image pairs Y2 and Y2', Y3 and Y3', ..., Y... n With Y n The same method is used for splitting, and randomness must be ensured in each allocation.

[0034] Furthermore, step S4 specifically includes:

[0035] S4.1 First, divide each image in image group A into M blocks according to the aspect ratio, and then use the SIFT algorithm to extract feature points for each block.

[0036] S4.2 uses the gradient histogram statistical method to obtain the position, scale and orientation information of each feature point, and then uses a set of vectors to represent the information of this feature point. The representation method uses a 4×4×8=128-dimensional vector. Finally, all feature point information extracted from M blocks of each image is statistically combined and represented by an M×128-dimensional vector.

[0037] Furthermore, step S5 specifically includes:

[0038] S5.1, the first convolution operation is performed on the input image, using a 5×5 convolution kernel to perform the convolution operation on the input image, with a stride of 2; then the ReLU activation function is used on the output image after convolution to remove negative values ​​in the image; finally, a 3×3 pooling window is set to perform pooling operation on the input image to achieve dimensionality reduction and preserve image features to the greatest extent.

[0039] S5.2, the output image of S5.1 is convolved again. The specific steps are as follows: the input image is convolved using a 4×4 convolution kernel with a stride of 1; then the ReLU activation function is applied to the convolved output image to remove negative values; finally, a 3×3 pooling window is used to pool the input image.

[0040] S5.3 performs a pure convolution operation on the output image of S5.2, using a 3×3 convolution kernel with a stride of 1, and then uses the ReLU function to remove negative values ​​after convolution.

[0041] S5.4 applies a fully connected layer three times to the output of S5.3 to obtain an M×128 dimensional vector.

[0042] Furthermore, the formula for calculating the error E in step S6 is as follows:

[0043]

[0044] The present invention has the following beneficial effects:

[0045] The method proposed in this invention can extract features that are robust to extreme changes in appearance from the original image without requiring labeled training data or training in a specific environment. This reduces the difficulty of operation and increases efficiency to a certain extent. In terms of network design, the use of a lightweight network greatly improves the running speed. Attached Figure Description

[0046] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0047] Figure 1 A flowchart of an image feature extraction method based on an unsupervised deep neural network according to the present invention is shown. Detailed Implementation

[0048] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] like Figure 1 The image feature extraction method based on unsupervised deep neural networks shown includes the following steps:

[0050] S1, acquire images {X1, X2, X3, ..., X...} from different viewpoints. n Adjust all images to a uniform size, and generate grayscale images {Y1, Y2, Y3, ..., Y} using grayscale conversion. n}

[0051] Specifically, environmental images {X1, X2, X3, ..., X...} are acquired from different perspectives of the robot using a monocular camera. n After obtaining the relevant images, formula (1) is used.

[0052]

[0053] Convert all images to grayscale and rename the grayscale images to {Y1, Y2, Y3, ..., Y...}. n}; where Gray(i,j) represents the pixel value at horizontal distance i and vertical distance j after the image is grayscaled, and R(i,j), G(i,j), and B(i,j) represent the pixel values ​​of the (i,j) coordinate in the red, green, and blue channels of the color image, respectively.

[0054] S2, for the grayscale image {Y1, Y2, Y3, ..., Y...} n The homography transformation is performed on image Y1 to generate a new image Y1', Y2 to generate Y2', and so on, finally generating n image pairs {(Y1, Y1'), (Y2, Y2'), (Y3, Y3'), ..., (Y... n Y n `)};

[0055] Specifically, first, read the grayscale image {Y1, Y2, Y3, ..., Y...} generated in the first step. n From the bounding boxes at the four corners of each image, four points are randomly selected, where the length and width of the bounding boxes are both 1 / 4 of the original grayscale image. Then, a projection transformation matrix is ​​calculated to warp the original image, making these four points the four corner points of the new grayscale image. The original grayscale image {Y1, Y2, Y3, ..., Y...} is then transformed... n In the image, all pixels are distorted by a projection transformation matrix to form a new grayscale image {Y1`, Y2`, Y3`, ..., Y... n Finally, the input and output images are combined to form a new image pair {(Y1, Y1`), (Y2, Y2`), (Y3, Y3`), ..., (Y...}. n Y n `)}.

[0056] The projection transformation matrix is ​​obtained by solving the matrix equation using four sets of matching points. The specific process is as follows:

[0057] S2.1, convert the original grayscale image {Y1, Y2, Y3, ..., Y...} into a single image. n Four points Pr1(x1, y1), Pr2(x2, y2), Pr3(x3, y3), and Pr4(x4, y4) are randomly selected from the corner bounding boxes of each image in the image and correspond to the four corner points Pc1(0, 0), Pc2(0, h), Pc3(w, 0), and Pc4(w, h) of the new grayscale image, respectively.

[0058] Where, x i y i (i = 1, 2, 3, 4) are the x and y coordinates of 4 random points, and w and h are the length and width of the original grayscale image, which are also the length and width of the new grayscale image;

[0059] S2.2, Calculate the parameters of the projection transformation matrix in groups.

[0060] The projection transformation matrix H is a 3×3 matrix, but it only has 8 degrees of freedom, so it is generally written as... From matrix equations have to Where x′ i y′ i For point (x) i y i The x and y coordinates of the corresponding points after projection transformation;

[0061] Substituting the coordinates of the four sets of matching points respectively, we have

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Thus, all parameters can be obtained, and the projection transformation matrix H can be derived.

[0071] S2.3, transform the coordinates of each pixel in the original grayscale image using matrix H to obtain the corresponding coordinates in the new grayscale image, thus obtaining new grayscale images {Y1`, Y2`, Y3`, ..., Y... from different viewpoints. n `};

[0072] S2.4, convert the grayscale images {Y1`, Y2`, Y3`, ..., Y... obtained from different viewpoints in step 2.3 into... n `} and the input grayscale image {Y1, Y2, Y3, ..., Y} n} are put together to form n image pairs {(Y1, Y1`), (Y2, Y2`), (Y3, Y3`), ..., (Y n Y n `)}.

[0073] S3, randomly select one image from each image pair to form image group A, and the other image to form image group B;

[0074] Specifically, to ensure the accuracy of the final result, the images are randomly assigned to two input groups. The specific implementation steps are as follows: For image pairs Y1 and Y1', if Y1 is assigned to image group A, then Y1' is assigned to image group B; for the remaining image pairs Y2 and Y2', Y3 and Y3', ..., Y... n With Y n The same method is used for splitting. It's important to note that randomness must be maintained during each allocation.

[0075] S4. Divide each image in image group A into blocks according to its aspect ratio, resulting in M ​​blocks. Perform scale-invariant feature transformation on each block to extract information from one key point in each image. Finally, compile the key point information extracted from the M blocks into an M×128 dimensional vector. The detailed steps are as follows:

[0076] Specifically, in S4.1, each image in image group A is first divided into M blocks according to its aspect ratio. Then, feature points are extracted from each block using the SIFT algorithm (i.e., scale-invariant feature transformation is performed on each block).

[0077] S4.2 uses the gradient histogram statistical method to obtain the position, scale and orientation information of each feature point, and then uses a set of vectors to represent the information of this feature point. The representation method uses a 4×4×8=128-dimensional vector. Finally, all feature point information extracted from M blocks of each image is statistically combined and represented by an M×128-dimensional vector.

[0078] S5, take image group B as input to unsupervised deep learning neural network, use unsupervised deep learning neural network to extract features from images, generate N×4×14×19 feature map, where N represents the number of input images, and then use three fully connected layers to generate feature vector of M×128 dimensional image.

[0079] Image group B is input into the training network, which consists of three convolutional layers, two pooling layers, and three fully connected layers. The proposed network first uses two convolutional-pooling operations, which effectively extracts feature information while making the network more lightweight. Then, a pure convolutional layer is added to reduce the output data before the fully connected layers. Finally, three consecutive fully connected layers output an M×128-dimensional vector. It is important to note that a corresponding activation function is added after each layer's output. The ReLU activation function is used for the output of the convolutional layers, while the Sigmoid function is chosen for the fully connected layers to better reconstruct the SIFT descriptor. The detailed steps are as follows:

[0080] Specifically, in S5.1, the input image is subjected to a first convolution operation, using a 5×5 convolution kernel to perform convolution operations on the input image with a stride of 2; then, the ReLU activation function is applied to the output image after convolution to remove negative values ​​in the image; finally, a 3×3 pooling window is set to perform pooling operations on the input image to achieve dimensionality reduction and preserve image features to the greatest extent.

[0081] S5.2, the output image of S5.1 is convolved again. The specific steps are as follows: the input image is convolved using a 4×4 convolution kernel with a stride of 1; then the ReLU activation function is applied to the convolved output image to remove negative values; finally, a 3×3 pooling window is used to pool the input image.

[0082] S5.3 performs a pure convolution operation on the output image of S5.2, using a 3×3 convolution kernel with a stride of 1, and then uses the ReLU function to remove negative values ​​after convolution.

[0083] S5.4 applies a fully connected layer three times to the output of S5.3 to obtain an M×128 dimensional vector.

[0084] S6. The L2 loss function is used to calculate the error of the two M×128 one-dimensional vectors generated for each image in image group A and image group B. The calculation formula is as follows:

[0085]

[0086] The calculated error E is then used to backpropagate the unsupervised deep learning neural network, adjusting the network parameters layer by layer.

[0087] This invention provides an image feature extraction method based on unsupervised deep neural networks. This method can be trained using public data in an unsupervised manner and requires only a few parameters. Specifically, during network construction, the original image is transformed into a pair of images from different viewpoints through projection transformation to simulate a robot recognizing the same location from different viewpoints. A portion of the transformed images is used to extract features using the SIFT algorithm, a traditional image processing technique, to maximize the preservation of salient feature information. The remaining images are input into a specially designed convolutional neural network to extract adjusted features. Finally, loss calculations are performed using the features extracted by the SIFT algorithm, and the network parameters are adjusted layer by layer through backpropagation. The designed network extracts feature information using only two simple convolutional and pooling layers; therefore, the constructed network is sufficiently lightweight, exhibits strong robustness in feature extraction under extreme changes in image appearance, and can construct real-time closed loops in resource-constrained systems.

[0088] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. An image feature extraction method based on an unsupervised deep neural network, characterized in that, Specifically comprising the following steps: S1, acquire images from different viewpoints {X1, X2, X3, ..., X...} n Adjust all images to a uniform size, and generate grayscale images {Y1, Y2, Y3, ..., Y} using grayscale conversion. n }; S2, for the grayscale image {Y1, Y2, Y3, ..., Y... n The homography transformation is performed on the image. Four points are randomly selected from the corner bounding boxes of each image in the original grayscale image and sequentially matched with the four corner points of the new grayscale image. Specifically, image Y1 is transformed to generate a new image Y1', Y2 to generate Y2', and so on, finally generating n image pairs {(Y1, Y1'), (Y2, Y2'), (Y3, Y3'), ..., (Y... n Y n `)}; S3, randomly select one image in each image pair to form image group A, and the other image to form image group B; S4, each picture in image group A is divided into M blocks according to the image aspect ratio, and the scale invariant feature transformation is performed on each block picture to extract the information of a key point in each block picture. Finally, the key point information extracted from the M block pictures is counted into an Mx128-dimensional vector. Through the gradient histogram statistical method, the position, scale and direction information of each feature point are obtained, and then a set of vectors is used to express the information of the feature point. The expression method adopts a 4x4x8=128-dimensional vector, and finally all the feature point information extracted from each picture M block is counted together and represented by an Mx128-dimensional vector; S5, image group B is used as the input of the unsupervised deep learning neural network, and the unsupervised deep learning neural network is used to extract features from the image to generate an Nx4x14x19 feature map, wherein N represents the number of input pictures, and then a three-layer fully connected layer is used to generate an Mx128-dimensional feature vector of the image; S6, using L2 loss function to calculate the error of two Mx128 one-dimensional vectors generated by each image in image group A and image group B, and using the calculated error E to perform back propagation on the unsupervised deep learning neural network to adjust the neural network parameters layer by layer. 2.The method of claim 1, wherein, Step S1 specifically is: The environment images of different perspectives of the robot are acquired by a monocular camera {X1, X2, X3, …, Xn} n After the relevant images are acquired, formula (1) is used （1） Convert all images to grayscale and rename the grayscale images to {Y1, Y2, Y3, ..., Y...}. n };in This represents the pixel value at horizontal distance i and vertical distance j after the image has been converted to grayscale. , , Representing the color image The pixel values ​​in the red, green, and blue channels. 3.The method of claim 1, wherein, Step S2 specifically includes: S2.1, randomly selecting 4 points Pr1( n ), Pr2( ), Pr3( ), Pr4( ) in the corner border frame of each image in the original gray image {Y1, Y2, Y3, …, Y }, corresponding to the 4 corners of the new gray image Pc1(0, 0), Pc2(0, ), Pc3( , 0), Pc4( , ) in turn; wherein, , (i = 1, 2, 3, 4) are the horizontal and vertical coordinate values of the 4 random points, and are the length and width of the original gray image, and are also the length and width of the new gray image. S2.2, group calculation of the parameters of the projection transformation matrix; The projection transformation matrix H is a 3x3 matrix, but only has 8 degrees of freedom, so it is generally written as From the matrix equation , we get , where are the horizontal and vertical coordinate values of the corresponding point of the point ( ) after the projection transformation. Substitute the coordinates of the four groups of matching points respectively, and （2） （3） （4） （5） （6） （7） （8） （9） Thus, all parameters are solved, and the projection transformation matrix H is obtained. S2.3, each pixel point coordinate in the original gray image is transformed through a matrix to obtain the coordinates corresponding to the new gray image, thereby obtaining the new gray images of different viewpoints {Y1`, Y2`, Y3`, …, Y n `}; S2.4, the gray scale images of different viewpoints {Y1', Y2', Y3',..., Y n} obtained by step 2.3 are put together with the input gray scale images {Y1, Y2, Y3,..., Y n} to form n image pairs {(Y1, Y1'), (Y2, Y2'), (Y3, Y3'),..., (Y n , Y n ')}. 4.The method of claim 1, wherein, Step S3 is specifically: for the image pair Y1 and Y1', if Y1 is divided into the image group A, then Y1' is divided into the image group B; for the remaining image pairs Y2 and Y2', Y3 and Y3', …, Yn and Yn', the same method is used to divide them, and the randomness is ensured in each time of allocation. n and Y n ' are also divided by the same method, and the randomness is ensured in each time of allocation. 5.The method of claim 1, wherein, Step S4 further includes: Each picture in image group A is divided into M blocks according to the image aspect ratio, and then the SIFT algorithm is used to extract feature points from each block picture. 6.The method of claim 1, wherein, Step S5 specifically includes: S5.1, the first convolution operation is performed on the input image, a 5x5 convolution kernel is used for convolution operation on the input image, and the step length is set to 2; then the Relu activation function is used on the output image after convolution to remove the negative values in the image, and finally a 3x3 pooling window is set for the input image to perform pooling operation, which realizes dimension reduction and maximally retains the image features; S5.2, the output image of S5.1 is convolved again, wherein the specific steps are: a 4x4 convolution kernel is used for convolution operation on the input image, and the step length is set to 1; then the Relu activation function is used on the output image after convolution to remove the negative values in the image, and finally a 3x3 pooling window is set for the input image to perform pooling operation; S5.3, the output image of S5.2 is subjected to pure convolution operation, wherein a 3x3 convolution kernel is used, the step length is set to 1, and the Relu function is used after convolution to remove negative values; S5.4, the output result of S5.3 is continuously used three times by a fully connected layer to obtain an Mx128-dimensional vector. 7.The method of claim 1, wherein, The error E calculation formula in step S6 is: （10）。

Images