A 6D recognition method and system with adaptive feature selection

Abstract

The present invention discloses a 6D recognition method and system for adaptive feature selection, including extracting features of the point cloud and image of the target object; combining the extracted features to form a pose estimation feature vector; encoding and decoding the features of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector; inputting the new pose estimation feature vector into a pre-established network model, and obtaining the pose recognition information of the target object through training. The above scheme can fully utilize the data features of the pose estimation of the target object based on the image and point cloud features, thereby reducing the number of training times, increasing the estimation accuracy, and reducing the estimation operation time.

Description

Technical Field

[0001] The present invention relates to a target recognition and position posture estimation technology based on deep learning in image processing, and in particular to a 6D recognition method and system for adaptive feature selection. Background Art

[0002] At present, the recognition and position and posture estimation of target objects based on deep learning mainly use point cloud and image data. The features extracted from the two types of data, point cloud and image matching, are all applied to the recognition and position and posture estimation of target objects, without selective application based on the richness of image texture and the number of point cloud features of target objects. In practical applications, the richness of image features and point cloud features varies in different scenes, different target objects, and different parts of target objects. Therefore, the importance of image and point cloud features will be different in the recognition and position and posture estimation of target objects, so the application ratio of image and point cloud features needs to be changed. Therefore, there is a need for an adaptive selection mechanism for point cloud and image features. Summary of the invention

[0003] In order to achieve the above-mentioned purpose, the present invention discloses a 6D recognition method and system with adaptive feature selection, and proposes an adaptive learning framework for the selection of point cloud features and image features in 6D recognition. In the process of 6D recognition, on the basis of applying deep learning methods to extract the features of point clouds and images, the extracted features are analyzed to determine the strength of point cloud features and image features. According to the strength of point cloud features and image features, a suitable feature extraction method is selected to realize the recognition of the target object.

[0004] The present invention solves the above technical problems by the following technical means:

[0005] A 6D recognition method for adaptive feature selection, the method comprising the following steps:

[0006] Extract the features of the point cloud and image of the target object;

[0007] The extracted features are combined to form a pose estimation feature vector;

[0008] Encoding and decoding the features of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector;

[0009] The new pose estimation feature vector is input into a pre-established network model, and pose recognition information of the target object is obtained through training.

[0010] Preferably, the extracting of the features of the point cloud and the image of the target object comprises: obtaining N pixel points (x1, y1), (x2, y2), ..., (x n ,y n ), surface texture features of N∈1,2,…,n;

[0011] Map the point cloud of the target scene to the image, apply the PointNet deep learning network, and obtain the points corresponding to N pixels (x1, y1), (x2, y2), ..., (x n ,y n )’s spatial characteristics.

[0012] Preferably, encoding and decoding the features of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector comprises:

[0013] The total dimension of the pose estimation feature vector formed by the extracted feature combination is set to p, where the feature of each pixel is (v1, v2, ..., v p );

[0014] The feature vector of each pixel (v1, v2, ..., v p ) Use restricted Boltzmann machine for training encoding and decoding to obtain a new pose estimation feature vector.

[0015] Furthermore, the feature vector of each pixel (v1, v2, ..., v p ) The restricted Boltzmann machine is used for training encoding and decoding to obtain a new pose estimation feature vector including:

[0016] Through training, the vector (v1, v2, ..., v p ) is encoded to obtain the encoding vector (v'1,v'2,......,v' q );

[0017] Decode the q-dimensional encoded vector into an r-dimensional feature vector.

[0018] Furthermore, the vector (v1, v2, ..., v p ) is encoded to obtain the encoding vector (v'1,v'2,......,v' q )include:

[0019] Assume that the input matrix of N pixels is N×p, that is: [V1, V2, ..., V p ];

[0020] Then the transformation matrix is:

[0021]

[0022] Through matrix transformation, the new eigenvector becomes [V1,V2,......,V p ]×A pq To obtain the q-dimensional encoding vector [V'1,V'2,......,V' q ].

[0023] Furthermore, the q-dimensional vector is decoded into an r-dimensional feature vector by the following transformation matrix:

[0024]

[0025] [S'1,S'2,......,S' r ]=[V'1,V'2,......,V' q ]×A qr ;

[0026] A pq ×A qr A feature transformation matrix is ​​obtained, and an r-dimensional local weight feature is obtained; then the features of N pixels are adaptively selected to obtain an N r-dimensional feature vector (v1, v2, ..., v r ).

[0027] Preferably, the step of inputting the new pose estimation feature vector into a pre-established network model to obtain the pose recognition information of the target object through training comprises:

[0028] The position and posture features of N pixels are processed by adapting the selected data, and the position and posture of the target object are trained through CNN;

[0029] After training, the position and posture estimation of the target object is obtained through image and point cloud data.

[0030] Furthermore, the pre-established network model includes a training loss function, which is expressed as follows:

[0031]

[0032] in: is the loss function of position and posture estimation, x j is the jth point selected from the target object; p = [R|t] is the true value, is the i-th 3D position point of the predicted position and attitude.

[0033] A 6D recognition system with adaptive feature selection, the system comprising:

[0034] A feature extraction module is used to extract features of the point cloud and image of the target object;

[0035] A combination module, used to combine the extracted features to form a pose estimation feature vector;

[0036] A processing module, used for encoding and decoding the features of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector;

[0037] The recognition module is used to input the new posture estimation feature vector into a pre-established network model to obtain the posture recognition information of the target object through training.

[0038] The beneficial effects of the present invention are embodied in:

[0039] The present invention proposes a 6D recognition method and system with adaptive feature selection, and proposes an adaptive learning framework for the selection of point cloud features and image features in 6D recognition. In the process of 6D recognition, on the basis of applying deep learning methods to extract the features of point clouds and images, the extracted features are analyzed to determine the strength of point cloud features and image features. According to the strength of point cloud features and image features, a suitable feature extraction method is selected to realize the recognition of the target object.

[0040] The present invention proposes a 6D recognition method and system for adaptive feature selection, which can fully utilize the data features of target object pose estimation based on image and point cloud features, thereby reducing the number of training times, increasing estimation accuracy, and reducing the estimation operation time. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0042] Figure 1 The overall flow chart of the method provided by the present invention;

[0043] Figure 2 A network diagram for encoding and decoding feature vectors in the feature selection stage provided in Embodiment 1 of the present invention;

[0044] Figure 3 A schematic diagram of a training process for encoding and decoding feature vectors in a feature selection phase provided in Embodiment 1 of the present invention;

[0045] Figure 4Schematic diagram of target object position and posture estimation based on image and point cloud features provided in Example 1 of the present invention.

[0046] Figure 5 This is a flow chart of position and posture estimation without adding feature selection provided in Example 1 of the present invention.

[0047] Figure 6 Provided for Example 1 of the present invention is a flow chart for implementing feature selection.

[0048] Figure 7 Provided for Example 1 of the present invention is a flow chart of position and posture estimation with added feature selection. DETAILED DESCRIPTION

[0049] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0050] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0051] like Figure 1 As shown, a 6D recognition method for adaptive feature selection is provided in a specific embodiment of the present invention, and the method comprises the following steps:

[0052] S1 extracts the features of the point cloud and image of the target object;

[0053] S2 combines the extracted features to form a pose estimation feature vector;

[0054] S3 encodes and decodes the features of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector;

[0055] S4 inputs the new posture estimation feature vector into a pre-established network model, and obtains the posture recognition information of the target object through training.

[0056] In step S1, the feature extraction of the point cloud and image of the target object includes: obtaining N pixel points (x1, y1), (x2, y2), ..., (x n ,y n ), surface texture features of N∈1,2,…,n;

[0057] Map the point cloud of the target scene to the image, apply the PointNet deep learning network, and obtain the points corresponding to N pixels (x1, y1), (x2, y2), ..., (xn ,y n )’s spatial characteristics.

[0058] In step S3, encoding and decoding the features of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector includes:

[0059] The total dimension of the pose estimation feature vector formed by the extracted feature combination is set to p, where the feature of each pixel is (v1, v2, ..., v p );

[0060] The feature vector of each pixel (v1, v2, ..., v p ) Use restricted Boltzmann machine for training encoding and decoding to obtain a new pose estimation feature vector.

[0061] Furthermore, the feature vector of each pixel (v1, v2, ..., v p ) The restricted Boltzmann machine is used for training encoding and decoding to obtain a new pose estimation feature vector including:

[0062] Through training, the vector (v1, v2, ..., v p ) is encoded to obtain the encoding vector (v'1,v'2,......,v' q );

[0063] Decode the q-dimensional encoded vector into an r-dimensional feature vector.

[0064] Furthermore, the vector (v1, v2, ..., v p ) is encoded to obtain the encoding vector (v'1,v'2,......,v' q )include:

[0065] Assume that the input matrix of N pixels is N×p, that is: [V1, V2, ..., V p ];

[0066] Then the transformation matrix is:

[0067]

[0068] Through matrix transformation, the new eigenvector becomes [V1,V2,......,V p ]×A pq To obtain the q-dimensional encoding vector [V'1,V'2,......,V' q ].

[0069] Furthermore, the q-dimensional vector is decoded into an r-dimensional feature vector by the following transformation matrix:

[0070]

[0071] [S'1,S'2,......,S' r ]=[V'1,V'2,......,V' q ]×A qr ;

[0072] A pq ×A qr A feature transformation matrix is ​​obtained, and an r-dimensional local weight feature is obtained; then the features of N pixels are adaptively selected to obtain an N r-dimensional feature vector (v1, v2, ..., v r ).

[0073] In step S4, the new pose estimation feature vector is input into the pre-established network model, and the pose recognition information of the target object obtained through training includes:

[0074] The position and posture features of N pixels are processed by adapting the selected data, and the position and posture of the target object are trained through CNN;

[0075] After training, the position and posture estimation of the target object is obtained through image and point cloud data.

[0076] Furthermore, the pre-established network model includes a training loss function, which is expressed as follows:

[0077]

[0078] in: is the loss function of position and posture estimation, x j is the jth point selected from the target object; p = [R|t] is the true value, is the i-th 3D position point of the predicted position and attitude.

[0079] Embodiment 1:

[0080] Based on the above inventive concept, the present invention provides an adaptive target object recognition and pose estimation method, such as Figure 4 and 7 As shown, the following steps are included:

[0081] ① Using RGB images, through the CNN deep learning network, we can obtain the surface texture features of N pixels in the image (x1, y1), (x2, y2), ..., (x n ,y n );

[0082] Therefore, the surface texture features of the target object are obtained: {(x1,y1),(x2,y2),......,(x n ,y n )}.

[0083] ② Map the point cloud of the target scene to the image and apply the deep learning network method of PointNet to obtain the points corresponding to (x1,y1),(x2,y2),......,(x n ,y n ) points to obtain the spatial features of the target object.

[0084] A. Feature selection stage:

[0085] For each 3D position point, the total dimension of the data input is p, half of which is the surface texture features of the target object (v1, v2, ..., v p / 2 The other half of the features are So the feature of a pixel is (v1,v2,......,v p ).like Figure 5 and 6 The training diagram is shown in the figure, where the network structure includes hidden layers and visible layers; (the visible layer is the original vector v, and the hidden layer is the implemented encoding h) each feature vector (v1, v2, ..., v p )Use restricted Boltzmann machines for training encoding and decoding.

[0086] Its loss function is: θ={W,a,b} is the parameter of the model; through training, the vector (v1,v2,......,v p ) to get the encoding vector (v'1,v'2,......,v' q ).

[0087] The input matrix of N pixels is N×p, that is: [V1, V2, ..., V p ], and get the transformation matrix:

[0088]

[0089] The new eigenvector becomes [V1,V2,......,V p ]×A pq , thus obtaining a q-dimensional vector [V'1,V'2,......,V' q ].

[0090] Similarly, the q-dimensional vector can be decoded into an r-dimensional feature vector, and its transformation matrix is:

[0091]

[0092] [S'1,S'2,......,S' r ]=[V'1,V'2,......,V' q ]×A qr ;

[0093] A pq ×A qr Obtaining a feature transformation matrix will result in an r-dimensional local weight feature.

[0094] In this way, the features of N pixels are adaptively selected to obtain N r-dimensional feature vectors (v1, v2, ..., v r ), thus realizing feature selection.

[0095] B.6D Pose Estimation Stage

[0096] The position and posture features of N pixels are processed by adaptive selection and CNN training to predict the position and posture of the target object. The loss function of the training is:

[0097]

[0098] is the loss function of position and posture estimation, where: x j is the jth point selected from the target object; p = [R|t] is the true value, is the i-th 3D position point of the predicted position and attitude.

[0099] After training is completed, the position and posture estimation of the target object can be achieved through image and point cloud data.

[0100] Embodiment 2:

[0101] Based on the same technical concept, the present invention also provides a 6D recognition system with adaptive feature selection, the system comprising:

[0102] A feature extraction module is used to extract features of the point cloud and image of the target object;

[0103] A combination module, used to combine the extracted features to form a pose estimation feature vector;

[0104] A processing module, used for encoding and decoding the features of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector;

[0105] The recognition module is used to input the new posture estimation feature vector into a pre-established network model to obtain the posture recognition information of the target object through training.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0107] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than to limit its protection scope. Although the present application has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present application, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, and these changes, modifications or equivalent substitutions are all within the scope of the claims to be approved by the application.

Claims

1. A 6D recognition method with adaptive feature selection, characterized in that: The method comprises the following steps: Extract the features of the point cloud and image of the target object; The extracted features are combined to form a pose estimation feature vector; Encoding and decoding the features of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector; Inputting the new pose estimation feature vector into a pre-established network model, and obtaining the pose recognition information of the target object through training; The extracting of features of the point cloud and image of the target object includes: Through the CNN deep learning network, we can obtain N pixel points (x1, y1), (x2, y2), ..., (x n ,y n ), surface texture features of N∈1,2,…,n; Map the point cloud of the target scene to the image, apply the PointNet deep learning network, and obtain the points corresponding to N pixels (x1, y1), (x2, y2), ..., (x n ,y n ), and obtaining the spatial features of the target object; The encoding and decoding of the feature of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector comprises: The total dimension of the pose estimation feature vector formed by the extracted feature combination is set to p, half of which is the surface texture feature of the target object, where the feature of each pixel is (v1, v2, ..., v p ); The feature vector of each pixel (v1, v2, ..., v p ) The restricted Boltzmann machine is used for training encoding and decoding to obtain a new pose estimation feature vector including: Through training, the vector (v1, v2, ..., v p ) is encoded to obtain the encoding vector (v'1,v'2,......,v' q ); Decode the q-dimensional encoding vector into an r-dimensional feature vector; The training vector (v1, v2, ..., v p ) is encoded to obtain the encoding vector (v'1,v'2,......,v' q )include: Assume that the input matrix of N pixels is N×p, that is: [V1, V2, ..., V p ]; Then the transformation matrix is: Through matrix transformation, the new eigenvector becomes [V1, V2, ..., V p ]×A pq To obtain the q-dimensional encoding vector [V'1,V'2,......,V' q ].

2. The method according to claim 1, characterized in that The q-dimensional vector is decoded into an r-dimensional feature vector by the following transformation matrix: [S'1,S'2,......,S' r ]=[V1',V'2,......,V' q ]×A qr ; A pq ×A qr A feature transformation matrix is ​​obtained, and an r-dimensional local weight feature is obtained; then the features of N pixels are adaptively selected to obtain an N r-dimensional feature vector (v1, v2, ..., v r ).

3. The method according to claim 1, characterized in that The new pose estimation feature vector is input into a pre-established network model, and the pose recognition information of the target object obtained through training includes: The position and posture features of N pixels are processed by adapting the selected data, and the position and posture of the target object are trained through CNN; After training, the position and posture estimation of the target object is obtained through image and point cloud data.

4. The method according to claim 3, characterized in that The pre-established network model includes a training loss function, which is expressed as follows: in: is the loss function of position and posture estimation, x j is the jth point selected from the target object; p = [R|t] is the true value, is the i-th 3D position point of the predicted position and attitude.

5. A 6D recognition system with adaptive feature selection, characterized in that: The system comprises: A feature extraction module is used to extract features of the point cloud and image of the target object; A combination module, used to combine the extracted features to form a pose estimation feature vector; A processing module, used for encoding and decoding the features of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector; A recognition module, used for inputting the new pose estimation feature vector into a pre-established network model to obtain the pose recognition information of the target object through training; The extracting of features of the point cloud and image of the target object includes: Through the CNN deep learning network, we can obtain N pixel points (x1, y1), (x2, y2), ..., (x n ,y n ), surface texture features of N∈1,2,…,n; Map the point cloud of the target scene to the image, apply the PointNet deep learning network, and obtain the points corresponding to N pixels (x1, y1), (x2, y2), ..., (x n ,y n ), and obtaining the spatial features of the target object; The encoding and decoding of the feature of each pixel in the pose estimation feature vector to obtain a new pose estimation feature vector comprises: The total dimension of the pose estimation feature vector formed by the extracted feature combination is set to p, half of which is the surface texture feature of the target object, where the feature of each pixel is (v1, v2, ..., v p ); The feature vector of each pixel (v1, v2, ..., v p ) The restricted Boltzmann machine is used for training encoding and decoding to obtain a new pose estimation feature vector including: Through training, the vector (v1, v2, ..., v p ) is encoded to obtain the encoding vector (v'1,v'2,......,v' q ); Decode the q-dimensional encoding vector into an r-dimensional feature vector; The training vector (v1, v2, ..., v p ) is encoded to obtain the encoding vector (v'1,v'2,......,v' q )include: Assume that the input matrix of N pixels is N×p, that is: [V1, V2, ..., V p ]; Then the transformation matrix is: Through matrix transformation, the new eigenvector becomes [V1,V2,......,V p ]×A pq To obtain the q-dimensional encoding vector [V1',V'2,......,V' q ].

Images