Robot multi-modal object recognition method based on greedy deep dictionary learning

Abstract

The present application relates to the technical field of object recognition, and particularly relates to a robot multi-modal object recognition method based on greedy deep dictionary learning, which comprises the following steps: visual modal and tactile modal data acquisition, data feature extraction, visual modal and tactile modal fusion, punishment dictionary greedy deep dictionary learning, feature representation merging and classifier design. The method overcomes the problem of low recognition rate of traditional visual modal in single modal recognition, and solves the problem of insufficient tactile training samples. The object recognition is carried out in a multi-modal fusion mode, the fusion of visual modal and tactile modal can obtain the shape, color, texture and roughness features of the object, and a large training test data set does not need to be created, time and labor are saved, and the accuracy of object recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of object recognition, and particularly relates to a robot multi-modal object recognition method based on greedy deep dictionary learning. BACKGROUND

[0002] The most common method of object recognition is to use a camera to obtain the image of an object, and then perform feature extraction, image classification and image recognition. However, object recognition relying only on vision has great challenges in actual interactive operation. For example, it is difficult to distinguish two bottles that are similar in appearance but different in attributes only by relying on pictures, and in addition, it is difficult to obtain accurate classification and recognition results in the case of occlusion or poor lighting conditions. Haptic information is also often used to identify objects, but object classification and recognition based purely on haptic sensors is unreliable because haptic signals are highly variable. With the development of technology, deep neural network algorithms are used for object recognition, and as the number of network parameters increases during the training process, a large number of parameters need to be learned, and therefore a large number of training samples are required to achieve good classification ability. However, a large data set has the problems of time-consuming and labor-intensive when used.

[0003] Traditional greedy deep dictionary learning uses the deepest feature representation as the input of the classifier, at this time only complex abstract representation can be applied to the classifier, and the shallow information cannot be saved, which affects the accuracy of object recognition. In addition, for the dictionary learned by the traditional greedy deep dictionary framework, sometimes invalid solutions are generated, and when the difference between the feature representations of different categories is small, the discrimination ability of the dictionary will be affected. The greedy deep dictionary learning framework often replaces the convolutional layer of the neural network, and relies on the fully connected layer of the neural network for classification, at this time not only a large number of training samples are required, but also the number of floating point operations per second increases, and real-time applications face great challenges. SUMMARY

[0004] In order to overcome the defects of the prior art, the present application provides a robot multi-modal object recognition method based on greedy deep dictionary learning, which overcomes the problem of low recognition rate of traditional visual modal in single modal recognition, solves the problem of insufficient haptic training samples, and does not need to create a large training test data set, saving time and effort.

[0005] The object recognition is performed in a multi-modal fusion manner, the fusion of the visual modal and the tactile modal can obtain the shape, color, texture and roughness features of the object, and the accuracy of recognizing the object is improved. The greedy deep dictionary learning algorithm with a penalty dictionary is used for training, a dictionary penalty term is added to the traditional greedy deep dictionary model, invalid solutions are avoided, and the difference between the feature representations of different objects is enhanced. The multi-layer dictionary learning framework is used instead of the deep neural network, and the limitation of the deep neural network requiring a large number of training samples is overcome. The shallow layer dictionary learning of the greedy deep dictionary learning with a penalty dictionary is used for edge, angle and line feature representation, and the deep layer dictionary learning is used for complex abstract feature representation. The first layer feature representation Z1 and the sparse representation Z L The obtained feature representation is combined as the input of the classifier, the information is saved across layers, and the accuracy of recognizing the object is improved.

[0006] To achieve the purpose, the technical scheme is adopted in the present application.

[0007] The object of the present application is to provide a robot multi-modal object recognition method based on greedy deep dictionary learning, mainly including the following steps:

[0008] S1, visual modal and tactile modal data acquisition

[0009] (1a) the data acquisition of the visual modal is completed by using a camera;

[0010] (1b) the data acquisition of the tactile modal is completed by using a dexterous hand equipped with a tactile sensor mounted on the end of a robot arm;

[0011] (1c) when the data acquisition work is performed, the robot arm uses a camera to acquire an object image, and uses a finger to touch it, so as to obtain an image sample and a tactile sample of the object to be recognized;

[0012] S2, data feature extraction;

[0013] (2a) the feature extraction of the image data adopts a scale invariant feature transform algorithm, which has scale invariance and can detect key points in the image;

[0014] (2b) after the column vector is obtained by pre-processing the tactile data, the principal component analysis algorithm is used to obtain the tactile sequence features, so as to reduce the redundant part in the tactile sequence and reduce the time cost of the overall classification algorithm;

[0015] S3, fusion of visual modal and tactile modal

[0016] The visual feature matrix and the tactile sequence feature matrix are horizontally combined and fused, the data set is obtained after fusion, the training set X and the test set X testThe shape, color, texture and roughness features of the object can be obtained, and the accuracy of identifying the object is improved.

[0017] S4, greedy depth dictionary learning of penalty dictionary

[0018] The greedy depth dictionary learning of penalty dictionary is to train each layer of the multi-layer dictionary framework, and each layer is optimized by iterative training to obtain the feature representation Z = {Z1, Z2,..., Z L} and the dictionary layer Ω = {D1, D2,..., D L} of the training set data, and the data x test of the input test sample n, and the all feature representations z L of the test sample n are obtained according to the dictionary layer Ω = {D1, D2,..., D test} solved by training. 1,test 2,test L,test

[0019] S5, feature representation merging

[0020] The feature representation merging is to merge the edge, angle and line feature representations of the shallow layer and the complex abstract feature representations of the deep layer to save information across layers and improve the accuracy of identifying the object, and the first layer feature representation Z1 of the training set data and the sparse representation Z L are merged to obtain Z S , and the first layer feature representation z 1,test of the test sample n and the sparse representation z L,test of the deepest layer are merged to obtain z S,test , and the representation is as follows:

[0021]

[0022]

[0023] S6, classifier design

[0024] The classifier is designed to obtain the label of the test sample for object recognition, and the label of the test sample is defined as follows:

[0025]

[0026] Where, there are M objects to be identified, the label number is M, m is the label of a certain test sample input, is the sub-vector of the corresponding label in Z S , Z S is the vector of the merged shallow and deep feature representations of the training set data, and z S,test is the vector of the merged shallow and deep feature representations of a certain test sample. ​​​

[0027] Preferably, in S4, the greedy depth dictionary learning of the penalty dictionary adds a dictionary penalty term to avoid generating invalid solutions, and enhances the distinction of feature representations of different objects, and an alternating least squares method is used for solving;

[0028] The greedy depth dictionary model training process of the penalty dictionary can be expressed as:

[0029] An input training set X, a dictionary coefficient λ1, a sparse coefficient λ2, and an initialized dictionary layer Ω={D1, D2, …, D L} are input, and there are L layers. The first layer of the greedy depth dictionary model of the penalty dictionary is as follows:

[0030]

[0031]

[0032] Where I is an identity matrix, and the process is repeated until convergence. Once the first layer converges, the next layer is trained.

[0033] The second to L-1 layers of the greedy depth dictionary learning model of the penalty dictionary are as follows, where l represents the layer number, and φ is an activation function, and Tanh is selected, and the mathematical definition is as follows: The formula is as follows:

[0034]

[0035]

[0036] The process is repeated until convergence. Once the lth layer converges, the next layer is trained.

[0037] The Lth layer of the greedy depth dictionary learning model of the penalty dictionary is as follows:

[0038]

[0039]

[0040] The process is repeated until convergence, and the training is completed.

[0041] Compared with the prior art, the present application has the following advantages:

[0042] (1) The present application provides a robot multi-modal object recognition method based on greedy depth dictionary learning, which adopts a multi-modal fusion method for object recognition. The fusion of visual modalities and tactile modalities can obtain shape, color, texture and roughness features of the object, overcoming the low recognition rate problem of traditional visual modalities in single-modal recognition, and solving the problem of insufficient tactile training samples.

[0043] (2) The application adopts a greedy deep dictionary learning algorithm for training, uses a multi-layer dictionary learning framework instead of a deep neural network, overcomes the limitation of the deep neural network requiring a large number of training samples, and does not need to create a large training test data set, saving time and effort. A dictionary penalty term is added to the traditional greedy deep dictionary learning model to avoid generating invalid solutions and enhance the difference between the feature representations of different objects.

[0044] (3) The application uses the combined feature representation as the input of the classifier, the shallow dictionary learning shallow edge, angle, line and other feature representations, and the deep dictionary learning complex abstract feature representation. The first layer feature representation Z1 and the sparse representation Z L The combined feature representation is used as the input of the classifier to save information across layers and improve the accuracy of identifying objects. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a specific flowchart of a robot multi-modal object recognition method based on greedy deep dictionary learning of the application;

[0046] Figure 2 is a dictionary-penalized greedy deep dictionary learning algorithm model provided by the application. DETAILED DESCRIPTION

[0047] The technical solutions of the application will be further described below in combination with the drawings and through specific embodiments.

[0048] Figure 1 is a flowchart of a robot multi-modal object recognition method based on greedy deep dictionary learning of the application, as shown in Figure 1 The method comprises:

[0049] Example 1: Visual modal and tactile modal data acquisition

[0050] The visual modal data acquisition is completed by a camera. In a stable environment, each object is placed at the approximate center of the table, and the image of the object is obtained by the camera.

[0051] The tactile modal data acquisition is completed by a dexterous hand equipped with a tactile sensor mounted on the end of a robot arm. During the tactile modal data collection process, each object is randomly placed on the table to ensure the variability of the tactile data.

[0052] Example 2: Data feature extraction

[0053] The scale-invariant feature transform algorithm is used for feature extraction of image data, which has scale invariance and can detect key points in the image. The tactile sequence is preprocessed, and the tactile data collected by a single finger sensor unit is taken as an example. The 3x8 matrix dimension is converted into a column vector, and the column matrix with 24 element dimensions is used to represent the tactile information collected by a single tactile sensor in a unit time.

[0054] After preprocessing, the principal component analysis algorithm is used for tactile feature extraction, which can reduce the redundant part of the tactile sequence and reduce the time cost of the overall classification algorithm while ensuring the lowest loss of principal components.

[0055] Embodiment 3: Fusion of visual and tactile modalities

[0056] The visual feature matrix and the tactile sequence feature matrix are merged horizontally for fusion, and the data set obtained after fusion is segmented to obtain the training set X and the test set X test The shape, color, texture, and roughness features of the object can be obtained, and the accuracy of object recognition can be improved.

[0057] Embodiment 4: Greedy deep dictionary learning with dictionary penalty

[0058] The greedy deep dictionary learning with dictionary penalty is a hierarchical training of dictionary learning, each layer is optimized by iterative training, and the alternating least squares method is used for optimization. Adding a dictionary penalty term avoids generating invalid solutions and enhances the discrimination ability of feature representation of different objects. In the optimization process of each layer, the feature representation Z is optimized first, then the dictionary D is optimized, and this process is repeated until convergence. Once the first layer converges, the same iterative training is started for the next layer, and it is repeated in all layers.

[0059] The training set X is input, the dictionary coefficient λ1 and the sparse coefficient λ2 are input, the dictionary layer Ω is initialized as {D1, D2, …, D L}, a total of L layers. D is the dictionary, and Z is the feature representation of data X in D. The greedy dictionary learning model uses L2 norm, which has a closed-form solution and is easy to minimize. The first layer optimal solution of the greedy deep dictionary learning model with dictionary penalty is as follows:

[0060]

[0061] This process is repeated until convergence. Once the first layer converges, the next layer is trained.

[0062] The second to L-1 layers, the number of layers is represented by l, and φ is the activation function. The l-th layer optimal solution is as follows:

[0063]

[0064] Repeat this process until convergence, once the first layer converges, start training the next layer.

[0065] Lth layer, feature representation Z L Sparse, in order to learn sparse features, need to feature L1 norm regularization, the optimal solution as follows:

[0066]

[0067] Repeat this process until convergence, training is complete.

[0068] Alternating least squares method for optimization, as shown in Figure 2 , Figure 2 The application provides a kind of algorithm model of greedy depth dictionary learning of penalty dictionary, specific model as follows:

[0069] A1, input layer.Input training set X, input dictionary coefficient λ1, sparse coefficient λ2, initialize dictionary layer Ω={D1, D2, …, D L}There are L layers.

[0070] A2, the first layer.First update feature representation Z1, its formula as follows:

[0071]

[0072] Then update dictionary D1, its formula as follows:

[0073]

[0074] Wherein, I is unit matrix, repeat this process until convergence, once the first layer converges, start training the next layer.

[0075] A3, the second layer ~ L-1 layer, layer number is represented by l, φ is activation function, selects Tanh, mathematical definition is The first layer updates feature representation Z l , its formula as follows:

[0076]

[0077] Then update dictionary D l , its formula as follows:

[0078]

[0079] Repeat this process until convergence, once the first layer converges, start training the next layer.

[0080] A4, the Lth layer, first update feature representation Z LThe formula is as follows:

[0081]

[0082] The dictionary D is then updated, and the formula is as follows: L

[0083]

[0084] This process is repeated until convergence, and the training is complete.

[0085] For the test sample n, the input test sample x test The calculation formula of the feature representation z 1,test of the first layer of the test sample n is as follows:

[0086]

[0087] The calculation formula of the feature representation z l,test of the second to L-1 layer of the test sample n is as follows, where l represents the layer number:

[0088]

[0089] The calculation formula of the feature representation z L,test of the L layer of the test sample n is as follows:

[0090]

[0091] Example 5: Feature representation merging

[0092] The first layer feature representation Z1 of the training set data is merged with the sparse representation Z L of the deepest layer to obtain Z S The first layer feature representation z 1,test of the test sample n is merged with the sparse representation z L,test of the deepest layer to obtain z S,test , which is represented as follows:

[0093]

[0094]

[0095] The shallow layer edge, angle, line feature representation is merged with the deep layer complex abstract feature representation to save information across layers and improve the accuracy of object recognition.

[0096] Example 6: Classifier design

[0097] The classifier is designed to obtain the label of the test sample for object recognition, and the label of the test sample is defined as follows:

[0098] ​

[0099] wherein, M is the total number of items to be identified, M is the label number, m is the label of a certain test sample input, is Z S the sub-vector of the corresponding label, Z S is the vector of the combination of the shallow and deep feature representations of the training set data, z S,test is the vector of the combination of the shallow and deep feature representations of a certain test sample.

[0100] The above has described the specific embodiments of the present application in detail, but it is only as an example, the present application is not limited to the above described specific embodiments. For those skilled in the art, without departing from the spirit and scope of the present application, various changes or equivalent replacements can be made to these features and embodiments. The present application is not limited by the specific embodiments disclosed herein, other embodiments falling within the scope of the claims of the present application are within the scope of the present application.

Claims

1. A method for robot multimodal object recognition based on greedy deep dictionary learning, characterized in that, The main steps include: S1, Visual and tactile modal data acquisition (1a) Visual modality data acquisition was accomplished using a camera; (1b) Data acquisition of tactile modalities is accomplished by using a dexterous hand equipped with tactile sensors, which is mounted on the end of a robotic arm; (1c) When performing data acquisition, the robotic arm uses a camera to acquire images of objects and touches them with its fingers to obtain image samples and tactile samples of the objects to be identified. S2, Data Feature Extraction; (2a) The scale-invariant feature transformation algorithm is used to extract features from image data. This algorithm has scale invariance and can detect key points in the image. (2b) After preprocessing the tactile data to obtain column vectors, principal component analysis algorithm is used to obtain tactile sequence features, reducing redundant parts in the tactile sequence and reducing the time cost of the overall classification algorithm. S3, fusion of visual and tactile modalities The visual feature matrix and the tactile sequence feature matrix are horizontally merged and fused to obtain the dataset. The dataset is then split into training set X and test set X. test It obtains the shape, color, texture, and roughness features of objects, thereby improving the accuracy of object recognition; S4, Greedy Deep Dictionary Learning with Penalized Dictionaries Greedy deep dictionary learning with penalty dictionaries trains a multi-layered dictionary framework layer by layer, optimizing each layer through iterative training to obtain the feature representation Z={Z1, Z2, ..., Zn} of the training set data. L } and dictionary layer Ω={D1, D2, ..., D L }, with a total of L layers, and input test sample n data x test Based on the dictionary layer Ω={D1, D2, ..., D...} obtained through training, L } Obtain all feature representations z of the test sample n. S,test ={z 1,test , z 2,test , ..., z L,test }; S5, Feature Representation Merging Feature representation merging combines shallow edge, corner, and line feature representations with deep, complex, and abstract feature representations to preserve information across layers and improve the accuracy of object recognition. This involves merging the first-layer feature representation Z1 of the training set data with the deepest sparse representation Z... L The result of merging is Z S The first layer feature representation z of the test sample n 1,test With the deepest sparse representation z L,test The result of merging is z S,test , means as follows: S6, Classifier Design Design a classifier to obtain labels for test samples for object recognition. The labels for the test samples are defined as follows: There are M items to be identified, with M tags, and m is the tag of a certain test sample. It is Z S The subvector of the corresponding label in the vector, Z S z is a vector formed by merging the shallow and deep feature representations of the training set data. S,test A vector representing the combined shallow and deep feature representations of a given test sample.

2. The robot multimodal object recognition method based on greedy deep dictionary learning as described in claim 1, characterized in that: In S4, the greedy deep dictionary learning of the penalty dictionary adds dictionary penalty terms to avoid generating invalid solutions, while enhancing the distinction of feature representations of different objects, and uses the alternating least squares method for solving; The training process of a greedy deep dictionary model for penalty dictionaries can be represented as: Input training set X, dictionary coefficients λ1, sparse coefficients λ2, initialize dictionary layer Ω={D1, D2, ..., D L The model consists of L layers, and is the first layer of a greedy depth-first dictionary model with a penalty dictionary. The formula is as follows: Where I is the identity matrix, this process is repeated until convergence, and once the first layer converges, the training of the next layer begins; The greedy deep dictionary learning model with penalty dictionary has layers 2 to L-1, where the layer number is denoted by l. As the activation function, we choose Tanh, which is mathematically defined as follows: The formula is as follows: Repeat this process until convergence. Once the l-th layer converges, start training the next layer. The formula for the Lth layer of the greedy deep dictionary learning model with penalty dictionary is as follows: Repeat this process until convergence, at which point training is complete.

Images