RGB-d image feature collaborative fusion method based on transfer learning and width learning

Abstract

The application provides an RGB-D image feature collaborative fusion method based on transfer learning and width learning, and comprises the following steps: obtaining an RGB-D data set, performing preliminary training through a neural network, and performing retraining in the data set after modifying the structure; after feature extraction, performing correlation analysis and fusion on RGB image features and depth image features; and using width learning to classify and identify the fused features. The application can reasonably fuse the features of RGB images and depth images, ensure that the feature information of color images and depth images can complement each other, improve the running speed of the system by using width learning, and finally make the classification result have higher accuracy and reliability.

Description

Technical Field

[0001] This invention relates to computer vision and image processing technology, and more particularly to a collaborative fusion method for RGB-D image features based on transfer learning and width learning. Background Technology

[0002] Computer vision refers to machine vision that uses cameras and computers to identify, track, and measure targets, replacing the human eye. It further performs image processing for target classification, pose determination, and dimensional measurement. With continuous technological advancements, computer vision research has made groundbreaking progress. Image classification is the most fundamental and important branch of computer vision, significantly contributing to the development of multimedia retrieval technology. In recent years, numerous studies have designed deep convolutional neural networks for RGB image recognition, achieving significant progress and gaining widespread application in various industries such as traffic monitoring, intelligent security, intelligent robotics, and automated assembly of parts.

[0003] However, in practical applications, due to the inherent limitations of RGB images, they are easily affected by factors such as lighting background, target occlusion, and overlap, resulting in low target recognition rates and inability to classify correctly, failing to meet the needs of practical applications. With the development of sensor technology, devices capable of acquiring three-dimensional images have emerged. Three-dimensional images add depth information of objects to two-dimensional images. Recent RGB-D cameras, such as Kinect, can simultaneously capture RGB and depth images of objects. The depth image contains spatial geometric information of the object, allowing RGB and depth images to effectively complement each other, improving the robustness and accuracy of image recognition. Therefore, how to fully fuse the information from RGB and depth images is a key issue. Currently, RGB-D fusion networks exploring cross-modal complementarity of RGB and depth image data fall into two categories: single-stream network architecture and dual-stream network architecture. The single-stream network architecture learns feature maps by concatenating the data of both RGB and depth images. However, this type of algorithm directly concatenates the RGB and depth image data, ignoring the differences between the two data sets, and the resulting feature maps cannot fully represent the image. Two-stream network architectures learn features from RGB and depth images through two independent branches, then learn a joint representation of the features from both through a shared network layer added early or late to obtain the final feature map. This type of algorithm extracts features from both depth and RGB images separately, fusing RGB and depth features at the feature layer, which can effectively improve the accuracy of image classification. However, the deep neural networks used for feature extraction suffer from problems such as long computation time and complex structure.

[0004] Therefore, this invention takes into account the characteristics of RGB images and depth images, selects width learning with a simple flat-layer network structure, fast learning speed and few steps, and introduces transfer learning to establish a network structure that integrates the image features of both. Experiments on the RGB-D dataset show that the network structure has higher classification accuracy and robustness. Summary of the Invention

[0005] This invention addresses the problem that existing RGB and depth images cannot be fully fused, and provides a collaborative fusion method for RGB-D image features based on transfer learning and width learning. Compared with traditional neural network models, it has higher accuracy and better stability.

[0006] The technical solution provided by this invention is a collaborative fusion method for RGB-D image features based on transfer learning and width learning, specifically implemented according to the following steps:

[0007] S1: Obtain the Washington RGB-D dataset from a public dataset, process the data to obtain the network input;

[0008] S2: Build a neural network ResNet, perform initial training on the ImageNet dataset, obtain the initially trained neural network ResNet, and save it;

[0009] S3: Fine-tune the ResNet neural network obtained from the initial training on the RGB-D dataset, remove the last layer of the ResNet neural network, and after inputting the data, the RGB image and the depth image will each output a feature vector in the network to obtain the features of the two images;

[0010] S4: Based on the two feature vectors of RGB image and depth image in step S3, they are used as two sets of variables with no obvious correlation as input to the canonical correlation analysis (CCA) layer. The correlation coefficient between the two sets of variables is maximized, and the two sets of variables are uniformly mapped to the feature space generated by CCA for learning. According to the CCA feature fusion strategy, the parallel matrix is ​​used as the fusion feature to achieve feature fusion and dimensionality reduction.

[0011] S5: Based on the fusion features in step S4, feature nodes and enhancement nodes are generated using width learning, and the final classification and recognition result is obtained using the generalized inverse of ridge regression.

[0012] Step S1 includes the following steps:

[0013] S11: Obtain the Washington RGB-D dataset from the public dataset to get RGB image and depth image data;

[0014] S12: Image preprocessing, resize the RGB image and depth image to 256*256 respectively.

[0015] Step S2 includes the following steps:

[0016] S21: Construct a ResNet neural network. The ResNet neural network is composed of modules of residual blocks. Each module is composed of several residual blocks with the same number of output channels. The construction process is as follows: The deep learning network ResNet is constructed using the PyTorch framework.

[0017] S22: Perform initial training on the ImageNet dataset. The specific process is as follows: load the publicly available ResNet parameters that have been trained on the ImageNet dataset; obtain the initially trained neural network ResNet and save it.

[0018] Step S3 includes the following steps:

[0019] S31: Remove the final average pooling layer and linear layer of the pre-trained ResNet neural network.

[0020] S32: Input the processed RGB image and depth image into the adjusted ResNet neural network model, respectively. The neural network model constructed in step S31 obtains the RGB image feature vector and the depth image feature vector. The two output feature vectors are then average-pooled and flattened.

[0021] Step S5 includes the following steps:

[0022] S51: Use the fused features from step S4 as input X for width learning. * Generate n sets of feature maps, each containing i feature nodes, and generate a random weight matrix W. ei If the values ​​of are Gaussian distributed, then the i-th group of mapping features is as shown in the equation: in, Let β represent the activation function. ei This represents a randomly generated bias. Meanwhile, width learning employs a sparse autoencoder approach to optimize the input weights W. ei .

[0023] S52: Feature Node Z i Enhanced node H is generated through nonlinear mapping. j Define and generate m groups of enhancement nodes, and calculate H. j =Z i W hj +β hj And scale H and calculate Using an activation function to activate the boosting nodes, the boosting nodes in the j-th group can be represented as:

[0024]

[0025] Where ξ represents the nonlinear activation function, W hj and β hj This represents a fixed set of randomly generated weights and biases;

[0026] Therefore, the output of width learning is represented as:

[0027]

[0028] Among them W bls The output layer weights represent the width-based learning. This represents all input features for width learning.

[0029] S53: This can be obtained from step S52. And W bls Through the The ridge regression approximation is calculated based on the formula: Training weights W bls The final classification results and accuracy are obtained: λ is the eigenvalue, and I is the identity matrix.

[0030] Compared with the prior art, the technical effects and advantages of the present invention are as follows:

[0031] This invention proposes a collaborative fusion method for RGB-D image features based on transfer learning and width learning. This model can balance efficiency and accuracy, achieve efficient feature extraction of RGB-D images, and use canonical correlation to achieve dimensionality reduction through the fusion of features from the two modalities. While completing the image classification and recognition task, it reduces training time and improves classification performance.

[0032] The technical solution proposed in this invention uses transfer learning to achieve efficient extraction of RGB and depth images, which can avoid the problem of insufficient data to train a good neural network, and effectively reduce training time.

[0033] The technical solution proposed in this invention uses typical correlation analysis to fuse features, which can effectively integrate RGB image features and depth image features, efficiently utilize the correlation between RGB images and depth images, and reduce feature dimensionality.

[0034] The technical solution proposed in this invention uses width learning to efficiently achieve image classification tasks, which simplifies the time complexity of fusion algorithms, accelerates training, and makes the model have good stability and robustness, greatly improving the accuracy of image classification and recognition and increasing work efficiency. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the specific embodiments of the present invention, the accompanying drawings used in the description of the specific embodiments are briefly introduced below:

[0036] Figure 1 This is a flowchart of an RGB-D image feature collaborative fusion method based on transfer learning and width learning according to the present invention.

[0037] Figure 2 This is a network architecture diagram of an RGB-D image feature collaborative fusion method based on transfer learning and width learning according to the present invention.

[0038] Figure 3 This is a flowchart illustrating the transfer learning approach used in this invention.

[0039] Figure 4 This is a flowchart illustrating how the present invention uses canonical correlation analysis to fuse RGB image features and depth image features.

[0040] Figure 5 This is a flowchart illustrating the classification and recognition process using width learning in this invention.

[0041] Specific implementation method

[0042] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0043] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0044] This invention discloses a collaborative fusion method for RGB-D image features based on transfer learning and width learning, such as... Figure 1 As shown, the main steps include:

[0045] S1: Obtain the Washington RGB-D dataset from a public dataset, process the data to obtain the network input;

[0046] S2: Build a neural network ResNet, perform initial training on the ImageNet dataset, obtain the initially trained neural network ResNet, and save it;

[0047] S3: Fine-tune the ResNet neural network obtained from the initial training on the RGB-D dataset, remove the last layer of the ResNet neural network, and after inputting the data, the RGB image and the depth image will each output a feature vector in the network to obtain the features of the two images;

[0048] S4: Based on the two feature vectors of RGB image and depth image in step S3, they are used as two sets of variables with no obvious correlation as input to the canonical correlation analysis (CCA) layer. The correlation coefficient between the two sets of variables is maximized, and the two sets of variables are uniformly mapped to the feature space generated by CCA for learning. According to the CCA feature fusion strategy, the parallel matrix is ​​used as the fusion feature to achieve feature fusion and dimensionality reduction.

[0049] S5: Based on the fusion features in step S4, feature nodes and enhancement nodes are generated using width learning, and the final classification and recognition result is obtained using the generalized inverse of ridge regression.

[0050] Step S1 includes the following steps:

[0051] S11: Obtain the Washington RGB-D dataset from a public dataset to acquire RGB and depth image data. The specific process for acquiring this dataset is as follows: each object is placed on a turntable, and a 3D camera records one complete rotation cycle of the turntable. Each object contains three video sequences, recorded from cameras at different heights. This yields RGB and depth images of common everyday items from different perspectives and under different lighting conditions, forming the dataset.

[0052] S12: During the experiment, the validation dataset was divided into two parts: a test set and a training set. RGB images and depth images were preprocessed, and the image sizes were adjusted to 256*256.

[0053] Step S2 includes the following steps:

[0054] S21: Construct a ResNet neural network. The ResNet neural network is composed of modules of residual blocks. Each module is composed of several residual blocks with the same number of output channels. The construction process is as follows: The deep learning network ResNet is constructed using the PyTorch framework.

[0055] S22: Perform initial training on the ImageNet dataset. The specific process is as follows: load the publicly available ResNet parameters that have been trained on the ImageNet dataset; obtain the ResNet neural network trained for the first time and save it.

[0056] Step S3 includes the following steps:

[0057] S31: Remove the final average pooling layer and linear layer of the pre-trained ResNet neural network.

[0058] S32: Input the processed RGB image and depth image into the adjusted ResNet neural network model, respectively. The neural network model constructed in step S31 yields the RGB image feature vector and the depth image feature vector. The two output feature vectors are then average-pooled and flattened to finally obtain the RGB image feature vector X = {x1, x2, ... x...} f} and depth image feature vector Y = {y1, y2, ... y f}

[0059] Step S4 includes the following steps:

[0060] S41: Based on the RGB image feature vector X = {x1, x2, ... x3} in step S3... f} and depth image feature vector Y = {y1, y2, ... y f} These are two groups of variables that do not have an explicit correlation, respectively, used as inputs to the CCA layer of canonical correlation analysis;

[0061] S42: Calculate the covariance matrix ∑ of each component of X. XX Calculate the covariance matrix ∑ of each component of Y. YY Calculate the cross-covariance matrix ∑ of X and Y. XY ;

[0062] S43: Calculate the matrix Calculate the singular value decomposition of matrix A to obtain the maximum singular value δ and the corresponding left singular vector u and right singular vector v;

[0063] S44: Calculate the projection vectors W of X and Y. X and W Y ,get and Therefore, we obtain the linear combination α = W X T X and β = W Y T The correlation between Y is the highest;

[0064] S45: Based on the CCA feature fusion strategy, the parallel matrix As a fusion feature, it achieves feature fusion and dimensionality reduction.

[0065] Step S5 includes the following steps:

[0066] S51: The steps in S4... The fused features are used as input for width learning. First, the input features are established. The mapping to feature nodes first involves Z-score standardization of the input features to ensure that the input data is normalized to between 0 and 1, and then... To augment the training set, an additional column is added at the end. This is done to ensure that when generating feature nodes, a bias term can be directly added through matrix operations, transforming it into X. * 2f+1 ;

[0067] S52: Generate a random weight matrix W ei W ei The value is Gaussian; calculation That is, perform a random convolution and bias on the features of each sample to generate n sets of feature maps, each set containing i feature nodes;

[0068] S53: To Z i Normalization and sparse representation are performed. The Lasso method is used to address the optimization problem in the sparsity process, fine-tuning the random features into a more compact set of features. This is solved using the following formula:

[0069]

[0070] S54: As seen in step S52, the generated feature nodes are linear. A characteristic of width learning is that augmentation nodes can supplement random feature nodes. Therefore, the purpose of introducing augmentation nodes is to increase the nonlinearity in the network, allowing feature nodes to be generated through nonlinear mapping. Unlike feature nodes, the coefficient matrix W of augmentation nodes... hj It is not a random matrix, but a random matrix that has been orthogonally normalized. The purpose is to map the feature nodes to a high-dimensional subspace through nonlinear mapping, so that the network has a stronger expressive power.

[0071] S55: Activate the augmentation nodes and set S to control the shrinkage size of the augmentation nodes to prevent redundancy caused by too many augmentation nodes. Define the generation of m groups of augmentation nodes and calculate H. j =Z i W hj +β hj And scale H and calculate Using the activation function tansig to activate the augmented nodes, the augmented nodes in the j-th group can be represented as:

[0072]

[0073] S56: The final input to the width learning network is: The output of width learning is represented as: And W bls Through the The ridge regression approximation is calculated based on the formula: Training weights W bls The final classification result and accuracy are obtained, where λ is the feature value and I is the identity matrix:

[0074]

[0075] This implementation uses the Washington RGB-D Object dataset. The RGB-D Object dataset consists of RGB-D images of 300 common daily necessities from 51 categories under different viewing angles and lighting conditions. All images were acquired using a Kinect camera, totaling 278,374 images. To evaluate the performance of the proposed model in multimodal fusion classification and recognition tasks, the classification accuracy of this invention compared to other methods is shown in Table 1. A comparative model, as shown in Table 2, was also designed, using BLS to learn and classify color images, depth images, and images obtained by simply stitching the two together. The comparison mainly focuses on recognition rate, training time, and testing time. As can be seen from Tables 1 and 2, the algorithm proposed in this invention can effectively learn the high-dimensional features of color and depth images, greatly improving the accuracy of classification and recognition. This sufficiently demonstrates the speed and effectiveness of the algorithm.

[0076] Table 1 Comparison of image classification accuracy on the RGB-D Object dataset

[0077] method accuracy Liner SVM 81.92％ Nonliner SVM 74.80％％ Random Forest 79.62％ DKD 86.00％ HKDES 82.40％ Ours 87.19％

[0078] Table 2 Comparison of classification accuracy on the RGB-D Object dataset

[0079] Image modality depth color splicing Fusion Training time / s 58.62 53.59 95.91 13.60 Test time / s 1.34 1.76 2.34 0.61 Recognition rate / % 53.07 65.32 73.62 87.19

[0080] In summary, this invention presents a collaborative fusion method for RGB-D image features based on transfer learning and width learning. It employs transfer learning, applying pre-trained ResNet neural network parameters trained on the large image dataset ImageNet to the feature extraction process of RGB and depth images, rapidly improving and learning image features. Canonical correlation analysis (CCA) effectively fuses the features of the two images, while CCA's dimensionality reduction properties facilitate model simplification. Furthermore, the introduction of width learning enhances the system's operating speed, ultimately resulting in higher accuracy and reliability of the classification results.

[0081] The examples described above are merely specific embodiments of the present invention, used to illustrate the technical solutions of the present invention, and are not intended to limit it. The scope of protection of the present invention is not limited thereto; the scope of protection of the present invention is determined by the scope of the claims. Within the understanding of those skilled in the art, any changes, modifications, substitutions, combinations, or simplifications made within the spirit and principles of the present invention should be considered equivalent substitutions and are all included within the scope of protection of the present invention.

Claims

1. A collaborative fusion method for RGB-D image features based on transfer learning and width learning, characterized in that, Includes the following steps: S1: Obtain the RGB-D dataset from a public dataset; S2: Build a ResNet neural network, perform initial training on the ImageNet dataset, obtain the initially trained ResNet neural network, and save it; S3: Fine-tune the ResNet neural network obtained from the initial training on the RGB-D dataset. Remove the last layer of the ResNet neural network. After inputting the data, the RGB image and the depth image will each output a feature vector in the network to obtain the features of the two images. S4: Based on the two feature vectors of RGB image and depth image in step S3, they are used as two sets of variables with no obvious correlation as input to the canonical correlation analysis (CCA) layer. The correlation coefficient between the two sets of variables is maximized, and the two sets of variables are uniformly mapped to the feature space generated by CCA for learning. According to the CCA feature fusion strategy, the parallel matrix is ​​used as the fusion feature to achieve feature fusion and dimensionality reduction. S5: Based on the fusion features in step S4, feature nodes and enhancement nodes are generated using width learning, and the final classification and recognition result is obtained using the generalized inverse of ridge regression.

2. The RGB-D image feature collaborative fusion method based on transfer learning and width learning according to claim 1, characterized in that, In step S1, the specific process of acquiring RGB-D data is as follows: each object is placed on a turntable, and a 3D camera is used to record one complete rotation cycle of the turntable. Each object contains 3 video sequences, which are recorded on cameras at different heights. This allows us to obtain RGB images and depth images from different perspectives and under different lighting conditions, forming a dataset.

3. The RGB-D image feature collaborative fusion method based on transfer learning and width learning according to claim 2, characterized in that, In step S2, the neural network ResNet is a module composed of residual blocks. Each module is composed of several residual blocks with the same number of output channels. The construction process is as follows: the neural network ResNet is constructed using the PyTorch framework.

4. The RGB-D image feature collaborative fusion method based on transfer learning and width learning according to claim 3, characterized in that, In step S2, the preliminary training process specifically involves loading and saving the publicly available ResNet parameters that have been trained on the ImageNet dataset.

5. The RGB-D image feature collaborative fusion method based on transfer learning and width learning according to claim 4, characterized in that, In step S3, after fine-tuning the network, the last layer of the ResNet neural network is removed. After inputting data, the RGB image and the depth image output a feature vector in the network respectively. The output feature vector is then averaged and flattened.

6. The RGB-D image feature collaborative fusion method based on transfer learning and width learning according to claim 5, characterized in that, In step S4, the specific method for fusing the two sets of feature vectors is as follows: The purpose of CCA is to fuse the two sets of feature vectors... and Find a pair of projection axes and Perform a linear transformation to make the linear combination and The correlation between them is the highest.

7. The RGB-D image feature collaborative fusion method based on transfer learning and width learning according to claim 6, characterized in that, In step S5, the specific process of width learning classification and recognition is as follows: the fusion features in step S4 are used as input samples. ,generate Group feature mapping, each group contains Each feature node generates a random weight matrix. If its value follows a Gaussian distribution, then the th Group mapping features ,in, This represents the activation function. The bias is represented by a randomly generated value. Meanwhile, the width learning employs a sparse autoencoder approach to optimize the random weight matrix. , Define and generate m sets of augmentation nodes, and calculate and to Scaling and calculation Using an activation function to activate the augmented nodes, the augmented nodes in the j-th group are represented as follows: ,in, Represents a non-linear activation function. and This represents fixed, randomly generated weights and biases; the output of width-learned training is represented as: ,in The output layer weights represent the width-based learning. This represents all input features for width learning.

Images