An object recognition method for visual-haptic-dynamic combined perception

Abstract

The application provides an object recognition method for visual-haptic-dynamic joint perception, adopts Gaussian noise of different degrees to enhance original visual, haptic and dynamic data without noise, and performs data preprocessing of enhancement, dimension reduction and unified representation on the visual, haptic and dynamic data respectively; through a visual-haptic-dynamic joint model based on a Transformer, object recognition based on visual, haptic and dynamic data is realized; the visual-haptic-dynamic joint model comprises a visual feature extractor, a haptic feature extractor, a dynamic feature extractor, a weight-learnable weighted summation operation, a Transformer encoder and a classifier, the input of which is the preprocessed visual, haptic and dynamic data, and the output of which is an object category; and a learning rate change trend of a triangular periodic learning rate adjustment model is constructed. The method can comprehensively understand the attributes of an object and the environment where the object is located, and improve the performance of object recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical fields of deep learning technology, multi-modal learning technology and visual-haptic interaction, and particularly relates to an object recognition method for visual-haptic-motion joint perception. BACKGROUND

[0002] With the rise of embodied artificial intelligence, the perception combining visual, haptic and motion sensation has become a research hotspot. The perception combining visual, haptic and motion sensation plays an important role in the robot perception of environment in the field of embodied intelligence. However, when the robot perceives the environment, the visual, haptic and motion data used will be of low quality, which will deepen the perception errors and decision-making errors of the machine. In the real world, the quality of different modalities is dynamically changing in different scenes. Processing multi-modal data of dynamic quality is an inevitable problem of embodied artificial intelligence. SUMMARY

[0003] Therefore, the purpose of the present application is to provide an object recognition method for visual-haptic-motion joint perception. The method constructs a visual-haptic-motion joint model based on Transformer, enhances and preprocesses the data through Gaussian noise, trains using a triangular periodic learning rate, and fuses visual, haptic and motion information to realize the performance and generalization of the model for multi-modal data of dynamic quality, improve the object recognition performance, and solve the problem of poor generalization ability of robots in low-quality visual, haptic and motion data scenes.

[0004] The overall implementation of the scheme includes the following steps: visual-haptic-motion data preprocessing: enhancing the original visual, haptic and motion data without noise with different degrees of Gaussian noise, and enhancing, reducing dimension and unifying the representation of the visual, haptic and motion data respectively. Visual-haptic-motion joint model construction: constructing a visual-haptic-motion joint model based on Transformer to realize object recognition based on visual, haptic and motion data. Visual-haptic-motion joint model training: constructing the learning rate change trend of the triangular periodic learning rate adjustment model. The visual-haptic-motion joint model based on Transformer of the present application can dynamically adjust the weights of the three kinds of information through weighted summation, and suppress potential noise interference. Through visual-haptic-motion joint perception, the model can more comprehensively understand the attributes of the object and the environment it is in, and improve the performance of object recognition.

[0005] The technical solutions adopted by the present application to solve its technical problems are as follows:

[0006] The application discloses an object recognition method for visual-haptic-dynamic joint perception, which enhances original visual, haptic and dynamic data with different degrees of Gaussian noise, and performs data preprocessing on the visual, haptic and dynamic data in the forms of enhancement, dimension reduction and unified representation; the object recognition is realized based on visual, haptic and dynamic data through a visual-haptic-dynamic joint model based on a Transformer; the visual-haptic-dynamic joint model takes the preprocessed visual, haptic and dynamic data as input and outputs object categories; the visual-haptic-dynamic joint model comprises a visual feature extractor, a haptic feature extractor, a dynamic feature extractor, a weight-learnable weighted summation operation, a Transformer encoder and a classifier.

[0007] Further, the enhancement comprises changing brightness, contrast and rotation angle, the dimension reduction comprises downsampling and PCA, and the unified representation comprises splicing.

[0008] Further, the visual-haptic-dynamic joint model takes the preprocessed visual, haptic and dynamic data as input and outputs object categories; during the training process of the visual-haptic-dynamic joint model, a cross-entropy loss function is used as a loss function, and a triangular periodic learning rate is constructed to adjust the learning rate change trend during the training process.

[0009] Further, the visual-haptic-dynamic joint model dynamically adjusts the weights of the three kinds of information through the weight-learnable weighted summation operation to suppress potential noise interference.

[0010] Further, the preprocessing process specifically comprises:

[0011] A noise factor is set, a Gaussian noise array with the same size as the visual, haptic and dynamic data is generated, the Gaussian noise array is superimposed with the original visual, haptic and dynamic data to form visual, haptic and dynamic data with noise;

[0012] The visual data without noise and with noise is enhanced by changing the brightness value, the contrast value and the rotation angle, and the enhanced data is stored in the same folder;

[0013] The haptic data without noise and with noise is dimensionally reduced through downsampling and PCA, and the dimensionally reduced data is appended row by row to form a haptic data collection file;

[0014] The dynamic data without noise and with noise is uniformly represented by splicing in the same direction, and the uniformly represented data is appended row by row to form a dynamic data collection file.

[0015] Further, the visual feature extractor includes a Vision Transformer model pre-trained on ImageNet and three one-dimensional convolution layers, the calculation formula is shown as (1), the tactile feature extractor includes three one-dimensional convolution layers and a BiGRU layer, the calculation formula is shown as (2), and the kinesthetic feature extractor includes three one-dimensional convolution layers, the calculation formula is shown as (3):

[0016] F V =f 1DCNNs (f ViT (I V )) (1)

[0017] F T =f BiGRU (f 1DCNNs (I T )) (2)

[0018] F K =f 1DCNNs (I K ) (3)

[0019] wherein I V , I T and I K represent input visual, tactile and kinesthetic data respectively, F V , F T and F K represent output features of the visual feature extractor, the tactile feature extractor and the kinesthetic feature extractor respectively, f ViT (·) represents the Vision Transformer model pre-trained on ImageNet, f BiGRU (·) represents the BiGRU layer, and f 1DCNNs (·) represents the three one-dimensional convolution layers;

[0020] The output of the visual feature extractor, the tactile feature extractor and the kinesthetic feature extractor is subjected to a weighted summation operation with learnable weights, the calculation formula is shown as (4):

[0021]

[0022] wherein F VTK represents the fusion feature of F V , F T and F K , ω i and F i represent the weight and the feature of the i-th modality respectively, the weight is a learnable weight, and when i is 1, 2 and 3, it represents the visual, tactile and kinesthetic modalities respectively;

[0023] The fusion feature FVTK The input is input to a Transformer encoder for dynamic feature learning, and the Transformer encoder includes four multi-head self-attention mechanism layers;

[0024] The output of the dynamic feature learning is input to a classifier, and the probability is calculated by a softmax function, and the highest probability category is taken as the recognition result.

[0025] Further, the cross-entropy loss function calculation process is shown in (5):

[0026]

[0027] Wherein, Loss represents the loss value, y c represents the one-hot encoded label vector of the cth category, p c represents the probability of the cth category in the calculation of the classifier.

[0028] Further, the principle of the triangular periodic learning rate is that the learning rate is periodically changed in a triangle between two boundaries according to a specified constant frequency, and after each period, the maximum value of the learning rate is halved.

[0029] And an object recognition system for visual-haptic-kinesthetic joint perception, comprising: a preprocessing module, which enhances, reduces dimension, and uniformly represents the data preprocessing of visual, haptic, and kinesthetic data by using different degrees of Gaussian noise to enhance the original visual, haptic, and kinesthetic data without noise; and a Transformer-based visual-haptic-kinesthetic joint model for realizing object recognition based on visual, haptic, and kinesthetic data; the input of the visual-haptic-kinesthetic joint model is the visual, haptic, and kinesthetic data obtained by preprocessing, and the output is the object category; including a visual feature extractor, a haptic feature extractor, a kinesthetic feature extractor, a weight-learnable weighted summation operation, a Transformer encoder, and a classifier.

[0030] And an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the steps of the object recognition method for visual-haptic-kinesthetic joint perception as described above.

[0031] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to realize the steps of the object recognition method for visual-haptic-kinesthetic joint perception as described above.

[0032] Compared with the prior art, the application and the preferred scheme thereof combine the information of vision, touch and motion in the noise and non-noise scene, enhance, reduce dimension and uniformly represent the vision, touch and motion data respectively, construct a vision-touch-motion joint model based on Transformer, train in combination with a triangular periodic learning rate, improve the performance of the model in the object recognition task, and improve the generalization ability of the robot in the low-quality vision, touch and motion data scene. BRIEF DESCRIPTION OF DRAWINGS

[0033] The application will be described in further detail below with reference to the drawings and specific embodiments:

[0034] Figure 1 The overall flowchart of the object recognition method for the vision-touch-motion joint perception of the embodiment of the application is shown in the figure.

[0035] Figure 2 The vision-touch-motion joint model based on Transformer constructed in the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0036] To make the features and advantages of the application more obvious and easy to understand, the following embodiments are specifically described as follows:

[0037] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used in the specification have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0038] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0039] The embodiment of the application provides an object recognition method for vision-touch-motion joint perception, and the overall implementation flow is as shown in the figure. Figure 1 The method comprises the following steps:

[0040] Step S1, vision-touch-motion data preprocessing, enhancing the original non-noise vision, touch and motion data with different degrees of Gaussian noise, enhancing, reducing dimension and uniformly representing the noise and non-noise vision, touch and motion data, wherein the enhancing operation includes changing brightness, contrast and rotation angle, the dimension reduction includes downsampling and PCA, and the uniform representation includes splicing;

[0041] Step S2, a visual-haptic-kinesthetic joint model is constructed, a visual-haptic-kinesthetic joint model based on a Transformer is built, and the visual, haptic and kinesthetic data obtained through the preprocessing operation in step S1 are input into the joint model for recognition and classification, so that the output of the object category is realized. The joint model includes a visual feature extractor, a haptic feature extractor, a kinesthetic feature extractor, a weight-learnable weighted summation operation, a Transformer encoder and a classifier.

[0042] Step S3, the visual-haptic-kinesthetic joint model training, the cross-entropy loss function is used as the loss function of the visual-haptic-kinesthetic joint model in step S2, and the triangular periodic learning rate is constructed to adjust the learning rate change trend of the visual-haptic-kinesthetic joint model in step S2 during the training process.

[0043] In this embodiment, step S1, visual-haptic-kinesthetic data preprocessing, the original visual, haptic and kinesthetic data without noise are enhanced with different degrees of Gaussian noise, and the visual, haptic and kinesthetic data with and without noise are respectively enhanced, dimensionally reduced and uniformly represented. The enhancement operation includes changing the brightness, contrast and rotation angle, the dimension reduction includes downsampling and PCA, and the uniform representation includes splicing, which specifically includes the following steps:

[0044] Step S11, setting a noise factor, generating a Gaussian noise array with the same size as the visual, haptic and kinesthetic data, and superimposing the Gaussian noise array and the original visual, haptic and kinesthetic data to form visual, haptic and kinesthetic data with noise;

[0045] Step S12, enhancing the visual data without noise and generated with noise in step S11 by changing the brightness value, contrast value and rotation angle, and storing the enhanced data in the same folder;

[0046] Step S13, dimensionally reducing the haptic data without noise and generated with noise in step S11 through downsampling and PCA, and appending the dimensionally reduced data row by row to form a haptic data collection file;

[0047] Step S14, uniformly representing the kinesthetic data without noise and generated with noise in step S11 by splicing in the same direction, and appending the unified data row by row to form a kinesthetic data collection file.

[0048] Next, step S2, a visual-haptic-kinesthetic joint model is constructed, a visual-haptic-kinesthetic joint model based on Transformer is built, and the visual, haptic and kinesthetic data obtained after the preprocessing operation in step S1 are input into the joint model for recognition and classification to realize the output of the object category. The joint model includes a visual feature extractor, a haptic feature extractor, a kinesthetic feature extractor, a weight-learnable weighted summation operation, a Transformer encoder and a classifier, and specifically includes the following steps:

[0049] Step S21, according to the input shape of the visual, haptic and kinesthetic data after preprocessing in step S1, a visual feature extractor, a haptic feature extractor and a kinesthetic feature extractor are respectively designed. The visual feature extractor includes a Vision Transformer model pre-trained on ImageNet and three one-dimensional convolutional layers, the calculation formula is shown in (1), the haptic feature extractor includes three one-dimensional convolutional layers and a double BiGRU layer, the calculation formula is shown in (2), and the kinesthetic feature extractor includes three one-dimensional convolutional layers, the calculation formula is shown in (3):

[0050] F V =f 1DCNNs (f ViT (I V ) (1)

[0051] F T =f BiGRU (f 1DCNNs (I T )) (2)

[0052] F K =f 1DCNNs (I K ) (3)

[0053] Wherein, I V , I T and I K represent the input visual, haptic and kinesthetic data in step S1 respectively, F V , F T and F K represent the output features of the visual feature extractor, the haptic feature extractor and the kinesthetic feature extractor respectively, f ViT (·) represents the Vision Transformer model pre-trained on ImageNet, f BiGRU (·) represents the BiGRU layer, and f 1DCNNs (·) represents three one-dimensional convolutional layers.

[0054] Step S22, the outputs of the visual feature extractor, the tactile feature extractor and the kinesthetic feature extractor in step S21 are subjected to a weighted summation operation with learnable weights, and the calculation formula is shown in (4):

[0055]

[0056] wherein F VTK represents the fusion feature of F V , F T and F K in step S21, ω i and F i respectively represent the weight and the feature of the i-th modality, and when i is 1, 2 and 3, they respectively represent the visual, tactile and kinesthetic modalities;

[0057] In this design scheme, the weighted summation operation is learnable in weight, and can eliminate the potential noise in the visual, tactile and kinesthetic data.

[0058] Step S23, the fusion feature F VTK obtained in step S22 is input to the Transformer encoder for dynamic feature learning, and the Transformer encoder includes four multi-head self-attention mechanism layers; it takes advantage of the dynamic fusion characteristics of the Transformer encoder, which can adapt to the visual, tactile and kinesthetic data with dynamically changing quality.

[0059] Step S24, the output result of step S23 is input to the classifier, the probability is calculated by the softmax function, the highest probability class is taken, and the recognition result is obtained.

[0060] Further, step S3, the visual-tactile-kinesthetic joint model is trained, the cross-entropy loss function is used as the loss function of the visual-tactile-kinesthetic joint model in step S2, the learning rate change trend of the visual-tactile-kinesthetic joint model in step S2 during the training process is adjusted by constructing a triangular periodic learning rate, and the specific steps include the following steps:

[0061] Step S31, the cross-entropy loss function is used as the loss function of the visual-tactile-kinesthetic joint model in step S2, and the calculation process of the cross-entropy loss function is shown in (5):

[0062]

[0063] wherein Loss represents the loss value, y c represents the one-hot encoded label vector of the c-th class, and p c represents the probability of the c-th class in step S25.

[0064] Step S32, the visual-haptic-kinesthetic joint model of step S2 is made to change the learning rate in a triangular periodic manner between two boundaries with a specified constant frequency, and the maximum value of the learning rate is halved after each period by constructing a triangular periodic learning rate.

[0065] Those skilled in the art will appreciate that embodiments of the application can be supplied as methods, systems, or computer program products. Accordingly, the application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can be embodied in the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage media, etc.) having computer usable program code embodied therein.

[0066] The present application is described in reference to the flowchart and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowchart and / or block diagrams, as well as combinations of flows and / or blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram block or blocks.

[0067] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions means which implement the function specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram block or blocks.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the function specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram block or blocks.

[0069] It should be noted that the technical terms or scientific terms used in the present application should be understood as the common meanings understood by those skilled in the art unless otherwise defined. The terms "first", "second", and the like used in the present application do not represent any order, number, or importance, but are only used to distinguish different components. The terms "include" or "contain" and the like mean that the elements or objects before the terms cover the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and the like are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right", and the like are only used to represent relative positional relationships, and when the absolute positions of the described objects are changed, the relative positional relationships can also be changed accordingly.

[0070] The above is only the preferred embodiment of the present application, and does not limit other forms of the present application. Any skilled person in the art can modify or change the above disclosed technology into equivalent embodiments with equivalent changes. However, any simple modification, equivalent change, and modification of the above embodiments made according to the technical essence of the present application without departing from the technical solution of the present application still falls within the protection scope of the present application.

[0071] The present application is not limited to the above best mode, and anyone can derive other various forms of an object recognition method for visual-tactile-kinesthetic combined perception under the inspiration of the present application. Any equivalent change and modification made according to the scope of the present application should be within the scope of the present application.

Claims

1. A method for object recognition for visual-haptic-dynamic combined perception, characterized in that: The original visual, tactile and dynamic data without noise is enhanced by different degrees of Gaussian noise, and the data preprocessing of enhancing, dimensionality reduction and unified representation of visual, tactile and dynamic data is performed; object recognition based on visual, tactile and dynamic data is realized through a visual-tactile-dynamic joint model based on Transformer; the input of the visual-tactile-dynamic joint model is the visual, tactile and dynamic data obtained by preprocessing, and the output is the object category; the visual-tactile-dynamic joint model comprises a visual feature extractor, a tactile feature extractor, a dynamic feature extractor, a weight-learnable weighted summation operation, a Transformer encoder and a classifier; The enhancement includes changing brightness, contrast and rotation angle, the dimensionality reduction includes downsampling and PCA, and the unified representation includes splicing; The preprocessing process specifically comprises: A noise factor is set, a Gaussian noise array with the same size as the visual, tactile and dynamic data is generated, and the Gaussian noise array is superimposed with the original visual, tactile and dynamic data to form visual, tactile and dynamic data with noise; The visual data without noise and with noise is enhanced by changing the brightness value, contrast value and rotation angle, and the enhanced data is stored in the same folder; The tactile data without noise and with noise is dimensionality reduced by downsampling and PCA, and the dimensionality reduced data is appended row by row to form a tactile data collection file; The dynamic data without noise and with noise is unified represented by splicing in the same direction, and the unified data is appended row by row to form a dynamic data collection file. 2.The object recognition method for visual-haptic-dynamic combined perception according to claim 1, wherein: In the training process of the visual-tactile-dynamic joint model, the cross-entropy loss function is used as the loss function, and the learning rate change trend in the training process is adjusted by constructing a triangular periodic learning rate. 3.The object recognition method for visual-haptic-dynamic combined perception according to claim 1, wherein: The visual-tactile-dynamic joint model dynamically adjusts the weights of the three kinds of information by the weight-learnable weighted summation operation, and suppresses potential noise interference.

4. The object recognition method for visual-tactile-dynamic joint perception according to claim 1, characterized in that: The visual feature extractor includes a Vision Transformer model pre-trained on ImageNet and three one-dimensional convolution layers, and the calculation formula is as shown in the following formula (1) : As shown in the following formula (2), the tactile feature extractor includes three one-dimensional convolution layers and a BiGRU layer, and the calculation formula is as shown in the following formula (2) : As shown in the following formula (3), the kinesthetic feature extractor includes three one-dimensional convolution layers, and the calculation formula is as shown in the following formula (3) : As shown in the following formula (3), the kinesthetic feature extractor includes three one-dimensional convolution layers, and the calculation formula is as shown in the following formula (3) : wherein, , and represent the input visual, tactile and kinesthetic data, respectively, , and represent the output features of the visual, tactile and kinesthetic feature extractors, respectively, represents a Vision Transformer model pre-trained on ImageNet, represents a BiGRU layer, represents three one-dimensional convolutional layers; The outputs of the visual feature extractor, the tactile feature extractor and the kinesthetic feature extractor are subjected to a weighted summation operation with learnable weights, the formula of which is shown as shown as: in, express , and The fusion characteristics and They represent the first The weights and features of each modality, wherein the weights are learnable weights, when When the values ​​are 1, 2, and 3, they represent visual, tactile, and kinesthetic modalities, respectively. The fused features are input to a Transformer encoder for dynamic feature learning, the Transformer encoder including four multi-head self-attention mechanism layers; The output result of the dynamic feature learning is input to a classifier, and a recognition result is obtained by The function calculates the probability, and takes the highest probability category as the recognition result.

5. The object recognition method of claim 2, wherein: The principle of the triangular periodic learning rate is that the learning rate changes periodically in a triangle between two boundaries according to a specified constant frequency, and the maximum value of the learning rate is halved after each period.

6. The object recognition method for visual-tactile-dynamic joint perception according to claim 2, characterized in that: The cross-entropy loss function calculation process is as shown in FIG.

2. wherein, represents a loss value, represents a label vector of one-hot encoding of the th class, represents a probability of the th class in the calculation of the classifier.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps of the object recognition method for visual-tactile-dynamic joint perception according to any one of claims 1-6.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the object recognition method for visual-tactile-dynamic joint perception according to any one of claims 1-6.

Images