Object identification method for visual-tactile-kinesthetic combined perception

Abstract

The invention provides an object identification method for visual-tactile-kinesthetic combined perception, which comprises the following steps of: enhancing original visual, tactile and kinesthetic data without noise by adopting Gaussian noise of different degrees, and respectively performing data preprocessing of enhancement, dimensionality reduction and unified representation on the visual, tactile and kinesthetic data; object identification based on visual, tactile and kinesthetic data is realized through a visual-tactile-kinesthetic joint model based on Transform; the vision-tactile-kinesthetic joint model comprises a visual feature extractor, a tactile feature extractor, a kinesthetic feature extractor, a weight learnable weighted summation operation, a Transform encoder and a classifier, the input of the vision-tactile-kinesthetic joint model is vision, tactile and kinesthetic data obtained through preprocessing, and the output of the vision-tactile-kinesthetic joint model is an object category; and the learning rate change trend of the model is adjusted by constructing a triangular periodic learning rate. The method can more comprehensively understand the attributes of the object and the environment where the object is located, and improves the object recognition performance.

Description

Technical Field

[0001] The present invention belongs to the technical fields of deep learning technology, multimodal learning technology and visual-tactile interaction, and specifically relates to an object recognition method for visual-tactile-kinesthetic joint perception. Background Art

[0002] With the rise of embodied artificial intelligence, perception combining vision, touch and kinesthetics has become a research hotspot. Combining vision, touch and kinesthetics plays an important role in robot perception of the environment in the field of embodied intelligence. However, when the visual, tactile and kinesthetic data used by robots to perceive the environment are of low quality, it will deepen the machine's perception errors and decision-making errors. In the real world, the quality of different modalities changes dynamically in different scenarios. Processing multimodal data with dynamically changing quality is an inevitable problem for embodied artificial intelligence. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide an object recognition method for visual-tactile-kinesthetic joint perception, which constructs a visual-tactile-kinesthetic joint model based on Transformer, enhances and preprocesses data through Gaussian noise, and utilizes triangular periodic learning rate training to integrate visual, tactile and kinesthetic information, thereby achieving the performance and generalization of the model for multimodal data with dynamically changing quality, improving the object recognition performance, and solving the problem of poor generalization ability of the robot in low-quality visual, tactile and kinesthetic data scenarios.

[0004] The overall implementation of the scheme includes the following steps: Visual-tactile-kinesthetic data preprocessing: using different degrees of Gaussian noise to enhance the original noise-free visual, tactile and kinesthetic data, respectively enhancing, reducing the dimension and uniformly representing the visual, tactile and kinesthetic data. Visual-tactile-kinesthetic joint model construction: constructing a Transformer-based visual-tactile-kinesthetic joint model to achieve object recognition based on visual, tactile and kinesthetic data. Visual-tactile-kinesthetic joint model training: constructing the learning rate change trend of the triangular periodic learning rate adjustment model. The Transformer-based visual-tactile-kinesthetic joint model of the present invention can dynamically adjust the weights of the three types of information through weighted summation to suppress potential noise interference. Through visual-tactile-kinesthetic joint perception, the model can more comprehensively understand the properties of the object and the environment in which it is located, and improve the performance of object recognition.

[0005] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0006] A method for object recognition for visual-tactile-kinesthetic joint perception, which uses different degrees of Gaussian noise to enhance original noise-free visual, tactile and kinesthetic data, and performs data preprocessing for enhancement, dimension reduction and unified representation of the visual, tactile and kinesthetic data respectively; through a Transformer-based visual-tactile-kinesthetic joint model, object recognition based on visual, tactile and kinesthetic data is realized; the input of the visual-tactile-kinesthetic joint model is the preprocessed visual, tactile and kinesthetic data, and the output is the object category; the method includes a visual feature extractor, a tactile feature extractor, a kinesthetic feature extractor, a weighted sum operation with learnable weights, a Transformer encoder and a classifier.

[0007] Furthermore, the enhancement includes changing brightness, contrast, and rotation angle, the dimensionality reduction includes downsampling and PCA, and the unified representation includes splicing.

[0008] Furthermore, the input of the visual-tactile-kinesthetic joint model is the visual, tactile and kinesthetic data obtained through preprocessing, and the output is the object category; during the training process of the visual-tactile-kinesthetic joint model, the cross entropy loss function is used as the loss function, and the learning rate change trend during the training process is adjusted by constructing a triangular periodic learning rate.

[0009] Furthermore, the visual-tactile-kinesthetic joint model dynamically adjusts the weights of the three types of information through a learnable weighted sum to suppress potential noise interference.

[0010] Furthermore, the pre-processing process specifically includes:

[0011] Setting a noise factor, generating a Gaussian noise array of the same size as the visual, tactile and kinesthetic data, and superimposing the Gaussian noise array with the original visual, tactile and kinesthetic data to form visual, tactile and kinesthetic data with noise;

[0012] Enhance the visual data with and without noise by changing the brightness value, contrast value, and rotation angle, and store the enhanced data in the same folder;

[0013] The tactile data without noise and with noise are reduced in dimension by downsampling and PCA, and the reduced-dimensional data are appended line by line to form a tactile data collection file;

[0014] The kinesthetic data without noise and with noise are uniformly represented by splicing in the same direction, and the unified data are appended line by line to form a kinesthetic data collection file.

[0015] Furthermore, the visual feature extractor includes a Vision Transformer model pre-trained on ImageNet and three one-dimensional convolutional layers, and the calculation formula is shown in (1). The tactile feature extractor includes three one-dimensional convolutional layers and one BiGRU layer, and the calculation formula is shown in (2). The kinesthetic feature extractor includes three one-dimensional convolutional layers, and the calculation formula is shown in (3):

[0016] F V =f 1DCNNs (f ViT (I V )) (1)

[0017] F T =f BiGRU (f 1DCNNs (I T )) (2)

[0018] F K =f 1DCNNs (I K ) (3)

[0019] Among them, I V ,I T and I K Represent the input visual, tactile and kinesthetic data respectively, and F V 、F T and F K denote the output features of the visual feature extractor, tactile feature extractor, and kinesthetic feature extractor, respectively, and f ViT (·) represents the Vision Transformer model pre-trained on ImageNet, f BiGRU (·) represents the BiGRU layer, f 1DCNNs (·) represents three one-dimensional convolutional layers;

[0020] The outputs of the visual feature extractor, the tactile feature extractor and the kinesthetic feature extractor are subjected to a weighted summation operation with learnable weights, and the calculation formula is shown in (4):

[0021]

[0022] Among them, F VTK Indicates F V 、F T and F K The fusion feature of i and F i Respectively represent the weight and feature of the i-th modality, the weight is a learnable weight, and when i is 1, 2, and 3, it represents the visual, tactile, and kinesthetic modalities, respectively;

[0023] The fusion feature FVTK Input to the Transformer encoder for dynamic feature learning. The Transformer encoder includes four multi-head self-attention mechanism layers;

[0024] The output results of dynamic feature learning are input into the classifier, the probability is calculated by the softmax function, and the category with the highest probability is taken as the recognition result.

[0025] Furthermore, the cross entropy loss function calculation process is shown in (5):

[0026]

[0027] Among them, Loss represents the loss value, y c represents the label vector of the one-hot encoding of the c-th category, p c Represents the probability of the cth category in the classifier calculation.

[0028] Furthermore, the principle of the triangular periodic learning rate is that the learning rate changes triangularly periodically between two boundaries at a specified constant frequency, and after each cycle, the maximum value of the learning rate is halved.

[0029] And, an object recognition system for visual-tactile-kinesthetic joint perception, including: a preprocessing module, which enhances the original noise-free visual, tactile and kinesthetic data by using different degrees of Gaussian noise to enhance, reduce the dimension and uniformly represent the visual, tactile and kinesthetic data respectively; and a Transformer-based visual-tactile-kinesthetic joint model, which is used to realize object recognition based on visual, tactile and kinesthetic data; the input of the visual-tactile-kinesthetic joint model is the visual, tactile and kinesthetic data obtained by preprocessing, and the output is the object category; it includes a visual feature extractor, a tactile feature extractor, a kinesthetic feature extractor, a weighted sum operation with learnable weights, a Transformer encoder and a classifier.

[0030] And, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the object recognition method for visual-tactile-kinesthetic combined perception as described above are implemented.

[0031] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the object recognition method for visual-tactile-kinesthetic joint perception as described above.

[0032] Compared with the prior art, the present invention and its preferred solution combine the information of vision, touch and kinesthetic sense in noisy and non-noisy scenes, and construct a Transformer-based visual-tactile-kinesthetic joint model by enhancing, reducing the dimension and uniformly representing the visual, tactile and kinesthetic data respectively. The model is trained with a triangular periodic learning rate, which improves the performance of the model in object recognition tasks and improves the generalization ability of the robot in low-quality visual, tactile and kinesthetic data scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] Figure 1 This is an overall flow chart of the object recognition method for visual-tactile-kinesthetic joint perception according to an embodiment of the present invention;

[0035] Figure 2 Diagram of the Transformer-based visual-tactile-kinesthetic joint model constructed for an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the features and advantages of the present invention more clearly understood, the following embodiments are specifically described in detail as follows:

[0037] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0038] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0039] The embodiment of the present invention provides an object recognition method for visual-tactile-kinesthetic joint perception, and its overall implementation process is as follows: Figure 1 As shown. The following steps are included:

[0040] Step S1, preprocessing visual-tactile-kinesthetic data, using different degrees of Gaussian noise to enhance the original visual, tactile and kinesthetic data without noise, and performing enhancement, dimensionality reduction and unified representation operations on the noisy and non-noisy visual, tactile and kinesthetic data respectively, wherein the enhancement operation includes changing brightness, contrast and rotation angle, the dimensionality reduction includes downsampling and PCA, and the unified representation includes splicing;

[0041] Step S2, constructing a visual-tactile-kinesthetic joint model, building a Transformer-based visual-tactile-kinesthetic joint model, inputting the visual, tactile and kinesthetic data obtained by the preprocessing operation in step S1 into the joint model for recognition and classification, and realizing the output of object categories. The joint model includes a visual feature extractor, a tactile feature extractor, a kinesthetic feature extractor, a weighted sum operation with learnable weights, a Transformer encoder and a classifier;

[0042] Step S3, visual-tactile-kinesthetic joint model training, using the cross entropy loss function as the loss function of the visual-tactile-kinesthetic joint model of step S2, and adjusting the learning rate change trend of the visual-tactile-kinesthetic joint model of step S2 during the training process by constructing a triangular periodic learning rate.

[0043] In this embodiment, step S1, visual-tactile-kinesthetic data preprocessing, using different degrees of Gaussian noise to enhance the original visual, tactile and kinesthetic data without noise, respectively performing enhancement, dimensionality reduction and unified representation operations on the noisy and non-noisy visual, tactile and kinesthetic data, the enhancement operation includes changing the brightness, contrast and rotation angle, the dimensionality reduction includes downsampling and PCA, and the unified representation includes splicing, specifically including the following steps:

[0044] Step S11, setting a noise factor, generating a Gaussian noise array of the same size as the visual, tactile and kinesthetic data, and superimposing the Gaussian noise array with the original visual, tactile and kinesthetic data to form visual, tactile and kinesthetic data with noise;

[0045] Step S12, enhancing the visual data without noise and the visual data with noise generated in step S11 by changing the brightness value, contrast value, and rotation angle, and storing the enhanced data in the same folder;

[0046] Step S13, reducing the dimension of the tactile data without noise and the tactile data with noise generated in step S11 by downsampling and PCA, and appending the reduced-dimensional data line by line to form a tactile data collection file;

[0047] Step S14, unifying the kinesthetic data without noise and the kinesthetic data with noise generated in step S11 by splicing in the same direction, appending the unified data line by line to form a kinesthetic data collection file.

[0048] Next, step S2, visual-tactile-kinesthetic joint model construction, build a Transformer-based visual-tactile-kinesthetic joint model, input the visual, tactile and kinesthetic data obtained by the preprocessing operation in step S1 into the joint model recognition classification to achieve the output of object category. The joint model includes a visual feature extractor, a tactile feature extractor, a kinesthetic feature extractor, a weighted sum operation with learnable weights, a Transformer encoder and a classifier, and specifically includes the following steps:

[0049] Step S21: According to the input shapes of the visual, tactile and kinesthetic data preprocessed in step S1, a visual feature extractor, a tactile feature extractor and a kinesthetic feature extractor are designed respectively. The visual feature extractor includes a Vision Transformer model pre-trained on ImageNet and three one-dimensional convolutional layers, and the calculation formula is shown in (1). The tactile feature extractor includes three one-dimensional convolutional layers and one double BiGRU layer, and the calculation formula is shown in (2). The kinesthetic feature extractor includes three one-dimensional convolutional layers, and the calculation formula is shown in (3):

[0050] F V =f 1DCNNs (f ViT (I V ) (1)

[0051] F T =f BiGRU (f 1DCNNs (I T )) (2)

[0052] F K =f 1DCNNs (I K ) (3)

[0053] Among them, I V ,I T and I K represent the visual, tactile and kinesthetic data in step S1, respectively, V 、F T and F K denote the output features of the visual feature extractor, tactile feature extractor, and kinesthetic feature extractor, respectively, and f ViT (·) represents the Vision Transformer model pre-trained on ImageNet, f BiGRU (·) represents the BiGRU layer, f 1DCNNs (·) represents three one-dimensional convolutional layers;

[0054] Step S22: Perform a weighted sum operation on the outputs of the visual feature extractor, the tactile feature extractor and the kinesthetic feature extractor in step S21, and the calculation formula is shown in (4):

[0055]

[0056] Among them, F VTK In step S21, F V 、F T and F K The fusion feature of i and F i They represent the weight and features of the i-th modality, respectively. When i is 1, 2, and 3, they represent the visual, tactile, and kinesthetic modalities, respectively.

[0057] In this design, the weighted summation operation is weight-learnable and can eliminate potential noise in visual, tactile, and kinesthetic data.

[0058] Step S23: The fusion feature F obtained in step S22 VTK , input to the Transformer encoder for dynamic feature learning. The Transformer encoder includes four multi-head self-attention mechanism layers; it utilizes the dynamic fusion characteristics of the Transformer encoder and can adapt to the visual, tactile and kinesthetic data of dynamically changing quality.

[0059] Step S24: Input the output result of step S23 to the classifier, calculate the probability through the softmax function, take the highest probability category, and obtain the recognition result.

[0060] Further, step S3, visual-tactile-kinesthetic joint model training, using the cross entropy loss function as the loss function of the visual-tactile-kinesthetic joint model of step S2, and adjusting the learning rate change trend of the visual-tactile-kinesthetic joint model of step S2 during the training process by constructing a triangular periodic learning rate, specifically comprising the following steps:

[0061] Step S31: Use the cross entropy loss function as the loss function of the visual-tactile-kinesthetic joint model of step S2. The cross entropy loss function calculation process is shown in (5):

[0062]

[0063] Among them, Loss represents the loss value, y c represents the label vector of the one-hot encoding of the c-th category, p c represents the probability of the cth category in step S25;

[0064] Step S32: construct a triangular periodic learning rate so that the learning rate of the visual-tactile-kinesthetic joint model in step S2 during training changes triangularly periodically between two boundaries at a specified constant frequency, and after each cycle, the maximum value of the learning rate is halved.

[0065] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0066] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0067] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0069] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0070] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0071] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone can derive other various forms of an object recognition method for visual-tactile-kinesthetic combined perception under the inspiration of the present invention. All equal changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A method for object recognition based on visual-tactile-kinesthetic joint perception, characterized in that: The original noise-free visual, tactile and kinesthetic data are enhanced by using Gaussian noise of different degrees, and the visual, tactile and kinesthetic data are respectively enhanced, reduced in dimension and uniformly represented by data preprocessing; object recognition based on visual, tactile and kinesthetic data is realized by a Transformer-based visual-tactile-kinesthetic joint model; the input of the visual-tactile-kinesthetic joint model is the preprocessed visual, tactile and kinesthetic data, and the output is the object category; the model includes a visual feature extractor, a tactile feature extractor, a kinesthetic feature extractor, a weighted sum operation with learnable weights, a Transformer encoder and a classifier.

2. The object recognition method for visual-tactile-kinesthetic combined perception according to claim 1, characterized in that: The enhancement includes changing brightness, contrast, and rotation angle, the dimensionality reduction includes downsampling and PCA, and the unified representation includes splicing.

3. The object recognition method for visual-tactile-kinesthetic combined perception according to claim 1, characterized in that: During the training process of the visual-tactile-kinesthetic joint model, a cross entropy loss function is used as a loss function, and a triangular periodic learning rate is constructed to adjust the learning rate change trend during the training process.

4. The object recognition method for visual-tactile-kinesthetic combined perception according to claim 1, characterized in that: The visual-tactile-kinesthetic joint model dynamically adjusts the weights of the three types of information through a learnable weighted sum to suppress potential noise interference.

5. The object recognition method for visual-tactile-kinesthetic combined perception according to claim 1, characterized in that: The pre-treatment process specifically includes: Setting a noise factor, generating a Gaussian noise array of the same size as the visual, tactile and kinesthetic data, and superimposing the Gaussian noise array with the original visual, tactile and kinesthetic data to form visual, tactile and kinesthetic data with noise; Enhance the visual data with and without noise by changing the brightness value, contrast value, and rotation angle, and store the enhanced data in the same folder; The tactile data without noise and with noise are reduced in dimension by downsampling and PCA, and the reduced-dimensional data are appended line by line to form a tactile data collection file; The kinesthetic data without noise and with noise are uniformly represented by splicing in the same direction, and the unified data are appended line by line to form a kinesthetic data collection file.

6. The object recognition method for visual-tactile-kinesthetic combined perception according to claim 1, characterized in that: The visual feature extractor includes a Vision Transformer model pre-trained on ImageNet and three one-dimensional convolutional layers. The calculation formula is as follows: ( 1 ) As shown in Figure 2, the tactile feature extractor includes three one-dimensional convolutional layers and one BiGRU layer. The calculation formula is as follows: ) As shown in Figure 2, the kinesthetic feature extractor includes three one-dimensional convolutional layers, and the calculation formula is as follows: ( 3 ) As shown: F V =f 1DCNNs (f ViT (I V )) (1) F T =f BiGRU (f 1DCNNs (I T )) (2) F K =f 1DCNNs (I K ) (3) Among them, I V ,I T and I K Represent the input visual, tactile and kinesthetic data respectively, and F V 、F T and F K denote the output features of the visual feature extractor, tactile feature extractor, and kinesthetic feature extractor, respectively, and f ViT (·) represents the Vision Transformer model pre-trained on ImageNet, f BiGRU (·) represents the BiGRU layer, f 1DCNNs (·) represents three one-dimensional convolutional layers; The outputs of the visual feature extractor, the tactile feature extractor and the kinesthetic feature extractor are subjected to a weighted summation operation with learnable weights, and the calculation formula is shown in (4): Among them, F VTK Indicates F V 、F T and F K The fusion feature of i and F i Respectively represent the weight and feature of the i-th modality, the weight is a learnable weight, and when i is 1, 2, and 3, it represents the visual, tactile, and kinesthetic modalities, respectively; The fusion feature F VTK Input to the Transformer encoder for dynamic feature learning. The Transformer encoder includes four multi-head self-attention mechanism layers; The output results of dynamic feature learning are input into the classifier, the probability is calculated by the softmax function, and the category with the highest probability is taken as the recognition result.

7. The object recognition method for visual-tactile-kinesthetic combined perception according to claim 3, characterized in that: The principle of the triangular periodic learning rate is that the learning rate changes triangularly periodically between two boundaries at a specified constant frequency, and after each cycle, the maximum value of the learning rate is halved.

8. The object recognition method for visual-tactile-kinesthetic combined perception according to claim 4, characterized in that: The cross entropy loss function calculation process is as follows ( 5 ) As shown: Among them, Loss represents the loss value, y c represents the label vector of the one-hot encoding of the c-th category, p c Represents the probability of the cth category in the classifier calculation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the object recognition method for visual-tactile-kinesthetic joint perception as described in any one of claims 1-8 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the object recognition method for visual-tactile-kinesthetic joint perception as described in any one of claims 1 to 8 are implemented.

Images