An object recognition method based on visual-haptic-text multi-modal fusion

By employing a vision-touch-text multimodal fusion method, and utilizing a multi-head cross-attention mechanism and a self-attention module for hierarchical fusion, the shortcomings of single visual perception and vision-touch fusion methods in object recognition under complex environments are addressed, achieving high accuracy and robust object recognition.

CN122174151APending Publication Date: 2026-06-09HUAZHONG UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
HUAZHONG UNIV OF SCI & TECH
Filing Date
2026-02-12
Publication Date
2026-06-09

AI Technical Summary

Technical Problem

In existing technologies, single visual perception significantly reduces object recognition performance in low light, occlusion, reflection, or scenarios where objects look similar but have significantly different physical properties. Furthermore, visual-tactile fusion methods struggle to fully model cross-modal relationships, and textual semantic information is not effectively incorporated into the object recognition framework, resulting in insufficient understanding of object attributes and semantics in robot object recognition.

Method used

A method based on visual-tactile-text multimodal fusion is adopted. Through a multi-head cross-attention mechanism with parameter sharing, a two-way deep interaction of visual, tactile and text information is realized. Six sets of cross-modal fusion features are extracted and hierarchically fused through a self-attention module to output object category prediction results.

Benefits of technology

It significantly improves the accuracy and robustness of object recognition, is applicable to complex real-world environments, and achieves collaborative modeling of physical perception and semantic perception, thereby enhancing the accuracy and stability of object recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122174151A_ABST
    Figure CN122174151A_ABST
Patent Text Reader

Abstract

This application relates to the field of robot perception and intelligent operation technology, and discloses an object recognition method based on visual-tactile-text multimodal fusion. The method includes acquiring multimodal input data of the object to be recognized, including visual, tactile, and text modal data; extracting visual, tactile, and text modal feature sequences based on the visual, tactile, and text modal data; employing a parameter-sharing multi-head cross-attention mechanism to obtain six sets of cross-modal fusion features; concatenating two sets of cross-modal fusion features corresponding to each modality in the visual, tactile, and text modal data to obtain enhanced visual, tactile, and text modal feature representations; cascading the enhanced visual, tactile, and text modal feature representations and inputting them into a self-attention module to output multimodal fusion features; and inputting the multimodal fusion features into a classification head network for recognition, outputting object category prediction results. This method can improve the accuracy and robustness of object recognition.
Need to check novelty before this filing date? Find Prior Art