Cross-modal endoscopic surgery image text description generation method and system

By using cross-modal feature fusion technology, instrument and tissue features in endoscopic surgical images are extracted to generate accurate text descriptions, solving the problem of lack of text descriptions in endoscopic surgical images and improving the completeness of surgical understanding and analysis.

CN121354119AActive Publication Date: 2026-01-16CHANGCHUN UNIV OF SCI & TECH
5 Cites 0 Cited by

Patent Information

Application Number
CN202511905804.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-17
Publication Date
2026-01-16
Estimated Expiration
2045-12-17

AI Technical Summary

Technical Problem

Endoscopic surgical images lack textual descriptions and annotations, and the text model training is insufficient. The interaction between surgical instruments and tissues is complex, making it difficult for existing technologies to generate accurate textual descriptions of endoscopic surgical images.

Method used

A cross-modal method for generating text descriptions of endoscopic surgical images is adopted. Global and local detail visual features are extracted through a transferable distilled visual Transformer, and category features are generated by combining the visual Transformer model. The feature representation is optimized through cross-modal fusion and squeeze-excited attention mechanism, and text descriptions are generated by using self-attention mechanism and feedforward neural network.

Benefits of technology

It improves the accuracy and completeness of endoscopic surgical image processing and analysis, and provides precise auxiliary information for surgical navigation, intraoperative decision support and surgical teaching.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention belongs to the technical field of image data processing and generation, and particularly relates to a cross-modal endoscopic surgery image text description generation method and system. Comprising a scene feature extraction module used for processing endoscopic surgery images in parallel and extracting global and local visual features of a surgery scene; the instance encoding module is used for accurately identifying key objects such as surgical instruments, tissues and organs in the endoscopic surgery image, generating corresponding classification information and encoding the classification information into image category features; the cross-modal fusion module is used for carrying out cross-modal feature fusion on the image category features and the global visual features in a self-adaptive channel superposition mode; and the text description module is used for decoding the cross-modal fusion features to generate text description of the endoscopic surgery image. Through the cross-modal feature fusion technology, the instrument category semantic information and the panoramic visual information in the surgery scene are effectively combined, and the accuracy of endoscopic surgery image processing and the integrity of analysis are improved.
Need to check novelty before this filing date? Find Prior Art