A cross-domain recognition interaction system based on multi-modal visual cues

Abstract

The application relates to the technical field of intelligent identification, in particular to a cross-domain identification interaction system based on multi-modal visual prompts, a dynamic modal alignment module is used for generating an adaptability prompt template and a vector, a mapping relationship between visual features and semantic features is established through a global-local two-stage alignment strategy, and multi-modal features are dynamically fused based on attention weights; a domain self-adaption generalization module is used for realizing cross-domain adaptation from a source domain to a target domain through domain prototype learning and adversarial training; a robustness enhancement module is used for repairing damaged modalities and performing consistency verification on inference results; a lightweight inference module is used for reducing calculation overhead and meeting real-time interaction requirements through a modular sparse architecture and an edge-cloud collaborative inference mode; and an interaction response output module is used for generating multi-modal interaction responses and receiving user feedback to iteratively optimize system parameters; the application provides a cross-domain identification interaction system with high reliability.

Description

Technical Field

[0001] This invention relates to the field of intelligent recognition technology, and in particular to a cross-domain recognition and interaction system based on multimodal visual cues. Background Technology

[0002] With the rapid development of artificial intelligence and multimodal fusion technologies, cross-domain recognition and interaction systems based on multimodal visual cues have been widely applied in various technical fields such as autonomous driving, smart healthcare, industrial quality inspection, and intelligent robotics. Their core objective is to achieve intelligent recognition and interactive response across scenarios and professional fields by integrating multi-source modal information such as images, text, and speech. These systems typically rely on visual cues and multimodal feature fusion mechanisms to associate the low-level features of visual data with high-level semantic information from modalities such as text, thereby completing recognition, reasoning, and interaction tasks and providing technical support for intelligent decision-making in complex scenarios.

[0003] The inherent heterogeneity between the representation spaces of visual and textual modalities makes it difficult to effectively bridge the semantic gap between them. Existing systems often employ static prompt templates or simple feature concatenation and weighted fusion to achieve modal interaction, lacking dynamic adaptation mechanisms for modal association and fine-grained semantic alignment strategies. This easily leads to the phenomenon of "modal silos," where visual features and textual semantics cannot be accurately matched, and alignment accuracy can decrease in complex scenarios due to feature interference or loss. The training of existing systems largely relies on large-scale labeled data from source domains, such as everyday scene images and general videos. When migrating to specialized target domains such as medical, industrial, and remote sensing, the differences in data distribution between domains and the scarcity of labeled data in the target domain cause a sharp drop in system recognition accuracy. At the same time, existing systems lack effective anti-interference mechanisms, resulting in a drastic performance collapse when modal integrity is compromised, or even slight perturbations leading to completely erroneous recognition results. Summary of the Invention

[0004] The purpose of this invention is to provide a cross-domain recognition and interaction system based on multimodal visual cues to solve the technical problems existing in the background art.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A cross-domain recognition and interaction system based on multimodal visual cues is characterized by comprising a multimodal data input module, a dynamic modality alignment module, a domain-adaptive generalization module, a robustness enhancement module, a lightweight inference module, and an interactive response output module that work sequentially and collaboratively; wherein, The dynamic modal alignment module is used to generate adaptive prompt templates and vectors based on the data input by the multimodal data input module. It establishes a mapping relationship between visual features and semantic features through a global-local two-level alignment strategy and dynamically fuses multimodal features based on attention weights. The domain adaptive generalization module is used to achieve cross-domain adaptation from the source domain to the target domain through domain prototype learning and adversarial training, in order to cope with domain offset and modality missing scenarios. The robustness enhancement module is used to repair damaged modes and perform consistency verification on inference results to improve the system's anti-interference capability. The lightweight inference module is used to reduce computational overhead and meet real-time interaction requirements through modular sparse architecture and edge-cloud collaborative inference. The interactive response output module is used to generate multimodal interactive responses and receive user feedback to iteratively optimize system parameters.

[0006] Furthermore, the dynamic modality alignment module includes a dynamic prompt generator, a fine-grained semantic alignment unit, and an adaptive fusion unit. The dynamic prompt generator generates prompts containing mappings of technical terms based on the domain type and generates completion prompts when a modality is missing. The fine-grained semantic alignment unit uses the InfoNCE loss function and the semantic consistency loss function to optimize the modality representation space. The adaptive fusion unit dynamically allocates modality weights through a gating fusion mechanism. When the modality is complete, the visual weight and text weight are set to 0.6 and 0.4, respectively, and when the modality is missing, the effective modality weight is increased to 0.9.

[0007] Furthermore, the domain adaptive generalization module includes a domain prototype learning unit, a domain adversarial adaptive unit, and a distribution drift detection and update unit; the domain prototype learning unit is used to construct a hybrid representation of "general prototype + domain prototype"; the distribution drift detection and update unit is used to monitor data distribution differences in real time, and when the distribution drift is detected to exceed a preset threshold, it triggers lightweight incremental learning to update the domain prototype and model parameters.

[0008] Furthermore, the robustness enhancement module includes a modality repair and anti-interference unit, an inference consistency verification unit, and a confidence calibration unit. The modality repair and anti-interference unit uses an autoencoder to repair damaged modalities and filters out prompts for disturbances through semantic similarity verification. The inference consistency verification unit is used to establish a cross-modal logic verification rule base and maintains cross-frame inference consistency in dynamic scenes through a temporal attention mechanism. The confidence calibration unit is used to evaluate the confidence of the recognition results and trigger secondary recognition when the confidence is lower than a threshold.

[0009] Furthermore, the lightweight inference module adopts a hybrid expert system architecture, which is divided into general experts and domain experts.

[0010] Furthermore, the interactive response output module includes an intent recognition submodule and a feedback iteration unit; the intent recognition submodule is used to parse the user's multimodal combination commands and prioritize responding to core requests; the feedback iteration unit is used to receive user feedback on the recognition results and automatically update the domain prototype library and dynamic modality alignment strategy, thereby achieving continuous optimization of system performance; the interactive response output module supports three output formats: text description, visual annotation, and voice broadcasting, to adapt to different application scenarios.

[0011] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention effectively solves the technical problems of insufficient cross-modal alignment accuracy and weak domain generalization adaptability by setting up a dynamic modal alignment module and a domain adaptive generalization module. The dynamic prompt generator can generate adaptive prompts according to the domain type and modal integrity. Combined with the global-local two-level alignment strategy and the InfoNCE+ semantic consistency loss function, it can achieve fine-grained and accurate mapping between visual features and semantic features, greatly bridging the modal semantic gap and avoiding modal silos. The domain prototype learning unit constructs a hybrid representation of "general prototype + domain prototype". With the cooperation of distribution drift detection and incremental learning, it significantly improves the system's adaptability to changes in professional domains and data distribution.

[0012] 2. This invention achieves a balanced optimization of robustness and real-time performance by setting up a robustness enhancement module and a lightweight inference module. The modal repair unit uses an autoencoder to repair damaged data, and the inference consistency verification and confidence calibration mechanism effectively filters disturbances and reduces phantom outputs, thus reducing the performance degradation of the system in scenarios with missing modalities and cue disturbances. The lightweight inference module adopts the MoE sparse architecture and feature quantization technology, compressing the model size to 1 / 4 of the original size while maintaining an accuracy loss of ≤3%, thereby improving inference speed. Combined with edge-cloud collaborative inference, latency is reduced, which not only meets the real-time requirements of autonomous driving, portable medical devices, and other scenarios, but also improves the reliability and deployment flexibility of the system in complex real-world environments. Attached Figure Description

[0013] Figure 1 This is the system architecture diagram of the present invention. Detailed Implementation

[0014] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0015] See Figure 1 As shown, this embodiment describes a cross-domain recognition and interaction system based on multimodal visual cues applied to a portable medical imaging terminal, as detailed below: (a) System Deployment Environment Hardware environment: The edge device uses a portable medical imaging terminal (processor is ARM Cortex-A78, memory is 8GB, storage is 64GB), and the cloud server uses CPU Intel Xeon Gold 6330 (32 cores) and GPU NVIDIA A100 (40GB). The edge and cloud communicate via 5G network (bandwidth ≥100Mbps).

[0016] Software environment: The edge device operates on Android 12 and uses the lightweight inference engine TensorRT 8.6; the cloud operating system is Ubuntu 20.04 and uses the PyTorch 2.0 deep learning framework. The database uses MySQL 8.0 to store domain prototype libraries and interaction logs.

[0017] (II) Implementation details of each module Multimodal data input module: Receives lung CT images (DICOM format, resolution 512×512) uploaded by doctors and text commands "identify ground-glass opacities". The preprocessing unit performs noise reduction (median filtering) and normalization (pixel values ​​are mapped to [0,1]) on the CT images. After checking the data integrity (no missing / damaged data), the images are converted to RGB format. The text commands are converted into semantic vectors through BPE word segmentation and RoPE positional encoding.

[0018] The dynamic modal alignment module: The dynamic prompt generator identifies medical domain attributes and generates a prompt with professional terminology mapping, such as "Ground-glass opacity → Visual features: blurred boundaries, slightly increased density area"; The fine-grained semantic alignment unit extracts 100 ROI region features from CT images through Faster R-CNN and maps them one-to-one with the text phrase "ground-glass opacity". It optimizes the modal representation space using InfoNCE loss (temperature parameter τ=0.07) + semantic consistency loss (weight 0.3); The adaptive fusion unit assigns weights (visual weight 0.6, text weight 0.4) through a gated fusion mechanism and outputs a fused feature vector (512 dimensions).

[0019] Domain Adaptive Generalization Module: The domain prototype learning unit calls a pre-trained general prototype library (containing 1000 categories of everyday object features) and a medical domain prototype library (containing 50 categories of lung lesion features, trained using 5-shot samples) to construct a hybrid prototype representation; the domain adversarial adaptive unit eliminates the distribution difference between the source domain (general image) and the target domain (CT image) through a domain discriminator (using a 3-layer fully connected network), with 100 iterations of adversarial training and a learning rate of 0.001; the distribution drift detection unit calculates the KL divergence between the input CT image and the training data in real time. When the divergence is >0.2 (preset threshold), lightweight incremental learning is triggered (only updating the ground-glass opacity-related features in the domain prototype library, iterating for 20 rounds).

[0020] Robustness enhancement module: The modal repair and anti-interference unit detects slight noise (no serious damage) in local areas of the CT image, which does not require repair. The semantic similarity check (threshold 0.85) confirms that the text command is not disturbed. The reasoning consistency check unit calls the cross-modal logic rule base (rule "text asks about lesion → visual features must include lesion area location information") to verify that the fused features contain the coordinate information of the ground-glass opacity area, and the logic is consistent. The confidence calibration unit evaluates the recognition result, and the confidence is 0.92 (higher than the threshold 0.7), so no secondary recognition is required.

[0021] Lightweight inference module: Adopting the MoE hybrid expert system architecture (including 4 general experts and 2 medical domain experts), only the medical domain expert module is activated to reduce redundant computation; the fused features are subjected to low-rank decomposition (rank=128) and 8-bit quantization processing, compressing the model size from the original 200MB to 50MB; edge devices complete image preprocessing and ROI extraction, and the cloud completes modality alignment and domain adaptation. Intermediate results are transmitted incrementally (only ROI features are transmitted, and the data volume is ≤1MB), with a total inference latency of 380ms.

[0022] Interactive response output module: The intent recognition submodule interprets the core requirement as "lesion recognition and annotation", and outputs a lung CT image (annotating the boundary box of the ground-glass opacity lesion in red) and a text report "1 ground-glass opacity lesion detected, location: upper lobe of right lung, size: 12mm×8mm", and supports voice broadcast (using a TTS engine, speech rate of 150 words / minute); The feedback iteration unit receives doctor feedback in real time. If the doctor corrects the lesion boundary box, it automatically updates the visual features of ground-glass opacity in the medical prototype library and optimizes the alignment strategy.

[0023] (III) Verification of Implementation Results Cross-modal alignment accuracy: The semantic matching accuracy of ground-glass opacity lesion identification reaches 93.5%, which is 21.5% higher than the existing system (accuracy 72%), and there is no "modal island" phenomenon.

[0024] Domain generalization capability: With only 5-shot lung lesion samples for matching, the cross-domain identification accuracy reaches 89%, which is 26% higher than the existing system (accuracy 63%).

[0025] Robustness: When 10% noise is added to CT images, the recognition accuracy drops to 82% (a decrease of 10%), with no erroneous annotations; when the text command is replaced with "recognize areas with increased fuzzy density" (synonym perturbation), ground-glass opacities can still be accurately identified, and the anti-interference capability is significantly improved.

[0026] Real-time performance: Inference latency of 380ms meets the real-time diagnostic requirements of portable medical terminals, and the power consumption of model deployment is reduced by 65% ​​compared with existing systems.

[0027] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A multi-modal visual cue based cross-domain recognition interaction system, characterized in that, It includes a multimodal data input module, a dynamic modality alignment module, a domain-adaptive generalization module, a robustness enhancement module, a lightweight inference module, and an interactive response output module that work sequentially and collaboratively; among them, The dynamic modal alignment module is used to generate adaptive prompt templates and vectors based on the data input by the multimodal data input module. It establishes a mapping relationship between visual features and semantic features through a global-local two-level alignment strategy and dynamically fuses multimodal features based on attention weights. The domain adaptive generalization module is used to achieve cross-domain adaptation from the source domain to the target domain through domain prototype learning and adversarial training, in order to cope with domain offset and modality missing scenarios. The robustness enhancement module is used to repair damaged modes and perform consistency verification on inference results to improve the system's anti-interference capability. The lightweight inference module is used to reduce computational overhead and meet real-time interaction requirements through modular sparse architecture and edge-cloud collaborative inference. The interactive response output module is used to generate multimodal interactive responses and receive user feedback to iteratively optimize system parameters.

2. The cross-domain recognition and interaction system based on multimodal visual cues according to claim 1, characterized in that, The dynamic modality alignment module includes a dynamic cue generator, a fine-grained semantic alignment unit, and an adaptive fusion unit; The dynamic prompt generator is used to generate prompts containing technical terminology mappings based on the domain type, and to generate completion prompts when a modality is missing; The fine-grained semantic alignment unit uses the InfoNCE loss function and the semantic consistency loss function to optimize the modal representation space; the adaptive fusion unit dynamically allocates modal weights through a gating fusion mechanism. When the modality is complete, the visual weight and text weight are set to 0.6 and 0.4 respectively, and when the modality is missing, the effective modal weight is increased to 0.

9.

3. The cross-domain recognition and interaction system based on multimodal visual cues according to claim 2, characterized in that, The domain adaptive generalization module includes a domain prototype learning unit, a domain adversarial adaptive unit, and a distribution drift detection and update unit. The domain prototype learning unit is used to construct a hybrid representation of "general prototype + domain prototype". The distribution drift detection and update unit is used to monitor data distribution differences in real time. When the distribution drift is detected to exceed a preset threshold, lightweight incremental learning is triggered to update the domain prototype and model parameters.

4. The cross-domain recognition and interaction system based on multimodal visual cues according to claim 3, characterized in that, The robustness enhancement module includes a modality repair and anti-interference unit, an inference consistency verification unit, and a confidence calibration unit. The modality repair and anti-interference unit uses an autoencoder to repair damaged modalities and filters out disturbances through semantic similarity verification. The inference consistency verification unit is used to establish a cross-modal logic verification rule base and maintains cross-frame inference consistency in dynamic scenarios through a temporal attention mechanism. The confidence calibration unit is used to evaluate the confidence of the recognition results and trigger secondary recognition when the confidence is lower than the threshold.

5. A cross-domain recognition and interaction system based on multimodal visual cues according to claim 4, characterized in that, The lightweight reasoning module adopts a hybrid expert system architecture, which is divided into general experts and domain experts.

6. A cross-domain recognition and interaction system based on multimodal visual cues according to claim 5, characterized in that, The interactive response output module includes an intent recognition submodule and a feedback iteration unit; the intent recognition submodule is used to parse the user's multimodal combination commands and prioritize responding to core requests; The feedback iteration unit is used to receive user feedback on the recognition results and automatically update the domain prototype library and dynamic modality alignment strategy, thereby achieving continuous optimization of system performance. The interactive response output module supports three output formats: text description, visual annotation, and voice broadcast, to adapt to different application scenarios.

Images