A method and system for automatically calibrating a basic visual model recognition result based on a multi-modal visual large model

Abstract

The application discloses a kind of based on multimodal vision big model automatic calibration basic vision model identification result method and system, belong to visual AI identification technical field.The present application aims to solve the problem of high false alarm rate of edge monitoring in complex scene caused by cross-modal feature alignment loss and prompt word static configuration in prior art.The method adopts "edge-platform" dual-mode architecture, in platform review system, by setting multi-level feature adapter, the cross-modal semantic alignment of image features and text instructions is realized;At the same time, through dynamic prompt word optimization system, according to scene context and reinforcement learning, the adaptive text instruction is automatically generated and optimized.The application realizes the secondary review of the identification result reported on the edge end through the joint action of the above-mentioned "feature adapter+prompt word optimization", significantly improves the accuracy and scene adaptability of visual recognition, effectively reduces the false alarm rate of the system.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a method and system for calibrating visual recognition results based on a multimodal large model. Background Technology

[0002] With the development of deep learning technology, a two-layer architecture of "initial early warning by a small model at the edge + secondary verification by a large model on the platform" is widely adopted in scenarios such as security monitoring to address the problems of limited computing resources and high false alarm rates of edge devices. For example, Chinese patent CN118433330B discloses a method for reducing the false alarm rate of edge monitoring using a large model. This method deploys a large model in the cloud to perform secondary verification of alarms uploaded from the edge, which alleviates the high false alarm problem caused by insufficient performance of small edge models to a certain extent.

[0003] However, existing technologies still face challenges in practice. First, current methods typically perform only simple linear transformations on image features before concatenating them with text features, failing to fully consider the heterogeneity of the feature spaces between the two. This resulting cross-modal semantic gap often leads to low accuracy in reviewing large models under varying lighting conditions and complex backgrounds, easily resulting in misjudgments. Furthermore, there are limitations in the configuration of prompt words. Most existing methods use fixed, templated, static prompt words, such as uniformly using "detect whether there is a fire in the image." This approach cannot adapt to dynamically changing scenarios. For example, the hazard level and morphological characteristics of smoke from a chemical plant are completely different from those of smoke from an ordinary office building. Fixed prompt words result in poor system scene adaptability, failing to accurately guide large models in reasoning, thus affecting the review effect.

[0004] Therefore, how to achieve deep semantic alignment between image features and text commands, and dynamically optimize prompt words according to specific scenarios, is a technical problem that urgently needs to be solved in the field of reducing false alarms in edge monitoring using large models. Summary of the Invention

[0005] To overcome the existing problems and shortcomings, this invention proposes a method for calibrating visual recognition results based on a multimodal large model, comprising the following steps:

[0006] At the edge, a small model is used to perform preliminary reasoning on the acquired images to generate structured alarm data containing the original images;

[0007] On the platform side, the structured alarm data is received, and the features of the image are processed through a multi-level feature adapter to achieve cross-modal semantic alignment between the image features and the text instruction features. At the same time, a dynamic prompt word optimization system automatically generates the text instruction that best matches the current scene based on the scene context and historical data. Using the multi-modal visual big model, a comprehensive judgment is made based on the aligned image features and the optimized text instruction, and the calibrated final recognition result is output. The text instruction is used to guide the multi-level feature adapter to perform dynamic feature projection.

[0008] Furthermore, the steps of the multi-level feature adapter in processing image features include:

[0009] Extract the initial feature map of the image;

[0010] The initial feature map is recalibrated using a spatial attention mechanism to enhance the feature response of key regions in the image;

[0011] Based on the text instructions, dynamic feature gating projection is performed on the recalibrated image features to output image features aligned with the text instruction features in both dimension and semantic space.

[0012] Furthermore, the recalibration of the initial feature map using the spatial attention mechanism is achieved by first compressing the initial feature map into channels, and then superimposing one or more 1x1 convolutional kernels to generate a single-channel attention heatmap.

[0013] Furthermore, the step of the dynamic prompt word optimization system generating text instructions includes:

[0014] The rule engine vectorizes structured historical events containing accident type, area code, and hazardous substance number to generate risk feature vectors.

[0015] The TF-IDF weighted algorithm is used to extract domain-specific keywords from a pre-built domain-specific thesaurus.

[0016] By integrating environmental features such as lighting and timestamps with extracted scene keywords, basic prompt words are generated;

[0017] Using a reinforcement learning model, the basic prompt words are iteratively optimized based on the execution effect of historical prompt words to output the final text instruction.

[0018] Furthermore, the reward mechanism of the reinforcement learning model includes: giving a positive reward when the optimized prompt words can improve the system's response speed, and giving a negative penalty when the optimized prompt words cause the false alarm rate to exceed a preset threshold.

[0019] Furthermore, the reinforcement learning model iteratively optimizes the basic prompt words by adjusting the syntax tree structure of the prompt words. The adjustment includes at least one of adding conditional clauses, replacing keywords, or inserting alarm levels.

[0020] Furthermore, the optimized text instruction is a composite instruction that includes the monitoring object, triggering condition, linkage operation, and response level.

[0021] A system for calibrating visual recognition results based on a multimodal large model includes:

[0022] An edge analysis system, configured at the edge, is used to perform preliminary reasoning on the acquired images and generate structured alarm data.

[0023] A platform review system, configured on the platform, is used to receive the structured alarm data and perform a second review on the preliminary inference results to output the final identification result. The platform review system further includes:

[0024] A multi-level feature adapter is used to process the image features in the alarm data to achieve cross-modal semantic alignment between the image features and the text command features;

[0025] A dynamic prompt word optimization system is used to automatically generate text instructions that best fit the current scene based on scene context and historical data, wherein the text instructions are used to guide the multi-level feature adapter to perform dynamic feature projection.

[0026] Beneficial effects:

[0027] This invention significantly improves recognition accuracy in complex scenes. By using a multi-level feature adapter and leveraging spatial attention and dynamic feature projection, it achieves cross-modal deep semantic alignment between image features and text commands. Experimental data shows that this design can increase the cosine similarity of cross-modal features from 0.52 to 0.87, effectively solving the misjudgment problem caused by feature space mismatch and significantly reducing the final false alarm rate in complex lighting and occlusion scenarios.

[0028] Building upon this foundation, the present invention further enhances the system's scene adaptability and automation level. Through a dynamic prompt word optimization system, the system can automatically generate and iteratively optimize the most suitable text instructions based on historical data and real-time scene context using reinforcement learning. Experimental data shows that this system can reduce scene semantic matching errors by 73%, enabling the system to automatically adjust its monitoring focus and judgment logic for different scenarios, much like a domain expert, thus improving automation and review efficiency.

[0029] This invention achieves efficient end-to-end identification and calibration. By seamlessly integrating the two modules into an "edge-platform" dual-mode architecture, it not only improves accuracy but also optimizes the overall performance of the system through a reinforcement learning reward mechanism, realizing a low-latency, high-precision closed-loop calibration process from edge early warning to precise platform verification. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of a system architecture for calibrating visual recognition results based on a multimodal large model, as provided in an embodiment of the present invention.

[0032] Figure 2 This is a schematic diagram of the workflow of the multi-level feature adapter in an embodiment of the present invention.

[0033] Figure 3 This is a schematic diagram of the workflow of the dynamic prompt word optimization system in an embodiment of the present invention. Detailed Implementation

[0034] The present application will be described below with reference to specific embodiments:

[0035] Example 1:

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0037] The system provided in this embodiment adopts an "edge-platform" dual-mode architecture, which mainly includes an edge analysis system deployed at the monitoring site and a platform review system deployed in the cloud or on a central server. At the edge, in this embodiment, the edge analysis system may include one or more image acquisition devices and an edge computing device. The image acquisition device may be a high-definition camera with explosion-proof function, and its resolution may be, for example, 2560×1440 or higher. The edge computing device may be an edge computing box equipped with an AI acceleration chip, and its computing power should be sufficient to support real-time inference of a lightweight visual model. These edge devices constitute the edge analysis system, which is responsible for performing preliminary inference on the images acquired in real time by the camera through the lightweight visual model. Once a suspected anomaly is detected, such as liquid leakage, unauthorized hot work, or personnel not wearing protective equipment as required, structured alarm data containing the original image, detection box, and preliminary label will be generated.

[0038] The generated alarm data is then pushed to the platform's review system via message middleware such as Kafka. This platform system is typically deployed on a central server, which, to support high-load computation of large models, is configured with a high-performance central processing unit and one or more graphics processing units (GPUs), and equipped with a large capacity of, for example, 512GB or more of memory. The core task of this platform review system is to perform accurate secondary review of alarms uploaded from the edge, and the key to achieving this task lies in its internally integrated multi-level feature adapter and dynamic prompt word optimization system.

[0039] When the platform's review system receives an alarm image from the edge, such as an image suggesting "liquid leakage at the valve connection of a transport tanker truck," the multi-level feature adapter in the platform's review system is activated to process the features of the alarm image, thereby reducing cross-modal semantic differences between the image features and the text command features used in subsequent analysis. Figure 2 As shown, firstly, the system can extract an initial feature map of the image using a standard visual backbone network (e.g., ResNet). This initial feature map can be a 256-channel, 64x64-dimensional feature map. To highlight key target regions in the image (such as a liquid leak) and suppress background noise (such as the vehicle body or the ground), the method further includes recalibrating the initial feature map using a spatial attention mechanism. In one specific implementation, the spatial attention mechanism can be implemented through local feature enhancement using one or more 1x1 convolutional kernels. The purpose is to generate an attention heatmap, which, after being multiplied point-by-point with the original feature map, can significantly improve the feature response intensity of key regions such as the leak point, while suppressing interference from background regions.

[0040] After the image features are enhanced, a multi-level feature adapter performs dynamic feature projection to align them semantically with the text instructions. Unlike the simple linear transformations in existing technologies, this embodiment employs a dynamic feature-gated projection mechanism. This mechanism uses the text instruction features generated by the dynamic prompt word optimization system as a control gate to perform non-linear dynamic weighting and selective projection on the image features after spatial attention recalibration. This approach allows the transformation path of the image features to adaptively adjust according to the semantics of the text instructions (e.g., whether the focus is on "temperature" or "pressure"), thereby outputting an image feature highly aligned with the text instruction features in both dimension and semantics, providing high-quality input for the final judgment of the multimodal visual large model.

[0041] A dynamic prompt optimization system is also working in parallel, aiming to automatically generate the most suitable text instructions based on the current scene context. For example... Figure 3 As shown, its workflow begins with the analysis of historical risk data. A rule engine parses and quantifies tens of thousands of historical events stored in the database (including accident type, berth code, hazardous materials CAS number, severity level, etc.) to generate a risk feature vector of, for example, 256 dimensions. Next, the system uses a TF-IDF weighted algorithm to analyze the scenario description text within a pre-built hazardous materials terminal safety thesaurus containing thousands of specialized terms. This extracts the keywords most relevant to the current scenario. For example, in the scenario of "liquid chlorine tank unloading," such as in the monitoring of hazardous chemical unloading operations, if the scenario description text contains "liquid chlorine tank unloading," the TF-IDF weighted algorithm can extract keywords such as "liquid chlorine," "tank," and "emergency shut-off valve" with high weights from the pre-built thesaurus.

[0042] The system integrates environmental features such as timestamps and lighting conditions with extracted scene keywords to generate a basic prompt word. However, this basic prompt word is not the final version; it is fed into a reinforcement learning optimizer for iterative optimization. This optimizer comprehensively considers the semantic features of the scene, historical risk profiles, and the execution effects of past prompt words (such as response latency) as its state space, and continuously explores and optimizes by adjusting the syntax tree structure of the prompt word (e.g., adding conditional clauses, replacing keywords, or inserting alarm levels) as its action space.

[0043] Specifically, the action space includes, but is not limited to, the following three operations: 1) adding conditional clauses, for example, adding "when temperature > 40℃" to the prompt; 2) replacing keywords, for example, optimizing the generalized "detect corrosion" to the more specific "ultrasonic thickness gauge scans weld corrosion"; 3) inserting alarm levels, for example, explicitly stating "pressure > 0.8MPa triggers L2 alarm" after the trigger condition. Through the combination and exploration of these atomic operations, the model can generate instructions with more complex structures and more rigorous logic.

[0044] The reward mechanism of this reinforcement learning model can include: when an optimized prompt word can speed up the system's emergency response, the model will receive a positive reward; and when it causes the false alarm rate to exceed a preset threshold, it will be negatively penalized. After thousands of training cycles, a prompt word that may initially be very simple (such as "monitor berth No. 3") may be iteratively optimized into a composite instruction containing multiple dimensions of information. The typical structure of this composite instruction includes four core elements: a clearly defined monitoring object, specific triggering conditions, linked operation instructions, and a preset response level. As an example, this composite instruction can be specifically expressed as: "Under specific environmental conditions (such as nighttime, strong winds), prioritize scanning the specified monitoring object (such as the liquid chlorine tank at berth No. 3); when key parameters (such as valve flange pressure, temperature rise rate) exceed preset thresholds (such as greater than 0.8 MPa, greater than or equal to 0.5℃ / min), trigger a linked operation (such as activating a gas detector for confirmation), and activate the preset response level (such as L3 response) after confirming the anomaly)." It should be understood that the specific instructions, objects, and thresholds here are all exemplary and can be adjusted according to the actual application scenario.

[0045] The multimodal visual big model in the final platform review system receives image features processed by multi-level feature adapters, as well as composite instruction text features generated by a dynamic prompt word optimization system. For comprehensive and in-depth reasoning and judgment, the big model internally executes a fusion and verification process based on a cross-attention mechanism. Specifically, the model uses the text instruction features as a query and the aligned image features as keys and values. It calculates attention scores to determine which regions of the image are most relevant to the semantics of the text instruction (e.g., "pressure > 0.8 MPa", "temperature rise rate ≥ 0.5℃ / min"). Subsequently, the model outputs a confidence score based on the fit between the visual features of these highly relevant regions and the text instruction. Only when this score exceeds a preset judgment threshold (e.g., 0.9) will the system confirm the validity of the alarm and output a precisely calibrated final recognition result, presented to monitoring personnel through a visual interface. Through this process, the present invention achieves low-latency, high-precision closed-loop calibration from initial edge warning to precise platform review, significantly improving the reliability and automation level of the visual recognition system in complex scenarios such as hazardous materials terminals. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0046] Example 2:

[0047] This invention also provides a system for calibrating visual recognition results based on a multimodal large model, used to execute the method provided in Embodiment 1. Figure 1 As shown, the system includes: an edge analysis system and a platform review system.

[0048] The edge analysis system is configured at the monitoring site to perform preliminary inference on the acquired images and generate structured alarm data. In this embodiment, the edge analysis system may consist of the following components:

[0049] One or more image acquisition devices: for example, a high-definition camera with explosion-proof capabilities and a resolution of 2560×1440 or higher.

[0050] Edge computing devices: For example, edge computing boxes equipped with AI acceleration chips. A lightweight vision model runs on this device, responsible for analyzing images in real time and generating initial alerts.

[0051] Data push module: For example, through message middleware such as Kafka, the generated structured alarm data is asynchronously pushed to the platform review system.

[0052] The platform review system is configured in the cloud or on a central server to receive alarm data and perform secondary review. In this embodiment, the platform review system can be deployed on a central server equipped with a high-performance CPU and GPU, and further includes:

[0053] Multi-level Feature Adapter: This adapter is used to process the features of alarm images. Internally, it can integrate a visual backbone network (such as ResNet) for extracting initial feature maps, a spatial attention module (such as implemented with 1x1 convolutional kernels) for recalibrating feature maps, and a dynamic feature gating projection module for non-linear projection and alignment of features according to text instructions.

[0054] Dynamic Prompt Optimization System: This system is used to automatically generate and optimize text instructions. Internally, it may include a rule engine for parsing historical risk data, a TF-IDF weighted algorithm module for extracting scene keywords, and a reinforcement learning optimizer for iteratively optimizing basic prompts. The action space of this optimizer can be defined as adjustments to the prompt lexical tree (such as adding conditional clauses, replacing keywords, etc.).

[0055] Multimodal vision large model: used to receive image features processed by the adapter and text instructions generated by the optimizer, make the final comprehensive judgment, and output the calibrated recognition result.

Claims

1. A method for automatically calibrating the recognition results of a basic visual model based on a large multimodal visual model, characterized in that, Includes the following steps: At the edge, a small model is used to perform preliminary reasoning on the acquired images to generate structured alarm data containing the original images; On the platform side, the structured alarm data is received, and the image features of the image are processed through a multi-level feature adapter to achieve cross-modal semantic alignment between the image features and the text instruction features. At the same time, a dynamic prompt word optimization system is used to automatically generate the text instruction most suitable for the current scene based on the scene context and historical data. The multi-modal visual big model is used to make a comprehensive judgment based on the aligned image features and the optimized text instruction, and output the calibrated final recognition result. The text instruction is used to guide the multi-level feature adapter to perform dynamic feature projection. The steps of the multi-level feature adapter in processing image features include: Extract the initial feature map of the image; The initial feature map is recalibrated using a spatial attention mechanism to enhance the feature response of key regions in the image; Based on the text instructions, dynamic feature gating projection is performed on the recalibrated image features to output image features aligned with the text instruction features in both dimension and semantic space. The steps of the dynamic prompt word optimization system in generating text instructions include: The rule engine vectorizes structured historical events containing accident type, area code, and hazardous substance number to generate risk feature vectors. The TF-IDF weighted algorithm is used to extract domain-specific keywords from a pre-built domain-specific thesaurus. By integrating environmental features with extracted scene keywords, basic prompt words are generated. The environmental features include lighting and timestamps. Using a reinforcement learning model, the basic prompt words are iteratively optimized based on the execution effect of historical prompt words to output the final text instruction.

2. The method according to claim 1, characterized in that, The recalibration of the initial feature map using the spatial attention mechanism is achieved by first compressing the initial feature map into channels, and then superimposing one or more 1x1 convolutional kernels to generate a single-channel attention heatmap.

3. The method according to claim 1, characterized in that, The reward mechanism of the reinforcement learning model includes: giving a positive reward when the optimized prompt words can improve the system's response speed, and giving a negative penalty when the optimized prompt words cause the false alarm rate to exceed a preset threshold.

4. The method according to claim 1, characterized in that, The reinforcement learning model iteratively optimizes the basic prompt words by adjusting the syntax tree structure of the prompt words. The adjustment includes at least one of adding conditional clauses, replacing keywords, or inserting alarm levels.

5. The method according to claim 1, characterized in that, The optimized text command is a composite command that includes the monitored object, triggering condition, linkage operation, and response level.

6. A system for automatically calibrating the recognition results of a basic visual model based on a large multimodal visual model, characterized in that, include: An edge analysis system, configured at the edge, is used to perform preliminary reasoning on the acquired images and generate structured alarm data. A platform review system, configured on the platform, receives the structured alarm data and utilizes the multimodal visual large model within it to perform a comprehensive judgment based on aligned image features and optimized text instructions. It then performs a secondary review of the preliminary inference result to output a calibrated final recognition result. The platform review system further includes: A multi-level feature adapter is used to: extract an initial feature map of the image; recalibrate the initial feature map using a spatial attention mechanism to enhance the feature response of key regions in the image; and perform dynamic feature gating projection on the recalibrated image features according to text instructions to output image features aligned with the text instruction features in dimensionality and semantic space. A dynamic prompt word optimization system is used to: vectorize structured historical events containing accident type, region code, and hazardous substance number through a rule engine to generate risk feature vectors; extract domain-specific keywords from a pre-set domain-specific thesaurus using a TF-IDF weighted algorithm; fuse environmental features such as illumination and timestamps with the extracted scene keywords to generate basic prompt words; and use a reinforcement learning model to iteratively optimize the basic prompt words based on the execution effect of historical prompt words to output the final text instruction. The text instructions are used to guide the multi-level feature adapter to perform dynamic feature projection.

Images