A visual multi-modal image detection method for AI training
By using a cross-modal attention mechanism and a large language model to generate a multi-agent framework, dynamic cue vectors are generated and iteratively optimized, solving the robustness and interpretability issues of multimodal detection methods in complex environments, and achieving efficient detection and adaptive capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SAIER DIGITAL (BEIJING) TECH CO LTD
- Filing Date
- 2026-05-25
- Publication Date
- 2026-07-31
AI Technical Summary
Existing multimodal detection methods lack robustness in complex environments, have weak generalization ability, rely on manually designed prompts and lack interpretability, making it difficult to adapt to rapidly changing detection environments and product lines.
A cross-modal attention mechanism is adopted to fuse features from heterogeneous sensors. A multi-agent framework is generated using a large language model. The prompt vector is dynamically generated and its self-consistency is verified through iterative optimization. Natural language explanatory text is then output.
It significantly improves the robustness and accuracy of detection, enhances the ability to generalize to unknown defects, reduces maintenance costs, and increases the transparency and reliability of the system.
Smart Images
Figure CN122265806B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image detection technology, specifically to a visual multimodal image detection method for AI training. Background Technology
[0002] With the rapid development of artificial intelligence technology, deep learning-based visual image detection methods have been widely used in industrial quality inspection, autonomous driving, and intelligent security. Traditional visual inspection methods usually rely on single-modal image data, such as visible light images or infrared images. However, single-modal sensors face inherent limitations in complex environments: visible light cameras are prone to losing key information under low light, strong exposure, or occlusion conditions, while infrared sensors, although capable of sensing thermal radiation, lack sufficient texture details and are difficult to accurately identify object boundaries and surface defects. Therefore, how to effectively integrate the complementary information of heterogeneous sensors such as visible light and infrared to improve the robustness of the detection system in complex environments has become a hot topic in current multimodal visual inspection research.
[0003] Meanwhile, most existing multimodal detection methods still employ fixed fusion strategies and manually designed prompts or query mechanisms. These methods have the following shortcomings: First, they have weak generalization ability to unknown scenarios or novel defects, making it difficult to dynamically adapt to rapid changes in product lines or detection environments; second, they rely on expert knowledge to manually write prompts, which is not only time-consuming and labor-intensive but also prone to subjective bias and semantic omissions; third, the detection process lacks interpretability, usually only outputting category or location information, failing to provide human-understandable judgment criteria, and reducing the transparency and credibility of artificial intelligence systems.
[0004] Therefore, there is an urgent need for a visual multimodal image detection method that can automatically fuse features from heterogeneous sensors, dynamically generate adaptive query prompts, and possess natural language interpretation capabilities, so as to simultaneously improve detection accuracy, automation level, domain adaptability, and result interpretability. Summary of the Invention
[0005] The purpose of this invention is to provide a visual multimodal image detection method for AI training, so as to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution: a visual multimodal image detection method for AI training, comprising the following steps:
[0007] S1. After obtaining the original multimodal data tensor, use a convolutional network to extract features, align them spatially and through channels, compensate for geometric deviations, and finally output the aligned multimodal feature set.
[0008] S2. Input the multimodal feature set into the multi-agent framework generated by the large language model. The image annotation agent generates scene text description, the defect description agent dynamically generates prompt vectors, and the anomaly evaluation agent outputs anomaly scores to quantify the defect confidence.
[0009] S3. Calculate the consistency loss through the meta-evaluator. If the loss exceeds the threshold, the defect description agent calculates the gradient and updates the prompt vector to drive the anomaly evaluation agent to evaluate the weight of the anomaly evaluation agent. Iterate and optimize until the termination condition is met.
[0010] S4. Obtain the cue vector and anomaly score, and fuse the scene description, qualitative words of score mapping and defect features of cue decoding through a word combination function to generate natural language explanation text, and output a detection window containing cue, score and explanation.
[0011] Preferably, step S1 specifically includes the following steps:
[0012] S101, at discrete time step The original multimodal data sequence was obtained. ,in , Indicates the first The raw data for each modality, express Corresponding image height, express The corresponding width, express The corresponding number of channels, Represent the field of real numbers, for the th Modality Deep features are extracted using a convolutional backbone network that matches the modality type.
[0013] S102, change the original mode Mapping to a high-dimensional feature space, the output is the original feature map. ,in , This represents the spatial height obtained after convolutional downsampling. This represents the spatial width obtained after convolutional downsampling. This represents a predefined number of feature channels that remains consistent across all modalities;
[0014] S103. Repeat the feature extraction operation for each modality in the sequence to obtain a set of original feature maps. ,in Indicates the first The feature map of the i-th modality is selected. Feature map of each modality and the Feature map of each modality Then, two learnable projection matrices are determined, one for querying the projection matrix and the other for learning the projection matrix. Bond projection matrix ,use To each Perform a linear transformation to obtain and ,pass and Calculate the similarity score matrix and count the feature dimensions after projection. Applying the softmax function, based on the similarity score matrix and Analyze the attention weight matrix ,in The attention weight matrix is calculated by repeating the process. ;
[0015] S104. After performing cross-modal attention alignment on all modal pairs, for each original feature map Generate aligned features ,in , Indicates the sequence number, and all aligned features After aggregation, an aligned multimodal feature set is constructed. .
[0016] Preferably, step S2 specifically includes the following steps:
[0017] S201, Set up multimodal features The data is fed into a multi-agent collaborative framework generated by instantiating a large language model. This framework leverages the instruction following and context learning capabilities of the large language model to instantly generate three agents with distinct roles: an image annotation agent, a defect description agent, and an anomaly assessment agent. These agents then collaboratively process the input data in a pipeline manner.
[0018] S202, Receive multimodal feature set through image annotation agent. As the sole input, the internal vision-to-language generation function decodes word by word in an autoregressive manner to output a text description.
[0019] Preferably, step S2 further includes the following steps:
[0020] S203. Using a defect description agent, the agent takes the multimodal feature set and the text description output by the image annotation agent as joint input, and dynamically constructs a cue vector through a generator function.
[0021] S204. Using an anomaly assessment agent, the cue vector generated by the defect description agent is jointly input with the multimodal feature set. The cue vector is used as a condition key and deeply integrated with the multimodal features through feature affine transformation to obtain the fused features. A conditional probability model is applied to analyze these features, thereby outputting a scalar score. Specifically, the scalar score is:
[0022]
[0023] in, This represents the original logical value output by the model for the normal category. This represents the original logical value output by the model for the anomaly category. Indicates temperature parameter, Indicates the score for abnormal decisions.
[0024] Preferably, step S3 specifically includes the following steps:
[0025] S301. Introduce a meta-evaluator, which is a multilayer perceptron containing only two fully connected layers. This meta-evaluator receives a multimodal feature set. As input, output the expected score;
[0026] S302, For multimodal feature sets Apply image enhancement techniques to derive and obtain Feature representation from different perspectives ,in Indicates the first Feature representation from each perspective, Each viewpoint is fed into the anomaly assessment agent, generating anomaly decision scores for each perspective. The meta-evaluator then calculates the expected score of its own output. The sum of the differences between the scores of the anomalous decisions yields the self-consistency loss, which is specifically:
[0027]
[0028] in, Indicates a loss of self-consistency. Indicates the first Anomaly decision scores from an enhanced perspective Indicates the expected score. Indicates the serial number. This indicates the total number of enhanced perspectives.
[0029] Preferably, step S3 further includes the following steps:
[0030] S303. After obtaining the self-consistency loss, compare it with a pre-set threshold. If the self-consistency loss is less than or equal to the threshold, it means that the current model's judgment has reached a self-consistent state, and the loop terminates early. If the self-consistency loss is greater than the threshold, it means that the robustness of the decision generated based on the prompt vector has not reached the expected effect, and the re-prompt mechanism is triggered. After the defect description agent calculates the gradient of the self-consistency loss with respect to the current prompt vector, it updates the prompt vector according to the gradient descent rule.
[0031] S304. After completing the cue vector update, use the cue vector to drive the anomaly evaluation agent again, re-evaluate the anomaly of the original multi-view feature set, generate a new set of anomaly scores, recalculate the self-consistency loss, and compare it with the threshold.
[0032] S305. Continue iteratively until one of the two termination conditions is met, then end the loop operation and output the prompt vector after multiple rounds of optimization and the corresponding anomaly score. The two termination conditions are that the self-consistency loss is less than or equal to the threshold and the number of iterations reaches the preset maximum allowed number of iterations.
[0033] Preferably, step S4 specifically includes the following steps:
[0034] S401. Construct the optimal parameters for this detection using the cue vectors optimized through multiple rounds and the corresponding anomaly scores. Call the predefined word combination function, which takes the aligned multimodal feature set and anomaly scores as input and outputs a natural language explanatory text.
[0035] S402. The internal working mechanism of this combination function integrates three information sources. The first information source extracts scene background information from the text description output by the image annotation agent. The second information source converts the final abnormality score in numerical form into easily understandable qualitative degree adverbs according to a predefined mapping table. The third information source decodes the converged cue vector from the continuous vector space back into readable text fragments to describe the specific characteristics of the defect.
[0036] Preferably, step S4 further includes the following steps:
[0037] S403. Combine the three information sources according to the preset template to generate a complete natural language explanation text, encapsulate all output information to form the final detection result window, which includes the prompt vector, anomaly score and user-oriented explanation text.
[0038] Compared with the prior art, the beneficial effects of the present invention are:
[0039] 1. This invention effectively integrates data features from heterogeneous sensors through a cross-modal attention mechanism, significantly improving detection robustness in complex environments. At the same time, it achieves automatic conversion from visual features to text descriptions by leveraging a multi-agent collaborative framework dynamically generated by a large language model. Combined with a dynamic prompting mechanism and iterative self-consistency verification, it avoids the bias and omissions of manually designed prompts, greatly improving the accuracy of anomaly detection and the ability to generalize to unknown defects.
[0040] 2. This invention automatically generates customized query prompts based on image content and iteratively optimizes them through gradient updates. When the detection scenario or product line changes, the method can quickly adapt on its own, significantly reducing maintenance costs. Furthermore, the output natural language explanation text combines the detection results, confidence scores, and specific defect features, enabling users to clearly understand the model's judgment basis and enhancing the transparency and credibility of the artificial intelligence system. Attached Figure Description
[0041] Figure 1 An overall method flowchart is provided for embodiments of the present invention. Detailed Implementation
[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] Example 1:
[0044] Please see Figure 1 This invention provides a technical solution: a visual multimodal image detection method for AI training, comprising the following steps:
[0045] S1. After obtaining the original multimodal data tensor, use a convolutional network to extract features, align them spatially and through channels, compensate for geometric deviations, and finally output the aligned multimodal feature set.
[0046] S2. Input the multimodal feature set into the multi-agent framework generated by the large language model. The image annotation agent generates scene text description, the defect description agent dynamically generates prompt vectors, and the anomaly evaluation agent outputs anomaly scores to quantify the defect confidence.
[0047] S3. Calculate the consistency loss through the meta-evaluator. If the loss exceeds the threshold, the defect description agent calculates the gradient and updates the prompt vector to drive the anomaly evaluation agent to evaluate the weight of the anomaly evaluation agent. Iterate and optimize until the termination condition is met.
[0048] S4. Obtain the cue vector and anomaly score, and fuse the scene description, qualitative words of score mapping and defect features of cue decoding through a word combination function to generate natural language explanation text, and output a detection window containing cue, score and explanation.
[0049] S1 specifically includes the following steps:
[0050] S101, at discrete time step The original multimodal data sequence was obtained. ,in , Indicates the first The raw data for each modality, express Corresponding image height, express The corresponding width, express The corresponding number of channels, Represent the field of real numbers, for the th Modality Deep features are extracted using a convolutional backbone network that matches the modality type.
[0051] S102, change the original mode Mapping to a high-dimensional feature space, the output is the original feature map. ,in , This represents the spatial height obtained after convolutional downsampling. This represents the spatial width obtained after convolutional downsampling. This represents a predefined number of feature channels that remains consistent across all modalities;
[0052] S103. Repeat the feature extraction operation for each modality in the sequence to obtain a set of original feature maps. ,in Indicates the first The feature map of the i-th modality is selected. Feature map of each modality and the Feature map of each modality Then, two learnable projection matrices are determined, one for querying the projection matrix and the other for learning the projection matrix. Bond projection matrix ,use To each Perform a linear transformation to obtain and ,pass and Calculate the similarity score matrix and count the feature dimensions after projection. Applying the softmax function, based on the similarity score matrix and Analyze the attention weight matrix ,in The attention weight matrix is calculated by repeating the process. ;
[0053] S104. After performing cross-modal attention alignment on all modal pairs, for each original feature map Generate aligned features ,in , Indicates the sequence number, and all aligned features After aggregation, an aligned multimodal feature set is constructed. ;
[0054] S2 specifically includes the following steps:
[0055] S201, Set up multimodal features The data is fed into a multi-agent collaborative framework generated by instantiating a large language model. This framework leverages the instruction following and context learning capabilities of the large language model to instantly generate three agents with distinct roles: an image annotation agent, a defect description agent, and an anomaly assessment agent. These agents then collaboratively process the input data in a pipeline manner.
[0056] S202, Receive multimodal feature set through image annotation agent. As the sole input, the internal vision-to-language generation function decodes word by word in an autoregressive manner to output a text description.
[0057] S2 also includes the following steps:
[0058] S203. Using a defect description agent, the agent takes the multimodal feature set and the text description output by the image annotation agent as joint input, and dynamically constructs a cue vector through a generator function.
[0059] S204. Using an anomaly assessment agent, the cue vector generated by the defect description agent is jointly input with the multimodal feature set. The cue vector is used as a condition key and deeply integrated with the multimodal features through feature affine transformation to obtain the fused features. A conditional probability model is then applied to analyze these features, thereby outputting a scalar score. Specifically, the scalar score is:
[0060]
[0061] in, This represents the original logical value output by the model for the normal category. This represents the original logical value output by the model for the anomaly category. Indicates temperature parameter, Indicates the score of abnormal decision;
[0062] S3 specifically includes the following steps:
[0063] S301. Introduce a meta-evaluator, which is a multilayer perceptron containing only two fully connected layers. This meta-evaluator receives a multimodal feature set. As input, output the expected score;
[0064] The meta-evaluator is a simple, small-scale validation neural network. It employs a multilayer perceptron architecture with only two fully connected layers. The specific layer connections are as follows: The first layer is the input layer, with the number of neurons equal to the dimension of the multimodal feature set D, used to receive the aligned multimodal feature set; the first and second layers are connected via a fully connected layer, and the ReLU activation function is used to perform a non-linear transformation on the intermediate features; the second layer is the hidden layer, with the number of neurons preset to one-quarter of the dimension of the input layer; the third layer is the output layer, fully connected to the hidden layer, without an activation function, directly outputting a scalar real number as the expected score.
[0065] The input to the meta-evaluator is a multimodal feature set D, which is obtained after feature extraction and attention fusion of the original multimodal data. Its physical meaning is a high-dimensional feature representation that integrates information from multiple sensors. The output of the meta-evaluator is the expected score, which serves as a reference benchmark and is used to compare with the anomalous decision scores output by the anomalous evaluation agent under each augmentation perspective. The expected score is the mapping result obtained after the input features undergo two layers of linear transformation and nonlinear activation. This output value itself is not directly used as the final detection conclusion, but is used to participate in the calculation of self-consistency loss, thereby evaluating whether the current decision is stable.
[0066] During the training phase, the training steps of the meta-evaluator are as follows: a training dataset containing normal and abnormal samples is collected, and the corresponding feature set D is extracted for each training sample. This feature set D is then input into the meta-evaluator to obtain the expected score of the output. The true label corresponding to the training sample (0 for normal and 1 for abnormal) is used as a supervision signal, and the mean squared error loss between the output and the true label is calculated. Finally, the weights and bias parameters of each layer of the meta-evaluator are updated using the backpropagation algorithm with stochastic gradient descent. The steps are repeated until the loss function converges. Through training, the meta-evaluator acquires the ability to map input features to expected scores.
[0067] S302, For multimodal feature sets Apply image enhancement techniques to derive and obtain Feature representation from different perspectives ,in Indicates the first Feature representation from each perspective, Each viewpoint is fed into the anomaly assessment agent, generating anomaly decision scores for each perspective. The meta-evaluator then calculates the expected score of its own output. The sum of the differences between the scores of the anomalous decisions yields the consistency loss, which is specifically:
[0068]
[0069] in, Indicates a loss of self-consistency. Indicates the first Anomaly decision scores from an enhanced perspective Indicates the expected score. Indicates the serial number. Indicates the total number of enhanced perspectives;
[0070] S3 also includes the following steps:
[0071] S303. After obtaining the self-consistency loss, compare it with a pre-set threshold. If the self-consistency loss is less than or equal to the threshold, it means that the current model's judgment has reached a self-consistent state, and the loop terminates early. If the self-consistency loss is greater than the threshold, it means that the robustness of the decision generated based on the prompt vector has not reached the expected effect, and the re-prompt mechanism is triggered. After the defect description agent calculates the gradient of the self-consistency loss with respect to the current prompt vector, it updates the prompt vector according to the gradient descent rule.
[0072] S304. After completing the cue vector update, use the cue vector to drive the anomaly evaluation agent again, re-evaluate the anomaly of the original multi-view feature set, generate a new set of anomaly scores, recalculate the self-consistency loss, and compare it with the threshold.
[0073] S305. Continue iteratively until one of the two termination conditions is met, then end the loop operation and output the prompt vector after multiple rounds of optimization and the corresponding anomaly score. The two termination conditions are that the self-consistency loss is less than or equal to the threshold and the number of iterations reaches the preset maximum allowed number of iterations.
[0074] S4 specifically includes the following steps:
[0075] S401. Construct the optimal parameters for this detection using the cue vectors optimized through multiple rounds and the corresponding anomaly scores. Call the predefined word combination function, which takes the aligned multimodal feature set and anomaly scores as input and outputs a natural language explanatory text.
[0076] S402. The internal working mechanism of this combination function integrates three information sources. The first information source extracts scene background information from the text description output by the image annotation agent. The second information source converts the final abnormal score in numerical form into easily understandable qualitative degree adverbs according to a predefined mapping table. The third information source decodes the converged cue vector from the continuous vector space back into readable text fragments to describe the specific characteristics of the defect.
[0077] S4 also includes the following steps:
[0078] S403. Combine the three information sources according to the preset template to generate a complete natural language explanation text, encapsulate all output information to form the final detection result window, which includes the prompt vector, anomaly score and user-oriented explanation text.
[0079] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A visual multimodal image detection method for AI training, characterized in that, The method includes the following steps: S1. After obtaining the original multimodal data tensor, use a convolutional network to extract features, align them spatially and through channels, compensate for geometric deviations, and finally output the aligned multimodal feature set. S1 specifically includes the following steps: S101, at discrete time step The original multimodal data sequence was obtained. ,in , Indicates the first The raw data for each modality, express Corresponding image height, express The corresponding width, express The corresponding number of channels, Represent the field of real numbers, for the th Modality Deep features are extracted using a convolutional backbone network that matches the modality type. S102, change the original mode Mapping to a high-dimensional feature space, the output is the original feature map. ,in , This represents the spatial height obtained after convolutional downsampling. This represents the spatial width obtained after convolutional downsampling. This represents a predefined number of feature channels that remains consistent across all modalities; S103. Repeat the feature extraction operation for each modality in the sequence to obtain a set of original feature maps. ,in Indicates the first The feature map of the i-th modality is selected. Feature map of each modality and the Feature map of each modality Then, two learnable projection matrices are determined, one for querying the projection matrix and the other for learning the projection matrix. Bond projection matrix ,use To each Perform a linear transformation to obtain and ,pass and Calculate the similarity score matrix and count the feature dimensions after projection. Applying the softmax function, based on the similarity score matrix and Analyze the attention weight matrix ,in The attention weight matrix is calculated by repeating the process. ; S104. After performing cross-modal attention alignment on all modal pairs, for each original feature map Generate aligned features ,in , Indicates the sequence number, and all aligned features After aggregation, an aligned multimodal feature set is constructed. ; S2. Input the multimodal feature set into the multi-agent framework generated by the large language model. The image annotation agent generates scene text description, the defect description agent dynamically generates prompt vectors, and the anomaly evaluation agent outputs anomaly scores to quantify the defect confidence. S3. Calculate the consistency loss through the meta-evaluator. If the loss exceeds the threshold, the defect description agent calculates the gradient and updates the prompt vector to drive the anomaly evaluation agent to evaluate the weight of the anomaly evaluation agent. Iterate and optimize until the termination condition is met. S4. Obtain the cue vector and anomaly score, and fuse the scene description, qualitative words of score mapping and defect features of cue decoding through a word combination function to generate natural language explanation text, and output a detection window containing cue, score and explanation.
2. The visual multimodal image detection method for AI training according to claim 1, characterized in that, S2 specifically includes the following steps: S201, Set up multimodal features The data is fed into a multi-agent collaborative framework generated by instantiating a large language model. This framework leverages the instruction following and context learning capabilities of the large language model to instantly generate three agents with distinct roles: an image annotation agent, a defect description agent, and an anomaly assessment agent. These agents then collaboratively process the input data in a pipeline manner. S202, Receive multimodal feature set through image annotation agent. As the sole input, the internal vision-to-language generation function decodes word by word in an autoregressive manner to output a text description.
3. The visual multimodal image detection method for AI training according to claim 2, characterized in that, S2 further includes the following steps: S203. Using a defect description agent, the agent takes the multimodal feature set and the text description output by the image annotation agent as joint input, and dynamically constructs a cue vector through a generator function. S204. Using an anomaly assessment agent, the cue vector generated by the defect description agent is jointly input with the multimodal feature set. The cue vector is used as a condition key and deeply integrated with the multimodal features through feature affine transformation to obtain the fused features. A conditional probability model is applied to analyze these features, thereby outputting a scalar score. Specifically, the scalar score is: in, This represents the original logical value output by the model for the normal category. This represents the original logical value output by the model for the anomaly category. Indicates temperature parameter, Indicates the score for abnormal decisions.
4. The visual multimodal image detection method for AI training according to claim 1, characterized in that, S3 specifically includes the following steps: S301. Introduce a meta-evaluator, which is a multilayer perceptron containing only two fully connected layers. This meta-evaluator receives a multimodal feature set. As input, output the expected score; S302, For multimodal feature sets Apply image enhancement techniques to derive and obtain Feature representation from different perspectives ,in Indicates the first Feature representation from each perspective, Each viewpoint is fed into the anomaly assessment agent, generating anomaly decision scores for each perspective. The meta-evaluator then calculates the expected score of its own output. The sum of the differences between the scores of the anomalous decisions yields the self-consistency loss, which is specifically: in, Indicates a loss of self-consistency. Indicates the first Anomaly decision scores from an enhanced perspective Indicates the expected score. Indicates the serial number. This indicates the total number of enhanced perspectives.
5. The visual multimodal image detection method for AI training according to claim 4, characterized in that, S3 specifically includes the following steps: S303. After obtaining the self-consistency loss, compare it with a pre-set threshold. If the self-consistency loss is less than or equal to the threshold, it means that the current model's judgment has reached a self-consistent state, and the loop terminates early. If the self-consistency loss is greater than the threshold, it means that the robustness of the decision generated based on the prompt vector has not reached the expected effect, and the re-prompt mechanism is triggered. After the defect description agent calculates the gradient of the self-consistency loss with respect to the current prompt vector, it updates the prompt vector according to the gradient descent rule. S304. After completing the cue vector update, use the cue vector to drive the anomaly evaluation agent again, re-evaluate the anomaly of the original multi-view feature set, generate a new set of anomaly scores, recalculate the self-consistency loss, and compare it with the threshold. S305. Continue iteratively until one of the two termination conditions is met, then end the loop operation and output the prompt vector after multiple rounds of optimization and the corresponding anomaly score. The two termination conditions are that the self-consistency loss is less than or equal to the threshold and the number of iterations reaches the preset maximum allowed number of iterations.
6. The visual multimodal image detection method for AI training according to claim 1, characterized in that, S4 specifically includes the following steps: S401. Construct the optimal parameters for this detection using the cue vectors optimized through multiple rounds and the corresponding anomaly scores. Call the predefined word combination function, which takes the aligned multimodal feature set and anomaly scores as input and outputs a natural language explanatory text. S402. The internal working mechanism of this combination function integrates three information sources. The first information source extracts scene background information from the text description output by the image annotation agent. The second information source converts the final abnormality score in numerical form into easily understandable qualitative degree adverbs according to a predefined mapping table. The third information source decodes the converged cue vector from the continuous vector space back into readable text fragments to describe the specific characteristics of the defect.
7. The visual multimodal image detection method for AI training according to claim 6, characterized in that, S4 specifically includes the following steps: S403. Combine the three information sources according to the preset template to generate a complete natural language explanation text, encapsulate all output information to form the final detection result window, which includes the prompt vector, anomaly score and user-oriented explanation text.