Image detection method based on criterion perception, electronic device and storage medium

Abstract

The application discloses a criterion perception-based image detection method, an electronic device and a storage medium. The method comprises the following steps: obtaining to-be-detected task information; wherein the to-be-detected task information comprises to-be-detected visual data, task description information and a reference support set associated with the to-be-detected task; the reference support set comprises at least one reference example visual data and a detection result label corresponding to the reference example visual data; inputting the to-be-detected task information into a fine-tuned multi-modal large model; perceiving a detection criterion used by the reference support set according to an indication of the task description information by using the fine-tuned multi-modal large model; and detecting the to-be-detected visual data according to the detection criterion, and outputting a detection result.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to a criterion-based image detection method, electronic device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, image-based automated inspection technology is being applied more and more widely, for example in business scenarios such as defect detection in industrial products, lesion identification in medical images, and abnormal behavior recognition in security videos. At the same time, advancements in large-scale model technology have enabled multimodal large models to simultaneously understand image content and text descriptions, providing stronger technical support for visual inspection tasks.

[0003] In related technologies, when multimodal large models perform visual inspection tasks, users only need to input the image or video to be inspected into the model, and the model can directly output the detection result, such as "qualified" or "unqualified", "normal" or "abnormal". The model's judgment is mainly based on the fixed knowledge and discrimination boundaries learned from massive amounts of data during its pre-training stage.

[0004] However, in actual business operations, the following situation often occurs: the detection task itself remains unchanged, but the stringency of the judgment varies depending on the application scenario, risk appetite, or industry standards. For example, in industrial defect detection, high-end exterior parts have "zero tolerance" for minor scratches, while internal structural parts only focus on severe cracks. Related technologies use fixed discrimination boundaries and cannot adjust the judgment scale according to the dynamic needs of the business scenario. This results in the model struggling to provide accurate detection results when faced with critical images that require different conclusions based on different standards. Summary of the Invention

[0005] This application provides a criterion-based image detection method, electronic device, and storage medium that can adaptively adjust the discrimination boundary according to the reference support set-based detection criteria, thereby improving the detection accuracy of critical images.

[0006] According to a first aspect of this application, a criterion-based image detection method is disclosed, the method comprising: Obtain information about the task to be detected; wherein, the information about the task to be detected includes: visual data to be detected, task description information, and a reference support set associated with the task to be detected; the reference support set contains at least one reference example visual data and the detection result label corresponding to the reference example visual data; The task information to be detected is input into the fine-tuned multimodal large model. The fine-tuned multimodal large model perceives the detection criteria used by the reference support set according to the instructions of the task description information, and detects the visual data to be detected according to the detection criteria, and outputs the detection results.

[0007] According to a second aspect of this application, an electronic device is disclosed, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the criterion-based image detection method as described in the first aspect.

[0008] According to a third aspect of this application, a computer-readable storage medium is disclosed having a computer program / instructions stored thereon, which, when executed by a processor, implements the criterion-based image detection method as described in the first aspect.

[0009] According to a fourth aspect of this application, a computer program product is disclosed, comprising a computer program / instructions that, when executed by a processor, implement the criterion-based image detection method as described in the first aspect.

[0010] In this embodiment, information about the task to be detected is obtained; wherein, the information about the task to be detected includes: visual data to be detected, task description information, and a reference support set associated with the task to be detected; the reference support set contains at least one reference example visual data and a detection result label corresponding to the reference example visual data; the information about the task to be detected is input into a fine-tuning multimodal large model, and the fine-tuning multimodal large model perceives the detection criteria used by the reference support set according to the instructions of the task description information, and detects the visual data to be detected according to the detection criteria, and outputs the detection result.

[0011] As can be seen, in this embodiment, by acquiring detection task information including visual data to be detected, task description information and reference support set, and inputting it into a fine-tuned multimodal large model for detection, the model can perceive the current detection criteria and adaptively adjust the discrimination boundary according to the reference support set, thereby improving the detection accuracy of critical images. At the same time, image detection can be achieved with only a small number of reference examples, reducing data annotation costs. Attached Figure Description

[0012] Figure 1 This is a flowchart of a criterion-aware image detection method provided by some embodiments of this application; Figure 2 This is one of the example diagrams of the reference support set provided by some embodiments of this application; Figure 3 This is the second example diagram of the reference support set provided by some embodiments of this application; Figure 4 This is a network structure diagram of a fine-tuned multimodal large model provided in some embodiments of this application; Figure 5 This is a flowchart of a training method for fine-tuning a multimodal large model provided in some embodiments of this application; Figure 6 This is a structural block diagram of a criterion-based image detection device provided in some embodiments of this application; Figure 7 This is a structural block diagram of an electronic device provided in some embodiments of this application. Detailed Implementation

[0013] It should be noted that the image data obtained in this application is accessed, collected, stored, and used for subsequent analysis and processing after the user or relevant data owner has been clearly informed of the content of the data collection, the purpose of the data, the processing method, etc., and with the consent and authorization of the user or relevant data owner. Furthermore, the application can provide the user or relevant data owner with the means to access, correct, or delete the data, as well as the method to revoke consent or authorization.

[0014] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0016] The criterion-based image detection method provided in this application can be applied to the following scenarios where detection criteria need to be dynamically adjusted according to business needs: industrial product defect detection, medical image lesion judgment, security video abnormal behavior recognition, and other visual inspection tasks.

[0017] In industrial product defect detection, high-end appearance parts are subject to a zero-tolerance criterion (minor scratches are considered unqualified), while internal structural parts are subject to a limited tolerance criterion (only severe damage is considered unqualified). The method provided in this application embodiment can adaptively output detection results according to the detection criteria corresponding to the product type.

[0018] In the identification of lesions in medical imaging, different medical institutions have different criteria for judging suspected lesions. For example, some institutions may adhere to the principle of "better to misdiagnose than to miss" or "only judge those with obvious features." The method provided in this application can be adapted to different judgment strategies. In the identification of abnormal behavior in security videos, different locations such as schools, banks, and factories have different definitions of abnormal behavior. The method provided in this application embodiment can make corresponding judgments based on the detection criteria corresponding to the location.

[0019] Furthermore, the method provided in this application embodiment can also be applied to tasks such as agricultural pest and disease detection, remote sensing image target recognition, and traffic violation judgment, and this application embodiment does not limit it in this regard.

[0020] The method provided in this application can be applied to a variety of electronic devices, including but not limited to: mobile terminals such as smartphones, tablets, and laptops, as well as computing devices such as servers and desktop computers. This application does not limit the application to these devices.

[0021] Figure 1 This is a flowchart of a criterion-aware image detection method provided by some embodiments of this application, such as... Figure 1 As shown, the method may include the following steps: step 101 and step 102.

[0022] In step 101, the information of the task to be detected is obtained; wherein, the information of the task to be detected includes: visual data to be detected, task description information, and a reference support set associated with the task to be detected; the reference support set contains at least one reference example visual data and the detection result label corresponding to the reference example visual data.

[0023] In this embodiment of the application, the task to be detected refers to the detection task that needs to be performed on the visual data to be detected.

[0024] In some embodiments, the task to be detected is a discriminative task, which requires making explicit judgments or classifications of the visual data to be detected. The output is a discrete decision conclusion, rather than generating a continuous description or open-ended answer. For example, in industrial defect detection scenarios, a discriminative task is to "determine whether the product is qualified," with an output of "qualified" or "unqualified"; in medical imaging diagnosis scenarios, a discriminative task is to "determine whether the lesion is malignant," with an output of "benign" or "malignant"; in security scenarios, a discriminative task is to "determine whether the behavior is abnormal," with an output of "normal" or "abnormal." All of the above discriminative tasks are characterized by clear task objectives, clear judgment criteria, and binary or multi-classified output results.

[0025] In this embodiment, the discrimination task can clearly define the specific task to be performed through the task description information, and flexibly indicate the discrimination criterion currently adopted through the reference support set, thereby realizing the separation of detection task and detection criterion, enabling the same model to quickly adapt to different discrimination criteria, and improving the model's versatility and deployment efficiency in various business scenarios.

[0026] In this embodiment of the application, the visual data to be detected refers to the visual data that needs to be detected and judged, that is, the processing object of the detection task.

[0027] In some embodiments, the visual data to be detected is an image or a video. Images may include still pictures, photographs, scanned images, etc.; videos may include continuous frame sequences, recordings, etc.

[0028] For example, in an industrial defect detection scenario, the visual data to be detected can be an image of the product's appearance; in a medical imaging diagnostic scenario, the visual data to be detected can be a patient's CT image or X-ray; and in a security scenario, the visual data to be detected can be a video clip captured by a camera.

[0029] In this embodiment, the visual data to be detected can be either a static image or a dynamic video, making the method applicable to both static image detection and dynamic video analysis scenarios.

[0030] In this embodiment, task description information refers to natural language instructions used to describe the detection task to be performed, indicating the detection task to be performed on the visual data to be detected. Through task description information, subsequent fine-tuning of the multimodal large model can clearly define the specific task that needs to be performed.

[0031] For example, the task description information could be "Please determine whether there is a defect in the image and output whether it is qualified or unqualified". In this case, the detection task is defect detection. The task description information could also be "Please determine whether the lesion is malignant". In this case, the detection task is lesion identification. The task description information could also be "Please determine whether the behavior is abnormal". In this case, the detection task is abnormal behavior identification.

[0032] In this embodiment, the reference support set refers to a set of example data used to express the current detection criteria. The reference support set contains N reference example visual data and the detection result label corresponding to each reference example visual data, where N is a positive integer. The value of N can be set according to the actual application scenario. The smaller N is, the higher the computational efficiency; the larger N is, the more fully the criteria are expressed. This embodiment does not limit the specific value of N.

[0033] In this embodiment of the application, reference example visual data refers to image or video data that serves as a reference example.

[0034] In this embodiment, the detection result label refers to the conclusion marker drawn after judging the visual data of the reference example. The specific content of the detection result label corresponds to the detection task. For example, in a defect detection task, the detection result label can be "qualified" or "unqualified"; in a lesion judgment task, the detection result label can be "benign" or "malignant"; in an abnormal behavior recognition task, the detection result label can be "normal" or "abnormal".

[0035] In this embodiment, the detection criteria refer to the judgment boundary or stringency used when judging the visual data to be inspected. Different business scenarios, risk preferences, or industry standards correspond to different detection criteria. For example, in defect detection, under the zero-tolerance criterion, even a slight scratch is judged as unqualified, while under the limited tolerance criterion, only serious damage is judged as unqualified.

[0036] In this embodiment, criterion perception refers to the ability of fine-tuning a multimodal large model to infer the currently used detection criterion by analyzing the correspondence between visual data of reference examples in the reference support set and the labels of detection results. Fine-tuning the multimodal large model does not rely on fixed discrimination boundaries formed during the pre-training phase, but rather flexibly perceives and adapts to different detection criteria through label patterns in the reference support set.

[0037] In this embodiment, the reference support set implicitly demonstrates, through the correspondence between reference example visual data and detection result labels, "what kind of visual data corresponds to what kind of detection conclusion" under the current detection criteria. This allows the fine-tuning multimodal large model to perceive the detection criteria corresponding to the visual data to be detected, without having to directly describe the criteria content in natural language. By analyzing this correspondence, the fine-tuning multimodal large model can infer the discrimination boundary. For example, it can infer that a zero-tolerance criterion is currently used from "minor scratch → unqualified"; and infer that a limited-tolerance criterion is currently used from "minor scratch → qualified".

[0038] For example, such as Figure 2 As shown, reference support set 1 expresses the zero tolerance criterion, which includes: a flawless image and its inspection result labeled "pass", a severe crack image and its inspection result labeled "fail", and a minor scratch image and its inspection result labeled "fail".

[0039] For example, such as Figure 3 As shown, reference support set 2 expresses the limited tolerance criterion, which includes: with Figure 2 The same images with minor scratches and their test results are labeled "qualified", the same images without defects and their test results are labeled "qualified", and the same images with severe cracks and their test results are labeled "unqualified".

[0040] In this embodiment of the application, by acquiring three elements—the visual data to be detected, the task description information, and the reference support set—the detection object, the task indication, and the criterion expression are separated, providing the model with complete input information. This enables the model to clearly define "what to do" (indicated by the visual data to be detected), "what task to perform" (indicated by the task description information), and "what standard to follow" (indicated by the reference support set).

[0041] In this embodiment of the application, the information of the task to be detected can be obtained in a variety of ways.

[0042] In some embodiments, the electronic device provides a human-computer interaction interface (such as a dialog box), through which users can upload visual data to be detected and the corresponding reference support set, and input task description information. For example, a user uploads an image to be detected and its corresponding reference support set (including example images and detection result labels) in a conversational interface, and inputs "Please determine whether there is a defect in the image" as task description information. The electronic device combines the above information into the task information to be detected.

[0043] In some embodiments, the electronic device is communicatively connected to a task management system. The task management system stores a task queue, which contains information on one or more tasks to be executed. Each task includes corresponding visual data to be inspected, task description information, and a reference support set. The electronic device reads each task from the task queue sequentially and automatically executes the inspection task. This method is suitable for automated inspection scenarios such as online quality inspection on production lines and batch interpretation of hospital images.

[0044] The embodiments of this application do not limit the specific method of obtaining the information of the detection task.

[0045] In step 102, the task information to be detected is input into the fine-tuned multimodal large model. The fine-tuned multimodal large model perceives the detection criteria used by the reference support set according to the instructions of the task description information, and performs detection on the visual data to be detected according to the detection criteria, and outputs the detection results.

[0046] In this embodiment, the fine-tuned multimodal large model refers to a multimodal large model further optimized through multi-criteria training based on the basic multimodal large model. This model can simultaneously process visual data such as images and videos, as well as natural language text, and has the ability to understand visual content and generate corresponding text descriptions. The basic multimodal large model is obtained by learning from massive amounts of image and text data during the pre-training phase, while the fine-tuned multimodal large model is obtained by performing multi-criteria supervised training on the basic multimodal large model using a small amount of training data (e.g., hundreds to thousands of sets of training data), enabling it to perceive detection criteria based on a reference support set.

[0047] For example, such as Figure 4 As shown, the fine-tuned multimodal large model 400 may include: a visual encoder 401, a projection layer 402, and a language model 403.

[0048] A visual encoder is used to extract visual features from visual data to be detected. Specifically, the visual encoder receives input visual data such as images or videos, extracts features from the visual data through a multi-layer neural network, and outputs the corresponding visual feature vectors. The visual encoder can adopt a visual model based on the Transformer architecture (such as ViT) or a visual model based on a convolutional neural network (CNN) (such as ResNet), and this application embodiment does not limit this.

[0049] The projection layer is used to map the visual features output by the visual encoder to the embedding space of the language model. Specifically, the projection layer receives the visual feature vectors output by the visual encoder and transforms them into feature vectors that match the embedding dimension of the language model through a multilayer perceptron (MLP) or a linear transformation layer, so that the visual features can be aligned and interact with the text features in the same semantic space.

[0050] The language model is used to generate corresponding text output based on the input visual features and text information. Specifically, the language model receives visual features output from the projection layer, as well as text data such as task description information input by the user and label information from the reference support set, and generates detection results word by word through an autoregressive approach. The language model can be a large language model based on the Transformer architecture (such as GPT, LLaMA), and this embodiment of the application does not limit this.

[0051] In this embodiment of the application, during the inference process, the visual data to be detected is first input into the visual encoder, which extracts its visual features; the visual features are then mapped by the projection layer and input into the language model; at the same time, the task description information and the example data and label information in the reference support set are also input into the language model in text form; the language model integrates the visual features and text information, processes them, and outputs the detection result of the visual data to be detected.

[0052] As can be seen, in this embodiment, through the three-part structure of the visual encoder, projection layer, and language model, the fine-tuned multimodal large model can simultaneously process visual and textual data, achieving cross-modal understanding and generation. Specifically, the visual encoder is responsible for understanding image content, the language model is responsible for understanding task instructions and criterion information and generating judgment results, and the projection layer acts as a bridge to effectively connect the two. This structure provides the basic architectural support for criterion-aware reasoning and multi-criterion training.

[0053] For example, the visual data to be detected is a picture of wood with slight scratches; the task description is "Please determine whether there are defects in the picture and output 'qualified' or 'unqualified'"; the reference support set 1 contains pictures with slight scratches and their corresponding detection result labels "unqualified". The model currently adopts a zero-tolerance criterion and outputs the detection result as "unqualified".

[0054] For example, the visual data to be detected is the same image of wood with slight scratches; the task description information is the same; reference support set 2 contains the same image with slight scratches and its corresponding detection result label "pass". The model perceives that it currently adopts a limited tolerance criterion and outputs the detection result as "pass".

[0055] It should be noted that during inference, the model parameters of the fine-tuned multimodal large model remain fixed and are not updated. That is, the model perceives the current detection criterion only by referring to the correspondence between example visual data in the support set and the detection result labels, and outputs the detection result according to the perceived criterion, without adjusting its own parameters according to the current input. This allows this method to quickly adapt to different detection criteria with a small number of reference examples without retraining, without requiring a large amount of labeled data, and is suitable for scenarios with few samples.

[0056] As can be seen from the above embodiments, in this embodiment, by acquiring detection task information including visual data to be detected, task description information and reference support set, and inputting it into a fine-tuned multimodal large model for detection, the model can perceive the current detection criteria and adaptively adjust the discrimination boundary according to the reference support set, thereby improving the detection accuracy of critical images. At the same time, image detection can be achieved with only a small number of reference examples, reducing the data annotation cost.

[0057] In some embodiments provided in this application, the detection result labels corresponding to the same reference example visual data are different under different detection criteria.

[0058] In this embodiment, the reference example visual data included in the reference support set is fixed, but the detection result labels corresponding to the reference example visual data can change depending on the detection criteria used. That is, the same reference example visual data can be labeled with different detection result labels under different detection criteria.

[0059] For example, using an image of wood with minor scratches as reference visual data, when applying the zero-tolerance criterion, the detection result label for this slightly scratched image is "unacceptable"; while when applying the limited tolerance criterion, the detection result label for the same slightly scratched image is "acceptable". It is evident that the reference visual data itself remains unchanged, but its detection result label changes with the detection criterion.

[0060] In this embodiment, the reference support set implicitly expresses the currently used detection criteria to the fine-tuning multimodal large model through the correspondence between "same image, different labels". After receiving the reference support set, the fine-tuning multimodal large model can infer the currently used detection criteria based on the correspondence between the reference example visual data and the detection result labels: when a slightly scratched image is labeled "unacceptable", the model infers that a stricter detection criterion is currently used; when the same image is labeled "acceptable", the model infers that a more lenient detection criterion is currently used.

[0061] As can be seen, in the embodiments of this application, the fine-tuned multimodal large model no longer relies on the fixed discrimination boundary formed in the pre-training stage, but can flexibly perceive and adapt to different detection criteria according to the label patterns in the reference support set, thereby solving the technical problem in related technologies that the model is difficult to accurately detect when facing critical images that need to make different conclusions according to different standards.

[0062] In some embodiments provided in this application, the reference example visual data and the visual data to be detected belong to the same category; wherein, the category is used to indicate the object type to which the detected object belongs.

[0063] In this embodiment, the category refers to the object type or application scope to which the object being detected belongs, such as object types like "wood," "metal," and "glass," or application scopes like "lung CT imaging diagnosis" and "security scenarios." Categories are used to distinguish different objects or scenarios being detected, and the classification is based on the type of the object itself or the application scope to which the detection task belongs. For example, a "mug" belongs to ceramic / glass products, while a "stainless steel mug" belongs to metal products; they belong to different categories. Similarly, "lung CT imaging diagnosis" and "wood defect detection" belong to different application scopes and are also different categories.

[0064] In this embodiment, visual data of the same category have similar feature distributions. For example, images of wood products all contain common visual features such as wood grain and texture; lung CT images all contain common medical features such as lung tissue and nodules. Therefore, when the reference example visual data and the visual data to be detected belong to the same category, the label information of the reference example visual data has high reference value for judging the visual data to be detected. Conversely, if the reference example visual data and the visual data to be detected belong to different categories, such as using lung CT images as a reference example for wood defect detection, due to the huge difference in feature distribution between the two, the label information of the reference example may not provide an effective reference for judging the visual data to be detected, and may even mislead the model.

[0065] As can be seen, in this embodiment of the application, by limiting the reference example visual data in the reference support set to belong to the same category as the visual data to be detected, the validity and relevance of the reference information are ensured, the judgment error caused by cross-category reference is avoided, and the accuracy and reliability of fine-tuning the multimodal large model in the detection task are improved.

[0066] In some embodiments provided in this application, any category includes at least one subcategory, and the subcategory information is used to represent the specific state type under the category; the detection criteria are used to characterize the tolerance of the task to be detected to whether there are anomalies in the subcategory information.

[0067] In this embodiment, the category refers to the object type to which the detected object belongs, such as "wood," "metal," or "glass." A subcategory refers to a more specific state type within a category, used to represent different states or defect types that may occur within that category. For example, under the category of "wood," subcategories may include specific state types such as "scratches," "cracks," and "dents"; under the category of "metal," subcategories may include specific state types such as "scratches," "rust," and "deformation."

[0068] In this embodiment, the detection criteria characterize the tolerance of the task to be detected for anomalies in each subcategory. That is, under different detection criteria, the detection results for the same subcategory can differ. For example, for the "scratches" subcategory under the "wood" category, under the zero-tolerance criterion, "scratches" are judged as "unacceptable" (abnormal); under the limited tolerance criterion, "scratches" are judged as "acceptable" (normal). Thus, the subcategory describes "what state exists," and the detection criteria describe "whether this state is tolerated," both jointly determining the final detection result label.

[0069] For example, taking the "timber" category as an example, the test result labels for different subcategories under different testing criteria are shown in Table 1 below.

[0070]

[0071] Table 1 As can be seen from Table 1, the detection result labels for the same subcategory may be different under different detection criteria, depending on the tolerance of the detection criteria for the state type represented by the subcategory.

[0072] As can be seen, in this embodiment of the application, by dividing the category into subcategories and defining the tolerance of the detection criteria for the existence of anomalies in the subcategories, the mapping relationship between the detection criteria and the subcategories is realized, which provides a foundation for multi-criteria training and criterion awareness.

[0073] In some embodiments provided in this application, step 102 may specifically include the following steps: step 1021.

[0074] In step 1021, the multimodal large model is fine-tuned to perceive the detection criteria used by the reference support set based on the correspondence between the reference example visual data and the detection result labels; wherein, the fine-tuned multimodal large model is trained on training data containing multiple detection criterion support sets and has the ability to perceive detection criteria.

[0075] In this embodiment, the fine-tuned multimodal large model possesses the ability to perceive detection criteria, which originates from the model's multi-criteria training. Specifically, the fine-tuned multimodal large model is obtained by performing multi-criteria supervised training on training data containing support sets for multiple detection criteria, based on the basic multimodal large model. The training data includes different detection result labels corresponding to the same visual data under different detection criteria, enabling the model to learn the rule that "the same visual data can correspond to different judgment conclusions under different criteria," thereby possessing the ability to perceive the current detection criterion based on the correspondence between example data and labels in the reference support set.

[0076] During the inference process, after receiving the reference support set, the fine-tuned multimodal large model perceives the detection criteria currently used by analyzing the correspondence between the reference example visual data and the detection result labels.

[0077] For example, when the reference support set contains a minor scratch image and its corresponding detection result label "unacceptable", the fine-tuned multimodal large model currently adopts a zero-tolerance criterion; when the reference support set contains the same minor scratch image and its corresponding detection result label "acceptable", the model currently adopts a limited tolerance criterion. The model detects the visual data to be detected according to the perceived detection criteria and outputs the detection results.

[0078] As can be seen, in the embodiments of this application, the fine-tuned multimodal large model has obtained the ability to perceive and detect criteria through multi-criteria training. During inference, the current criteria can be perceived by analyzing the correspondence between example data and labels in the reference support set, without the need for retraining or parameter updates.

[0079] Figure 5 This is a flowchart of a training method for fine-tuning a multimodal large model provided in some embodiments of this application. In the embodiments of this application, a multi-criteria training method is adopted, such as... Figure 5 As shown, the method may include the following steps: step 501, step 502 and step 503.

[0080] In step 501, the basic multimodal large model is obtained.

[0081] In this embodiment, the basic multimodal large model refers to a pre-trained multimodal model. This type of model learns from massive amounts of image and text data during the pre-training phase, gaining the ability to understand visual content and generate corresponding text descriptions. This embodiment uses this basic multimodal large model as the starting point for training, and through subsequent multi-criteria training, it acquires criterion-aware capabilities, ultimately obtaining a fine-tuned multimodal large model.

[0082] In this embodiment, the basic multimodal large model can receive visual data (such as images or videos) and text data as input and generate corresponding prediction results. For example, given a product image and a prompt text "Please determine if there is a defect," the basic multimodal large model can output a prediction result of "qualified" or "unqualified."

[0083] In this embodiment, a pre-trained multimodal model is used as the basic multimodal large model, eliminating the need to train from scratch. This allows full utilization of the visual understanding and language generation capabilities of existing models, reducing training costs and improving training efficiency.

[0084] In this embodiment of the application, multi-criteria training refers to a model training method in which the same training visual data is combined with at least two sample support sets expressing different detection criteria and input into a basic multimodal large model, and the model is supervised to output detection results corresponding to the detection criteria expressed by each sample support set for the training visual data.

[0085] In step 502, a training set is constructed; wherein the training set contains multiple sets of training data, each set of training data includes: a sample visual data, at least two sample support sets corresponding to the sample visual data, and detection result labels of the sample visual data under each sample support set; at least two sample support sets correspond to different detection criteria, and the sample support sets contain at least one sample example visual data and the corresponding detection result labels; all sample support sets in the same set of training data use the same sample example visual data, but the detection result labels of the sample example visual data are different.

[0086] In this embodiment, the number of training data sets can be set according to the actual application scenario and task complexity. More training data means a richer sample set for the model, generally resulting in stronger criterion perception; however, this also increases training time and computational resource consumption. In practical applications, the number of training data sets can range from several hundred to several thousand. For example, for a wood defect detection scenario, approximately 500 training data sets can be constructed; for a more complex medical imaging lesion judgment scenario, more than 1000 training data sets can be constructed. This embodiment does not limit the specific number of training data sets; it can be flexibly set according to specific needs in actual applications.

[0087] In this embodiment, multi-criteria training enables the model to learn the ability to "perceive the current detection criterion from the label patterns in the support set," rather than memorizing specific image-label mapping relationships. This ability, once learned, can be reused in different scenarios. The design of "same image, different labels"—where all sample support sets use the same sample example visual data, but the detection result labels change depending on the detection criterion—allows the model to focus only on the label changes with the criterion, without needing to relearn image features. Fixed parameters during inference ensure that the model does not need to be retrained in new scenarios; it only needs to replace the reference support set to quickly adapt to new detection criteria. The combined effect of these features allows this embodiment to enable the model to possess criterion-aware capabilities with only a small amount of training data.

[0088] Compared to traditional methods that require hundreds of thousands or even millions of labeled samples, the embodiments of this application only require hundreds to thousands of sets of training data, reducing the cost and time of data collection and labeling. This is especially suitable for actual business scenarios where labeled data is scarce or it is difficult to obtain a large amount of high-quality labeled data, thereby improving training efficiency and lowering the threshold for model deployment.

[0089] In this embodiment, sample visual data refers to image or video data used for training, i.e., the visual data that the model needs to judge. For example, in a training scenario for industrial defect detection, sample visual data could be product images with minor scratches.

[0090] In this embodiment, the sample support set is a set of example data used to express a specific detection criterion. Each training data corresponds to at least M sample support sets, and each sample support set expresses a different detection criterion, where M is an integer greater than 1. The value of M can be set according to training requirements. The larger M is, the more types of detection criteria the model encounters, and the stronger its ability to perceive changes in criteria. However, the scale and complexity of the training data will also increase accordingly.

[0091] Preferably, M can be 2 or 3, meaning each training data point corresponds to 2 or 3 different detection criteria. Taking M=2 as an example, a set of training data can simultaneously contain a support set of samples expressing the zero-tolerance criterion and a support set of samples expressing the finite-tolerance criterion, allowing the model to learn that the same visual data should output different detection results under different criteria. This application does not limit the specific value of M in its embodiments.

[0092] In this embodiment, each sample support set contains at least one sample example visual data and a corresponding detection result label. Sample example visual data refers to image or video data used as a reference example; the detection result label refers to the conclusion drawn after judging the sample example visual data. For sample visual data in the training data, the true detection result under the detection criterion corresponding to each sample support set is also represented in the form of a detection result label. That is, for the same sample visual data, different true labels correspond to different detection criteria.

[0093] In this embodiment, all sample support sets in the same training data use the same sample example visual data, but the detection result labels of the sample example visual data are different. Through this "same image, different labels" design, the model can learn the rule that the image content remains unchanged while the label changes with the detection criteria, thereby possessing criterion-awareness capabilities.

[0094] For example, taking wood defect detection as an example, a set of training data can be constructed as follows: Sample visual data: A picture of wood with slight scratches; The sample support set A (expressing the zero tolerance criterion) contains: a picture with a slight scratch and its test result labeled "unacceptable", a picture with no defects and its test result labeled "acceptable", and a picture with a severe crack and its test result labeled "unacceptable". The sample support set B (expressing the limited tolerance criterion) contains: the same minor scratch image and its label "acceptable", the same flawless image and its label "acceptable", and the same severe crack image and its label "unacceptable". Test result label: Under the zero tolerance criterion, the true label of the visual data of this sample is "unacceptable"; under the limited tolerance criterion, the true label of the visual data of this sample is "acceptable".

[0095] As can be seen, in this embodiment of the application, by constructing training data containing a multi-criteria support set, the model can be exposed to the situation that "the same visual data should have different judgment results under different detection criteria" during the training phase, thereby learning to adjust the discrimination boundary according to the criteria expressed by the support set, rather than memorizing the fixed mapping relationship between images and labels.

[0096] In step 503, each group of training data is input into the basic multimodal large model to obtain the prediction detection result. The loss value is calculated based on the prediction detection result and the detection result label of the visual data of the samples in the training data. The parameters of the basic multimodal large model are updated based on the loss value. The above training steps are repeated until the model converges to obtain the fine-tuned multimodal large model.

[0097] In this embodiment, sample visual data from the training data and a set of sample support sets are input into a basic multimodal large model. The basic multimodal large model infers the current detection criterion based on the correspondence between example visual data in the sample support sets and detection result labels, judges the sample visual data, and outputs the predicted detection result. Since each set of training data contains at least two sample support sets (corresponding to different detection criteria), it is necessary to combine the sample visual data with each sample support set and input them into the model separately to obtain the corresponding predicted detection results.

[0098] In this embodiment, the loss value is calculated based on the predicted detection result output by the model and the actual detection result labels annotated in the training data. The loss function can be a commonly used classification loss function such as cross-entropy loss, and this embodiment is not limited to this. For the same set of training data, since it contains multiple sample support sets, the model will produce multiple prediction results, and correspondingly, multiple loss values ​​will be calculated. These loss values ​​can be summed or averaged to obtain the final loss value.

[0099] In this embodiment, the parameters of the basic multimodal large model are updated using the backpropagation algorithm based on the calculated loss value. Training can be performed using a full-parameter fine-tuning method, i.e., updating all parameters of the model; or it can be performed using a parameter-efficient fine-tuning method such as Low-Rank Adaptation (LoRA), i.e., updating only some parameters to reduce training costs. This embodiment does not limit the training to this method.

[0100] In this embodiment, the forward propagation, loss calculation, and parameter update steps described above are repeated, and multiple sets of training data from the training set are input into the model multiple times for training until the model converges. The criteria for judging model convergence can be: the loss value decreases below a preset threshold, or the accuracy on the validation set no longer improves.

[0101] In this embodiment of the application, after training is completed, a fine-tuned multimodal large model with criterion awareness is obtained. This model can infer the current detection criteria based on the input reference support set during the inference stage and output the detection results accordingly.

[0102] In this embodiment of the application, through multi-criteria training, the model no longer relies on the fixed discrimination boundary formed in the pre-training stage, but learns to perceive the current criteria from the label patterns in the support set. This enables the model to adaptively adjust the discrimination boundary and output the correct detection result when facing critical images that require different conclusions based on different criteria.

[0103] As can be seen, the training method for fine-tuning a multimodal large model provided in this application constructs training data containing a multi-criteria support set and performs multi-criteria supervised training on the basic multimodal large model, enabling the model to learn to perceive detection criteria from the correspondence between example data and labels in the support set. The trained fine-tuned multimodal large model can adaptively adjust the discrimination boundary according to dynamically changing detection criteria during the inference phase, solving the problem in related technologies where models struggle to accurately detect critical images.

[0104] In some embodiments provided in this application, the basic multimodal large model includes: a visual encoder, a projection layer, and a language model. The pre-training of the basic multimodal large model may include the following stages: The first stage is the visual encoder pre-training: The visual encoder is pre-trained using massive amounts of image and text data. Specifically, images and their corresponding text descriptions are used as training data, and a contrastive learning approach is employed to train the visual encoder. A commonly used loss function is the InfoNCE contrastive loss function, which brings matching image-text pairs closer together in the feature space and keeps mismatched image-text pairs further apart. Through this stage, the visual encoder learns to understand image content and extract effective visual features.

[0105] The second stage is language model pre-training: The language model is pre-trained using massive amounts of text data. Specifically, a large-scale text corpus is used as training data, and an autoregressive language modeling task is employed for training. A commonly used loss function is Next Token Prediction Loss, which predicts the next token (such as a word or subword) based on the generated text content, enabling the model to learn to understand language patterns and semantic information.

[0106] The third stage, cross-modal alignment, connects the visual encoder and language model through a projection layer and trains them using massive amounts of image and text data for cross-modal alignment. Specifically, images and their corresponding text descriptions are used as training data, and Next Token Prediction Loss is employed for training. The model predicts the next text token based on image features and the generated text content. Through this stage, visual and text features are aligned in a unified semantic space, enabling the model to achieve image-text understanding.

[0107] The fourth stage, instruction fine-tuning, involves fine-tuning the model using dialogue-formatted image and text data. Specifically, user-inputted instructions (such as "Please determine if there are defects in the image") are input into the model along with the image, and the model is trained to generate corresponding outputs according to the instructions. A commonly used loss function is Next Token Prediction Loss. Through this stage, the model learns to understand and follow human instructions, enabling it to perform specific visual detection tasks according to the task requirements.

[0108] The fifth stage, human preference alignment, involves training the model using image and text data labeled with human preferences. Specifically, it collects human preference data for different model outputs (e.g., which answer humans prefer for the same image and the same question), and trains the model using the Direct Preference Optimization (DPO) loss function. Through this stage, the model's output better aligns with human expectations and preferences, resulting in safer, more useful, and more accurate answers.

[0109] After completing the above five stages of pre-training, the basic multimodal large model has the following capabilities: understanding the visual content of images or videos; understanding the semantics of natural language text; performing detection tasks according to human instructions; and generating text output that meets human expectations.

[0110] However, pre-trained basic multimodal large models suffer from a "rigid discrimination boundary" problem, meaning the model relies on fixed judgment criteria formed during the pre-training stage (e.g., considering "scratches" as "unacceptable") and cannot adjust the discrimination scale according to the dynamic needs of business scenarios. The multi-criteria training in the embodiments of this application can solve this problem by further fine-tuning the basic multimodal large model to enable it to have criterion-aware capabilities.

[0111] In this embodiment, through the aforementioned multi-stage training, the basic multimodal large model acquires strong visual understanding and language generation capabilities, providing a solid foundation for subsequent multi-criteria training. Compared to training from scratch, using a pre-trained basic multimodal large model can reduce training costs, improve training efficiency, and achieve better detection performance.

[0112] In some embodiments provided in this application, step 502 may specifically include the following steps: step 5021, step 5022, step 5023 and step 5024.

[0113] In step 5021, category definition information is obtained; wherein, the category is used to indicate the object type to which the detected object belongs, and any category contains at least one subcategory, and the subcategory is used to indicate the specific state type under the category.

[0114] In this embodiment, "category" refers to the object type or application scope to which the detected object belongs, such as object types like "wood," "metal," and "glass," or application scopes like "industrial product defect detection," "medical image lesion judgment," and "security video abnormal behavior recognition." Category is a higher-level concept used to distinguish different detected objects or detection scenarios. The category is unrelated to the specific shape, size, material, or other physical properties of the detected object; rather, it depends on the type of the detected object itself or the application scope to which the detection task belongs.

[0115] In this embodiment, a subcategory refers to a more specific state type within a certain category, used to represent different states or defect types that may occur within that category. A subcategory is a lower-level concept and can be mapped to different detection result labels under different detection criteria. For example, under the category of "wood," subcategories may include specific state types such as "scratches," "cracks," and "dents"; under the category of "metal," subcategories may include specific state types such as "scratches," "rust," and "deformation"; and under the category of "lung CT images," subcategories may include specific lesion types such as "nodules," "ground-glass opacities," and "consolidation."

[0116] In this embodiment, categories and subcategories form a two-level structure: categories are at the upper level, representing the object type to which the detected object belongs; subcategories are at the lower level, representing the specific state type under that category. A single category can contain multiple subcategories, with different subcategories representing different states or defect types that may occur under the same object type.

[0117] For example, taking wood defect detection as an example, the category is "wood", which means that the current detection task is for wood products; the subcategories include "scratches", "cracks" and "dents", which represent different types of defects that may occur in wood products.

[0118] In this embodiment, a two-level structure of categories and subcategories is used. Categories remain stable to represent the type of the detected object, providing a foundation for label mapping of subcategories under different detection criteria. Subcategories describe the specific states within that type, allowing different subcategories within the same category to be mapped to different detection result labels under different detection criteria. This hierarchical structure separates the detected object type from the specific state type. With the task object remaining unchanged, different detection criteria can be constructed simply by adjusting the label mapping of subcategories under different criteria, thus providing a foundation for flexibly adapting to various business scenarios.

[0119] In step 5022, labels are mapped to each subcategory according to different detection criteria to determine the corresponding detection result labels for each subcategory under different detection criteria.

[0120] In this embodiment, label mapping refers to the operation of mapping a subcategory (such as "scratch") to a specific test result label (such as "pass" or "fail"). Under different testing criteria, the same subcategory can be mapped to different test result labels.

[0121] In this embodiment of the application, the definition of the testing criteria is described in the relevant section of step 101, and will not be repeated here. For example, the zero-tolerance criterion means that there is zero tolerance for any defects, and minor defects are judged as unqualified; the limited tolerance criterion means that only serious structural damage is judged as unqualified, and minor defects are acceptable.

[0122] For example, taking the subcategories under the "timber" category as an example, the label mapping under different detection criteria is shown in Table 2 below.

[0123]

[0124] Table 2 As shown in Table 2, "cracks" are "unacceptable" under both criteria, belonging to the category of criteria-invariant samples; "scratches" and "dents" have different labels under different criteria, belonging to the category of criteria-sensitive samples. Criterion-invariant samples refer to samples whose test results labels remain unchanged under different testing criteria, while criteria-sensitive samples refer to samples whose test results labels change under different testing criteria.

[0125] In this embodiment, label mapping allows for the systematic definition of the judgment results for each sub-category under different detection criteria, providing a standardized basis for the subsequent construction of the support set and ensuring the consistency and accuracy of the training data. Simultaneously, label mapping distinguishes between criterion-invariant samples and criterion-sensitive samples, providing a foundation for model training and evaluation.

[0126] In step 5023, based on label mapping, the same sample example visual data is copied into at least two copies, and labeled according to the detection result labels under different detection criteria, to obtain at least two sample support sets; wherein each sample support set expresses a detection criterion.

[0127] In this embodiment of the application, sample example visual data refers to image or video data used as a reference example, belonging to a defined subcategory. For example, an image of wood with scratches belongs to the "scratches" subcategory under the "wood" category.

[0128] In this embodiment, the same sample example visual data is copied into at least two copies. For each copy, according to the label mapping determined in step 5022, the corresponding detection result label is labeled according to different detection criteria. For example, using an image of wood with scratches as sample example visual data, one copy is made, labeled as "unqualified" according to the zero tolerance criterion, and placed in sample support set A; the other copy is made, labeled as "qualified" according to the limited tolerance criterion, and placed in sample support set B.

[0129] In this embodiment of the application, a sample support set typically contains multiple sample example visual data and their corresponding detection result labels, which are used to comprehensively express a detection criterion.

[0130] For example, sample support set A (expressing zero tolerance criteria) may contain: a scratch image and its label "unacceptable", a flawless image and its label "acceptable", and a crack image and its label "unacceptable". Sample support set B (expressing limited tolerance criteria) may contain: the same scratch image and its label "acceptable", the same flawless image and its label "acceptable", and the same crack image and its label "unacceptable".

[0131] In this embodiment, all sample support sets in the same set of training data use the same sample example visual data, but the detection result labels of the sample example visual data are different, ensuring that the model learns the change in detection criteria rather than the change in image content.

[0132] As can be seen, in this embodiment of the application, by copying the same sample example visual data and labeling it with different labels according to different criteria, multiple support sets expressing different detection criteria can be constructed, ensuring that the differences between support sets only come from label changes (i.e. criterion changes) rather than image content changes, thereby enabling the model to focus on learning criterion perception capabilities.

[0133] In step 5024, the sample support set is associated with the corresponding sample visual data to obtain a set of training data.

[0134] In this embodiment of the application, a complete set of training data includes three parts: a sample visual data (the object to be detected), at least two sample support sets (expressing different detection criteria), and the detection result label (true label) of the sample visual data under the detection criterion corresponding to each sample support set.

[0135] In this embodiment, at least two sample support sets constructed in step 5023 are associated with a sample visual data. This sample visual data belongs to the same category as the example visual data in the sample support sets, but can be different specific images. For example, the sample visual data is a picture of wood with slight scratches (another scratched image different from the sample example visual data). Sample support set A is a set of examples expressing the zero-tolerance criterion, and sample support set B is a set of examples expressing the limited tolerance criterion. The true labels are: "unacceptable" under the zero-tolerance criterion and "acceptable" under the limited tolerance criterion.

[0136] For example, taking wood defect detection as an example, a complete set of training data is shown in Table 3 below.

[0137]

[0138] Table 3 In this embodiment, complete training data is formed by associating sample support sets with sample visual data. Each set of training data contains complete information on "object-criteria-result," enabling the model to learn that for the same object, different detection results should be output under different criteria, thus providing a standardized data format for multi-criteria training.

[0139] As can be seen, the training set construction method provided in this application constructs a training set for multi-criteria training through four steps: obtaining category definition information, performing label mapping, constructing a multi-criteria support set, and associating training data. This ensures the consistency and scalability of the training data, enabling the model to encounter situations where "the same visual data should have different judgment results under different criteria" during the training phase. As a result, the model learns to adjust the discrimination boundary according to the criteria expressed by the support set, rather than memorizing a fixed mapping relationship between images and labels.

[0140] In some embodiments provided in this application, a model evaluation method is also provided for evaluating a fine-tuned multimodal large model obtained by multi-criteria training. The method may include the following steps: step 504, step 505 and step 506.

[0141] In step 504, a test set is obtained; wherein the test set includes criterion-invariant samples and criterion-sensitive samples; wherein the criterion-invariant samples have the same detection result label under different detection criteria, and the criterion-sensitive samples have different detection result labels under different detection criteria.

[0142] In this embodiment, the test set is a dataset used to evaluate the performance of fine-tuning a multimodal large model. The test set is independent of the training set, does not participate in model training, and is only used to test the model's generalization ability and criterion-awareness.

[0143] In this embodiment, a criterion-invariant sample refers to a sample whose test result label remains unchanged under different test criteria. That is, regardless of whether a strict or lenient criterion is used, the judgment conclusion for this sample is consistent. For example, taking an image of wood with severe cracks as an example, under a zero-tolerance criterion, the test result label is "unacceptable"; under a limited-tolerance criterion, the test result label is also "unacceptable". Regardless of the criterion used, the test result for this sample is "unacceptable," therefore it belongs to the criterion-invariant sample category. The judgment conclusion of a criterion-invariant sample does not depend on the test criterion; the model should be able to correctly judge regardless of whether it understands the criterion change. Therefore, criterion-invariant samples are mainly used to evaluate the stability of the model.

[0144] In this embodiment, a criterion-sensitive sample refers to a sample whose detection result label changes under different detection criteria. That is, for this sample, applying strict and lenient criteria will lead to different judgment conclusions. For example, taking an image of wood with minor scratches as an example, under a zero-tolerance criterion, the detection result label is "unacceptable"; under a limited-tolerance criterion, the detection result label is "acceptable." The detection result of this sample differs under different criteria, therefore it is a criterion-sensitive sample. The judgment conclusion of a criterion-sensitive sample depends on the detection criterion; the model must understand the currently applied detection criterion to make a correct judgment. Therefore, criterion-sensitive samples are mainly used to evaluate the model's adaptability.

[0145] For example, Table 4 below shows a test set.

[0146]

[0147] Table 4 In this embodiment of the application, by constructing a test set that includes criterion-invariant samples and criterion-sensitive samples, two different dimensions of the model can be evaluated separately: stability (detection accuracy performance on criterion-invariant samples) and adaptability (detection accuracy performance on criterion-sensitive samples). This separate test set design makes the evaluation results more targeted and interpretable.

[0148] In step 505, the first detection accuracy of the fine-tuned multimodal large model on criterion-sensitive samples and the second detection accuracy of the fine-tuned multimodal large model on criterion-unchanged samples are statistically analyzed. The first detection accuracy is used to characterize the adaptability of the fine-tuned multimodal large model to changes in detection criteria, and the second detection accuracy is used to characterize the ability of the fine-tuned multimodal large model to maintain detection conclusions.

[0149] In this embodiment, the first detection accuracy reflects the model's performance on criterion-sensitive samples. For each criterion-sensitive sample in the test set, it is input into the model along with a reference support set expressing different detection criteria, and the model's output detection result is checked to see if it matches the true label. Specifically, for the same criterion-sensitive sample, when paired with a reference support set expressing a zero-tolerance criterion, the model should output "unqualified"; when paired with a reference support set expressing a limited-tolerance criterion, the model should output "qualified". If the model outputs correct results under both criteria, the sample is counted in the number of correctly responding samples. The first detection accuracy is the proportion of correctly responding criterion-sensitive samples to the total number of criterion-sensitive samples. The higher this value, the better the model can correctly switch judgment conclusions according to changes in detection criteria.

[0150] In this embodiment, the second detection accuracy reflects the model's performance on criterion-invariant samples. For each criterion-invariant sample in the test set, it is input into the model along with a reference support set expressing different detection criteria, and the model's output detection results are checked for consistency and correctness. Specifically, for the same criterion-invariant sample, when paired with a reference support set expressing a zero-tolerance criterion, the model should output a correct result; when paired with a reference support set expressing a limited-tolerance criterion, the model should output the same correct result. If the model outputs the same correct result under both criteria, the sample is counted in the number of consistently correct response samples. The second detection accuracy is the proportion of consistently correct response criterion-invariant samples to the total number of criterion-invariant samples. The higher this value, the more the model is unaffected by changes in detection criteria and maintains the stability of its judgments.

[0151] For example, consider a criterion-sensitive sample with slight scratches, as shown in Table 5 below.

[0152]

[0153] Table 5 As can be seen from Table 5, the model can correctly switch detection results according to changes in detection criteria on criterion-sensitive samples, indicating that it has good adaptability.

[0154] For example, consider a criterion-invariant sample with severe cracks, as shown in Table 6 below.

[0155]

[0156] Table 6 As can be seen from Table 6, the model can maintain consistent and correct judgments on samples with unchanged criteria, indicating that it has good stability.

[0157] In this embodiment, the model's criterion-awareness capability can be comprehensively evaluated by separately calculating the first detection accuracy and the second detection accuracy. The first detection accuracy reflects the model's sensitivity to criterion changes and its ability to switch correctly, while the second detection accuracy reflects the model's ability to maintain its inherent conclusions. Together, they constitute a comprehensive performance evaluation of the model in dynamic detection scenarios.

[0158] In step 506, based on the first detection accuracy and the second detection accuracy, it is determined whether the fine-tuned multimodal large model meets the preset detection requirements.

[0159] In this embodiment, the preset detection requirement refers to a performance threshold set according to a specific business scenario. Different business scenarios may have different requirements for the first detection accuracy and the second detection accuracy. For example, for scenarios that require frequent switching of detection criteria (such as simultaneously inspecting multiple product types), the requirement for the first detection accuracy is higher; for scenarios with relatively fixed criteria, the requirement for the second detection accuracy is higher.

[0160] In this embodiment, the statistically obtained first and second detection accuracies are compared with preset detection requirements: if the first detection accuracy is greater than or equal to the first threshold and the second detection accuracy is greater than or equal to the second threshold, the fine-tuned multimodal large model is determined to meet the preset detection requirements, indicating that the model has good criterion awareness capabilities and can be deployed to actual business scenarios; otherwise, the model is determined not to meet the requirements. If the first detection accuracy is low, it indicates that the model's ability to perceive changes in criteria is insufficient, and it may be necessary to increase the proportion of criterion-sensitive samples or adjust the training strategy; if the second detection accuracy is low, it indicates that the model's judgment of inherent conclusions is not stable enough, and it may be necessary to increase the proportion of criterion-invariant samples or adjust the training strategy. Based on the analysis results, the training data or training parameters are adjusted and retrained, and the evaluation is performed again until the requirements are met.

[0161] In this embodiment, by setting preset detection requirements and comparing them with evaluation results, it is possible to objectively and quantitatively determine whether a model can be put into use. This provides a scientific basis for model quality control and deployment decisions, avoiding the uncertainty caused by subjective judgment. Furthermore, for models that do not meet the requirements, training strategies can be adjusted in a targeted manner based on the evaluation results to improve training efficiency.

[0162] As can be seen, the model evaluation method provided in this application constructs a test set containing criterion-invariant samples and criterion-sensitive samples, statistically analyzes the detection accuracy of the model on the two types of samples respectively, and judges whether the model meets the usage conditions based on preset detection requirements. It can comprehensively and objectively evaluate the criterion perception capability of fine-tuning multimodal large models, and provide a reliable basis for verifying the model training effect and making decisions on model deployment.

[0163] The criterion-aware image detection method provided in this application can be executed by a criterion-aware image detection device. This application uses an example of a criterion-aware image detection device executing the criterion-aware image detection method to illustrate the criterion-aware image detection device provided in this application.

[0164] Figure 6 This is a structural block diagram of a criterion-based image detection device provided in some embodiments of this application, such as... Figure 6 As shown, the criterion-based image detection device 600 may include: an acquisition module 601 and a detection module 602; The acquisition module 601 is used to acquire information about the task to be detected; wherein, the information about the task to be detected includes: visual data to be detected, task description information, and a reference support set associated with the task to be detected; the reference support set includes at least one reference example visual data and a detection result label corresponding to the reference example visual data; The detection module 602 is used to input the task information to be detected into the fine-tuned multimodal large model, and through the fine-tuned multimodal large model, perceive the detection criteria used by the reference support set according to the instructions of the task description information, and detect the visual data to be detected according to the detection criteria, and output the detection result.

[0165] As can be seen from the above embodiments, in this embodiment, by acquiring detection task information including visual data to be detected, task description information and reference support set, and inputting it into a fine-tuned multimodal large model for detection, the model can perceive the current detection criteria and adaptively adjust the discrimination boundary according to the reference support set, thereby improving the detection accuracy of critical images. At the same time, image detection can be achieved with only a small number of reference examples, reducing the data annotation cost.

[0166] Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device includes a processing component 722, which further includes one or more processors, and memory resources represented by a memory 732 for storing instructions executable by the processing component 722, such as application programs. The application programs stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform the above-described criterion-based image detection method.

[0167] The electronic device may also include a power supply 726 configured to perform power management of the electronic device, a wired or wireless network interface 750 configured to connect the electronic device to a network, and an input / output (I / O) interface 758. The electronic device may operate on an operating system stored in memory 732, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or similar.

[0168] According to another embodiment of this application, this application also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps in the criterion-based image detection method as described in any of the above embodiments.

[0169] According to another embodiment of this application, this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps in the criterion-based image detection method as described in any of the above embodiments.

[0170] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Those skilled in the art will understand that the embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, the embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0172] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. Although preferred embodiments of the embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0173] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0174] The foregoing has provided a detailed description of the criterion-based image detection method, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A criterion-based image detection method, characterized in that, The method includes: Obtain information about the task to be detected; wherein, the information about the task to be detected includes: visual data to be detected, task description information, and a reference support set associated with the task to be detected; the reference support set contains at least one reference example visual data and the detection result label corresponding to the reference example visual data; The task information to be detected is input into the fine-tuned multimodal large model. The fine-tuned multimodal large model perceives the detection criteria used by the reference support set according to the instructions of the task description information, and detects the visual data to be detected according to the detection criteria, and outputs the detection results.

2. The method according to claim 1, characterized in that, The task to be detected is a discrimination task; Furthermore, the visual data to be detected is an image or video.

3. The method according to claim 1, characterized in that, Under different detection criteria, the detection result labels corresponding to the same reference example visual data are different.

4. The method according to claim 1, characterized in that, The reference example visual data and the detected object in the visual data to be detected belong to the same category; wherein, the category is used to indicate the object type to which the detected object belongs.

5. The method according to claim 4, characterized in that, Each of the categories contains at least one subcategory, and the subcategory information is used to represent the specific state type under the category; The detection criteria are used to characterize the tolerance of the task to be detected for anomalies in the sub-category information.

6. The method according to claim 1, characterized in that, The detection criteria used to perceive the reference support set through the fine-tuned multimodal large model include: The fine-tuned multimodal large model perceives the detection criteria used by the reference support set based on the correspondence between the reference example visual data and the detection result labels. The fine-tuned multimodal large model is trained on training data containing multiple detection criterion support sets and has the ability to perceive detection criteria.

7. The method according to claim 1, characterized in that, The fine-tuned multimodal large model is obtained through multi-criteria training: Obtain a basic multimodal large model; wherein the basic multimodal large model is a pre-trained multimodal model used to generate prediction results based on input visual data and text; Construct a training set; wherein the training set contains multiple sets of training data, each set of training data including: a sample visual data, at least two sample support sets corresponding to the sample visual data, and detection result labels of the sample visual data under each sample support set; the at least two sample support sets correspond to different detection criteria, and the sample support sets contain at least one sample example visual data and a corresponding detection result label; all sample support sets in the same set of training data use the same sample example visual data, but the detection result labels of the sample example visual data are different; Each set of training data is input into the basic multimodal large model to obtain the prediction detection result. The loss value is calculated based on the prediction detection result and the detection result label of the visual data of the samples in the training data. The parameters of the basic multimodal large model are updated based on the loss value. The above training steps are repeated until the model converges to obtain the fine-tuned multimodal large model.

8. The method according to claim 7, characterized in that, The construction of the training set includes: Obtain category definition information; wherein, the category is used to represent the object type to which the detected object belongs, and any category contains at least one subcategory, the subcategory being used to represent the specific state type under the category; Based on different detection criteria, labels are mapped to each subcategory to determine the corresponding detection result labels for each subcategory under different detection criteria. Based on the label mapping, the visual data of the same sample example is copied into at least two copies, and labeled according to the detection result labels under different detection criteria, to obtain at least two sample support sets; wherein, each sample support set expresses a detection criterion; The sample support set is associated with the corresponding sample visual data to obtain a set of training data.

9. The method according to claim 7, characterized in that, The method further includes: Obtain a test set; wherein the test set includes criterion-invariant samples and criterion-sensitive samples; wherein the criterion-invariant samples have the same detection result label under different detection criteria, and the criterion-sensitive samples have different detection result labels under different detection criteria; The first detection accuracy of the fine-tuned multimodal large model on the criterion-sensitive samples is statistically analyzed, and the second detection accuracy of the fine-tuned multimodal large model on the criterion-unchanged samples is statistically analyzed. The first detection accuracy is used to characterize the adaptability of the fine-tuned multimodal large model to changes in the detection criteria, and the second detection accuracy is used to characterize the ability of the fine-tuned multimodal large model to maintain the detection conclusion. Based on the first detection accuracy and the second detection accuracy, determine whether the fine-tuned multimodal large model meets the preset detection requirements.

10. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the criterion-based image detection method as described in any one of claims 1-9.

11. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the criterion-based image detection method as described in any one of claims 1-9.

Images