Model training method and device, image detection method and device, electronic equipment and storage medium
By expanding the training data and generating difference prompts, the problem of artifact detection in the image generation process in the existing technology is solved, the robustness and detection accuracy of the model are improved, and effective detection of image anomalies is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PENG CHENG LAB
- Filing Date
- 2025-12-19
- Publication Date
- 2026-05-01
AI Technical Summary
Existing technologies struggle to effectively detect artifacts such as limb misalignment, missing limbs, or redundancy during image generation. Traditional detection methods rely on manual features and are costly, while supervised learning-based methods require large-scale annotation of artifact samples, resulting in poor detection performance.
By expanding the training data and combining normal and abnormal image samples, the model is trained, difference prompts are generated, and fine-tuning is performed to enhance the model's tolerance to noise and improve its robustness and generalization performance.
It significantly improves the model's detection accuracy in noisy data scenarios and effectively detects abnormal parts in images.
Smart Images
Figure CN121962799A_ABST
Abstract
Description
Model training methods, image detection methods, devices, electronic equipment and storage media Technical Field
[0001] This application belongs to the field of image processing technology, and in particular relates to a model training method, an image detection method, an apparatus, an electronic device, and a storage medium. Background Technology
[0002] AIGC (AI Generated Content) refers to the technology and application fields that automatically generate various types of content such as text, images, audio, and video using artificial intelligence technology. Among them, AI image generation technology has developed rapidly and, with its efficient content production capabilities, has been widely applied in various practical scenarios such as e-commerce virtual try-on, film and television creation, and virtual idol creation. Current image generation processes are prone to artifacts such as: limb misalignment, manifested as joint angles exceeding the limits of human anatomy; missing or redundant limbs, such as missing finger joints, extra toes, or missing skull bones; disproportionate body proportions; and entangled limbs in multi-person scenes, resulting in limb penetration, head and neck swapping, and other problems. Current detection solutions for these image anomalies have significant shortcomings: traditional detection relies on manually defined features, which is difficult to adapt to the varied shapes of artifacts, easily leading to missed detections or false positives; while supervised learning-based detection models require large-scale annotation of artifact samples to complete training, resulting in high annotation costs and failing to effectively detect abnormal parts in images. Summary of the Invention
[0003] This application provides a model training method, an image detection method, an apparatus, an electronic device, and a storage medium. By expanding the training data and training the model with normal and abnormal image samples, the tolerance of the large model to input noise is significantly enhanced, the robustness and generalization performance of the model in noisy data scenarios are improved, and the detection accuracy of the large model is improved, effectively detecting anomalies in images.
[0004] In a first aspect, embodiments of this application provide a model training method, the method comprising: acquiring preset training samples, the preset training samples including a preset image, a preset anomaly detection prompt statement, and anomaly detection labels, wherein the preset image is generated by a preset image generation model; adding noise to image regions in the preset image other than anomaly detection regions according to the anomaly detection labels to obtain a processed target image; generating a difference prompt statement based on the difference between the target image and the preset image, and updating the preset anomaly detection prompt statement according to the difference prompt statement to obtain a target anomaly detection prompt statement; generating a set of target training samples by combining the target image, the target anomaly detection prompt statement, and the anomaly detection labels; and fine-tuning a preset large model by combining the preset training samples and the target training samples to obtain a target large model, the target large model being used to detect images to be detected generated by the preset image generation model.
[0005] In one embodiment of this application, the step of adding noise to the image region other than the anomaly detection region in the preset image according to the anomaly detection label to obtain the processed target image includes: generating image mask information for the anomaly detection region in the preset image according to the anomaly detection label; and adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image.
[0006] In one embodiment of this application, the step of adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image includes: generating a random number using a random algorithm; randomly selecting a quality value from a preset quality range as a target value when the random number meets preset conditions; and compressing the image region other than the anomaly detection region in the preset image based on the target value and the image mask information to obtain the processed target image.
[0007] In one embodiment of this application, the step of adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image includes: generating random numbers using a random algorithm; and, if the random numbers meet preset conditions, adjusting the brightness of the image region other than the anomaly detection region in the preset image to a preset brightness, and / or adjusting the saturation to a preset saturation, and / or adjusting the contrast to a preset contrast based on the image mask information to obtain the processed target image.
[0008] In one embodiment of this application, the step of adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image includes: generating random numbers using a random algorithm; and, if the random numbers meet preset conditions, modifying the pixel values of at least one pixel within a target range region in the image region other than the anomaly detection region in the preset image to preset values based on the image mask information to obtain the processed target image.
[0009] In one embodiment of this application, the step of adding noise to the image region of the preset image excluding the anomaly detection region based on the image mask information to obtain the processed target image includes: generating random numbers using a random algorithm; randomly selecting a variance within a preset variance range as the target variance when the random numbers meet preset conditions; and adding Gaussian noise to the image region of the preset image excluding the anomaly detection region based on the target variance and the image mask information to obtain the processed target image.
[0010] In one embodiment of this application, the step of fine-tuning the preset large model by combining the preset training samples and the target training samples to obtain the target large model includes: during the training of the preset large model using the preset training samples and the target training samples, determining a loss value by combining the anomaly detection result output by the preset large model for the anomaly detection region with the anomaly detection label; if the loss value is greater than or equal to the preset loss value, fine-tuning the preset large model until the loss value is less than the preset loss value, and using the currently fine-tuned preset large model as the target large model.
[0011] Secondly, embodiments of this application provide an image detection method, the method comprising: acquiring an image to be detected, the image to be detected being generated by a preset image generation model; inputting the image to be detected and a preset anomaly detection prompt statement into a target large model to obtain a target anomaly detection result of the image to be detected, the target large model being obtained based on the model training method described in the first aspect.
[0012] Thirdly, embodiments of this application provide a model training apparatus, the apparatus comprising: a first acquisition module, configured to acquire preset training samples, the preset training samples including a preset image, a preset anomaly detection prompt statement, and anomaly detection labels, wherein the preset image is generated by a preset image generation model; a first processing module, configured to add noise to image regions in the preset image other than anomaly detection regions according to the anomaly detection labels, to obtain a processed target image; a second processing module, configured to generate a difference prompt statement according to the difference between the target image and the preset image, and update the preset anomaly detection prompt statement according to the difference prompt statement, to obtain a target anomaly detection prompt statement; a third processing module, configured to combine the target image, the target anomaly detection prompt statement, and the anomaly detection labels to generate a set of target training samples; and a training module, configured to combine the preset training samples and the target training samples to fine-tune and train a preset large model to obtain a target large model, the target large model being used to detect images to be detected generated by the preset image generation model.
[0013] Fourthly, embodiments of this application provide an image detection device, the device comprising: a second acquisition module for acquiring an image to be detected, the image to be detected being generated by a preset image generation model; and a detection module for inputting the image to be detected and a preset anomaly detection prompt statement into a target large model to obtain a target anomaly detection result of the image to be detected, the target large model being obtained by the model training method described in the first aspect.
[0014] Fifthly, embodiments of this application provide an electronic device, including: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the model training method as described in the first aspect, or the image detection method as described in the second aspect.
[0015] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the model training method as described in the first aspect, or the image detection method as described in the second aspect.
[0016] In a seventh aspect, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the model training method as described in the first aspect, or the image detection method as described in the second aspect.
[0017] This embodiment provides a model training method, image detection method, apparatus, electronic device, and storage medium. By acquiring preset training samples, which include preset images, preset anomaly detection prompts, and anomaly detection labels, noise is added to the image regions in the preset images excluding anomaly detection areas based on the anomaly detection labels, resulting in a processed target image. Further, based on the differences between the target image and the preset image, a difference prompt is generated, and the preset anomaly detection prompt is updated based on the difference prompt, resulting in a target anomaly detection prompt. Thus, a set of target training samples is generated by combining the target image, the target anomaly detection prompt, and the anomaly detection labels. The preset large model is then fine-tuned using the preset training samples and the target training samples to obtain the target large model. In the above steps, expanding the training data in this way, and training the model using both normal and anomaly image samples, significantly enhances the large model's tolerance to input noise, improves its robustness and generalization performance in noisy data scenarios, and ultimately improves the detection accuracy of the large model, effectively detecting anomalies in images. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 is a flowchart of a model training method provided in an embodiment of this application; Figure 2 is a framework diagram of an image detection method provided in an embodiment of this application; Figure 3 is a flowchart of an image detection method provided in an embodiment of this application; Figure 4 is a structural diagram of a model training device provided in an embodiment of this application; Figure 5 is a structural diagram of an image detection device provided in an embodiment of this application; Figure 6 is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0022] In all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user images, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. Additionally, when embodiments of this application require access to sensitive personal information, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments obtained.
[0023] To address the problems of the prior art, embodiments of this application provide a model training method, an image detection method, an apparatus, an electronic device, and a storage medium. The model training method provided in this application embodiment will be described first below.
[0024] Figure 1 shows a flowchart of a model training method provided in an embodiment of this application. As shown in Figure 1, the model training method provided in this embodiment of the application is applied to an electronic device, such as a server, and includes the following steps 101-105, wherein: Step 101, obtaining a preset training sample, the preset training sample includes a preset image, a preset anomaly detection prompt statement, and anomaly detection labels, wherein the preset image is generated by a preset image generation model.
[0025] In this embodiment, preset training samples are obtained. There are multiple sets of preset training samples. Each set of preset training samples includes preset images, preset anomaly detection prompts, and anomaly detection labels. The preset images are generated by a preset image generation model. The preset images are Artificial Intelligence Generated Content (AIGC). The core of AIGC is to enable the preset image generation model, i.e., the AI model, to autonomously complete the creation and generation of content by learning patterns and rules in massive amounts of data.
[0026] Optionally, the preset anomaly detection prompt statement may include the following: System: You are an anomaly detection assistant; User: <image>; Task: List all abnormal body parts in the format "[category name cx cy wh]", with each part separated by a comma, using normalized coordinates (0-1); Assistant: Abnormal head 0.34 0.52 0.09 0.21, abnormal neck 0.67 0.43 0.12 0.18.
[0027] The image in the preset anomaly detection prompt is the aforementioned preset image, with anomaly head dimensions of 0.34 0.52 0.09 0.21 and anomaly neck dimensions of 0.67 0.43 0.12 0.18, used to induce the large model to output this format; cx is the x-coordinate of the bounding box center, cy is the y-coordinate of the bounding box center, w is the width of the bounding box, and h is the height of the bounding box. The area enclosed by the bounding box is the anomaly detection area.
[0028] Step 102: Based on the anomaly detection label, noise is added to the image regions in the preset image other than the anomaly detection region to obtain the processed target image.
[0029] In this embodiment, the anomaly detection label refers to the standard answer or target result, which can be the coordinate information of the target area in the image, including the horizontal coordinate, vertical coordinate, width, and height. Based on the anomaly detection label, the image area in the preset image other than the anomaly detection area is subjected to noise addition processing to obtain the processed target image. The noise addition processing includes at least one of compression processing, color perturbation processing, noise removal processing, and adding Gaussian noise.
[0030] The anomaly detection region can be the detection area corresponding to the anomaly detection label in the preset image. This region generally contains the object to be detected. For example, taking a person as the object to be detected, the anomaly detection region contains the person to be detected. This includes detecting the position of the person in the image, whether there are any limb misalignments, whether the person's facial features conform to objective facts, etc., depending on the actual detection task. Furthermore, the image area in the preset image other than the anomaly detection region can be understood as the background area in the preset image.
[0031] Step 103: Generate a difference prompt statement based on the difference between the target image and the preset image, and update the preset anomaly detection prompt statement based on the difference prompt statement to obtain the target anomaly detection prompt statement.
[0032] In this embodiment, a difference prompt statement is generated based on the difference between the target image and the preset image. For example, the noise addition process includes compression processing. The icon image is a compressed image, and the difference between it and the preset image is compression. The generated difference prompt statement could be "This image has compression artifacts." Alternatively, if four noise addition processes are performed, the generated difference prompt statement could be "This image has compression artifacts, color jitter, cropped regions, and noise." Here, compression artifacts are obtained through compression processing, color jitter is obtained through color perturbation processing, cropped regions are obtained through elimination processing, and noise is obtained by adding Gaussian noise.
[0033] Furthermore, the preset anomaly detection prompt statement is updated based on the difference prompt statement. For example, the difference prompt statement is merged with the preset anomaly detection prompt statement to obtain the target anomaly detection prompt statement, or the preset anomaly detection prompt statement is modified using the difference prompt statement to obtain the target anomaly detection prompt statement.
[0034] Step 104: Combine the target image, the target anomaly detection prompt statement, and the anomaly detection label to generate a set of target training samples.
[0035] In this embodiment, a set of target training samples is generated by combining the target image, the target anomaly detection prompt statement, and the anomaly detection label. There are multiple sets of target training samples, and each set of target training samples includes the target image, the target anomaly detection prompt statement, and the anomaly detection label.
[0036] Step 105: Combine the preset training samples and the target training samples to fine-tune the preset large model to obtain the target large model. The target large model is used to detect the image to be detected generated by the preset image generation model.
[0037] In this embodiment, preset training samples and target training samples are combined as training samples for a preset large model. The images in the preset training samples are normal image samples, and the images in the target training samples are abnormal image samples. The preset large model is fine-tuned and trained to obtain a target large model. The target large model is used to detect the image to be detected, wherein the image to be detected is generated by a preset image generation model.
[0038] In this embodiment, a preset training sample is obtained, which includes a preset image, a preset anomaly detection prompt statement, and anomaly detection labels. Based on the anomaly detection labels, noise is added to the image regions in the preset image except for the anomaly detection area to obtain the processed target image. Further, based on the difference between the target image and the preset image, a difference prompt statement is generated, and the preset anomaly detection prompt statement is updated based on the difference prompt statement to obtain the target anomaly detection prompt statement. Thus, a set of target training samples is generated by combining the target image, the target anomaly detection prompt statement, and the anomaly detection labels. The preset large model is fine-tuned and trained by combining the preset training samples and the target training samples to obtain the target large model. In the above steps, by expanding the training data in the above manner and training the model with normal image samples and anomaly image samples, the tolerance of the large model to input noise can be significantly enhanced, the robustness and generalization performance of the model in noisy data scenarios can be improved, and the detection accuracy of the large model can be improved, effectively detecting anomalies in images.
[0039] In one embodiment of this application, the step of adding noise to the image region other than the anomaly detection region in the preset image according to the anomaly detection label to obtain the processed target image includes: generating image mask information for the anomaly detection region in the preset image according to the anomaly detection label; and adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image.
[0040] In this embodiment, based on the anomaly detection label, image mask information for the anomaly detection area in the preset image is generated. Specifically, a segmentation model is used for processing, such as using the SAM (Segment Anything Model) segmentation model to obtain the mask information. Any region is accurately segmented using points, boxes, text prompts, etc. The core logic for obtaining the image anomaly detection area mask is: first, locate the anomaly detection area in the image, then segment the area using the SAM segmentation model, and finally process it into a binary mask template mask M, that is, obtain the image mask information of the anomaly detection area in the preset image. The image mask information is a two-dimensional matrix, and the elements in the matrix only contain two values: 0 and 1. 0 is the background area, and 1 is the anomaly detection area. The size of the mask matrix is consistent with the resolution of the corresponding preset image. For example, if the input image is an H×W (height × width) image, then mask M is an H×W two-dimensional matrix. mask M marks the anomaly area (1) and the background area (0). Subsequently, the anomaly area (0) can be only processed by the image × mask operation. The noise is added to the region, while the anomaly detection region (1 region) is completely preserved, and the processing range is precisely controlled. The background region is the image region in the preset image other than the anomaly detection region.
[0041] Furthermore, based on the image mask information, noise is added to the image regions in the preset image other than the anomaly detection region to increase the noise in the background region, thus obtaining the processed target image.
[0042] By clearly distinguishing the anomaly detection area from the image area outside the anomaly detection area using mask information, noise is added only to the image area outside the anomaly detection area, i.e., the background area. This preserves the complete features of the anomaly area while avoiding disturbance and destruction of key anomaly information, forcing the model to focus on learning the core features of the anomaly area and improving the targeting of anomaly detection.
[0043] In one embodiment of this application, the step of adding noise to the image region of the preset image excluding the anomaly detection region based on the image mask information to obtain a processed target image includes: generating a random number using a random algorithm; randomly selecting a quality value from a preset quality range as a target value when the random number meets preset conditions; and compressing the image region of the preset image excluding the anomaly detection region based on the target value and the image mask information to obtain the processed target image.
[0044] In this embodiment, a random algorithm is used to generate random numbers. The random algorithm generates random numbers based on the interval [0, 1] and determines whether the random numbers meet the preset conditions. For example, if the generated random number is less than or equal to 0.5, the preset conditions are met; if the generated random number is greater than 0.5, the preset conditions are not met.
[0045] If the random number meets the preset conditions, a quality value is randomly selected from the preset quality range as the target value. The preset quality range can be quality_range=(75, 100). A quality value is selected from (75, 100) as the target value. (75, 100) means that the range of image quality values is limited to between 75 and 100. This is the quality control for JPEG compression format. The smaller the value, the more severe the compression and the stronger the distortion.
[0046] Based on the target value and image mask information, the image regions in the preset image, excluding the anomaly detection region, are compressed. The mask information can accurately distinguish the background region that needs to be compressed and the anomaly detection region that needs to be retained, thus obtaining the processed target image.
[0047] For example, the `albumentations.ImageCompression` method can be used for compression processing. The `albumentations.ImageCompression` method is a core data augmentation method in the Albumentations library used to simulate JPEG compression distortion and add controllable compression noise to the image.
[0048] Compression processing only damages the image quality of the background area, while preserving the original clarity of the anomaly detection area. This avoids interfering with the features of the anomaly detection area and can improve the model's robustness to scenes with background compression distortion.
[0049] In one embodiment of this application, the step of adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image includes: generating random numbers using a random algorithm; and, if the random numbers meet preset conditions, adjusting the brightness of the image region other than the anomaly detection region in the preset image to a preset brightness, and / or adjusting the saturation to a preset saturation, and / or adjusting the contrast to a preset contrast based on the image mask information to obtain the processed target image.
[0050] In this embodiment, a random algorithm is used to generate random numbers. The random algorithm generates random numbers based on the interval [0, 1] and determines whether the random numbers meet the preset conditions. For example, if the generated random number is less than or equal to 0.5, the preset conditions are met; if the generated random number is greater than 0.5, the preset conditions are not met.
[0051] When the random number meets the preset conditions, the image regions in the preset image other than the anomaly detection region are adjusted based on the image mask information. Specifically, the brightness of the background region is adjusted to the preset brightness, and / or the saturation is adjusted to the preset saturation, and / or the contrast is adjusted to the preset contrast, so as to obtain the processed target image.
[0052] For example, you can use `albumentations.ColorJitter` to adjust the brightness, contrast, and saturation of an image. `albumentations.ColorJitter` is a method in the Albumentations library used to randomly adjust the color attributes of an image. Combined with image mask information, it adds color perturbation only to the background area. For example, it can adjust the brightness of the background area to 0.4, and / or adjust the saturation to 0.4, and / or adjust the contrast to 0.4.
[0053] Color perturbation processing only destroys the color features of the background area, simulating color interference in real-world scenes. This avoids the model relying on background color to identify anomalies, fully preserving the original color features of the anomaly detection area. This ensures that the model learns the features of the anomaly itself rather than the background features. Moreover, the training data contains diverse background color perturbations, enabling the model to accurately identify anomalies when faced with images under different lighting conditions or from different devices during training.
[0054] In one embodiment of this application, the step of adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image includes: generating random numbers using a random algorithm; and, if the random numbers meet preset conditions, modifying the pixel values of at least one pixel within a target range region in the image region other than the anomaly detection region in the preset image to preset values based on the image mask information to obtain the processed target image.
[0055] In this embodiment, a random algorithm is used to generate random numbers. The random algorithm generates random numbers based on the interval [0, 1] and determines whether the random numbers meet the preset conditions. For example, if the generated random number is less than or equal to 0.5, the preset conditions are met; if the generated random number is greater than 0.5, the preset conditions are not met.
[0056] When the random number meets the preset conditions, at least one target range region is determined in the image region outside the anomaly detection region in the preset image. For example, the target range region consists of 64×64 pixels. The pixel values of the pixels in at least one target range region are modified to preset values to obtain the processed target image. For example, the pixel values of the pixels in each of the two target range regions are modified by 128.
[0057] Alternatively, the Cutout method can be used, which is a lightweight image data augmentation technique. The Cutout method randomly erases image regions outside the anomaly detection area, forcing the model to learn the global features of the target rather than relying on local details.
[0058] Modifying pixel values in image regions outside of anomaly detection areas reduces background noise interference, thereby improving the model's accuracy and robustness in target recognition and detection tasks.
[0059] In one embodiment of this application, the step of adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image includes: generating random numbers using a random algorithm; randomly selecting a variance within a preset variance range as the target variance when the random numbers meet preset conditions; and adding Gaussian noise to the image region other than the anomaly detection region in the preset image based on the target variance and the image mask information to obtain the processed target image.
[0060] In this embodiment, a random algorithm is used to generate random numbers. The random algorithm generates random numbers based on the interval [0, 1] and determines whether the random numbers meet the preset conditions. For example, if the generated random number is less than or equal to 0.3, the preset conditions are met; if the generated random number is greater than 0.3, the preset conditions are not met.
[0061] If the random number meets the preset conditions, a variance is randomly selected as the target variance within the preset variance range. The preset variance range consists of multiple variance values. The preset variance range is used to limit the upper and lower limits of the intensity of a certain type of enhancement operation. For example, the preset variance range is var_limit=(1.0, 3.0). When the target variance is 1, the noise intensity is weak, only slightly perturbs the image, and retains most of the original features. When the target variance is 3, the noise intensity is moderate, which can effectively improve the model's anti-interference ability and will not completely cover up the image details.
[0062] Based on the target variance and image mask information, Gaussian noise is added to the image region outside the anomaly detection region in the preset image, that is, the background region in the image is processed to obtain the processed target image.
[0063] Set a variance range, select the target variance based on the variance range to process the image, avoid the noise being too weak to have no enhancement effect, and prevent the noise being too strong to destroy image features.
[0064] In one embodiment of this application, the step of fine-tuning the preset large model by combining the preset training samples and the target training samples to obtain the target large model includes: during the training of the preset large model using the preset training samples and the target training samples, determining a loss value by combining the anomaly detection result output by the preset large model for the anomaly detection region with the anomaly detection label; if the loss value is greater than or equal to the preset loss value, fine-tuning the preset large model until the loss value is less than the preset loss value, and using the currently fine-tuned preset large model as the target large model.
[0065] In this embodiment, during the training of the preset large model using preset training samples and target training samples, the loss value is calculated by combining the anomaly detection results and anomaly detection labels output by the preset large model for the anomaly detection region, such as calculating the cross-entropy loss value. If the loss value is greater than or equal to the preset loss value, the preset large model is fine-tuned until the loss value is less than the preset value. The current fine-tuned preset large model is used as the target large model, which is used to detect the image to be detected. The image to be detected is generated by the preset image generation model.
[0066] Referring to Figure 2, the preset images and anomaly detection labels are obtained from the object detection dataset. The object detection dataset can be the HumanRefiner dataset, which consists of two parts: train and val. The train / val directory contains two directories: images and labels. The images contain the images, i.e., the preset images mentioned above, and the labels contain the ground truth annotations of each corresponding image, i.e., the anomaly detection labels mentioned above. The anomaly detection labels can be the coordinate information of the anomaly detection region or the coordinate information of the bounding box (bbox) corresponding to the anomaly detection region. According to the path of the provided HumanRefiner dataset, the paths of all images are read.
[0067] A set of preset training samples is generated based on preset images, preset anomaly detection prompts, and anomaly detection labels. The preset anomaly detection prompts are as follows: System: You are an anomaly detection assistant; User: <image>; Task: List all abnormal body parts in the format "[category name cx cy wh]", with each part separated by a comma, using normalized coordinates (0-1); Assistant: Abnormal head 0.34 0.52 0.09 0.21, Abnormal neck 0.67 0.43 0.12 0.18.
[0068] Furthermore, noise is added to the image regions in the preset image excluding the anomaly detection area to obtain the processed target image. Based on the difference between the target image and the preset image, a difference prompt statement is generated. The preset anomaly detection prompt statement is then updated based on the difference prompt statement to obtain the target anomaly detection prompt statement. The target anomaly detection prompt statement is as follows: System: You are an anomaly detection assistant; User: <Image>; Task: List all abnormal body parts in the format "[Category Name cx cy wh]", separated by commas, using normalized coordinates (0-1); Assistant: Abnormal head 0.34 0.52 0.09 0.21, Abnormal neck 0.67 0.43 0.12 0.18; This image contains compression artifacts.
[0069] The image in question contains compression artifacts, which is indicated by the difference message.
[0070] By combining the target image, the target anomaly detection prompt, and the anomaly detection label, a set of target training samples is generated. The preset large model is then fine-tuned by combining the preset training samples and the target training samples, as shown in Figure 2. The preset large model is a multimodal large model, which can be an LLaVANext-8B model. The preset large model includes a visual encoder, a projection layer (MLP Projector), and a large language model (LLM). The visual encoder can be a contrastive language-image pre-training (CLIP) model, such as the CLIP-ViT-L model, which is about 300M in size.
[0071] The embedding layer transforms the target anomaly detection prompt into a statement feature vector; the visual encoder transforms the target image into an image feature vector; and the projection layer maps the image feature vector to obtain a first feature vector, ensuring that the first feature vector and the statement feature vector reside in the same vector space (see Figure 2). The statement feature vector and the first feature vector are then concatenated to obtain a concatenated vector. It should be noted that the structure of the pre-defined large model is not limited to the above structure and can also be other large models.
[0072] The concatenated vector is input into the large language model. The loss value is calculated by combining the anomaly detection results and anomaly detection labels output by the preset large model for the anomaly detection region. If the loss value is greater than or equal to the preset loss value, the preset large model is fine-tuned until the loss value is less than the preset loss value. The current fine-tuned preset large model is then used as the target large model.
[0073] To address the training process of large-scale anomaly detection tasks, this paper proposes a data augmentation method that maintains coordinate conservation in the anomaly detection region. All noise addition is confined to the image region outside the anomaly detection region. Current conventional object detection data augmentation methods, such as those in the pure vision domain, typically alter the bounding box, requiring corresponding transformations. In this application, the bounding box coordinates remain unchanged. This data augmentation method expands the training data, enhances the noise resistance, robustness, and generalization of the trained large-scale model, and improves the model's detection accuracy.
[0074] To ensure semantic consistency between the multimodal image-text pair (i.e., the preset images and anomaly detection prompts), the image enhancement techniques used on the text side are explained, providing a semantically consistent multimodal data enhancement technique. It is necessary not only to enhance the image but also to maintain the semantic consistency of the anomaly detection prompts during enhancement.
[0075] Figure 3 shows a flowchart of an image detection method provided in an embodiment of this application. As shown in Figure 3, the image detection method provided in this embodiment of the application is applied to an electronic device, such as a server, and includes the following steps 301-302, wherein: Step 301, acquiring an image to be detected, the image to be detected being generated by a preset image generation model.
[0076] In this embodiment, an image to be detected is acquired. The image to be detected is generated by a preset image generation model. The image to be detected is Artificial Intelligence Generated Content (AIGC). The core of AIGC is to enable the preset image generation model, i.e., the AI model, to autonomously complete the creation and generation of content by learning patterns and rules in massive amounts of data.
[0077] Step 302: Input the image to be detected and the preset anomaly detection prompt statement into the target large model to obtain the target anomaly detection result of the image to be detected. The target large model is obtained based on the model training method.
[0078] In this embodiment, a pre-trained large model, namely the target large model, is used for image detection. The target large model is obtained through the model training method described above.
[0079] Specifically, the image to be detected and the preset anomaly detection prompt statement are input into the target large model to obtain the target anomaly detection result of the image to be detected output by the target large model. The anomaly detection result can be the coordinate information of the anomaly detection area.
[0080] Optionally, the preset anomaly detection prompt statement may include the following: System: You are an anomaly detection assistant; User: <image>; Task: List all abnormal body parts in the format "[category name cx cy wh]", with each part separated by a comma and using normalized coordinates (0-1).
[0081] It should be noted that if no abnormal detection area is identified in the image to be detected, the target large model will output a detection result indicating that the image has no abnormalities.
[0082] The target large model is obtained through the model training method described above. Expanding the training data in this way significantly enhances the large model's tolerance to input noise, improves its robustness and generalization performance in noisy data scenarios, and thus increases the detection accuracy of the large model, effectively detecting anomalies in images. Image detection using the target large model yields more accurate detection results.
[0083] Figure 4 shows a structural diagram of the model training device provided in an embodiment of this application. As shown in Figure 4, the model training device 400 includes: a first acquisition module 401, used to acquire preset training samples, the preset training samples including a preset image, a preset anomaly detection prompt statement, and anomaly detection labels, wherein the preset image is generated by a preset image generation model; a first processing module 402, used to add noise to the image region of the preset image excluding the anomaly detection region according to the anomaly detection labels, to obtain a processed target image; a second processing module 403, used to generate a difference prompt statement according to the difference between the target image and the preset image, and update the preset anomaly detection prompt statement according to the difference prompt statement, to obtain a target anomaly detection prompt statement; a third processing module 404, used to combine the target image, the target anomaly detection prompt statement, and the anomaly detection labels to generate a set of target training samples; and a training module 405, used to combine the preset training samples and the target training samples to fine-tune the preset large model to obtain a target large model, the target large model being used to detect the image to be detected generated by the preset image generation model.
[0084] In one embodiment of this application, the first processing module 402 is further configured to generate image mask information for the anomaly detection area in the preset image based on the anomaly detection label; and to perform noise addition processing on the image area in the preset image other than the anomaly detection area based on the image mask information to obtain the processed target image.
[0085] In one embodiment of this application, the first processing module 402 is specifically used to generate random numbers using a random algorithm; when the random numbers meet preset conditions, a quality value is randomly selected from a preset quality range as a target value; based on the target value and the image mask information, the image region in the preset image other than the anomaly detection region is compressed to obtain the processed target image.
[0086] In one embodiment of this application, the first processing module 402 is specifically used to generate random numbers using a random algorithm; when the random numbers meet preset conditions, based on the image mask information, the brightness of the image region in the preset image excluding the anomaly detection region is adjusted to a preset brightness, and / or the saturation is adjusted to a preset saturation, and / or the contrast is adjusted to a preset contrast, to obtain the processed target image.
[0087] In one embodiment of this application, the first processing module 402 is specifically used to generate random numbers using a random algorithm; when the random numbers meet preset conditions, based on the image mask information, the pixel values of at least one pixel in a target range region of the image region other than the anomaly detection region in the preset image are modified to preset values to obtain the processed target image.
[0088] In one embodiment of this application, the first processing module 402 is specifically used to generate random numbers using a random algorithm; when the random numbers meet preset conditions, a variance is randomly selected within a preset variance range as the target variance; based on the target variance and the image mask information, Gaussian noise is added to the image region in the preset image other than the anomaly detection region to obtain the processed target image.
[0089] In one embodiment of this application, the training module 405 is specifically used to determine a loss value by combining the anomaly detection result output by the preset large model for anomaly detection region with the anomaly detection label during the training process of the preset large model using the preset training samples and the target training samples; and to fine-tune the preset large model when the loss value is greater than or equal to the preset loss value, until the loss value is less than the preset loss value, and to use the currently fine-tuned preset large model as the target large model.
[0090] The model training apparatus provided in this application embodiment can implement all the processes implemented in the aforementioned model training method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.
[0091] Figure 5 shows a structural diagram of the image detection device provided in an embodiment of this application. As shown in Figure 5, the image detection device 500 includes: a second acquisition module 501, used to acquire an image to be detected, the image to be detected being generated by a preset image generation model; and a detection module 502, used to input the image to be detected and a preset anomaly detection prompt statement into a target large model to obtain the target anomaly detection result of the image to be detected, the target large model being obtained based on the model training method described above.
[0092] The image detection apparatus provided in this application embodiment can implement all the processes implemented in the aforementioned image detection method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.
[0093] Figure 6 shows a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application.
[0094] The electronic device may include a processor 601 and a memory 602 storing computer program instructions.
[0095] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0096] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.
[0097] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this disclosure.
[0098] The processor 601 implements any of the methods described above in the above embodiments by reading and executing computer program instructions stored in the memory 602.
[0099] In one example, the electronic device may also include a communication interface 603 and a bus 610. As shown in Figure 6, the processor 601, memory 602, and communication interface 603 are connected via the bus 610 and communicate with each other.
[0100] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0101] Bus 610 includes hardware, software, or both, that couples components of a method or electronic device as described above together. For example, and not as a limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0102] Alternatively, embodiments of this application can be implemented using a computer storage medium. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the model training methods described in the above embodiments.
[0103] Alternatively, this application embodiment can provide a computer program product for implementation, wherein the instructions in the computer program product, when executed by the processor of an electronic device, cause the electronic device to implement any of the model training methods in the above embodiments.
[0104] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described as examples. However, the method process of this application is not limited to the specific steps described. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0105] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0106] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0107] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0108] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A model training method, characterized in that, The method includes: acquiring preset training samples, the preset training samples comprising a preset image, a preset anomaly detection prompt statement, and anomaly detection labels, wherein the preset image is generated by a preset image generation model; adding noise to image regions in the preset image other than anomaly detection regions according to the anomaly detection labels to obtain a processed target image; generating a difference prompt statement based on the difference between the target image and the preset image, and updating the preset anomaly detection prompt statement according to the difference prompt statement to obtain a target anomaly detection prompt statement; generating a set of target training samples by combining the target image, the target anomaly detection prompt statement, and the anomaly detection labels; and fine-tuning a preset large model by combining the preset training samples and the target training samples to obtain a target large model, the target large model being used to detect the image to be detected generated by the preset image generation model.
2. The model training method according to claim 1, characterized in that, The step of adding noise to the image region other than the anomaly detection region in the preset image according to the anomaly detection label to obtain the processed target image includes: generating image mask information for the anomaly detection region in the preset image according to the anomaly detection label; and adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image.
3. The model training method according to claim 2, characterized in that, The step of adding noise to the image region of the preset image, excluding the anomaly detection region, based on the image mask information to obtain the processed target image includes: generating a random number using a random algorithm; randomly selecting a quality value from a preset quality range as a target value when the random number meets preset conditions; and compressing the image region of the preset image, excluding the anomaly detection region, based on the target value and the image mask information to obtain the processed target image.
4. The model training method according to claim 2, characterized in that, The step of adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image includes: generating random numbers using a random algorithm; and, if the random numbers meet preset conditions, adjusting the brightness of the image region other than the anomaly detection region in the preset image to a preset brightness, and / or adjusting the saturation to a preset saturation, and / or adjusting the contrast to a preset contrast based on the image mask information to obtain the processed target image.
5. The model training method according to claim 2, characterized in that, The step of adding noise to the image region other than the anomaly detection region in the preset image based on the image mask information to obtain the processed target image includes: generating random numbers using a random algorithm; and modifying the pixel values of at least one pixel in a target range region of the image region other than the anomaly detection region in the preset image to a preset value based on the image mask information when the random numbers meet preset conditions, thereby obtaining the processed target image.
6. The model training method according to claim 2, characterized in that, The step of adding noise to the image regions in the preset image other than the anomaly detection region based on the image mask information to obtain the processed target image includes: generating random numbers using a random algorithm; randomly selecting a variance within a preset variance range as the target variance if the random numbers meet preset conditions; and adding Gaussian noise to the image regions in the preset image other than the anomaly detection region based on the target variance and the image mask information to obtain the processed target image.
7. The model training method according to any one of claims 1 to 6, characterized in that, The step of fine-tuning the preset large model by combining the preset training samples and the target training samples to obtain the target large model includes: during the training of the preset large model using the preset training samples and the target training samples, determining a loss value by combining the anomaly detection result output by the preset large model for the anomaly detection region with the anomaly detection label; if the loss value is greater than or equal to the preset loss value, fine-tuning the preset large model until the loss value is less than the preset loss value, and using the currently fine-tuned preset large model as the target large model.
8. An image detection method, characterized in that, The method includes: acquiring an image to be detected, the image to be detected being generated by a preset image generation model; inputting the image to be detected and a preset anomaly detection prompt statement into a target large model to obtain the target anomaly detection result of the image to be detected, wherein the target large model is obtained based on the model training method according to any one of claims 1-7.
9. A model training device, characterized in that, The apparatus includes: a first acquisition module for acquiring preset training samples, the preset training samples including a preset image, a preset anomaly detection prompt statement, and anomaly detection labels, wherein the preset image is generated by a preset image generation model; a first processing module for adding noise to image regions in the preset image other than anomaly detection regions according to the anomaly detection labels to obtain a processed target image; a second processing module for generating a difference prompt statement based on the difference between the target image and the preset image, and updating the preset anomaly detection prompt statement according to the difference prompt statement to obtain a target anomaly detection prompt statement; a third processing module for generating a set of target training samples by combining the target image, the target anomaly detection prompt statement, and the anomaly detection labels; and a training module for fine-tuning a preset large model by combining the preset training samples and the target training samples to obtain a target large model, the target large model being used to detect the image to be detected generated by the preset image generation model.
10. An image detection device, characterized in that, The device includes: a second acquisition module for acquiring an image to be detected, the image to be detected being generated by a preset image generation model; and a detection module for inputting the image to be detected and a preset anomaly detection prompt statement into a target large model to obtain a target anomaly detection result of the image to be detected, the target large model being obtained based on the model training method according to any one of claims 1-7.
11. An electronic device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the model training method as described in any one of claims 1-7, or the image detection method as described in claim 8.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the model training method as described in any one of claims 1-7, or the image detection method as described in claim 8.