Image processing method, training method and device for multimodal image generation model

Abstract

The present invention provides an image processing method, a training method and a device for a multimodal image generation model. The image processing method includes: inputting a text to be processed into a pre-trained multimodal image generation model, and outputting a target defect image; the multimodal image generation model is a model that combines a stable diffusion model and a target LoRA model; the target LoRA model is used to fine-tune the weight parameters of the stable diffusion model; the target LoRA model is a model that matches the defect category to which the defect feature information belongs. In this method, since the target LoRA model is a model that matches the defect category to which the defect feature information belongs, the multimodal image generation model that combines the target LoRA model and the stable diffusion model can accurately include the defects described by the defect feature information in the generated target defect image, thereby improving the convenience and controllability of generating the target defect image, and thus providing a number of defect images that meet the needs.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method, a training method and a device for a multimodal image generation model. Background Art

[0002] In industrial defect detection, the defect image to be detected is usually input into a pre-trained target detection model to detect the relevant defect data of the defect image to be detected. To ensure the accuracy of the detection results, sufficient defect image samples need to be provided when training the target detection model. In related technologies, defect images can be generated using an image generation model based on a generative adversarial network, but this method is difficult to accurately capture complex industrial defect features, resulting in a large difference between the generated defect images and the actual situation; traditional data enhancement algorithms can also be used for data enhancement, but this method is usually based on simple geometric transformations and image processing operations, which is difficult to accurately simulate the real features of industrial defects and cannot provide sufficient data diversity. Therefore, the problem of insufficient number of defect images required for industrial defect detection cannot be solved based on related technologies. Summary of the invention

[0003] The object of the present invention is to provide an image processing method, a training method and a device for a multimodal image generation model to solve the problem of insufficient number of defect images required for industrial defect detection.

[0004] The present invention provides an image processing method, which includes: obtaining a text to be processed; wherein the text to be processed includes defect feature information; inputting the text to be processed into a pre-trained multimodal image generation model, so as to output a target defect image corresponding to the text to be processed through the multimodal image generation model; wherein the multimodal image generation model is a model combining a stable diffusion model and a target LoRA model; the target LoRA model is used to fine-tune the weight parameters of the stable diffusion model; and the target LoRA model is a model that matches the defect category to which the defect feature information belongs.

[0005] The present invention provides a training method for a multimodal image generation model, the method comprising: obtaining a sub-defect image sample set of multiple defect categories; for each defect category, determining a sample image based on the sub-defect image sample set of the defect category; wherein the sample image has a corresponding target naming label; inputting the target naming label into an initial image generation model, so as to output a predicted image corresponding to the target naming label through the initial image generation model; wherein the initial image generation model includes a stable diffusion model and an initial LoRA model; the initial LoRA model is embedded in the stable diffusion model for fine-tuning the weight parameters of the stable diffusion model; based on the sample image and the predicted image, updating the parameters of the initial LoRA model, and continuing to perform the step of determining the sample image based on the sub-defect image sample set of the defect category until the initial LoRA model converges to obtain a target LoRA model; and obtaining a trained multimodal image generation model based on the target LoRA model and the stable diffusion model.

[0006] Furthermore, the step of obtaining sub-defect image sample sets of multiple defect categories includes: obtaining a target defect image sample set; classifying the target defect image sample set according to a preset classification method to obtain sub-defect image sample sets of multiple defect categories; wherein the preset classification method includes at least one of the following: classification by defect morphology, classification by defect size, and classification by defect color.

[0007] Furthermore, the step of obtaining a target defect image sample set includes: obtaining a first defect image sample; wherein the first defect image sample contains a first defect feature; cropping the first defect image sample to crop out a first defect area image corresponding to the first defect feature from the first defect image sample; performing a similarity comparison between the first defect area image and each image in a preset defect image set according to a preset similarity comparison algorithm to obtain a comparison result; and determining a target defect image sample set based on the comparison result.

[0008] Furthermore, the step of determining the target defect image sample set based on the comparison results includes: determining the image whose comparison result is greater than a preset similarity threshold as an enhanced defect image sample; cropping the enhanced defect image sample to crop out the second defect area image corresponding to the first defect feature from the enhanced defect image sample; and screening the first defect area image and the second defect area image to obtain the target defect image sample set.

[0009] Furthermore, the target naming label is determined in the following manner: for each defect category, an initial naming label is set for each sample image in the sub-defect image sample set of the defect category according to a preset label naming rule; for each sample image, the initial naming label of the sample image is sent to the basic model of the stable diffusion model to output a first result through the basic model; if the first result does not match the initial naming label, the initial naming label is determined as the target naming label of the sample image; if the first result matches the naming label, the initial naming label of the sample image is updated, and the step of sending the naming label of the sample image to the basic model in the stable diffusion model is repeated until the target naming label of the sample image is determined.

[0010] Furthermore, the tag naming rule includes: the initial naming tag includes project information and defect description information corresponding to the defect category.

[0011] The present invention provides an image processing device, which includes: a first acquisition module, used to acquire a text to be processed; wherein the text to be processed includes defect feature information; a first output module, used to input the text to be processed into a pre-trained multimodal image generation model, so as to output a target defect image corresponding to the text to be processed through the multimodal image generation model; wherein the multimodal image generation model is a model combining a stable diffusion model and a target LoRA model; the target LoRA model is used to fine-tune the weight parameters of the stable diffusion model; and the target LoRA model is a model that matches the defect category to which the defect feature information belongs.

[0012] The present invention provides a training device for a multimodal image generation model, which includes: a second acquisition module, used to acquire a sub-defect image sample set of multiple defect categories; a determination module, used to determine a sample image based on the sub-defect image sample set of each defect category; wherein the sample image has a corresponding target naming label; a second output module, used to input the target naming label into an initial image generation model, so as to output a predicted image corresponding to the target naming label through the initial image generation model; wherein the initial image generation model includes a stable diffusion model and an initial LoRA model; the initial LoRA model is embedded in the stable diffusion model and is used to fine-tune the weight parameters of the stable diffusion model; an update module, used to update the parameters of the initial LoRA model based on the sample image and the predicted image, and continue to perform the step of determining the sample image based on the sub-defect image sample set of the defect category until the initial LoRA model converges to obtain a trained multimodal image generation model.

[0013] An electronic device provided by the present invention includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned image processing method, or any of the above-mentioned multimodal image generation model training methods.

[0014] The present invention provides an image processing method, a training method and a device for a multimodal image generation model, wherein the image processing method comprises: obtaining a text to be processed; wherein the text to be processed includes defect feature information; inputting the text to be processed into a pre-trained multimodal image generation model, so as to output a target defect image corresponding to the text to be processed through the multimodal image generation model; wherein the multimodal image generation model is a model combining a stable diffusion model and a target LoRA model; the target LoRA model is used to fine-tune the weight parameters of the stable diffusion model; and the target LoRA model is a model that matches the defect category to which the defect feature information belongs. In this method, since the target LoRA model is a model that matches the defect category to which the defect feature information belongs, the multimodal image generation model combining the target LoRA model and the stable diffusion model can make the generated target defect image accurately contain the defects described by the defect feature information, thereby improving the convenience and controllability of generating the target defect image, and thus providing a number of defect images that meet the requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of an image processing method provided by an embodiment of the present invention;

[0018] Figure 3 A flowchart of a method for training a multimodal image generation model provided by an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of an initial naming label storage format and an initial naming label provided by an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of a training process of a multimodal image generation model provided by an embodiment of the present invention;

[0021] Figure 6 A flowchart of a method for training a multimodal image generation model provided by an embodiment of the present invention;

[0022] Figure 7 A flowchart of an image processing method provided by an embodiment of the present invention;

[0023] Figure 8 A schematic diagram of a defect image provided by an embodiment of the present invention;

[0024] Fig. 9 A schematic diagram of a defect image provided by an embodiment of the present invention;

[0025] Fig.10 A schematic diagram of a defect image provided by an embodiment of the present invention;

[0026] Fig.11 A schematic diagram of a defect image provided by an embodiment of the present invention;

[0027] Fig.12 A schematic diagram of a defect image provided by an embodiment of the present invention;

[0028] Fig.13 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention;

[0029] Fig.14 A schematic diagram of the structure of a training device for a multimodal image generation model provided by an embodiment of the present invention;

[0030] Fig.15 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In industrial defect detection, when training the target detection model, it is necessary to provide sufficient defect image samples. In the related art, an image generation model based on a generative adversarial network (GAN) and a traditional data enhancement method can be used to improve the problem of insufficient number of defect images in industrial defect detection. However, these methods have certain limitations. Specifically, the generative adversarial network performs well in generating images, but there are still certain challenges in the generation of industrial defects, especially in terms of increasing data diversity while maintaining defect features. It is often difficult to accurately capture complex defect features, resulting in the generated defect image not being close enough to the actual situation. Traditional data enhancement methods usually implement data enhancement based on simple geometric transformations and image processing operations, which makes it difficult to accurately simulate the real features of industrial defects, and usually cannot provide sufficient data diversity, and cannot meet the requirements of industrial defect detection algorithms for a large amount of labeled data. These shortcomings limit the performance and application scope of industrial defect detection algorithms. Therefore, the related art cannot solve the problem of insufficient number of defect images required for industrial defect detection, resulting in limited training and application of target detection models. Based on this, an embodiment of the present invention provides an image processing method, a training method and a device for a multimodal image generation model, which can be applied to applications that require the generation of industrial defect images.

[0033] To facilitate understanding of this embodiment, an image processing method disclosed in an embodiment of the present invention is first introduced. Figure 1 As shown, the method comprises the following steps:

[0034] Step S102, obtaining a text to be processed; wherein the text to be processed includes defect feature information;

[0035] The above-mentioned text to be processed is usually a text description related to the target defect image to be generated, and the text description usually includes some prompt words, such as: subject description, subject feature, background description, light, viewing angle, etc.; the above-mentioned defect feature information is usually related defect features in industrial applications, such as the screw area of ​​the car chassis is not tightened, the end face of the sleeve has crack features, etc. In actual implementation, when it is necessary to generate a target defect image, the relevant text to be processed can be obtained first. The text to be processed can be input by the user through a text box, or can be obtained from a pre-stored text set.

[0036] Step S104, inputting the text to be processed into a pre-trained multimodal image generation model to output a target defect image corresponding to the text to be processed through the multimodal image generation model; wherein the multimodal image generation model is a model combining a stable diffusion model and a target LoRA model; the target LoRA model is used to fine-tune the weight parameters of the stable diffusion model; the target LoRA model is a model that matches the defect category to which the defect feature information belongs.

[0037] The above-mentioned stable diffusion model can be specifically a Stable Diffusion model, which is a machine learning model that can generate corresponding images according to text descriptions. After training, it can gradually denoise random Gaussian noise to obtain images of interest. The above-mentioned target LoRA model is mainly used to deal with the problem of fine-tuning large models. The target LoRA model usually recommends freezing the weights of the stable diffusion model and injecting a trainable layer (rank-decomposition matrix) into each Transformer block. Since there is no need to calculate gradients for most model weights, the number of parameters that need to be trained can be greatly reduced, and the memory requirements of the GPU (graphics processing unit) are reduced. The above-mentioned defect categories can be: crack defects with a long length, untightened screw defects, etc.; specifically, multiple defect categories can be set according to actual needs. Usually, each defect category has its own corresponding target LoRA model. It can also be understood that there can be multiple target LoRA models, and each target LoRA model corresponds to a defect category, that is, the target LoRA model is trained based on sample images belonging to the defect category. In actual implementation, the text to be processed obtained above can be input into a multimodal image generation model that combines a stable diffusion model and a target LoRA model, and the target defect image corresponding to the text to be processed can be output, that is, the target defect image contains the defects described by the defect feature information.

[0038] See also Figure 2 The schematic diagram of an image processing method shown in FIG. 1 inputs a text description of an image to be generated (corresponding to the above-mentioned text to be processed) into a multimodal image generation model that combines a stable diffusion model Stable Diffusion and a target LoRA model, and outputs a target defect image corresponding to the text description.

[0039] The above-mentioned image processing method obtains the text to be processed; wherein the text to be processed includes defect feature information; the text to be processed is input into a pre-trained multimodal image generation model, so as to output the target defect image corresponding to the text to be processed through the multimodal image generation model; wherein the multimodal image generation model is a model combining a stable diffusion model and a target LoRA model; the target LoRA model is used to fine-tune the weight parameters of the stable diffusion model; the target LoRA model is a model that matches the defect category to which the defect feature information belongs. In this method, since the target LoRA model is a model that matches the defect category to which the defect feature information belongs, the multimodal image generation model combining the target LoRA model and the stable diffusion model can make the generated target defect image accurately contain the defects described by the defect feature information, thereby improving the convenience and controllability of generating the target defect image, and thus providing a number of defect images that meet the needs.

[0040] The embodiment of the present invention provides a training method for a multimodal image generation model, such as Figure 3 As shown, the method comprises the following steps:

[0041] Step S302, obtaining a sub-defect image sample set of multiple defect categories;

[0042] The number of the above defect categories can be set according to actual needs. For example, the defect categories may include: crack defects with a long length, untightened screw defects, etc.; in actual implementation, multiple defect categories can be pre-set, and sub-defect image sample sets corresponding to each defect category can be obtained; each sub-defect image sample set usually includes a large number of sample images.

[0043] Step S304, for each defect category, determining a sample image based on the sub-defect image sample set of the defect category; wherein the sample image has a corresponding target naming label;

[0044] The above-mentioned target naming labels can be understood as the result of labeling the defects corresponding to the sample images; in actual implementation, when the initial image generation model needs to be trained, it is usually necessary to collect a large number of sample images. For each defect category, the sample image can be determined according to the sub-defect image sample set of the defect category.

[0045] Step S306, inputting the target naming label into the initial image generation model, so as to output the predicted image corresponding to the target naming label through the initial image generation model; wherein the initial image generation model includes a stable diffusion model and an initial LoRA model; the initial LoRA model is embedded in the stable diffusion model, and is used to fine-tune the weight parameters of the stable diffusion model;

[0046] In actual implementation, after determining the sample image, the target naming label corresponding to the sample image can be input into the initial image generation model that combines the stable diffusion model and the initial LoRA model, and the predicted image is generated by the initial image generation model.

[0047] Step S308, based on the sample image and the predicted image, update the parameters of the initial LoRA model, continue to perform the step of determining the sample image based on the sub-defect image sample set of the defect category, until the initial LoRA model converges to obtain the target LoRA model; obtain the trained multimodal image generation model based on the target LoRA model and the stable diffusion model.

[0048] In actual implementation, the initial LoRA model can be trained based on the sample image. When training the initial LoRA model, it is usually necessary to update the weight parameters of the initial LoRA model based on the above sample image and the predicted image to train the initial LoRA model, and then continue to perform the step of determining the sample image based on the sub-defect image sample set of the defect category until the initial LoRA model converges to obtain the converged target LoRA model, and embed the weight parameters of the target LoRA model into the stable diffusion model to fine-tune the weight parameters of the stable diffusion model to obtain a trained multimodal image generation model. According to the above training process, the target LoRA model corresponding to each defect category can be trained.

[0049] The training method of the above-mentioned multimodal image generation model obtains a sample set of sub-defect images of multiple defect categories; for each defect category, a sample image is determined based on the sample set of sub-defect images of the defect category; wherein the sample image has a corresponding target naming label; the target naming label is input into the initial image generation model to output a predicted image corresponding to the target naming label through the initial image generation model; wherein the initial image generation model includes a stable diffusion model and an initial LoRA model; the initial LoRA model is embedded in the stable diffusion model for fine-tuning the weight parameters of the stable diffusion model; based on the sample image and the predicted image, the parameters of the initial LoRA model are updated, and the step of determining the sample image based on the sample set of sub-defect images of the defect category is continued until the initial LoRA model converges to obtain the target LoRA model; and a trained multimodal image generation model is obtained based on the target LoRA model and the stable diffusion model. This method can train a target LoRA model corresponding to each defect category, and fine-tune the weight parameters of the stable diffusion model based on the target LoRA model to obtain a trained multimodal image generation model. On the one hand, it improves the accuracy of the image generation model, and on the other hand, it is beneficial to model learning and improves the generalization ability of the multimodal image generation model.

[0050] The embodiment of the present invention also provides another method for training a multimodal image generation model, the method comprising the following steps:

[0051] Step 1, obtaining a target defect image sample set;

[0052] The step 1 can be implemented by the following steps 10 to 13:

[0053] Step 10, obtaining a first defect image sample; wherein the first defect image sample includes a first defect feature;

[0054] The above-mentioned first defect image samples are usually small batches of real defect images collected; each first defect image sample contains a first defect feature, and the first defect feature contained in each first defect image sample may be the same or different. For example, the first defect image sample is an image of a car chassis, and the first defect feature is that the screws in the screw area of ​​the car chassis are not tightened, etc.

[0055] Step 11, performing a cropping process on the first defect image sample to crop a first defect region image corresponding to the first defect feature from the first defect image sample;

[0056] Each first defect image sample may be cropped to crop a first defect region image containing a first defect feature from each first defect image sample. The first defect region image is usually a rectangular image or the like.

[0057] Step 12, performing a similarity comparison between the first defect area image and each image in a preset defect image set according to a preset similarity comparison algorithm to obtain a comparison result;

[0058] The above-mentioned similarity comparison algorithm can be implemented by using an SSIM (Structural Similarity) algorithm, etc., wherein the SSIM algorithm is an indicator for measuring the similarity between two images, which measures image similarity from three aspects: brightness, contrast, and structure. The above-mentioned defect image set usually contains a large number of defect images, which can be pre-stored in a database. In actual implementation, for each first defect area image, the first defect area image can be compared with each image in the defect image set by using an SSIM or other algorithm to obtain a comparison result corresponding to each image. The comparison result can be used to indicate the degree of similarity between the first defect area image and the image. For example, the comparison result can be between -1 and 1, and the larger the value, the higher the similarity.

[0059] Step 13: determine the target defect image sample set according to the comparison result.

[0060] The step 13 can be specifically implemented through steps A to C:

[0061] Step A, determining an image whose comparison result is greater than a preset similarity threshold as an enhanced defect image sample;

[0062] The above-mentioned similarity threshold can be set according to actual needs, for example, it can be 0.6, 0.7, etc. In actual implementation, if the comparison result is greater than the preset similarity threshold, it can be considered that the image is similar to the first defect area image, and the image with a comparison result greater than the preset similarity threshold can be extracted from the multiple comparison results obtained to obtain an enhanced defect image.

[0063] Step B, performing a cropping process on the enhanced defect image sample to crop a second defect region image corresponding to the first defect feature from the enhanced defect image sample;

[0064] Each enhanced defect image sample may be cropped to crop a second defect region image containing the first defect feature from each enhanced defect image sample. The second defect region image is usually a rectangular image.

[0065] Step C: screening the first defect area image and the second defect area image to obtain a target defect image sample set.

[0066] In actual implementation, an image blur algorithm can be used to evaluate the image clarity of each first defect area image and each second defect area image. For example, a variance gradient algorithm, an edge gradient algorithm, and other algorithms can be used to screen out images with unclear imaging quality from each first defect area image and each second defect area image to obtain a target defect image sample set. In order to further ensure the accuracy and feature consistency of the target defect image sample set obtained after screening, the screened target defect image sample set can be manually inspected and verified again to ensure the reliability of subsequent processing and training.

[0067] Step 2: classify the target defect image sample set according to a preset classification method to obtain sub-defect image sample sets of multiple defect categories; wherein the preset classification method includes at least one of the following: classification by defect morphology, classification by defect size, and classification by defect color.

[0068] The above-mentioned defect morphology can be understood as the appearance state of the defect; for example, cracks, scratches, etc.; in actual implementation, the target defect image sample set can be classified according to defect morphology, defect size and / or defect color, etc., to obtain sub-defect image sample sets corresponding to multiple defect categories, wherein each sub-defect image sample set usually includes multiple sample images.

[0069] Step 3, for each defect category, determine a sample image based on a sub-defect image sample set of the defect category; wherein the sample image has a corresponding target naming label;

[0070] The target naming label is determined by the following steps 30 to 33:

[0071] Step 30, for each defect category, according to a preset label naming rule, an initial naming label is set for each sample image in the sub-defect image sample set of the defect category;

[0072] The above-mentioned tag naming rules include: the initial naming tag includes project information and defect description information corresponding to the defect category; in one embodiment, the initial naming tag includes project number information, project name information and defect description information corresponding to the defect category. The above-mentioned project number information can be understood as the number of the project to which the sample image belongs; the above-mentioned project name information indicates the name of the project to which the sample image belongs; the defect description information can be understood as the description information used to indicate the defect category, for example, "long" can represent a crack defect with a longer length, etc.; in actual implementation, for each defect category, a txt file corresponding to the sample image can be configured according to the image name of each sample image in the sub-defect image sample set of the defect category. The name of the txt file is the same as the name of the corresponding sample image. The txt file is used as a carrier to store the initial naming label of the sample image in the txt file. The initial naming label can be set according to the above-mentioned label naming rules. For example, the above-mentioned initial naming label can be set according to the project number information, the project name information, the defect description information, and the underscore as a separator. For example, the project number information is "XX", the pinyin abbreviation of the project name information is "kl", and the defect description information is "long", then the initial naming label can be set to: "xx_kl_long" according to the naming rules.

[0073] For example, see Figure 4 A schematic diagram of an initial naming label storage format and an initial naming label is shown, wherein, in the left picture, in each pair of sample images and txt files with the same name, what is saved in the txt file is the initial naming label of the sample image, for example, the initial naming label of the first sample image saved in the first txt file is "chongya_kl_long".

[0074] The reason for naming with this label naming rule is to make the initial image generation model completely distinguishable from natural language words during training. Because in the initial image generation model, natural language information has corresponding image information, so this embodiment standardizes a symbol that can be effectively distinguished from natural language and can represent the defect category to name it. The initial naming label made according to this specification can enable the initial image generation model to effectively learn the correspondence between the defect category and its unique corresponding language symbol, thereby preventing semantic duplication between the language symbol and other natural language words.

[0075] Step 31, for each sample image, sending the initial naming label of the sample image to the basic model of the stable diffusion model, so as to output a first result through the basic model;

[0076] The basic model of the above-mentioned stable diffusion model can specifically be the base model in the Stable Diffusion model, and the base model can be used for text-generated images. In specific implementation, the base model of the Stable Diffusion model can be used in the text-generated images and the image local redrawing functions, and the initial naming labels can be combined to perform multimodal image generation to obtain the first result.

[0077] Step 32, if the first result does not match the initial naming label, determine the initial naming label as the target naming label of the sample image;

[0078] If the first result does not match the initial naming label, it can be considered that the generated first result has no real symbolic meaning. In this case, the initial naming label can be used as the target naming label of the sample image.

[0079] Step 33: if the first result matches the named label, the initial named label of the sample image is updated, and the step of sending the named label of the sample image to the base model in the stable diffusion model is repeated until the target named label of the sample image is determined.

[0080] If the first result matches the named label, it can be considered that the generated first result has realistic symbolic meaning, indicating that the initial named label has been trained in the base model of the Stable Diffusion model and has a specific image feature meaning. At this time, the initial named label needs to be updated. For example, the information in the initial named label can be further amplified, and the steps of sending the named label of the sample image to the base model in the stable diffusion model are repeated until the target named label of the sample image is determined. This can avoid the initial image generation model from training named labels that already have specific meanings, thereby preventing the generated defect features from being unstable.

[0081] Step 4: input the target naming label into the initial image generation model, so as to output the predicted image corresponding to the target naming label through the initial image generation model; wherein the initial image generation model includes a stable diffusion model and an initial LoRA model; the initial LoRA model is embedded in the stable diffusion model to fine-tune the weight parameters of the stable diffusion model;

[0082] Step five, based on the sample image and the predicted image, update the parameters of the initial LoRA model, and continue to perform the step of determining the sample image based on the sub-defect image sample set of the defect category until the initial LoRA model converges to obtain the target LoRA model; based on the target LoRA model and the stable diffusion model, obtain the trained multimodal image generation model.

[0083] For easier understanding, see Figure 5 The diagram shows a training process of a multimodal image generation model. The Stable Diffusion model is a generation model that can be trained to denoise objects such as images to obtain images of interest. The Stable Diffusion model is trained to slightly denoise the image in each step until the image of interest is obtained. The LoRA large model fine-tuning algorithm is used to train the parameter nodes of the additional initial LoRA model based on the Stable Diffusion model as the base model, so that the Stable Diffusion model has the ability to generate specific defect features after combining the trained target LoRA model. Figure 5 In the method, the input target naming label and the sample image are one-to-one corresponding. The initial image generation model includes a stable diffusion model Stable Diffusion and an initial LoRA model. The weight of the initial LoRA model is embedded in the stable diffusion model Stable Diffusion. According to the input target naming label, the output predicted image can be obtained. The loss value can be determined according to the predicted image and the sample image. The weight parameters of the initial LoRA model are continuously updated until the loss value converges. Then, the trained target LoRA model can be obtained. The weight parameters of the target LoRA model are embedded in the stable diffusion model StableDiffusion to fine-tune the weight parameters of the stable diffusion model Stable Diffusion. Then, the trained multimodal image generation model can be obtained.

[0084] It can be seen from the above embodiments that this method has the ability of defect-oriented learning. In order to enable the multimodal image generation model to have the ability to generate specified defect features in a targeted manner, the target LoRA model corresponding to the defect category can be trained separately for different defect categories. In this way, a target LoRA model can have the ability to generate corresponding defect categories. By using the stable diffusion model Stable Diffusion combined with the target LoRA model, new enhanced defect image samples can be generated, which can be used to expand the training data set or test data set to improve the detection ability of subsequent target detection models for different defect categories.

[0085] In actual implementation, for the convenience of training, the pixel size of all sample images can be uniformly adjusted to 256×256; the rank of the LoRA model is set to 256. A larger rank is more conducive to adjusting the ability of the stable diffusion model. Although it will lose a certain amount of training efficiency, it will enable the LoRA model to learn more information from the training data.

[0086] In order to accurately generate target defect images that meet the needs based on the stable diffusion model and the target LoRA model, according to the project characteristics, local image redrawing can be used for areas where defects are prone to occur and difficult to detect. Combining the Stable Diffusion base model with the trained target LoRA model and the label characters of the corresponding features, defect features can be generated in the specified area of ​​the image. Among them, specifying different target LoRA models can generate defects with different features, and the size of the generated defects can be controlled by the size of the redrawing area, realizing precise control of defect feature generation.

[0087] In actual implementation, the weight of the target LoRA model can be set to 1.2. Since the defect features are relatively simple and the cosine similarity of the features is similar, a larger weight can make the generated defect features more similar to the real features. The redraw area is set to the mask area that is only painted. The sampling step is set to 24. The sampling method is set to "DPM SDE Karras". The generated width and height are set to be consistent with the LoRA training width and height, that is, 256×256. The prompt word relevance is set to 27.5. The redraw amplitude is set to 0.85. In the LoRA layer control, the OUTS layer weight of the target LoRA model is set from 1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1 to 1,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0. The last two weights of the deep layer of the network are changed to 0, and the others remain unchanged. This can change the overall morphology while retaining the detailed information of the generated features, thereby increasing the generalization of the defect features generated by the model.

[0088] The above settings are all targeted setting methods suitable for defect generation, which are generally related and not suitable for natural image generation. It should be noted that the above parameters are preferred parameters and can be adjusted according to actual needs in practical applications, and are not limited here.

[0089] like Figure 6 A flowchart of a training method for a multimodal image generation model is shown, which includes four stages, namely: a data collection stage, a data annotation stage, a model training stage and a defect generation stage; in the data collection stage, a small batch of defect images to be detected can be collected, and a first defect local rectangular image can be cropped out (corresponding to the above-mentioned acquisition of the first defect image sample, the first defect image sample is cropped to crop out a first defect area image corresponding to the first defect feature from the first defect image sample), a similar enhanced defect image sample is matched in the defect database using the SSIM algorithm, and a second defect local rectangular image is cropped out (corresponding to the above-mentioned determination of an image with a comparison result greater than a preset similarity threshold as an enhanced defect image sample; the enhanced defect image sample is cropped to crop out a second defect area image corresponding to the first defect feature from the enhanced defect image sample), the first defect local rectangular image and the second defect local rectangular image are manually screened to extract a target defect image sample set. In the data annotation stage, the collected small batch defect images and enhanced defect image samples can be named, and can be named according to the following label naming rules: project name information_defect category 1; the defective parts in the small batch defect images and enhanced defect image samples can be masked and labeled with initial naming labels, and can be named according to the following label naming rules: project name information_defect category 2; the initial naming labels can be corrected. Specifically, the initial naming labels can be sent to the base model of the stable diffusion model StableDiffusion for image generation. If meaningless noise is generated, there is no need to change the initial naming labels, and they can be directly used as target naming labels; if meaningful images are generated, the initial naming labels need to be updated and repeatedly verified through the base model until the target naming labels are obtained. The initial LoRA model parameter adjustment training is performed based on the sorted target naming labels to obtain a trained target LoRA model. The trained target LoRA models of different defect categories are combined with the stable diffusion model StableDiffusion to generate target defect images of different defect categories.

[0090] See also Figure 7The flowchart of an image processing method shown in the figure can use the generated target defect image to train a preset target detection model, infer the real defect data of the target defect image through the target detection model, deploy the trained target detection model to the device, and test the existing real defects to determine whether there are any missed detections. If there are missed detections, it means that the defect actually exists, but the defect is not included in the target defect image generated above. In this way, the similarity of the missed defect feature image and each image in the defect image set in the database is compared, and the images that meet the similarity threshold are extracted. The steps in the above embodiment are repeated to construct a new target detection model to form a closed-loop management.

[0091] For easier understanding, see Figure 8 The schematic diagram of a defect image shown in FIG. 1 shows that in the image of the chassis of a car, the trained multimodal image generation model is used to generate defect features of no screws in the area with screws by using local redrawing. Figure 8 The left image in the figure shows a defect in which the screw is not actually tightened. Figure 8 The right image in the figure is a target defect image of an untightened screw in the screw area generated by a multimodal image generation model after systematic training.

[0092] See also Fig. 9 A schematic diagram of a defect image is shown, wherein: Fig. 9 The left image in the figure shows a defect in which the screw is not actually tightened. Fig. 9 The right image in the figure is a target defect image of an untightened screw in the screw area generated by a multimodal image generation model after systematic training.

[0093] See also Fig.10 A schematic diagram of a defect image is shown, wherein: Fig.10 The left image in the figure shows a defect in which the screw is not actually tightened. Fig.10 The right image in the figure is a target defect image of an untightened screw in the screw area generated by a multimodal image generation model after systematic training.

[0094] See also Fig.11 The schematic diagram of a defect image shown in FIG. 1 generates a crack characteristic defect for a local tensile area prone to cracking on a stamping image of a car body. Fig.12 The schematic diagram of a defect image shown is a crack characteristic defect generated on the end face of the sleeve.

[0095] In the related technology, the stable diffusion model Stable Diffusion is currently generally used in the fields of illustrations and film and television, and has not yet been applied to the field of industrial defect generation, and the generated defects also require corresponding defect data sets, which have certain barriers. When generating target defect images, a special generation mode is required; this embodiment provides a training method for a multimodal image generation model. First, combined with the real defect data of a specific project, the feature similarity algorithm is used to extract relevant images in the database and annotate the target naming labels. Then, on the basis of the stable diffusion model Stable Diffusion, the LoRA large model fine-tuning technology is used to fine-tune the data to the stable diffusion model Stable Diffusion, so that the stable diffusion model Stable Diffusion can generate defect images corresponding to the target naming labels. Through the synergy of multiple subsystems such as LoRA model training, defect image generation and target detection model training, the generation, optimization and expansion of industrial defect image data are realized. The defect images generated by this system can effectively solve the problem of insufficient data in industrial defect detection, thereby improving the accuracy and robustness of the target detection model.

[0096] The training method of the multimodal image generation model achieves efficient generation and optimization of industrial defect image data through the combination and synergy of the above-mentioned processing stages. By training the multimodal image generation model for different defect features and using the defect images generated by the multimodal image generation model, the problem of lack of defect images in industrial defect detection is solved. The systematic solution of this embodiment makes the generation of defect images easy, controllable and rich. The generated defect images can meet the needs of industrial defect detection algorithms for a large amount of annotated data. By combining the generated defect images with the original images for training, the accuracy and robustness of the target detection model in industrial defect detection are improved.

[0097] Different from traditional industrial defect image generation methods such as GAN generation, image pasting or traditional Poisson fusion, this method uses a multimodal generation algorithm based on the stable diffusion model Stable Diffusion and the target LoRA model to generate industrial defect image features; by using the large model fine-tuning technology of the LoRA algorithm and combining it with the stable diffusion model Stable Diffusion, it is possible to accurately control the size, shape, position, etc. of the defect feature generation.

[0098] This method provides a set of multimodal image generation model training methods suitable for generating defect images, which includes a label naming rule for defects, which can meet the training of defect detection algorithms in different demand scenarios. This method provides a method and steps for controlling the generation of specific defects, and provides a parameter setting suitable for defect generation, so that the generated defects are more consistent with the distribution of real features.

[0099] The training method of the multimodal image generation model proposed in this way is a standardized system, in which the data collection stage corresponds to the "intelligent defect data preprocessing and selection system", the data annotation stage corresponds to the "generation algorithm data annotation system", the model training stage corresponds to the "defect feature directional training system", and the defect generation stage corresponds to the "accurate defect generation system based on stable diffusion module and target LoRA model". Based on this system, high-quality defect images can be efficiently generated to solve the problem of lack of defect images in target detection algorithms in defect detection in the industrial field.

[0100] This method provides a method for constructing the richness of defect images. It uses an image similarity algorithm to extract and classify a large number of related defect images in the database. The feature richness of defects can be guaranteed based on the trained target LoRA module. The local redrawing function of the stable diffusion model Stable Diffusion can generate defects with the required features in any area of ​​the image, ensuring the position richness of the defects. This method is innovative and has application value in the field of industrial defect image generation. The comprehensive application of the above technical means makes the generated defect images more realistic and credible, providing better data support for the training and optimization of subsequent target detection models.

[0101] The embodiment of the present invention provides an image processing device, such as Fig.13 As shown, the device comprises:

[0102] The first acquisition module 1300 is used to acquire the text to be processed; wherein the text to be processed includes defect feature information;

[0103] The first output module 1301 is used to input the text to be processed into a pre-trained multimodal image generation model, so as to output a target defect image corresponding to the text to be processed through the multimodal image generation model;

[0104] Among them, the multimodal image generation model is a model that combines the stable diffusion model and the target LoRA model; the target LoRA model is used to fine-tune the weight parameters of the stable diffusion model; the target LoRA model is a model that matches the defect category to which the defect feature information belongs.

[0105] The above-mentioned image processing device, since the target LoRA model is a model that matches the defect category to which the defect feature information belongs, therefore, through a multimodal image generation model that combines the target LoRA model with a stable diffusion model, the generated target defect image can accurately include the defects described by the defect feature information, thereby improving the convenience and controllability of generating the target defect image, and thus being able to provide a number of defect images that meet the needs.

[0106] The embodiment of the present invention provides a training device for a multimodal image generation model, such as Fig.14 As shown, the device comprises:

[0107] The second acquisition module 141 is used to acquire a sub-defect image sample set of multiple defect categories;

[0108] A determination module 142 is used to determine, for each defect category, a sample image based on a sub-defect image sample set of the defect category; wherein the sample image has a corresponding target naming label;

[0109] The second output module 143 is used to input the target naming label into the initial image generation model, so as to output the predicted image corresponding to the target naming label through the initial image generation model; wherein the initial image generation model includes a stable diffusion model and an initial LoRA model; the initial LoRA model is embedded in the stable diffusion model, and is used to fine-tune the weight parameters of the stable diffusion model;

[0110] The updating module 144 is used to update the parameters of the initial LoRA model based on the sample image and the predicted image, and continue to perform the step of determining the sample image based on the sub-defect image sample set of the defect category until the initial LoRA model converges to obtain the target LoRA model; and obtain the trained multimodal image generation model based on the target LoRA model and the stable diffusion model.

[0111] The training device of the above-mentioned multimodal image generation model can train a target LoRA model corresponding to each defect category, and fine-tune the weight parameters of the stable diffusion model based on the target LoRA model to obtain a trained multimodal image generation model. On the one hand, it improves the accuracy of the image generation model, and at the same time, it is also beneficial to model learning and improves the generalization ability of the multimodal image generation model.

[0112] Furthermore, the second acquisition module is also used to: obtain a target defect image sample set; classify the target defect image sample set according to a preset classification method to obtain sub-defect image sample sets of multiple defect categories; wherein the preset classification method includes at least one of the following: classification by defect morphology, classification by defect size, and classification by defect color.

[0113] Furthermore, the second acquisition module is also used to: acquire a first defect image sample; wherein the first defect image sample contains a first defect feature; perform cropping processing on the first defect image sample to crop out a first defect area image corresponding to the first defect feature from the first defect image sample; perform similarity comparison between the first defect area image and each image in a preset defect image set according to a preset similarity comparison algorithm to obtain a comparison result; and determine a target defect image sample set based on the comparison result.

[0114] Furthermore, the second acquisition module is also used to: determine an image whose comparison result is greater than a preset similarity threshold as an enhanced defect image sample; crop the enhanced defect image sample to crop out a second defect area image corresponding to the first defect feature from the enhanced defect image sample; and screen the first defect area image and the second defect area image to obtain a target defect image sample set.

[0115] Furthermore, the device also includes a target naming label determination module, and the target naming label is determined by the target naming label determination module. The target naming label determination module is used to: for each defect category, according to a preset label naming rule, set an initial naming label for each sample image in the sub-defect image sample set of the defect category; for each sample image, send the initial naming label of the sample image to the basic model of the stable diffusion model to output a first result through the basic model; if the first result does not match the initial naming label, determine the initial naming label as the target naming label of the sample image; if the first result matches the naming label, update the initial naming label of the sample image, and repeat the step of sending the naming label of the sample image to the basic model in the stable diffusion model until the target naming label of the sample image is determined.

[0116] Furthermore, the tag naming rule includes: the initial naming tag includes project information and defect description information corresponding to the defect category.

[0117] The training device for a multimodal image generation model provided in an embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned training method embodiment for a multimodal image generation model. For the sake of brief description, for matters not mentioned in the embodiment of the training device for a multimodal image generation model, reference may be made to the corresponding contents in the aforementioned training method embodiment for a multimodal image generation model.

[0118] The embodiment of the present invention further provides an electronic device, see Fig.15 As shown, the electronic device includes a processor 130 and a memory 131, wherein the memory 131 stores machine executable instructions that can be executed by the processor 130, and the processor 130 executes the machine executable instructions to implement the above-mentioned image processing method and the training method of the multimodal image generation model.

[0119] Further, Fig.15 The electronic device shown further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

[0120] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0121] The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a multimodal image generation model, characterized in that: The method comprises: Obtaining a sub-defect image sample set of multiple defect categories; For each defect category, a sample image is determined based on a sub-defect image sample set of the defect category; wherein the sample image has a corresponding target naming label; Inputting the target naming label into an initial image generation model to output a predicted image corresponding to the target naming label through the initial image generation model; wherein the initial image generation model includes a stable diffusion model and an initial LoRA model; the initial LoRA model is embedded in the stable diffusion model to fine-tune the weight parameters of the stable diffusion model; Based on the sample image and the predicted image, the parameters of the initial LoRA model are updated, and the step of determining the sample image based on the sub-defect image sample set of the defect category is continued until the initial LoRA model converges to obtain a target LoRA model; based on the target LoRA model and the stable diffusion model, a trained multimodal image generation model is obtained; The target naming label is determined in the following manner: For each defect category, an initial naming label is set for each sample image in the sub-defect image sample set of the defect category according to a preset label naming rule; For each sample image, the initial naming label of the sample image is sent to the basic model of the stable diffusion model to output the first result through the basic model; wherein the initial naming label includes project number information, project name information and defect description information corresponding to the defect category; the project number information is the number of the project to which the sample image belongs; the project name information indicates the name of the project to which the sample image belongs; the defect description information is used to indicate the description information of the defect category; according to the image name of the sample image, a txt file corresponding to the sample image is configured, and the initial naming label of the sample image is stored in the txt file; If the first result does not match the initial naming label, determining the initial naming label as the target naming label of the sample image; If the first result matches the named label, the initial named label of the sample image is updated, and the step of sending the named label of the sample image to the base model in the stable diffusion model is repeated until a target named label of the sample image is determined.

2. The method according to claim 1, characterized in that The steps of obtaining a sub-defect image sample set of multiple defect categories include: Obtain a target defect image sample set; Classifying the target defect image sample set according to a preset classification method to obtain sub-defect image sample sets of multiple defect categories; Among them, the preset classification method includes at least one of the following: classification by defect shape, classification by defect size, and classification by defect color.

3. The method according to claim 2, characterized in that The steps of obtaining a target defect image sample set include: Acquire a first defect image sample; wherein the first defect image sample contains a first defect feature; Performing cropping processing on the first defect image sample to crop a first defect region image corresponding to the first defect feature from the first defect image sample; According to a preset similarity comparison algorithm, the first defect area image is compared with each image in a preset defect image set for similarity to obtain a comparison result; A target defect image sample set is determined according to the comparison result.

4. The method according to claim 3, characterized in that The step of determining a target defect image sample set according to the comparison result comprises: An image whose comparison result is greater than a preset similarity threshold is determined as an enhanced defect image sample; Performing a cropping process on the enhanced defect image sample to crop a second defect region image corresponding to the first defect feature from the enhanced defect image sample; The first defect area image and the second defect area image are screened to obtain a target defect image sample set.

5. The method according to claim 1, characterized in that: The tag naming rule includes: the initial naming tag includes project information and defect description information corresponding to the defect category.

6. A training device for a multimodal image generation model, characterized in that: The device comprises: The second acquisition module is used to acquire sub-defect image sample sets of multiple defect categories; A determination module, configured to determine, for each defect category, a sample image based on a sub-defect image sample set of the defect category; wherein the sample image has a corresponding target naming label; A second output module is used to input the target naming label into the initial image generation model to output the predicted image corresponding to the target naming label through the initial image generation model; wherein the initial image generation model includes a stable diffusion model and an initial LoRA model; the initial LoRA model is embedded in the stable diffusion model to fine-tune the weight parameters of the stable diffusion model; An updating module, used for updating the parameters of the initial LoRA model based on the sample image and the predicted image, and continuing to perform the step of determining the sample image based on the sub-defect image sample set of the defect category until the initial LoRA model converges to obtain a trained multimodal image generation model; The device further comprises: a target naming label determination module; the target naming label determination module is used to: For each defect category, an initial naming label is set for each sample image in the sub-defect image sample set of the defect category according to a preset label naming rule; For each sample image, the initial naming label of the sample image is sent to the basic model of the stable diffusion model to output the first result through the basic model; wherein the initial naming label includes project number information, project name information and defect description information corresponding to the defect category; the project number information is the number of the project to which the sample image belongs; the project name information indicates the name of the project to which the sample image belongs; the defect description information is used to indicate the description information of the defect category; according to the image name of the sample image, a txt file corresponding to the sample image is configured, and the initial naming label of the sample image is stored in the txt file; If the first result does not match the initial naming label, determining the initial naming label as the target naming label of the sample image; If the first result matches the named label, the initial named label of the sample image is updated, and the step of sending the named label of the sample image to the base model in the stable diffusion model is repeated until a target named label of the sample image is determined.

7. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the training method of the multimodal image generation model described in any one of claims 1-5.

Images