A method and system for fine-tuning a text-to-image diffusion model based on decay testing

Abstract

The application discloses a kind of based on text generation graph diffusion model fine-tuning method and system of transmutation test, method includes the following steps: constructing the use case set including transmutation relationship set and entity set, test text is constructed based on use case set;Test text is input into text generation graph diffusion model and the generated image is obtained, image analysis is carried out to the generated image using detection model, and transmutation relationship check is executed based on image analysis result;Optimization objective function of text generation graph diffusion model is constructed based on transmutation relationship check result, and the parameter of text generation graph diffusion model is iteratively updated using optimization objective function and its gradient, and finally the fine-tuned text generation graph diffusion model is obtained.The application forms a complete automated text generation graph diffusion model fine-tuning scheme, can effectively enhance the understanding of text details of text generation graph diffusion model, to improve the performance of model on specific task.

Description

Technical Field

[0001] This invention belongs to the field of computer artificial intelligence technology, specifically relating to a method and system for fine-tuning a text-based graph diffusion model based on metamorphic testing. Background Technology

[0002] Text-to-image (TPE) models are generative models designed to generate images by learning from large amounts of text and image data. Their development has gone through several stages, with one significant advancement being the diffusion model. Inspired by the physics of gas diffusion, the diffusion model, compared to previous TPE models, better captures long-term dependencies and global structures in language, thus generating more artistic images. Only when the model accurately captures and parses the semantic information in the text can it generate images that match the text description; otherwise, the generated images may not match the text description, severely impacting user experience and the model's practical application value. Therefore, how to align the model with the semantics of the input text and generate high-quality images is a crucial research direction for researchers.

[0003] In real-world applications, users often utilize pre-trained text-based image diffusion models, which perform exceptionally well across various tasks, such as generating high-quality images from text descriptions in creative art design, advertising and marketing, game development, assisted writing, education and science popularization, and virtual and augmented reality. However, pre-trained text-based images on large-scale datasets may not fully meet the requirements of specific tasks. Therefore, fine-tuning the text-based image diffusion model is necessary to adapt it to specific domains or tasks. Fine-tuning allows adjustments to the model's parameters, making it better suited to the needs of specific tasks and enhancing its practicality and applicability. Thus, fine-tuning the text-based image diffusion model is a crucial step in improving its effectiveness in practical applications.

[0004] Among them, fine-tuning methods based on reinforcement learning, which can improve the alignment problem between input text and images, are a current research hotspot. However, current reinforcement learning strategies have obvious drawbacks: First, most existing methods rely on human feedback or additionally trained reward models, a process that consumes a lot of human and material resources; second, current fine-tuning goals based on reinforcement learning focus on the artistry of the generated images and the degree of alignment between the input text and the generated images, but in practice, it has been found that simply emphasizing the alignment between text and images is insufficient to enhance the text-to-image diffusion model's ability to perceive text details. In real-world scenarios, users often require the text-to-image diffusion model to generate corresponding images based on details in the text, such as a specific number of entities, a special scene, and a style. Therefore, a high-efficiency fine-tuning method for text-to-image diffusion models that does not rely on additional human intervention and can better enable the model to understand text is needed.

[0005] In the field of software testing, metamorphic testing is an effective automated testing method. Its purpose is to test whether a software system can correctly handle changes without introducing errors by subjecting it to multiple transformations (changing program code or input data). The core idea of ​​metamorphic testing is to verify the robustness and correctness of the system under various conditions through different modifications. In metamorphic testing, testers establish inherent metamorphic relationships based on requirements; that is, when program inputs satisfy certain relationships, their corresponding outputs must also satisfy certain relationships. This relationship is called a metamorphic relationship. Metamorphic relationships describe the behavioral changes of the system under different input conditions, which helps in determining the selection and design of test cases. Common metamorphic relationships include equivalence class metamorphism, boundary value metamorphism, and sequential metamorphism. By understanding and applying metamorphic relationships, metamorphic testing can be performed more effectively, improving test coverage and efficiency. Metamorphic testing is essentially a method of constructing multiple sets of test cases based on metamorphic relationships, and then discovering and locating defects in the program by analyzing the degree to which the outputs of different test cases satisfy the metamorphic relationships. Based on the understanding of metamorphic relationships, how to improve the effectiveness and efficiency of fine-tuning the text-based diffusion model has become an urgent problem to be solved. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a method and system for fine-tuning a text-based image diffusion model based on metamorphic testing. In order to improve the understanding of text details by the text-based image diffusion model during fine-tuning, a specific set of metamorphic relationships is provided according to requirements. Input text is automatically generated using this set and then input into the text-based image diffusion model. A metamorphic test is performed on the generated image to check whether the metamorphic relationships are satisfied. The loss function is calculated using the results to update the model parameters, thereby completing the fine-tuning. This improves the model's ability to perceive text metamorphic relationships and enhances the image generation capability of the text-based image diffusion model.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a method for fine-tuning a textural image diffusion model based on metamorphic testing, comprising the following steps:

[0009] Construct a use case set that includes a metamorphic relationship set and an entity set, and build test text based on the use case set;

[0010] The test text is input into the text-generated image diffusion model to obtain the generated image. The generated image is then analyzed using the detection model, and a metamorphic relationship check is performed based on the image analysis results.

[0011] Based on the results of the metamorphic relationship check, an optimization objective function for the text-based image diffusion model is constructed. The parameters of the text-based image diffusion model are iteratively updated using the optimization objective function and its gradient, and finally the fine-tuned text-based image diffusion model is obtained.

[0012] Preferably, the metamorphic relationship includes quantitative relationship, attribute relationship and / or style relationship, which are used to represent the correspondence between the description of the test text and the number, attributes and style of entities in the generated image.

[0013] Preferably, the construction of a use case set including a metamorphic relationship set and an entity set, and the generation of test text based on the use case set, includes:

[0014] Construct a set of metamorphic relations to represent metamorphic relations. The set of metamorphic relations includes several sets of sentence templates. Construct an entity set to limit the objects generated by the text-generated graph diffusion model. The entity set includes several nouns. Combine the set of metamorphic relations and the entity set to form a set of use cases for fine-tuning the text-generated graph diffusion model.

[0015] The test text consists of a set of sentences and nouns selected from the test case set, arranged according to a pre-defined logical structure to satisfy a metamorphic relationship.

[0016] Preferably, the sentence template contains zero or at least one placeholder. When constructing the test text, the placeholders will be replaced by different words in the entity set. If there are zero placeholders, no replacement will be performed.

[0017] Preferably, the step of performing image analysis on the generated image using a detection model includes:

[0018] The detection model includes an object detection model, an attribute detection model, and a style detection model. The generated image is input into the object detection model, the attribute detection model, and / or the style detection model respectively to analyze the number of entities, the attributes of the entities, and / or the overall style in the generated image, and obtain the image analysis results.

[0019] Preferably, the step of performing metamorphic relationship checks based on image analysis results includes:

[0020] Perform corresponding checks based on different metamorphic relationships, record whether the corresponding entities in the generated image satisfy the metamorphic relationship, and combine the check results of whether the metamorphic relationship is satisfied with the generated image into a tuple as the basis for constructing the objective function of the optimization algorithm for the text-based image diffusion model.

[0021] Preferably, the optimization objective function of the textural image diffusion model is constructed based on the metamorphic relationship check results. The parameters of the textural image diffusion model are iteratively updated using the optimization objective function and its gradient to finally obtain the fine-tuned textural image diffusion model, including:

[0022] The denoising process in the text-to-image diffusion model is modeled as a multi-step Markov decision process to optimize the text-to-image process in the model. Combining the optimization principles of Markov decision processes, the final optimization objective function of the text-to-image diffusion model is determined. Defined as:

[0023]

[0024] in, Let c be the input test text and x0 be the expected detection result, and π be the value of the current given state. The conditional distribution of the next step denoising result a t Represented as a strategy The conditional distribution of the next denoising result is considered as action a. t ,Right now R(x0,c) is the reward function, expressed as:

[0025]

[0026] Where b(x0,c) is the metamorphic relationship check result under generated image x0 and test text c;

[0027] Calculate gradient Where θ represents the parameters of the text-based diffusion model, p θ (·) represents the conditional distribution of the next denoising result in a certain state during the diffusion process, x t Let T be the sampling result at step t of the textural image diffusion model, where T is the total number of sampling steps in the model. The parameters of the textural image diffusion model are updated using the objective function and its gradient, such that... To maximize the effect, the parameters of the text-based diffusion model are iteratively updated to obtain a fine-tuned text-based diffusion model.

[0028] Secondly, in order to achieve the above-mentioned objectives, the present invention also provides a text-based graph diffusion model fine-tuning system based on metamorphosis testing, including: a test text generation module, a metamorphosis testing module, and a model fine-tuning module;

[0029] The test text generation module is used to construct a test case set including a metamorphic relationship set and an entity set, and to construct test text based on the test case set;

[0030] The metamorphosis test module is used to input test text into the text-generated image diffusion model to obtain a generated image, use the detection model to perform image analysis on the generated image, and perform metamorphosis relationship checks based on the image analysis results.

[0031] The model fine-tuning module is used to construct an optimization objective function for the text-based image diffusion model based on the metamorphic relationship check results. The optimization objective function and its gradient are used to iteratively update the parameters of the text-based image diffusion model, and finally the fine-tuned text-based image diffusion model is obtained.

[0032] Thirdly, to achieve the above-mentioned objectives, embodiments of the present invention also provide a fine-tuning device for a textural image diffusion model based on metamorphic testing, including a memory and a processor. The memory is used to store a computer program, and the processor is used to implement the above-mentioned fine-tuning method for a textural image diffusion model based on metamorphic testing when the computer program is executed.

[0033] Fourthly, to achieve the above-mentioned objectives, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-mentioned method for fine-tuning the text-based diffusion model based on metamorphic testing.

[0034] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0035] (1) By constructing a test case set including a metamorphic relation set and an entity set, this invention can accurately capture and define semantic relationships in text, ensuring the comprehensiveness and effectiveness of testing. The input text is automatically generated based on the test case set and input into the text-generated graph diffusion model to obtain generated images. Image analysis and metamorphic relation checks are performed based on the generated images, which can improve the accuracy and reliability of metamorphic testing. The results of the metamorphic relation check are used as an important basis for fine-tuning the text-generated graph diffusion model. Gradient updates are used to update model parameters to achieve fine-tuning. The process-oriented design reduces computational resources and time costs, ensuring the effectiveness and efficiency of model fine-tuning.

[0036] (2) In the entire fine-tuning process of this invention, only the generation process of the test case set requires the provision of a corpus to set the metamorphic relation data and entity data. The whole process is highly automated and the desired model can be obtained without much adjustment. In addition, common model fine-tuning methods rely on manual or reward models that require additional training. This invention only requires the use of a detection model in the metamorphic testing stage. The detection model can be a general detection model, which does not require additional training and saves manpower and resources.

[0037] (3) In the past, the fine-tuning of text-to-image diffusion models has mostly focused on artistic quality or text-image matching. This invention takes the text details of the text-to-image diffusion model as the evaluation standard, and considers the model's learning of quantitative relationships, attribute relationships and style relationships, thereby indirectly quantifying the text-to-image diffusion model's understanding of the input text.

[0038] (4) This invention introduces metamorphosis testing, a software testing method, as a reward function into the fine-tuning of the text-based image diffusion model for the first time. This opens up a new direction for subsequent reinforcement learning ideas, builds an end-to-end solution, integrates multiple technologies such as software testing, reinforcement learning and computer vision, and forms a complete automated text-based image diffusion model fine-tuning scheme to improve the performance of the model on specific tasks and ensure that the generated image is highly matched with the input text description. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a general framework diagram of the text-based diffusion model fine-tuning method based on metamorphic testing provided in this embodiment of the invention;

[0041] Figure 2 This is a schematic diagram of the test text generation method provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of the text-based diffusion model fine-tuning system based on metamorphic testing provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0044] The inventive concept of this invention addresses the problem that existing technologies, in the fine-tuning of text-based image diffusion models, often rely on human preferences or additional training rewards, neglecting the model's ability to perceive the detailed semantics within the text. In the actual generation process, whether the model can learn the semantic logic present in the text, such as quantitative relationships, attribute relationships, and stylistic relationships, often affects the accuracy of the final generated image. Accurate image generation based on text is crucial for cost savings, efficiency improvement, and increased model feasibility. Based on this, this invention provides a method and system for fine-tuning a text-based image diffusion model based on metamorphic testing. It automatically generates input text and analyzes the generation results of the text-based image diffusion model based on the input text, checking whether the generated images satisfy metamorphic relationships. This is used as the basis for fine-tuning the parameters of the text-based image diffusion model. Through this method, the text-based image diffusion model's ability to understand text can be effectively enhanced, improving the accuracy of the generated images.

[0045] Figure 1 This is a general framework diagram of the textural graph diffusion model fine-tuning method based on metamorphic testing provided in this embodiment of the invention. For example... Figure 1 As shown in the embodiment, a method for fine-tuning a textural graph diffusion model based on metamorphic testing is provided, including the following steps:

[0046] S1, construct a use case set including a metamorphic relation set and an entity set, and construct test text based on the use case set.

[0047] Specifically, the specific process for the test text generation stage in step S1 is as follows:

[0048] S1.1, Constructing the Use Case Set. Based on user needs, a corresponding use case set is provided. The system will filter the pre-given corpus to generate a metamorphic relation set P. M and an entity set P O This constitutes a use case set. Among them, the metamorphic relation set P M The metamorphic relation set P contains the metamorphic relations needed for a single fine-tuning. M Each element in the set is composed of a pair p. i =[N i ,r i ] indicates that N i It is a set of sentence templates N i =[n1,n2,...,n n Each template in the model is transformed into a sentence. Subsequent tests will use these sentences as input text, feeding them into the text-generated graph diffusion model. The sentence templates are n. i It may contain placeholders {Object j}, different subscripts j represent different placeholders, meaning they will be replaced by different words in the entity set, ri This represents a metamorphic relationship, which can be a quantitative relationship, an attribute relationship, or a style relationship. Entity set P O This is responsible for providing the entities that need to generate the image, limiting the objects generated by the model, the entity set P. O Each element in the text is a word, and words are entity nouns such as cat, dog, and rabbit.

[0049] S1.2, Generate test text. Each time test text is generated, it is first generated from the metamorphic relation set P. M Select a metamorphic relation p from the entity set P, and then, based on the relation type, perform a transformation on the entity set P. O Select zero or at least one word, and using pre-defined logic, form a test text C = [c1, c2, ..., c n The test text C and the metamorphic relation p are used as a test pair in the next stage. The test text is generated as follows: Figure 2 As shown, Figure 2 In this context, XXXX and YYYY represent the text content.

[0050] In the embodiment, for quantitative relationships, such as the metamorphic relationship r: s1 + s2 = s3, this means that the number of entities s1 in the generated image corresponding to sentence one plus the number of entities s2 in the generated image corresponding to sentence two equals the number of entities s3 in the generated image corresponding to sentence three. For example, assuming the selected sentence template group is N = ["One {Object0} is in the sky", "Two {Object0} are in the sky", "Three {Object0} are in the sky"], then the next step is to... O Choose one word to replace the placeholder {Object0} in three sentence templates. When the selected word is "wild goose", the final test text group is C = ["One {wild goose} in the sky", "Two {wild geese} in the sky", "Three {wild geese} in the sky"].

[0051] In the embodiment, for attribute relations, such as the metamorphic relation r: s1≠s2, this means that the attribute s1 of the entity in the image generated from sentence one is inconsistent with the attribute s2 of the entity in the image generated from sentence two. For example, assuming the selected sentence template group is N = ["{Object0} is happy", "{Object0} is unhappy"], then the next step is to... O Choose one word to replace the placeholder {Object0} in two sentence templates. When the selected word is "puppy", the final test text group is C = ["{puppy} is happy", "{puppy} is unhappy"].

[0052] In the embodiment, for style relations, such as the metamorphic relation r: s1 = s2, this means that the style s1 of the image generated by sentence one is consistent with the style s2 of the image generated by sentence two. For example, suppose the selected sentence template group is N = ["A person is taking a walk, pixel style", "A person is taking a walk, 8-bit style"]. Because no placeholders appear, it will not appear in the entity set P. O Select entities from the list, and the final test text group is C = ["A person is walking, pixel style", "A person is walking, 8-bit style"].

[0053] S2, input the test text into the text-generated image diffusion model to obtain the generated image, use the detection model to perform image analysis on the generated image, and perform metamorphic relationship check based on the image analysis results.

[0054] Specifically, the specific process of the transformation test phase in step S2 is as follows:

[0055] S2.1, Image Generation. The set of test text C = [c1, c2, ..., c...] obtained in step S1.2 is used... n The text is sequentially input into the pre-trained text-based image diffusion model M, which is to be fine-tuned. For each sentence in the test text C, only one image is generated, resulting in a set of output images X = [x...]. 1 ,x 2 ,...,x n The image and sentence text are linked by subscripts, representing image x. j From sentence c j The image is generated after inputting the Wensheng image diffusion model M.

[0056] S2.2, Image Analysis. Based on the different metamorphic relationships, the image X = [x] generated in step S2.1 is analyzed. 1 ,x 2 ,...,x n These are then incorporated into different detection models. Each detection model is responsible for analyzing the number of entities in an image, the attributes of those entities, or the overall style. In this embodiment, the detection models are implemented using open-source general-purpose detection models, specifically: an object detection model (for quantity relationships, the input is an image, and the output is the objects present in the image and their quantity s′), an attribute detection model (for attribute relationships, the input is an image, and the output is the objects present in the image and their attributes s′), and a style detection model (for style relationships, the input is an image, and the output is the overall style s′ of the image).

[0057] S2.3, Relationship Check. Different checks are performed on the image analysis results based on different metamorphic relationships to determine whether a set of images exhibits a metamorphic relationship. The metamorphic relationship check result is defined as a binary variable b, used to represent whether a metamorphic relationship is satisfied. In this embodiment, when corresponding entities in a set of generated images satisfy a quantitative relationship, the metamorphic relationship check result is recorded as b. + Conversely, the result of the metamorphosis relationship check is recorded as b. - The result b (where b is a variable between 0 and 1) will be checked. + (where b represents satisfying the metamorphic relation, and b- represents not satisfying the metamorphic relation) and the generated image X = [x 1 ,x 2 ,...,x n This information is passed along with the binary pair to the next step, serving as an important basis for model fine-tuning.

[0058] In this embodiment, assuming the metamorphic relationship is a quantitative relationship, the corresponding metamorphic relationship r is s1+s2=s3, and the input test text group is C=[“One {goose} in the sky”, “Two {goose} in the sky”, “Three {goose} in the sky”], then it will check whether the metamorphic relationship s′1+s′2=s′3 of the corresponding generated image is true, where s′ i Step S2.2 involves the object detection model inspecting and generating image x. i The number of "geese" in the middle, when s′1+s′2=s′3 is true, the metamorphic relation check result is recorded as b=b + Conversely, if the metamorphic relation is not satisfied, record the metamorphic relation check result as b = b. - , where b + and b - These are hyperparameters, and are pre-set values.

[0059] S3. Based on the results of the metamorphic relationship check, construct the optimization objective function of the Wensheng image diffusion model, and use the optimization objective function and its gradient to iteratively update the parameters of the Wensheng image diffusion model, finally obtaining the fine-tuned Wensheng image diffusion model.

[0060] Specifically, the detailed process of the model fine-tuning stage in step S3 is as follows:

[0061] S3.1 Parameter Update. Based on the results obtained in step S2.3, the parameters of the textural image diffusion model are updated using a reinforcement learning strategy. In this embodiment, the DDPO (Denoising Diffusion Policy Optimization) algorithm is applied to fine-tune the textural image diffusion model. The DDPO algorithm models the denoising process of the textural image diffusion model as a multi-step Markov decision process (multi-step MDP), and the specific modeling process is as follows:

[0062] A pre-trained Wensheng image diffusion model μ θ (x t The t-th step sampling x in the model of the diffusion of the text image (where θ is a parameter in the model of the text image diffusion) t The input text c and the sampling process t are represented as a state s. t ,Right now The conditional distribution of the next denoising result under the current given state is represented as a strategy, namely... The conditional distribution of the next denoising result is considered as action a. t ,Right now Because the next denoising result x is sampled t-1 After that, the state transition is determined, so the state transition probability is represented by three Dirac delta distributions, i.e. Given the prior distribution, total sampling time step T, and x satisfying the input text c, T The initial state distribution is a Gaussian noise distribution, i.e.

[0063] After completing the modeling, the optimization principle of Markov decision processes is used to refine the final optimization objective function of the Wensheng graph diffusion model. Defined as:

[0064]

[0065] in, Given the input test text c and the expected detection result under the generated image x0, where generated image x0 is the output image after the last sampling step, corresponding to the generated image x of the text-to-image diffusion model. i Let π be the policy and R(x0,c) be the reward function, expressed as:

[0066]

[0067] Where b(x0,c) is the metamorphic relationship check result under generated image x0 and test text c;

[0068] Calculate gradient Where θ represents the parameters of the textural diffusion model, p θ (·) represents the conditional distribution of the next denoising result in a certain state during the diffusion process, x t Let T be the sampling result at step t of the textural image diffusion model, where T is the total number of sampling steps in the model. The parameters of the textural image diffusion model are updated using the objective function and its gradient, such that... maximize.

[0069] S3.2, Iterative optimization. Repeat steps S1.1-S3.1 to iteratively update the parameters of the textural image diffusion model, ultimately obtaining a fine-tuned textural image diffusion model M. * .

[0070] In summary, a text-based graph diffusion model fine-tuning method based on metamorphic testing combines the idea of ​​metamorphic testing with the use of automatically generated semantic samples to detect the alignment ability between the model-generated images and text at a fine-grained word level. This is then used as feedback to fine-tune the model, thereby improving the text-based graph diffusion model's ability to perceive metamorphic relationships in text semantics. This helps the model better capture details in the input text, thereby generating images with higher matching granularity, enhancing the model's generation ability, improving the efficiency of model fine-tuning, and making it suitable for various task application scenarios of text-based graphs.

[0071] Based on the same inventive concept, such as Figure 3 As shown, this embodiment of the invention also provides a text-based graph diffusion model fine-tuning system 300 based on metamorphosis testing, including: a test text generation module 310, a metamorphosis testing module 320, and a model fine-tuning module 330.

[0072] Among them, the test text generation module 310 is used to construct a test case set including a metamorphic relationship set and an entity set, and to construct test text based on the test case set;

[0073] The metamorphosis test module 320 is used to input the test text into the text-generated image diffusion model to obtain the generated image, use the detection model to perform image analysis on the generated image, and perform metamorphosis relationship checks based on the image analysis results.

[0074] The model fine-tuning module 330 is used to construct the optimization objective function of the textual image diffusion model based on the metamorphic relationship check results. The optimization objective function and its gradient are used to iteratively update the parameters of the textual image diffusion model, and finally the fine-tuned textual image diffusion model is obtained.

[0075] Based on the same inventive concept, this embodiment of the invention also provides a fine-tuning device for a textural image diffusion model based on metamorphic testing, including a memory and a processor. The memory is used to store a computer program, and the processor is used to implement the above-described fine-tuning method for a textural image diffusion model based on metamorphic testing when the computer program is executed.

[0076] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for fine-tuning the text-based diffusion model based on metamorphic testing.

[0077] It should be noted that the text-based image diffusion model fine-tuning system, the text-based image diffusion model fine-tuning device, and the computer-readable storage medium provided in the above embodiments all belong to the same inventive concept as the text-based image diffusion model fine-tuning method based on metamorphic testing. For details of their specific implementation process, please refer to the embodiments of the text-based image diffusion model fine-tuning method based on metamorphic testing, which will not be repeated here.

[0078] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for fine-tuning a text-based image diffusion model based on metamorphic testing, characterized in that, Includes the following steps: Construct a use case set including a metamorphic relation set and an entity set. Based on the use case set, construct test text, including: constructing a metamorphic relation set to represent metamorphic relations, which includes several sets of sentence templates; constructing an entity set to limit the objects generated by the text-generated graph diffusion model, which includes several nouns; combining the metamorphic relation set and the entity set to form a use case set for fine-tuning the text-generated graph diffusion model; and combining the sentence templates and nouns selected from the use case set with preset logic to form a set of statements that satisfy the metamorphic relations as test text. The test text is input into the text-generated image diffusion model to obtain the generated image. The generated image is then analyzed using a detection model. Based on the image analysis results, a metamorphic relationship check is performed. The metamorphic relationship includes quantitative relationship, attribute relationship, and / or style relationship, which are used to represent the correspondence between the description of the test text and the number, attributes, and style of entities in the generated image, respectively. Based on the metamorphic relationship check results, an optimization objective function for the textural graph diffusion model is constructed. The parameters of the textural graph diffusion model are iteratively updated using the optimization objective function and its gradient, ultimately yielding a fine-tuned textural graph diffusion model. This includes: modeling the denoising process in the textural graph diffusion model as a multi-step Markov decision process for optimizing the textural graph process; and combining the optimization principles of Markov decision processes to refine the final optimization objective function of the textural graph diffusion model. Defined as: ， in, For the input test text and generating images Expectations for the test results To make the current given state The conditional distribution of the next step denoising result Represented as a strategy The conditional distribution of the next denoising result is regarded as an action. ,Right now , The reward function is expressed as: ， in, To generate an image and test text Results of the metamorphic relationship examination; Calculate gradient ,in, The parameters of the text-based image diffusion model, This represents the conditional distribution of the next denoising result under a certain state during the diffusion process. For text-based graph diffusion model No. The sampling results of the step, The total number of sampling steps for the textural image diffusion model is given. The parameters of the textural image diffusion model are updated using the optimization objective function and its gradient, so that... To maximize the effect, the parameters of the text-based diffusion model are iteratively updated to obtain a fine-tuned text-based diffusion model.

2. The method for fine-tuning the textural image diffusion model based on metamorphic testing according to claim 1, characterized in that, The sentence template contains zero or at least one placeholder, which will be replaced by different words from the entity set when constructing the test text.

3. The method for fine-tuning the text-based diffusion model based on metamorphic testing according to claim 1, characterized in that, The image analysis of the generated image using the detection model includes: The detection model includes an object detection model, an attribute detection model, and a style detection model. The generated image is input into the object detection model, the attribute detection model, and / or the style detection model respectively to analyze the number of entities, the attributes of the entities, and / or the overall style in the generated image to obtain the image analysis results.

4. The method for fine-tuning the text-based diffusion model based on metamorphic testing according to claim 1, characterized in that, The metamorphic relationship check based on image analysis results includes: Perform corresponding checks based on different metamorphic relationships, record whether the corresponding entities in the generated image satisfy the metamorphic relationship, and combine the check results of whether the metamorphic relationship is satisfied with the generated image into a tuple as the basis for constructing the objective function of the optimization algorithm for the text-based image diffusion model.

5. A fine-tuning system for a textural image diffusion model based on metamorphic testing, used to implement the fine-tuning method for a textural image diffusion model based on metamorphic testing as described in any one of claims 1 to 4, characterized in that, include: Test the text generation module, the metamorphosis testing module, and the model fine-tuning module; The test text generation module is used to construct a test case set including a metamorphic relation set and an entity set, and to construct test text based on the test case set; The metamorphosis test module is used to input test text into the text-generated image diffusion model to obtain a generated image, use the detection model to perform image analysis on the generated image, and perform metamorphosis relationship checks based on the image analysis results. The model fine-tuning module is used to construct an optimization objective function for the text-to-image diffusion model based on the metamorphic relationship check results. The optimization objective function and its gradient are used to iteratively update the parameters of the text-to-image diffusion model, and finally the fine-tuned text-to-image diffusion model is obtained.

6. A device for fine-tuning a textural diffusion model based on metamorphic testing, comprising a memory and a processor, wherein the memory is used to store a computer program, characterized in that, The processor is configured to implement, when executing the computer program, the textural diffusion model fine-tuning method based on metamorphic testing as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the text-based diffusion model fine-tuning method based on metamorphic testing as described in any one of claims 1-4.

Images