Personalized gram model construction and gram method based on causal fine-tuning

Abstract

The present invention relates to the field of generative artificial intelligence technology, and discloses a personalized vincent graph model construction and vincent graph method based on causal fine-tuning, including: merging and enhancing multiple reference images and text descriptions of multiple subjects to obtain multiple merged enhanced images and merged text descriptions; using a preset causal decoupling algorithm, extracting identity-protecting text representations, identity-related and identity-irrelevant visual representations from the text descriptions, merged text descriptions and merged enhanced images of each subject; inputting them into a preset causal reasoning model to obtain a cross-attention map; aligning the cross-attention map with the visual representation related to the corresponding subject identity. The present invention identifies subject identity-related information and identity-irrelevant information through subject-perceived causal decoupling, protects subject identity information, and causally aligns the text attributes and visual attributes of each subject through cross-attention guidance, thereby avoiding confusion between subjects and improving the effectiveness and controllability of vincent graphs.

Description

Technical Field

[0001] The present invention relates to the field of generative artificial intelligence technology, and in particular to personalized causal fine-tuning-based venerable graph model construction and a venerable graph method. Background Art

[0002] In recent years, revolutionary large model technology has promoted the development of artificial intelligence generated content (AIGC). As an important research content in the field of AIGC, the subject-driven personalized subject-driven subject-driven subject-driven image generation requires customized images to be generated based on the reference image provided by the user and the text description containing personalized needs, which increases the difficulty of the subject-driven subject-driven subject-driven image generation task. In particular, compared with the single-subject personalized subject-driven ... Summary of the invention

[0003] In view of this, the present invention provides a personalized text graph model construction and text graph method based on causal fine-tuning to solve the problem of subject missing or distortion in multi-agent text graphs.

[0004] In a first aspect, the present invention provides a method for constructing a personalized cultural graph model based on causal fine-tuning, the method comprising:

[0005] Acquire multiple reference images of multiple subjects and text descriptions of each subject, merge and enhance the multiple reference images of each subject to obtain multiple merged enhanced images, and merge and enhance the text descriptions of each subject to obtain a merged text description, wherein the merged enhanced image includes a reference image of each subject;

[0006] Using the preset causal decoupling algorithm, the identity-protected text representation, identity-related visual representation, and identity-irrelevant visual representation are extracted from the text description, merged text description, and merged enhanced image of each subject respectively;

[0007] The merged enhanced image, identity-protected text representation, identity-related visual representation, and identity-independent visual representation are input into a preset causal inference model to complete causal denoising reasoning and obtain a cross-attention map.

[0008] According to the text representation of the protected identity, the cross-attention map of the corresponding subject is extracted, and the cross-attention map of the corresponding subject is aligned with the visual representation related to the corresponding subject identity to complete the construction of the multi-subject personalized text graph model.

[0009] The personalized text graph model construction method based on causal fine-tuning provided by the present invention distinguishes information related to the subject's identity and information unrelated to the identity through the causal decoupling of subject perception, thereby protecting the subject's identity information. The text attributes and visual attributes of each subject are causally aligned through cross-attention guidance to avoid confusion between subjects, thereby significantly improving the accuracy of the multi-subject personalized text graph model.

[0010] In an optional implementation, a preset causal decoupling algorithm is used to extract identity-protected text representations, identity-related visual representations, and identity-irrelevant visual representations from the text descriptions, merged text descriptions, and merged enhanced images of each subject, respectively, including:

[0011] Inputting the text description of each subject and the combined text description into a preset text encoder to obtain a text representation that protects the identity;

[0012] Inputting the merged enhanced image into a preset image encoder to obtain a visual representation of the merged enhanced image;

[0013] Using a preset mask set to perform causal intervention on the visual representation, we obtain the counterfactual visual representation of each subject and the merged subject;

[0014] The pre-trained model is used to transform the counterfactual visual representations of each subject and the merged subject into a latent space similar to the text representation, thus obtaining identity-related visual representations and identity-independent visual representations.

[0015] By using a preset causal decoupling function, a first similarity between the identity-protecting textual representation and the identity-related visual representation is maximized, and a second similarity between the identity-protecting textual representation and the identity-irrelevant visual representation is minimized;

[0016] According to the first similarity and the second similarity, the identity-related visual representation and the identity-irrelevant visual representation are decoupled.

[0017] The personalized text graph model construction method based on causal fine-tuning provided by the present invention determines identity-related visual representations and identity-irrelevant visual representations according to the text representation of identity protection, which is conducive to excluding identity-irrelevant visual representations, reducing the interference of the picture background on the picture generation process, and improving the accuracy of the multi-agent personalized text graph model.

[0018] In an optional implementation, a preset mask set is used to perform causal intervention on the visual representation to obtain counterfactual visual representations of each subject and the combined subject, including:

[0019] Design preset mask collection Among them, M i (i=1,2,...,n) represents the mask of the ith subject, Mn+1 Masks representing the merged subjects, each mask ranges from (0,1) and has the same latent space dimension as the visual representation corresponding to the merged enhanced image;

[0020] According to the preset mask of each subject and the visual representation of the merged enhanced image, the counterfactual visual representation of each subject is obtained, and according to the preset mask of the merged subject and the visual representation of the merged enhanced image, the counterfactual visual representation of the merged subject is obtained.

[0021] The personalized cultural graph model construction method based on causal fine-tuning provided by the present invention realizes causal intervention on visual representation by using a preset mask set, and finally obtains the counterfactual visual signs of each subject and the merged subject, thereby filtering out the visual representations irrelevant to the subject identity, alleviating the interference of the visual representations irrelevant to the subject identity in the reference image on the image generation process, and effectively solving the problem of subject identity distortion.

[0022] In an optional implementation, the counterfactual visual representations of each subject and the merged subject are converted into a latent space similar to the textual representation using a pre-trained model, including:

[0023] Using multi-layer perceptrons and residual connections, the counterfactual visual representations of each subject and the merged subject are converted to a latent space similar to the text representation. The formula is:

[0024] f i =MLP(M i *f ms +MLP(M i *f ms )),i=1,2,...,n+1

[0025]

[0026] in, f i (i=1,2,...,n), They represent the identity-related visual representation and identity-independent visual representation of the i-th subject, respectively, and f i (i=n+1), They represent the identity-related visual representation and identity-irrelevant visual representation of the merged subject respectively.

[0027] The personalized text graph model construction method based on causal fine-tuning provided by the present invention utilizes a multi-layer perceptron and residual connections to convert counterfactual visual representations into a latent space similar to text representations, thereby alleviating the distribution differences of different representations in different latent spaces, facilitating the comparison of visual representations with text representations, and more accurately determining identity-related visual representations and identity-irrelevant visual representations.

[0028] In an optional embodiment, aligning the cross-attention map of the corresponding subject with the visual representation related to the identity of the corresponding subject includes:

[0029] Calculate the average of the cross-attention maps of all subjects;

[0030] The average value is input into the preset image encoder to obtain the attention representation corresponding to the cross-attention map of each subject;

[0031] According to the attention representation corresponding to the cross-attention map of each subject and the identity-related visual representation, the attention representation corresponding to the cross-attention map of each subject is aligned with the identity-related visual representation.

[0032] The personalized text graph model construction method based on causal fine-tuning provided by the present invention avoids interference between subjects by aligning the attention representation corresponding to the cross-attention map of each subject with the identity-related visual representation, enhances the causal alignment of text representation and visual representation between subjects, and is conducive to using a multi-subject personalized text graph model to generate images that conform to the text representation.

[0033] In a second aspect, the present invention provides a method for a culture graph, the method comprising:

[0034] Get a text description containing multiple subjects;

[0035] The text descriptions of multiple subjects are input into a preset multi-subject personalized text graph model to generate an image corresponding to the text description. The preset multi-subject personalized text graph model is obtained by using the personalized text graph model construction method based on causal fine-tuning of the first aspect.

[0036] The text image method provided by the present invention uses a picture obtained by a multi-subject personalized text image model, reduces the interference of the background on the subject and the interference between the subjects, avoids the problem of subject missing, and improves the effectiveness and controllability of the multi-subject personalized text image.

[0037] In a third aspect, the present invention provides a personalized cultural graph model construction system based on causal fine-tuning, the system comprising:

[0038] A data enhancement module, used to obtain multiple reference images of multiple subjects and text descriptions of each subject, and merge and enhance the multiple reference images of each subject to obtain multiple merged enhanced images, and merge and enhance the text descriptions of each subject to obtain a merged text description, wherein the merged enhanced image includes a reference image of each subject;

[0039] A subject-aware causal decoupling module is used to extract identity-protected text representations, identity-related visual representations, and identity-irrelevant visual representations from the text descriptions, merged text descriptions, and merged enhanced images of each subject using a preset causal decoupling algorithm;

[0040] The diffusion feedforward module is used to input the merged enhanced image, the identity-protected text representation, the identity-related visual representation, and the identity-independent visual representation into a preset causal reasoning model to complete the causal denoising reasoning and obtain the cross-attention map;

[0041] The causally inspired cross-attention guidance module is used to extract the cross-attention map of the corresponding subject based on the textual representation of the protected identity, and align the cross-attention map of the corresponding subject with the visual representation related to the corresponding subject's identity to complete the construction of the multi-agent personalized text graph model.

[0042] In a fourth aspect, the present invention provides a cultural map system, the system comprising:

[0043] A text acquisition module is used to acquire a text description containing multiple subjects;

[0044] The text graph module is used to input the text descriptions of multiple subjects into a preset multi-subject personalized text graph model to generate an image corresponding to the text description. The preset multi-subject personalized text graph model is obtained by using the personalized text graph model construction method based on causal fine-tuning of the first aspect.

[0045] In a fifth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the personalized Vincent graph model construction method based on causal fine-tuning of the above-mentioned first aspect or any corresponding embodiment thereof, and the Vincent graph method of the second aspect or any corresponding embodiment thereof by executing the computer instructions.

[0046] In a sixth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the personalized Vincent graph model construction method based on causal fine-tuning of the first aspect or any corresponding embodiment thereof, and the Vincent graph method of the second aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is a comparison chart of the effects of related technologies on multi-subject personality text map tasks;

[0049] Figure 2 is a flow chart of a method for constructing a personalized cultural graph model based on causal fine-tuning according to an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of a specific embodiment of a method for constructing a personalized cultural graph model based on causal fine-tuning according to an embodiment of the present invention;

[0051] Figure 4 is a flow chart of another personalized cultural graph model construction method based on causal fine-tuning according to an embodiment of the present invention;

[0052] Figure 5 is a schematic flow chart of a Vincent graph method according to an embodiment of the present invention;

[0053] Figure 6 is a comparison diagram of the effects of the Wensheng diagram method according to an embodiment of the present invention and related technologies in a single-subject personalized Wensheng diagram task;

[0054] Figure 7 is a comparison diagram of the effects of the Wensheng diagram method according to an embodiment of the present invention and related technologies in a multi-agent personalized Wensheng diagram task;

[0055] Figure 8 is a structural block diagram of a personalized cultural graph model construction system based on causal fine-tuning according to an embodiment of the present invention;

[0056] Fig. 9 is a structural block diagram of a culture graph system according to an embodiment of the present invention;

[0057] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0059] To achieve effective personalized cultural maps, relevant methods include:

[0060] (1) Personalized text graph based on fine-tuning, that is, it is necessary to fine-tune the parameters of the text graph model according to the reference image and text description provided by the user, and then use the fine-tuned text graph model to infer and generate personalized images corresponding to the target text description. However, related methods only simply jointly optimize the training objectives of multiple subjects, ignoring the mutual interference between subjects, or only train the pre-trained text vector of the target object, ignoring the interference of identity-irrelevant visual information such as the background in the reference image on the subject concept, which easily leads to identity distortion problems of multiple subjects.

[0061] (2) Personalized text graphs that do not require fine-tuning, that is, the entire text graph model does not require fine-tuning training. It only needs to input the reference image and the target text description into the text graph model, complete the reasoning by optimizing the encoding parameters, and generate the target image. A related method is to embed the visual representation of the reference image into the text space by constructing a global mapping network, and then embed the fine-grained visual representation into the cross-attention layer by constructing a local mapping network. This method also has good personalized text graph performance. However, this method is mainly aimed at single-subject personalized text graphs, and is easily affected by visual information unrelated to the subject's identity and interference between subjects, resulting in distortion problems for multiple subjects.

[0062] In summary, existing personalized vinyasa methods mainly focus on single-subject personalized vinyasa, and there are still obvious deficiencies in multi-subject personalized vinyasa. The problem of missing or distorted subject identities is particularly serious. This problem is mainly caused by two reasons: on the one hand, the identity-irrelevant information within the subject (such as the background in the image) is overly coupled with the subject identity information, resulting in missing or distorted subject identity information in the generated target image; on the other hand, interference is prone to exist between different subjects, resulting in confusion or identity loss of the subject in the generated target image. Therefore, how to effectively protect the subject identity information while alleviating the interference between different subjects, and thus improve the effectiveness and controllability of multi-subject personalized vinyasa is an important challenge in this field. Figure 1 The comparison of the effects of ELITE, DreamBooth, Custom Diffusion and DisenBooth on the multi-agent personality text map task is shown. Figure 1 It can be seen that most related technologies fail to effectively solve the problem of missing or distorted subject identity.

[0063] Based on the above problems, the embodiments of the present invention provide a personalized vinyasa graph model construction and vinyasa graph method based on causal fine-tuning, which achieves the effect of improving the effectiveness and controllability of multi-agent personalized vinyasa graphs through subject-perceived causal decoupling and causally inspired cross-attention guidance.

[0064] According to an embodiment of the present invention, an embodiment of a method for constructing a personalized cultural graph model based on causal fine-tuning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0065] In this embodiment, a personalized cultural graph model construction method based on causal fine-tuning is provided, which can be used in the above-mentioned computer device. Figure 2 is a flowchart of a method for constructing a personalized cultural graph model based on causal fine-tuning according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0066] Step S101, obtain multiple reference images of multiple subjects and text descriptions of each subject, merge and enhance the multiple reference images of each subject to obtain multiple merged enhanced images, and merge and enhance the text descriptions of each subject to obtain a merged text description, wherein the merged enhanced image contains a reference image of each subject.

[0067] Specifically, each reference image contains a subject. Usually, 2-4 reference images are sufficient for each subject. A large number of images are not required as training samples, which simplifies the number of training times. This is only an example, but not limited to this. The merged enhanced image contains all subjects. The merged text description is obtained by merging and enhancing the text descriptions of each subject, and contains all subjects. Figure 3 As shown, take two subjects "cat" and "dog" as an example: the text description of one subject is "a dog", and the text description of the other subject is "a cat", then the merged text description is "a dog and a cat", which is only an example, but not limited to this. A cat image and a dog image are merged to obtain a merged enhanced image. During the model training process, it is necessary to input the merged enhanced image, the text description of each subject, and the text description of the merged subject.

[0068] Step S102 , using a preset causal decoupling algorithm, extracting identity-protecting text representations, identity-related visual representations, and identity-irrelevant visual representations from the text descriptions, merged text descriptions, and merged enhanced images of each subject.

[0069] Specifically, a text representation that protects the identity is extracted from the text description of each subject to achieve the goal of causal decoupling. For example, if the subject is a "cat", the text representation that protects the identity is the text representation obtained based on the text label "cat". This is only an example, but not limited to this. By performing causal intervention on the merged enhanced image, a counterfactual visual representation of each subject is generated, and then a visual representation that is aligned with the text representation that protects the identity is decoupled from the counterfactual visual representation, that is, the identity-related visual representation, and the remaining counterfactual visual representations are identity-independent visual representations.

[0070] In step S103, the merged enhanced image, the identity-protected text representation, the identity-related visual representation, and the identity-irrelevant visual representation are input into a preset causal reasoning model to complete the causal denoising reasoning and obtain a cross-attention map.

[0071] Specifically, the merged enhanced image is input into the stsblediffusion model, the merged enhanced image is denoised, the identity-protected text representation, identity-related visual representation, and identity-independent visual representation are input into the U-Net network for causal denoising, and the image encoder ε in the stsblediffusion model (i.e., the encoder in the variational autoencoder) is used to obtain the latent space representation z = ε(x) of the merged enhanced image x, which is then subjected to traditional diffusion noise sampling and U-Net network ε θ Decoding, realizing reasoning, the objective function is as follows:

[0072]

[0073] Among them, C s represents the reference image set provided by the user. For each subject, this embodiment requires 2-4 corresponding reference images to complete the training. ε~N(0,1) represents the Gaussian noise of diffusion sampling. The fusion of the identity-protecting text representation and the identity-independent visual representation of the merged enhanced image after causal intervention is represented, and the fused data is input into the U-Net network, and cross-attention reasoning is performed in the U-Net network to obtain a cross-attention map. It should be noted that the traditional stsblediffusion model realizes content generation based on variational reasoning of Gaussian noise, mainly including noise addition and denoising. This embodiment uses the features of causal decoupling (identity-protecting text representation, identity-related visual representation, identity-independent visual representation) to complete the reasoning of the denoising stage, so the result is more accurate.

[0074] Step S104, extracting the cross-attention map of the corresponding subject according to the text representation of the protected identity, and aligning the cross-attention map of the corresponding subject with the visual representation related to the corresponding subject identity, so as to complete the construction of the multi-subject personalized text graph model.

[0075] Specifically, the subject identity-preserving textual representation is utilized to extract the cross-attention maps of its corresponding subject to align each subject's identity-preserving textual attributes with its causally decoupled visual attributes.

[0076] The personalized text graph model construction method based on causal fine-tuning provided in this embodiment distinguishes information related to the subject's identity and information unrelated to the identity through the causal decoupling of subject perception, thereby protecting the subject's identity information. The text attributes and visual attributes of each subject are causally aligned through cross-attention guidance to avoid confusion between subjects, thereby significantly improving the accuracy of the multi-subject personalized text graph model.

[0077] In this embodiment, a personalized cultural graph model construction method based on causal fine-tuning is provided, which can be used in the above-mentioned computer device. Figure 4 is a flowchart of a method for constructing a personalized cultural graph model based on causal fine-tuning according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0078] Step S201, obtain multiple reference images of multiple subjects and text descriptions of each subject, merge and enhance the multiple reference images of each subject to obtain multiple merged enhanced images, and merge and enhance the text descriptions of each subject to obtain a merged text description, wherein the merged enhanced image includes a reference image of each subject. For details, please refer to Figure 2 Step S101 of the illustrated embodiment will not be described in detail here.

[0079] Step S202 , using a preset causal decoupling algorithm, extracting identity-protecting text representations, identity-related visual representations, and identity-irrelevant visual representations from the text descriptions, merged text descriptions, and merged enhanced images of each subject.

[0080] Specifically, the above step S202 includes:

[0081] Step S2021, input the text description of each subject and the merged text description into a preset text encoder to obtain a text representation of the protected identity.

[0082] Specifically, for a given n subject text descriptions, a merged text description is obtained after merging and enhancing, that is, a set ρ = {ρ 1 ,...,ρ n ,ρn+1}, input the n+1 text descriptions into the text encoder E of the multimodal pre-trained neural network (Contrastive Language-Image Pre-Training, CLIP) model T , get the identity-protected text representation: f i s =E T (ρ i ),i=1,2,...,n+1, where f i s (i=1,2,...,n) represents the text representation of the protected identity of each subject, A textual representation of the protected identity of the merged subject. Figure 3 As shown, f i s (i=1,2,3) corresponds to the identity-preserving text representation output by the CLIP text encoder, where f 1 s , f 3 s The text representations of the subject-protected identities of “a cat”, “a dog”, and “a cat and a dog”, respectively. Figure 3 The entities represented by the subscripts in are the same.

[0083] Step S2022: input the combined enhanced image into a preset image encoder to obtain a visual representation of the combined enhanced image.

[0084] Specifically, for a given merged enhanced image x, the merged enhanced image x is input into the image encoder E of the CLIP model I , and obtain the visual representation f of the merged enhanced image ms =E I (x ms ).

[0085] Step S2023, using a preset mask set to perform causal intervention on the visual representation to obtain the counterfactual visual representation of each subject and the merged subject.

[0086] Specifically, in order to alleviate the interference of visual representations in the reference image that are irrelevant to the subject's identity (such as the background of the subject in the reference image) on the vincent map process, it is necessary to perform causal intervention on the visual representation to filter out the visual representations that are irrelevant to the subject's identity.

[0087] In some optional implementations, the above step S2023 includes:

[0088] Step a1: Design a preset mask set Among them, Mi (i=1,2,...,n) represents the mask of the ith subject, M n+1 Masks representing the merged subjects, each mask ranges from (0,1) and has the same latent space dimension as the visual representation corresponding to the merged augmented image.

[0089] Specifically, in order to enhance the visual representation of the image from the merge ms =E I (x ms ) to filter out visual information related to the subject identity and visual information unrelated to the subject identity, and design a subject-specific mask set To achieve causal intervention, each mask is not fixed, but a learnable mask composed of a neural network and trained with the model.

[0090] Step a2, obtaining a counterfactual visual representation of each subject based on the preset mask of each subject and the visual representation of the merged enhanced image, and obtaining a counterfactual visual representation of the merged subject based on the preset mask of the merged subject and the visual representation of the merged enhanced image.

[0091] Specifically, by multiplying the preset mask of each subject with the visual representation of the merged enhanced image, the counterfactual visual representation M of the i-th subject is obtained: i *f ms , and the counterfactual visual representation M of the merged subject n+1 *f ms .

[0092] The personalized text graph model construction method based on causal fine-tuning provided in this embodiment uses a preset mask set to achieve causal intervention on visual representations, and finally obtains counterfactual visual signs of each subject and the merged subject, thereby filtering out visual representations that are not related to the subject identity, alleviating the interference of visual representations that are not related to the subject identity in the reference image on the image generation process, and effectively solving the problem of subject identity distortion.

[0093] Step S2024, using the pre-trained model, the counterfactual visual representations of each subject and the merged subject are converted into a latent space similar to the text representation, so as to obtain identity-related visual representations and identity-independent visual representations.

[0094] Specifically, this embodiment uses the publicly available pre-trained Stable Diffusion as the basic model, which includes a variational autoencoder for encoding and decoding images, a text encoder, and a U-Net network for noise reasoning. Since the text encoder and U-Net are jointly trained together in the pre-training stage of Stable Diffusion, while the variational autoencoder is not jointly trained with the U-Net, in order to alleviate the distribution difference of representations in different latent spaces, the counterfactual visual representations of each subject and the merged subject are converted to a latent space similar to the text representation.

[0095] In some optional implementations, the above step S2024 includes:

[0096] Using multi-layer perceptrons and residual connections, the counterfactual visual representations of each subject and the merged subject are converted to a latent space similar to the text representation. The formula is:

[0097] f i =MLP(M i *f ms +MLP(M i *f ms )),i=1,2,...,n+1

[0098]

[0099] in, f i (i=1,2,...,n), They represent the identity-related visual representation and identity-independent visual representation of the i-th subject, respectively, and f i (i=n+1), They represent the identity-related visual representation and identity-irrelevant visual representation of the merged subject respectively.

[0100] Specifically, Figure 3 As shown, f i (i=1,2,3), They correspond to the visual representation related to the subject identity and the visual representation unrelated to the subject identity output after causal intervention, respectively.

[0101] The personalized text graph model construction method based on causal fine-tuning provided in this embodiment uses a multi-layer perceptron and residual connections to convert counterfactual visual representations into a latent space similar to text representations, thereby alleviating the distribution differences of different representations in different latent spaces, facilitating the comparison of visual representations with text representations, and more accurately determining identity-related visual representations and identity-irrelevant visual representations.

[0102] Step S2025 , using a preset causal decoupling function, maximizes a first similarity between the identity-protecting text representation and the identity-related visual representation, and minimizes a second similarity between the identity-protecting text representation and the identity-irrelevant visual representation.

[0103] Specifically, based on the identity-protecting textual representation and the subject identity-related / irrelevant counterfactual visual representation, a comparison objective is designed to decouple the identity-related visual representation and the identity-irrelevant visual representation. The goal is to maximize the subject identity-protecting textual representation f i s (i=1,2,...,n+1) and the visual representation f related to the subject identity i (i=1,2,...,n+1), while minimizing the subject's identity-preserving text representation f i s (i=1,2,...,n+1) Visual representation independent of subject identity The similarities between.

[0104] Step S2026: Decouple the identity-related visual representation and the identity-irrelevant visual representation according to the first similarity and the second similarity.

[0105] Specifically, the decoupling objective function is:

[0106]

[0107] Among them, cosine is the cosine similarity metric, and α is a hyperparameter of the decoupling objective. Figure 3 As shown, the contrastive target of causal decoupling is represented by the text f i s (i=1,2,3) and visual representation f i (i=1,2,3) or The dotted double-headed arrow between them indicates that the subject protects the identity of the text representation f i s (i=1,2,3) Visual representation f related to subject identity i The dotted double-headed arrow line between (i=1,2,3) indicates the distance between them, which corresponds to the denominator of the decoupling objective function, the text representation of the subject's protected identity f i s (i=1,2,3) Visual representations that are independent of subject identity The double-headed dotted arrow between indicates the distance between the two is increased, which corresponds to the numerator of the decoupling objective function. The closer the distance, the greater the similarity, and the farther the distance, the smaller the similarity, so the first similarity is greater than the second similarity, thereby decoupling the visual representation related to the subject identity and the visual representation unrelated to the subject identity.

[0108] The personalized text graph model construction method based on causal fine-tuning provided in this embodiment determines identity-related visual representations and identity-irrelevant visual representations according to the text representation of identity protection, which is conducive to excluding identity-irrelevant visual representations, reducing the interference of the image background on the image generation process, and improving the accuracy of the multi-agent personalized text graph model.

[0109] Step S203: Input the combined enhanced image, identity-protected text representation, identity-related visual representation, and identity-irrelevant visual representation into a preset causal inference model to complete causal denoising reasoning and obtain a cross-attention map. Figure 2 Step S103 of the illustrated embodiment will not be described in detail here.

[0110] Step S204, extracting the cross-attention map of the corresponding subject according to the text representation of the protected identity, and aligning the cross-attention map of the corresponding subject with the visual representation related to the corresponding subject identity, so as to complete the construction of the multi-subject personalized text graph model.

[0111] Specifically, the Stable diffusion model uses the cross-attention map of the U-Net network to realize the interaction between the textual representation of the subject's identity protection and the identity-related visual representation in the feedforward process. To avoid interference between the attributes of different subjects, it is necessary to extract the cross-attention map corresponding to the textual representation of each subject's identity protection and align it with the identity-related visual representation of the corresponding subject obtained by causal decoupling.

[0112] Specifically, the above step S204 includes:

[0113] Step S2041, calculate the average value of the cross-attention maps of all subjects.

[0114] Specifically, for the text description ρ of the i-th subject i (i=1,2,...,n), the cross attention map corresponding to the subject identity can be extracted from the U-Net network Compute the average of all cross-attention maps corresponding to subject identities.

[0115] Step S2042, input the average value into a preset image encoder to obtain the attention representation corresponding to the cross-attention map of each subject.

[0116] Specifically, the average value is input into the CLIP image encoder to obtain the attention representation corresponding to the cross-attention map of the i-th subject:

[0117]

[0118] Step S2043, aligning the attention representation corresponding to the cross-attention map of each subject with the visual representation related to the identity based on the attention representation corresponding to the cross-attention map of each subject and the visual representation related to the identity.

[0119] Specifically, the objective function of cross-attention guidance is used to align the attention representation corresponding to the cross-attention map of each subject with the identity-related visual representation. The objective function of cross-attention guidance is as follows:

[0120]

[0121] Among them, β is a hyperparameter, and MSE represents the mean square error loss. Figure 3 As shown in Figure 2, the subject-perceived causal decoupling and causally inspired cross-attention guide the two modules, and the two bidirectional dashed arrows draw the identity-related visual representations closer. i (i=1,2) and cross attention map Atten i The distance between them is (i=1,2) to achieve alignment.

[0122] The overall fine-tuning objective function of the multi-agent personalized text graph model is:

[0123] L=L cdm +L attn +L cst

[0124] Before using the multi-agent personalized text-based graph model each time, one only needs to use new text descriptions and reference images to train the text encoder, causal intervention submodule, and cross-attention layer in the CLIP model's U-Net network, which reduces the fine-tuning overhead compared to most existing fine-tuning methods.

[0125] The personalized text graph model construction method based on causal fine-tuning provided in this embodiment avoids interference between subjects by aligning the attention representation corresponding to the cross-attention map of each subject with the identity-related visual representation, enhances the causal alignment of text representation and visual representation between subjects, and is conducive to using a multi-subject personalized text graph model to generate images that conform to the text representation.

[0126] In this embodiment, a method for generating a text map is provided, which can be used in the above-mentioned computer device. Figure 5 is a flow chart of a Wensheng diagram method according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0127] Step S501, obtaining a text description containing multiple subjects.

[0128] Step S502, input the text descriptions of multiple subjects into a preset multi-agent personalized text graph model to generate an image corresponding to the text description. The preset multi-agent personalized text graph model is obtained by using any embodiment of the personalized text graph model construction method based on causal fine-tuning.

[0129] Specifically, in the actual text-generated image process, a text description containing multiple subjects is directly obtained, and then it is input into a preset multi-subject personalized text-generated image model to obtain an image corresponding to the text description. It should be noted that the text description can also contain a subject, and an image corresponding to the text can be obtained, and the subject in the obtained image is not interfered by the background in the reference image and the other subject attributes, which alleviates the problem of missing or distorted subject identity.

[0130] like Figure 6 As shown in FIG. , the effect comparison between this embodiment and the related technology in the single-subject personalized text map task is shown in FIG. Figure 7 As shown in FIG. 1 , the effect comparison between this embodiment and the related technology in the multi-agent personalized text map task is shown in FIG. Figure 6 and Figure 7 It can be seen that the embodiments of the present invention can significantly improve the effectiveness and controllability of single-agent and multi-agent personalized cultural maps.

[0131] The text image method provided in this embodiment uses the image obtained by the multi-subject personalized text image model, reduces the interference of the background on the subject and the interference between the subjects, avoids the problem of subject missing, and improves the effectiveness and controllability of the multi-subject personalized text image.

[0132] In this embodiment, a personalized culture graph model construction system based on causal fine-tuning is also provided, which is used to implement the above-mentioned personalized culture graph model construction method embodiment and preferred implementation method based on causal fine-tuning, which have been described and will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0133] This embodiment provides a personalized cultural graph model construction system based on causal fine-tuning, such as Figure 8 As shown, including:

[0134] The data enhancement module 801 is used to obtain multiple reference images of multiple subjects and text descriptions of each subject, and merge and enhance the multiple reference images of each subject to obtain multiple merged enhanced images, and merge and enhance the text descriptions of each subject to obtain a merged text description, wherein the merged enhanced image contains a reference image of each subject.

[0135] The subject-aware causal decoupling module 802 is used to extract identity-protected text representations, identity-related visual representations, and identity-irrelevant visual representations from the text descriptions, merged text descriptions, and merged enhanced images of each subject using a preset causal decoupling algorithm.

[0136] The diffusion feedforward module 803 is used to input the merged enhanced image, the identity-protected text representation, the identity-related visual representation, and the identity-irrelevant visual representation into a preset causal reasoning model to complete the causal denoising reasoning and obtain a cross-attention map.

[0137] The causally inspired cross-attention guidance module 804 is used to extract the cross-attention map of the corresponding subject based on the text representation of the protected identity, and align the cross-attention map of the corresponding subject with the visual representation related to the corresponding subject identity, so as to complete the construction of the multi-subject personalized text graph model.

[0138] In some optional implementations, the subject-perceived causal decoupling module 802 includes:

[0139] The text representation determination unit is used to input the text description of each subject and the combined text description into a preset text encoder to obtain a text representation for protecting the identity.

[0140] The visual representation determining unit is used to input the combined enhanced image into a preset image encoder to obtain a visual representation of the combined enhanced image.

[0141] The causal intervention unit is used to perform causal intervention on the visual representation using a preset mask set to obtain the counterfactual visual representation of each subject and the merged subject.

[0142] The spatial conversion unit is used to convert the counterfactual visual representations of each subject and the merged subject into a latent space similar to the text representation using a pre-trained model, so as to obtain identity-related visual representations and identity-independent visual representations.

[0143] The similarity judgment unit is used to maximize the first similarity between the text representation of the identity protection and the visual representation related to the identity, and minimize the second similarity between the text representation of the identity protection and the visual representation unrelated to the identity by using a preset causal decoupling function.

[0144] The causal decoupling unit is used to decouple the identity-related visual representation and the identity-irrelevant visual representation according to the first similarity and the second similarity.

[0145] In some optional embodiments, the causal intervention unit includes:

[0146] Mask determination subunit, used to design a preset mask set Among them, i represents the i-th subject, i=n+i represents the merged subject, the range of each mask is (0,1), and the visual representation corresponding to the merged enhanced image has the same latent space dimension.

[0147] The counterfactual visual representation subunit is used to obtain the counterfactual visual representation of each subject based on the preset mask of each subject and the visual representation of the merged enhanced image, and to obtain the counterfactual visual representation of the merged subject based on the preset mask of the merged subject and the visual representation of the merged enhanced image.

[0148] In some optional implementations, the causal-inspired cross-attention guidance module 804 includes:

[0149] The mean calculation unit is used to calculate the average of the cross-attention maps of all subjects.

[0150] The attention representation determination unit is used to input the average value into a preset image encoder to obtain the attention representation corresponding to the cross-attention map of each subject.

[0151] A representation alignment unit is used to align the attention representations corresponding to the cross-attention maps of each subject with the identity-related visual representations based on the attention representations corresponding to the cross-attention maps of each subject and the identity-related visual representations.

[0152] In this embodiment, a Wensheng diagram system is also provided, which is used to implement the Wensheng diagram method embodiment and preferred implementation mode described above, which have been described and will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0153] This embodiment provides a cultural graph system, such as Fig. 9 As shown, including:

[0154] The text acquisition module 901 is used to acquire a text description containing multiple subjects.

[0155] The text graph module 902 is used to input the text descriptions of multiple subjects into a preset multi-subject personalized text graph model to generate an image corresponding to the text description. The preset multi-subject personalized text graph model is obtained by using any embodiment of the personalized text graph model construction method based on causal fine-tuning described above.

[0156] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0157] The personalized gram model construction system based on causal fine-tuning and the gram system in the above-mentioned embodiments are presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0158] The embodiment of the present invention also provides a computer device having the above Figure 8 The personalized cultural graph model construction system based on causal fine-tuning shown above Fig. 9 The Vincent diagram system shown.

[0159] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0160] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0161] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0162] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0163] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0164] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0165] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0166] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A personalized cultural graph model construction method based on causal fine-tuning, characterized in that: The method comprises: Acquire multiple reference images of multiple subjects and text descriptions of each subject, merge and enhance the multiple reference images of each subject to obtain multiple merged enhanced images, and merge and enhance the text descriptions of each subject to obtain a merged text description, wherein the merged enhanced image includes a reference image of each subject; Using the preset causal decoupling algorithm, the identity-protected text representation, identity-related visual representation, and identity-irrelevant visual representation are extracted from the text description, merged text description, and merged enhanced image of each subject respectively; Inputting the combined enhanced image, the identity-protected text representation, the identity-related visual representation, and the identity-irrelevant visual representation into a preset causal reasoning model, completing causal denoising reasoning, and obtaining a cross-attention map; The cross-attention map of the corresponding subject is extracted according to the text representation of the protected identity, and the cross-attention map of the corresponding subject is aligned with the visual representation related to the corresponding subject identity to complete the construction of the multi-subject personalized text graph model.

2. The method according to claim 1, characterized in that: Using the preset causal decoupling algorithm, the identity-protected text representation, identity-related visual representation, and identity-irrelevant visual representation are extracted from the text description, merged text description, and merged enhanced image of each subject, including: Inputting the text description of each subject and the combined text description into a preset text encoder to obtain a text representation that protects the identity; Inputting the merged enhanced image into a preset image encoder to obtain a visual representation of the merged enhanced image; Causally intervening the visual representation using a preset mask set to obtain counterfactual visual representations of each subject and the combined subject; The counterfactual visual representations of the subjects and the merged subjects are converted into a latent space similar to the text representation by using a pre-trained model, so as to obtain identity-related visual representations and identity-independent visual representations; By using a preset causal decoupling function, a first similarity between the identity-protecting textual representation and the identity-related visual representation is maximized, and a second similarity between the identity-protecting textual representation and the identity-irrelevant visual representation is minimized; According to the first similarity and the second similarity, the identity-related visual representation and the identity-irrelevant visual representation are decoupled.

3. The method according to claim 2, characterized in that The visual representation is causally intervened by using a preset mask set to obtain counterfactual visual representations of each subject and the combined subject, including: Design preset mask collection Among them, M i (i=1,2,...,n) represents the mask of the ith subject, M n+1 Masks representing the merged subjects, each mask ranges from (0,1) and has the same latent space dimension as the visual representation corresponding to the merged enhanced image; According to the preset mask of each subject and the visual representation of the merged enhanced image, the counterfactual visual representation of each subject is obtained, and according to the preset mask of the merged subject and the visual representation of the merged enhanced image, the counterfactual visual representation of the merged subject is obtained.

4. The method according to claim 2, characterized in that: The method of using the pre-trained model to convert the counterfactual visual representations of each subject and the merged subject into a latent space similar to the textual representation includes: Using multi-layer perceptrons and residual connections, the counterfactual visual representations of the subjects and the merged subjects are converted into a latent space similar to the text representation. The formula is: f i =MLP(M i *f ms +MLP(M i *f ms )),i=1,2,...,n+1 in, f i (i=1,2,...,n), They represent the identity-related visual representation and identity-independent visual representation of the i-th subject, respectively, and f i (i=n+1), They represent the identity-related visual representation and identity-irrelevant visual representation of the merged subject respectively.

5. The method according to claim 1, characterized in that: The aligning the cross-attention map of the corresponding subject with the visual representation associated with the identity of the corresponding subject comprises: Calculate the average of the cross-attention maps of all subjects; Inputting the average value into a preset image encoder to obtain an attention representation corresponding to the cross-attention map of each subject; According to the attention representation corresponding to the cross-attention map of each subject and the identity-related visual representation, the attention representation corresponding to the cross-attention map of each subject is aligned with the identity-related visual representation.

6. A Vincent diagram method, characterized in that: The method comprises: Get a text description containing multiple subjects; The text descriptions of the multiple subjects are input into a preset multi-subject personalized text graph model to generate an image corresponding to the text description. The preset multi-subject personalized text graph model is obtained by using the personalized text graph model construction method based on causal fine-tuning according to any one of claims 1 to 5.

7. A personalized cultural graph model construction system based on causal fine-tuning, characterized in that: The system comprises: A data enhancement module, used to obtain multiple reference images of multiple subjects and text descriptions of each subject, and merge and enhance the multiple reference images of each subject to obtain multiple merged enhanced images, and merge and enhance the text descriptions of each subject to obtain a merged text description, wherein the merged enhanced image includes a reference image of each subject; A subject-aware causal decoupling module is used to extract identity-protected text representations, identity-related visual representations, and identity-irrelevant visual representations from the text descriptions, merged text descriptions, and merged enhanced images of each subject using a preset causal decoupling algorithm; A diffusion feedforward module, used for inputting the combined enhanced image, the identity-protected text representation, the identity-related visual representation, and the identity-irrelevant visual representation into a preset causal reasoning model, completing causal denoising reasoning, and obtaining a cross-attention map; A causally inspired cross-attention guidance module is used to extract the cross-attention map of the corresponding subject based on the text representation of the protected identity, and align the cross-attention map of the corresponding subject with the visual representation related to the corresponding subject identity, so as to complete the construction of a multi-subject personalized text graph model.

8. A Wensheng graph system, characterized in that: The system comprises: A text acquisition module is used to acquire a text description containing multiple subjects; A text graph module is used to input the text descriptions of the multiple subjects into a preset multi-subject personalized text graph model to generate an image corresponding to the text description, wherein the preset multi-subject personalized text graph model is obtained by using the personalized text graph model construction method based on causal fine-tuning according to any one of claims 1 to 5.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the personalized Vincent graph model construction method based on causal fine-tuning according to any one of claims 1 to 5, or executes the Vincent graph method according to claim 6 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the personalized Vincent graph model construction method based on causal fine-tuning according to any one of claims 1 to 5, or to execute the Vincent graph method according to claim 6.

Images