Conditional image generation method and apparatus, electronic device, and storage medium

Abstract

The present application relates to the technical field of image processing, and discloses a conditional image generation method and device, electronic equipment and storage medium, comprising: obtaining a target image generation model comprising a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network and an image decoding network; processing image generation requirement data through the feature encoding network to obtain a requirement feature encoding vector, and converting the requirement feature encoding vector into an image feature vector through the intermediate feature extraction network; obtaining a feature correlation causal graph by operating the requirement feature encoding vector and the image feature vector through the causal discovery network; distributing the image feature weight by using the causal perception adaptive attention network and the feature correlation causal graph; projecting and modulating the image feature vector according to the image feature weight to obtain a target image feature, and finally synthesizing the target image through the image decoding network. The present application can be applied to the financial technology scene and the health care scene, and improves the accuracy of conditional image generation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology and image processing technology, and can be applied to fintech and healthcare scenarios. In particular, it relates to a conditional image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Conditional image generation methods are used to generate images that meet semantic requirements based on given conditional information, thus satisfying the need for customized image content in different scenarios. Taking the risk control sample expansion scenario in the financial field as an example, conditional image generation methods can generate rare and abnormal financial document images according to user-specified document formats and fraud characteristics. This enriches the training sample pool of the risk control recognition model, improving its accuracy in fraud risk detection. Taking the medical image data construction scenario in the healthcare field as an example, conditional image generation can generate medical image samples of rare cases according to user-specified lesion types and lesion degrees. This compensates for the lack of sufficient clinical case data and also supports the optimized training of medical auxiliary diagnostic models.

[0003] Currently, most conditional image generation methods rely on statistical patterns presented in the training data to build fixed generation logic. Therefore, they can only adapt to limited scenarios covered by the training data. For example, in a task of generating animal images by category, all images labeled as cats in the training data have green grass backgrounds. In this case, the image generation model, during its learning process, will use the statistical associations presented in this batch of training data to form a fixed generation logic that generates cat images with grass backgrounds. However, when generating cat images with snowy backgrounds, because this scene is not covered by this batch of training data, the image generation model cannot generate qualified images that meet the requirements. Due to this limitation, when faced with new scenes not covered by the training data, the images generated by the image generation model will deviate significantly from the real image features, inherent semantic logic, and actual usage requirements, resulting in inaccurate final images. Therefore, improving the accuracy of conditional image generation has become an urgent technical problem to be solved. Summary of the Invention

[0004] This invention provides a conditional image generation method, apparatus, computer device, and medium for artificial intelligence, aiming to solve the technical problems of existing conditional image generation methods that rely on statistical regularities of training data for feature association, lack explicit modeling of the causal structure of features, resulting in decreased generation quality, insufficient generalization ability, and lack of interpretability of feature selection process in distribution offset scenarios, thereby improving the accuracy of conditional image generation.

[0005] Firstly, a conditional image generation method is provided, including: Obtain a target image generation model, wherein the target image generation model includes a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network, and an image decoding network; Based on the feature coding network, feature coding is performed on the pre-acquired image generation requirement data to obtain the requirement feature coding vector; Based on the intermediate feature extraction network, the required feature encoding vector is transformed into an image feature vector. Based on the causal discovery network, causal association operations are performed on the demand feature encoding vector and the image feature vector to obtain a feature association causal graph. Based on the causal perception adaptive attention network and the feature association causal graph, attention is allocated to obtain image feature weights; Based on the image feature weights, feature projection modulation is performed on the image feature vector to obtain the target image features; The target image is obtained by image decoding and synthesis based on the image decoding network and the target image features.

[0006] Secondly, a conditional image generation apparatus is provided, comprising: The model loading module is used to obtain the target image generation model, wherein the target image generation model includes a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network, and an image decoding network; The feature encoding module is used to perform feature encoding on the pre-acquired image generation requirement data based on the feature encoding network to obtain the requirement feature encoding vector; The feature transformation module is used to perform image feature transformation on the required feature encoding vector based on the intermediate feature extraction network to obtain an image feature vector; The causal association module is used to perform causal association operations on the demand feature encoding vector and the image feature vector based on the causal discovery network to obtain a feature association causal graph. The quantitative evaluation module is used to perform attention allocation based on the causal perception adaptive attention network and the feature association causal graph to obtain image feature weights; The projection modulation module is used to perform feature projection modulation on the image feature vector based on the image feature weights to obtain the target image features; The decoding and synthesis module is used to perform image decoding and synthesis based on the image decoding network and the target image features to obtain the target image.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described conditional image generation method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described conditional image generation method.

[0009] In the aforementioned scheme implemented by the conditional image generation method, apparatus, computer equipment, and storage medium, a feature correlation causal graph between the requirement feature encoding vector and the image feature vector can be constructed first through a causal discovery network. This can fundamentally distinguish core features with a real causal relationship to the image generation requirement data from interference features with only statistical correlation, thereby avoiding feature selection bias caused by false correlations and ensuring the accuracy of feature selection. Secondly, relying on a causal perception adaptive attention network, image feature weights are dynamically allocated based on the causal importance of the feature correlation causal graph, and feature projection modulation is completed. This allows the generation process to prioritize the expression of core causal features and suppress the influence of non-causal interference features, thereby improving the rationality of feature aggregation and image representation. Finally, the target image generation model generates a target image that matches the input image generation requirement data based on the target image features. This effectively improves the problem of the generated content deviating from the requirements in out-of-distribution scenarios using traditional methods. In addition, it also improves the accuracy of conditional image generation. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a conditional image generation method according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of a conditional image generation method according to an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S10; Figure 4 yes Figure 2 A schematic diagram of a specific implementation of step S40; Figure 5 yes Figure 2 A schematic diagram of a specific implementation method for step S50; Figure 6 This is a schematic diagram of a conditional image generation device in one embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The conditional image generation method provided in this application embodiment can be applied to, for example, Figure 1In this application environment, the client communicates with the server via a network. The server can receive image generation requirement data input by the user from the client, and based on the feature encoding network in the pre-retrieved target image generation model, encode the image generation requirement data to obtain a requirement feature encoding vector; based on the intermediate feature extraction network, perform image feature transformation on the requirement feature encoding vector to obtain an image feature vector; based on the causal discovery network, perform causal association operations on the requirement feature encoding vector and the image feature vector to obtain a feature association causal graph; based on the causal perception adaptive attention network and the feature association causal graph, perform attention allocation to obtain image feature weights; based on the image feature weights, perform feature projection modulation on the image feature vector to obtain the target image features; finally, based on the image decoding network and the target image features, perform image decoding and synthesis to obtain the target image. In this application, for the conditional image generation business entity, a feature association causal graph is constructed by mining the true causal dependencies between demand features and visual features through a causal discovery network. Then, image feature weights are assigned by quantifying causal importance to complete feature projection modulation, prioritizing the expression of core causal features and effectively suppressing the influence of non-causal interference features. Finally, the target image is synthesized, ensuring a high degree of matching between the generated content and the input semantic constraints. This solves the problem that traditional image generation methods are easily affected by spurious correlations in out-of-distribution scenarios, leading to deviations from the demand in the generated content. It improves the accuracy, visual fidelity, and model generalization ability of conditional image generation, and enhances the efficiency and user experience of image sample expansion and intelligent design. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments further illustrates this application.

[0014] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the conditional image generation method provided in this embodiment of the invention includes the following steps: S10: Obtain the target image generation model, which includes a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network, and an image decoding network. S20: Based on the feature coding network, feature coding is performed on the pre-acquired image generation requirement data to obtain the requirement feature coding vector; S30: Based on the intermediate feature extraction network, the required feature encoding vector is transformed into an image feature vector. S40: Based on the causal discovery network, causal association operations are performed on the demand feature encoding vector and the image feature vector to obtain the feature association causal graph; S50: Based on the causal perception adaptive attention network and feature association causal graph, attention is allocated to obtain image feature weights; S60: Based on image feature weights, feature projection modulation is performed on the image feature vector to obtain the target image features; S70: Based on the image decoding network and target image features, image decoding and synthesis are performed to obtain the target image.

[0015] Steps S10 to S70 of this embodiment first construct a feature association causal graph between the requirement feature encoding vector and the image feature vector through a causal discovery network. This can fundamentally distinguish core features with a real causal relationship with the image generation requirement data from interference features with only statistical relationships, thereby avoiding feature selection bias caused by false correlations and ensuring the accuracy of feature selection. Second, relying on a causal perception adaptive attention network, image feature weights are dynamically allocated based on the causal importance of the feature association causal graph, and feature projection modulation is completed. This allows the generation process to prioritize the expression of core causal features and suppress the influence of non-causal interference features, thereby improving the rationality of feature aggregation and image representation. Finally, the target image generation model generates a target image that matches the input image generation requirement data based on the target image features. This effectively improves the problem of the generated content deviating from the requirements in out-of-distribution scenarios in traditional methods. In addition, it also improves the accuracy of conditional image generation.

[0016] The conditional image generation method provided in this application can be applied to the image sample generation backend server in fields such as smart healthcare or financial documents. This backend server can receive various image generation requirement data input by users in real time. For example, in the field of smart healthcare image sample expansion applications, medical research institutions often need to generate highly realistic medical image samples for specific rare lesions to train auxiliary diagnostic models. They often need to rely on the backend server to accurately extract the core pathological constraints in the image generation requirement data in order to accurately control the pathological morphology and tissue structure details of the generated images, ensuring the medical accuracy and usability of the generated images.

[0017] For example, a medical research team plans to generate a batch of MRI medical image samples that reflect early-stage liver malignant tumors and have blurred edges. The research team submits image generation request data containing pathological descriptions such as early-stage liver tumors and blurred edges to the backend server. After receiving the image generation request data, the backend server immediately calls a pre-trained target image generation model, which fully includes a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network, and an image decoding network. Subsequently, the backend server first performs feature encoding processing on the above image generation request data based on the feature encoding network, transforming it into a requirement feature encoding vector in the latent space. Then, based on the intermediate feature extraction network, it performs image feature transformation on the requirement feature encoding vector to generate an image feature vector that carries the image grayscale and preliminary tissue structure.

[0018] Subsequently, the backend server performs causal association calculations on the aforementioned required feature encoding vectors and image feature vectors based on a causal discovery network. This accurately uncovers the true causal dependency between the pathological description semantics and the local shadows in the image, and eliminates spurious statistical associations caused by differences in scanning equipment, thereby generating a feature association causal graph. Next, the backend server performs attention allocation based on a causal perception adaptive attention network and the aforementioned feature association causal graph, calculating the image feature weights for the tumor lesion area of ​​focus. Based on these image feature weights, it performs feature projection modulation on the image feature vector, amplifying the lesion feature components and suppressing background artifact noise, resulting in a clean target image feature. Finally, the backend server performs high-resolution image decoding and synthesis based on an image decoding network and the target image features, automatically generating a target MRI medical image that accurately matches the pathological characteristics of early liver tumors. The generated results are continuously synchronized to the sample database of the medical research team, effectively avoiding the risks of pathological feature distortion and artifact errors that are prone to occur in traditional generation models.

[0019] In step S10 of some embodiments, the target image generation model refers to a deep learning network model that has stable conditional image generation capabilities after training and optimization. For example, in the scenario of generating financial document samples, the target image generation model can be a generative adversarial network model specifically used to generate various types of financial document images, while in the scenario of expanding medical image samples, the target image generation model can be a deep generative network model specifically used to generate various types of lesion medical images.

[0020] Feature encoding networks are network components deployed at the front end of a target image generation model to convert raw input image generation requirement data into high-dimensional feature vectors that the target image generation model can process. For example, in the scenario of financial document generation, a feature encoding network can be a multilayer perceptron network that converts image generation requirement data such as document type and format requirements into latent space feature vectors. In the scenario of medical image generation, a feature encoding network can be an encoding network component that converts image generation requirement data such as lesion type and lesion degree into latent space feature vectors.

[0021] Intermediate feature extraction network refers to a network component deployed inside the target image generation model to convert the demand feature encoding vector into intermediate features in the image domain. It mainly undertakes the function of converting semantic features into visual features. For example, in the scenario of financial document generation, the intermediate feature extraction network can be a convolutional network component that converts the demand feature encoding vector corresponding to the document into the document's texture structure features. In the scenario of medical image generation, the intermediate feature extraction network can be a convolutional network component that converts the demand feature encoding vector corresponding to the lesion into the medical image tissue features.

[0022] Causal discovery networks are network components deployed within a target image generation model to uncover the true causal dependencies between demand feature encoding vectors and image feature vectors. They are primarily used to distinguish between causal features and non-causal interference features. For example, in a financial document generation scenario, a causal discovery network can be a network component that uncovers the causal relationship between the demand feature encoding vector corresponding to the document and the image feature vector corresponding to the core layout feature. In a medical image generation scenario, a causal discovery network can be a network component that uncovers the causal relationship between the demand feature encoding vector corresponding to the lesion and the image feature vector corresponding to the core image feature.

[0023] Causal perception adaptive attention network refers to a network component deployed inside the target image generation model that can dynamically adjust the image feature weights based on the feature association causal graph. It is mainly used to strengthen the weights of causal features and suppress the interference of non-causal features. For example, in the financial document generation scenario, the causal perception adaptive attention network can be a network component that dynamically adjusts the image feature weights corresponding to the layout features based on the feature association causal graph corresponding to the document. In the medical image generation scenario, the causal perception adaptive attention network can be a network component that dynamically adjusts the image feature weights corresponding to the lesion features based on the feature association causal graph corresponding to the image.

[0024] An image decoding network is a network component deployed on the backend of a target image generation model to convert the features of the target image into a visualized pixel image. For example, in a financial document generation scenario, the image decoding network can be an upsampling network that decodes the target image features corresponding to the document into a pixel-level electronic document image. In a medical image generation scenario, the image decoding network can be an upsampling network that decodes the target image features corresponding to the medical image into a pixel-level medical image.

[0025] In this embodiment, the conditional image generation system can retrieve a pre-trained and optimized target image generation model from a preset storage area. The training and optimization process of this target image generation model typically involves first acquiring conditional image generation training data, which includes sample requirement data and corresponding real sample images. Then, the conditional image generation training data is input into a pre-constructed initial image generation model for forward propagation calculation to obtain predicted sample images and predicted image feature weights. Next, pixel-level differences between the predicted sample images and real sample images are calculated to obtain the image reconstruction loss, which is used to constrain the generated image. The visual similarity between the generated image and the real image is used to ensure the authenticity of the generated content. Then, the causal regularization loss is calculated by sparse regularization of the predicted image feature weights. The causal regularization loss is used to constrain the sparsity of the feature association causal graph and ensure the effectiveness of causal feature selection. The image reconstruction loss and the causal regularization loss are then weighted and fused to obtain the total loss. Gradient backpropagation is then performed based on the total loss to update all network parameters of the initial image generation model. Finally, after multiple rounds of iterative training until the loss converges, the target image generation model, which includes a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network, and an image decoding network, can be obtained.

[0026] In detail, such as Figure 3 As shown, in step S10, the training process of the target conditional image generation model includes the following steps: S11: Obtain conditional image generation training data, wherein the conditional image generation training data includes sample requirement data and real sample images corresponding to the sample requirement data; S12: Based on the pre-built initial image generation model, forward propagation is performed on the sample requirement data to obtain the predicted sample image and the predicted image feature weights; S13: Calculate the difference between the predicted sample image and the real sample image to obtain the image reconstruction loss; S14: Regularization calculation is performed based on the predicted image feature weights to obtain the causal regularization loss; S15: Based on image reconstruction loss and causal regularization loss, the initial image generation model is optimized to obtain the target image generation model.

[0027] In step S11 of some embodiments, conditional image generation training data refers to a set of supervised learning samples used to optimize and update the parameters of the initial image generation model. For example, in the scenario of generating financial document samples, the conditional image generation training data may be a set of samples containing document attribute labels and corresponding document images, while in the scenario of expanding medical image samples, the conditional image generation training data may be a set of samples containing medical record description text and corresponding medical images.

[0028] Sample requirement data refers to the source data in the conditional image generation training data used to characterize the semantic constraints of the expected generated features. For example, in the financial document generation scenario, sample requirement data can be text labels that record the specified document format and anti-counterfeiting features, while in the medical image generation scenario, sample requirement data can be text descriptions that record specific disease types and lesion ranges.

[0029] Real sample images refer to pixel images in the conditional image generation training data that have an objective physical correspondence with the sample requirement data and serve as the benchmark for effect evaluation. For example, in the scenario of financial document generation, a real sample image can be an image of a real financial voucher that has been stamped and sealed, while in the scenario of medical image generation, a real sample image can be a medical image collected clinically that contains corresponding real lesion features.

[0030] In this embodiment, pre-collected and processed conditional image generation training data can be read from a preset storage area. The conditional image generation training data fully includes sample requirement data and real sample images that have a clear correspondence with the sample requirement data, thereby facilitating the provision of a standard reference benchmark for the subsequent network parameter update process.

[0031] In step S12 of some embodiments, the initial image generation model refers to a deep learning structure that has not yet been fully optimized with training data and whose parameters of each network component are in a state of random initialization or fine-tuning.

[0032] Predictive sample images refer to tentatively generated images synthesized by the initial image generation model based on the input sample requirement data through forward propagation. For example, in the scenario of financial document generation, the predictive sample image may be image data with blurred document outlines initially synthesized by the initial image generation model, while in the scenario of medical image generation, the predictive sample image may be image data containing suspected lesions initially output by the initial image generation model.

[0033] Predicted image feature weights refer to the tentative attention values ​​that the initial image generation model assigns to each feature dimension when processing sample demand data. For example, in the scenario of financial document generation, the predicted image feature weights can be the gating values ​​that the initial image generation model currently assigns to the texture features of financial seals, while in the scenario of medical image generation, the predicted image feature weights can be the evaluation values ​​that the initial image generation model currently assigns to the features of tumor edges.

[0034] In this embodiment, the extracted sample requirement data can be input into the pre-built initial image generation model. Then, the initial image generation model controls its internal feature encoding network and causal discovery network and other components to perform feature mapping and attention allocation calculations in sequence according to the set calculation logic, thereby completing a single forward propagation process and finally outputting the synthesized predicted sample image and the predicted image feature weights generated during the forward calculation process.

[0035] In step S13 of some embodiments, the image reconstruction loss refers to the error calculation result obtained by quantifying the difference between the predicted sample image and the real sample image in the visual feature space.

[0036] In this embodiment of the application, after successfully obtaining the predicted sample image, a predetermined loss function can be called to perform a pixel-level spatial alignment comparison with the corresponding real sample image. Then, differential numerical calculations are performed at the visual appearance and structural texture levels to accurately measure the specific degree to which the predicted sample image deviates from the real sample image, thereby deriving the image reconstruction loss used to constrain the authenticity of the generated content.

[0037] In step S14 of some embodiments, the causal regularization loss refers to the penalty value calculated by imposing specific constraints on the weights of predicted image features to induce the initial image generation model to tend to retain core causal features.

[0038] In this embodiment of the application, after receiving the predicted image feature weights output by the initial image generation model, a sparse computing paradigm can be used to perform sparse regularization calculation on them, thereby generating a causal regularization loss specifically used to constrain the sparsity of the causal association structure. It should be noted that the calculation mechanism of the causal regularization loss can force the initial image generation model to automatically filter out statistically false features that are irrelevant to the target generation task during the learning period.

[0039] Specifically, regularization is calculated based on the predicted image feature weights to obtain the causal regularization loss, including: The attention distribution matrix is ​​obtained by reshaping and parsing the predicted image feature weights. Sparse regularization calculations are performed based on the attention distribution matrix to obtain the causal regularization loss.

[0040] In this embodiment of the application, the attention distribution matrix refers to a two-dimensional data matrix obtained by reconstructing the spatial dimensions and algebraically analyzing the predicted image feature weights, which is used to quantitatively characterize the distribution state of attention energy among the feature dimensions. For example, in the financial document generation scenario, the attention distribution matrix can be a numerical matrix used to characterize the weight distribution relationship between document text elements and layout structure features, while in the medical image generation scenario, the attention distribution matrix can be a numerical matrix used to characterize the degree of weight correlation between the lesion anatomical region and background tissue features.

[0041] In this embodiment of the application, after the initial image generation model obtains the output predicted image feature weights, it can call the matrix reshaping module to perform multi-dimensional spatial analysis and rearrangement of the predicted image feature weights, converting them from a low-dimensional vector form into a two-dimensional matrix structure that can fully map the interaction intensity between each feature node, and thus successfully outputting an attention distribution matrix that can reflect the spatial distribution law of attention.

[0042] Furthermore, in this embodiment, the generated attention distribution matrix can be fed into the sparse penalty calculation unit of the initial image generation model as input data. The sparse penalty calculation unit performs regularization algebra operations on the attention distribution matrix by introducing a sparse benchmark operator based on the first or second norm. This allows for the precise extraction of the causal regularization loss value used to measure the redundancy of the causal structure, thereby providing a quantitative optimization pointer for the subsequent structural simplification and false feature suppression of the initial image generation model.

[0043] This application first performs matrix reshaping analysis on the predicted image feature weights output by the forward propagation process, converting the one-dimensional weight values ​​into an attention distribution matrix that can completely and intuitively map the attention distribution relationship between each feature dimension. This provides an input carrier with both structured and spatial interactive features for the subsequent refined quantitative evaluation of causal dependence. Secondly, based on this, sparse regularization calculation is performed on the attention distribution matrix to obtain causal regularization loss. This can force the weight components of non-core features that only have statistical pseudo-correlation with a rigorous mathematical penalty mechanism, causing the attention resources inside the network to be highly concentrated on core feature nodes with real causal dependence. This greatly improves the accuracy and guiding efficiency of causal regularization loss in constraining the sparsity of the feature causal relationship structure, laying a solid discriminant benchmark for training an image generation model with high anti-interference ability.

[0044] In step S15 of some embodiments, the calculated image reconstruction loss and causal regularization loss can be weighted and fused according to a set ratio to obtain a global loss value. Then, based on the global loss value, the parameter gradient backpropagation update operation is performed on all components inside the initial image generation model, so that the structural parameters of the initial image generation model gradually approach the optimal mapping state. Finally, after multiple iterations until the loss converges, the target image generation model with excellent and stable conditional image generation capabilities can be output.

[0045] It should be noted that the sample requirement data includes local semantic entities. Before optimizing the initial image generation model based on image reconstruction loss and causal regularization loss to obtain the target image generation model, the embodiments of this application may also perform the following operations: By performing variable substitution on local semantic entities, counterfactual demand data can be obtained. Based on the initial image generation model, feature weights are extracted from the counterfactual demand data to obtain the counterfactual image feature weights. Based on the predicted image feature weights and the counterfactual image feature weights, a distribution difference comparison is performed to obtain the counterfactual consistency loss.

[0046] In this embodiment of the application, a local semantic entity refers to an independent text word, symbol, or label unit in the input data that carries the core business meaning or characteristic attributes of a specific local area. For example, in the scenario of generating financial documents, a local semantic entity may be a core field such as the amount, company name, or seal type in the document, while in the scenario of generating medical images, a local semantic entity may be a key clinical term such as tumor lesion, nodule size, or lesion location in the medical record.

[0047] Counterfactual data refers to fictitious informational data that does not exist in objective reality but conforms to logical boundaries, constructed by replacing, falsifying, or modifying specific business entities in the original input text or label data. For example, in the scenario of generating financial documents, counterfactual data may be text data after artificially replacing the actual small amount of an invoice with a huge amount or modifying the invoice header to a fictitious entity. In the scenario of generating medical images, counterfactual data may be medical record description data after artificially replacing lung nodules with benign ones or magnifying the diameter of lesions by a factor of two.

[0048] Counterfactual image feature weights refer to the attention importance coefficients calculated and assigned to each visual feature dimension by the internal attention mechanism of the initial image generation model when receiving and processing counterfactual demand data. For example, in the scenario of financial document generation, the counterfactual image feature weights can be the weight values ​​assigned by the model to the feature of the amount layout area when faced with modified large invoice text. In the scenario of medical image generation, the counterfactual image feature weights can be the weight values ​​assigned by the model to the feature of the corresponding tissue area in the image when faced with artificially magnified description of lesions.

[0049] Counterfactual consistency loss refers to the error value measured by quantitatively comparing the difference between the attention distribution output by the initial image generation model under real sample demand data and the attention distribution output under counterfactual demand data.

[0050] In this embodiment, natural language processing or label recognition algorithms can be used to perform semantic parsing on the pre-acquired sample requirement data to lock the local semantic entities that restrict the local generation of the image. Then, while keeping other contextual semantics unchanged, preset alternative words or dummy variables with the same attributes are used to replace the above-mentioned local semantic entities, thereby synthesizing counterfactual requirement data for intervention testing.

[0051] Subsequently, in this embodiment of the application, the generated counterfactual requirement data can be input into the initial image generation model to be optimized. During the forward propagation process, the initial image generation model activates its core components such as the feature encoding network, intermediate feature extraction network, and causal discovery network, and performs nonlinear mapping and causal dependency measurement specifically for the natural semantic or category features of the counterfactual side, and finally outputs the counterfactual image feature weights corresponding to the counterfactual requirement data.

[0052] Furthermore, after the initial image generation model simultaneously obtains the predicted image feature weights generated on the real side and the counterfactual image feature weights generated on the counterfactual side, it can call the probability distribution comparison module to perform deep correlation analysis and difference calculation on the weight space distribution of the two using relative entropy, mean square error or other distribution difference measurement operators. This quantifies the degree of attention fluctuation of the model before and after semantic intervention, and thus derives the counterfactual consistency loss used to constrain causal invariance.

[0053] Based on the above scheme, the initial image generation model is optimized using image reconstruction loss and causal regularization loss to obtain the target image generation model, including: Based on counterfactual consistency loss, image reconstruction loss, and causal regularization loss, parameter gradient backpropagation is performed on the initial image generation model to obtain the target image generation model.

[0054] Specifically, in this embodiment, the counterfactual consistency loss, image reconstruction loss, and causal regularization loss can be combined and fused together by weighted summation to generate a global composite loss value that guides the optimization direction of the entire initial image generation model. Then, this global composite loss value is backpropagated layer by layer from the back end to the front end along the topology of the initial image generation model. This can correct the internal parameters of each component in the initial image generation model, such as the feature encoding network, the causal discovery network, and the image decoding network, until the overall performance of the initial image generation model converges, and finally the target image generation model can be obtained.

[0055] This application first constructs counterfactual demand data by substituting variables for local semantic entities in the sample demand data, thus creating external intervention conditions for the network model to actively explore non-real distribution data. Second, by extracting the corresponding counterfactual image feature weights and comparing their distribution differences with the predicted image feature weights, a counterfactual consistency loss is obtained. This allows for in-depth examination and constraint of the model's inherent causal stability in the face of semantic fine-tuning from the perspective of causal intervention, preventing the model from shifting its attention due to environmental interference or background pseudo-correlation. Finally, this counterfactual consistency loss is deeply integrated with the image reconstruction loss and causal regularization loss to jointly guide the backpropagation of parameter gradients and the optimization of overall parameters in the initial image generation model. This results in the final refined target image generation model that not only accurately controls the similarity of real data in pixel space but also possesses strong causal logic robustness and inference resilience, significantly improving the image synthesis accuracy and generalization generation performance when facing out-of-distribution data or adversarial demand attacks.

[0056] Steps S11 to S15 of this embodiment first obtain conditional image generation training data containing sample requirement data and real sample images, providing complete and reliable supervision guidance for parameter mapping and feature learning of deep neural networks. Second, regularization calculation is performed on the predicted image feature weights obtained from forward propagation to obtain causal regularization loss, so that the initial image generation model can be affected by sparsity constraints during iteration, thereby actively suppressing the overexpression of non-causal interference features and ensuring the purity of feature selection. Next, the causal regularization loss that constrains the causal discovery ability of the network is combined with the image reconstruction loss that constrains the visual realism of the generated image for joint optimization, which can comprehensively correct the joint calculation error of the initial image generation model in causal logic discrimination and pixel spatial distribution. Finally, iterative optimization processing is performed based on the above composite loss information, which effectively overcomes the inherent defect that the model is prone to fitting false correlation features in the dataset under the traditional training mode, thereby greatly improving the comprehensive generalization performance of the final target image generation model in accurately outputting high-quality target images when facing complex or even unseen sample requirement data.

[0057] In step S20 of some embodiments, image generation requirement data refers to input data used to limit the semantics, category, attributes, etc. of the generated image. It can serve as a constraint basis for image generation. For example, in the financial document generation scenario, image generation requirement data can be descriptive data used to limit the document type and anti-counterfeiting features of the generated image, while in the medical image generation scenario, image generation requirement data can be descriptive data used to limit the imaging modality and lesion attributes of the generated image.

[0058] The demand feature encoding vector refers to the high-dimensional vector data obtained after the image generation demand data is mapped by the feature encoding network. It carries the core semantic information of the image generation demand data. For example, in the financial document generation scenario, the demand feature encoding vector can be the latent space encoding vector that carries the format and attribute information of the financial document, while in the medical image generation scenario, the demand feature encoding vector can be the latent space encoding vector that carries the lesion and modality information of the medical image.

[0059] In this embodiment, after receiving the input image generation requirement data, the feature encoding network can adopt a three-layer multilayer perceptron network structure to perform linear transformation and nonlinear activation processing on the image generation requirement data layer by layer. The activation function can be a Gaussian error linear unit. Then, the image generation requirement data is mapped from the original space of natural semantics or category labels to the latent space dimension that the target image generation model can process, so as to output a requirement feature encoding vector with fixed dimension.

[0060] In step S30 of some embodiments, the image feature vector refers to the high-dimensional feature data of the image's own visual attributes generated by the demand feature encoding vector. It can be used to characterize the image's inherent features such as texture, structure, color, and grayscale. For example, in the above-mentioned financial document generation scenario, the image feature vector can be the visual feature vector of the texture and layout structure information of the corresponding generated financial document image. In the above-mentioned medical image generation scenario, the image feature vector can be the visual feature vector of the grayscale and tissue structure information of the corresponding generated medical image.

[0061] In this embodiment, after receiving the required feature encoding vector, the intermediate feature extraction network can rely on a progressively growing convolutional generation architecture to perform multi-layer convolution operations and upsampling on the required feature encoding vector to achieve feature domain transformation, thereby gradually converting the semantic dimension required feature encoding vector into image dimension visual features. Then, as the resolution level gradually increases, intermediate feature data corresponding to different resolutions can be generated sequentially, and finally, image feature vectors carrying visual attributes such as image texture, structure, and color can be obtained.

[0062] In step S40 of some embodiments, the feature association causal graph refers to a data structure that presents the causal dependency relationship between the corresponding variable of the demand feature encoding vector and the corresponding node of the image feature vector in the form of a directed acyclic graph. For example, in the financial document generation scenario, the feature association causal graph can be a graph structure that presents the causal relationship between the corresponding variable of the document category and the corresponding node of the layout feature. In the medical image generation scenario, the feature association causal graph can be a graph structure that presents the causal relationship between the corresponding variable of the lesion type and the corresponding node of the image feature.

[0063] In this embodiment, the causal discovery network can first extract semantic nodes from the demand feature encoding vector and visual nodes from the image feature vector. Then, the causal discovery network can adopt a twelve-layer encoder architecture based on a self-attention mechanism, and calculate the causal dependency strength between the nodes using a differentiable structural equation model. Next, the causal dependency strength between the nodes is optimized using a causal loss function that includes feature reconstruction terms and sparse constraint terms. Relaxation techniques are used to achieve differentiable updates of the graph structure. After iterative optimization, the graph structure removes false connections that only have statistical correlations and retains node connections with real causal dependencies. Finally, a feature association causal graph presented in the form of a directed acyclic graph can be obtained.

[0064] In detail, such as Figure 4 As shown, in step S40, which is based on the causal discovery network, causal association operations are performed on the demand feature encoding vector and the image feature vector to obtain the feature association causal graph, including: S41: Extract nodes from the demand feature encoding vector to obtain semantic nodes; S42: Extract nodes from the image feature vector to obtain visual nodes; S43: Calculate the mutual information between semantic nodes and visual nodes to obtain conditional mutual information; S44: Construct a causal structure based on conditional mutual information to obtain a feature association causal graph.

[0065] In steps S41 and S42 of some embodiments, a semantic node refers to an independent variable or feature unit extracted from the demand feature encoding vector to represent the core semantic concept in the original input text or label in the graph structure. For example, in the financial document generation scenario, a semantic node may be a text feature variable representing the amount of the document, the company name, or the type of invoice stamp, while in the medical image generation scenario, a semantic node may be a medical descriptive variable representing the type of lesion, the severity of the lesion, or the anatomical location of the tissue.

[0066] A visual node is an independent variable or feature unit extracted from an image feature vector to represent the visual spatial features or local texture attributes of an image in a graph structure. For example, in a financial document generation scenario, a visual node can be a local visual variable representing the color of a seal, the layout border, or the character line features in a financial document image. In a medical image generation scenario, a visual node can be a local visual variable representing the grayscale distribution of tissues, the shadow edge of a lesion, or the texture density features in a medical image.

[0067] In this embodiment, after receiving the required feature encoding vector output by the feature encoding network, the causal discovery network can call the semantic parsing submodule to perform internal dimensional structured decomposition and attribute decoupling on the aforementioned required feature encoding vector, dividing it into multiple high-dimensional feature subspaces with independent business meanings or semantic constraints, thereby successfully extracting semantic nodes that represent the core semantic concepts of specific texts or tags. In this embodiment, the causal discovery network can synchronously receive the image feature vector output by the intermediate feature extraction network, and use spatial pooling or local window sliding extraction algorithms to perform visual channel and spatial dimension feature mapping on the aforementioned image feature vector, decomposing it into a series of local feature units carrying local image texture, structure, color, or organizational morphology information, thereby successfully extracting visual nodes that can reflect the visual attributes of the image itself.

[0068] In step S43 of some embodiments, conditional mutual information refers to a mathematical measure used to quantitatively measure the true statistical correlation or information sharing degree between two random variables after excluding interference from third parties, under the premise of introducing known third-party dependent variables. For example, in the scenario of financial document generation, conditional mutual information can be a calculated value used to measure the information association strength between the semantic node of document amount and the visual node of amount layout after excluding the interference of background noise variables. In the scenario of medical image generation, conditional mutual information can be a measure used to measure the information association strength between the semantic node of tumor type and the visual node of image lesion area after excluding the interference of device artifact noise variables.

[0069] In this embodiment, after successfully separating the semantic nodes and visual nodes, the causal discovery network can introduce a nonlinear information measurement mechanism to calculate the difference between the joint probability distribution and the marginal probability distribution between the semantic nodes and the visual nodes under the condition of other relevant environmental variables. By performing high-order statistical integration or neural network approximation fitting operations, the conditional mutual information that can reflect the true correlation strength between the two types of nodes after excluding external interference can be accurately obtained.

[0070] In step S44 of some embodiments, the causal discovery network can use the calculated conditional mutual information as the basis for edge weight initialization, combine differentiable structural equation model or graph search algorithm, perform edge pruning and direction derivation processing on the complete graph structure composed of the semantic nodes and the visual nodes, and optimize the graph topology structure through a causal loss function containing sparse constraints, and finally construct and output a feature association causal graph presented in the form of a directed acyclic graph.

[0071] Steps S41 to S44 of this embodiment first extract semantic nodes and visual nodes from the requirement feature encoding vector and image feature vector respectively, providing a fine-grained, attribute-decoupled structured data carrier for subsequent cross-modal causal relationship mining. Second, based on the semantic nodes and visual nodes, nonlinear conditional mutual information calculation is performed, which can accurately quantify the true statistical correlation strength between the two types of cross-modal nodes under the premise of excluding third-party interference factors such as environmental background or noise variables. Then, based on the conditional mutual information, a causal structure is constructed to obtain a feature association causal graph, which can rigorously transform the complex nonlinear statistical correlation between the two types of cross-modal nodes into a directed acyclic causal dependency with a clear direction at the graph topology level. Thus, at the root of the feature layer, the discrimination and isolation of causal core features and non-causal false interference features are completed, providing extremely accurate and robust causal graph support for the subsequent model to adaptively allocate reasonable attention weights and ensure the robustness and generalization of conditional image generation.

[0072] In step S50 of some embodiments, image feature weight refers to the importance coefficient of image feature vectors in each dimension. For example, in the financial document generation scenario, image feature weight can be the importance coefficient of each format feature dimension of the corresponding financial document, while in the medical image generation scenario, image feature weight can be the importance coefficient of each tissue feature dimension of the corresponding medical image.

[0073] In this embodiment, the causal perception adaptive attention network can obtain the causal importance score corresponding to each visual node by parsing the feature association causal graph. Then, the causal importance score is concatenated with the image feature vector and the demand feature encoding vector after dimensionality reduction by average pooling. The concatenated vector is then fed into a multilayer perceptron to perform mapping calculation. After activation function, the attention gating coefficients corresponding to each feature dimension are output. The activation function can be, but is not limited to, the logistic activation function. Finally, the attention gating coefficients are normalized to obtain the image feature weights corresponding to each image feature dimension. It should be noted that the value of the image feature weight is positively correlated with the causal importance of the corresponding feature.

[0074] In detail, such as Figure 5As shown, in step S50, attention is assigned based on the causal perception adaptive attention network and the feature association causal graph to obtain image feature weights, including: S51: Perform path identification on the feature association causal graph to obtain pseudo-related confusing paths; S52: Eliminate the causal effect of spurious correlation confusion paths to obtain the target causal effect value; S53: Map the target causal effect value to obtain the attention gating coefficient; S54: Normalize the attention gating coefficients to obtain the image feature weights.

[0075] In step S51 of some embodiments, a pseudo-correlation confusion path refers to an indirect connection path in a feature association causal graph that is not caused by a real causal relationship but generates a statistical dependency association between semantic nodes and visual nodes due to the existence of common cause confusion variables. For example, in the scenario of financial document generation, a pseudo-correlation confusion path may be a false association path formed between handwriting features and background features because a certain printer model simultaneously affects the clarity of handwriting and the paper texture. In the scenario of medical image generation, a pseudo-correlation confusion path may be a false association path formed between tissue features and edge features because a certain scanner model simultaneously interferes with tissue grayscale and image edges.

[0076] In this embodiment, after obtaining the feature association causal graph, the causal perception adaptive attention network can call the graph path search algorithm to traverse and parse the topology of the feature association causal graph, accurately identify the non-causal directed paths between semantic nodes and visual nodes caused by confusion factors, and thus accurately locate the pseudo-related confusion paths that cause the feature selection deviation.

[0077] In step S52 of some embodiments, the target causal effect value refers to the quantitative causal value that accurately reflects the real and direct driving effect of semantic nodes on visual nodes after implementing causal intervention or path blocking and completely eliminating the interference of confounding variables in the causal perception adaptive attention network. For example, in the financial document generation scenario, the target causal effect value may be the net causal influence of the semantic node of document amount value on the visual features of amount layout after removing the interference of printing equipment. In the medical image generation scenario, the target causal effect value may be the pure causal dependence strength of the semantic node of lesion type on the visual features of tumor area in the image after removing the interference of scanning noise.

[0078] In this embodiment, after locking the aforementioned spurious correlation confusion path, the causal perception adaptive attention network can implement a causal intervention mechanism based on the backdoor adjustment criterion or the frontdoor adjustment criterion. By truncating the connection edges pointing from the confusion variables to the relevant nodes, the path blocking is performed on the feature association causal graph, thereby calculating the target causal effect value of the pure causal dependency strength transmitted from the semantic node to the visual node after excluding false statistical associations.

[0079] In step S53 of some embodiments, the attention gating coefficient refers to a nonlinear adjustment coefficient obtained by mapping the target causal effect value using a mathematical function. This coefficient is used to dynamically control and adjust the information of each visual feature channel or spatial dimension through the valve size. For example, in the financial document generation scenario, the attention gating coefficient can be a nonlinear gating coefficient value used to indicate that the model should focus on amplifying the core features of the amount and turn off the irrelevant background texture channel. In the medical image generation scenario, the attention gating coefficient can be a nonlinear gating coefficient value used to indicate that the model should highly activate the lesion contour features and suppress the irrelevant artifact noise channel.

[0080] In this embodiment, the causal perception adaptive attention network can use the calculated target causal effect value as an independent variable and feed it into a linear projection or multilayer perceptron for nonlinear coefficient mapping calculation. Through function transformation, it can project the value onto a specified numerical gating interval, thereby successfully outputting the attention gating coefficients corresponding to each image feature dimension.

[0081] In step S54 of some embodiments, after the causal perception adaptive attention network produces the above attention gating coefficients, it can call the normalization processing module, use the logistic activation function or the flexible maximum function to perform numerical normalization transformation on the above attention gating coefficients in all feature channels or spatial dimensions, eliminate feature scale differences, and finally obtain image feature weights that carry the true causal importance.

[0082] Steps S51 to S54 of this embodiment first obtain pseudo-correlation confusion paths by path identification of the feature association causal graph. This can accurately capture non-causal false connections in the model caused by the environment or data distribution, thus providing a precise target for subsequent elimination of feature dependency bias. Second, by eliminating the causal effect of the above pseudo-correlation confusion paths, the target causal effect value is obtained. This can successfully cut off the interference of non-causal confounding factors on feature expression, so that the extracted effect value can purely and truly reflect the causal driving force of semantics on the visual dimension. Then, the above target causal effect value is mapped to obtain attention gating coefficients, realizing the non-linear expression of causal strength. Finally, the image feature weights are output through normalization processing, ensuring the physical meaning and algebraic stability of the weight distribution. This allows attention resources to be highly consistent with the core causal features, greatly improving the accuracy of feature selection in the conditional image generation process and effectively avoiding distortion of generated content caused by false correlations in out-of-distribution scenes.

[0083] In step S60 of some embodiments, the target image feature refers to the image feature data after the image feature vector has been filtered to enhance the causal core features. For example, in the financial document generation scenario, the target image feature may be the financial document image feature that enhances the core layout and anti-counterfeiting attributes, while in the medical image generation scenario, the target image feature may be the medical image feature that enhances the core lesion and tissue attributes.

[0084] In this embodiment, the image feature vector can be processed by three independent linear projections to obtain three types of feature projection matrices: query feature projection matrix, key feature projection matrix, and value feature projection matrix. Then, the image feature weights are aligned with these three types of feature projection matrices to ensure that the weight dimensions correspond one-to-one with the feature dimensions of the feature projection matrices. Next, the three types of feature projection matrices are modulated in an element-wise weighted manner to amplify the projection components of features with high causal importance and suppress the projection components of non-causal interference features. Finally, the modulated three types of projection matrices are integrated to obtain the target image features.

[0085] In detail, based on image feature weights, feature projection modulation is performed on the image feature vector to obtain the target image features, including: The image feature vector is subjected to feature projection processing to obtain the feature projection matrix, which contains multiple feature dimensions. Based on multiple feature dimensions, the image feature weights are aligned to obtain aligned feature weights. Based on the alignment feature weights, feature modulation processing is performed on the feature projection matrix to obtain the target image features.

[0086] In some embodiments, the feature projection matrix refers to a data matrix generated by linear transformation or matrix mapping of image feature vectors, used to characterize multidimensional information of local attributes and global structure of an image in different representation subspaces. For example, in the scenario of financial document generation, the feature projection matrix may be a data matrix containing the expression forms of features such as financial document seals, background patterns, and text in different query or key-value spaces. In the scenario of medical image generation, the feature projection matrix may be a data matrix containing features such as lesion edges and tissue density in medical images after transformation in different representation spaces.

[0087] Feature dimension refers to the independent vector channel or feature component axis in the feature projection matrix used to characterize the multi-level attributes of an image. For example, in the scenario of financial document generation, the feature dimension can be the feature channel corresponding to specific local layout attributes such as document character outlines and table lines, while in the scenario of medical imaging, the feature dimension can be the feature channel corresponding to specific pathological features such as the gray-scale distribution of tissue nodules and the details of lesion texture.

[0088] Alignment feature weights refer to consistent weight data that perfectly matches the feature projection matrix in both channel and spatial dimensions after the initially acquired image feature weights have undergone shape reshaping or spatial expansion. For example, in the scenario of financial document generation, alignment feature weights can be importance coefficient matrices that perfectly match each channel of the document anti-counterfeiting feature projection matrix and can be directly used for element-level multiplication. In the scenario of medical image generation, alignment feature weights can be causal importance matrices that perfectly correspond to each component axis of the lesion morphology feature projection matrix and can be directly used for feature modulation.

[0089] In this embodiment, the conditional image generation system can first receive the image feature vector output by the intermediate feature extraction network, and then perform three independent linear projection processes on it using a linear mapping layer. Through the multiplication operation of different weight matrices, the original image feature vector is mapped to different representation subspaces, thereby obtaining three types of feature projection matrices containing multiple feature dimensions: query feature projection matrix, key feature projection matrix, and value feature projection matrix. This greatly enriches the expression space and semantic tension of image features.

[0090] Next, after obtaining the aforementioned feature projection matrix and the image feature weights output by the causal perception adaptive attention network, the conditional image generation system can perform spatial and channel-level parsing and matching operations on the image feature weights based on the specific tensor shape of each feature dimension in the feature projection matrix, using matrix reshaping, dimension expansion, or tensor broadcasting mechanisms. This ensures that the weight dimension of the image feature weights corresponds one-to-one with the feature dimension of the three types of feature projection matrices, thereby successfully generating aligned feature weights.

[0091] Finally, the conditional image generation system can utilize the aforementioned alignment feature weights to perform feature modulation processing on the three types of feature projection matrices respectively in an element-wise weighted or matrix multiplication manner. During the modulation process, the system will adaptively amplify those projection components with high causal importance features, while significantly suppressing those projection components with non-causal interference features. Furthermore, by fusing and integrating the three types of feature projection matrices that have been precisely modulated, the system can successfully output target image features that enhance the core causal elements.

[0092] This application first obtains a feature projection matrix containing multiple feature dimensions by performing feature projection processing on the image feature vector. This maps the original visual features to a richer and more multi-dimensional representation subspace, greatly improving the expressive capacity and flexibility of image features. Second, based on multiple feature dimensions, the image feature weights are dimensionally aligned to obtain aligned feature weights. This effectively breaks down the dimensional barrier between the weight data and the feature matrix, providing a precise mathematical matching basis for subsequent fine-grained feature intervention. Finally, based on the aligned feature weights, the feature projection matrix is ​​subjected to feature modulation processing. This can accurately amplify the expressive energy of causal core features in an element-wise weighted manner and effectively suppress the noise components of non-causal interference features. This results in the synthesis of semantically clear and visually robust target image features, significantly enhancing the feature focusing ability of the conditional image generation model under complex requirements and the visual fidelity of the generated image.

[0093] In step S70 of some embodiments, the target image refers to the final image result that meets the input image generation requirement data constraints. For example, in the financial document generation scenario, the target image may be a financial electronic document image that meets the specified format and attribute requirements, while in the medical image generation scenario, the target image may be a medical image sample image that meets the specified lesion and modality requirements.

[0094] In this embodiment, the image decoding network can first receive the input target image features, and then gradually improve the spatial resolution of the target image features through multi-layer upsampling convolution and feature fusion operations. Next, it gradually fills in the image's detailed texture and structural information at each resolution level, and then maps the high-dimensional feature values ​​of the target image features to the standard pixel value range through the activation function of the output layer. This completes the transformation from high-dimensional abstract features to visualized pixel images, and finally obtains a target image that matches the semantic constraints of the image generation requirements.

[0095] As can be seen, in the above scheme, a causal correlation causal graph between the requirement feature encoding vector and the image feature vector can be constructed first through a causal discovery network. This can fundamentally distinguish the core features that have a real causal relationship with the image generation requirement data from the interference features that only have a statistical relationship, thereby avoiding feature selection bias caused by false correlations and ensuring the accuracy of feature selection. Secondly, relying on a causal perception adaptive attention network, the image feature weights are dynamically allocated based on the causal importance of the feature correlation causal graph and feature projection modulation is completed. This allows the generation process to prioritize the expression of core causal features and suppress the influence of non-causal interference features, thereby improving the rationality of feature aggregation and image representation. Finally, the target image generation model generates a target image that matches the input image generation requirement data based on the target image features. This effectively improves the problem of the generated content deviating from the requirements in out-of-distribution scenarios in traditional methods. In addition, it also improves the accuracy of conditional image generation.

[0096] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0097] In one embodiment, a conditional image generation apparatus is provided, which corresponds one-to-one with the conditional image generation methods described in the above embodiments. For example... Figure 6 As shown, the conditional image generation device includes a model loading module, a feature encoding module, a feature transformation module, a causal correlation module, a quantization evaluation module, a projection modulation module, and a decoding and synthesis module. Detailed descriptions of each functional module are as follows: The model loading module is used to obtain the target image generation model, which includes a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network, and an image decoding network. The feature encoding module is used to perform feature encoding on the pre-acquired image to generate the required feature encoding vector based on the feature encoding network; The feature transformation module is used to transform the required feature encoding vector into an image feature vector based on the intermediate feature extraction network. The causal association module is used to perform causal association operations on the demand feature encoding vector and image feature vector based on the causal discovery network to obtain the feature association causal graph. The quantitative evaluation module is used to perform attention allocation based on the causal perception adaptive attention network and feature association causal graph to obtain image feature weights; The projection modulation module is used to perform feature projection modulation on the image feature vector based on image feature weights to obtain the target image features; The decoding and synthesis module is used to decode and synthesize images based on the image decoding network and the features of the target image to obtain the target image.

[0098] In one embodiment, the model loading module is specifically used for: Obtain conditional image generation training data, which includes sample requirement data and real sample images corresponding to the sample requirement data; Based on the pre-built initial image generation model, the sample requirement data is forward propagated to obtain the predicted sample image and the predicted image feature weights. The image reconstruction loss is obtained by calculating the difference between the predicted sample image and the real sample image; The causal regularization loss is obtained by performing regularization calculation based on the predicted image feature weights. Based on image reconstruction loss and causal regularization loss, the initial image generation model is optimized to obtain the target image generation model.

[0099] In one embodiment, the model loading module is further configured to: The attention distribution matrix is ​​obtained by reshaping and parsing the predicted image feature weights. Sparse regularization calculations are performed based on the attention distribution matrix to obtain the causal regularization loss.

[0100] In one embodiment, the model loading module is further configured to: By performing variable substitution on local semantic entities, counterfactual demand data can be obtained. Based on the initial image generation model, feature weights are extracted from the counterfactual demand data to obtain the counterfactual image feature weights. Based on the predicted image feature weights and the counterfactual image feature weights, the distribution difference is compared to obtain the counterfactual consistency loss; Based on counterfactual consistency loss, image reconstruction loss, and causal regularization loss, parameter gradient backpropagation is performed on the initial image generation model to obtain the target image generation model.

[0101] In one embodiment, the causal association module is further configured to: Semantic nodes are obtained by extracting nodes from the demand feature encoding vector; Node extraction is performed on the image feature vector to obtain visual nodes; Conditional mutual information is obtained by calculating mutual information between semantic nodes and visual nodes. Based on conditional mutual information, a causal structure is constructed to obtain a feature-related causal graph.

[0102] In one embodiment, the quantitative evaluation module is specifically used for: Path identification is performed on the feature association causal graph to obtain pseudo-related confusing paths; The causal effect of spurious correlation confounding paths is eliminated to obtain the target causal effect value; By mapping the target causal effect value to coefficients, attention gating coefficients are obtained; The attention gating coefficients are normalized to obtain the image feature weights.

[0103] In one embodiment, the projection modulation module is specifically used for: The image feature vector is subjected to feature projection processing to obtain the feature projection matrix, which contains multiple feature dimensions. Based on multiple feature dimensions, the image feature weights are aligned to obtain aligned feature weights. Based on the alignment feature weights, feature modulation processing is performed on the feature projection matrix to obtain the target image features.

[0104] This invention provides a conditional image generation device. First, a causal discovery network is used to construct a feature association causal graph between the requirement feature encoding vector and the image feature vector. This can fundamentally distinguish core features with a real causal relationship to the image generation requirement data from interference features with only statistical correlation, thereby avoiding feature selection bias caused by false correlations and ensuring the accuracy of feature selection. Second, relying on a causal perception adaptive attention network, image feature weights are dynamically allocated based on the causal importance of the feature association causal graph, and feature projection modulation is completed. This allows the generation process to prioritize the expression of core causal features and suppress the influence of non-causal interference features, thereby improving the rationality of feature aggregation and image representation. Finally, the target image generation model generates a target image that matches the input image generation requirement data based on the target image features. This effectively improves the problem of the generated content deviating from the requirements in out-of-distribution scenarios in traditional methods. In addition, it also improves the accuracy of conditional image generation.

[0105] Specific limitations regarding the conditional image generation device can be found in the limitations of the conditional image generation method described above, and will not be repeated here. Each module in the aforementioned conditional image generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0106] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a conditional image generation method on the server side.

[0107] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a conditional image generation method on the client side. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the target image generation model, which includes a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network, and an image decoding network; Based on the feature coding network, the pre-acquired image generation requirement data is feature-coded to obtain the requirement feature coding vector; Based on the intermediate feature extraction network, the required feature encoding vector is transformed into an image feature vector. Based on the causal discovery network, causal association operations are performed on the demand feature encoding vector and the image feature vector to obtain the feature association causal graph. Based on the causal perception adaptive attention network and the feature association causal graph, attention is assigned to obtain image feature weights; Based on image feature weights, feature projection modulation is performed on the image feature vector to obtain the target image features; The target image is obtained by decoding and synthesizing images based on image decoding networks and target image features.

[0108] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the target image generation model, which includes a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network, and an image decoding network; Based on the feature coding network, the pre-acquired image generation requirement data is feature-coded to obtain the requirement feature coding vector; Based on the intermediate feature extraction network, the required feature encoding vector is transformed into an image feature vector. Based on the causal discovery network, causal association operations are performed on the demand feature encoding vector and the image feature vector to obtain the feature association causal graph. Based on the causal perception adaptive attention network and the feature association causal graph, attention is assigned to obtain image feature weights; Based on image feature weights, feature projection modulation is performed on the image feature vector to obtain the target image features; The target image is obtained by decoding and synthesizing images based on image decoding networks and target image features.

[0109] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0112] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0113] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A conditional image generation method, characterized in that, include: Obtain a target image generation model, wherein the target image generation model includes a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network, and an image decoding network; Based on the feature coding network, feature coding is performed on the pre-acquired image generation requirement data to obtain the requirement feature coding vector; Based on the intermediate feature extraction network, the required feature encoding vector is transformed into an image feature vector. Based on the causal discovery network, causal association operations are performed on the demand feature encoding vector and the image feature vector to obtain a feature association causal graph. Based on the causal perception adaptive attention network and the feature association causal graph, attention is allocated to obtain image feature weights; Based on the image feature weights, feature projection modulation is performed on the image feature vector to obtain the target image features; The target image is obtained by image decoding and synthesis based on the image decoding network and the target image features.

2. The conditional image generation method as described in claim 1, characterized in that, The training steps of the target image generation model include: Acquire conditional image generation training data, wherein the conditional image generation training data includes sample requirement data and real sample images corresponding to the sample requirement data; Based on the pre-built initial image generation model, the sample requirement data is forward-propagated to obtain the predicted sample image and the predicted image feature weights. The image reconstruction loss is obtained by calculating the difference between the predicted sample image and the real sample image. Regularization calculation is performed based on the predicted image feature weights to obtain the causal regularization loss. Based on the image reconstruction loss and the causal regularization loss, the initial image generation model is optimized to obtain the target image generation model.

3. The conditional image generation method as described in claim 2, characterized in that, The regularization calculation based on the predicted image feature weights to obtain the causal regularization loss includes: The attention distribution matrix is ​​obtained by performing matrix reshaping and parsing on the predicted image feature weights; Based on the attention distribution matrix, sparse regularization calculation is performed to obtain the causal regularization loss.

4. The conditional image generation method as described in claim 2, characterized in that, The sample requirement data includes local semantic entities; The training steps of the target image generation model also include: By performing variable substitution on the local semantic entities, counterfactual demand data is obtained; Based on the initial image generation model, feature weights are extracted from the counterfactual demand data to obtain counterfactual image feature weights. Based on the predicted image feature weights and the counterfactual image feature weights, a distribution difference comparison is performed to obtain the counterfactual consistency loss; The optimization process based on the image reconstruction loss and the causal regularization loss to obtain the target image generation model includes: Based on the counterfactual consistency loss, the image reconstruction loss, and the causal regularization loss, the parameter gradient backpropagation is performed on the initial image generation model to obtain the target image generation model.

5. The conditional image generation method as described in claim 1, characterized in that, The step of performing causal association operations on the demand feature encoding vector and the image feature vector based on the causal discovery network to obtain a feature association causal graph includes: Node extraction is performed on the demand feature encoding vector to obtain semantic nodes; Node extraction is performed on the image feature vector to obtain visual nodes; Mutual information is calculated between the semantic node and the visual node to obtain conditional mutual information; Based on the conditional mutual information, a causal structure is constructed to obtain the feature association causal graph.

6. The conditional image generation method according to any one of claims 1-5, characterized in that, The process of allocating attention based on the causal perception adaptive attention network and the feature association causal graph to obtain image feature weights includes: Path identification is performed on the feature association causal graph to obtain pseudo-related confusion paths; The causal effect of the pseudo-related confusion path is eliminated to obtain the target causal effect value; The attention gating coefficients are obtained by mapping the target causal effect value. The attention gating coefficients are normalized to obtain the image feature weights.

7. The conditional image generation method according to any one of claims 1-5, characterized in that, The step of performing feature projection modulation on the image feature vector based on the image feature weights to obtain target image features includes: The image feature vector is subjected to feature projection processing to obtain a feature projection matrix, wherein the feature projection matrix contains multiple feature dimensions; Based on multiple feature dimensions, the image feature weights are dimension aligned to obtain aligned feature weights; Based on the alignment feature weights, feature modulation processing is performed on the feature projection matrix to obtain the target image features.

8. A conditional image generation apparatus, characterized in that, include: The model loading module is used to obtain the target image generation model, wherein the target image generation model includes a feature encoding network, an intermediate feature extraction network, a causal discovery network, a causal perception adaptive attention network, and an image decoding network; The feature encoding module is used to perform feature encoding on the pre-acquired image generation requirement data based on the feature encoding network to obtain the requirement feature encoding vector; The feature transformation module is used to perform image feature transformation on the required feature encoding vector based on the intermediate feature extraction network to obtain an image feature vector; The causal association module is used to perform causal association operations on the demand feature encoding vector and the image feature vector based on the causal discovery network to obtain a feature association causal graph. The quantitative evaluation module is used to perform attention allocation based on the causal perception adaptive attention network and the feature association causal graph to obtain image feature weights; The projection modulation module is used to perform feature projection modulation on the image feature vector based on the image feature weights to obtain the target image features; The decoding and synthesis module is used to perform image decoding and synthesis based on the image decoding network and the target image features to obtain the target image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the conditional image generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the conditional image generation method as described in any one of claims 1 to 7.

Images