Conditional image generation method and apparatus, electronic device, and storage medium

Abstract

The present application relates to the technical field of image processing, and discloses a conditional image generation method and device, electronic equipment and storage medium, comprising: obtaining a target image generation model, which comprises a conditional coding network and a self-attention network containing a target orthogonal projection matrix; performing affine transformation on a preset target image generation condition based on the conditional coding network to obtain a conditional affine feature; performing conditional normalization processing on a preset initial base image based on the conditional affine feature to obtain a conditional normalized feature map; performing self-attention feature extraction on the conditional normalized feature map based on the target orthogonal projection matrix to obtain a self-attention feature map; and performing upsampling synthesis processing on the self-attention feature map to obtain a target image. The present application can be applied to the financial technology scene and the health care scene, and improves the accuracy of conditional image generation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology and image processing technology, and can be applied to fintech and healthcare scenarios. In particular, it relates to a conditional image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Conditional image generation methods are used to generate images that meet semantic requirements based on given conditional information, thus satisfying the need for customized image content in different scenarios. Taking digital service scenarios in the financial sector as an example, conditional image generation methods can generate highly realistic financial promotional materials based on specific business scenario needs, providing financial institutions with efficient content production capabilities. Simultaneously, it reduces the construction cost and cycle of visual assets. In the field of healthcare, for example, conditional image generation methods can generate corresponding simulated medical images based on some physiological indicators or pathological features of patients, providing precise visual data supplementation for medical teaching, surgical planning, and rare pathology analysis.

[0003] Currently, in the process of conditional image generation, existing image generation models typically employ simple feature concatenation for conditional fusion. The fusion method between conditional features and basic image features is rather rigid, leading to issues such as interference between global semantics and local details in the generated images, as well as unstable feature distribution under conditional guidance. Therefore, improving the accuracy of conditional image generation has become an urgent technical problem to be solved. Summary of the Invention

[0004] This invention provides a conditional image generation method, apparatus, computer device, and medium based on artificial intelligence, aiming to solve the technical problems of unstable feature distribution and low image generation accuracy caused by rigid feature fusion and easy interference between global semantics and local details in the existing conditional image generation process, so as to improve the accuracy of conditional image generation.

[0005] Firstly, a conditional image generation method is provided, including: Obtain a target image generation model, wherein the target image generation model includes a conditional coding network and a self-attention network, and the self-attention network includes a target orthogonal projection matrix; Based on the conditional coding network, an affine transformation is performed on the preset target image generation conditions to obtain conditional affine features; Based on the conditional affine features, the preset initial base image is subjected to conditional normalization processing to obtain a conditional normalized feature map. Based on the target orthogonal projection matrix, self-attention feature extraction is performed on the conditionally normalized feature map to obtain a self-attention feature map; The self-attention feature map is upsampled and synthesized to obtain the target image.

[0006] Secondly, a conditional image generation apparatus is provided, comprising: The model loading module is used to obtain the target image generation model, wherein the target image generation model includes a conditional coding network and a self-attention network, and the self-attention network includes a target orthogonal projection matrix; The conditional transformation module is used to perform affine transformation on the preset target image generation conditions based on the conditional coding network to obtain conditional affine features; The normalization processing module is used to perform conditional normalization processing on the preset initial base image based on the conditional affine features to obtain a conditional normalized feature map. The feature extraction module is used to perform self-attention feature extraction on the conditionally normalized feature map based on the target orthogonal projection matrix to obtain a self-attention feature map; The image generation module is used to perform upsampling and synthesis processing on the self-attention feature map to obtain the target image.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described conditional image generation method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described conditional image generation method.

[0009] In the scheme implemented by the above-mentioned conditional image generation method, apparatus, computer equipment, and storage medium, the global and local features of the target image generation conditions can be extracted and cross-injected into the mapping through a conditional coding network. This effectively separates and accurately obtains the guiding information, thereby avoiding the problem of mutual interference between global semantics and local details in complex scenes. Secondly, the initial base image is subjected to zero-mean stripping and affine transformation processing based on the conditional affine features, realizing deep alignment and fusion of base image features and conditional constraints. This avoids the problem of stiff fusion caused by traditional simple stitching and ensures the stability of feature distribution under conditional guidance. Finally, the conditional normalized feature map is subjected to subspace projection and self-attention aggregation based on the target orthogonal projection matrix, eliminating the redundant coupling phenomenon in the high-dimensional feature extraction process. After upsampling and synthesis processing, the target image is obtained, thereby gradually realizing the goal of making the generated image highly consistent with the initial guiding conditions. In addition, the accuracy of conditional image generation is significantly improved. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a conditional image generation method according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of a conditional image generation method according to an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S10; Figure 4 yes Figure 2 A schematic diagram of a specific implementation method for step S20; Figure 5 yes Figure 2 A schematic diagram of a specific implementation method for step S30; Figure 6 This is a schematic diagram of a conditional image generation device in one embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The conditional image generation method provided in this embodiment of the invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can receive image generation requirement data or preset target image generation conditions from the client, and based on the target image generation conditions, performs layer-by-layer feature transformation and multi-dimensional semantic modulation using a pre-trained and optimized target image generation model to obtain the target image. The target image generation model includes a conditional coding network, a self-attention network, and an image decoding network. Based on the conditional coding network, global semantics and local structure are extracted from the target image generation conditions, and affine transformation calculations are performed to obtain conditional affine features containing scale and offset control parameters. Based on the conditional affine features, channel dimension statistics and zero-mean stripping are performed on the initial base image, followed by conditional normalization processing to obtain a conditionally normalized feature map that removes global background interference and incorporates conditional constraints. Based on the target orthogonal projection matrix in the self-attention network, orthogonal feature space projection, point distribution calculation, and nonlinear sharpening filtering are performed on the conditionally normalized feature map, followed by self-attention feature extraction to obtain a feature map that incorporates global long-range dependencies. The invention employs a self-attention feature map that is dependent on relationships and eliminates false statistical associations. Finally, based on an image decoding network, multi-layer upsampling convolution and detail texture filling are applied to the self-attention feature map, followed by upsampling synthesis to obtain a target image that matches the semantic constraints of the image generation requirements. In this invention, for the conditional image generation business entity, affine transformation and conditional normalization are performed based on pre-acquired target image generation conditions to obtain a highly normalized conditional normalized feature map. Then, orthogonal decoupling and self-attention feature extraction are performed on the features using the target orthogonal projection matrix, transforming it into a self-attention feature map that eliminates non-causal noise interference and possesses accurate global structural constraints. Finally, image decoding and synthesis are performed based on this self-attention feature map, ensuring the generated image highly matches the input requirements. This solves the problem of content deviation and structural ambiguity in out-of-distribution scenarios using traditional image generation methods, improving the accuracy and rationality of conditional image generation, and enhancing the quality and user experience of deep learning network image synthesis. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention is described in detail below through specific embodiments.

[0014] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the conditional image generation method provided in this embodiment of the invention includes the following steps: S10: Obtain the target image generation model, wherein the target image generation model includes a conditional coding network and a self-attention network, and the self-attention network contains the target orthogonal projection matrix; S20: Based on the conditional coding network, perform affine transformation on the preset target image generation conditions to obtain conditional affine features; S30: Based on conditional affine features, conditional normalization is performed on the preset initial base image to obtain a conditional normalized feature map. S40: Based on the target orthogonal projection matrix, perform self-attention feature extraction on the conditionally normalized feature map to obtain the self-attention feature map; S50: Upsample and synthesize the self-attention feature map to obtain the target image.

[0015] Steps S10 to S50 of this embodiment first extract global and local features of the target image generation conditions through a conditional coding network and cross-inject mapping, which can effectively separate and accurately obtain guidance information, thereby avoiding the problem of mutual interference between global semantics and local details in complex scenes. Secondly, based on the conditional affine features, the initial base image is stripped with zero mean and processed by affine transformation, realizing deep alignment and fusion of base image features and conditional constraints, thereby avoiding the stiff fusion problem caused by traditional simple splicing and ensuring the stability of feature distribution under conditional guidance. Finally, based on the target orthogonal projection matrix, the conditional normalized feature map is subjected to subspace projection and self-attention aggregation, eliminating the redundant coupling phenomenon in the high-dimensional feature extraction process, and the target image is obtained after upsampling and synthesis processing, thereby gradually realizing the purpose of making the generated image highly consistent with the initial guidance conditions. In addition, it significantly improves the accuracy of conditional image generation.

[0016] The conditional image generation method provided by this invention can be applied to the backend server of various image processing devices, such as image synthesis, data augmentation, and sample enhancement. This backend server can perform cross-modal conditional modulation of high-dimensional image features in real time. For example, in the fields of financial document sample generation or medical image sample augmentation, enterprises and institutions often need to synthesize high-fidelity document images or lesion images in batches. They often need to rely on a backend server to accurately control the local layout, structure, or tissue texture of the generated images in order to obtain high-quality sample data in real time and ensure that the semantic constraints of image generation meet the design requirements.

[0017] For example, a medical research institution plans to perform data augmentation on lesion images of a specific modality to expand the training samples for rare lesions. After submitting its image generation request to the backend server, the backend server first retrieves a pre-trained and optimized target image generation model from a preset storage area. This model includes a conditional encoding network and a self-attention network containing a target orthogonal projection matrix. Then, based on the conditional encoding network, the backend server extracts and cross-injects global semantics and local spatial structures from preset target image generation conditions such as lesion type and lesion degree, thereby performing affine transformations to successfully obtain conditional affine features carrying multidimensional constraints. Next, based on the scale affine matrix and offset affine matrix in these conditional affine features, the backend server performs channel dimension statistics and zero-mean stripping on a preset initial base image with basic anatomical structures, thereby performing conditional normalization. After processing, a conditionally normalized feature map that initially conforms to the lesion constraints was successfully obtained. Subsequently, the backend server, based on the target orthogonal projection matrix, performed orthogonal subspace projection, point distribution calculation, contrast stretching and sharpening, and weighted aggregation on the conditionally normalized feature map to perform self-attention feature extraction. This completely eliminated non-causal noise interference that only had statistical correlations, and successfully obtained a self-attention feature map that integrated the global long-distance anatomical shape contour and the specific lesion conditional response. Finally, the backend server performed upsampling synthesis processing on the self-attention feature map, including multi-layer upsampling convolution, detail texture filling, and pixel interval mapping. Ultimately, the target image that matched the semantic constraints of the input image generation requirement data was successfully output, which greatly improved the technical defects of traditional generation methods in the case of content deviation from the requirements in out-of-distribution scenarios.

[0018] In step S10 of some embodiments, the target image generation model refers to a deep learning network architecture that, after training and optimization, can generate matching visual features according to the input condition constraints. For example, in a financial digital service scenario, the target image generation model may be a deep network used to generate customized virtual digital human customer service images, while in a medical auxiliary analysis scenario, the target image generation model may be a deep network architecture used to generate simulated medical images of specific parts.

[0019] Conditional coding networks are network components deployed within a target image generation model to extract and transform semantic and structural information of guiding conditions. For example, in the scenario of generating financial promotional materials, a conditional coding network can be a network module that converts user profile descriptions into high-dimensional latent space features, while in the scenario of medical auxiliary diagnosis, a conditional coding network can be a component that converts patient pathological feature text into high-dimensional mapping features.

[0020] Self-attention networks are network components deployed within a target image generation model to capture long-distance contextual relationships between various locations within high-dimensional features. For example, in a financial document generation scenario, a self-attention network can be a network component used to calculate the feature dependency weights of different document layout regions, while in a medical image simulation scenario, a self-attention network can be a network component used to calculate the association weights of lesion features in different tissues.

[0021] The target orthogonal projection matrix refers to the feature subspace transformation matrix obtained after orthogonal regularization and Riemannian manifold gradient iterative optimization during the model training phase. For example, in the scenario of generating virtual human in finance, the target orthogonal projection matrix can be a parameter matrix used to map the facial features of the virtual human without redundancy to a subspace, while in the scenario of generating medical images, the target orthogonal projection matrix can be a parameter matrix used to map the features of medical image slices to an independent subspace.

[0022] In this embodiment, a target image generation model that has undergone multiple rounds of iterative optimization can be obtained from a preset repository. Then, the network architecture of the target image generation model is analyzed to confirm that the target image generation model includes a conditional encoding network for conditional parsing and a self-attention network for feature purification. Next, the parameters of the target image generation model are loaded in the system memory so that the self-attention network can directly call the internally stored target orthogonal projection matrix. Finally, the initial deployment preparation for the image generation calculation task is completed.

[0023] In detail, such as Figure 3 As shown, in step S10, the training process of the target conditional image generation model includes the following steps: S11: Obtain the sample image generation conditions, basic sample features, and the real target image corresponding to the sample image generation conditions; S12: Obtain the initial image generation model, wherein the initial image generation model includes an initial self-attention network, and the initial self-attention network contains an initial projection matrix; S13: Based on the initial image generation model, perform image prediction on the sample image generation conditions and basic features of the sample to obtain the predicted image; S14: Calculate the difference between the predicted image and the real target image to obtain the basic generation loss; S15: Calculate the orthogonality norm difference of the initial projection matrix to obtain the orthogonal regularization loss; S16: Perform joint loss calculation based on the basic generation loss and orthogonal regularization loss to obtain the global joint loss; S17: Based on the global joint loss, the initial projection matrix is ​​iteratively optimized to obtain the target orthogonal projection matrix; S18: Based on the global joint loss, the parameters of the initial image generation model are iteratively optimized to obtain the convergent network parameters; S19: Generate a pre-trained image generation model based on the target orthogonal projection matrix and convergent network parameters.

[0024] In step S11 of some embodiments, the sample image generation condition refers to the prior guidance information used to constrain the direction of model generation during the model training phase. For example, in the financial bill synthesis scenario, the sample image generation condition may be text annotation data containing the face value and date of the bill, while in the medical pathology simulation scenario, the sample image generation condition may be prior diagnostic attributes containing lesion type and grade.

[0025] The basic features of a sample refer to the low-dimensional structural representation or noise tensor that serves as the starting point for generation during the training process. For example, in the scenario of generating digital humans in finance, the basic features of a sample can be key point features representing the basic posture of a person, while in the scenario of medical image augmentation, the basic features of a sample can be a mask tensor representing the basic outline of an organ.

[0026] A real target image refers to the real reference visual content that matches the conditions for generating the sample image and serves as the target for model optimization. For example, in the scenario of generating financial promotional charts, the real target image can be a standard high-quality chart image designed by humans, while in the scenario of medical image enhancement, the real target image can be a clear medical scan image acquired by high-precision instruments.

[0027] In this embodiment, batch training data can be extracted from a pre-built training database to obtain sample image generation conditions containing text or label descriptions and sample basic features as underlying spatial support. At the same time, the system will simultaneously extract real target images that completely match the above sample image generation conditions as supervision signals for subsequent calculation of error loss.

[0028] In steps S12 and S13 of some embodiments, the initial image generation model refers to a deep learning network architecture that has not been sufficiently trained or whose parameters have not yet reached a convergent state.

[0029] A predicted image is an intermediate output with certain visual features that the model attempts to synthesize based on the input data during each forward propagation. For example, in the scenario of generating financial document samples, the predicted image may be a draft document with possible layout flaws in the initial generation, while in the scenario of generating medical imaging, the predicted image may be a simulated imaging image with possible blurred tissue edges.

[0030] In this embodiment, the conditional image generation system initializes and builds a network structure in memory to obtain an initial image generation model that has not been sufficiently optimized by parameters. This ensures that the initial image generation model is configured with an initial self-attention network containing an initial projection matrix. Then, the system inputs the extracted sample image generation conditions and sample basic features into the initial image generation model to perform forward propagation network calculations. Finally, the network outputs a predicted image with specific preliminary visual features.

[0031] In steps S14 to S16 of some embodiments, the basic generation loss refers to a value used to quantify the degree of difference between the predicted image output by the model and the real target image at the pixel level or feature level. For example, in the financial digital human generation scenario, the basic generation loss can be a value that measures the difference between the predicted digital human and the real material in facial color distribution, while in the medical ultrasound generation scenario, the basic generation loss can be a value that measures the difference between the predicted image and the real image in lesion texture structure.

[0032] Orthogonal regularization loss refers to the penalty value used to constrain the internal projection matrix of the self-attention network to maintain the mutual orthogonality of each column vector. For example, in the scenario of generating financial document images, orthogonal regularization loss can be a penalty value used to prevent the facial features of the document from being coupled with the background features, while in the scenario of generating medical imaging images, orthogonal regularization loss can be a penalty value used to avoid the confusion between vascular features and tissue features.

[0033] Global joint loss refers to the overall optimization index obtained by combining the basic generation loss that measures visual reconstruction error with the orthogonal regularization loss that measures parameter structure error. For example, in the scenario of financial promotional image synthesis, global joint loss can be the total gradient objective that guides the model to simultaneously consider color accuracy and decoupling of image and text features. In the scenario of medical diagnostic image generation, global joint loss can be the comprehensive optimization objective that guides the model to simultaneously consider lesion realism and the independence of various tissue features.

[0034] In this embodiment, the initial image generation model performs difference calculations on the predicted image and the real target image pixel by pixel or at the feature mapping level to quantify the reconstruction error between the two and obtain the basic generation loss. At the same time, the system performs orthogonal norm difference calculations on the initial projection matrix in the initial self-attention network to evaluate the degree of deviation from the ideal orthogonal norm, and then obtains the orthogonal regularization loss. Then, the system performs joint weighting and adaptive mapping calculation on the basic generation loss and the orthogonal regularization loss to obtain the global joint loss used to comprehensively guide the network weight update.

[0035] Step S16, which involves jointly calculating the loss based on the basic generation loss and the orthogonal regularization loss to obtain the global joint loss, includes: Data offset calculation is performed on the basic features of the sample to obtain the feature drift amount; Dynamic mapping is performed based on feature drift to obtain orthogonal penalty adaptive weights; Based on orthogonal penalty adaptive weights, scaling constraints are applied to the orthogonal regularization loss to obtain the constrained regularization loss; The global joint loss is obtained by weighting the basic generation loss and the constraint regularization loss.

[0036] In some embodiments, feature drift refers to a value obtained by quantifying the changes in the distribution of basic features of samples during the model training phase. It can reflect the degree of deviation of the underlying data. For example, in the scenario of financial bill generation, feature drift can be a statistical value that reflects the difference in the distribution of feature patterns of different batches of bills. In the scenario of medical image augmentation, feature drift can be a quantitative value that reflects the degree of deviation of lesion contrast features under different imaging devices.

[0037] Orthogonal penalty adaptive weight refers to a penalty coefficient dynamically calculated based on the feature distribution offset to adjust the strength of feature decoupling in the model. For example, in the scenario of generating financial promotional materials, orthogonal penalty adaptive weight can be an adjustment coefficient that dynamically adjusts the isolation strength between foreground and background features according to the degree of change in the background environment of the digital human. In the scenario of generating medical scan images, orthogonal penalty adaptive weight can be a dynamic parameter that automatically adjusts the decoupling strength between lesion and normal tissue features according to the changes in organ morphology of different patients.

[0038] Constraint regularization loss refers to the penalty value adjusted by dynamic weight scaling, which is used to provide the best matching feature orthogonal constraint under different data distribution states. For example, in the scenario of generating financial tables, constraint regularization loss can be a penalty value after weight scaling to strictly maintain the independence of table outline and text content features. In the scenario of generating medical pathology slides, constraint regularization loss can be an independent penalty value after adaptive adjustment to ensure that cell nucleus and cytoplasm features do not interfere with each other.

[0039] In this embodiment of the application, after the initial image generation model obtains the basic features of the sample, it compares and analyzes the current data distribution state with the baseline distribution features, performs data offset calculation through specific mathematical and statistical methods, so as to accurately quantify the change range of the input data in the multidimensional feature space, and finally obtains the feature drift amount that can intuitively reflect the difference in data distribution.

[0040] Next, the initial image generation model extracts the feature drift amount calculated above and inputs it into a pre-set dynamic mapping function for nonlinear transformation and evaluation calculation. It automatically assigns a reasonable penalty intensity according to the severity of the data offset. Finally, through this adaptive mapping process, it obtains orthogonal penalty adaptive weights specifically used to adjust the feature independence of the model.

[0041] Then, after obtaining the orthogonal penalty adaptive weights, the initial image generation model can directly apply them as scaling factors to the previously calculated orthogonal regularization loss, perform precise multiplication or scaling constraint calculations, so that the model can apply stronger orthogonal penalties to separate coupled features when the data distribution shift is large, and finally obtain a dynamically adjusted constraint regularization loss that is more in line with the current training state.

[0042] Finally, the initial image generation model extracts the basic generation loss used to constrain the visual reconstruction quality of the image, as well as the constraint regularization loss that is dynamically adjusted to constrain the feature decoupling strength, and performs weighted calculation on the two according to a preset ratio fusion strategy to integrate and unify the optimization objectives of different dimensions, thus obtaining the global joint loss function used to guide the iterative optimization of the entire network parameters and the backpropagation of gradients.

[0043] This application first calculates the feature drift by performing data offset calculation on the basic features of the samples, enabling the system to perceive the distribution changes and fluctuations of the underlying input data in real time. Secondly, based on the feature drift, dynamic mapping is performed to obtain orthogonal penalty adaptive weights, giving the system the ability to automatically adjust the structure optimization strategy according to the complexity of the data. Next, the orthogonal regularization loss is scaled based on the orthogonal penalty adaptive weights to obtain the constraint regularization loss, effectively avoiding the risk of underfitting or overfitting the model caused by using a fixed penalty strength in variable data scenarios. Finally, the basic generation loss and constraint regularization loss are weighted and integrated to obtain the global joint loss function, enabling the model to adaptively maintain the absolute independence of various high-dimensional features while ensuring the realism of image generation, thereby significantly improving the robustness and generation stability of the target image generation model in complex data environments with distribution offsets.

[0044] In steps S17 to S19 of some embodiments, after obtaining the global joint loss, the initial image generation model starts the backpropagation mechanism based on it, and uses a specific manifold optimization algorithm to iteratively optimize the initial projection matrix to obtain a target orthogonal projection matrix with strict orthogonal properties. At the same time, the system also uses the global joint loss to iteratively optimize the gradient descent parameters of the remaining network weights of the initial image generation model until the overall loss function value no longer decreases significantly and stable converged network parameters are obtained. Finally, the system saves and encapsulates the target orthogonal projection matrix and converged network parameters obtained above, and successfully generates a target image generation model with highly reliable inference capabilities.

[0045] Step S17, which involves iteratively optimizing the initial projection matrix based on the global joint loss to obtain the target orthogonal projection matrix, includes: Based on the global joint loss function, the manifold gradient of the initial projection matrix is ​​solved to obtain the Riemann gradient; Based on the Riemann gradient, the parameters of the initial projection matrix are optimized to obtain the target orthogonal projection matrix.

[0046] In one embodiment of this application, the Riemann gradient refers to the direction and step size vector obtained by solving in a specific geometric manifold space, which is used to guide parameter updates.

[0047] In this embodiment, after obtaining the global joint loss function covering the overall optimization objective, the initial image generation model, in order to ensure that the projection matrix inside the model can always maintain strict orthogonality during the parameter optimization process, abandons the traditional Euclidean gradient descent method, which is prone to destroying the matrix structure. Instead, it treats the initial projection matrix as a point on a specific orthogonal manifold space and performs mathematical differentiation and tangent space projection operations on the initial projection matrix in manifold space based on the global joint loss function. This accurately calculates the parameter update direction that can maintain the orthogonality of the matrix and obtains the Riemann gradient. For example, in the scenario of financial digital human generation, this method can be used to accurately calculate the orthogonal update vector used to separate the human posture and facial expression features. Subsequently, the initial image generation model will rely on the Riemann gradient calculated above to optimize and iteratively update the internal parameters of the initial projection matrix along the manifold geodesic or by using the recapitulation mapping technique. This ensures that the matrix structure always fits the orthogonal manifold surface while reducing the global loss, and finally successfully obtains the target orthogonal projection matrix with strong feature decoupling capabilities.

[0048] This application first obtains the Riemann gradient by performing derivative calculations on the initial projection matrix in manifold space based on the global joint loss function. This effectively overcomes the inherent defect that traditional Euclidean space parameter updates easily destroy the orthogonality of the matrix, ensuring that the direction of parameter optimization perfectly matches the geometric constraints of the orthogonal manifold space. Secondly, iterative optimization of the internal parameters of the initial projection matrix is ​​performed based on the precisely obtained Riemann gradient, ensuring that the parameters are always strictly constrained on a specific orthogonal surface during multiple rounds of optimization. Finally, the target orthogonal projection matrix that meets the high standard structural constraints is successfully obtained. This progressive process not only significantly improves the mathematical rigor and convergence stability of the underlying parameter optimization of the deep learning network, but also fundamentally ensures the absolute independence and decoupling between different dimensions of information when the model performs feature extraction, thereby significantly improving the precise control capability and synthesis quality of the conditional image generation process.

[0049] Steps S11 to S19 of this embodiment first obtain sample training data and use the initial image generation model to perform forward prediction to generate predicted images. This provides basic comparison materials and operating environment for the model's parameter optimization. Secondly, the difference between the predicted image and the real target image and the initial projection matrix is ​​calculated to obtain the basic generation loss and the orthogonal regularization loss. After joint calculation, the global joint loss is obtained, which comprehensively measures the model's performance in visual reconstruction accuracy and feature space independence. This effectively avoids the feature coupling and overfitting risks caused by a single optimization target. Finally, based on the global joint loss, the initial projection matrix and the entire network parameters are iteratively optimized to obtain the converged network parameters and the target orthogonal projection matrix. This enables the encapsulated image generation model to have strong feature decoupling ability and accurate content fitting ability, thereby achieving the ultimate goal of providing a highly stable and accurate target image generation model for the conditional image generation system.

[0050] In step S20 of some embodiments, the target image generation condition refers to prior constraint information used to guide the model generation process. For example, in the scenario of financial asset digitization, the target image generation condition may be constraint text containing the color of the clothing and facial contours of the financial digital human, while in the scenario of medical teaching images, the target image generation condition may be a labeled mask containing specific rare lesion categories and size requirements.

[0051] Conditional affine features refer to the modulation parameters output by the conditional coding network, which include scale factors and offset factors. For example, in the financial scenario, conditional affine features can be transformation parameters used to control the background color tone and layout ratio of financial posters, while in the medical scenario, conditional affine features can be transformation parameters used to control the grayscale distribution and lesion contrast of the inferred image.

[0052] In this embodiment, after receiving the preset target image generation conditions, the conditional coding network can first perform global semantic extraction on the target image generation conditions to obtain a global semantic vector, then perform local structure extraction on the target image generation conditions to obtain a local spatial structure tensor, then perform cross-injection mapping calculation on the global semantic vector and the local spatial structure tensor to output a feature space affine matrix, and finally perform feature integration on the feature space affine matrix to obtain a conditional affine feature containing a scale affine matrix and an offset affine matrix.

[0053] In detail, such as Figure 4 As shown, in step S20, that is, based on the conditional coding network, an affine transformation is performed on the conditions for generating the target image to obtain conditional affine features, including the following steps: S21: Based on the conditional coding network, global semantic extraction is performed on the generation conditions of the target image to obtain a global semantic vector; S22: Based on the conditional coding network, local structure extraction is performed on the target image generation conditions to obtain the local spatial structure tensor; S23: Perform cross-injection mapping on the global semantic vector and the local spatial structure tensor to obtain the feature space affine matrix; S24: Perform feature integration on the affine matrix of the feature space to obtain conditional affine features.

[0054] In step S21 of some embodiments, the global semantic vector refers to high-dimensional fixed-length vector data obtained after extracting macroscopic concepts, overall categories, or overall attributes of the target image generation conditions through a conditional coding network. It carries the global core semantics of the generation conditions. For example, in a financial document generation scenario, the global semantic vector can be a latent space vector carrying the overall document business category such as invoices, drafts, or deposit slips. In a medical image generation scenario, the global semantic vector can be a latent space vector representing specific imaging modalities such as MRI and computed tomography scans, as well as the overall disease classification.

[0055] In this embodiment, after receiving the preset target image generation conditions, the conditional coding network first formats and normalizes the category labels, text descriptions, or attribute maps and maps them to a preset latent space. Then, it performs macroscopic semantic interaction and encoding on the target image generation conditions through a multilayer perceptron, convolutional coding layer, or self-attention mechanism module to extract high-dimensional abstract semantics that can represent the overall category, overall style, core business attributes, and macroscopic constraints of the target image. Next, it performs dimensional normalization on the high-dimensional abstract semantics through continuous transformations of global pooling, linear mapping, and nonlinear activation functions. Finally, it outputs a global semantic vector with fixed dimensions.

[0056] In step S22 of some embodiments, the local spatial structure tensor refers to multidimensional feature data obtained by extracting spatial location, region boundaries, layout morphology, or fine-grained geometric constraints in the target image generation conditions through a conditional coding network. It can reflect the positional correspondence of various local regions in the image. For example, in a financial document generation scenario, the local spatial structure tensor can be tensor data representing the spatial geometric arrangement of table frames, amount input fields, date filling areas, and stamping areas in the document. In a medical image generation scenario, the local spatial structure tensor can be a spatial feature tensor representing the coordinates of the lesion center, the boundary contour of the lesion area, and the position of the surrounding tissue anatomical structures.

[0057] In this embodiment, the conditional coding network can further parse the spatial layout, region boundaries, and local morphological constraints contained in the target image generation conditions. Then, the relative positional relationships of local regions in the image generation conditions are modeled layer by layer through a feature extraction network containing multiple convolutional layers. Next, the structured fields, region masks, or layout key points in the conditional data are transformed into multi-channel feature maps carrying spatial position responses, so that each channel or each spatial coordinate can accurately correspond to the spatial physical constraints of specific structural parts in the target image. Subsequently, the multi-channel feature maps are scale-aligned and spatial resolution-normalized. Finally, a local spatial structure tensor that can finely represent the local spatial geometric distribution is formed.

[0058] In step S23 of some embodiments, the feature space affine matrix refers to a parameter matrix generated by cross-fusion and interactive mapping of global semantic vectors and local spatial structure tensors, used to perform scaling and offset modulation on basic features in the feature latent space. For example, in a financial document generation scenario, the feature space affine matrix can be a parameter matrix used to adaptively scale and shift the intensity and corresponding position of different text blocks, layout frames, and other visual features in the document image. In a medical image generation scenario, the feature space affine matrix can be a parameter matrix used to adaptively control the grayscale distribution and texture contrast of specific tissues and lesion areas in the image feature map.

[0059] In this embodiment, the conditional coding network first converts the obtained global semantic vector into a spatial dimension form that can be aligned with the local spatial structure tensor through linear mapping or dimensional copying. Then, the converted global semantic features are injected into each spatial location or feature channel of the local spatial structure tensor, so that the local structural features carry the overall category and global attributes of the target image while preserving geometric constraints. Next, the injected global semantic expression is dynamically modulated regionally according to the response intensity of different spatial locations in the local spatial structure tensor. Subsequently, nonlinear feature transformation is performed through a convolutional mapping layer or a spatial attention mapping module. Finally, a feature space affine matrix is ​​generated to guide subsequent image feature transformation.

[0060] In step S24 of some embodiments, the conditional coding network performs channel fusion, region fusion, and hierarchical fusion on the feature space affine matrix generated after cross-injection mapping to eliminate redundant affine parameters at different semantic levels and spatial dimensions. Then, through continuous mapping of convolutional integration layers, normalization layers, and nonlinear activation functions, the affine parameters are stabilized within a reasonable numerical range, thereby forming a conditional modulation feature with a unified structure and stable expression. Finally, the integrated feature is divided into a scale affine matrix and an offset affine matrix, which are used to control the feature amplitude change and the basic position offset, respectively, thereby outputting the final conditional affine feature.

[0061] Steps S21 to S24 as shown in the embodiments of this application firstly extract the global semantic vector from the target image generation conditions using a conditional coding network. This allows for the macroscopic locking of the overall category and core attributes of the generated image, ensuring that the image generation has a clear semantic direction. Then, the local structure is extracted from the target image generation conditions using a conditional coding network to obtain a local spatial structure tensor. This captures precise spatial layout, region boundaries, and fine-grained morphological constraints, significantly enhancing the controllability and accuracy of the target image at the local structure level. Finally, a feature space affine is obtained by cross-injection mapping of the global semantic vector and the local spatial structure tensor. The matrix enables deep interaction and organic fusion of global semantic constraints and local spatial constraints within the feature space, effectively avoiding local structural distortion caused by relying solely on global conditions, or overall semantic deviation caused by relying solely on local structures. Finally, by integrating the features of the affine matrix in the feature space to obtain conditional affine features, the target image generation conditions can be transformed into highly consistent scale and offset control parameters for subsequent fine-tuning of image features. This greatly improves the sufficiency of conditional information utilization and the accuracy of the expression of local and global image structures, thus effectively ensuring a high degree of consistency between the final output target image and the target image generation conditions.

[0062] In step S30 of some embodiments, the initial base image refers to a low-resolution image or noise tensor that serves as the starting point for generation. For example, in a financial scenario, the initial base image may be a randomly sampled Gaussian noise map, while in a medical scenario, the initial base image may be a sketch of the patient's basic anatomical structure.

[0063] Conditional normalized feature maps refer to the depth-aligned feature tensors obtained after statistical feature stripping and incorporating conditional affine features from an initial base image. For example, in a financial scenario, a conditional normalized feature map can be an intermediate feature map of a customer service image that incorporates features of the target customer profile, while in a medical scenario, a conditional normalized feature map can be an intermediate feature map of organ texture that incorporates features of lesion categories.

[0064] In this embodiment, after obtaining the conditional affine features, the channel dimension data statistics can be performed on the preset initial base image to obtain the channel feature mean and channel feature variance. Then, based on the channel feature mean and channel feature variance, a zero-mean stripping operation is performed on the initial base image to obtain the basic residual features. Next, the scale affine matrix and the offset affine matrix are subjected to affine transformation calculation with the basic residual features. Finally, the depth alignment and fusion of the basic features and the conditional constraints are completed to obtain the conditional normalized feature map.

[0065] In detail, such as Figure 5 As shown, in step S30, the conditional affine features include the scale affine matrix and the offset affine matrix. Based on this premise, the initial base image is conditionally normalized based on the conditional affine features to obtain the conditionally normalized feature map, including the following steps: S31: Perform channel dimension data statistics on the initial base image to obtain the channel feature mean and channel feature variance; S32: Based on the channel feature mean and channel feature variance, the initial base image is stripped to zero mean to obtain the base residual features; S33: Based on the scale affine matrix and the offset affine matrix, perform affine transformation on the basic residual features to obtain the conditionally normalized feature map.

[0066] In step S31 of some embodiments, the scale affine matrix refers to a matrix-type control parameter included in the conditional affine features, used to dynamically scale or adjust the amplitude or variance of image features within the feature space. For example, in a financial document generation scenario, the scale affine matrix can be a control matrix used to adaptively amplify or reduce the contrast or response intensity of local text block features or wireframe features in the document image. In a medical image generation scenario, the scale affine matrix can be a control matrix used to adaptively scale the texture contrast and grayscale variation amplitude of specific tissues and lesion areas in the image feature map.

[0067] An offset affine matrix is ​​a matrix-type control parameter included in conditional affine features, used to dynamically shift and correct the baseline or mean value of image features within the feature space. For example, in financial document generation scenarios, an offset affine matrix can be a correction matrix used to adjust the background color intensity and field physical position offset of local areas of the document. In medical image generation scenarios, an offset affine matrix can be a correction matrix used to adjust the base gray center value and background noise level of specific organs and lesion areas.

[0068] The channel feature mean refers to the average value of each feature channel calculated after sampling and statistically analyzing the data in the channel dimension of the initial base image. It can reflect the global background intensity of each channel feature.

[0069] Channel feature variance refers to the variance or standard deviation of each feature channel calculated after sampling and statistically analyzing the data in the channel dimension of the initial base image. It can reflect the data dispersion and contrast of each channel feature.

[0070] In this embodiment, after the target image generation model obtains the initial base image and the conditional affine features output by the conditional coding network, it first traverses and samples all pixel feature points contained in each feature channel along the spatial dimension of the initial base image. Then, by accumulating and summing and dividing equally, it calculates the channel feature mean for each feature channel in real time. Next, based on the channel feature mean, it calculates the squared deviation between each pixel feature point and the channel feature mean. Subsequently, by averaging the squared deviation, it obtains the channel feature variance that can accurately reflect the data dispersion and information contrast within each feature channel.

[0071] In step S32 of some embodiments, the basic residual features refer to the standard intrinsic features retained after the initial basic image has been stripped of global background interference by subtracting the channel feature mean and dividing by the channel feature variance to achieve zero mean stripping. These features can represent pure relative structural information. For example, in a financial document generation scenario, the basic residual features may be feature data that has been stripped of global background color interference and retains only the pure text skeleton and wireframe topology. In a medical image generation scenario, the basic residual features may be feature data that has been stripped of global grayscale bias and retains only the pure tissue boundaries and anatomical contours.

[0072] In this embodiment, after statistically obtaining the channel feature mean and channel feature variance, the target image generation model first uses the original feature value of the initial base image to subtract the channel feature mean corresponding to its feature channel point by point, so as to completely erase the global background component and the reference brightness bias from the initial base image. Then, it uses the calculated deviation result to divide point by the corresponding channel feature variance, or divides it by the combined square root value containing the channel feature variance and the small stability factor, so as to achieve normalization and alignment of the feature amplitude scales of different channels, thereby completing the zero mean stripping process. Finally, it outputs the basic residual features that have removed the style differences between channels and only carry pure geometric shape and topological structure information.

[0073] In step S33 of some embodiments, while outputting the basic residual features, the target image generation model can extract the scale affine matrix and offset affine matrix pre-deconstructed from the conditional affine features and align them with the channel dimension and spatial dimension of the basic residual features. Then, the basic residual features and the scale affine matrix are multiplied element-wise so that the scale control parameters in the target image generation conditions can be precisely applied to each spatial position of the feature map, thereby completing the adaptive amplification or reduction modulation of contrast and amplitude. Next, the product-modulated features and the offset affine matrix are added element-wise to seamlessly inject the position translation and mean bias information in the target image generation conditions into the feature space. Finally, through this fine affine transformation processing, a conditionally normalized feature map that conforms to the preset image generation condition constraints is output.

[0074] Steps S31 to S33, as illustrated in this embodiment, firstly, channel dimension data statistics are performed on the initial base image to obtain the channel feature mean and channel feature variance. This accurately captures the global background distribution and contrast state of the current image features. Based on this, the initial base image is stripped using zero-mean stripping based on the channel feature mean and channel feature variance to obtain basic residual features. This completely eliminates the inherent channel style bias and background noise interference of the initial base image, making the feature space regularized into a highly standardized pure structural expression. Furthermore, after extracting the scale affine matrix and offset affine matrix from the conditional affine features, the abstract target image can be generated into a striped structure. The components are transformed into feature scaling and translation parameters with explicit physical guidance. Finally, based on the scale affine matrix and the offset affine matrix, the basic residual features are subjected to affine transformation to obtain the conditionally normalized feature map. This enables the standardized basic residual features to achieve adaptive reconstruction under the precise drive of the conditional parameters. This avoids the problem of conditional failure caused by the residual style of the original features, and also avoids the technical defect that the conditional information cannot deeply penetrate the interior of the features. In addition, it greatly improves the consistency between the conditionally normalized feature map and the target image generation conditions, thus laying a solid feature foundation for subsequent high-quality and high-accuracy self-attention feature extraction and target image synthesis.

[0075] In step S40 of some embodiments, the self-attention feature map refers to a high-order feature tensor that carries global contextual information after point integration and weighted aggregation processing. For example, in a financial scenario, the self-attention feature map can be a feature tensor that enhances the coherence of financial anti-counterfeiting texture details, while in a medical scenario, the self-attention feature map can be a feature tensor that enhances the continuity of pathological tissue boundaries.

[0076] In this embodiment, the self-attention network can first perform lossless projection mapping of the feature subspace on the conditionally normalized feature map based on the target orthogonal projection matrix to obtain the query tensor, key tensor, and value tensor respectively. Then, it performs point distribution calculation on the associated dimension of the query tensor and key tensor to generate a point distribution matrix. Next, it performs sharpening processing on the point distribution matrix to eliminate redundant attention weights to obtain a sharpening distribution matrix. Finally, it performs dimension-wise weighted aggregation on the sharpening distribution matrix and the value tensor to obtain the self-attention feature map.

[0077] Specifically, based on the target orthogonal projection matrix, self-attention feature extraction is performed on the conditionally normalized feature map to obtain a self-attention feature map, including: Based on the target orthogonal projection matrix, feature subspace projection is performed on the conditionally normalized feature map to obtain the query tensor, key tensor and value tensor. The point integral distribution matrix is ​​obtained by performing point integral distribution calculation on the query tensor and the key tensor; The point integral distribution matrix is ​​sharpened to obtain the sharpened distribution matrix; The sharpening distribution matrix and value tensor are weighted and aggregated to obtain the self-attention feature map.

[0078] In some embodiments, the query tensor refers to the feature matrix obtained by projecting the conditionally normalized feature map into its feature subspace using the target orthogonal projection matrix in a self-attention mechanism. This feature matrix is ​​used as an active query term to match other locations in the feature map.

[0079] The key tensor refers to the feature matrix obtained by projecting the conditionally normalized feature map into its feature subspace through the target orthogonal projection matrix in the self-attention mechanism. It is used as the query term to respond to the matching request of the query tensor.

[0080] The value tensor refers to the feature matrix obtained by projecting the conditionally normalized feature map into the feature subspace through the target orthogonal projection matrix in the self-attention mechanism. It carries the original visual or structural information to be weighted and aggregated.

[0081] The dot product matrix is ​​a correlation matrix obtained by performing a matrix dot product operation between the query tensor and the key tensor. It is used to initially characterize the strength of statistical correlation between any two spatial locations in the feature map.

[0082] The sharpened distribution matrix refers to the probability distribution matrix obtained after the point integral distribution matrix has been normalized, scaled, or mathematically sharpened. It can significantly amplify the core correlation weights and suppress long-tail noise interference.

[0083] In this embodiment, after obtaining the conditionally normalized feature map carrying conditional constraints, the target image generation model can first retrieve the pre-trained converged target orthogonal projection matrix and decompose it into three mutually orthogonal independent projection sub-matrices. Then, the three projection sub-matrices are used to perform three independent linear projection calculations on the conditionally normalized feature map, thereby mapping the original features to three mutually orthogonal and non-overlapping feature subspaces. This minimizes information redundancy within the features. Finally, through this orthogonal subspace projection, the corresponding outputs are a query tensor representing the matching initiator, a key tensor representing the matched index item, and a value tensor carrying the core visual attributes.

[0084] After obtaining the query tensor and key tensor, the target image generation model can first transpose the spatial dimension of the key tensor so that the matrix dimension of the key tensor can be aligned with the matrix dimension of the query tensor by multiplying. Then, the query tensor and the transposed key tensor are subjected to matrix dot product operation. By calculating the vector similarity between any two spatial physical locations in the feature map, the long-distance semantic associations and layout dependencies within the image are dynamically captured. Next, the result of the dot product calculation is mapped to the relation matrix formed by multiplying the spatial dimensions. Finally, a dot product distribution matrix that can comprehensively represent the initial association strength between various regions within the feature map is successfully formed.

[0085] After generating the initial point distribution matrix, the target image generation model, in order to prevent the probability distribution from flattening due to long-distance computation or being disturbed by non-core statistical correlations, first divides each element in the point distribution matrix by the square root scaling factor of the corresponding feature dimension. This stabilizes the gradient backpropagation and performs preliminary contrast stretching. Then, the scaled point distribution matrix is ​​input into the normalized activation function to perform exponential and normalization operations, thereby greatly amplifying the core correlation weights with high response in a non-linear manner, while significantly suppressing false alarm connections with low response. Finally, through this refined sharpening and filtering mechanism, a sharpened distribution matrix with highly focused distribution and clear connection relationships is successfully output.

[0086] In this embodiment, after the target image generation model outputs the sharpening distribution matrix, it can first use the sharpening distribution matrix as a dynamic weight coefficient matrix and adjust the value tensor to an aggregation dimension that matches the weight coefficient matrix. Then, it performs matrix multiplication on the sharpening distribution matrix and the value tensor, so that each precise association weight in the sharpening distribution matrix can act on each spatial feature vector in the value tensor, thereby completing the dynamic aggregation of information based on global context association. Next, the weighted aggregated feature map is subjected to residual connection or linear mapping transformation to further enhance the depth of feature expression. Finally, a self-attention feature map that integrates precise global structural constraints and conditional responses is successfully output.

[0087] This application embodiment first uses the target orthogonal projection matrix to project the conditionally normalized feature map into a feature subspace to obtain the query tensor, key tensor, and value tensor. Relying on the unique geometric decoupling properties of orthogonal matrices, image features can be divided into independent orthogonal subspaces that do not interfere with each other. This fundamentally avoids information redundancy and basis collapse problems in self-attention computation. Based on this, a point distribution matrix is ​​obtained by performing point distribution calculation on the query tensor and key tensor. This matrix can comprehensively capture the implicit spatial layout associations and semantic constraints between arbitrary distances and regions within the image. Furthermore, a sharpening distribution matrix is ​​obtained by sharpening the point distribution matrix, which can achieve nonlinear... The form of sex mapping forcibly amplifies the difference between core associations and spurious associations, thereby completely eliminating non-causal noise that only has statistical associations, ensuring the high focus of attention weight allocation. Finally, by weighted aggregation of the sharpening distribution matrix and value tensor, a self-attention feature map is obtained, which can accurately assign the selected strong causal long-distance structural information to the original visual features, so that each pixel in the feature map has a contextual semantic that conforms to global condition constraints and physical logic. This greatly improves the accuracy of self-attention feature extraction and the rationality of feature aggregation, and also provides a decisive feature guarantee for synthesizing target images with content that does not deviate from the requirements and with highly accurate structural details.

[0088] In step S50 of some embodiments, the target image refers to the final visual image result that meets preset constraints and has been upsampled and decoded. For example, in a financial scenario, the target image may be a highly realistic financial digital human poster image that fully meets business requirements, while in a medical scenario, the target image may be an image of a precise lesion inference that fully meets the requirements of pathological diagnosis.

[0089] In this embodiment, the image generation system can first obtain a self-attention feature map carrying high-dimensional context information, then gradually expand the spatial resolution of the self-attention feature map through a series of multi-layer upsampling convolutional networks, then gradually fill each resolution level with pixel-level detail texture and edge structure information that meet the target image generation conditions, then use an activation function to perform interval mapping on the feature values ​​of the high-order tensor, and finally complete the decoding conversion from abstract features to a visual grid, outputting a target image that fully meets the preset generation requirements.

[0090] As can be seen, in the above scheme, the global and local features of the target image generation conditions can be extracted and cross-injected into the mapping through the conditional coding network, which can effectively separate and accurately obtain the guiding information, thereby avoiding the problem of mutual interference between global semantics and local details in complex scenes. Secondly, the initial base image is stripped by zero-mean stripping and affine transformation based on the conditional affine features, which realizes the deep alignment and fusion of base image features and conditional constraints, thereby avoiding the stiff fusion problem caused by traditional simple stitching and ensuring the stability of feature distribution under conditional guidance. Finally, the conditional normalized feature map is subspace projected and self-attention aggregated based on the target orthogonal projection matrix, which eliminates the redundant coupling phenomenon in the high-dimensional feature extraction process, and the target image is obtained after upsampling and synthesis processing, thereby gradually realizing the goal of making the generated image highly consistent with the initial guiding conditions. In addition, it also significantly improves the accuracy of conditional image generation.

[0091] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0092] In one embodiment, a conditional image generation apparatus is provided, which corresponds one-to-one with the conditional image generation methods described in the above embodiments. For example... Figure 6 As shown, the conditional image generation device includes a model loading module, a conditional transformation module, a normalization processing module, a feature extraction module, and an image generation module. Detailed descriptions of each functional module are as follows: The model loading module is used to obtain the target image generation model, which includes a conditional coding network and a self-attention network. The self-attention network contains the target orthogonal projection matrix. The conditional transformation module is used to perform affine transformation on the preset target image generation conditions based on the conditional coding network to obtain conditional affine features. The normalization processing module is used to perform conditional normalization processing on the preset initial base image based on conditional affine features to obtain a conditional normalized feature map. The feature extraction module is used to perform self-attention feature extraction on the conditionally normalized feature map based on the target orthogonal projection matrix, so as to obtain the self-attention feature map. The image generation module is used to upsample and synthesize the self-attention feature map to obtain the target image.

[0093] In one embodiment, the model loading module is specifically used for: Obtain the sample image generation conditions, basic sample features, and the real target image corresponding to the sample image generation conditions; Obtain an initial image generation model, wherein the initial image generation model includes an initial self-attention network, and the initial self-attention network contains an initial projection matrix; Based on the initial image generation model, image prediction is performed on the sample image generation conditions and basic features of the sample to obtain the predicted image; The difference between the predicted image and the real target image is calculated to obtain the basic generation loss; The orthogonal norm difference is calculated on the initial projection matrix to obtain the orthogonal regularization loss; The global joint loss is obtained by jointly calculating the basic generation loss and the orthogonal regularization loss. Based on the global joint loss, the initial projection matrix is ​​iteratively optimized to obtain the target orthogonal projection matrix; Based on the global joint loss, the parameters of the initial image generation model are iteratively optimized to obtain the convergent network parameters. A pre-trained image generation model is generated based on the target orthogonal projection matrix and the convergent network parameters.

[0094] In one embodiment, the model loading module is further configured to: Data offset calculation is performed on the basic features of the sample to obtain the feature drift amount; Dynamic mapping is performed based on feature drift to obtain orthogonal penalty adaptive weights; Based on orthogonal penalty adaptive weights, scaling constraints are applied to the orthogonal regularization loss to obtain the constrained regularization loss; The global joint loss is obtained by weighting the basic generation loss and the constraint regularization loss.

[0095] In one embodiment, the model loading module is further configured to: Based on the global joint loss function, the manifold gradient of the initial projection matrix is ​​solved to obtain the Riemann gradient; Based on the Riemann gradient, the parameters of the initial projection matrix are optimized to obtain the target orthogonal projection matrix.

[0096] In one embodiment, the condition transformation module is specifically used for: Based on a conditional coding network, global semantic extraction is performed on the generation conditions of the target image to obtain a global semantic vector. Based on the conditional coding network, local structure extraction is performed on the generation conditions of the target image to obtain the local spatial structure tensor. Cross-injection mapping is performed on the global semantic vector and the local spatial structure tensor to obtain the feature space affine matrix; By integrating the features of the affine matrix in the feature space, conditional affine features are obtained.

[0097] In one embodiment, the normalization processing module is specifically used for: Perform channel dimension data statistics on the initial base image to obtain the channel feature mean and channel feature variance; Based on the channel feature mean and channel feature variance, the initial base image is stripped with zero mean to obtain the base residual features; Based on the scale affine matrix and the offset affine matrix, an affine transformation is performed on the basic residual features to obtain a conditionally normalized feature map.

[0098] In one embodiment, the feature extraction module is specifically used for: Based on the target orthogonal projection matrix, feature subspace projection is performed on the conditionally normalized feature map to obtain the query tensor, key tensor and value tensor. The point integral distribution matrix is ​​obtained by performing point integral distribution calculation on the query tensor and the key tensor; The point integral distribution matrix is ​​sharpened to obtain the sharpened distribution matrix; The sharpening distribution matrix and value tensor are weighted and aggregated to obtain the self-attention feature map.

[0099] This invention provides a conditional image generation device. First, a conditional coding network is used to extract global and local features of the target image generation conditions and cross-inject and map them, which can effectively separate and accurately obtain guiding information, thereby avoiding the problem of mutual interference between global semantics and local details in complex scenes. Second, based on the conditional affine features, the initial base image is subjected to zero-mean stripping and affine transformation processing, realizing deep alignment and fusion of base image features and conditional constraints, thereby avoiding the stiff fusion problem caused by traditional simple stitching and ensuring the stability of feature distribution under conditional guidance. Finally, based on the target orthogonal projection matrix, the conditional normalized feature map is subjected to subspace projection and self-attention aggregation, eliminating the redundant coupling phenomenon in the high-dimensional feature extraction process, and the target image is obtained after upsampling and synthesis processing, thereby gradually realizing the goal of making the generated image highly consistent with the initial guiding conditions. In addition, it significantly improves the accuracy of conditional image generation.

[0100] Specific limitations regarding the conditional image generation device can be found in the limitations of the conditional image generation method described above, and will not be repeated here. Each module in the aforementioned conditional image generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0101] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a conditional image generation method on the server side.

[0102] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a conditional image generation method on the client side. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the target image generation model, which includes a conditional coding network and a self-attention network, wherein the self-attention network contains the target orthogonal projection matrix; Conditional affine features are obtained by performing affine transformation on preset target image generation conditions based on a conditional coding network. Based on conditional affine features, the preset initial base image is conditionally normalized to obtain a conditionally normalized feature map. Based on the target orthogonal projection matrix, self-attention feature extraction is performed on the conditionally normalized feature map to obtain the self-attention feature map. The self-attention feature map is upsampled and synthesized to obtain the target image.

[0103] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the target image generation model, which includes a conditional coding network and a self-attention network, wherein the self-attention network contains the target orthogonal projection matrix; Conditional affine features are obtained by performing affine transformation on preset target image generation conditions based on a conditional coding network. Based on conditional affine features, the preset initial base image is conditionally normalized to obtain a conditionally normalized feature map. Based on the target orthogonal projection matrix, self-attention feature extraction is performed on the conditionally normalized feature map to obtain the self-attention feature map. The self-attention feature map is upsampled and synthesized to obtain the target image.

[0104] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0105] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0107] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0108] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A conditional image generation method, characterized in that, include: Obtain a target image generation model, wherein the target image generation model includes a conditional coding network and a self-attention network, and the self-attention network includes a target orthogonal projection matrix; Based on the conditional coding network, an affine transformation is performed on the preset target image generation conditions to obtain conditional affine features; Based on the conditional affine features, the preset initial base image is subjected to conditional normalization processing to obtain a conditional normalized feature map. Based on the target orthogonal projection matrix, self-attention feature extraction is performed on the conditionally normalized feature map to obtain a self-attention feature map; The self-attention feature map is upsampled and synthesized to obtain the target image.

2. The conditional image generation method as described in claim 1, characterized in that, The training steps of the target image generation model include: Obtain the sample image generation conditions, the basic features of the sample, and the real target image corresponding to the sample image generation conditions; Obtain an initial image generation model, wherein the initial image generation model includes an initial self-attention network, and the initial self-attention network contains an initial projection matrix; Based on the initial image generation model, image prediction is performed on the sample image generation conditions and the basic features of the sample to obtain the predicted image; The difference between the predicted image and the real target image is calculated to obtain the basic generation loss; The orthogonal norm difference is calculated on the initial projection matrix to obtain the orthogonal regularization loss; Based on the basic generation loss and the orthogonal regularization loss, a joint loss calculation is performed to obtain the global joint loss; Based on the global joint loss, the initial projection matrix is ​​iteratively optimized to obtain the target orthogonal projection matrix; Based on the global joint loss, the parameters of the initial image generation model are iteratively optimized to obtain the convergent network parameters; The pre-trained image generation model is generated based on the target orthogonal projection matrix and the convergent network parameters.

3. The conditional image generation method as described in claim 2, characterized in that, The global joint loss is obtained by jointly calculating the loss based on the basic generation loss and the orthogonal regularization loss, including: The data offset is calculated on the basic features of the sample to obtain the feature drift amount; Dynamic mapping is performed based on the feature drift to obtain orthogonal penalty adaptive weights; Based on the orthogonal penalty adaptive weights, the orthogonal regularization loss is calculated by scaling constraints to obtain the constraint regularization loss; The global joint loss is obtained by weighting the basic generation loss and the constraint regularization loss.

4. The conditional image generation method as described in claim 2, characterized in that, The step of iteratively optimizing the initial projection matrix based on the global joint loss to obtain the target orthogonal projection matrix includes: Based on the global joint loss function, the manifold gradient of the initial projection matrix is ​​solved to obtain the Riemann gradient; Based on the Riemann gradient, the parameters of the initial projection matrix are optimized to obtain the target orthogonal projection matrix.

5. The conditional image generation method as described in claim 1, characterized in that, The step of performing an affine transformation on the target image generation conditions based on the conditional coding network to obtain conditional affine features includes: Based on the conditional coding network, global semantic extraction is performed on the target image generation conditions to obtain a global semantic vector; Based on the conditional coding network, local structure extraction is performed on the target image generation conditions to obtain the local spatial structure tensor; The global semantic vector and the local spatial structure tensor are cross-injected and mapped to obtain the feature space affine matrix. The conditional affine features are obtained by integrating the affine matrix in the feature space.

6. The conditional image generation method according to any one of claims 1-5, characterized in that, The conditional affine feature includes a scale affine matrix and an offset affine matrix; The conditional normalization process, based on the conditional affine features, is performed on the initial base image to obtain a conditional normalized feature map, including: The initial base image is statistically analyzed using channel dimension data to obtain the mean and variance of channel features; Based on the mean and variance of the channel features, the initial base image is stripped to zero mean to obtain the base residual features; Based on the scale affine matrix and the offset affine matrix, the basic residual features are subjected to an affine transformation to obtain the conditionally normalized feature map.

7. The conditional image generation method according to any one of claims 1-5, characterized in that, The step of extracting self-attention features from the conditionally normalized feature map based on the target orthogonal projection matrix to obtain a self-attention feature map includes: Based on the target orthogonal projection matrix, the conditionally normalized feature map is projected into the feature subspace to obtain the query tensor, key tensor, and value tensor. The point distribution matrix is ​​obtained by performing point integration distribution calculation on the query tensor and the key tensor; The point distribution matrix is ​​sharpened to obtain a sharpened distribution matrix; The self-attention feature map is obtained by weighted aggregation of the sharpening distribution matrix and the value tensor.

8. A conditional image generation apparatus, characterized in that, include: The model loading module is used to obtain the target image generation model, wherein the target image generation model includes a conditional coding network and a self-attention network, and the self-attention network includes a target orthogonal projection matrix; The conditional transformation module is used to perform affine transformation on the preset target image generation conditions based on the conditional coding network to obtain conditional affine features; The normalization processing module is used to perform conditional normalization processing on the preset initial base image based on the conditional affine features to obtain a conditional normalized feature map. The feature extraction module is used to perform self-attention feature extraction on the conditionally normalized feature map based on the target orthogonal projection matrix to obtain a self-attention feature map; The image generation module is used to perform upsampling and synthesis processing on the self-attention feature map to obtain the target image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the conditional image generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the conditional image generation method as described in any one of claims 1 to 7.

Images