Conditional image generation method, apparatus, electronic device and storage medium
By combining a hierarchical sparse encoder and a multi-scale reversible generative network, the problems of feature interference and spatial matching in conditional image generation are solved, achieving high-precision image reconstruction and improved texture clarity.
Patent Information
- Application Number
- CN202611340841.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
In existing conditional image generation methods, the high-dimensional dense conditional feature maps obtained by feature extraction are mixed with a large amount of non-critical interference information, and it is difficult to achieve depth space matching during reconstruction, resulting in blurred texture, artifacts and local misalignment in the generated image.
A hierarchical sparse encoder is used for joint convolutional mapping and sparse gating modulation to filter out non-critical information. A multi-scale reversible generative network is combined to perform cross-level scale routing and reversible affine transformation to ensure the accuracy of feature matching and reconstruction.
It improves the accuracy and texture clarity of conditional image generation, reduces image artifacts, and enhances the local alignment accuracy between the image and the source conditional image.
Smart Images

Figure CN123118798A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence technology and image processing technology, and can be applied to fintech and healthcare scenarios. In particular, it relates to a conditional image generation method, apparatus, electronic device, and storage medium. Background Technology
[0002] Conditional image generation methods are used to generate images that meet semantic requirements based on given conditional information, thus satisfying the need for customized image content in different scenarios. Taking intelligent insurance claims in the financial sector as an example, conditional image generation can generate complete and clear damage assessment images based on accident descriptions and blurry scene photos, assisting claims personnel in quickly assessing losses, thereby improving claims efficiency and reducing fraud risk. In the healthcare sector, for example, in medical image-assisted diagnosis, conditional image generation can generate high-resolution pathological images based on low-resolution scan images or tumor segmentation masks, helping doctors to more accurately identify lesions.
[0003] Currently, conditional image generation methods typically employ fully dense convolutional networks to extract features from an initial low-quality image and the corresponding conditional constraints, resulting in a high-dimensional dense conditional feature map. This map is then fed into a generative network for image feature reconstruction via single-level feature concatenation or direct cascading, yielding the final high-quality target image. However, this method suffers from drawbacks. The high-dimensional dense conditional feature map obtained from feature extraction contains a large amount of non-critical interference information, and the dense conditional feature map struggles to achieve depth space matching with the image reconstruction process. This leads to interference from irrelevant information and a lack of high-precision feature constraints during the reconstruction stage, resulting in texture blurring, artifacts, and local misalignment with the source conditional image in the final high-quality target image. Therefore, improving the accuracy of conditional image generation has become a pressing technical problem. Summary of the Invention
[0004] This invention provides a method, apparatus, computer device, and medium for generating conditional images using artificial intelligence, aiming to suppress information interference during the conditional reconstruction process, thereby improving the texture clarity of the final generated image, reducing image artifacts, and enhancing its local alignment accuracy with the source conditional image.
[0005] Firstly, a conditional image generation method is provided, including: Obtain the segmented component configuration dataset, wherein the segmented component configuration dataset includes sub-item configuration information and interaction strategies; Dynamic layout calculations are performed based on the segmented component configuration dataset to obtain the sub-item rendering coordinate set; Based on the sub-item rendering coordinate set and the sub-item configuration information, view reuse binding is performed to obtain the target sub-item view instance set; Gesture event listening is performed on the target sub-item view instance set to obtain gesture sliding displacement features; Based on the gesture sliding displacement characteristics, the deformation parameters of the indicator are estimated to obtain the deformation parameters of the target indicator. Graphics rendering is performed based on the deformation parameters of the target indicator and the rendering coordinate set of the sub-items.
[0006] Secondly, a conditional image generation apparatus is provided, comprising: The data acquisition module is used to acquire the segmented component configuration dataset, wherein the segmented component configuration dataset includes sub-item configuration information and interaction strategies; The layout calculation module is used to perform dynamic layout calculations based on the segmented component configuration dataset to obtain the sub-item rendering coordinate set. The view reuse binding module is used to perform view reuse binding based on the sub-item rendering coordinate set and the sub-item configuration information to obtain a target sub-item view instance set; The event listening module is used to listen for gesture events on the target sub-item view instance set and obtain gesture sliding displacement features; The parameter estimation module is used to estimate the indicator deformation parameters based on the gesture sliding displacement features to obtain the target indicator deformation parameters. The graphics rendering module is used to perform graphics rendering based on the deformation parameters of the target indicator and the rendering coordinate set of the sub-items.
[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described conditional image generation method.
[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described conditional image generation method.
[0009] In the scheme implemented by the above-mentioned conditional image generation method, apparatus, computer equipment, and storage medium, the initial image features are first obtained by jointly convolving the initial low-quality image and conditional constraint information using the structured sparse weights in the hierarchical sparse encoder. Then, the initial image features are gated and modulated using a sparse gating unit to filter out non-critical interference information and obtain sparse conditional image features. This effectively solves the defect of mixed irrelevant information in the feature extraction process and improves the extraction accuracy of sparse conditional image features. Secondly, by splitting and mapping the sparse conditional image features to different spatial resolution scales of the multi-scale reversible generator network to form multi-scale conditional routing image features, the bottleneck of difficulty in achieving spatial matching between features and image reconstruction process is overcome. This ensures that the multi-scale reversible generator network can obtain high-precision feature constraints at each spatial resolution scale during reconstruction. Finally, guided by the multi-scale conditional routing image features, the initial random noise features are subjected to stepwise reversible affine transformation in the multi-scale reversible generator network to reconstruct and output the target high-quality image. This avoids the problems of interference from irrelevant information and lack of feature constraints in the reconstruction stage, thereby improving the accuracy of conditional image generation. Attached Figure Description
[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Under the premise of sex, other figures can be obtained from these figures.
[0012] Figure 1 This is a schematic diagram of an application environment for a conditional image generation method according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of a conditional image generation method according to an embodiment of the present invention; Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S20; Figure 4 yes Figure 2 A schematic diagram of a specific implementation of step S40; Figure 5 yes Figure 2 A schematic diagram of a specific implementation method for step S50; Figure 6 This is a schematic diagram of a conditional image generation device in one embodiment of the present invention; Figure 7This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0014] The conditional image generation method provided in this embodiment of the invention can be applied to, for example... Figure 1In the user interface rendering application environment, the client communicates with the server via a network. The server can receive the initial low-quality image and corresponding conditional constraints from the user input through the client. Based on the initial low-quality image and conditional constraints, it generates a high-quality target image by calling a pre-trained target conditional image generation model. The target conditional image generation model includes a hierarchical sparse encoder and a multi-scale reversible generative network. The hierarchical sparse encoder includes convolutional units and sparse gating units, with each convolutional unit containing structured sparse weights. The initial low-quality image and conditional constraints are jointly convolved and mapped using the convolutional units and structured sparse weights, resulting in feature fusion and local texture extraction to obtain initial image features. Based on the sparse gating units, the initial image features are subjected to channel-dimensional gating modulation, automatically attenuating unnecessary background channels and amplifying key conditional channels to obtain sparse conditional image features. Based on the multi-scale reversible generative network, the sparse conditional image features are routed across hierarchical scales through stride convolution and feature projection operations to obtain multi-scale conditional routed image features. Finally, based on… This invention relates to a multi-scale reversible generative network and multi-scale conditional routing image features. It performs a reversible affine transformation on preset initial random noise features, converting abstract features losslessly into visual pixels without losing feature information, thus obtaining a high-quality target image. In this application, for the conditional image generation business entity, based on a pre-acquired initial low-quality image and corresponding conditional constraint information, a hierarchical sparse encoder is invoked for joint convolutional mapping and sparse gating modulation. Then, by quantifying the feature importance of each channel, it is transformed into multi-scale conditional routing image features that can serve as precise guiding constraints. Finally, based on these multi-scale conditional routing image features, a reversible affine transformation is performed on the random noise features, making the generated high-quality target image highly correlated with the conditional constraint information. This solves the problems of insufficient single-scale feature representation, redundant noise interference, and loss of detail due to cross-level information transmission in traditional image generation methods. It improves the accuracy of conditional image generation and the sparsity precision of feature representation, thereby improving the efficiency of image processing and user experience. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The present invention will now be described in detail through specific embodiments.
[0015] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the conditional image generation method provided in this embodiment of the invention includes the following steps: S10: Obtain the initial low-quality image and the corresponding condition constraint information; S20: Obtain a pre-trained target conditional image generation model, wherein the target conditional image generation model includes a hierarchical sparse encoder and a multi-scale reversible generation network. The hierarchical sparse encoder includes convolutional units and sparse gating units, and the convolutional units contain structured sparse weights. S30: Perform joint convolution mapping on the initial low-quality image and conditional constraint information based on convolutional units and structured sparse weights to obtain the initial image features; S40: Based on sparse gating units, the initial image features are subjected to gating modulation processing to obtain sparse conditional image features; S50: Based on a multi-scale reversible generative network, cross-scale routing of sparse conditional image features is performed to obtain multi-scale conditional routed image features. S60: Based on multi-scale reversible generative networks and multi-scale conditional routing image features, a reversible affine transformation is performed on the preset initial random noise features to obtain a high-quality target image.
[0016] Steps S10 to S60 of this embodiment first obtain initial image features by jointly convolving the initial low-quality image and conditional constraint information using structured sparse weights in a hierarchical sparse encoder. Then, sparse gating units are used to perform gating modulation processing on the initial image features to filter out non-critical interference information and obtain sparse conditional image features. This effectively solves the defect of mixed irrelevant information in the feature extraction process and improves the extraction accuracy of sparse conditional image features. Secondly, by splitting and mapping the sparse conditional image features to different spatial resolution scales of a multi-scale reversible generator network to form multi-scale conditional routing image features, the bottleneck of difficulty in achieving spatial matching between features and image reconstruction is overcome. This ensures that the multi-scale reversible generator network can obtain high-precision feature constraints at each spatial resolution scale during reconstruction. Finally, guided by the multi-scale conditional routing image features, the initial random noise features undergo stepwise reversible affine transformation in the multi-scale reversible generator network to reconstruct and output a high-quality target image. This avoids the problems of interference from irrelevant information and lack of feature constraints in the reconstruction stage, thereby improving the accuracy of conditional image generation.
[0017] The conditional image generation method provided in this application can be applied to the backend server of various intelligent image generation and restoration systems, such as multimedia image processing, computer vision, generative artificial intelligence, and digital content creation. This backend server can receive user conditional commands and low-quality reference images to achieve high-quality, high-accuracy conditional image generation.
[0018] For example, in the scenario of image restoration of financial documents and invoices, financial institutions often need to perform high-definition digital reconstruction and restoration of a large number of historical, low-quality financial documents that are blurry, incomplete, or creased due to aging scanning equipment or damp paper. In this case, a back-end server is needed to perform high-fidelity reconstruction under specific financial rules to ensure the accuracy of financial audit data. For instance, a commercial bank's financial reimbursement review back-end system plans to perform high-definition restoration on a blurry image of a value-added tax invoice with ink reflection interference. The system first acquires the initial low-quality invoice image and the corresponding conditional constraint information, including pre-input printed text data, invoice codes, and specified red-ink stamp formats. Next, the system reads a pre-trained target conditional image generation model from a preset storage area. This target conditional image generation model includes a hierarchical sparse encoder and a multi-scale reversible generative network. The hierarchical sparse encoder is further composed of convolutional units with structured sparse weights and sparse gating units. Subsequently, the system's convolutional units, based on their structured sparse weights, perform joint convolutional mapping on the input low-quality invoice image and text constraint information. By performing element-level feature fusion and local texture extraction in the matrix feature space, the residual spatial information of the low-quality image is correlated and aligned with the text conditions, thereby outputting initial image features that fuse low-level details and high-level guidance information. Then, the system's sparse gating unit performs gated modulation processing on this initial image feature. This step automatically suppresses useless interference channels caused by creases, ink reflections, etc., in the feature matrix by calculating the global statistical properties of the feature channels and generating adaptive gating probabilities, while significantly amplifying key information channels such as text edges and stamp areas, thus outputting sparse conditional image features that have completed noise removal and feature simplification. Next, the system's multi-scale reversible generative network performs cross-level scale routing on these sparse conditional image features. This step involves progressively reducing the spatial resolution of features using convolution operators with a set stride, thereby constructing a multi-scale feature pyramid within the network that includes local details of invoice corners, character structure, and global layout. This pyramid is then projected onto a unified feature space to obtain multi-scale conditional routing image features. Finally, a multi-scale reversible generative network performs multi-level reversible affine transformations on preset initial random noise features based on the obtained multi-scale conditional routing image features. Without losing feature information, the multi-scale conditions in the routing image features guide the evolution of the probability density of loss and pixel-level affine reconstruction of the random noise features, thus losslessly converting abstract sparse features into high-resolution quantifiable pixels. The final output is a high-quality VAT invoice image with clear text, no noise interference, and fully conforming to the constraints of printed text.
[0019] In step S10 of some embodiments, the initial low-quality image refers to the input original image data that urgently needs to be improved in quality and reconstructed in features. For example, in the scenario of financial document image restoration, the initial low-quality image may be a low-resolution electronic invoice image with blurry text due to shooting angle, insufficient lighting or equipment limitations. In the scenario of medical image super-resolution reconstruction, the initial low-quality image may be an initial medical ultrasound image with a lot of noise and low spatial resolution due to limited scanning time.
[0020] Conditional constraint information refers to multimodal auxiliary information such as text descriptions, category labels, or structural indications used to guide, limit, and regulate the image generation process. For example, in the scenario of financial document and bill image restoration, conditional constraint information may be text label information specifying that the document belongs to a value-added tax invoice or a bank deposit slip, while in the scenario of medical image super-resolution reconstruction, conditional constraint information may be prior constraint data specifying the imaging modality, human anatomical location, or disease tissue type.
[0021] In this embodiment of the application, the conditional image generation system obtains the input initial low-quality image to be processed through a data interface or from a preset storage area, and simultaneously retrieves conditional constraint information that has a strong correlation with the initial low-quality image.
[0022] In step S20 of some embodiments, the target conditional image generation model refers to a deep learning network model that, after training and optimization, has the ability to reconstruct low-quality images into high-quality images under specific conditional constraints. For example, in the scenario of financial document image restoration, the target conditional image generation model may be a deep generative network model specifically designed to reconstruct blurry, low-resolution document images into high-definition images. In the scenario of medical image super-resolution reconstruction, the target conditional image generation model may be a deep reversible generative network model specifically designed to reconstruct low-dose, low-definition magnetic resonance or computed tomography images into high-fidelity lesion images.
[0023] A hierarchical sparse encoder is a network component deployed at the front end of a target conditional image generation model. It is constructed by convolutional units and sparse gating units and is used to perform high-dimensional feature transformation on the input data and adaptively select core conditions.
[0024] Multiscale reversible generative networks refer to network components deployed at the back end of target conditional image generation models that can perform cross-level multi-resolution feature routing and achieve lossless information transfer and high-quality image synthesis through reversible affine transformations.
[0025] Structured sparse weights refer to the convolutional kernel parameter matrix that has been optimized by a pruning controller to remove unnecessary redundant channels and present a specific sparse arrangement.
[0026] In this embodiment, the conditional image generation system can load and instantiate a pre-trained target conditional image generation model in memory. The target conditional image generation model integrates a hierarchical sparse encoder specifically responsible for extracting and filtering features, and a multi-scale reversible generative network responsible for performing lossless feature reconstruction. The hierarchical sparse encoder simplifies the parameters of its convolutional units through a pre-trained and optimized lottery hypothesis pruning algorithm, so that it contains structured sparse weights that can accurately capture core features, thereby ensuring that the target conditional image generation model has high computational efficiency while possessing high generalization reconstruction capability.
[0027] In detail, such as Figure 3 As shown, in step S20, the training process of the target conditional image generation model includes the following steps: S21: Obtain the initial conditional image generation model, wherein the initial conditional image generation model contains initial convolutional units, and the initial convolutional units contain initial convolutional weights; S22: Based on the pre-acquired training sample images and the training constraint information corresponding to the training sample images, the initial convolution weights are updated using a preset optimizer to obtain the updated convolution weights. S23: Based on the preset lottery pruning controller, the updated convolution weights are dynamically thresholded to obtain the pruning threshold; S24: Based on the lottery pruning controller, the difference between the updated convolution weights and the pruning threshold is binarized to obtain a binary pruning mask; S25: Multiply the binary pruning mask and updated convolution weights by matrix elements to obtain structured sparse weights; S26: The initial conditional image generation model is replaced with parameters based on structured sparse weights to obtain the target conditional image generation model.
[0028] In step S21 of some embodiments, the initial conditional image generation model refers to the original deep learning model in the early stage of model training or parameter optimization process, before the network parameters have been pruned, distilled or fine-tuned, and is still in a state of waiting to be optimized.
[0029] The initial convolutional unit refers to the basic feedforward computation component contained in each convolutional layer within the initial conditional image generation model, which undertakes the most basic feature mapping function.
[0030] Initial convolution weights refer to the original feature mapping parameter matrix in the initial convolution unit that has not been filtered by the lottery pruning algorithm and contains high redundancy.
[0031] In this embodiment, the conditional image generation system can first retrieve the initial conditional image generation model to be optimized, and then use internal components to retrieve the initial conditional image generation model containing initial convolutional units responsible for basic operations. At the same time, it is further clarified that the initial convolutional units contain initial convolutional weights distributed in a dense state, which are used as the operator objects for subsequent lottery pruning optimization evolution.
[0032] In step S22 of some embodiments, the training sample images refer to the paired image datasets that are pre-acquired during the network training phase and used to guide model parameter learning and loss calculation. For example, in the scenario of financial document image restoration, the training sample images may be low-quality invoice images containing a large number of scan stains and corresponding high-quality invoice images without stains. In the scenario of medical image super-resolution reconstruction, the training sample images may be low-dose computed tomography images containing high noise and corresponding high-quality medical images with standard dose.
[0033] Training constraint information refers to text labels or modal feature data that accompany the training sample images and are used to indicate the generation target during the model optimization training process. For example, in the scenario of financial document and bill image restoration, training constraint information can be a category one-hot encoding representing a specific financial document format, while in the scenario of medical image super-resolution reconstruction, training constraint information can be anatomical prior labels indicating specific anatomical tissues or lesion areas of the human body.
[0034] Updating convolution weights refers to the initial convolution weights being updated after gradient calculation via backpropagation and iterative gradient descent using a pre-defined optimizer. This process initially preserves the feature mapping parameters of the sample features. For example, in the scenario of financial document image restoration, updated convolution weights can be intermediate parameters that have initially acquired the ability to extract text contours after several rounds of training with invoice samples. In the scenario of medical image super-resolution reconstruction, updated convolution weights can be calculated weights that can initially fit the tissue structure after iterative optimization using medical image data.
[0035] In this embodiment, after loading the initial conditional image generation model, the conditional image generation system can call the preset optimizer in the initial conditional image generation model based on the pre-acquired training sample images and the training constraint information corresponding to the training sample images. The system then performs loss alignment between the reconstructed image obtained by forward prediction and the high-quality real image in the training sample images. The obtained image loss data is then backpropagated in reverse gradient, which can accurately update the initial convolution weights and thus smoothly output the updated convolution weights.
[0036] Specifically, the training sample images include low-quality sample images and corresponding high-quality sample images. Based on this, and using the pre-acquired training sample images and the corresponding training constraint information, a preset optimizer is used to update the initial convolution weights using gradients, resulting in updated convolution weights, including: Based on the initial conditional image generation model, forward prediction is performed on the low-quality sample image and training constraint information to obtain the reconstructed image; Loss calculations are performed on the reconstructed image and the high-quality sample image to obtain image loss data; The updated convolution weights are obtained by calculating the inverse gradient of the initial convolution weights based on the image loss data.
[0037] In some embodiments, a low-quality sample image refers to a low-resolution or high-noise initial example image that is input during the model training phase and serves as the source of learning degenerate features for the network to be optimized.
[0038] High-quality sample images refer to target example images that exist in pairs with low-quality sample images, achieve ideal standards in visual clarity and texture structure, and serve as ground truth for network training supervision.
[0039] Reconstructed images refer to the preliminary feature restoration or image synthesis results output by the initial conditional image generation model after forward prediction calculation, which contain the mapping capabilities of the network at the current stage. For example, in the scenario of financial document and invoice image restoration, the reconstructed image can be a predicted invoice and invoice image that has been preliminarily synthesized by the initial conditional image generation model and whose handwriting defects have been alleviated but still need further optimization. In the scenario of medical image super-resolution reconstruction, the reconstructed image can be a predicted medical organ image that has been preliminarily deduced by the initial conditional image generation model and whose noise has been suppressed but whose tissue texture still needs to be refined.
[0040] Image loss data refers to the scalar or tensor numerical results obtained after calculating the loss function from the reconstructed image and the high-quality sample image, which are used to quantitatively characterize the prediction error of the current network.
[0041] In this embodiment, after loading the initial conditional image generation model into the training system, the acquired low-quality sample images and their corresponding training constraints can be simultaneously input into the initial conditional image generation model. Then, relying on the initial convolution operator layers currently included in the initial conditional image generation model, layer-by-layer forward prediction operations are performed on these two input data, thereby successfully obtaining a reconstructed image that has initially recovered some visual features at the output. After obtaining the reconstructed image, it can be simultaneously input into a preset loss function estimator along with pre-retrieved high-quality sample images. Then, the loss function estimator is used to perform pixel-to-pixel or multi-scale structural similarity loss calculation on the reconstructed image and the high-quality sample images, thereby accurately and quantitatively outputting image loss data to characterize the current network's reconstruction capability error. After capturing the image loss data, the backhaul link of the computation graph can be started. Based on the image loss data and using the automatic differentiation algorithm, the backhaul gradient calculation of the chain partial derivatives is performed in reverse along the network layers of the initial condition image generation model. Finally, the calculated gradient values of each layer are sent back to the system's preset optimizer, which can perform directional parameter fine-tuning and gradient update of the initial convolution weights, thereby smoothly outputting the updated convolution weights.
[0042] This application first utilizes an initial conditional image generation model to perform full forward prediction calculations on low-quality sample images and training constraint information to obtain reconstructed images. Then, a preset loss function is used to perform multi-dimensional difference measurement and loss calculations on the reconstructed images and high-quality sample images to obtain image loss data. This effectively solves the problem of unclear training objectives due to the lack of high-precision error measurement tools during model optimization, improving the accuracy of image loss data in representing spatial structural differences. Secondly, by performing inverse chain derivatives and backward gradient calculations on the initial convolutional weights in the initial conditional image generation model based on the image loss data, updated convolutional weights are obtained. This overcomes the bottleneck of traditional blind parameter tuning methods failing to accurately lock the direction of parameter optimization, ensuring that the updated convolutional weights can perfectly capture the causal mapping between cross-modal conditions and high-quality images. Finally, guided by the updated convolutional weights as the benchmark for subsequent lottery pruning, multiple rounds of iterative gradient updates achieve robust convergence of the network to output highly expressive reconstructed parameters. This avoids artifacts or illogical textures in the generated images due to untimely supervision feedback during the training evolution stage, thereby improving the overall training stability of the conditional image generation model and the visual realism of the generated images.
[0043] In step S23 of some embodiments, the lottery pruning controller refers to a lightweight parameter management component based on lottery hypothesis theory, which removes non-core redundant weights in a deep network through dynamic threshold calculation and binary mask generation. For example, in the scenario of financial document image restoration, the lottery pruning controller can be a parameter compression algorithm module that automatically strips useless document background channels and locks only the text skeleton channel. In the scenario of medical image super-resolution reconstruction, the lottery pruning controller can be a dynamic pruning network that automatically identifies and retains the high-frequency texture channels of key lesions and cuts off low-contribution noise channels.
[0044] The pruning threshold is a dynamic numerical benchmark calculated by the lottery pruning controller based on the weight importance assessment results, used to determine whether weight parameters need to be pruned or removed. For example, in the scenario of financial document image restoration, the pruning threshold can be a numerical boundary extracted according to the distribution law of the absolute value of the parameters of the entire network, used to eliminate parameters with low absolute value contribution. In the scenario of medical image super-resolution reconstruction, the pruning threshold can be a dynamic percentile extracted according to a specific sparsity ratio, used to distinguish the core weight of lesions from the weight of redundant noise.
[0045] In this embodiment, after calculating the updated convolution weights, they can be directly input into a preset lottery pruning controller. Then, the weight importance evaluator integrated within the lottery pruning controller is used to sort the matrix elements of the updated convolution weights in ascending or descending order of absolute value, thereby obtaining the corresponding weight sorting distribution. Subsequently, based on the sparsity ratio parameter preset by the system, the weight sorting distribution is dynamically truncated to percentiles, so that the pruning threshold corresponding to the current training stage can be intelligently output in the real number domain.
[0046] In detail, based on a preset lottery pruning controller, dynamic threshold calculations are performed on the updated convolution weights to obtain the pruning threshold, including: The weights are sorted by absolute value based on the lottery pruning controller to obtain the weight sorting distribution. Based on a preset sparsity ratio parameter, the weighted sorting distribution is dynamically truncated by percentiles to obtain the pruning threshold.
[0047] In some embodiments, weight sorting distribution refers to the statistical sequence data formed after the updated convolution weights are arranged according to specific numerical rules by the lottery pruning controller, which can intuitively reflect the order and hierarchical structure of the weights of the entire network.
[0048] The sparsity ratio parameter refers to a pre-defined baseline hyperparameter in the system that specifies the proportion of redundant parameters to be pruned or eliminated during network pruning, or the target sparsity level. For example, in the scenario of financial document image restoration, the sparsity ratio parameter can be a pre-defined ratio used to prune 7 / 10 of the invoice background channel. In the scenario of medical image super-resolution reconstruction, the sparsity ratio parameter can be a pre-defined truncation coefficient used to remove medical artifact interference and maintain the model's lightweightness by 2 / 3.
[0049] In this embodiment, after obtaining the updated convolution weights, the conditional image generation system can load them into the lottery pruning controller. Then, based on the weight importance evaluator configured within the lottery pruning controller, the matrix elements of the updated convolution weights are sorted in ascending or descending order of absolute value, thereby accurately and intuitively outputting the weight ranking distribution that reflects the differences in parameter importance. After obtaining the weight ranking distribution, the conditional image generation system can automatically retrieve the pre-set sparsity ratio parameter in the storage area, and then use the sparsity ratio parameter as a truncation boundary operator to dynamically perform percentile truncation and numerical segmentation calculations on the weight ranking distribution. This allows for intelligent extraction and output of the pruning threshold in the current model state.
[0050] It should also be noted that in this embodiment, the pruning threshold is not fixed, but rather adaptively learned through a trainable parameter. Specifically, the lottery pruning controller can calculate the pruning threshold for the current level using the following formula:
[0051] in, The pruning threshold for level l. Let be the trainable pruning bias parameters of the l-th level. These pruning bias parameters are internal learnable variables of the lottery pruning controller, unconstrained by any manually preset sparsity ratio, and updated entirely by the training loss. , These are the upper and lower limits of the preset pruning ratio for level l, respectively.
[0052] This application first uses a weight importance evaluator to sort the absolute values of the updated convolutional weights to obtain the weight sorting distribution, and then dynamically truncates the weight sorting distribution by percentile based on a preset sparsity ratio parameter to obtain the pruning threshold. This effectively solves the defect of traditional static pruning methods that cannot adaptively match the differences in parameter distribution at different levels due to a fixed hard threshold, and improves the adaptive calculation accuracy of the pruning threshold at different network depths.
[0053] In step S24 of some embodiments, the binary pruning mask refers to a matrix selection matrix composed of zero-value and one-value elements that is completely consistent with the dimension of the weight matrix. It is used to indicate whether the convolution parameters at the corresponding spatial location should be retained or pruned to zero. For example, in the scenario of financial document image restoration, the binary pruning mask can be a matrix template that marks the position corresponding to the invoice text channel as one and marks the position of the reflective background channel as zero. In the scenario of medical image super-resolution reconstruction, the binary pruning mask can be a mask selection matrix that ensures that the weight path corresponding to the lesion contour remains unobstructed while directly blocking the redundant interference path.
[0054] In this embodiment, after determining the pruning threshold, the numerical difference between the convolution weights and the pruning threshold can be automatically read and updated by the lottery pruning controller inside the lottery pruning controller. Then, the lottery pruning controller performs a continuous smooth scaling operation on the difference to generate a smooth probability in the real number domain. Finally, the real number domain smooth probability is deterministically discretized into binary values using a fixed threshold, which can accurately obtain a binary pruning mask composed of 0s and 1s.
[0055] In detail, based on the difference between the updated convolutional weights and the pruning threshold, the lottery pruning controller performs binarization selection to obtain a binary pruning mask, including: Based on the lottery pruning controller, the difference between the updated convolution weights and the pruning threshold is continuously and smoothly scaled to obtain the real-domain smoothing probability. The binary pruning mask is obtained by deterministically thresholding the smoothing probability in the real number field.
[0056] In some embodiments, the real-domain smoothing probability refers to a numerical matrix data that is located in a continuous real-valued interval and can characterize the probability of the weight parameters being retained or pruned after the difference between the updated convolution weights and the pruning threshold is continuously scaled by a smoothing scale. For example, in the scenario of image restoration of financial documents and invoices, the real-domain smoothing probability can be a continuous real-valued matrix that reflects the probability that the text edge channel parameter of the invoice is retained is 0.99 while the probability that the background interference channel is retained is 0.01.
[0057] In this embodiment, after obtaining the updated convolution weights and calculating the corresponding pruning threshold, the conditional image generation system can automatically read the numerical difference between the updated convolution weights and the pruning threshold using the gradient continuous gating function integrated within the lottery pruning controller. Then, the gradient continuous gating function performs a continuous smooth scaling operation on this difference, smoothly projecting the discrete numerical difference into a continuous probability interval, thereby stably outputting the smoothed probability in the real-number domain. After obtaining the smoothed probability in the real-number domain, the lottery pruning controller can automatically call a preset hard-decision operator or sign function to perform a deterministic fixed-threshold discretization binary selection operation on the smoothed probability in the real-number domain, forcibly mapping the continuous probability values between 0 and 1 to absolute zero or one values. This allows for the accurate acquisition of a binary pruning mask that can be directly used for matrix multiplication.
[0058] This application first uses a lottery pruning controller to continuously and smoothly scale the difference between the updated convolution weights and the pruning threshold to obtain the real-domain smoothing probability, and then performs deterministic threshold discretization on the real-domain smoothing probability to obtain the binary pruning mask. This effectively solves the defect of traditional hard threshold pruning in the discretization stage where the gradient cannot propagate back normally due to non-differentiability, and improves the mathematical smoothing transformation accuracy of the binary pruning mask in the binarization selection process.
[0059] It's important to know that the binary pruning mask can be calculated using the following formula:
[0060] in, This is the binary pruning mask for the l-th level. This is an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise. This indicates taking the maximum absolute value among the updated convolution weights.
[0061] In step S25 of some embodiments, structured sparse weights refer to an efficient convolution kernel parameter matrix that eliminates unnecessary redundant channels and presents a specific geometric sparse shape after the convolution weights are updated and the binary pruning mask is multiplied element by element.
[0062] In this embodiment, after obtaining the binary pruning mask, the binary pruning mask and the updated convolution weights can be multiplied element-wise by a matrix parallel computing unit. At this time, the updated convolution weight parameters corresponding to the positions with zero values in the binary pruning mask will be directly erased and set to zero, while the positions with one values will be perfectly preserved. This can eliminate redundant parameter channels with low contribution and output structured sparse weights that conform to the lottery hypothesis.
[0063] It is important to know that the structured sparse weights can be calculated using the following formula:
[0064] in, For the structured sparse weights of the l-th level, This is used to update the convolution weights for the l-th layer.
[0065] In step S26 of some embodiments, after the structured sparse weights are smelted, they can be used as the core lightweight structural assets and backfilled to the corresponding convolutional layers in the initial conditional image generation model. This allows the structured sparse weights to replace all or part of the parameters of the initial conditional image generation model, thus seamlessly transforming and evolving to output a target conditional image generation model that combines strong generalization and extremely high computing power throughput.
[0066] Steps S21 to S26 of the embodiments of this application firstly update the initial convolutional weights in the initial conditional image generation model by iterative gradient updates based on training sample images and training constraint information using a preset optimizer. Then, a preset lottery pruning controller performs absolute value sorting and dynamic percentile truncation based on sparse ratio parameters on the updated convolutional weights to obtain the pruning threshold. This effectively solves the defect that the threshold is fixed and cannot adapt to changes in the model training state when network parameters are simplified, improving the adaptive dynamic adjustment accuracy of the pruning threshold calculation. Secondly, the difference between the updated convolutional weights and the pruning threshold is continuously and smoothly scaled and deterministically thresholded by the lottery pruning controller. Value discretization is used to obtain a binary pruning mask, overcoming the bottleneck of traditional hard threshold pruning in the discretization stage where smooth transformation is difficult due to non-differentiability. This ensures that the binary pruning mask can specifically and accurately distinguish between core and redundant channels. Finally, guided by the multiplication of the binary pruning mask and the matrix elements for updating convolution weights, structured sparse weights are formed by eliminating redundant computational paths. The structured sparse weights are then used to replace the parameters of the initial conditional image generation model to output the target conditional image generation model. This avoids the performance collapse and high computational redundancy caused by structured pruning in the network simplification and reconstruction stage, thereby improving the computational throughput and inference efficiency of the conditional image generation model in edge device deployment.
[0067] In step S30 of some embodiments, the initial image feature refers to the high-dimensional spatial feature vector containing multimodal information obtained by initially fusing the initial low-quality image and conditional constraint information through multiple layers of mapping by the convolutional units in the hierarchical sparse encoder. For example, in the scenario of financial document and bill image restoration, the initial image feature may be a hybrid latent space feature that fuses low-frequency pixels of the invoice and the invoice category label. In the scenario of medical image super-resolution reconstruction, the initial image feature may be a preliminary visual feature that fuses low-dose image structure and prior knowledge of lesions.
[0068] In this embodiment, after the initial low-quality image and conditional constraint information are synchronously input into the target conditional image generation model, joint convolutional mapping operations can be performed on the two different modal input data, the initial low-quality image and conditional constraint information, according to the structured sparse weights in the target conditional image generation model. This enables early deep fusion of cross-modal information, thereby obtaining the initial image features.
[0069] It should be noted that in this embodiment, the hierarchical sparse encoder contains L levels, where L is a positive integer greater than 1. For a level with l levels (l=1, 2, ..., L), when l=1, the input features of that level are the initial image features. It should also be noted that the l-th level, after processing by the aforementioned convolutional unit and structured sparse weights, can be represented as:
[0070] in, The intermediate feature output by the l-th level convolutional unit is, as it is important to know, process data within the hierarchical sparse encoder, used as input to subsequent sparse gating units. is a convolutional unit of the l-th level, which contains structured sparse weights.
[0071] In step S40 of some embodiments, sparse conditional image features refer to high-purity sparse vectors that amplify key causal features and suppress redundant interference features after the initial image features are gated, modulated, and filtered at the channel level by a sparse gating unit. For example, in the scenario of financial document image restoration, sparse conditional image features can be refined features that enhance text readability and reduce paper creases and reflection interference. In the scenario of medical image super-resolution reconstruction, sparse conditional image features can be pure features that highlight the core tissue features of lesions and shield scanner noise and physiological artifacts.
[0072] In this embodiment, after obtaining the initial image features, the initial image features can be adaptively gated modulation processed by relying on the sparse gating unit inside the hierarchical sparse encoder. In this process, the initial image features can first be subjected to channel global average pooling to obtain channel description image features. Then, the channel description image features are subjected to nonlinear activation mapping to output the corresponding channel activation probabilities. Finally, based on the channel activation probabilities and the initial image features, element-wise multiplication is performed on each channel to remove redundant interference with high precision and output sparse conditional image features.
[0073] It is important to know that the output of each level in a hierarchical sparse encoder can be obtained using the following formula:
[0074]
[0075] in, This represents the output feature of each level in a hierarchical sparse encoder, when l=L. This refers to sparse conditional image features. Let l be the gating function of the l-th level. The input to the gate function is usually... , This means that the L2 norm is calculated first along the channel dimension and then summed along the spatial dimension. It can be used to measure the overall activity of each feature channel. The learnable temperature coefficient for the l-th level is a trainable parameter of the sparse gating unit, used to control the sharpness of the gating decision. It is the Sigmoid activation function. This indicates Bernoulli sampling.
[0076] In detail, such as Figure 4 As shown, in step S40, that is, based on the sparse gating unit, the initial image features are subjected to gating modulation processing to obtain sparse conditional image features, including the following steps: S41: Perform channel-based global average pooling on the initial image features based on sparse gating units to obtain channel-described image features; S42: Perform nonlinear activation mapping on the channel description image features to obtain the channel activation probabilities; S43: Multiply the initial image features by channel elements based on the channel activation probability to obtain sparse conditional image features.
[0077] In step S41 of some embodiments, the channel description image feature refers to the scalar description vector obtained by channel global average pooling of the initial image features, which is used to indicate the global spatial distribution and intensity of each feature channel.
[0078] In this embodiment, after receiving the initial image features, the sparse gating unit can first read the pixel distribution of the initial image features in the width and height dimensions, and then use the channel global average pooling operator to calculate the average pixel value of the feature matrix of each channel in the whole image space, thereby reducing and compressing the multidimensional spatial distribution features into a one-dimensional scalar that can represent the global statistical characteristics of each channel, and thus outputting the channel description image features.
[0079] In step S42 of some embodiments, the channel activation probability refers to a real weight coefficient that is obtained by nonlinear mapping of the channel description image features, and whose value ranges from zero to one, and is used to indicate the importance or retention probability of each channel.
[0080] In this embodiment, after obtaining the channel description image features, the sparse gating unit can use a multilayer perceptron network composed of two fully connected layers to perform feature dimension shrinking and expansion transformation on the channel description image features, and then input them into the logistic activation function for nonlinear activation mapping processing, thereby converting the response intensity of each feature channel into a channel activation probability with a value range limited to zero to one.
[0081] In step S43 of some embodiments, after obtaining the channel activation probability, the sparse gating unit can perform element-wise multiplication broadcast modulation by multiplying the activation probability of each channel with all spatial pixel values of the corresponding channel in the initial image features. The probability coefficient is used as a gating weight to automatically attenuate unnecessary background channels and amplify key condition channels, and finally outputs sparse condition image features with channel reconstruction weighting.
[0082] Steps S41 to S43 of the embodiments of this application first perform channel global average pooling on the initial image features through a sparse gating unit to extract channel descriptive image features, and then perform nonlinear activation mapping on them to obtain channel activation probabilities. Finally, based on the channel activation probabilities, the initial image features are multiplied and weighted by channel elements to output sparse conditional image features. This effectively solves the defect of blurred image reconstruction caused by ignoring the correlation between channels and adding redundant and useless features during the feature extraction process, and improves the selective expression ability of sparse conditional image features for useful conditional constraint information and the sparse accuracy of feature representation.
[0083] In step S50 of some embodiments, multi-scale conditional routing image features refer to sparse conditional image features that, after undergoing multi-resolution transformation and cross-level transmission in a multi-scale reversible generative network, carry rich feature expressions of details with different spatial resolutions. For example, in the scenario of financial document image restoration, multi-scale conditional routing image features can be multi-resolution features that simultaneously contain macroscopic invoice table frames and microscopic digital stroke textures. In the scenario of medical image super-resolution reconstruction, multi-scale conditional routing image features can be multi-level features that simultaneously contain the overall contour of organs and cellular-level lesion details.
[0084] In this embodiment, after obtaining sparse conditional image features, the cascaded architecture of a multi-scale reversible generative network can be used to perform cross-level scale routing and hierarchical projection processing on the sparse conditional image features. In specific operation, the multi-scale reversible generative network can be used to perform stride convolution on the sparse conditional image features to build a multi-scale image feature pyramid. Furthermore, cross-resolution feature projection is performed on the multi-scale image feature pyramid to complete the deep fusion of global semantic features and local detail features, and finally, multi-scale conditional routed image features can be obtained.
[0085] In detail, such as Figure 5 As shown, in step S50, which is based on a multi-scale reversible generative network, cross-scale routing of sparse conditional image features is performed to obtain multi-scale conditional routed image features, including the following steps: S51: Based on a multi-scale reversible generative network, stride convolution is performed on sparse conditional image features to obtain a multi-scale image feature pyramid. S52: Based on a multi-scale reversible generative network, feature projection is performed on the multi-scale image feature pyramid to obtain multi-scale conditional routing image features.
[0086] In step S51 of some embodiments, the multi-scale image feature pyramid refers to a set of image feature maps with different spatial resolutions and different numbers of channels, arranged in a pyramid-like hierarchy, obtained by multi-level downsampling or stride convolution operations. For example, in the scenario of financial document image restoration, the multi-scale image feature pyramid can be a multi-level feature set containing high-resolution character corner features, medium-resolution character structure features, and low-resolution page layout features. In the scenario of medical image super-resolution reconstruction, the multi-scale image feature pyramid can be a progressive scale feature map array containing clear blood vessel boundary features, solid organ contour features, and global anatomical orientation information.
[0087] In this embodiment, after receiving sparse conditional image features, the multi-scale reversible generative network can first use its multi-layer stride convolution operators with set sliding strides to perform stride convolution extraction on the sparse conditional image features. By sliding the convolution kernels in the horizontal and vertical directions with a span of more than one pixel, the spatial resolution of the sparse conditional image features is compressed layer by layer while increasing the channel depth, thereby constructing a multi-scale image feature pyramid containing high-resolution local details to low-resolution global semantic features within the network.
[0088] In step S52 of some embodiments, after obtaining the multi-scale image feature pyramid, the multi-scale reversible generative network can further call the feature projection component, use a one-to-one pixel-by-pixel channel convolution kernel to perform dimensionality reduction or dimensionality increase mapping on the feature maps of different channel numbers in each layer of the multi-scale image feature pyramid, force the features of each scale level to be projected into the same preset hidden layer feature space to complete feature alignment, and finally output multi-scale conditional routing image features for subsequent reversible affine transformation layers to perform scale adaptive routing.
[0089] Steps S51 to S52 as shown in the embodiments of this application firstly construct a multi-scale image feature pyramid by performing stride convolution on sparse conditional image features through a multi-scale reversible generative network, and then perform feature projection on the multi-scale image feature pyramid to output multi-scale conditional routing image features. This effectively solves the problem of detail distortion caused by insufficient single-scale feature expression ability and loss of cross-level information transmission during the conditional image generation process, realizes the collaborative alignment and fusion of multi-scale spatial features, and improves the multi-level semantic constraint accuracy of the output conditional routing image features when guiding noisy images to perform reversible affine transformations.
[0090] In step S60 of some embodiments, the target high-quality image refers to the final reconstructed image that meets the conditional constraint information and achieves the preset high standard requirements in terms of visual clarity and texture realism. For example, in the scenario of financial document image restoration, the target high-quality image can be an ultra-high resolution electronic document image with clear and legible handwriting that meets the requirements of automatic optical character recognition. In the scenario of medical image super-resolution reconstruction, the target high-quality image can be a super-resolution medical image with complete image structure and clear lesion edges that can provide high-precision assistance for clinical diagnosis.
[0091] In this embodiment, after obtaining the multi-scale conditional routing image features, a preset initial random noise feature can be generated by calling a Gaussian distribution sampler. Then, a multi-scale reversible generator network is used to use the multi-scale conditional routing image features as modulation parameters to perform multiple iterative reversible affine transformations on the initial random noise feature. While ensuring that the information flow does not degrade, the high-frequency structure of the image is gradually restored, and finally, a high-quality target image that meets the original condition constraints and has extremely high pixel clarity can be synthesized.
[0092] Specifically, the multi-scale reversible generative network can first obtain the sparse conditional image features of each level and the corresponding binary pruning mask. Then, it applies the binary pruning mask to the sparse conditional image features of each level for secondary filtering. After learningable adaptive pooling to unify the spatial size, it further performs element-wise addition fusion to obtain the fused conditional features, which can be calculated using the following formula:
[0093] in, The fused conditional features are the sparse conditional image features from each level that have been pooled, uniformly sized, and fused together. For learnable adaptive pooling, This represents the total number of levels in a hierarchical sparse encoder. Indicates the current level.
[0094] Then, the multi-scale reversible generative network concatenates the current latent variable with the fusion conditional feature along the channel dimension and inputs it into the multilayer perceptron network to generate the scale transformation parameters and displacement transformation parameters required for the reversible affine transformation. The initial current latent variable is the preset initial random noise feature, and the scale transformation parameters and displacement transformation parameters can be calculated using the following formulas:
[0095] in, These are the scaling transformation parameters. For displacement transformation parameters, This indicates the number of the current invertible affine transformation. Let be the current latent variable before the i-th invertible affine transformation. It is a multilayer perceptron network. This indicates a feature splicing operation along the channel dimension.
[0096] Building upon this, multi-scale reversible generative networks can utilize scale transformation parameters and displacement transformation parameters to perform affine coupling updates on the current latent variables in order to obtain the current latent variables of the next level:
[0097] in, Let be the current hidden variable of the affine coupling layer after the i-th invertible affine transformation or before the (i+1)-th invertible affine transformation. It is a natural exponential function, which guarantees that the scaling factor is positive.
[0098] Finally, after multiple reversible affine transformations, the current latent variables are then mapped through the output of a multi-scale reversible generative network to obtain a high-quality target image.
[0099] As can be seen, in the above scheme, the initial image features are obtained by first using the structured sparse weights in the hierarchical sparse encoder to perform joint convolution mapping on the initial low-quality image and conditional constraint information. Then, the initial image features are gated and modulated using sparse gating units to filter out non-critical interference information and obtain sparse conditional image features. This effectively solves the defect of mixed irrelevant information in the feature extraction process and improves the extraction accuracy of sparse conditional image features. Secondly, by splitting and mapping the sparse conditional image features to different spatial resolution scales of the multi-scale reversible generator network to form multi-scale conditional routing image features, the bottleneck of difficulty in achieving spatial matching between features and image reconstruction process is overcome. This ensures that the multi-scale reversible generator network can obtain high-precision feature constraints at each spatial resolution scale during reconstruction. Finally, under the guidance of the multi-scale conditional routing image features, the initial random noise features are subjected to stepwise reversible affine transformation in the multi-scale reversible generator network to reconstruct and output the target high-quality image. This avoids the problems of interference from irrelevant information and lack of feature constraints in the reconstruction stage, thereby improving the accuracy of conditional image generation.
[0100] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0101] In one embodiment, a conditional image generation apparatus is provided, which corresponds one-to-one with the conditional image generation methods described in the above embodiments. For example... Figure 6 As shown, the conditional image generation device includes a data acquisition module, a model loading module, a convolution mapping module, a gating processing module, a scale routing module, and a reversible transformation module. Detailed descriptions of each functional module are as follows: The data acquisition module is used to acquire the initial low-quality image and the corresponding conditional constraint information. The model loading module is used to obtain a pre-trained target conditional image generation model, which includes a hierarchical sparse encoder and a multi-scale reversible generation network. The hierarchical sparse encoder includes convolutional units and sparse gating units, and the convolutional units contain structured sparse weights. The convolutional mapping module is used to perform joint convolutional mapping on the initial low-quality image and conditional constraint information based on convolutional units and structured sparse weights to obtain the initial image features; The gating processing module is used to perform gating modulation processing on the initial image features based on sparse gating units to obtain sparse conditional image features; The scale routing module is used to perform cross-level scale routing on sparse conditional image features based on a multi-scale reversible generative network, so as to obtain multi-scale conditional routed image features. The reversible transformation module is used to perform a reversible affine transformation on preset initial random noise features based on multi-scale reversible generator networks and multi-scale conditional routing image features to obtain a high-quality target image.
[0102] In one embodiment, the model loading module is specifically used for: Obtain an initial conditional image generation model, wherein the initial conditional image generation model contains initial convolutional units, and the initial convolutional units contain initial convolutional weights; Based on the pre-acquired training sample images and the training constraint information corresponding to the training sample images, the initial convolution weights are updated by gradient using a preset optimizer to obtain the updated convolution weights. The pruning threshold is obtained by dynamically calculating the updated convolution weights based on the preset lottery pruning controller. The difference between the updated convolutional weights and the pruning threshold is binarized based on the lottery pruning controller to obtain a binary pruning mask. The binary pruning mask and the updated convolution weights are multiplied by matrix elements to obtain structured sparse weights. The initial conditional image generation model is replaced with structured sparse weights to obtain the target conditional image generation model.
[0103] In one embodiment, the model loading module is further configured to: Based on the initial conditional image generation model, forward prediction is performed on the low-quality sample image and training constraint information to obtain the reconstructed image; Loss calculations are performed on the reconstructed image and the high-quality sample image to obtain image loss data; The updated convolution weights are obtained by calculating the inverse gradient of the initial convolution weights based on the image loss data.
[0104] In one embodiment, the model loading module is further configured to: The weights are sorted by absolute value based on the lottery pruning controller to obtain the weight sorting distribution. Based on a preset sparsity ratio parameter, the weighted sorting distribution is dynamically truncated by percentiles to obtain the pruning threshold.
[0105] In one embodiment, the model loading module is further configured to: Based on the lottery pruning controller, the difference between the updated convolution weights and the pruning threshold is continuously and smoothly scaled to obtain the real-domain smoothing probability. The binary pruning mask is obtained by deterministically thresholding the smoothing probability in the real number field.
[0106] In one embodiment, the gating processing module is specifically used for: Channel-based global average pooling is performed on the initial image features using sparse gating units to obtain channel-specific image features. The channel activation probabilities are obtained by performing nonlinear activation mapping on the channel description image features; The initial image features are multiplied element-wise based on the channel activation probability to obtain sparse conditional image features.
[0107] In one embodiment, the scale routing module is specifically used for: Based on a multi-scale reversible generative network, stride convolution is performed on sparse conditional image features to obtain a multi-scale image feature pyramid. Based on a multi-scale reversible generative network, feature projection is performed on a multi-scale image feature pyramid to obtain multi-scale conditional routing image features.
[0108] This invention provides a conditional image generation device. First, it uses structured sparse weights in a hierarchical sparse encoder to perform joint convolution mapping on an initial low-quality image and conditional constraint information to obtain initial image features. Then, it uses a sparse gating unit to perform gating modulation processing on the initial image features to filter out non-critical interference information, obtaining sparse conditional image features. This effectively solves the defect of mixed irrelevant information during feature extraction and improves the extraction accuracy of sparse conditional image features. Second, by splitting and mapping the sparse conditional image features to different spatial resolution scales of a multi-scale reversible generative network to form multi-scale conditional routing image features, it overcomes the bottleneck of difficulty in achieving spatial matching between features and image reconstruction, ensuring that the multi-scale reversible generative network can obtain high-precision feature constraints at each spatial resolution scale during reconstruction. Finally, guided by the multi-scale conditional routing image features, the initial random noise features undergo stepwise reversible affine transformations in the multi-scale reversible generative network to reconstruct and output a high-quality target image. This avoids the problems of interference from irrelevant information and lack of feature constraints during the reconstruction stage, thereby improving the accuracy of conditional image generation.
[0109] Specific limitations regarding the conditional image generation device can be found in the limitations of the conditional image generation method described above, and will not be repeated here. Each module in the aforementioned conditional image generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0110] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a conditional image generation method on the server side.
[0111] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a conditional image generation method on the client side. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the initial low-quality image and the corresponding conditional constraint information; Obtain a pre-trained target conditional image generation model, wherein the target conditional image generation model includes a hierarchical sparse encoder and a multi-scale reversible generation network. The hierarchical sparse encoder includes convolutional units and sparse gating units, and the convolutional units contain structured sparse weights. The initial image features are obtained by performing joint convolution mapping on the initial low-quality image and conditional constraint information based on convolutional units and structured sparse weights. Based on sparse gating units, the initial image features are subjected to gating modulation processing to obtain sparse conditional image features. Based on a multi-scale reversible generative network, cross-scale routing of sparse conditional image features is performed to obtain multi-scale conditional routed image features. Based on multi-scale reversible generative networks and multi-scale conditional routing image features, a reversible affine transformation is performed on the preset initial random noise features to obtain a high-quality target image.
[0112] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the initial low-quality image and the corresponding conditional constraint information; Obtain a pre-trained target conditional image generation model, wherein the target conditional image generation model includes a hierarchical sparse encoder and a multi-scale reversible generation network. The hierarchical sparse encoder includes convolutional units and sparse gating units, and the convolutional units contain structured sparse weights. The initial image features are obtained by performing joint convolution mapping on the initial low-quality image and conditional constraint information based on convolutional units and structured sparse weights. Based on sparse gating units, the initial image features are subjected to gating modulation processing to obtain sparse conditional image features. Based on a multi-scale reversible generative network, cross-scale routing of sparse conditional image features is performed to obtain multi-scale conditional routed image features. Based on multi-scale reversible generative networks and multi-scale conditional routing image features, a reversible affine transformation is performed on the preset initial random noise features to obtain a high-quality target image.
[0113] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0114] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0115] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0116] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.
[0117] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A conditional image generation method, characterized in that, include: Obtain an initial low-quality image and the corresponding conditional constraint information; Obtain a pre-trained target conditional image generation model, wherein the target conditional image generation model includes a hierarchical sparse encoder and a multi-scale reversible generation network, the hierarchical sparse encoder includes convolutional units and sparse gating units, and the convolutional units contain structured sparse weights; Based on the convolutional unit and the structured sparse weights, the initial low-quality image and the conditional constraint information are jointly convolutionally mapped to obtain the initial image features; Based on the sparse gating unit, the initial image features are subjected to gating modulation processing to obtain sparse conditional image features; Based on the multi-scale reversible generative network, cross-level scale routing is performed on the sparse conditional image features to obtain multi-scale conditional routed image features. Based on the multi-scale reversible generative network and the multi-scale conditional routing image features, a reversible affine transformation is performed on the preset initial random noise features to obtain a high-quality target image.
2. The conditional image generation method as described in claim 1, characterized in that, The target conditional image generation model is obtained through the following training steps, which include: Obtain an initial conditional image generation model, wherein the initial conditional image generation model includes an initial convolutional unit, and the initial convolutional unit includes initial convolutional weights; Based on the pre-acquired training sample images and the training constraint information corresponding to the training sample images, the initial convolution weights are updated using a preset optimizer to obtain the updated convolution weights. The updated convolution weights are dynamically thresholded based on a preset lottery pruning controller to obtain the pruning threshold. The difference between the updated convolutional weights and the pruning threshold is binarized based on the lottery pruning controller to obtain a binary pruning mask. The structured sparse weights are obtained by multiplying the binary pruning mask and the updated convolution weights element by matrix element. The initial conditional image generation model is replaced with parameters based on the structured sparse weights to obtain the target conditional image generation model.
3. The conditional image generation method as described in claim 2, characterized in that, The training sample images include low-quality sample images and high-quality sample images corresponding to the low-quality sample images; The step of updating the initial convolutional weights by using a preset optimizer to update the gradient of the initial convolutional weights based on the pre-acquired training sample images and the corresponding training constraint information, includes: Based on the initial conditional image generation model, the sample low-quality image and the training constraint information are used to perform forward prediction to obtain the reconstructed image; Loss calculations are performed on the reconstructed image and the high-quality sample image to obtain image loss data; The updated convolution weights are obtained by performing inverse gradient calculation on the initial convolution weights based on the image loss data.
4. The conditional image generation method as described in claim 2, characterized in that, The preset lottery pruning controller performs dynamic threshold calculations on the updated convolutional weights to obtain the pruning threshold, including: Based on the lottery pruning controller, the updated convolution weights are sorted by absolute value to obtain the weight sorting distribution; Based on a preset sparsity ratio parameter, the weighted sorting distribution is dynamically truncated to obtain the pruning threshold.
5. The conditional image generation method as described in claim 2, characterized in that, The step of performing binarization selection on the difference between the updated convolutional weights and the pruning threshold based on the lottery pruning controller to obtain a binary pruning mask includes: Based on the lottery pruning controller, the difference between the updated convolution weights and the pruning threshold is continuously and smoothly scaled to obtain the real-number domain smooth probability. The binary pruning mask is obtained by deterministically thresholding the smoothed probability in the real number field.
6. The conditional image generation method according to any one of claims 1-5, characterized in that, The process of performing gated modulation processing on the initial image features based on the sparse gating unit to obtain sparse conditional image features includes: Based on the sparse gating unit, channel global average pooling is performed on the initial image features to obtain channel description image features; The channel description image features are subjected to nonlinear activation mapping to obtain the channel activation probability; The initial image features are multiplied element-wise based on the channel activation probabilities to obtain the sparse conditional image features.
7. The conditional image generation method according to any one of claims 1-5, characterized in that, The method of performing cross-scale routing on the sparse conditional image features based on the multi-scale reversible generative network to obtain multi-scale conditional routed image features includes: Based on the multi-scale reversible generative network, stride convolution is performed on the sparse conditional image features to obtain a multi-scale image feature pyramid. Based on the multi-scale reversible generative network, feature projection is performed on the multi-scale image feature pyramid to obtain the multi-scale conditional routing image features.
8. A conditional image generation apparatus, characterized in that, include: The data acquisition module is used to acquire an initial low-quality image and the condition constraint information corresponding to the initial low-quality image; The model loading module is used to obtain a pre-trained target conditional image generation model, wherein the target conditional image generation model includes a hierarchical sparse encoder and a multi-scale reversible generative network, the hierarchical sparse encoder includes convolutional units and sparse gating units, and the convolutional units contain structured sparse weights. The convolutional mapping module is used to perform joint convolutional mapping on the initial low-quality image and the conditional constraint information based on the convolutional unit and the structured sparse weights to obtain the initial image features; The gating processing module is used to perform gating modulation processing on the initial image features based on the sparse gating unit to obtain sparse conditional image features; The scale routing module is used to perform cross-level scale routing on the sparse conditional image features based on the multi-scale reversible generative network to obtain multi-scale conditional routed image features. The reversible transformation module is used to perform a reversible affine transformation on the preset initial random noise features based on the multi-scale reversible generation network and the multi-scale conditional routing image features to obtain a high-quality target image.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the conditional image generation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the conditional image generation method as described in any one of claims 1 to 7.
